TestGrid: exploring in-depth metadata#

Note: This notebook can be run and experimented with directly using the “ocp-ci-analysis” image at our jupyterhub instance hosted on the MOC.

This notebook explores metadata present at a Test level within the existing TestGrid data at testgrid.k8s.io.

For initial exploration of the Testgrid data visit testgrid_EDA.ipynb.

import requests
import datetime as dt

from IPython.display import Image

What is Testgrid? According to the project’s readme it is a, “highly configurable, interactive dashboard for viewing your test results in a grid!” In other words, its an aggregation and visualization platform for CI data. Our hope is that this aggregation encodes some of the subject matter experts’ knowledge, and will provide better initial features than going straight to the more complex underlying CI data artifacts here.

Lets look into the various metadata fields present on a test level within each job for the testgrid dataset.

To pull the desired json we can follow this syntax:

https://testgrid.k8s.io/{dashboard_name}/table?&show-stale-tests=&tab={job_name}

Here we will get the data for 2 dashboard and jobs for further exploration as examples.

  1. dashboard : “redhat-openshift-ocp-release-4.2-informing” , job : “release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2”

  2. dashboard : “jetstack-cert-manager-master” , job : “ci-cert-manager-bazel”

openshift_testgrid = requests.get(
    "https://testgrid.k8s.io/redhat-openshift-ocp-release-4.2-informing/table? \
    &show-stale-tests=&tab=release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2"
)

jetstack_testgrid = requests.get(
    "https://testgrid.k8s.io/jetstack-cert-manager-master/table? \
    &show-stale-tests=&tab=ci-cert-manager-bazel"
)

openshift_testgrid has all the fields expected by TestGrid’s javascript client. It represents a grid of test results, with additional annotations for metadata.

The first thing we want to do is see what what keys we have in our detailed job data. Below we can see that there are a number of fields in this json.

for i in openshift_testgrid.json().keys():
    print(i)

test-group-name
query
status
phase-timer
cached
summary
bugs
changelists
column_ids
custom-columns
column-header-names
groups
metrics
tests
row_ids
timestamps
clusters
test_id_map
test-metadata
stale-test-threshold
num-stale-tests
add-tabular-names-option
show-tabular-names
description
bug-component
code-search-path
open-test-template
file-bug-template
attach-bug-template
results-url-template
code-search-url-template
about-dashboard-url
open-bug-template
context-menu-template
results-text
latest-green
triage-enabled
notifications
overall-status

We will now explore each metadata within the json and try to understand the significance of the values it contains

The summary field is always empty, hence we will ignore that here.

test-group-name#

print(openshift_testgrid.json()["test-group-name"])

releaseopenshiftorigininstallere2eawsupgraderollback41to42

As we can see above, thetest-group-name is a string value which contains the particular job name for which all the tests are executed.

query#

print(openshift_testgrid.json()["query"])

origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2

The above query is a string value which contains the gcs_prefix pertaining to the job. The gcs_prefix is the path to the test data stored in gcs (some-bucket/some/optional/path). The metadata can also be seen here and is documented here.

status#

print(openshift_testgrid.json()["status"])

Served 1174 results in 0.27 seconds

The status aggregates the tests for which a result - Pass, Fail, Flaky, Running was displayed and the time taken to serve the status. The time taken is calculated by aggregating times for different phases, which are further broken down in phase-timer. The status line is displayed on the Status bar for the job.

phase-timer#

print(openshift_testgrid.json()["phase-timer"])

{'phases': ['config load', 'cache peek (stale)', 'state fetch', 'state decompress', 'state unmarshal', 'table creation', 'table serialize (gob+snappy: 13KiB)', 'table querying'], 'delta': [2.5623e-05, 0.06963295, 0.17168032, 0.000316809, 0.000535445, 0.000140227, 0.000774511, 0.025254081], 'total': 0.268359966}

The phase-timer is a dictionary that contains the duration of each update cycle and stores the time taken for different phases. The aggregate time is used to populate the status displayed on the status bar of the job.

cached#

print(openshift_testgrid.json()["cached"])

False

cached holds a True or False value to determine if the response is loaded from a cache.

bugs#

print(openshift_testgrid.json()["bugs"])

{}

bugs is an empty dictionary here, but it should contain any linked bugs created for the job

changelists#

print(openshift_testgrid.json()["changelists"])

['1372535122330390528', '1372353507234942976', '1372171845851156480', '1371990292286148608', '1371808757339656192', '1371627211211149312', '1371445677782994944', '1371264283366658048', '1371082838371209216', '1370901281094242304', '1370719715751628800', '1370538213357129728', '1370358921604108288', '1370302741443776512', '1369994168893444096', '1369811862551531520', '1369630643398381568', '1369449096883998720', '1369267681374507008', '1369086139733905408', '1368904775671746560', '1368723410221273088', '1368542014391455744', '1368360542443409408', '1368179004518961152', '1367997486991413248', '1367816194010124288', '1367634626750713856', '1367453420490854400', '1367271859506122752']

changelists list consisting of IDs for each test within a job. The changelists can be used in a URL template to visit the Prow Dashboard after clicking on a cell such as https://prow.k8s.io/view/gcs/<gcs_prefix>/<changelists> for eg. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2/1365457253460611072

column_ids#

print(openshift_testgrid.json()["column_ids"])

['\ue0001372535122330390528', '\ue0001372353507234942976', '\ue0001372171845851156480', '\ue0001371990292286148608', '\ue0001371808757339656192', '\ue0001371627211211149312', '\ue0001371445677782994944', '\ue0001371264283366658048', '\ue0001371082838371209216', '\ue0001370901281094242304', '\ue0001370719715751628800', '\ue0001370538213357129728', '\ue0001370358921604108288', '\ue0001370302741443776512', '\ue0001369994168893444096', '\ue0001369811862551531520', '\ue0001369630643398381568', '\ue0001369449096883998720', '\ue0001369267681374507008', '\ue0001369086139733905408', '\ue0001368904775671746560', '\ue0001368723410221273088', '\ue0001368542014391455744', '\ue0001368360542443409408', '\ue0001368179004518961152', '\ue0001367997486991413248', '\ue0001367816194010124288', '\ue0001367634626750713856', '\ue0001367453420490854400', '\ue0001367271859506122752']

len(openshift_testgrid.json()["column_ids"])

30

print("\ue0001369630643398381568")

1369630643398381568

ord("\ue000")

57344

column_ids are timestamps from changelists preceded by a unicode character \ue000.

custom-columns#

print(openshift_testgrid.json()["custom-columns"])

[['1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', 'missing', '1', 'missing', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'], ['', '', '', '', '', '', '', '', '', '', '', '', 'missing', '', 'missing', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']]

print(jetstack_testgrid.json()["custom-columns"])

[['442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '442d7dbc0', '433cc2fa0', '433cc2fa0', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', 'fedea03a1', '03b3e2f20', '38ebe359a', '65a7a880f', '65a7a880f', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', '51340d0c8', 'f40ffc87c', 'f40ffc87c', 'f40ffc87c', 'f40ffc87c', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', '9f343ec58', 'c01603dd3', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', 'c2634d353', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '07964edea', '75a46ff90', 'deb55110a', 'deb55110a', 'deb55110a', 'deb55110a', 'deb55110a', 'deb55110a', 'deb55110a', 'deb55110a', 'deb55110a', '28fc97699', '28fc97699', '28fc97699', '28fc97699', 'eced645c8', 'eced645c8', 'eced645c8', 'eced645c8', 'eced645c8', 'eced645c8', 'eced645c8'], ['missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing', 'missing']]

For the first dashboard, custom-columns did not contain any values, but for the second dashboard, we see that custom-columns is nothing but a list of column headers with commit-ids for each test.

For example, as we see below ‘9f343ec58’ is common header and commit-id for the first tests.

Image(filename="../../../docs/assets/images/metadata-3.png")
../../../_images/testgrid_metadata_EDA_42_0.png 
len(openshift_testgrid.json()["custom-columns"][0])

30

column-header-names#

print(openshift_testgrid.json()["column-header-names"])

['Date', 'Time', 'Build Number', 'Commit', 'Infra-Commit']

This is not relevant to our current dashboards which we are looking at, since the values for these column headers are vacant as seen below.

Image(filename="../../../docs/assets/images/metadata-1.png")
../../../_images/testgrid_metadata_EDA_47_0.png

However, in other dashboards like https://testgrid.k8s.io/jetstack-cert-manager-master#ci-cert-manager-bazel, these consist of ‘Date’, ‘Time’, ‘Build Number’, ‘Commit’, ‘Infra-Commit’ for each Test.

Image(filename="../../../docs/assets/images/metadata-2.png")
../../../_images/testgrid_metadata_EDA_49_0.png

groups#

print(jetstack_testgrid.json()["groups"])

['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

Not sure what this means

metrics#

print(jetstack_testgrid.json()["metrics"])

['test-duration-minutes']

metrics is a list consisting of string labels for viewing additional metrics such as test-duration-minutes

Image(filename="../../../docs/assets/images/metadata-4.png")
../../../_images/testgrid_metadata_EDA_56_0.png

row_ids#

print(openshift_testgrid.json()["row_ids"])

['Overall', 'Operator results.operator conditions authentication', 'Operator results.operator conditions cloud-credential', 'Operator results.operator conditions monitoring', 'Operator results.operator conditions network', 'Operator results.operator conditions openshift-samples', 'openshift-tests.Monitor cluster while tests execute', 'openshift-tests.[Disruptive] Cluster upgrade should maintain a functioning cluster [Feature:ClusterUpgrade] [Suite:openshift] [Serial]', 'operator.Run template e2e-aws-upgrade - e2e-aws-upgrade container test', 'Operator results.operator conditions dns', 'Operator results.operator conditions operator-lifecycle-manager-packageserver', 'Operator results.operator conditions console', 'Operator results.operator conditions service-ca', 'Operator results.operator conditions image-registry', 'Pod', 'Operator results.operator conditions machine-config', 'Operator results.operator conditions cluster-autoscaler', 'Operator results.operator conditions ingress', 'Operator results.operator conditions insights', 'Operator results.operator conditions kube-apiserver', 'Operator results.operator conditions kube-controller-manager', 'Operator results.operator conditions kube-scheduler', 'Operator results.operator conditions machine-api', 'Operator results.operator conditions marketplace', 'Operator results.operator conditions node-tuning', 'Operator results.operator conditions openshift-apiserver', 'Operator results.operator conditions openshift-controller-manager', 'Operator results.operator conditions operator-lifecycle-manager', 'Operator results.operator conditions operator-lifecycle-manager-catalog', 'Operator results.operator conditions service-catalog-apiserver', 'Operator results.operator conditions service-catalog-controller-manager', 'Operator results.operator conditions storage', 'Symptom Detection.Bug 1812261: iptables is segfaulting', 'Symptom Detection.Infrastructure - AWS simulate policy rate-limit', 'Symptom Detection.Infrastructure - GCP quota exceeded (route to forum-gcp)', 'Symptom Detection.Node process segfaulted', 'Symptom Detection.Undiagnosed panic detected in pod', 'operator.All images are built and tagged into stable', 'operator.Find all of the input images from ocp/4.1:${component} and tag them into the output image stream', 'operator.Import the release payload "initial" from an external source', 'operator.Import the release payload "latest" from an external source', 'operator.Run template e2e-aws-upgrade - e2e-aws-upgrade container setup', 'operator.Run template e2e-aws-upgrade - e2e-aws-upgrade container teardown']

row_ids is a list consisting of all the various tests executed for the job release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2. Each job also has an Overall row which summarises all the test runs for the job.

len(openshift_testgrid.json()["row_ids"])

43

We see that as of right now, 43 tests were run for the job release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2 in the dashboard redhat-openshift-ocp-release-4.2-informing

print(jetstack_testgrid.json()["row_ids"])

['Pod', 'Overall', '//cmd/ctl/pkg/create/certificaterequest:go_default_test', '//cmd/ctl/pkg/inspect/secret:go_default_test', '//cmd/ctl/pkg/renew:go_default_test', '//cmd/ctl/pkg/status/certificate:go_default_test', '//hack:verify-bazel', '//hack:verify-boilerplate', '//hack:verify-codegen', '//hack:verify-crds', '//hack:verify-errexit', '//hack:verify-gofmt', '//pkg/acme/accounts:go_default_test', '//pkg/acme/util:go_default_test', '//pkg/api/util:go_default_test', '//pkg/controller/acmechallenges/scheduler:go_default_test', '//pkg/controller/acmechallenges:go_default_test', '//pkg/controller/acmeorders/selectors:go_default_test', '//pkg/controller/acmeorders:go_default_test', '//pkg/controller/certificaterequests/acme:go_default_test', '//pkg/controller/certificaterequests/approver:go_default_test', '//pkg/controller/certificaterequests/ca:go_default_test', '//pkg/controller/certificaterequests/selfsigned:go_default_test', '//pkg/controller/certificaterequests/util:go_default_test', '//pkg/controller/certificaterequests/vault:go_default_test', '//pkg/controller/certificaterequests/venafi:go_default_test', '//pkg/controller/certificaterequests:go_default_test', '//pkg/controller/certificates/internal/secretsmanager:go_default_test', '//pkg/controller/certificates/issuing:go_default_test', '//pkg/controller/certificates/keymanager:go_default_test', '//pkg/controller/certificates/readiness:go_default_test', '//pkg/controller/certificates/requestmanager:go_default_test', '//pkg/controller/certificates/trigger/policies:go_default_test', '//pkg/controller/certificates/trigger:go_default_test', '//pkg/controller/certificates:go_default_test', '//pkg/controller/clusterissuers:go_default_test', '//pkg/controller/ingress-shim:go_default_test', '//pkg/controller/issuers:go_default_test', '//pkg/internal/api/mutation:go_default_test', '//pkg/internal/api/validation:go_default_test', '//pkg/internal/apis/acme/install:go_default_test', '//pkg/internal/apis/acme/validation:go_default_test', '//pkg/internal/apis/certmanager/identity/certificaterequests:go_default_test', '//pkg/internal/apis/certmanager/install:go_default_test', '//pkg/internal/apis/certmanager/validation/util:go_default_test', '//pkg/internal/apis/certmanager/validation:go_default_test', '//pkg/internal/apis/meta/install:go_default_test', '//pkg/internal/vault:go_default_test', '//pkg/issuer/acme/dns/acmedns:go_default_test', '//pkg/issuer/acme/dns/akamai:go_default_test', '//pkg/issuer/acme/dns/azuredns:go_default_test', '//pkg/issuer/acme/dns/clouddns:go_default_test', '//pkg/issuer/acme/dns/cloudflare:go_default_test', '//pkg/issuer/acme/dns/digitalocean:go_default_test', '//pkg/issuer/acme/dns/rfc2136:go_default_test', '//pkg/issuer/acme/dns/route53:go_default_test', '//pkg/issuer/acme/dns/util:go_default_test', '//pkg/issuer/acme/dns:go_default_test', '//pkg/issuer/acme/http:go_default_test', '//pkg/issuer/venafi/client:go_default_test', '//pkg/issuer/venafi:go_default_test', '//pkg/issuer:go_default_test', '//pkg/metrics:go_default_test', '//pkg/scheduler:go_default_test', '//pkg/util/pki:go_default_test', '//pkg/util/predicate:go_default_test', '//pkg/util:go_default_test', '//pkg/webhook/authority:go_default_test', '//pkg/webhook/handlers/testdata/apis/testgroup/install:go_default_test', '//pkg/webhook/handlers/testdata/apis/testgroup/validation:go_default_test', '//pkg/webhook/handlers:go_default_test', '//pkg/webhook/server/tls:go_default_test', '//pkg/webhook/server:go_default_test', '//test/integration/certificates:go_default_test', '//test/integration/conversion:go_default_test', '//test/integration/ctl:go_default_test', '//test/integration/validation:go_default_test', '//test/integration/webhook:go_default_test', '//tools/cobra:go_default_test']

len(jetstack_testgrid.json()["row_ids"])

79

Similary, we see the various tests executed for the job ci-cert-manager-bazel in dashboard jetstack-cert-manager-master.

timestamps#

print(openshift_testgrid.json()["timestamps"])

[1616072850000, 1616029551000, 1615986240000, 1615942953000, 1615899672000, 1615856389000, 1615813107000, 1615769859000, 1615726599000, 1615683312000, 1615640024000, 1615596750000, 1615554026000, 1615540610000, 1615467683000, 1615423576000, 1615380370000, 1615337085000, 1615293832000, 1615250550000, 1615207308000, 1615164068000, 1615120820000, 1615077555000, 1615034271000, 1614990999000, 1614947770000, 1614904482000, 1614861279000, 1614817991000]

timestamps returns the timestamp of when the test runs were executed.

len(openshift_testgrid.json()["timestamps"])

30

t1 = openshift_testgrid.json()["timestamps"]
dates = [dt.datetime.fromtimestamp(item / 1000).strftime("%d-%m-%y") for item in t1]
dates

['18-03-21',
 '18-03-21',
 '17-03-21',
 '17-03-21',
 '16-03-21',
 '16-03-21',
 '15-03-21',
 '15-03-21',
 '14-03-21',
 '14-03-21',
 '13-03-21',
 '13-03-21',
 '12-03-21',
 '12-03-21',
 '11-03-21',
 '11-03-21',
 '10-03-21',
 '10-03-21',
 '09-03-21',
 '09-03-21',
 '08-03-21',
 '08-03-21',
 '07-03-21',
 '07-03-21',
 '06-03-21',
 '06-03-21',
 '05-03-21',
 '05-03-21',
 '04-03-21',
 '04-03-21']

We can convert them to datetime format for further analysis.

print(jetstack_testgrid.json()["timestamps"])

[1616097568000, 1616090308000, 1616083050000, 1616075827000, 1616068570000, 1616061307000, 1616054049000, 1616046787000, 1616039527000, 1616032268000, 1616025068000, 1616017807000, 1616010576000, 1616003321000, 1615996053000, 1615988868000, 1615981605000, 1615974348000, 1615967085000, 1615959825000, 1615952564000, 1615945305000, 1615938046000, 1615930784000, 1615923526000, 1615916264000, 1615909010000, 1615901797000, 1615894537000, 1615887337000, 1615880137000, 1615872938000, 1615865738000, 1615858540000, 1615851338000, 1615844077000, 1615836878000, 1615829617000, 1615822359000, 1615815127000, 1615807930000, 1615800668000, 1615793866000, 1615786147000, 1615778888000, 1615771629000, 1615764369000, 1615757109000, 1615749850000, 1615742665000, 1615735466000, 1615728265000, 1615721004000, 1615713745000, 1615706485000, 1615699225000, 1615691964000, 1615684704000, 1615677444000, 1615670184000, 1615662924000, 1615655660000, 1615648448000, 1615641189000, 1615633928000, 1615626668000, 1615619409000, 1615612149000, 1615604888000, 1615597628000, 1615590369000, 1615583108000, 1615575848000, 1615568632000, 1615561428000, 1615554229000, 1615547027000, 1615539827000, 1615532568000, 1615525368000, 1615518107000, 1615510847000, 1615503587000, 1615496327000, 1615489068000, 1615481837000, 1615474638000, 1615467437000, 1615460238000, 1615453038000, 1615438637000, 1615431437000, 1615424177000, 1615359061000, 1615351801000, 1615344541000, 1615337283000, 1615330024000, 1615322760000, 1615315502000, 1615308246000, 1615300982000, 1615293730000, 1615286578000, 1615279310000, 1615272050000, 1615264852000, 1615257591000, 1615250331000, 1615243131000, 1615235932000, 1615228732000, 1615221531000, 1615214272000, 1615207072000, 1615199810000, 1615192620000, 1615185361000, 1615178103000, 1615170840000, 1615163580000, 1615156321000, 1615149060000, 1615141800000, 1615134572000, 1615127312000, 1615120051000, 1615112789000, 1615105569000, 1615098308000, 1615091050000, 1615083792000, 1615076528000, 1615069269000, 1615062029000, 1615054771000, 1615047514000, 1615040251000, 1615032990000, 1615025731000, 1615018470000, 1615011211000, 1615003951000, 1614996691000, 1614989431000, 1614982172000, 1614974924000, 1614967665000, 1614960398000, 1614953138000, 1614945878000, 1614938621000, 1614931358000, 1614924097000, 1614916838000, 1614909578000, 1614902318000, 1614895059000, 1614887798000, 1614880537000, 1614873315000, 1614866057000, 1614858866000, 1614851658000, 1614844455000, 1614837262000, 1614830002000, 1614822741000, 1614815482000, 1614808225000]

len(jetstack_testgrid.json()["timestamps"])

170

clusters#

print(openshift_testgrid.json()["clusters"])

[]

print(jetstack_testgrid.json()["clusters"])

[]

Not very sure what clusters should return. According to: https://github.com/GoogleCloudPlatform/testgrid/blob/ce1b1405fc889f371fa047ac03adfab23447c804/README.md#clustered-failures, clusters should probably return the clustered failure tests. However, this does not seem to be the case since for release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2 in the dashboard redhat-openshift-ocp-release-4.2-informing, we see that when we filter the dashboard tab to “Display Clustered Failure Tests” it returns the following:

Image(filename="../../../docs/assets/images/testgrid_02.png")
../../../_images/testgrid_metadata_EDA_77_0.png

test_id_map#

print(openshift_testgrid.json()["test_id_map"])

None

print(jetstack_testgrid.json()["test_id_map"])

None

This seems to be a lookup map for shortening long test IDs in each row: https://github.com/GoogleCloudPlatform/testgrid/blob/master/pb/response/types.proto#L133

test-metadata#

print(openshift_testgrid.json()["test-metadata"])

{}

print(jetstack_testgrid.json()["test-metadata"])

{}

stale-test-threshold#

print(openshift_testgrid.json()["stale-test-threshold"])

10

print(jetstack_testgrid.json()["stale-test-threshold"])

10

stale-test-threshold probably means the number of times a test is not run for a particular job, before it gets labeled as “stale” and hidden in the UI. For both the jobs in this example, this threshold is 10.

num-stale-tests#

print(openshift_testgrid.json()["num-stale-tests"])

0

We see that for jobrelease-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2 in the dashboard redhat-openshift-ocp-release-4.2-informing, the number of stale tests is 0.

print(jetstack_testgrid.json()["num-stale-tests"])

0

Similary, for the job ci-cert-manager-bazel in dashboard jetstack-cert-manager-master, the number of stale tests were 0.

add-tabular-name-option#

print(openshift_testgrid.json()["add-tabular-names-option"])

False

print(jetstack_testgrid.json()["add-tabular-names-option"])

False

add-tabular-names-option seems to return a True/False value, indicating whether additional tabular names can be added.

show-tabular-names#

print(openshift_testgrid.json()["show-tabular-names"])

False

print(jetstack_testgrid.json()["show-tabular-names"])

False

show-tabular-names also returns a True/False value, but doesn’t seem too relevant for our analysis.

description#

print(openshift_testgrid.json()["description"])



print(jetstack_testgrid.json()["description"])

Runs 'bazel test //...'

The description field seems to be a general description of the job. However, most of them seem to return empty values.

bug-component#

print(openshift_testgrid.json()["bug-component"])

0

print(jetstack_testgrid.json()["bug-component"])

0

Not too sure what this should return.

code-search-path#

print(openshift_testgrid.json()["code-search-path"])

https://github.com/openshift/origin/search

print(jetstack_testgrid.json()["code-search-path"])

github.com/kubernetes/kubernetes/search

The code-search-path seems to return the relevant GitHub repository path related to the job and the test runs executed.

overall-status#

The overall-status field returns a status code describing the entire “job” (the individual tabs within a dashboard in TestGrid are jobs). The mapping and definition of the overall-status codes are defined here: https://github.com/GoogleCloudPlatform/testgrid/blob/1dbc0e8b6ef997b9c90b3fb79518c35ce4d27ea2/pb/summary/summary.proto#L163

  • 0 = NOT_SET

  • 1 = UNKNOWN

  • 2 = PASS

  • 3 = FAIL

  • 4 = FLAKY

  • 5 = STALE

  • 6 = BROKEN

print(openshift_testgrid.json()["overall-status"])

3

Hence, for the job release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2 in the dashboard redhat-openshift-ocp-release-4.2-informing, the overall status is 3 which is Fail.

print(jetstack_testgrid.json()["overall-status"])

3

Similarly, the overall status for the job ci-cert-manager-bazel in dashboard jetstack-cert-manager-master is 3, which attributes it as Fail.

What is interesting to note is that the overall-status field is different from the Overall row we see in the TestGrid dashboard. The overall-status defines the status on a “job” level, whereas the Overall defines the status on a “build” level.

For eg: in the dashboard redhat-openshift-ocp-release-4.2-informing, under the tab aka job release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2 there is an “Overall” row as highlighted below, which defines the overall status for each “build” executed in the job. Each build comprises of multiple tests and the overall status of the build is an aggregate of all its test statuses.

Image(filename="../../../docs/assets/images/testgrid_03.png")
../../../_images/testgrid_metadata_EDA_121_0.png

about-dashboard-url#

openshift_testgrid.json()["about-dashboard-url"]

''

jetstack_testgrid.json()["about-dashboard-url"]

''

Not entirely sure, but maybe this field shows the URL which describes this particular dashboard.

results-text#

print(openshift_testgrid.json()["results-text"])

See these results on Prow

This field usually just says “see results on prow”. If we manually go the testgrid website : https://testgrid.k8s.io/redhat-openshift-ocp-release-4.2-informing#release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2&show-stale-tests= and click on any test, we will be re-directed to the prow dashboard (https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-rollback-4.1-to-4.2/1371445677782994944) where we can find these results.

latest-green#

print(openshift_testgrid.json()["latest-green"])



print(jetstack_testgrid.json()["latest-green"])

Not too sure what this should be returning. Manually checked this for a few other dashboards/tabs, but all were empty.

triage-enabled#

print(openshift_testgrid.json()["triage-enabled"])

False

print(openshift_testgrid.json()["triage-enabled"])

False

In addition to the testgrid dashboard, Google has another dashboard called triage. This dashboard “shows clusters of similar test failures across all jobs”, and suggests what general subject these failures related to e.g. storage. So this the value, True or False, returned in this field probably tells whether triage is enabled or not for this particular dashboard/tab.

notifications#

print(openshift_testgrid.json()["notifications"])

[]

print(jetstack_testgrid.json()["notifications"])

[]

This value looks like should be returning the notifications if any. For various dashboards and jobs we looked at it however returned None

xyz-template-type fields#

These fields are instances of the LinkTemplate struct on the testgrid repo. These structs/jsons have three keys:

  1. url

  2. name

  3. options

Together, these keys describe the format of various URLs where additional information regarding the tests can be found, where bugs can be filed, etc.

open-test-template#

openshift_testgrid.json()["open-test-template"]

{'url': 'https://prow.ci.openshift.org/view/gcs/<gcs_prefix>/<changelist>',
 'name': '',
 'options': {}}

jetstack_testgrid.json()["open-test-template"]

{'url': 'https://prow.k8s.io/view/gs/<gcs_prefix>/<changelist>',
 'name': '',
 'options': {}}

This field seems to be the template for the URL where more details for a particular test can be found. That is, it’s the “general format” for the Prow page for each test.

file-bug-template#

openshift_testgrid.json()["file-bug-template"]

{'url': 'https://bugzilla.redhat.com/enter_bug.cgi',
 'name': '',
 'options': {'cf_environment': 'test: <test-name>',
  'cf_internal_whiteboard': 'buildcop',
  'classification': 'Red Hat',
  'comment': 'test: <test-name> failed, see job: <link>',
  'product': 'OpenShift Container Platform',
  'short_desc': 'test: <test-name>'}}

jetstack_testgrid.json()["file-bug-template"]

{'url': 'https://github.com/kubernetes/kubernetes/issues/new',
 'name': '',
 'options': {'body': '<test-url>', 'title': 'E2E: <test-name>'}}

This field seems to be the template for the URL where bugs can be filed for the respective organization/dashboard/job.

attach-bug-template#

openshift_testgrid.json()["attach-bug-template"]

{'url': '', 'name': '', 'options': {}}

jetstack_testgrid.json()["attach-bug-template"]

{'url': '', 'name': '', 'options': {}}

This is probably the template for URL where bugs can be attached. However, it’s empty for all of the instances we looked into.

results-url-template#

openshift_testgrid.json()["results-url-template"]

{'url': 'https://prow.ci.openshift.org/job-history/<gcs_prefix>',
 'name': '',
 'options': {}}

jetstack_testgrid.json()["results-url-template"]

{'url': 'https://prow.k8s.io/job-history/<gcs_prefix>',
 'name': '',
 'options': {}}

This field seems to be the template for the URL where historical results can be viewed. That is, the “Job History” tab on Prow for each test.

code-search-url-template#

openshift_testgrid.json()["code-search-url-template"]

{'url': 'https://github.com/openshift/origin/compare/<start-custom-0>...<end-custom-0>',
 'name': '',
 'options': {}}

jetstack_testgrid.json()["code-search-url-template"]

{'url': 'https://github.com/jetstack/cert-manager/compare/<start-custom-0>...<end-custom-0>',
 'name': '',
 'options': {}}

This field is the template for the URL where you can view diffs / changes across branches (basically to the git “Compare changes” page).

open-bug-template#

openshift_testgrid.json()["open-bug-template"]

{'url': 'https://github.com/openshift/origin/issues/',
 'name': '',
 'options': {}}

jetstack_testgrid.json()["open-bug-template"]

{'url': 'https://github.com/jetstack/cert-manager/issues/',
 'name': '',
 'options': {}}

This field seems to be the template for the URL that contains the link to issues page for the respective repository, however the name and options look empty.

context-menu-template#

openshift_testgrid.json()["context-menu-template"]

{'url': '', 'name': '', 'options': {}}

jetstack_testgrid.json()["context-menu-template"]

{'url': '', 'name': '', 'options': {}}

This field is mostly empty for all intances we looked into. So not sure what this URL is supposed to be.

Metadata within the statuses field#

print(openshift_testgrid.json()["tests"][0].keys())

dict_keys(['name', 'original-name', 'alert', 'linked_bugs', 'messages', 'short_texts', 'statuses', 'target', 'user_property'])

print((jetstack_testgrid.json()["tests"][0].keys()))

dict_keys(['name', 'original-name', 'alert', 'linked_bugs', 'messages', 'short_texts', 'statuses', 'target', 'user_property'])

linked_bugs#

print((openshift_testgrid.json()["tests"][0])["linked_bugs"])

[]

print((jetstack_testgrid.json()["tests"][0])["linked_bugs"])

[]

# is this non-empty for any other test in any other tabs?
print("Checking OpenShift dashboard")
for t in openshift_testgrid.json()["tests"]:
    if len(t["linked_bugs"]):
        print(t["linked_bugs"])

print("Checking non-OpenShift dashboard")
for t in jetstack_testgrid.json()["tests"]:
    if len(t["linked_bugs"]):
        print(t["linked_bugs"])

Checking OpenShift dashboard
Checking non-OpenShift dashboard

Linked bugs is empty.

messages#

print((openshift_testgrid.json()["tests"][0])["messages"])

['', '', '', '', '', '', '', '', '', '', '', '', 'Build failed outside of test results', '', 'Build failed outside of test results', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

print((jetstack_testgrid.json()["tests"][0])["messages"])

["podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter", "podinfo.json not found, please install prow's GCS reporter"]

len((jetstack_testgrid.json()["tests"][0])["messages"])

170

Messages contains messages for first few tests of openshift testgrid but good information for jetstack testgrid.

short_texts#

print((openshift_testgrid.json()["tests"][0])["short_texts"])

['', '', '', '', '', '', '', '', '', '', '', '', 'F', '', 'F', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

print((jetstack_testgrid.json()["tests"][0])["short_texts"])

['!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!']

print((jetstack_testgrid.json()["tests"][1])["short_texts"])

['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'F', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'F', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

Not sure what F or ! signify here. No documentation found about what this could signify.

user_property#

print((openshift_testgrid.json()["tests"][0])["user_property"])

None

print((jetstack_testgrid.json()["tests"][0])["user_property"])

None

# is this non-None for any other test in any other tabs?
print("Checking OpenShift dashboard")
for t in openshift_testgrid.json()["tests"]:
    if t["user_property"] is not None:
        print(t["user_property"])

print("Checking non-OpenShift dashboard")
for t in jetstack_testgrid.json()["tests"]:
    if t["user_property"] is not None:
        print(t["user_property"])

Checking OpenShift dashboard
Checking non-OpenShift dashboard

This field is empty for all tests in all the dashboards we looked at. Therefore we can’t determine exactly what this field means. Nonetheless, based off of the comment here, it seems that user_property is any custom, user-defined property that can be associated with a test result.

target#

print((openshift_testgrid.json()["tests"][0])["target"])

Overall

print((jetstack_testgrid.json()["tests"][0])["target"])

Pod

# is "target" same as "name"?
print("Checking OpenShift dashboard")
for t in openshift_testgrid.json()["tests"]:
    if t["target"] != t["name"]:
        print(t["target"])
        print(t["name"])

print("Checking non-OpenShift dashboard")
for t in jetstack_testgrid.json()["tests"]:
    if t["target"] != t["name"]:
        print(t["target"])
        print(t["name"])

Checking OpenShift dashboard
Checking non-OpenShift dashboard

This value in this field is the same as the value in name for all tests in all the dashboards we looked at. We couldn’t find any strong evidence in the testgrid repo that could explain the difference. Therefore we can’t determine exactly how this field is different than test name.

original-name#

print((openshift_testgrid.json()["tests"][0])["original-name"])

Overall

print((jetstack_testgrid.json()["tests"][0])["original-name"])

Pod

# is "original-name" same as "name"?
print("Checking OpenShift dashboard")
for t in openshift_testgrid.json()["tests"]:
    if t["original-name"] != t["name"]:
        print(t["original-name"])
        print(t["name"])

print("Checking non-OpenShift dashboard")
for t in jetstack_testgrid.json()["tests"]:
    if t["original-name"] != t["name"]:
        print(t["original-name"])
        print(t["name"])

Checking OpenShift dashboard
Checking non-OpenShift dashboard

This value in this field is the same as the value in name for all tests in all the dashboards we looked at. We couldn’t find any strong evidence in the testgrid repo that could explain the difference. Therefore we can’t determine exactly how this field is different than test name

Conclusion#

In this notebook, we explored all the metadata fields in the testgrid json data. From the results above, it seems that not all metadata is created equal and that some fields are more useful for data analysis than the others. Specifically, the following fields seem to have more valueable information:

  • query: The “gcs_prefix” value here can be used for contstructing prow URL for each test

  • changelists: The “ids” value here can be used for contstructing prow URL for each test

  • custom-column: Although not properly populated for OpenShift dashboards, in general the values here show the commit ids corresponding to each test run. These can later be used for other types of analysis such as determining what code resulted in passing tests vs failing test, etc.

  • metrics: The “test-duration-minutes” value here is useful

  • row_ids: This provides the list of test names. Maybe this can be used in the labelwise_encode function (instead of getting test names in a for loop, we can directly set test names to be the value in this field)

  • timestamps: Run timestamps

  • clusters: Seems useful because it provides similar tests, but it’s not heavily populated for OpenShift dashboards

  • num-stale-tests: Gives an idea of “health” of test runs (are all tests getting run? how many have not been run in a long time?)

  • stale-test-threshold: Might be useful, as it gives context for how severe of a job health indicator the number above is

  • code-search-path: Provides the GitHub link for diff’ing across PRs/commits

  • overall-status: Gives job summary

  • open-test-template: The template here can be used with query and changelists values to get relevant URLs

  • file-bug-template: The template here can be used with query and changelists values to get relevant URLs

  • open-bug-template: The template here can be used with query and changelists values to get relevant URLs

  • results-url-template: The template here can be used with query and changelists values to get relevant URLs

  • code-search-url-template: The template here can be used with query and changelists values to get relevant URLs

  • messages: Contains some error messages from running the test, but not very densely populated.

Lastly, the following fields seem to be either noise or empty or repetiions of data already included in other fields.

  • linked-bugs

    • NOTE: Maybe this could have been a very useful field, but it was found to be always empty.

  • bugs

    • NOTE: Maybe this could have been a very useful field, but it was found to be always empty.

  • cached

  • phase-timer

  • test-group-name

  • status

  • column_ids

  • column-header-names

  • groups

  • test-id-map

  • test-metadata

  • add-tabular-name-option

  • show-tabular-names

  • description

  • bug-component

  • about-dashboard-url

  • results-text

  • latest-green

  • triage-enabled

  • notifications

  • short_texts

  • attach-bug-template

  • context-menu-template

  • target

  • original-name

  • user_property