Project Onboarding Procedure

Below are step-by-step instructions on how to onboard a project to report test metrics.

Prerequisites

To report test metrics for a project, ensure the following requirements are met:

  • Test results must be in JUnit format.
  • Coverage results must be in JSON format.
  • Supported test frameworks are listed in the Metric Interpretation Guide.

1. Setup CICD to push test result and coverage data to GCS

  • Create a directory for the repository in the ecosystem-test-eng-metrics GCP Cloud Bucket.
  • Set up a service account for the project with Storage Object Creator and Storage Object Viewer permissions. Store the credentials in the ETE 1Password vault.
  • Update project CICD jobs to push Coverage JSON files and JUnit XML files to the GCS repository directory, under coverage and junit subdirectories respectively.
  • Coverage JSON files must follow a strict naming convention:
    {job_number}__{utc_epoch_datetime}__{repository}__{workflow}__{test_suite}__coverage.xml
    
    • Example: 15592__1724283071__autopush-rs__build-test-deploy__integration__coverage.json
  • JUnit XML files must follow a strict naming convention:
    {job_number}__{utc_epoch_datetime}__{repository}__{workflow}__{test_suite}__results{-index}.xml
    
    • Example: 15592__1724283071__autopush-rs__build-test-deploy__integration__results.xml
    • The index is optional and can be used in cases of parallel test execution

2. Create and Populate Tables in the ETE BigQuery Dataset

  • Create Tables:
    • For test results, create one empty table named {project_name}_results.
    • For coverage results, create one empty table named {project_name}_coverage.
    • These tables should be created in the test_metrics dataset of the ETE BigQuery instance. Reference the official documentation to create empty tables with schema definitions. Schemas can be copied from existing project tables.
  • Populate Tables:
    • Execute the following command to populate the tables with data:
      make run_metric_reporter
      

3. Create a Looker Dashboard

  • License Requirement: A developer license is required to create and edit dashboards in Looker. Instructions for obtaining a license and resources for learning Looker are available on the Mozilla Confluence. Additional help can be found in the #data-help and #looker-platform-discussion Slack channels.

  • Update Looker Project:

  • Create Dashboard: Create a new dashboard, populate it with looks for the new project data, and add it to the ETE Test Metrics Board.