Introduction

The ecosystem-test-scripts repository is maintained by Mozilla’s Ecosystem Test Engineering (ETE) Team to support their testing operations and the service teams they work with. It provides tools and documentation for managing tests in Mozilla services.

Command-Line Tools: Scripts for tasks such as metric collection and report updates.
Developer Guides: Instructions for configuring environments and contributing to the repository.
Reference Guides: Documentation for interpreting metrics and SOPs.

Command Line Tool

COMMANDS

check -- Run linting, formatting, security, and type checks.
clean -- Clean up installation and cache files.
format -- Apply formatting.
install -- Install dependencies.
run_circleci_scraper -- Run the CircleCI scraper.
run_google_sheet_uploader -- Run the Test Metric Reporter.
run_metric_reporter -- Run the Test Metric Reporter.
run_report_merger -- Run the Report Merger.
test -- Run tests.
test_coverage -- Run tests with coverage reporting.
test_coverage_html -- Run tests and generate HTML coverage report.

`install`

Install dependencies.

USAGE

make install

`clean`

Clean up installation and cache files.

USAGE

make clean

`check`

Run linting, formatting, security, and type checks.

This script uses the following tools:

ruff -- check for linting issues and code formatting.
bandit -- check for security issues.
mypy -- check for type issues.

USAGE

make check

`format`

Apply formatting.

This script will use ruff to automatically fix linting issues and format the code.

USAGE

make format

`test`

Run tests.

USAGE

make test

`test_coverage`

Run tests with coverage reporting.

USAGE

make test_coverage

`test_coverage_html`

Run tests and generate HTML coverage report.

USAGE

make test_coverage_html

`run_circleci_scraper`

Run the CircleCI scraper.

USAGE

make run_circleci_scraper

In order to use this command, you need to make sure you set your personal CircleCI token in your local config.ini file, as seen below:

[circleci_scraper]
token = <YoUr_tOkEn_hErE>

If the days_of_data option is not present in the config.ini file, the default of "all available data" will be fetched. If you want to customize how many previous days are fetched, you can set the days_of_data option in your local config.ini file:

[circleci_scraper]
;(optional) Get data starting from x days past from now (default: all available data)
days_of_data = 7

If you have the previous day's data stored locally, the cached data will be used and not re-fetched from CircleCI.

`run_google_sheet_uploader`

Run the Google Sheet Uploader.

USAGE

make run_google_sheet_uploader

`run_metric_reporter`

Run the Test Metric Reporter.

USAGE

make run_metric_reporter

`run_report_merger`

Run the Report Merger.

USAGE

make run_report_merger

Developer Setup

Below are step-by-step instructions on how to set up a development environment in order to be able to successfully contribute to and execute the ecosystem test scripts.

1. Clone the ecosystem-test-scripts repository

The ecosystem test scripts are hosted on the Mozilla Github and can be cloned using the method of your choice (see Github Cloning A Repository). Contributors should follow the Contributing Guidelines and Community Participation Guidelines for the repository.

2. Create a CircleCI API Token

In order to execute the circleci_scraper script, a personal CircleCI API Token is needed. To create a token, follow the creating a personal api token CircleCI instructions. Store the key value in a safe place.

DO NOT SHARE YOUR CIRCLECI API TOKEN

3. Copy the BigQuery and Google Sheet Service Account JSON Keys

The metric_reporter and google_sheet_uploader scripts are set up using the ecosystem-test-eng GCP project with the metric-reporter and metric-gsheet service accounts respectively. In order to execute these scripts, keys for these service account, in the form of JSON files, need to be copied from the 1Password Ecosystem Test Engineering Team Vault into the root directory of the ecosystem-test-scripts project.

4. Set up the config.ini

All settings for the ecosystem-test-scripts are defined in the config.ini file. To set up a local config.ini file:

4.1 Make a copy of the config.ini.sample file found in the root directory of the ecosystem-test-scripts project and rename it to config.ini
4.2 Under the [circleci_scraper] section of the file, set the token value to the CircleCI API key created in step 2

5. Copy the latest raw data locally

By default, CircleCI has a retention policy of 30 days for artifacts and 90 days for uploaded test results; However, we have over a years worth of data gathered for some projects. In order to produce reports with full trend data and reduce scraping time, copy the latest raw_data from the ETE team folder to the root directory of the ecosystem-test-scripts project.

6. Set up the python virtual environment

This project uses Poetry for dependency management in conjunction with a pyproject.toml file. While you can use virtualenv to set up the dev environment, it is recommended to use pyenv and pyenv-virtualenv, as they work nicely with Poetry. Once poetry is installed, dependencies can be installed using the following Make command from the root directory:

make install

For more information on Make commands, run:

make help

7. Start Developing!

Metric Interpretation Guide

The purpose of this document is to help individuals understand and interpret the metrics that represent the health of their test suites and ensure that tests contribute to rapid development and high product quality. Test metrics provide insights into test performance, helping teams address potential issues early and monitor improvement efforts. While these metrics offer valuable data about the health of test suites, they do not necessarily measure the effectiveness of the test cases themselves.

Test Suite Size & Success Rates

Supported Test Frameworks: jest, mocha, nextest, playwright, pytest, tap
Supported CI: CircleCI

Test Suite Size

Test suite size refers to the number of tests in a suite and serves as a control measure. Unexplained changes, such as sudden growth or shrinkage, may indicate test runner issues or attempts to manipulate the scope or quality of the suite. Test suite size should correspond to the state of the product. For example, a product under active development should show a gradual increase in the size of its test suite, while a product in maintenance should exhibit more stable trends.

Success Rates

Success rates provide a quick indication of the test suite's health. A low success rate or high failure rate signals potential quality issues, either in the product or within the test suite itself. This metric can be tracked on both a test-by-test basis and for the entire test suite.

Average Success Rates

To avoid noise from isolated failures and spot trends more easily, it's helpful to calculate averaged success rates over time. This allows teams to act early if trends toward failure begin to emerge, preventing the test suite from becoming unreliable or mistrusted. Success rate averages are calculated as:

100 x (Successful Runs / (All Runs - Cancelled Runs))

These averages can be calculated over 30-day, 60-day, and 90-day periods, with the 90-day trend being preferred. Average success rates are typically interpreted as follows, though teams may adjust thresholds based on their specific needs:

Threshold	Interpretation
>= 95%	Healthy - Tests pass the majority of the time
90% - 95%	Caution - Tests show signs of instability, requiring investigation
< 90%	Critical - Tests are faulty and need intervention

Time Measurements

Supported Test Frameworks: jest, mocha, nextest, playwright, pytest
Supported CI: CircleCI

Time measurements track how long it takes for tests to run. Ideally, these times should be proportional to the size of the test suite and remain stable over time. Significant increases or variations in execution time may indicate performance issues or inefficiencies within the test suite. Monitoring execution times allows teams to identify and address bottlenecks to keep test suites efficient.

Run Time

The cumulative time of all test runs in a suite.

Execution Time

The total time taken for the test suite to execute. If tests are not run in parallel, the execution time should match the run time.

Job Time

The time taken for the test job to complete in CI. Job time thresholds are typically interpreted as follows:

Threshold	Interpretation
> 10m	Slow - The test suite may require optimization
<= 10m	Fast - The test suite runs within an acceptable time frame

Coverage Metrics

Supported Coverage Frameworks: pytest-cov, llvm-cov

Coverage metrics measure the percentage of the codebase covered by tests. They help identify untested areas of the code, allowing teams to determine whether critical paths are adequately covered.

While high coverage percentages are generally good, they don’t always guarantee that the tests are meaningful. The quality and relevance of tests should be balanced with coverage goals. The following thresholds for line coverage provide general guidance but can be adjusted according to project needs:

Threshold	Interpretation
>= 95%	High - Potential for metric gaming or diminishing returns. * For teams using pytest-cov, the `line excluded` measure may offer further insight into coverage gaming.
80% - 95%	Good - Suitable for high-risk or high-incident projects
60% - 79%	Acceptable - Suitable for low-risk or low-incident projects
< 60%	Low - Coverage should be improved

Skip Rates

Supported Test Frameworks: jest, mocha, nextest, playwright, pytest
Supported CI: CircleCI

Skip rates indicate how often tests are temporarily excluded from execution. While skipping tests can be a necessary short-term solution to prevent flaky tests from disrupting workflows, high or sustained skip rates can signal deeper issues with the test suite's sustainability.

Long-term skips may indicate that tests have fallen into disrepair, and an increasing skip rate can point to team capacity or prioritization problems. Monitoring skip rates ensures that skipped tests are revisited and resolved promptly. Thresholds for skip rates are typically interpreted as follows:

Threshold	Interpretation
> 2%	Critical - Test coverage is compromised, requiring immediate intervention
1% - 2%	Caution - Test coverage is at risk, and the suite may become prone to silent failures
<= 1%	Healthy - Most of the test suite is running, ensuring comprehensive coverage

Note: Playwright offers both skipme and fixme annotations, allowing for further refinement of this metric.

Retry Rates

Supported Test Frameworks: playwright

Retry rates track how often tests are re-executed following a failure. While retries can help address transient issues, such as network errors, elevated retry rates may indicate flakiness in the test suite or performance regressions in the product. High retry rates can increase execution times and negatively impact developer workflows. Monitoring retry rates helps teams identify and fix unstable tests, ensuring predictable test execution.

Metric Update Procedure

Please follow the steps below to update the ETE Looker Dashboards. The dashboards are typically updated on Monday mornings (North America ET/PT) to ensure the values are available for team check-in meetings. The process of updating the dashboards can take up to 10 minutes.

Prerequisites

Before updating the test metrics, ensure that:

Your development environment is setup with proper permissions (see the Developer Setup).
You are on the latest version of the main branch
Your config.ini file in the root directory is up-to-date
You have the latest raw data in the ecosystem-test-scripts root directory.
- The raw data should be found in the test_result_dir specified in the config.ini file and is typically named raw_data.
- The latest raw data is available in the ETE team folder

1. Scrape for New Raw Test Data

To retrieve the latest test and coverage results for local parsing, execute the following command from the ecosystem-test-scripts root directory:

make run_circleci_scraper

Notes:

Set the days_of_data option in the config.ini file to the appropriate number of days. This is typically 8 days since the update cadence is weekly on Mondays.

2. Push updates to BigQuery

To push data to BigQuery and generate CSV reports with the latest test results, test averages, and test coverages, execute the following command from the ecosystem-test-scripts root directory:

make run_metric_reporter

Notes:

Average data is produced only after 90 days of data is available. Therefore, some test suites may not have these values.
Coverage results are produced only for Autopush-rs unit tests and Merino-py unit and integration tests.

3. Backup the latest `test_result_dir` to the ETE team folder

Compress the contents of the test_result_dir, typically called 'raw_data,' and replace the file located in the ETE team folder.

Project Onboarding Procedure

Below are step-by-step instructions on how to onboard a project to report test metrics.

Prerequisites

To report test metrics for a project, ensure the following requirements are met:

The project uses CircleCI for test execution, and members of the ETE team have access to the CircleCI pipelines.
Testing and coverage results are stored as artifacts in CircleCI jobs:
- Test results must be in JUnit format.
- Coverage results must be in JSON format.
- Supported test frameworks are listed in the Metric Interpretation Guide.

1. Add CircleCI Pipelines to the Config File and Scrape Historical Data

Add the new CircleCI pipelines under the circleci_scraper section in your local config.ini and config.ini.sample. Push changes to the config.ini.sample to the repository to keep contributors up-to-date.
Execute the circleci_scraper using the following command. Ensure the days_of_data option in the circleci_scraper section of config.ini is omitted to scrape all available historical data. Test and coverage data will be stored locally in the test_result_dir (typically called raw_data at the project root).
```
make run_circleci_scraper
```
Once scraping is complete, compress the contents of test_result_dir, and replace the corresponding file in the ETE team folder.

2. Create and Populate Tables in the ETE BigQuery Dataset

Create Tables:
- For test results, create two empty tables using the following naming conventions:
  - {project_name}_averages
  - {project_name}_results
- For coverage results, create one empty table named {project_name}_coverage.
- These tables should be created in the test_metrics dataset of the ETE BigQuery instance. Reference the official documentation to create empty tables with schema definitions. Schemas can be copied from existing project tables.
Populate Tables:
- Execute the following command to populate the tables with data:
```
make run_metric_reporter
```

3. Create a Looker Dashboard

License Requirement: A developer license is required to create and edit dashboards in Looker. Instructions for obtaining a license and resources for learning Looker are available on the Mozilla Confluence. Additional help can be found in the #data-help and #looker-platform-discussion Slack channels.
Update Looker Project:
- Update the ecosystem-test-eng Looker project Looker project model and add the required views for the new project test data. Related repository: looker-ecosystem-test-eng.
Create Dashboard: Create a new dashboard, populate it with looks for the new project data, and add it to the ETE Test Metrics Board.

Contributors

The ecosystem-test-scripts repository is owned by the Ecosystem Test Engineering Team.

See https://github.com/mozilla/ecosystem-test-scripts/blob/main/CONTRIBUTING.md for contribution guidelines.