The purpose of this document is to help individuals understand and interpret the metrics that represent the health of their test suites and ensure that tests contribute to rapid development and high product quality. Test metrics provide insights into test performance, helping teams address potential issues early and monitor improvement efforts. While these metrics offer valuable data about the health of test suites, they do not necessarily measure the effectiveness of the test cases themselves.

Test Suite Size & Success Rates

Supported Test Frameworks: jest, mocha, nextest, playwright, pytest, tap
Supported CI: CircleCI

Test Suite Size

Test suite size refers to the number of tests in a suite and serves as a control measure. Unexplained changes, such as sudden growth or shrinkage, may indicate test runner issues or attempts to manipulate the scope or quality of the suite. Test suite size should correspond to the state of the product. For example, a product under active development should show a gradual increase in the size of its test suite, while a product in maintenance should exhibit more stable trends.

Success Rates

Success rates provide a quick indication of the test suite's health. A low success rate or high failure rate signals potential quality issues, either in the product or within the test suite itself. This metric can be tracked on both a test-by-test basis and for the entire test suite.

Average Success Rates

To avoid noise from isolated failures and spot trends more easily, it's helpful to calculate averaged success rates over time. This allows teams to act early if trends toward failure begin to emerge, preventing the test suite from becoming unreliable or mistrusted. Success rate averages are calculated as:

100 x (Successful Runs / (All Runs - Cancelled Runs))

These averages can be calculated over 30-day, 60-day, and 90-day periods, with the 90-day trend being preferred. Average success rates are typically interpreted as follows, though teams may adjust thresholds based on their specific needs:

Threshold	Interpretation
>= 95%	Healthy - Tests pass the majority of the time
90% - 95%	Caution - Tests show signs of instability, requiring investigation
< 90%	Critical - Tests are faulty and need intervention

Time Measurements

Supported Test Frameworks: jest, mocha, nextest, playwright, pytest
Supported CI: CircleCI

Time measurements track how long it takes for tests to run. Ideally, these times should be proportional to the size of the test suite and remain stable over time. Significant increases or variations in execution time may indicate performance issues or inefficiencies within the test suite. Monitoring execution times allows teams to identify and address bottlenecks to keep test suites efficient.

Run Time

The cumulative time of all test runs in a suite.

Execution Time

The total time taken for the test suite to execute. If tests are not run in parallel, the execution time should match the run time.

Job Time

The time taken for the test job to complete in CI. Job time thresholds are typically interpreted as follows:

Threshold	Interpretation
> 10m	Slow - The test suite may require optimization
<= 10m	Fast - The test suite runs within an acceptable time frame

Coverage Metrics

Supported Coverage Frameworks: pytest-cov, llvm-cov

Coverage metrics measure the percentage of the codebase covered by tests. They help identify untested areas of the code, allowing teams to determine whether critical paths are adequately covered.

While high coverage percentages are generally good, they don’t always guarantee that the tests are meaningful. The quality and relevance of tests should be balanced with coverage goals. The following thresholds for line coverage provide general guidance but can be adjusted according to project needs:

Threshold	Interpretation
>= 95%	High - Potential for metric gaming or diminishing returns. * For teams using pytest-cov, the `line excluded` measure may offer further insight into coverage gaming.
80% - 95%	Good - Suitable for high-risk or high-incident projects
60% - 79%	Acceptable - Suitable for low-risk or low-incident projects
< 60%	Low - Coverage should be improved

Skip Rates

Supported Test Frameworks: jest, mocha, nextest, playwright, pytest
Supported CI: CircleCI

Skip rates indicate how often tests are temporarily excluded from execution. While skipping tests can be a necessary short-term solution to prevent flaky tests from disrupting workflows, high or sustained skip rates can signal deeper issues with the test suite's sustainability.

Long-term skips may indicate that tests have fallen into disrepair, and an increasing skip rate can point to team capacity or prioritization problems. Monitoring skip rates ensures that skipped tests are revisited and resolved promptly. Thresholds for skip rates are typically interpreted as follows:

Threshold	Interpretation
> 2%	Critical - Test coverage is compromised, requiring immediate intervention
1% - 2%	Caution - Test coverage is at risk, and the suite may become prone to silent failures
<= 1%	Healthy - Most of the test suite is running, ensuring comprehensive coverage

Note: Playwright offers both skipme and fixme annotations, allowing for further refinement of this metric.

Retry Rates

Supported Test Frameworks: playwright

Retry rates track how often tests are re-executed following a failure. While retries can help address transient issues, such as network errors, elevated retry rates may indicate flakiness in the test suite or performance regressions in the product. High retry rates can increase execution times and negatively impact developer workflows. Monitoring retry rates helps teams identify and fix unstable tests, ensuring predictable test execution.

Metric Update Procedure

Please follow the steps below to update the following Google Sheets:

The sheets are typically updated on Monday mornings (North America ET/PT) to ensure the values are available for team check-in meetings. The process of updating all three Google Sheets can take up to 30 minutes.

Prerequisites

Before updating the test metrics, ensure that:

You are on the latest version of the main branch
Your config.ini file in the root directory is up to date
You have the latest raw data in the ecosystem-test-scripts root directory.
- The raw data should be found in the test_result_dir specified in the config.ini file and is typically named raw_data.
- The latest raw data is available in the ETE team folder

1. Scrape for New Raw Test Data

To retrieve the latest test and coverage results for local parsing, execute the following command from the ecosystem-test-scripts root directory:

make run_circleci_scraper

Notes:

Set the days_of_data option in the config.ini file to the appropriate number of days. This is typically 8 days since the update cadence is weekly on Mondays.

2. Create new CSV reports

To generate CSV reports with the latest test results, test averages, and test coverages, execute the following command from the ecosystem-test-scripts root directory:

make run_metric_reporter

Notes:

The reports will be output to the reports_dir specified in the config.ini file. Typically, this is a reports directory in the ecosystem-test-scripts root.
Average reports are produced only after 90 days of data is available. Therefore, some test suites may not have these reports.
Coverage reports are produced only for Autopush-rs unit tests and Merino-py unit and integration tests.

3. Import CSVs to Google Spreadsheets

To update the latest graphs and values for development teams, import each report in the reports_dir directory to the corresponding Google Sheet. To import a CSV file:

3.1 Select the corresponding sheet in Autopush-rs Metrics, FxA Metrics or Merino-py Metrics

3.2 Import the CSV

In the top menu bar, select File > Import, a file selection dialogue should appear
Select the Upload tab and click Browse, a local file selection dialogue should appear
Select the corresponding csv file from the reports_dir, an Import file dialogue should appear
For Import location, select Replace current sheet, for Separator type, select Detect automatically, leave the Convert text to numbers, dates, and formulas box checked, and click Import data. The Google Sheet should now reflect the latest data.

3.3 If the report being imported is a results or coverage report. Convert the type of the date column to 'plain text' so that the graphs display at an even cadence.

Highlight the Date column and in the top menu par select Format > Number > Plain text

4. Backup the latest `test_result_dir` to the ETE team folder

Compress the contents of the test_result_dir, typically called 'raw_data,' and replace the file located in the ETE team folder.

Command Line Tool

REQUIREMENTS

Personal CircleCI API token
Python 3
Grit and determination

INSTALLATION

TODO

COMMANDS

check -- Run linting, formatting, security, and type checks.
clean -- Clean up installation and cache files.
format -- Apply formatting.
install -- Install dependencies.
run_circleci_scraper -- Run the CircleCI scraper.
run_metric_reporter -- Run the Test Metric Reporter.
test -- Run tests.
test_coverage -- Run tests with coverage reporting.
test_coverage_html -- Run tests and generate HTML coverage report.

`install`

Install dependencies.

USAGE

make install

`clean`

Clean up installation and cache files.

USAGE

make clean

`check`

Run linting, formatting, security, and type checks.

This script uses the following tools:

ruff -- check for linting issues and code formatting.
bandit -- check for security issues.
mypy -- check for type issues.

USAGE

make check

`format`

Apply formatting.

This script will use ruff to automatically fix linting issues and format the code.

USAGE

make format

`test`

Run tests.

USAGE

make test

`test_coverage`

Run tests with coverage reporting.

USAGE

make test_coverage

`test_coverage_html`

Run tests and generate HTML coverage report.

USAGE

make test_coverage_html

`run_circleci_scraper`

Run the CircleCI scraper.

USAGE

make run_circleci_scraper

In order to use this command, you need to make sure you set your personal CircleCI token in your local config.ini file, as seen below:

[circleci_scraper]
token = <YoUr_tOkEn_hErE>

If the days_of_data option is not present in the config.ini file, the default of "all available data" will be fetched. If you want to customize how many previous days are fetched, you can set the days_of_data option in your local config.ini file:

[circleci_scraper]
;(optional) Get data starting from x days past from now (default: all available data)
days_of_data = 7

If you have the previous day's data stored locally, the cached data will be used and not re-fetched from CircleCI.

`run_metric_reporter`

Run the Test Metric Reporter.

USAGE

make run_metric_reporter

Contributors

The ecosystem-test-scripts repository is owned by the Ecosystem Test Engineering Team.

See https://github.com/mozilla/ecosystem-test-scripts/blob/main/CONTRIBUTING.md for contribution guidelines.