Metric Update Procedure
Please follow the steps below to update the following Google Sheets:
The sheets are typically updated on Monday mornings (North America ET/PT) to ensure the values are available for team check-in meetings. The process of updating all three Google Sheets can take up to 30 minutes.
Prerequisites
Before updating the test metrics, ensure that:
- You are on the latest version of the
main
branch - Your
config.ini
file in the root directory is up to date - You have the latest raw data in the ecosystem-test-scripts root directory.
- The raw data should be found in the
test_result_dir
specified in the config.ini file and is typically namedraw_data
. - The latest raw data is available in the ETE team folder
- The raw data should be found in the
1. Scrape for New Raw Test Data
To retrieve the latest test and coverage results for local parsing, execute the following command from the ecosystem-test-scripts root directory:
make run_circleci_scraper
Notes:
- Set the
days_of_data
option in the config.ini file to the appropriate number of days. This is typically8
days since the update cadence is weekly on Mondays.
2. Create new CSV reports
To generate CSV reports with the latest test results, test averages, and test coverages, execute the following command from the ecosystem-test-scripts root directory:
make run_metric_reporter
Notes:
- The reports will be output to the
reports_dir
specified in the config.ini file. Typically, this is a reports directory in the ecosystem-test-scripts root. - Average reports are produced only after 90 days of data is available. Therefore, some test suites may not have these reports.
- Coverage reports are produced only for Autopush-rs unit tests and Merino-py unit and integration tests.
3. Import CSVs to Google Spreadsheets
To update the latest graphs and values for development teams, import each report in the
reports_dir
directory to the corresponding Google Sheet. To import a CSV file:
3.1 Select the corresponding sheet in Autopush-rs Metrics, FxA Metrics or Merino-py Metrics
3.2 Import the CSV
- In the top menu bar, select
File > Import
, a file selection dialogue should appear - Select the
Upload
tab and clickBrowse
, a local file selection dialogue should appear - Select the corresponding csv file from the
reports_dir
, anImport file
dialogue should appear - For
Import location
, selectReplace current sheet
, forSeparator type,
selectDetect automatically
, leave theConvert text to numbers, dates, and formulas
box checked, and clickImport data.
The Google Sheet should now reflect the latest data.
3.3 If the report being imported is a results or coverage report. Convert the type of the date column to 'plain text' so that the graphs display at an even cadence.
- Highlight the
Date
column and in the top menu par selectFormat > Number > Plain text
4. Backup the latest test_result_dir
to the ETE team folder
Compress the contents of the test_result_dir
, typically called 'raw_data,' and replace the file
located in the ETE team folder.