Unit testing Glean metrics
In order to support unit testing inside of client applications using the Glean SDK, a set of testing API functions have been included. The intent is to make the Glean SDKs easier to test 'out of the box' in any client application it may be used in. These functions expose a way to inspect and validate recorded metric values within the client application. but are restricted to test code only. (Outside of a testing context, Glean APIs are otherwise write-only so that it can enforce semantics and constraints about data).
To encourage using the testing APIs, it is also possible to generate testing coverage reports to show which metrics in your project are tested.
Example of using the test API
In order to enable metrics testing APIs in each SDK, Glean must be reset and put in testing mode. For documentation on how to do that, refer to Initializing - Testing API.
Check out full examples of using the metric testing API on each Glean SDK. All examples omit the step of resetting Glean for tests to focus solely on metrics unit testing.
// Record a metric value with extra to validate against GleanMetrics.BrowserEngagement.click.record( BrowserEngagementExtras(font = "Courier") ) // Record more events without extras attached BrowserEngagement.click.record() BrowserEngagement.click.record() // Retrieve a snapshot of the recorded events val events = BrowserEngagement.click.testGetValue()!! // Check if we collected all 3 events in the snapshot assertEquals(3, events.size) // Check extra key/value for first event in the list assertEquals("Courier", events.elementAt(0).extra["font"])
// Record a metric value with extra to validate against GleanMetrics.BrowserEngagement.click.record([.font: "Courier"]) // Record more events without extras attached BrowserEngagement.click.record() BrowserEngagement.click.record() // Retrieve a snapshot of the recorded events let events = BrowserEngagement.click.testGetValue()! // Check if we collected all 3 events in the snapshot XCTAssertEqual(3, events.count) // Check extra key/value for first event in the list XCTAssertEqual("Courier", events.extra?["font"])
from glean import load_metrics metrics = load_metrics("metrics.yaml") # Record a metric value with extra to validate against metrics.url.visit.add(1) # Check if we collected any events into the 'click' metric assert metrics.url.visit.test_get_value() is not Null # Retrieve a snapshot of the recorded events assert 1 == metrics.url.visit.test_get_value()
Generating testing coverage reports
Glean can generate coverage reports to track which metrics are tested in your unit test suite.
There are three steps to integrate it into your continuous integration workflow: recording coverage, post-processing the results, and uploading the results.
Glean testing coverage is enabled by setting the
GLEAN_TEST_COVERAGE environment variable to the name of a file to store results.
It is good practice to set it to the absolute path to a file, since some testing harnesses (such as
cargo test) may change the current working directory.
GLEAN_TEST_COVERAGE=$(realpath glean_coverage.txt) make test
Post-processing the results
A post-processing step is required to convert the raw output in the file specified by
GLEAN_TEST_COVERAGE into usable output for coverage reporting tools. Currently, the only coverage reporting tool supported is codecov.io.
This post-processor is available in the
coverage subcommand in the
For some build systems,
glean_parser is already installed for you by the build system integration at the following locations:
- On Android/Gradle,
- On iOS,
- For other systems, install
pip install glean_parser
glean_parser coverage command requires the following parameters:
-f: The output format to produce, for example
codecovioto produce codecov.io's custom format.
-o: The path to the output file, for example
-c: The input raw coverage file.
glean_coverage.txtin the example above.
- A list of the
metrics.yamlfiles in your repository.
For example, to produce output for codecov.io:
glean_parser coverage -f codecovio -o glean_coverage.json -c glean_coverage.txt app/metrics.yaml
In this example, the
glean_coverage.json file is now ready for uploading to codecov.io.
codecov.io, the uploader doesn't send coverage results for YAML files by default. Pass the
-X yaml option to the uploader to make sure they are included:
bash <(curl -s https://codecov.io/bash) -X yaml