Unit testing Glean metrics

In order to support unit testing inside of client applications using the Glean SDK, a set of testing API functions have been included. The intent is to make the Glean SDK easier to test 'out of the box' in any client application it may be used in. These functions expose a way to inspect and validate recorded metric values within the client application but are restricted to test code only through visibility annotations (@VisibleForTesting(otherwise = VisibleForTesting.NONE) for Kotlin, internal methods for Swift). (Outside of a testing context, Glean APIs are otherwise write-only so that it can enforce semantics and constraints about data).

To encourage using the testing API, it is also possible to generate testing coverage reports to show which metrics in your project are tested.

General test API method semantics

Using the Glean SDK's unit testing API requires adding Robolectric 4.0 or later as a testing dependency. In Gradle, this can be done by declaring a testImplementation dependency:

dependencies {
    testImplementation "org.robolectric:robolectric:4.3.1" 
}

In order to prevent issues with async calls when unit testing the Glean SDK, it is important to put the Glean SDK into testing mode by applying the JUnit GleanTestRule to your test class. When the Glean SDK is in testing mode, it enables uploading and clears the recorded metrics at the beginning of each test run. The rule can be used as shown below:

@RunWith(AndroidJUnit4::class)
class ActivityCollectingDataTest {
    // Apply the GleanTestRule to set up a disposable Glean instance.
    // Please note that this clears the Glean data across tests.
    @get:Rule
    val gleanRule = GleanTestRule(ApplicationProvider.getApplicationContext())

    @Test
    fun checkCollectedData() {
      // The Glean SDK testing API can be called here.
    }
}

This will ensure that metrics are done recording when the other test functions are used.

To check if a value exists (i.e. it has been recorded), there is a testHasValue() function on each of the metric instances:

assertTrue(GleanMetrics.Search.defaultSearchEngineUrl.testHasValue())

To check the actual values, there is a testGetValue() function on each of the metric instances. It is important to check that the values are recorded as expected, since many of the metric types may truncate or error-correct the value. This function will return a datatype appropriate to the specific type of the metric it is being used with:

assertEquals("https://example.com/search?", GleanMetrics.Search.defaultSearchEngineUrl.testGetValue())

Note that each of these functions has its visibility limited to the scope of unit tests by making use of the @VisibleForTesting annotation, so the IDE should complain if you attempt to use them inside of client code.

NOTE: There's no automatic test rule for Glean tests implemented.

In order to prevent issues with async calls when unit testing the Glean SDK, it is important to put the Glean SDK into testing mode. When the Glean SDK is in testing mode, it enables uploading and clears the recorded metrics at the beginning of each test run.

Activate it by resetting Glean in your test's setup:

@testable import Glean
import XCTest

class GleanUsageTests: XCTestCase {
    override func setUp() {
        Glean.shared.resetGlean(clearStores: true)
    }

    // ...
}

This will ensure that metrics are done recording when the other test functions are used.

To check if a value exists (i.e. it has been recorded), there is a testHasValue() function on each of the metric instances:

XCTAssertTrue(GleanMetrics.Search.defaultSearchEngineUrl.testHasValue())

To check the actual values, there is a testGetValue() function on each of the metric instances. It is important to check that the values are recorded as expected, since many of the metric types may truncate or error-correct the value. This function will return a datatype appropriate to the specific type of the metric it is being used with:

XCTAssertEqual("https://example.com/search?", try GleanMetrics.Search.defaultSearchEngineUrl.testGetValue())

Note that each of these functions is marked as internal, you need to import Glean explicitly in test mode:

@testable import Glean

It is generally a good practice to "reset" the Glean SDK prior to every unit test that uses the Glean SDK, to prevent side effects of one unit test impacting others. The Glean SDK contains a helper function glean.testing.reset_glean() for this purpose. It has two required arguments: the application ID, and the application version. Each reset of the Glean SDK will create a new temporary directory for Glean to store its data in. This temporary directory is automatically cleaned up the next time the Glean SDK is reset or when the testing framework finishes.

The instructions below assume you are using pytest as the test runner. Other test-running libraries have similar features, but are different in the details.

Create a file conftest.py at the root of your test directory, and add the following to reset Glean at the start of every test in your suite:

import pytest
from glean import testing

@pytest.fixture(name="reset_glean", scope="function", autouse=True)
def fixture_reset_glean():
    testing.reset_glean(application_id="my-app-id", application_version="0.1.0")

To check if a value exists (i.e. it has been recorded), there is a test_has_value() function on each of the metric instances:

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# ...

assert metrics.search.search_engine_url.test_has_value()

To check the actual values, there is a test_get_value() function on each of the metric instances. It is important to check that the values are recorded as expected, since many of the metric types may truncate or error-correct the value. This function will return a datatype appropriate to the specific type of the metric it is being used with:

assert (
    "https://example.com/search?" ==
    metrics.search.default_search_engine_url.test_get_value()
)

TODO. To be implemented in bug 1648448.

Testing metrics for custom pings

In order to test metrics where the metric is included in more than one ping, the test functions take an optional pingName argument (ping_name in Python). This is the name of the ping that the metric is being sent in, such as "events" for the events ping, or "metrics" for the metrics ping. This could also be a custom ping name that the metric is being sent in. In most cases you should not have to supply the ping name to the test function and can just use the default which is the "default" ping that this metric is sent in. You should only need to provide a pingName if the metric is being sent in more than one ping in order to identify the correct metric store.

You can call the testHasValue() and testGetValue() functions with pingName like this:

GleanMetrics.Foo.uriCount.testHasValue("customPing")
GleanMetrics.Foo.uriCount.testGetValue("customPing")

Example of using the test API

Here is a longer example to better illustrate the intended use of the test API:

// Record a metric value with extra to validate against
GleanMetrics.BrowserEngagement.click.record(
    mapOf(
        BrowserEngagement.clickKeys.font to "Courier"
    )
)

// Record more events without extras attached
BrowserEngagement.click.record()
BrowserEngagement.click.record()

// Check if we collected any events into the 'click' metric
assertTrue(BrowserEngagement.click.testHasValue())

// Retrieve a snapshot of the recorded events
val events = BrowserEngagement.click.testGetValue()

// Check if we collected all 3 events in the snapshot
assertEquals(3, events.size)

// Check extra key/value for first event in the list
assertEquals("Courier", events.elementAt(0).extra["font"])

Here is a longer example to better illustrate the intended use of the test API:

// Record a metric value with extra to validate against
GleanMetrics.BrowserEngagement.click.record([.font: "Courier"])

// Record more events without extras attached
BrowserEngagement.click.record()
BrowserEngagement.click.record()

// Check if we collected any events into the 'click' metric
XCTAssertTrue(BrowserEngagement.click.testHasValue())

// Retrieve a snapshot of the recorded events
let events = try! BrowserEngagement.click.testGetValue()

// Check if we collected all 3 events in the snapshot
XCTAssertEqual(3, events.count)

// Check extra key/value for first event in the list
XCTAssertEqual("Courier", events[0].extra?["font"])

Here is a longer example to better illustrate the intended use of the test API:

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Record a metric value with extra to validate against
metrics.url.visit.add(1)

# Check if we collected any events into the 'click' metric
assert metrics.url.visit.test_has_value()

# Retrieve a snapshot of the recorded events
assert 1 == metrics.url.visit.test_get_value()

TODO. To be implemented in bug 1648448.

Generating testing coverage reports

Glean can generate coverage reports to track which metrics are tested in your unit test suite.

There are three steps to integrate it into your continuous integration workflow: recording coverage, post-processing the results, and uploading the results.

Recording coverage

Glean testing coverage is enabled by setting the GLEAN_TEST_COVERAGE environment variable to the name of a file to store results. It is good practice to set it to the absolute path to a file, since some testing harnesses (such as cargo test) may change the current working directory.

GLEAN_TEST_COVERAGE=$(realpath glean_coverage.txt) make test

Post-processing the results

A post-processing step is required to convert the raw output in the file specified by GLEAN_TEST_COVERAGE into usable output for coverage reporting tools. Currently, the only coverage reporting tool supported is codecov.io.

This post-processor is available in the coverage subcommand in the glean_parser tool.

For some build systems, glean_parser is already installed for you by the build system integration at the following locations:

  • On Android/Gradle, $GRADLE_HOME/glean/bootstrap-4.5.11/Miniconda3/bin/glean_parser
  • On iOS/Carthage, $PROJECT_ROOT/.venv/bin/glean_parser
  • For other systems, install glean_parser using pip install glean_parser

The glean_parser coverage command requires the following parameters:

  • -f: The output format to produce, for example codecovio to produce codecov.io's custom format.
  • -o: The path to the output file, for example codecov.json.
  • -c: The input raw coverage file. glean_coverage.txt in the example above.
  • A list of the metrics.yaml files in your repository.

For example, to produce output for codecov.io:

glean_parser coverage -f codecovio -o glean_coverage.json -c glean_coverage.txt app/metrics.yaml

In this example, the glean_coverage.json file is now ready for uploading to codecov.io.

Uploading coverage

If using codecov.io, the uploader doesn't send coverage results for YAML files by default. Pass the -X yaml option to the uploader to make sure they are included:

bash <(curl -s https://codecov.io/bash) -X yaml