Glean SDK

Glean logo

The Glean SDK is a modern approach for a Telemetry library and is part of the Glean project.

To contact us you can:

The source code is available on GitHub.

Using this book

This book is specifically about the Glean SDK (the client side code for collecting telemetry). Documentation about the broader end-to-end Glean project.

This book is divided into 5 main chapters:

Using the Glean SDK

If you want to use the Glean SDK to report data then this is the section you should read. It explains the first steps from integrating Glean into your project, contains details about all available metric types and how to do send your own custom pings.

Metrics collected by the Glean SDK

This chapter lists all metrics collected by the Glean SDK itself.

Developing the Glean SDK

This chapter describes how to develop the Glean SDK and its various implementations. This is relevant if you plan to contribute to the Glean SDK code base.

API Reference Documentation

Reference documentation for the API in its various language bindings.

This Week in Glean

“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean.

License

The Glean SDK Source Code is subject to the terms of the Mozilla Public License v2.0. You can obtain a copy of the MPL at https://mozilla.org/MPL/2.0/.

Using the Glean SDK

In this chapter we describe how to use the Glean SDK in your own libraries and applications.

Adding Glean to your project

Glean integration checklist

The Glean integration checklist can help to ensure your Glean SDK-using product is meeting all of the recommended guidelines.

Products (applications or libraries) using the Glean SDK to collect telemetry must:

  1. Integrate the Glean SDK into the build system. Since the Glean SDK does some code generation for your metrics at build time, this requires a few more steps than just adding a library.

  2. Include the markdown-formatted documentation generated from the metrics.yaml and pings.yaml files in the project's documentation.

  3. Go through data review process for all newly collected data.

  4. Ensure that telemetry coming from automated testing or continuous integration is either not sent to the telemetry server or tagged with the automation tag using the sourceTag feature.

Additionally, applications (but not libraries) must:

  1. File a data engineering bug to enable your product's application id.

  2. Provide a way for users to turn data collection off (e.g. providing settings to control Glean.setUploadEnabled()). The exact method used is application-specific.

Usage

Integrating with your project

Setting up the dependency

The Glean SDK is published on maven.mozilla.org. To use it, you need to add the following to your project's top-level build file, in the allprojects block (see e.g. Glean SDK's own build.gradle):

repositories {
    maven {
       url "https://maven.mozilla.org/maven2"
    }
}

Each module that uses Glean SDK needs to specify it in its build file, in the dependencies block. Add this to your Gradle configuration:

implementation "org.mozilla.components:service-glean:{latest-version}"

Important: the {latest-version} placeholder in the above link should be replaced with the version of Android Components used by the project.

The Glean SDK is released as part of android-components. Therefore, it follows android-components' versions. The android-components release page can be used to determine the latest version.

For example, if version 33.0.0 is used, then the include directive becomes:

implementation "org.mozilla.components:service-glean:33.0.0"

Size impact on the application APK: the Glean SDK APK ships binary libraries for all the supported platforms. Each library file measures about 600KB. If the final APK size of the consuming project is a concern, please enable ABI splits.

Requirements

  • Python >= 3.6.

Setting up the dependency

The Glean SDK can be consumed through Carthage, a dependency manager for macOS and iOS. For consuming the latest version of the Glean SDK, add the following line to your Cartfile:

github "mozilla/glean" "{latest-version}"

Important: the {latest-version} placeholder should be replaced with the version number of the latest Glean SDK release. You can find the version number on the release page.

Then check out and build the new dependency:

carthage update --platform iOS

Integrating with the build system

For integration with the build system you can follow the Carthage Quick Start steps.

  1. After building the dependency one drag the built .framework binaries from Carthage/Build/iOS into your application's Xcode project.

  2. On your application targets' Build Phases settings tab, click the + icon and choose New Run Script Phase. If you already use Carthage for other dependencies, extend the existing step. Create a Run Script in which you specify your shell (ex: /bin/sh), add the following contents to the script area below the shell:

    /usr/local/bin/carthage copy-frameworks
    
  3. Add the path to the Glean framework under "Input Files":

    $(SRCROOT)/Carthage/Build/iOS/Glean.framework
    
  4. Add the paths to the copied framework to the "Output Files":

    $(BUILT_PRODUCTS_DIR)/$(FRAMEWORKS_FOLDER_PATH)/Glean.framework
    

We recommend using a virtual environment for your work to isolate the dependencies for your project. There are many popular abstractions on top of virtual environments in the Python ecosystem which can help manage your project dependencies.

The Glean SDK Python bindings currently have prebuilt wheels on PyPI for Windows (i686 and x86_64), Linux (x86_64) and macOS (x86_64). For other platforms, the glean_sdk package will be built from source on your machine. This requires that Cargo and Rust are already installed. The easiest way to do this is through rustup.

Once you have your virtual environment set up and activated, you can install the Glean SDK into it using:

$ python -m pip install glean_sdk

The Glean SDK Python bindings make extensive use of type annotations to catch type related errors at build time. We highly recommend adding mypy to your continuous integration workflow to catch errors related to type mismatches early.

TODO. To be implemented in bug 1643568.

Adding new metrics

All metrics that your project collects must be defined in a metrics.yaml file.

To learn more, see adding new metrics. See the metric parameters documentation which provides reference information about the contents of that file.

Important: as stated before, any new data collection requires documentation and data-review. This is also required for any new metric automatically collected by the Glean SDK.

In order for the Glean SDK to generate an API for your metrics, two Gradle plugins must be included in your build:

The Glean Gradle plugin is distributed through Mozilla's Maven, so we need to tell your build where to look for it by adding the following to the top of your build.gradle:

buildscript {
    repositories {
        // Include the next clause if you are tracking snapshots of android components
        maven {
            url "https://snapshots.maven.mozilla.org/maven2"
        }
        maven {
            url "https://maven.mozilla.org/maven2"
        }

        dependencies {
            classpath "org.mozilla.components:tooling-glean-gradle:{android-components-version}"
        }
    }
}

Important: as above, the {android-components-version} placeholder in the above link should be replaced with the version number of android components used in your project.

The JetBrains Python plugin is distributed in the Gradle plugin repository, so it can be included with:

plugins {
    id "com.jetbrains.python.envs" version "0.0.26"
}

Right before the end of the same file, we need to apply the Glean Gradle plugin. Set any additional parameters to control the behavior of the Glean Gradle plugin before calling apply plugin.

// Optionally, set any parameters to send to the plugin.
ext.gleanGenerateMarkdownDocs = true
apply plugin: "org.mozilla.telemetry.glean-gradle-plugin"

Note: Earlier versions of Glean used a Gradle script (sdk_generator.gradle) rather than a Gradle plugin. Its use is deprecated and projects should be updated to use the Gradle plugin as described above.

Note: The Glean Gradle plugin has limited support for offline builds of applications that use the Glean SDK.

The metrics.yaml file is parsed at build time and Swift code is generated. Add a new metrics.yaml file to your Xcode project.

Follow these steps to automatically run the parser at build time:

  1. Download the sdk_generator.sh script from the Glean repository:

    https://raw.githubusercontent.com/mozilla/glean/{latest-release}/glean-core/ios/sdk_generator.sh
    

    Important: as above, the {latest-version} placeholder should be replaced with the version number of Glean SDK release used in this project.

  2. Add the sdk_generator.sh file to your Xcode project.

  3. On your application targets' Build Phases settings tab, click the + icon and choose New Run Script Phase. Create a Run Script in which you specify your shell (ex: /bin/sh), add the following contents to the script area below the shell:

    bash $PWD/sdk_generator.sh
    
  4. Add the path to your metrics.yaml and (optionally) pings.yaml under "Input files":

    $(SRCROOT)/{project-name}/metrics.yaml
    $(SRCROOT)/{project-name}/pings.yaml
    
  5. Add the path to the generated code file to the "Output Files":

    $(SRCROOT)/{project-name}/Generated/Metrics.swift
    

    Important: The parser now generates a single file called Metrics.swift (since Glean v31.0.0).

  6. If you are using Git, add the following lines to your .gitignore file:

    .venv/
    {project-name}/Generated
    

    This will ignore files that are generated at build time by the sdk_generator.sh script. They don't need to be kept in version control, as they can be re-generated from your metrics.yaml and pings.yaml files.

Important information about Glean and embedded extensions: Metric collection is a no-op in application extensions and Glean will not run. Since extensions run in a separate sandbox and process from the application, Glean would run in an extension as if it were a completely separate application with different client ids and storage. This complicates things because Glean doesn’t know or care about other processes. Because of this, Glean is purposefully prevented from running in an application extension and if metrics need to be collected from extensions, it's up to the integrating application to pass the information to the base application to record in Glean.

For Python, the metrics.yaml file must be available and loaded at runtime.

If your project is a script (i.e. just Python files in a directory), you can load the metrics.yaml using:

from glean import load_metrics

metrics = load_metrics("metrics.yaml")

# Use a metric on the returned object
metrics.your_category.your_metric.set("value")

If your project is a distributable Python package, you need to include the metrics.yaml file using one of the myriad ways to include data in a Python package and then use pkg_resources.resource_filename() to get the filename at runtime.

from glean import load_metrics
from pkg_resources import resource_filename

metrics = load_metrics(resource_filename(__name__, "metrics.yaml"))

# Use a metric on the returned object
metrics.your_category.your_metric.set("value")

Automation steps

Documentation

The documentation for your application or library's metrics and pings are written in metrics.yaml and pings.yaml. However, you should also provide human-readable markdown files based on this information, and this is a requirement for Mozilla projects using the Glean SDK. For other languages and platforms, this transformation is done automatically as part of the build. However, for Python the integration to automatically generate docs is an additional step.

The Glean SDK provides a commandline tool for automatically generating markdown documentation from your metrics.yaml and pings.yaml files. To perform that translation, run glean_parser's translate command:

python3 -m glean_parser translate -f markdown -o docs metrics.yaml pings.yaml

To get more help about the commandline options:

python3 -m glean_parser translate --help

We recommend integrating this step into your project's documentation build. The details of that integration is left to you, since it depends on the documentation tool being used and how your project is set up.

Metrics linting

Glean includes a "linter" for metrics.yaml and pings.yaml files called the glinter that catches a number of common mistakes in these files.

As part of your continuous integration, you should run the following on your metrics.yaml and pings.yaml files:

python3 -m glean_parser glinter metrics.yaml pings.yaml

A new build target needs to be added to the project csproj file in order to generate the metrics and pings APIs from the registry files (e.g. metrics.yaml, pings.yaml).

<Project>
  <!-- ... other directives ... -->

  <Target Name="GleanIntegration" BeforeTargets="CoreCompile">
    <ItemGroup>
      <!--
        Note that the two files are not required: Glean will work just fine
        with just the 'metrics.yaml'. A 'pings.yaml' is only required if custom
        pings are defined.
        Please also note that more than one metrics file can be added.
      -->
      <GleanRegistryFiles Include="metrics.yaml" />
      <GleanRegistryFiles Include="pings.yaml" />
    </ItemGroup>
    <!-- This is what actually runs the parser. -->
    <GleanParser RegistryFiles="@(GleanRegistryFiles)" OutputPath="$(IntermediateOutputPath)Glean" Namespace="csharp.GleanMetrics" />

    <!--
      And this adds the generated files to the project, so that they can be found by
      the compiler and Intellisense.
    -->
    <ItemGroup>
      <Compile Include="$(IntermediateOutputPath)Glean\**\*.cs" />
    </ItemGroup>
  </Target>
</Project>

This is using the Python 3 interpreter found in PATH under the hood. The GLEAN_PYTHON environment variable can be used to provide the location of the Python 3 interpreter.

Adding custom pings

Please refer to the custom pings documentation.

Important: as stated before, any new data collection requires documentation and data-review. This is also required for any new metric automatically collected by the Glean SDK.

Parallelism

All of the Glean SDK's target languages use a separate worker thread to do most of its work, including any I/O. This thread is fully managed by the Glean SDK as an implementation detail. Therefore, users should feel free to use the Glean SDK wherever it is most convenient, without worrying about the performance impact of updating metrics and sending pings.

Since the Glean SDK performs disk and networking I/O, it tries to do as much of its work as possible on separate threads and processes. Since there are complex trade-offs and corner cases to support Python parallelism, it is hard to design a one-size-fits-all approach.

Default behavior

When using the Python bindings, most of the Glean SDK's work is done on a separate thread, managed by the Glean SDK itself. The Glean SDK releases the Global Interpreter Lock (GIL) for most of its operations, therefore your application's threads should not be in contention with the Glean SDK's worker thread.

The Glean SDK installs an atexit handler so that its worker thread can cleanly finish when your application exits. This handler will wait up to 30 seconds for any pending work to complete.

By default, ping uploading is performed in a separate child process. This process will continue to upload any pending pings even after the main process shuts down. This is important for commandline tools where you want to return control to the shell as soon as possible and not be delayed by network connectivity.

Cases where subprocesses aren't possible

The default approach may not work with applications built using PyInstaller or similar tools which bundle an application together with a Python interpreter making it impossible to spawn new subprocesses of that interpreter. For these cases, there is an option to ensure that ping uploading occurs in the main process. To do this, set the allow_multiprocessing parameter on the glean.Configuration object to False.

Using the multiprocessing module

Additionally, the default approach does not work if your application uses the multiprocessing module for parallelism. The Glean SDK can not wait to finish its work in a multiprocessing subprocess, since atexit handlers are not supported in that context.
Therefore, if the Glean SDK detects that it is running in a multiprocessing subprocess, all of its work that would normally run on a worker thread will run on the main thread. In practice, this should not be a performance issue: since the work is already in a subprocess, it will not block the main process of your application.

Testing metrics

In order to make testing metrics easier 'out of the box', all metrics include a set of test API functions in order to facilitate unit testing. These include functions to test whether a value has been stored, and functions to retrieve the stored value for validation. For more information, please refer to Unit testing Glean metrics.

The General API

The Glean SDK has a minimal API available on its top-level Glean object. This API allows one to enable and disable upload, register custom pings and set experiment data.

Important: The Glean SDK should only be initialized from the main application, not individual libraries.

If you are adding Glean SDK support to a library, you can safely skip this section.

The API

The Glean SDK provides a general API that supports the following operations. See below for language-specific details.

OperationDescriptionNotes
initializeConfigure and initialize the Glean SDK.Initializing the Glean SDK
setUploadEnabledEnable or disable Glean collection and upload.Enabling and disabling Metrics
registerPingsRegister custom pings generated from pings.yaml.Custom pings
setExperimentActiveIndicate that an experiment is running.Using the Experiments API
setExperimentInactiveIndicate that an experiment is no longer running..Using the Experiments API

Initializing the Glean SDK

The following steps are required for applications using the Glean SDK, but not libraries.

Note: The initialize function must be called, even if telemetry upload is disabled. Glean needs to perform maintenance tasks even when telemetry is disabled, and because Glean does this as part of its initialization, it is required to always call the initialize function. Otherwise, Glean won't be able to clean up collected data, disable queuing of pre-init tasks, or perform other required operations.

Note: The Glean SDK does not support use across multiple processes, and must only be initialized on the application's main process. Initializing in other processes is a no-op. Additionally, Glean must be initialized on the main (UI) thread of the applications main process. Failure to do so will throw an IllegalThreadStateException.

An excellent place to initialize Glean is within the onCreate method of the class that extends Android's Application class.

import org.mozilla.yourApplication.GleanMetrics.Pings

class SampleApplication : Application() {

    override fun onCreate() {
        super.onCreate()

        // If you have custom pings in your application, you must register them
        // using the following command. This command should be omitted for
        // applications not using custom pings.
        Glean.registerPings(Pings)

        // Initialize the Glean library.
        Glean.initialize(
            applicationContext,
            // Here, `settings()` is a method to get user preferences, specific to
            // your application and not part of the Glean SDK API.
            uploadEnabled = settings().isTelemetryEnabled
        )
    }
}

Once initialized, if uploadEnabled is true, the Glean SDK will automatically start collecting baseline metrics and sending its pings, according to their respective schedules.
If uploadEnabled is false, any persisted metrics, events and pings (other than first_run_date) are cleared, and subsequent calls to record metrics will be no-ops.

The Glean SDK should be initialized as soon as possible, and importantly, before any other libraries in the application start using Glean. Library code should never call Glean.initialize, since it should be called exactly once per application.

Note: if the application has the concept of release channels and knows which channel it is on at run-time, then it can provide the Glean SDK with this information by setting it as part of the Configuration object parameter of the Glean.initialize method. For example:

Glean.initialize(applicationContext, Configuration(channel = "beta"))

Note: When the Glean SDK is consumed through Android Components, it is required to configure an HTTP client to be used for upload. For example:

// Requires `org.mozilla.components:concept-fetch`
import mozilla.components.concept.fetch.Client
// Requires `org.mozilla.components:lib-fetch-httpurlconnection`.
// This can be replaced by other implementations, e.g. `lib-fetch-okhttp`
// or an implementation from `browser-engine-gecko`.
import mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClient

import mozilla.components.service.glean.config.Configuration
import mozilla.components.service.glean.net.ConceptFetchHttpUploader

val httpClient = ConceptFetchHttpUploader(lazy { HttpURLConnectionClient() as Client })
val config = Configuration(httpClient = httpClient)
Glean.initialize(context, uploadEnabled = true, configuration = config)

Note: The Glean SDK does not support use across multiple processes, and must only be initialized on the application's main process.

An excellent place to initialize Glean is within the application(_:) method of the class that extends the UIApplicationDelegate class.

import Glean
import UIKit

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {
    func application(_: UIApplication, didFinishLaunchingWithOptions _: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        // If you have custom pings in your application, you must register them
        // using the following command. This command should be omitted for
        // applications not using custom pings.
        Glean.shared.registerPings(GleanMetrics.Pings)

        // Initialize the Glean library.
        Glean.shared.initialize(
            // Here, `Settings` is a method to get user preferences specific to
            // your application, and not part of the Glean SDK API.
            uploadEnabled = Settings.isTelemetryEnabled
        )
    }
}

Once initialized, if uploadEnabled is true, the Glean SDK will automatically start collecting baseline metrics and sending its pings, according to their respective schedules.
If uploadEnabled is false, any persisted metrics, events and pings (other than first_run_date) are cleared, and subsequent calls to record metrics will be no-ops.

The Glean SDK should be initialized as soon as possible, and importantly, before any other libraries in the application start using Glean. Library code should never call Glean.shared.initialize, since it should be called exactly once per application.

Note: if the application has the concept of release channels and knows which channel it is on at run-time, then it can provide the Glean SDK with this information by setting it as part of the Configuration object parameter of the Glean.shared.initialize method. For example:

Glean.shared.initialize(Configuration(channel: "beta"))

The main control for the Glean SDK is on the glean.Glean singleton.

The Glean SDK should be initialized as soon as possible, and importantly, before any other libraries in the application start using Glean. Library code should never call Glean.initialize, since it should be called exactly once per application.

from glean import Glean

Glean.initialize(
    application_id="my-app-id",
    application_version="0.1.0",
    upload_enabled=True,
)

Once initialized, if upload_enabled is true, the Glean SDK will automatically start collecting baseline metrics. If upload_enabled is false, any persisted metrics, events and pings (other than first_run_date) are cleared, and subsequent calls to record metrics will be no-ops.

Additional configuration is available on the glean.Configuration object, which can be passed into Glean.initialize().

Unlike Android and Swift, the Python bindings do not automatically send any pings. See the custom pings documentation about adding custom pings and sending them.

The main control for the Glean SDK is on the GleanInstance singleton.

The Glean SDK should be initialized as soon as possible, and importantly, before any other libraries in the application start using Glean. Library code should never call Glean.initialize, since it should be called exactly once per application.

using static Mozilla.Glean.Glean;

GleanInstance.Initialize(
    applicationId: "my.app.id",
    applicationVersion: "0.1.1",
    uploadEnabled: true,
    configuration: new Configuration(),
    dataDir: gleanDataDir
    );

Behavior when uninitialized

Metric recording that happens before the Glean SDK is initialized is queued and applied at initialization. To avoid unbounded memory growth the queue is bounded (currently to a maximum of 100 tasks), and further recordings are dropped. The number of recordings dropped, if any, is recorded in the glean.error.preinit_tasks_overflow metric.

Custom ping submission will not fail before initialization. Collection and upload of the custom ping is delayed until the Glean SDK is initialized. Built-in pings are only available after initialization.

Enabling and disabling metrics

Glean.setUploadEnabled() should be called in response to the user enabling or disabling telemetry.

Note: If called before Glean.initialize() the call to Glean.setUploadEnabled() will be ignored. Set the initial state using uploadEnabled on Glean.initialize().

Glean.shared.setUploadEnabled() should be called in response to the user enabling or disabling telemetry.

Note: If called before Glean.shared.initialize() the call to Glean.shared.setUploadEnabled() will be ignored. Set the initial state using uploadEnabled on Glean.shared.initialize().

Glean.set_upload_enabled() should be called in response to the user enabling or disabling telemetry.

Note: If called before Glean.initialize() the call to Glean.set_upload_enabled() will be ignored. Set the initial state using upload_enabled on Glean.initialize().

GleanInstance.SetUploadEnabled() should be called in response to the user enabling or disabling telemetry.

Note: If called before GleanInstance.initialize() the call to GleanInstance.SetUploadEnabled() will be ignored. Set the initial state using uploadEnabled on GleanInstance.initialize().

The application should provide some form of user interface to call this method.

When going from enabled to disabled, all pending events, metrics and pings are cleared, except for first_run_date. When re-enabling, core Glean metrics will be recomputed at that time.

Adding new metrics

When adding a new metric, the workflow is:

  • Consider the question you are trying to answer with this data, and choose the metric type and parameters to use.
  • Add a new entry to metrics.yaml.
  • Add code to your project to record into the metric by calling the Glean SDK.

Important: Any new data collection requires documentation and data-review. This is also required for any new metric automatically collected by the Glean SDK.

Choosing a metric type

The following is a set of questions to ask about the data being collected to help better determine which metric type to use.

Is it a single measurement?

If the value is true or false, use a boolean metric.

If the value is a string, use a string metric. For example, to record the name of the default search engine.

Beware: string metrics are exceedingly general, and you are probably best served by selecting the most specific metric for the job, since you'll get better error checking and richer analysis tools for free. For example, avoid storing a number in a string metric --- you probably want a counter metric instead.

If you need to store multiple string values in a metric, use a string list metric. For example, you may want to record the list of other Mozilla products installed on the device.

For all of the metric types in this section that measure single values, it is especially important to consider how the lifetime of the value relates to the ping it is being sent in. Since these metrics don't perform any aggregation on the client side, when a ping containing the metric is submitted, it will contain only the "last known" value for the metric, potentially resulting in data loss. There is further discussion of metric lifetimes below.

Are you counting things?

If you want to know how many times something happened, use a counter metric. If you are counting a group of related things, or you don't know what all of the things to count are at build time, use a labeled counter metric.

If you need to know when the things being counted happened relative to other things, consider using an event.

Are you measuring time?

If you need to record an absolute time, use a datetime metric. Datetimes are recorded in the user's local time, according to their device's real time clock, along with a timezone offset from UTC. Datetime metrics allow specifying the resolution they are collected at, and to stay lean, they should only be collected at the minimum resolution required to answer your question.

If you need to record how long something takes you have a few options.

If you need to measure the total time spent doing a particular task, look to the timespan metric. Timespan metrics allow specifying the resolution they are collected at, and to stay lean, they should only be collected at the minimum resolution required to answer your question. Note that this metric should only be used to measure time on a single thread. If multiple overlapping timespans are measured for the same metric, an invalid state error is recorded.

If you need to measure the relative occurrences of many timings, use a timing distribution. It builds a histogram of timing measurements, and is safe to record multiple concurrent timespans on different threads.

If you need to know the time between multiple distinct actions that aren't a simple "begin" and "end" pair, consider using an event.

Do you need to know the order of events relative to other events?

If you need to know the order of actions relative to other actions, such as, the user performed tasks A, B, and then C, and this is meaningfully different from the user performing tasks A, C and then B, (in other words, the order is meaningful beyond just the fact that a set of tasks were performed), use an event metric.

Important: events are the most expensive metric type to record, transmit, store and analyze, so they should be used sparingly, and only when none of the other metric types are sufficient for answering your question.

For how long do you need to collect this data?

Think carefully about how long the metric will be needed, and set the expires parameter to disable the metric at the earliest possible time. This is an important component of Mozilla's lean data practices.

When the metric passes its expiration date (determined at build time), it will automatically stop collecting data.

When a metric's expiration is within in 14 days, emails will be sent from telemetry-alerts@mozilla.com to the notification_emails addresses associated with the metric. At that time, the metric should be removed, which involves removing it from the metrics.yaml file and removing uses of it in the source code. Removing a metric does not affect the availability of data already collected by the pipeline.

If the metric is still needed after its expiration date, it should go back for another round of data review to have its expiration date extended.

When should the Glean SDK automatically clear the measurement?

The lifetime parameter of a metric defines when its value will be cleared. There are three lifetime options available:

  • ping (default): The metric is cleared each time it is submitted in the ping. This is the most common case, and should be used for metrics that are highly dynamic, such as things computed in response to the user's interaction with the application.
  • application: The metric is related to an application run, and is cleared after the application restarts and any Glean-owned ping, due at startup, is submitted. This should be used for things that are constant during the run of an application, such as the operating system version. In practice, these metrics are generally set during application startup. A common mistake---using the ping lifetime for these type of metrics---means that they will only be included in the first ping sent during a particular run of the application.
  • user: Reach out to the Glean team before using this.. The metric is part of the user's profile and will live as long as the profile lives. This is often not the best choice unless the metric records a value that really needs to be persisted for the full lifetime of the user profile, e.g. an identifier like the client_id, the day the product was first executed. It is rare to use this lifetime outside of some metrics that are built in to the Glean SDK.

While lifetimes are important to understand for all metric types, they are particularly important for the metric types that record single values and don't aggregate on the client (boolean, string, labeled_string, string_list, datetime and uuid), since these metrics will send the "last known" value and missing the earlier values could be a form of unintended data loss.

A lifetime example

Let's work through an example to see how these lifetimes play out in practice. Let's suppose we have a user preference, "turbo mode", which defaults to false, but the user can turn it to true at any time. We want to know when this flag is true so we can measure its affect on other metrics in the same ping. In the following diagram, we look at a time period that sends 4 pings across two separate runs of the application. We assume here, that like the Glean SDK's built-in metrics ping, the developer writing the metric isn't in control of when the ping is submitted.

In this diagram, the ping measurement windows are represented as rectangles, but the moment the ping is "submitted" is represented by its right edge. The user changes the "turbo mode" setting from false to true in the first run, and then toggles it again twice in the second run.

Metric lifetime timeline

  • A. Ping lifetime, set on change: The value isn't included in Ping 1, because Glean doesn't know about it yet. It is included in the first ping after being recorded (Ping 2), which causes it to be cleared.

  • B. Ping lifetime, set on init and change: The default value is included in Ping 1, and the changed value is included in Ping 2, which causes it to be cleared. It therefore misses Ping 3, but when the application is started, it is recorded again and it is included in Ping 4. However, this causes it to be cleared again and it is not in Ping 5.

  • C. Application lifetime, set on change: The value isn't included in Ping 1, because Glean doesn't know about it yet. After the value is changed, it is included in Pings 2 and 3, but then due to application restart it is cleared, so it is not included until the value is manually toggled again.

  • D. Application, set on init and change: The default value is included in Ping 1, and the changed value is included in Pings 2 and 3. Even though the application startup causes it to be cleared, it is set again, and all subsequent pings also have the value.

  • E. User, set on change: The default value is missing from Ping 1, but since user lifetime metrics aren't cleared unless the user profile is reset (e.g. on Android, when the product is uninstalled), it is included in all subsequent pings.

  • F. User, set on init and change: Since user lifetime metrics aren't cleared unless the user profile is reset, it is included in all subsequent pings. This would be true even if the "turbo mode" preference were never changed again.

Note that for all of the metric configurations, the toggle of the preference off and on during Ping 4 is completely missed. If you need to create a ping containing one, and only one, value for this metric, consider using a custom ping to create a ping whose lifetime matches the lifetime of the value.

What if none of these lifetimes are appropriate?

If the timing at which the metric is sent in the ping needs to closely match the timing of the metrics value, the best option is to use a custom ping to manually control when pings are sent.

This is especially useful when metrics need to be tightly related to one another, for example when you need to measure the distribution of frame paint times when a particular rendering backend is in use. If these metrics were in different pings, with different measurement windows, it is much harder to do that kind of reasoning with much certainty.

What should this new metric be called?

Reuse names from other applications

There's a lot of value using the same name for analogous metrics collected across different products. For example, BigQuery makes it simple to join columns with the same name across multiple tables. Therefore, we encourage you to investigate if a similar metric is already being collected by another product. If it is, there may be an opportunity for code reuse across these products, and if all the projects are using the Glean SDK, it's easy for libraries to send their own metrics. If sharing the code doesn't make sense, at a minimum we recommend using the same metric name for similar actions and concepts whenever possible.

Make names unique within an application

Metric identifiers (the combination of a metric's category and name) must be unique across all metrics that are sent by a single application. This includes not only the metrics defined in the app's metrics.yaml, but the metrics.yaml of any Glean SDK-using library that the application uses, including the Glean SDK itself. Therefore, care should be taken to name things specifically enough so as to avoid namespace collisions. In practice, this generally involves thinking carefully about the category of the metric, more than the name.

Note: Duplicate metric identifiers are not currently detected at build time. See bug 1578383 for progress on that. However, the probe_scraper process, which runs nightly, will detect duplicate metrics and e-mail the notification_emails associated with the given metrics.

Be as specific as possible

More broadly, you should choose the names of metrics to be as specific as possible. It is not necessary to put the type of the metric in the category or name, since this information is retained in other ways through the entire end-to-end system.

For example, if defining a set of events related to search, put them in a category called search, rather than just events or search_events. The events word here would be redundant.

What if none of these metric types is the right fit?

The current set of metrics the Glean SDK supports is based on known common use cases, but new use cases are discovered all the time.

Please reach out to us on #glean:mozilla.org. If you think you need a new metric type, we have a process for that.

How do I make sure my metric is working?

The Glean SDK has rich support for writing unit tests involving metrics. Writing a good unit test is a large topic, but in general, you should write unit tests for all new telemetry that does the following:

  • Performs the operation being measured.

  • Asserts that metrics contain the expected data, using the testGetValue API on the metric.

  • Where applicable, asserts that no errors are recorded, such as when values are out of range, using the testGetNumRecordedErrors API.

In addition to unit tests, it is good practice to validate the incoming data for the new metric on a pre-release channel to make sure things are working as expected.

Adding the metric to the metrics.yaml file

The metrics.yaml file defines the metrics your application or library will send. They are organized into categories. The overall organization is:

# Required to indicate this is a `metrics.yaml` file
$schema: moz://mozilla.org/schemas/glean/metrics/1-0-0

toolbar:
  click:
    type: event
    description: |
      Event to record toolbar clicks.
    notification_emails:
      - CHANGE-ME@example.com
    bugs:
      - https://bugzilla.mozilla.org/123456789/
    data_reviews:
      - http://example.com/path/to/data-review
    expires: 2019-06-01  # <-- Update to a date in the future

  double_click:
    ...

category2.subcategory:  # Categories can contain subcategories using `.`
  metric:
    ...

The details of the metric parameters are described in metric parameters.

The metrics.yaml file is used to generate code in the target language (e.g. Kotlin, Swift, ...) that becomes the public API to access your application's metrics.

Using the metric from your code

The reference documentation for each metric type goes into detail about using each metric type from your code.

One thing to note is that we try to adhere to the coding conventions of each language wherever possible, to the metric name in the metrics.yaml (which is in snake_case) may be changed to some other case convention, such as camelCase, when used from code.

Category and metric names in the metrics.yaml are in snake_case, but given the Kotlin coding standards defined by ktlint, these identifiers must be camelCase in Kotlin. For example, the metric defined in the metrics.yaml as:

views:
  login_opened:
    ...

is accessible in Kotlin as:

import org.mozilla.yourApplication.GleanMetrics.Views
GleanMetrics.Views.loginOpened...

Category and metric names in the metrics.yaml are in snake_case, but given the Swift coding standards defined by swiftlint, these identifiers must be camelCase in Swift. For example, the metric defined in the metrics.yaml as:

views:
  login_opened:
    ...

is accessible in Kotlin as:

GleanMetrics.Views.loginOpened...

Category and metric names in the metrics.yaml are in snake_case, which matches the PEP8 standard, so no translation is needed for Python.

TODO. To be implemented in this bug.

Metric parameters

Required metric parameters

  • type: Required. Specifies the type of a metric, like "counter" or "event". This defines which operations are valid for the metric, how it is stored and how data analysis tooling displays it. See the list of supported metric types.

Important: Once a metric is released in a product, its type should not be changed. If any data was collected locally with the older type, and hasn't yet been sent in a ping, recording data with the new type may cause any old persisted data to be lost for that metric. See this comment for an extended explanation of the different scenarios.

  • description: Required. A textual description of the metric for humans. It should describe what the metric does, what it means for analysts, and its edge cases or any other helpful information.

    The description field may contain markdown syntax.

  • notification_emails: Required. A list of email addresses to notify for important events with the metric or when people with context or ownership for the metric need to be contacted.

  • bugs: Required. A list of bugs (e.g. Bugzilla or GitHub) that are relevant to this metric. For example, bugs that track its original implementation or later changes to it.

    Each entry should be the full URL to the bug in an issue tracker. The use of numbers alone is deprecated and will be an error in the future.

  • data_reviews: Required. A list of URIs to any data collection reviews responses relevant to the metric.

  • expires: Required. May be one of the following values:

    • <build date>: An ISO date yyyy-mm-dd in UTC on which the metric expires. For example, 2019-03-13. This date is checked at build time. Except in special cases, this form should be used so that the metric automatically "sunsets" after a period of time. Emails will be sent to the notification_emails addresses when the metric is about to expire. Generally, when a metric is no longer needed, it should simply be removed. This does not affect the availability of data already collected by the pipeline.
    • never: This metric never expires.
    • expired: This metric is manually expired.

Optional metric parameters

  • lifetime: Defines the lifetime of the metric. Different lifetimes affect when the metrics value is reset.

    • ping (default): The metric is cleared each time it is submitted in the ping. This is the most common case, and should be used for metrics that are highly dynamic, such as things computed in response to the user's interaction with the application.
    • application: The metric is related to an application run, and is cleared after the application restarts and any Glean-owned ping, due at startup, is submitted. This should be used for things that are constant during the run of an application, such as the operating system version. In practice, these metrics are generally set during application startup. A common mistake---using the ping lifetime for these type of metrics---means that they will only be included in the first ping sent during a particular run of the application.
    • user: Reach out to the Glean team before using this.. The metric is part of the user's profile and will live as long as the profile lives. This is often not the best choice unless the metric records a value that really needs to be persisted for the full lifetime of the user profile, e.g. an identifier like the client_id, the day the product was first executed. It is rare to use this lifetime outside of some metrics that are built in to the Glean SDK.
  • send_in_pings: Defines which pings the metric should be sent on. If not specified, the metric is sent on the "default ping", which is the events ping for events and the metrics ping for everything else. Most metrics don't need to specify this unless they are sent on custom pings.

  • disabled: (default: false) Data collection for this metric is disabled. This is useful when you want to temporarily disable the collection for a specific metric without removing references to it in your source code. Generally, when a metric is no longer needed, it should simply be removed. This does not affect the availability of data already collected by the pipeline.

  • version: (default: 0) The version of the metric. A monotonically increasing integer value. This should be bumped if the metric changes in a backward-incompatible way.

  • data_sensitivity: (default: []) A list of data sensitivity categories that the metric falls under. There are four data collection categories related to data sensitivity defined in Mozilla's data collection review process:

    • Category 1: Technical Data: (technical) Information about the machine or Firefox itself. Examples include OS, available memory, crashes and errors, outcome of automated processes like updates, safe browsing, activation, version #s, and build id. This also includes compatibility information about features and APIs used by websites, add-ons, and other 3rd-party software that interact with Firefox during usage.

    • Category 2: Interaction Data: (interaction) Information about the user’s direct engagement with Firefox. Examples include how many tabs, add-ons, or windows a user has open; uses of specific Firefox features; session length, scrolls and clicks; and the status of discrete user preferences.

    • Category 3: Web activity data: (web_activity) Information about user web browsing that could be considered sensitive. Examples include users’ specific web browsing history; general information about their web browsing history (such as TLDs or categories of webpages visited over time); and potentially certain types of interaction data about specific webpages visited.

    • Category 4: Highly sensitive data: (highly_sensitive) Information that directly identifies a person, or if combined with other data could identify a person. Examples include e-mail, usernames, identifiers such as google ad id, apple id, Firefox account, city or country (unless small ones are explicitly filtered out), or certain cookies. It may be embedded within specific website content, such as memory contents, dumps, captures of screen data, or DOM data.

Unit testing Glean metrics

In order to support unit testing inside of client applications using the Glean SDK, a set of testing API functions have been included. The intent is to make the Glean SDK easier to test 'out of the box' in any client application it may be used in. These functions expose a way to inspect and validate recorded metric values within the client application but are restricted to test code only through visibility annotations (@VisibleForTesting(otherwise = VisibleForTesting.NONE) for Kotlin, internal methods for Swift).

General test API method semantics

In order to prevent issues with async calls when unit testing the Glean SDK, it is important to put the Glean SDK into testing mode by applying the JUnit GleanTestRule to your test class. When the Glean SDK is in testing mode, it enables uploading and clears the recorded metrics at the beginning of each test run. The rule can be used as shown below:

@RunWith(AndroidJUnit4::class)
class ActivityCollectingDataTest {
    // Apply the GleanTestRule to set up a disposable Glean instance.
    // Please note that this clears the Glean data across tests.
    @get:Rule
    val gleanRule = GleanTestRule(ApplicationProvider.getApplicationContext())

    @Test
    fun checkCollectedData() {
      // The Glean SDK testing API can be called here.
    }
}

This will ensure that metrics are done recording when the other test functions are used.

To check if a value exists (i.e. it has been recorded), there is a testHasValue() function on each of the metric instances:

assertTrue(GleanMetrics.Search.defaultSearchEngineUrl.testHasValue())

To check the actual values, there is a testGetValue() function on each of the metric instances. It is important to check that the values are recorded as expected, since many of the metric types may truncate or error-correct the value. This function will return a datatype appropriate to the specific type of the metric it is being used with:

assertEquals("https://example.com/search?", GleanMetrics.Search.defaultSearchEngineUrl.testGetValue())

Note that each of these functions has its visibility limited to the scope of unit tests by making use of the @VisibleForTesting annotation, so the IDE should complain if you attempt to use them inside of client code.

NOTE: There's no automatic test rule for Glean tests implemented.

In order to prevent issues with async calls when unit testing the Glean SDK, it is important to put the Glean SDK into testing mode. When the Glean SDK is in testing mode, it enables uploading and clears the recorded metrics at the beginning of each test run.

Activate it by resetting Glean in your test's setup:

@testable import Glean
import XCTest

class GleanUsageTests: XCTestCase {
    override func setUp() {
        Glean.shared.resetGlean(clearStores: true)
    }

    // ...
}

This will ensure that metrics are done recording when the other test functions are used.

To check if a value exists (i.e. it has been recorded), there is a testHasValue() function on each of the metric instances:

XCTAssertTrue(GleanMetrics.Search.defaultSearchEngineUrl.testHasValue())

To check the actual values, there is a testGetValue() function on each of the metric instances. It is important to check that the values are recorded as expected, since many of the metric types may truncate or error-correct the value. This function will return a datatype appropriate to the specific type of the metric it is being used with:

XCTAssertEqual("https://example.com/search?", try GleanMetrics.Search.defaultSearchEngineUrl.testGetValue())

Note that each of these functions is marked as internal, you need to import Glean explicitly in test mode:

@testable import Glean

It is generally a good practice to "reset" the Glean SDK prior to every unit test that uses the Glean SDK, to prevent side effects of one unit test impacting others. The Glean SDK contains a helper function glean.testing.reset_glean() for this purpose. It has two required arguments: the application ID, and the application version. Each reset of the Glean SDK will create a new temporary directory for Glean to store its data in. This temporary directory is automatically cleaned up the next time the Glean SDK is reset or when the testing framework finishes.

The instructions below assume you are using pytest as the test runner. Other test-running libraries have similar features, but are different in the details.

Create a file conftest.py at the root of your test directory, and add the following to reset Glean at the start of every test in your suite:

import pytest
from glean import testing

@pytest.fixture(name="reset_glean", scope="function", autouse=True)
def fixture_reset_glean():
    testing.reset_glean(application_id="my-app-id", application_version="0.1.0")

To check if a value exists (i.e. it has been recorded), there is a test_has_value() function on each of the metric instances:

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# ...

assert metrics.search.search_engine_url.test_has_value()

To check the actual values, there is a test_get_value() function on each of the metric instances. It is important to check that the values are recorded as expected, since many of the metric types may truncate or error-correct the value. This function will return a datatype appropriate to the specific type of the metric it is being used with:

assert (
    "https://example.com/search?" ==
    metrics.search.default_search_engine_url.test_get_value()
)

TODO. To be implemented in bug 1648448.

Testing metrics for custom pings

In order to test metrics where the metric is included in more than one ping, the test functions take an optional pingName argument (ping_name in Python). This is the name of the ping that the metric is being sent in, such as "events" for the events ping, or "metrics" for the metrics ping. This could also be a custom ping name that the metric is being sent in. In most cases you should not have to supply the ping name to the test function and can just use the default which is the "default" ping that this metric is sent in. You should only need to provide a pingName if the metric is being sent in more than one ping in order to identify the correct metric store.

You can call the testHasValue() and testGetValue() functions with pingName like this:

GleanMetrics.Foo.uriCount.testHasValue("customPing")
GleanMetrics.Foo.uriCount.testGetValue("customPing")

Example of using the test API

Here is a longer example to better illustrate the intended use of the test API:

// Record a metric value with extra to validate against
GleanMetrics.BrowserEngagement.click.record(
    mapOf(
        BrowserEngagement.clickKeys.font to "Courier"
    )
)

// Record more events without extras attached
BrowserEngagement.click.record()
BrowserEngagement.click.record()

// Check if we collected any events into the 'click' metric
assertTrue(BrowserEngagement.click.testHasValue())

// Retrieve a snapshot of the recorded events
val events = BrowserEngagement.click.testGetValue()

// Check if we collected all 3 events in the snapshot
assertEquals(3, events.size)

// Check extra key/value for first event in the list
assertEquals("Courier", events.elementAt(0).extra["font"])

Here is a longer example to better illustrate the intended use of the test API:

// Record a metric value with extra to validate against
GleanMetrics.BrowserEngagement.click.record([.font: "Courier"])

// Record more events without extras attached
BrowserEngagement.click.record()
BrowserEngagement.click.record()

// Check if we collected any events into the 'click' metric
XCTAssertTrue(BrowserEngagement.click.testHasValue())

// Retrieve a snapshot of the recorded events
let events = try! BrowserEngagement.click.testGetValue()

// Check if we collected all 3 events in the snapshot
XCTAssertEqual(3, events.count)

// Check extra key/value for first event in the list
XCTAssertEqual("Courier", events[0].extra?["font"])

Here is a longer example to better illustrate the intended use of the test API:

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Record a metric value with extra to validate against
metrics.url.visit.add(1)

# Check if we collected any events into the 'click' metric
assert metrics.url.visit.test_has_value()

# Retrieve a snapshot of the recorded events
assert 1 == metrics.url.visit.test_get_value()

TODO. To be implemented in bug 1648448.

Debugging products using the Glean SDK

Platform-specific debugging

  1. Debugging Android applications using the Glean SDK
  2. Debugging iOS applications using the Glean SDK
  3. Debugging Python applications using the Glean SDK

General debugging information

Available debugging features

The Glean SDK has 4 available debugging features:

  • logPings: This is either true or false and will cause all subsequent pings that are submitted, to also be echoed to the device's log. Once enabled, the only way to disable this feature is to restart or manually reset the application.
  • debugViewTag: This will tag all subsequent outgoing pings with the provided value, in order to identify them in the Glean Debug View. Once enabled, the only way to disable this feature is to restart or manually reset the application.
  • sourceTags: This will tag all subsequent outgoing pings with a maximum of 5 comma-separated tags. Once enabled, the only way to disable this feature is to restart or manually reset the application.
  • sendPing: This expects the name of a ping and forces its immediate collection and submission. This feature is only available for Android and iOS.

Different platforms may have different ways to enable these features.

Enabling debugging features through environment variables

Some of the debugging features described above may be enabled using environment variables:

  • logPings: May be set by the GLEAN_LOG_PINGS environment variable. The accepted values are true or false. Any other value will be ignored.
  • debugViewTag: May be set by the GLEAN_DEBUG_VIEW_TAG environment variable. Any valid HTTP header value is expected here (e.g. any value that matches the regex [a-zA-Z0-9-]{1,20}). Invalid values will be ignored.
  • sourceTags: May be set by the GLEAN_SOURCE_TAGS environment variable. A comma-separated list of valid HTTP header values is expected here (e.g. any value that matches the regex [a-zA-Z0-9-]{1,20}). Invalid values will be ignored.

These variables must be set at runtime, not at compile time. They will be checked upon Glean initialization.

Enabling debugging features using environment variables is available for all supported platforms.

Note Although it is technically possible to use the environment variables described here to enable debugging features in Android, the Glean team is not currently aware of a proper way to set environment variables in Android devices or emulators.

Important considerations when using Glean SDK debug features

  • Enabled features will persist until the application is closed or manually reset. When enabled by environment variables, the variables need to be cleared upon resetting for the feature to be disabled.

  • There are a couple different ways in which to send pings using the Glean SDK debug tools.

    1. You can tag pings using the debug tools and trigger them manually using the UI. This should always produce a ping with all required fields.
    2. You can tag and send pings using the debug tools. This has the side effect of potentially sending a ping which does not include all fields because sendPing triggers pings to be sent before certain application behaviors can occur which would record that information. For example, duration is not calculated or included in a baseline ping sent with sendPing because it forces the ping to be sent before the duration metric has been recorded. Keep in mind that there may be nothing to send, in which case no ping is generated.
    3. You can trigger a command while the instrumented application is still running. This is useful for toggling commands or for triggering pings that have schedules that are difficult to trigger manually. This is especially useful if you need to trigger a ping submission after some activity within the application, such as with the metrics ping.

Glean SDK Log messages

The Glean SDK logs warnings and errors through the platform-specific logging frameworks. See the platform-specific instructions for information on how to view the logs.

Implementation details

See Debug Pings.

Glean Debug Ping View

The Glean Debug Ping View enables you to easily see in real-time what data your application is sending.

This data is what actually arrives in our data pipeline, shown in a web interface that is automatically updated when new data arrives. Any data sent from a Glean SDK-instrumented application usually shows up within 10 seconds, updating the pages automatically. Pings are retained for 3 weeks.

What setup is needed for applications?

You need to tag each ping with a debugging tag. See the documentation on debugging for information on how to do this for each platform and/or language.

Troubleshooting

If nothing is showing up on the dashboard, and you see Glean must be enabled before sending pings. in the logs, then the application has disabled Glean. Check with the application author on how to re-enable it.

Debugging Android applications using the Glean SDK

The Glean SDK exports the GleanDebugActivity that can be used to toggle debugging features on or off. Users can invoke this special activity, at run-time, using the following adb command:

adb shell am start -n [applicationId]/mozilla.telemetry.glean.debug.GleanDebugActivity [extra keys]

In the above:

  • [applicationId] is the product's application id as defined in the manifest file and/or build script. For the Glean sample application, this is org.mozilla.samples.gleancore for a release build and org.mozilla.samples.gleancore.debug for a debug build.

  • [extra keys] is a list of extra keys to be passed to the debug activity. See the documentation for the command line switches used to pass the extra keys. These are the currently supported keys:

keytypedescription
logPingsboolean (--ez)If set to true, pings are dumped to logcat; defaults to false
sendPingstring (--es)Sends the ping with the given name immediately
debugViewTagstring (--es)Tags all outgoing pings as debug pings to make them available for real-time validation, on the Glean Debug View. The value must match the pattern [a-zA-Z0-9-]{1,20}. Important: in older versions of the Glean SDK, this was named tagPings
sourceTagsstring array (--esa)Tags outgoing pings with a maximum of 5 comma-separated tags. The tags must match the pattern [a-zA-Z0-9-]{1,20}. The automation tag is meant for tagging pings generated on automation: such pings will be specially handled on the pipeline (i.e. discarded from non-live views). Tags starting with glean are reserved for future use. Subsequent calls of this overwrite any previously stored tag
startNextstring (--es)The name of an exported Android Activity, as defined in the product manifest file, to start right after the GleanDebugActivity completes. All the options provided are propagated to this next activity as well. When omitted, the default launcher activity for the product is started instead.

All the options provided to start the activity are passed over to the main activity for the application to process. This is useful if SDK users wants to debug telemetry while providing additional options to the product to enable specific behaviors.

Note: Due to limitations on Android logcat message size, pings larger than 4KB are broken into multiple log messages when using logPings.

For example, to direct a release build of the Glean sample application to (1) dump pings to logcat, (2) tag the ping with the test-metrics-ping tag, and (3) send the "metrics" ping immediately, the following command can be used:

adb shell am start -n org.mozilla.samples.gleancore/mozilla.telemetry.glean.debug.GleanDebugActivity \
  --ez logPings true \
  --es sendPing metrics \
  --es debugViewTag test-metrics-ping

The logPings command doesn't trigger ping submission and you won't see any output until a ping has been sent. You can use the sendPing command to force a ping to be sent, but it could be more desirable to trigger the pings submission on their normal schedule. For instance, the baseline and events pings can be triggered by moving the app out of the foreground and the metrics ping can be triggered normally if it is overdue for the current calendar day.

Note: The device or emulator must be connected to the internet for this to work. Otherwise the job that sends the pings won't be triggered.

If no metrics have been collected, no pings will be sent unless send_if_empty is set on your ping. See the ping documentation for more information on ping scheduling to learn when pings are sent.

Options that are set using the adb flags are not immediately reset and will persist until the application is closed or manually reset.

Glean SDK Log messages

When running a Glean SDK-powered app in the Android emulator or on a device connected to your computer via cable, there are several ways to read the log output.

Android Studio

Android Studio can show the logs of a connected emulator or device. To display the log messages for an app:

  1. Run an app on your device.
  2. Click View > Tool Windows > Logcat (or click Logcat in the tool window bar).

The Logcat window will show all log messages and allows to filter those by the application ID. Select the application ID of the product you're debugging. You can also filter by Glean only.

More information can be found in the View Logs with Logcat help article.

Command line

On the command line you can show all of the log output using:

adb logcat

This is the unfiltered output of all log messages. You can match for glean using grep:

adb logcat | grep -i glean

A simple way to filter for only the application that is being debugged is by using pidcat, a wrapper around adb, which adds colors and proper filtering by application ID and log level. Run it like this to filter for an application:

pidcat [applicationId]

In the above [applicationId] is the product's application id as defined in the manifest file and/or build script. For the Glean sample application, this is org.mozilla.samples.gleancore for a release build and org.mozilla.samples.gleancore.debug for a debug build.

Enabling debugging features in iOS through environment variables

Debugging features in iOS can be enabled using environment variables. For more information on the available features accessible through this method and how to enable them, see Enabling debugging features through environment variables.

These environment variables must be set on the device that is running the application.

Note To set environment variables to the process running your app in an iOS device or emulator you need to edit the scheme for your app. In the Xcode IDE, you can use the shortcut Cmd + < to open the scheme editor popup. The environment variables editor is under the Arguments tab on this popup.

Debugging iOS applications using the Glean SDK

For debugging and validation purposes on iOS, the Glean SDK makes use of a custom URL scheme which is implemented within the application that is consuming the Glean SDK. The Glean SDK provides some convenience functions to facilitate this, but it's up to the consuming application to enable this functionality. Applications that enable this Glean SDK feature will be able to launch the application from a URL with the Glean debug commands embedded in the URL itself.

Available commands and query format

There are 4 available commands that you can use with the Glean SDK debug tools

  • logPings: This is either true or false and will cause pings that are submitted to also be echoed to the device's log
  • debugViewTag: This command will tag outgoing pings with the provided value, in order to identify them in the Glean Debug View. Tags need to be string with upper and lower case letters, numbers and dashes, with a max length of 20 characters. Important: in older versions of the Glean SDK, this was named tagPings.
  • sourceTags: This command tags outgoing pings with a maximum of 5 comma-separated tags. The tags must match the pattern [a-zA-Z0-9-]{1,20}. The automation tag is meant for tagging pings generated on automation: such pings will be specially handled on the pipeline (i.e. discarded from non-live views). Tags starting with glean are reserved for future use.
  • sendPing: This command expects a string name of a ping to force immediate collection and submission of.

The structure of the custom URL uses the following format:

<protocol>://glean?<command 1>=<paramter 1>&<command 2>=<parameter 2> ...

Where:

  • <protocol> is the "URL Scheme" that has been added for your app (see Instrumenting the application below), such as glean-sample-app.
  • This is followed by :// and then glean which is required for the Glean SDK to recognize the command is meant for it to process.
  • Following standard URL query format, the next character after glean is the ? indicating the beginning of the query.
  • This is followed by one or more queries in the form of <command>=<parameter>, where the command is one of the commands listed above, followed by an = and then the value or parameter to be used with the command.

There are a few things to consider when creating the custom URL:

  • Invalid commands will log an error and cause the entire URL to be ignored.
  • Not all commands are required to be encoded in the URL, you can mix and match the commands that you need.
  • Multiple instances of commands are not allowed in the same URL and, if present, will cause the entire URL to be ignored.
  • The logPings command doesn't trigger ping submission and you won't see any output until a ping has been submitted. You can use the sendPings command to force a ping to be sent, but it could be more desirable to trigger the pings submission on their normal schedule. For instance, the baseline and events pings can be triggered by moving the app out of the foreground and the metrics ping can be triggered normally if it is overdue for the current calendar day. See the ping documentation for more information on ping scheduling to learn when pings are sent.
  • Enabling debugging features through custom URLs overrides any debugging features set through environment variables.

Instrumenting the application for Glean SDK debug functionality

In order to enable the debugging features in a Glean SDK consuming iOS application, it is necessary to add some information to the application's Info.plist, and add a line and possibly an override for a function in the AppDelegate.swift.

Register custom URL scheme in Info.plist

Note: If your application already has a custom URL scheme implemented, there is no need to implement a second scheme, you can simply use that and skip to the next section about adding the convenience method. If the app doesn't have a custom URL scheme implemented, then you will need to perform the following instructions to register your app to receive custom URLs.

Find and open the application's Info.plist and right click any blank area and select Add Row to create a new key.

You will be prompted to select a key from a drop-down menu, scroll down to and select URL types. This creates an array item, which can be expanded by clicking the triangle disclosure icon.

Select Item 0, click on it and click the disclosure icon to expand it and show the URL identifier line. Double-click the value field and fill in your identifier, typically the same as the bundle ID.

Right-click on Item 0 and select Add Row from the context menu. In the dropdown menu, select URL Schemes to add the item.

Click on the disclosure icon of URL Schemes to expand the item, double-click the value field of Item 0 and key in the value for your application's custom scheme. For instance, the Glean sample app uses glean-sample-app, which allows for custom URLs to be crafted using that as a protocol, for example: glean-sample-app://glean?logPings=true

Add the Glean.handleCustomUrl() convenience function and necessary overrides

In order to handle the incoming Glean SDK debug commands, it is necessary to implement the override in the application's AppDelegate.swift file. Within that function, you can make use of the convenience function provided in Glean handleCustomUrl(url: URL).

An example of a simple implementation of this would look like this:

func application(_: UIApplication,
                 open url: URL,
                 options _: [UIApplication.OpenURLOptionsKey: Any] = [:]) -> Bool {
    // ...

    // This does nothing if the url isn't meant for Glean.
    Glean.shared.handleCustomUrl(url: url)

    // ...

    return true
}

If you need additional help setting up a custom URL scheme in your application, please refer to Apple's documentation.

Invoking the Glean-iOS debug commands

Now that the app has the Glean SDK debug functionality enabled, there are a few ways in which we can invoke the debug commands.

Using a web browser

Perhaps the simplest way to invoke the Glean SDK debug functionality is to open a web browser and type/paste the custom URL into the address bar. This is especially useful on an actual device because there isn't a good way to launch from the command line and process the URL for an actual device.

Using the glean-sample-app as an example: to activate ping logging, tag the pings to go to the Glean Debug View, and force the events ping to be sent, enter the following URL in a web browser on the iOS device:

glean-sample-app://glean?logPings=true&debugViewTag=My-ping-tag&sendPing=events

This should cause iOS to prompt you with a dialog asking if you want to open the URL in the Glean Sample App, and if you select "Okay" then it will launch (or resume if it's already running) the application with the indicated commands and parameters and immediately force the collection and submission of the events ping.

Note: This method does not work if the browser you are using to input the command is the same application you are attempting to pass the Glean debug commands to. So, you couldn't use Firefox for iOS to trigger commands within Firefox for iOS.

It is also possible to encode the URL into a 2D barcode or QR code and launch the app via the camera app. After scanning the encoded URL, the dialog prompting to launch the app should appear as if the URL were entered into the browser address bar.

Using the command line

This method is useful for testing via the Simulator, which typically requires a Mac with Xcode installed, including the Xcode command line tools. In order to perform the same command as above with using the browser to input the URL, you can use the following command in the command line terminal of the Mac:

xcrun simctl openurl booted "glean-sample-app://glean?logPings=true&debugViewTag=My-ping-tag&sendPing=events"

This will launch the simulator and again prompt the user with a dialog box asking if you want to open the URL in the Glean Sample App (or whichever app you are instrumenting and testing).

Glean log messages

The Glean SDK integrates with the unified logging system available on iOS. There are various ways to retrieve log information, see the official documentation.

If debugging in the simulator, the logging messages can be seen in the console window within Xcode.

When running a Glean-powered app in the iOS Simulator or on a device connected to your computer via cable you can use Console.app to view the system log. You can filter the logs with category:glean to only see logs from the Glean SDK.

You can also use the command line utility log to stream the log output. Run the following in a shell:

log stream --predicate 'category contains "glean"'

See Diagnosing Issues Using Crash Reports and Device Logs for more information about debugging deployed iOS apps.

Debugging Python applications using the Glean SDK

Debugging features in Python can be enabled using environment variables. For more information on the available features and how to enable them, see Enabling debugging features through environment variables.

Sending pings

Unlike other platforms, Python doesn't expose convenience methods to send pings on demand.

In case that is necessary, calling the submit function for a given ping, such as pings.custom_ping.submit(), will send it.

Logging pings

If the GLEAN_LOG_PINGS environment variable is set to true, pings are logged to the console on DEBUG level whenever they are submitted.

Make sure that when you configure logging in your application, you set the level for the glean logger to DEBUG or higher. Otherwise pings won't be logged even if GLEAN_LOG_PINGS is set to true.

You can set the logging level for Glean to DEBUG as follows:

import logging

logging.getLogger("glean").setLevel(logging.DEBUG)

See the Python logging documentation for more information.

Error reporting

The Glean SDK records the number of errors that occur when metrics are passed invalid data or are otherwise used incorrectly. This information is reported back in special labeled counter metrics in the glean.error category. Error metrics are included in the same pings as the metric that caused the error. Additionally, error metrics are always sent in the metrics ping ping.

The following categories of errors are recorded:

  • invalid_value: The metric value was invalid.
  • invalid_label: The label on a labeled metric was invalid.
  • invalid_state: The metric caught an invalid state while recording.
  • invalid_overflow: The metric value to be recorded overflows the metric-specific upper range.

For example, if you had a string metric and passed it a string that was too long:

MyMetrics.stringMetric.set("this_string_is_longer_than_the_limit_for_string_metrics")

The following error metric counter would be incremented:

Glean.error.invalidOverflow["my_metrics.string_metric"].add(1)

Resulting in the following keys in the ping:

{
  "metrics": {
    "labeled_counter": {
      "glean.error.invalid_overflow": {
        "my_metrics.string_metric": 1
      }
    }
  }
}

If you have a debug build of the Glean SDK, details about the errors being recorded are included in the logs. This detailed information is not included in Glean pings.

Using the experiments API

The Glean SDK supports tagging all its pings with experiments annotations. The annotations are useful to report that experiments were active at the time the measurement were collected. The annotations are reported in the optional experiments entry in the ping_info section of all the Glean SDK pings.

Important: the experiment annotations set through this API are not persisted by the Glean SDK. The application or consuming library is responsible for setting the relevant experiment annotations at each run.

API

// Annotate Glean pings with experiments data.
Glean.setExperimentActive(
  experimentId = "blue-button-effective",
  branch = "branch-with-blue-button",
  extra: mapOf(
    "buttonLabel" to "test"
  )
)
// After the experiment terminates, the annotation
// can be removed.
Glean.setExperimentInactive("blue-button-effective")

Important: Experiment IDs and branches don't need to be pre-defined in the Glean SDK registry files. Please also note that the extra map is a non-nested arbitrary String to String map. It also has limits on the size of the keys and values defined below.

There are test APIs available too:

// Was the experiment annotated in Glean pings?
assertTrue(Glean.testIsExperimentActive("blue-button-effective"))
// Was the correct branch reported?
assertEquals(
  "branch-with-blue-button", Glean.testGetExperimentData("blue-button-effective")?.branch
)
// Annotate Glean pings with experiments data.
Glean.shared.setExperimentActive(
  experimentId: "blue-button-effective",
  branch: "branch-with-blue-button",
  extra: ["buttonLabel": "test"]
)
// After the experiment terminates, the annotation
// can be removed.
Glean.shared.setExperimentInactive(experimentId: "blue-button-effective")

Important: Experiment IDs and branch don't need to be pre-defined in the Glean SDK registry files. Please also note that the extra is a non-nested Dictionary of type [String: String].

There are test APIs available too:

// Was the experiment annotated in Glean pings?
XCTAssertTrue(Glean.shared.testIsExperimentActive(experimentId: "blue-button-effective"))
// Was the correct branch reported?
XCTAssertEqual(
  "branch-with-blue-button",
  Glean.testGetExperimentData(experimentId: "blue-button-effective")?.branch
)
from glean import Glean

# Annotate Glean pings with experiments data.
Glean.set_experiment_active(
  experiment_id="blue-button-effective",
  branch="branch-with-blue-button",
  extra={
    "buttonLabel": "test"
  }
)

# After the experiment terminates, the annotation
# can be removed.
Glean.set_experiment_inactive("blue-button-effective")

Important: Experiment IDs and branch don't need to be pre-defined in the Glean SDK registry files. Please also note that the extra dict is non-nested arbitrary str to str mapping.

There are test APIs available too:

from glean import Glean

# Was the experiment annotated in Glean pings?
assert Glean.test_is_experiment_active("blue-button-effective")
# Was the correct branch reported?
assert (
    "branch-with-blue-button" ==
    Glean.test_get_experiment_data("blue-button-effective").branch
)
// Annotate Glean pings with experiments data.
GleanInstance.SetExperimentActive(
  experimentId: "blue-button-effective",
  branch: "branch-with-blue-button",
  extra: new Dictionary<string, string>() {
    { "buttonLabel", "test"}
  }
);
// After the experiment terminates, the annotation
// can be removed.
GleanInstance.SetExperimentInactive("blue-button-effective");

Important: Experiment IDs and branches don't need to be pre-defined in the Glean SDK registry files. Please also note that the extra map is a non-nested arbitrary string to string dictionary. It also has limits on the size of the keys and values defined below.

There are test APIs available too:

// Was the experiment annotated in Glean pings?
Assert.True(GleanInstance.TestIsExperimentActive("blue-button-effective"));
// Was the correct branch reported?
Assert.Equal(
  "branch-with-blue-button", GleanInstance.TestGetExperimentData("blue-button-effective").Branch
);

Limits

  • experimentId, branch, and the keys and values of the 'extra' field are fixed at a maximum length of 100 bytes. Longer strings used as ids, keys, or values are truncated to their respective maximum length. (Specifically, this is measured in the number of bytes when the string is encoded in UTF-8.)
  • extra map is limited to 20 entries. If passed a map which contains more elements than this, it is truncated to 20 elements. WARNING Which items are truncated is nondeterministic due to the unordered nature of maps and what's left may not necessarily be the first elements added.

NOTE: Any truncation that occurs when recording to the Experiments API will result in an invalid_value error being recorded. See Error Reporting for more information about this type of error.

Reference

Metrics

Not sure which metric type to use? These docs contain a series of questions that can help. Reference information about each metric type is linked below.

There are different metrics to choose from, depending on what you want to achieve:

  • Boolean: Records a single truth value, for example "is a11y enabled?"

  • Counter: Used to count how often something happens, for example, how often a certain button was pressed.

  • Labeled counter: Used to count how often something happens, for example which kind of crash occurred ("uncaught_exception" or "native_code_crash").

  • JWE: Records an instance of a JWE encrypted value.

  • String: Records a single Unicode string value, for example the name of the OS.

  • Labeled strings: Records multiple Unicode string values, for example to record which kind of error occurred in different stages of a login process.

  • String List: Records a list of Unicode string values, for example the list of enabled search engines.

  • Timespan: Used to measure how much time is spent in a single task.

  • Timing Distribution: Used to record the distribution of multiple time measurements.

  • Memory Distribution: Used to record the distribution of memory sizes.

  • UUID: Used to record universally unique identifiers (UUIDs), such as a client ID.

  • Datetime: Used to record an absolute date and time, such as the time the user first ran the application.

  • Events: Records events e.g. individual occurrences of user actions, say every time a view was open and from where.

  • Custom Distribution: Used to record the distribution of a value that needs fine-grained control of how the histogram buckets are computed. Custom distributions are only available for values that come from Gecko.

  • Quantity: Used to record a single non-negative integer value. For example, the width of the display in pixels. Quantities are only available for values that come from Gecko.

Labeled metrics

There are two types of metrics listed above - labeled and unlabeled metrics. If a metric is labeled, it means that for a single metric entry you define in metrics.yaml, you can record into multiple metrics under the same name, each of the same type and identified by a different string label.

This is useful when you need to break down metrics by a label known at build time or run time. For example:

  • When you want to count a different set of sub-views that users interact with, you could use viewCount["view1"].add() and viewCount["view2"].add().

  • When you want to count errors that might occur for a feature, you could use errorCount[errorName].add().

Labeled metrics come in two forms:

  • Static labels: The labels are specified at build time in the metrics.yaml file. If a label that isn't part of this set is used at run time, it is converted to the special label __other__.

  • Dynamic labels: The labels aren't known at build time, so are set at run time. Only the first 16 labels seen by the Glean SDK will be tracked. After that, any additional labels are converted to the special label __other__.

Note: Be careful with using arbitrary strings as labels and make sure they can't accidentally contain identifying data (like directory paths or user input).

Label format

To ensure maximum support in database columns, labels must be made up of dot-separated identifiers with lowercase ASCII alphanumerics, containing underscores and dashes.

Specifically, they must conform to this regular expression:

^[a-z_][a-z0-9_-]{0,29}(\\.[a-z_][a-z0-9_-]{0,29})*$

Adding or changing metric types

Glean has a well-defined process for requesting changes to existing metric types or suggesting the implementation of new metric types:

  1. Glean consumers need to file a bug in the Data platforms & tools::Glean Metric Types component, filling in the provided form;
  2. The triage owner of the Bugzilla component prioritizes this within 6 business days and kicks off the decision making process.
  3. Once the decision process is completed, the bug is closed with a comment outlining the decision that was made.

Boolean

Booleans are used for simple flags, for example "is a11y enabled"?.

Configuration

Say you're adding a boolean to record whether a11y is enabled on the device. First you need to add an entry for the boolean to the metrics.yaml file:

flags:
  a11y_enabled:
    type: boolean
    description: >
      Records whether a11y is enabled on the device.
    lifetime: application
    ...

API

import org.mozilla.yourApplication.GleanMetrics.Flags

Flags.a11yEnabled.set(System.isAccesibilityEnabled())

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Flags

// Was anything recorded?
assertTrue(Flags.a11yEnabled.testHasValue())
// Does it have the expected value?
assertTrue(Flags.a11yEnabled.testGetValue())
import org.mozilla.yourApplication.GleanMetrics.Flags;

Flags.INSTANCE.a11yEnabled.set(System.isAccessibilityEnabled());

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Flags;

// Was anything recorded?
assertTrue(Flags.INSTANCE.a11yEnabled.testHasValue());
// Does it have the expected value?
assertTrue(Flags.INSTANCE.a11yEnabled.testGetValue());
Flags.a11yEnabled.set(self.isAccessibilityEnabled)

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssertTrue(Flags.a11yEnabled.testHasValue())
// Does the counter have the expected value?
XCTAssertTrue(try Flags.a11yEnabled.testGetValue())
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

metrics.flags.a11y_enabled.set(is_accessibility_enabled())

There are test APIs available too:

# Was anything recorded?
assert metrics.flags.a11y_enabled.test_has_value()
# Does it have the expected value?
assert True is metrics.flags.a11y_enabled.test_get_value()
using static Mozilla.YourApplication.GleanMetrics.FlagsOuter;

Flags.a11yEnabled.Set(System.IsAccessibilityEnabled());

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.FlagsOuter;

// Was anything recorded?
Assert.True(Flags.a11yEnabled.TestHasValue());
// Does it have the expected value?
Assert.True(Flags.a11yEnabled.TestGetValue());

Limits

  • None.

Examples

  • Is a11y enabled?

Recorded errors

  • None.

Reference

Labeled Booleans

Labeled booleans are used to record different related boolean flags.

Configuration

For example, you may want to record a set of flags related to accessibility (a11y).

accessibility:
  features:
    type: labeled_boolean
    description: >
      a11y features enabled on the device. ...
    labels:
      - screen_reader
      - high_contrast
    ...

Note: removing or changing labels, including their order in the registry file, is permitted. Avoid reusing labels that were removed in the past. It is best practice to add documentation about removed labels to the description field so that analysts will know of their existence and meaning in historical data. Special care must be taken when changing GeckoView metrics sent through the Glean SDK, as the index of the labels is used to report Gecko data through the Glean SDK.

API

Now you can use the labeled boolean from the application's code:

import org.mozilla.yourApplication.GleanMetrics.Accessibility
Accessibility.features["screen_reader"].set(isScreenReaderEnabled())
Accessibility.features["high_contrast"].set(isHighContrastEnabled())

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Accessibility
// Was anything recorded?
assertTrue(Accessibility.features["screen_reader"].testHasValue())
assertTrue(Accessibility.features["high_contrast"].testHasValue())
// Do the booleans have the expected values?
assertEquals(True, Accessibility.features["screen_reader"].testGetValue())
assertEquals(False, Accessibility.features["high_contrast"].testGetValue())
// Did we record any invalid labels?
assertEquals(0, Accessibility.features.testGetNumRecordedErrors(ErrorType.InvalidLabel))
Accessibility.features["screen_reader"].set(isScreenReaderEnabled())
Accessibility.features["high_contrast"].set(isHighContrastEnabled())

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Accessibility.features["screen_reader"].testHasValue())
XCTAssert(Accessibility.features["high_contrast"].testHasValue())
// Do the booleans have the expected values?
XCTAssertEqual(true, try Accessibility.features["screen_reader"].testGetValue())
XCTAssertEqual(false, try Accessibility.features["high_contrast"].testGetValue())
// Were there any invalid labels?
XCTAssertEqual(0, Accessibility.features.testGetNumRecordedErrors(.invalidLabel))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

metrics.accessibility.features["screen_reader"].set(
    is_screen_reader_enabled()
)
metrics.accessibility.features["high_contrast"].set(
    is_high_contrast_enabled()
)

There are test APIs available too:

# Was anything recorded?
assert metrics.accessibility.features["screen_reader"].test_has_value()
assert metrics.accessibility.features["high_contrast"].test_has_value()
# Do the booleans have the expected values?
assert metrics.accessibility.features["screen_reader"].test_get_value()
assert not metrics.accessibility.features["high_contrast"].test_get_value()
# Did we record any invalid labels?
assert 0 == metrics.accessibility.features.test_get_num_recorded_errors(
    ErrorType.INVALID_LABEL
)
using static Mozilla.YourApplication.GleanMetrics.Accessibility;

Accessibility.features["screen_reader"].Set(isScreenReaderEnabled());
Accessibility.features["high_contrast"].Set(isHighContrastEnabled());

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Accessibility;
// Was anything recorded?
Assert.True(Accessibility.features["screen_reader"].TestHasValue());
Assert.True(Accessibility.features["high_contrast"].TestHasValue());
// Do the booleans have the expected values?
Assert.Equal(true, Accessibility.features["screen_reader"].TestGetValue());
Assert.Equal(false, Accessibility.features["high_contrast"].TestGetValue());
// Did we record any invalid labels?
Assert.Equal(0, Accessibility.features.TestGetNumRecordedErrors(ErrorType.InvalidLabel));

Limits

  • Labels must conform to the label formatting regular expression.

  • Labels support lowercase alphanumeric characters; they additionally allow for dots (.), underscores (_) and/or hyphens (-).

  • Each label must have a maximum of 60 bytes, when encoded as UTF-8.

  • If the labels are specified in the metrics.yaml, using any label not listed in that file will be replaced with the special value __other__.

  • If the labels aren't specified in the metrics.yaml, only 16 different dynamic labels may be used, after which the special value __other__ will be used.

Examples

  • Record a related set of boolean flags.

Recorded Errors

  • invalid_label: If the label contains invalid characters.

  • invalid_label: If the label exceeds the maximum number of allowed characters.

Reference

Counter

Used to count how often something happens, say how often a certain button was pressed. A counter always starts from 0. Each time you record to a counter, its value is incremented. Unless incremented by a positive value, a counter will not be reported in pings.

IMPORTANT: When using a counter metric, it is important to let the Glean metric do the counting. Using your own variable for counting and setting the counter yourself could be problematic because it will be difficult to reset the value at the exact moment that the value is sent in a ping. Instead, just use counter.add to increment the value and let Glean handle resetting the counter.

Configuration

Say you're adding a new counter for how often the refresh button is pressed. First you need to add an entry for the counter to the metrics.yaml file:

controls:
  refresh_pressed:
    type: counter
    description: >
      Counts how often the refresh button is pressed.
    ...

API

import org.mozilla.yourApplication.GleanMetrics.Controls

Controls.refreshPressed.add() // Adds 1 to the counter.
Controls.refreshPressed.add(5) // Adds 5 to the counter.

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Controls

// Was anything recorded?
assertTrue(Controls.refreshPressed.testHasValue())
// Does the counter have the expected value?
assertEquals(6, Controls.refreshPressed.testGetValue())
// Did the counter record an negative value?
assertEquals(
    1, Controls.refreshPressed.testGetNumRecordedErrors(ErrorType.InvalidValue)
)
import org.mozilla.yourApplication.GleanMetrics.Controls;

Controls.INSTANCE.refreshPressed.add(); // Adds 1 to the counter.
Controls.INSTANCE.refreshPressed.add(5); // Adds 5 to the counter.

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Controls;

// Was anything recorded?
assertTrue(Controls.INSTANCE.refreshPressed.testHasValue());
// Does the counter have the expected value?
assertEquals(6, Controls.INSTANCE.refreshPressed.testGetValue());
// Did the counter record an negative value?
assertEquals(
    1, Controls.INSTANCE.refreshPressed.testGetNumRecordedErrors(ErrorType.InvalidValue)
);
Controls.refreshPressed.add() // Adds 1 to the counter.
Controls.refreshPressed.add(5) // Adds 5 to the counter.

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Controls.refreshPressed.testHasValue())
// Does the counter have the expected value?
XCTAssertEqual(6, try Controls.refreshPressed.testGetValue())
// Did the counter record a negative value?
XCTAssertEqual(1, Controls.refreshPressed.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

metrics.controls.refresh_pressed.add()  # Adds 1 to the counter.
metrics.controls.refresh_pressed.add(5) # Adds 5 to the counter.

There are test APIs available too:

# Was anything recorded?
assert metrics.controls.refresh_pressed.test_has_value()
# Does the counter have the expected value?
assert 6 == metrics.controls.refresh_pressed.test_get_value()
# Did the counter record an negative value?
from glean.testing import ErrorType
assert 1 == metrics.controls.refresh_pressed.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Controls;

Controls.refreshPressed.Add(); // Adds 1 to the counter.
Controls.refreshPressed.Add(5); // Adds 5 to the counter.

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Controls;

// Was anything recorded?
Assert.True(Controls.refreshPressed.TestHasValue());
// Does the counter have the expected value?
Assert.Equal(6, Controls.refreshPressed.TestGetValue());
// Did the counter record a negative value?
Assert.Equal(
    1, Controls.refreshPressed.TestGetNumRecordedErrors(ErrorType.InvalidValue)
);

Limits

  • Only increments, saturates at the largest value that can be represented as a 32-bit signed integer (2147483647).

Examples

  • How often was a certain button was pressed?

Recorded errors

  • invalid_value: If the counter is incremented by 0 or a negative value.

Reference

Labeled Counters

Labeled counters are used to record different related counts that should sum up to a total.

Configuration

For example, you may want to record a count of different types of crashes for your Android application, such as native code crashes and uncaught exceptions:

stability:
  crash_count:
    type: labeled_counter
    description: >
      Counts the number of crashes that occur in the application. ...
    labels:
      - uncaught_exception
      - native_code_crash
    ...

Note: removing or changing labels, including their order in the registry file, is permitted. Avoid reusing labels that were removed in the past. It is best practice to add documentation about removed labels to the description field so that analysts will know of their existence and meaning in historical data. Special care must be taken when changing GeckoView metrics sent through the Glean SDK, as the index of the labels is used to report Gecko data through the Glean SDK.

API

Now you can use the labeled counter from the application's code:

import org.mozilla.yourApplication.GleanMetrics.Stability
Stability.crashCount["uncaught_exception"].add() // Adds 1 to the "uncaught_exception" counter.
Stability.crashCount["native_code_crash"].add(3) // Adds 3 to the "native_code_crash" counter.

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Stability
// Was anything recorded?
assertTrue(Stability.crashCount["uncaught_exception"].testHasValue())
assertTrue(Stability.crashCount["native_code_crash"].testHasValue())
// Do the counters have the expected values?
assertEquals(1, Stability.crashCount["uncaught_exception"].testGetValue())
assertEquals(3, Stability.crashCount["native_code_crash"].testGetValue())
// Were there any invalid labels?
assertEquals(0, Stability.crashCount.testGetNumRecordedErrors(ErrorType.InvalidLabel))
Stability.crashCount["uncaught_exception"].add() // Adds 1 to the "uncaught_exception" counter.
Stability.crashCount["native_code_crash"].add(3) // Adds 3 to the "native_code_crash" counter.

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Stability.crashCount["uncaught_exception"].testHasValue())
XCTAssert(Stability.crashCount["native_code_crash"].testHasValue())
// Do the counters have the expected values?
XCTAssertEqual(1, try Stability.crashCount["uncaught_exception"].testGetValue())
XCTAssertEqual(3, try Stability.crashCount["native_code_crash"].testGetValue())
// Were there any invalid labels?
XCTAssertEqual(0, Stability.crashCount.testGetNumRecordedErrors(.invalidLabel))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Adds 1 to the "uncaught_exception" counter.
metrics.stability.crash_count["uncaught_exception"].add()
# Adds 3 to the "native_code_crash" counter.
metrics.stability.crash_count["native_code_crash"].add(3)

There are test APIs available too:

# Was anything recorded?
assert metrics.stability.crash_count["uncaught_exception"].test_has_value()
assert metrics.stability.crash_count["native_code_crash"].test_has_value()
# Do the counters have the expected values?
assert 1 == metrics.stability.crash_count["uncaught_exception"].test_get_value()
assert 3 == metrics.stability.crash_count["native_code_crash"].test_get_value()
# Were there any invalid labels?
assert 0 == metrics.stability.crash_count.test_get_num_recorded_errors(
    ErrorType.INVALID_LABEL
)
using static Mozilla.YourApplication.GleanMetrics.Stability;
Stability.crashCount["uncaught_exception"].Add(); // Adds 1 to the "uncaught_exception" counter.
Stability.crashCount["native_code_crash"].Add(3); // Adds 3 to the "native_code_crash" counter.

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Stability;
// Was anything recorded?
Assert.True(Stability.crashCount["uncaught_exception"].TestHasValue());
Assert.True(Stability.crashCount["native_code_crash"].TestHasValue());
// Do the counters have the expected values?
Assert.Equal(1, Stability.crashCount["uncaught_exception"].TestGetValue());
Assert.Equal(3, Stability.crashCount["native_code_crash"].TestGetValue());
// Were there any invalid labels?
Assert.Equal(0, Stability.crashCount.TestGetNumRecordedErrors(ErrorType.InvalidLabel));

Limits

  • Labels must conform to the label formatting regular expression.

  • Labels support lowercase alphanumeric characters; they additionally allow for dots (.), underscores (_) and/or hyphens (-).

  • Each label must have a maximum of 60 bytes, when encoded as UTF-8.

  • If the labels are specified in the metrics.yaml, using any label not listed in that file will be replaced with the special value __other__.

  • If the labels aren't specified in the metrics.yaml, only 16 different dynamic labels may be used, after which the special value __other__ will be used.

Examples

  • Record the number of times different kinds of crashes occurred.

Recorded Errors

  • invalid_label: If the label contains invalid characters.

  • invalid_label: If the label exceeds the maximum number of allowed characters.

Reference

JWE

JWE metrics are supposed to be used as a transport for JWE encrypted data.

The encryption should happen before setting the metric and the decryption happens on the pipeline. This means that the Glean SDK is only in charge of validating that a given input is valid JWE and encryption and decryption are not part of its scope.

Configuration

You first need to add an entry for it to the metrics.yaml file:

user:
  anon_id:
    type: jwe
    description: >
      The JWE encrypted value of an anonymized ID.
    lifetime: user
    decrypted_name: anon_id_decrypted
    ...

API

Now that the JWE is defined in metrics.yaml, you can use the metric to record values in the application's code.

import org.mozilla.yourApplication.GleanMetrics.User

// Build a JWE from its elements and set the metric to it
User.anonId.set(
    "eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ",
    "",
    "48V1_ALb6US04U3b",
    "5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A",
    "XFBoMYUZodetZdvTiFvSkQ"
)

// Set the metric from a JWE compact representation
User.anonId.setWithCompactRepresentation("eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ..48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ")

There are test APIs available too.

import org.mozilla.yourApplication.GleanMetrics.User

// Was anything recorded?
assertTrue(User.anonId.testHasValue())

// Was it the expected value?

// Get a snapshot of the data
let snapshot = User.anonId.testGetValue()
assertEquals(header, snapshot.header)
assertEquals(key, snapshot.key)
assertEquals(initVector, snapshot.initVector)
assertEquals(cipherText, snapshot.cipherText)
assertEquals(authTag, snapshot.authTag)

// Or check against the compact representation
assertEquals(jwe, User.anonId.testGetCompactRepresentation())
import org.mozilla.yourApplication.GleanMetrics.User;

// Build a JWE from its elements and set the metric to it
User.INSTANCE.anonId.set(
    "eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ",
    "",
    "48V1_ALb6US04U3b",
    "5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A",
    "XFBoMYUZodetZdvTiFvSkQ"
);  

// Set the metric from a JWE compact representation
User.INSTANCE.anonId.setWithCompactRepresentation("eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ..48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ")

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.User;

// Was anything recorded?
assertTrue(User.INSTANCE.anonId.testHasValue());

// Was it the expected value?

// Get a snapshot of the data
JweData snapshot = User.INSTANCE.anonId.testGetValue();
assertEquals(header, snapshot.header);
assertEquals(key, snapshot.key);
assertEquals(initVector, snapshot.initVector);
assertEquals(cipherText, snapshot.cipherText);
assertEquals(authTag, snapshot.authTag);

// Or check against the compact representation
assertEquals(anonId, User.INSTANCE.anonId.testGetCompactRepresentation());
// Build a JWE from its elements and set the metric to it
User.anonId.set(
    "eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ",
    "",
    "48V1_ALb6US04U3b",
    "5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A",
    "XFBoMYUZodetZdvTiFvSkQ"
)

// Set the metric from a JWE compact representation
User.anonId.setWithCompactRepresentation("eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ..48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ")

There are test APIs available too.

@testable import Glean

// Was anything recorded?
XCTAssert(User.anonId.testHasValue())

// Was it the expected value?

// Get a snapshot of the data
let snapshot = try User.anonId.testGetValue()
XCTAssertEqual(header, snapshot.header)
XCTAssertEqual(key, snapshot.key)
XCTAssertEqual(initVector, snapshot.initVector)
XCTAssertEqual(cipherText, snapshot.cipherText)
XCTAssertEqual(authTag, snapshot.authTag)

// Or check against the compact representation
XCTAssertEqual(jwe, try User.anonId.testGetCompactRepresentation())
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Build a JWE from its elements and set the metric to it
metrics.user.anon_id.set(
  "eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ",
  "",
  "48V1_ALb6US04U3b",
  "5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A",
  "XFBoMYUZodetZdvTiFvSkQ"
)

# Set the metric from a JWE compact representation
metrics.user.anon_id.set_with_compact_representation("eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ..48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ")

There are test APIs available too.

# Was anything recorded?
assert metrics.user.anon_id.test_has_value()

# Was it the expected value?

# Get a snapshot of the data
snapshot = metrics.user.anon_id.test_get_value()
assert header == snapshot.header
assert key == snapshot.key
assert init_vector == snapshot.init_vector
assert cipher_text == snapshot.cipher_text
assert auth_tag == snapshot.auth_tag

# Or check against the compact representation
assert anon_id == metrics.user.anon_id.test_get_compact_representation()
using static Mozilla.YourApplication.GleanMetrics.User;

// Build a JWE from its elements and set the metric to it
User.anonId.set(
    "eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ",
    "",
    "48V1_ALb6US04U3b",
    "5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A",
    "XFBoMYUZodetZdvTiFvSkQ"
)

// Set the metric from a JWE compact representation
User.anonId.setWithCompactRepresentation("eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ..48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ")

There are test APIs available too:

using Mozilla.Glean;
using static Mozilla.YourApplication.GleanMetrics.User;

// Was anything recorded?
Assert.True(User.anonId.TestHasValue());

// Was it the expected value?

// Get a snapshot of the data
Mozilla.Glean.Private.JweData snapshot = User.anonId.testGetValue()
Assert.Equals(header, snapshot.Header)
Assert.Equals(key, snapshot.Key)
Assert.Equals(initVector, snapshot.InitVector)
Assert.Equals(cipherText, snapshot.CipherText)
Assert.Equals(authTag, snapshot.AuthTag)

// Or check against the compact representation
Assert.Equals(jwe, User.anonId.testGetCompactRepresentation())

Limits

  • Variable sized elements of the JWE must not exceed 1024 characters. These elements are header, key and cipher_text.

Examples

  • An anonymized identifier.

Recorded errors

  • invalid_value:
    • if the compact representation passed to set_with_compact_representation is not valid;
    • if any one of the elements of the JWE is not valid BASE64URL;
    • if header or cipher_text elements are empty strings.
  • invalid_overflow:
    • if header, key or cipher_text elements exceed 1024 characters;
    • if init_vector element is not empty or exactly 96-bits;
    • if auth_tag element is not empty or exactly 128-bits.

Reference

Strings

This allows recording a Unicode string value with arbitrary content.

Note: Be careful using arbitrary strings and make sure they can't accidentally contain identifying data (like directory paths or user input).

Note: This is does not support recording JSON blobs - please get in contact with the Telemetry team if you're missing a type.

Configuration

Say you're adding a metric to find out what the default search in a browser is. First you need to add an entry for the metric to the metrics.yaml file:

search.default:
  name:
    type: string
    description: >
      The name of the default search engine.
    lifetime: application
    ...

API

import org.mozilla.yourApplication.GleanMetrics.SearchDefault

// Record a value into the metric.
SearchDefault.name.set("duck duck go")
// If it changed later, you can record the new value:
SearchDefault.name.set("wikipedia")

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.SearchDefault

// Was anything recorded?
assertTrue(SearchDefault.name.testHasValue())
// Does the string metric have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
assertEquals("wikipedia", SearchDefault.name.testGetValue())
// Was the string truncated, and an error reported?
assertEquals(1, SearchDefault.name.testGetNumRecordedErrors(ErrorType.InvalidValue))
import org.mozilla.yourApplication.GleanMetrics.SearchDefault;

// Record a value into the metric.
SearchDefault.INSTANCE.name.set("duck duck go");
// If it changed later, you can record the new value:
SearchDefault.INSTANCE.name.set("wikipedia");

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.SearchDefault

// Was anything recorded?
assertTrue(SearchDefault.INSTANCE.name.testHasValue());
// Does the string metric have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
assertEquals("wikipedia", SearchDefault.INSTANCE.name.testGetValue());
// Was the string truncated, and an error reported?
assertEquals(
    1,
    SearchDefault.INSTANCE.name.testGetNumRecordedErrors(
        ErrorType.InvalidValue
    )
);
// Record a value into the metric.
SearchDefault.name.set("duck duck go")
// If it changed later, you can record the new value:
SearchDefault.name.set("wikipedia")

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(SearchDefault.name.testHasValue())
// Does the string metric have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
XCTAssertEqual("wikipedia", try SearchDefault.name.testGetValue())
// Was the string truncated, and an error reported?
XCTAssertEqual(1, SearchDefault.name.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Record a value into the metric.
metrics.search_default.name.set("duck duck go")
# If it changed later, you can record the new value:
metrics.search_default.name.set("wikipedia")

There are test APIs available too:

# Was anything recorded?
assert metrics.search_default.name.test_has_value()
# Does the string metric have the expected value?
# IMPORTANT: It may have been truncated -- see "Limits" below
assert "wikipedia" == metrics.search_default.name.test_get_value()
# Was the string truncated, and an error reported?
assert 1 == metrics.search_default.name.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.SearchDefault;

// Record a value into the metric.
SearchDefault.name.Set("duck duck go");
// If it changed later, you can record the new value:
SearchDefault.name.Set("wikipedia");

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.SearchDefault;

// Was anything recorded?
Assert.True(SearchDefault.name.TestHasValue());
// Does the string metric have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
Assert.Equal("wikipedia", SearchDefault.name.TestGetValue());
// Was the string truncated, and an error reported?
Assert.Equal(
    1,
    SearchDefault.name.TestGetNumRecordedErrors(
        ErrorType.InvalidValue
    )
);

Limits

  • Fixed maximum string length: 100. Longer strings are truncated. This is measured in the number of bytes when the string is encoded in UTF-8.

Examples

  • Record the operating system name with a value of "android".

  • Recording the device model with a value of "SAMSUNG-SGH-I997".

Recorded errors

  • invalid_overflow: if the string is too long. (Prior to Glean 31.5.0, this recorded an invalid_value).

Reference

Labeled Strings

Labeled strings record multiple Unicode string values, each under a different label.

Configuration

For example to record which kind of error occurred in different stages of a login process - "RuntimeException" in the "server_auth" stage or "invalid_string" in the "enter_email" stage:

login:
  errors_by_stage:
    type: labeled_string
    description: Records the error type, if any, that occur in different stages of the login process.
    labels:
      - server_auth
      - enter_email
    ...

API

Now you can use the labeled string from the application's code:

import org.mozilla.yourApplication.GleanMetrics.Login

Login.errorsByStage["server_auth"].set("Invalid password")

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Login

// Was anything recorded?
assertTrue(Login.errorsByStage["server_auth"].testHasValue())

// Were there any invalid labels?
assertEquals(0, Login.errorsByStage.testGetNumRecordedErrors(ErrorType.InvalidLabel))
Login.errorsByStage["server_auth"].set("Invalid password")

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Login.errorsByStage["server_auth"].testHasValue())

// Were there any invalid labels?
XCTAssertEqual(0, Login.errorsByStage.testGetNumRecordedErrors(.invalidLabel))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

metrics.login.errors_by_stage["server_auth"].set("Invalid password")

There are test APIs available too:

# Was anything recorded?
assert metrics.login.errors_by_stage["server_auth"].test_has_value()

# Were there any invalid labels?
assert 0 == metrics.login.errors_by_stage.test_get_num_recorded_errors(
    ErrorType.INVALID_LABEL
)
using static Mozilla.YourApplication.GleanMetrics.Login;

Login.errorsByStage["server_auth"].Set("Invalid password");

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Login;

// Was anything recorded?
Assert.True(Login.errorsByStage["server_auth"].TestHasValue());

// Were there any invalid labels?
Assert.Equal(0, Login.errorsByStage.TestGetNumRecordedErrors(ErrorType.InvalidLabel));

Limits

  • Labels must conform to the label formatting regular expression.

  • Each label must have a maximum of 60 bytes, when encoded as UTF-8.

  • If the labels are specified in the metrics.yaml, using any label not listed in that file will be replaced with the special value __other__.

  • If the labels aren't specified in the metrics.yaml, only 16 different dynamic labels may be used, after which the special value __other__ will be used.

Examples

  • What kind of errors occurred at each step in the login process?

Recorded Errors

  • invalid_label: If the label contains invalid characters.

  • invalid_label: If the label exceeds the maximum number of allowed characters.

Reference

String List

Strings lists are used for recording a list of Unicode string values, such as the names of the enabled search engines.


Note: Be careful using arbitrary strings and make sure they can't accidentally contain identifying data (like directory paths or user input).


Configuration

First you need to add an entry for the counter to the metrics.yaml file:

search:
  engines:
    type: string_list
    description: >
      Records the name of the enabled search engines.
    lifetime: application
    ...

API

import org.mozilla.yourApplication.GleanMetrics.Search

// Add them one at a time
engines.forEach {
  Search.engines.add(it)
}

// Set them in one go
Search.engines.set(engines)

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Search

// Was anything recorded?
assertTrue(Search.engines.testHasValue())
// Does it have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
assertEquals(listOf("Google", "DuckDuckGo"), Search.engines.testGetValue())
// Were any of the values too long, and thus an error was recorded?
assertEquals(1, Search.engines.testGetNumRecordedErrors(ErrorType.InvalidValue))
// Add them one at a time
for engine in engines {
    Search.engines.add(engine)
}

// Set them in one go
Search.engines.set(engines)

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Search.engines.testHasValue())
// Does it have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
XCTAssertEqual(["Google", "DuckDuckGo"], try Search.engines.testGetValue())
// Were any of the values too long, and thus an error was recorded?
XCTAssertEqual(1, Search.engines.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics

metrics = load_metrics("metrics.yaml")

# Add them one at a time
for engine in engines:
    metrics.search.engines.add(engine)

# Set them in one go
metrics.search.engines.set(engines)

There are test APIs available too:

# Was anything recorded?
assert metrics.search.engines.test_has_value()
# Does it have the expected value?
# IMPORTANT: It may have been truncated -- see "Limits" below
assert ["Google", "DuckDuckGo"] == metrics.search.engines.test_get_value()
# Were any of the values too long, and thus an error was recorded?
assert 1 == metrics.search.engines.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Search;

// Set a string array into the metric.
Search.engines.Set(new string[] { "Google", "DuckDuckGo" });
// Add another string into the metric.
Search.engines.Add("Baidu");

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Search;

// Was anything recorded?
Assert.True(Search.engines.TestHasValue());
// Does the string list metric have the expected value?
// IMPORTANT: It may have been truncated -- see "Limits" below
var snapshot = Search.engines.TestGetValue();
Assert.Equal(3, Search.engines.Length);
Assert.Equal("Google", Search.engines[0]);
Assert.Equal("DuckDuckGo", Search.engines[1]);
Assert.Equal("Baidu", Search.engines[2]);
// Was the string truncated, and an error reported?
Assert.Equal(
    1,
    Search.engines.TestGetNumRecordedErrors(
        ErrorType.InvalidValue
    )
);

Limits

  • Fixed maximum string length: 50. Longer strings are truncated. This is measured in the number of bytes when the string is encoded in UTF-8.

  • Fixed maximum list length: 20 items. Additional strings are dropped.

Examples

  • The names of the enabled search engines.

Recorded errors

  • invalid_overflow: if the string is too long. (Prior to Glean 31.5.0, this recorded an invalid_value).

  • invalid_value: if the list is too long

Reference

Timespan

Timespans are used to make a measurement of how much time is spent in a particular task.

To measure the distribution of multiple timespans, see Timing Distributions. To record absolute times, see Datetimes.

It is not recommended to use timespans in multiple threads, since calling start or stop out of order will be recorded as an invalid_state error.

Configuration

Timespans have a required time_unit parameter to specify the smallest unit of resolution that the timespan will record. The allowed values for time_unit are:

  • nanosecond
  • microsecond
  • millisecond
  • second
  • minute
  • hour
  • day

Consider the resolution that is required by your metric, and use the largest possible value that will provide useful information so as to not leak too much fine-grained information from the client. It is important to note that the value sent in the ping is truncated down to the nearest unit. Therefore, a measurement of 500 nanoseconds will be truncated to 0 microseconds.

Say you're adding a new timespan for the time spent logging into the app. First you need to add an entry for the counter to the metrics.yaml file:

auth:
  login_time:
    type: timespan
    description: >
      Measures the time spent logging in.
    time_unit: milliseconds
    ...

API

import org.mozilla.yourApplication.GleanMetrics.Auth

fun onShowLogin() {
    Auth.loginTime.start()
    // ...
}

fun onLogin() {
    Auth.loginTime.stop()
    // ...
}

fun onLoginCancel() {
    Auth.loginTime.cancel()
    // ...
}

The time reported in the telemetry ping will be timespan recorded during the lifetime of the ping.

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Auth

// Was anything recorded?
assertTrue(Auth.loginTime.testHasValue())
// Does the timer have the expected value
assertTrue(Auth.loginTime.testGetValue() > 0)
// Was the timing recorded incorrectly?
assertEquals(1, Auth.loginTime.testGetNumRecordedErrors(ErrorType.InvalidValue))
import org.mozilla.yourApplication.GleanMetrics.Auth;

void onShowLogin() {
    Auth.INSTANCE.loginTime.start();
    // ...
}

void onLogin() {
    Auth.INSTANCE.loginTime.stop();
    // ...
}

void onLoginCancel() {
    Auth.INSTANCE.loginTime.cancel();
    // ...
}

The time reported in the telemetry ping will be timespan recorded during the lifetime of the ping.

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Auth;

// Was anything recorded?
assertTrue(Auth.INSTANCE.loginTime.testHasValue());
// Does the timer have the expected value
assertTrue(Auth.INSTANCE.loginTime.testGetValue() > 0);
// Was the timing recorded incorrectly?
assertEquals(
    1,
    Auth.INSTANCE.loginTime.testGetNumRecordedErrors(
        ErrorType.InvalidValue
    )
);
func onShowLogin() {
    Auth.loginTime.start()
    // ...
}

func onLogin() {
    Auth.loginTime.stop()
    // ...
}

func onLoginCancel() {
    Auth.loginTime.cancel()
    // ...
}

The time reported in the telemetry ping will be timespan recorded during the lifetime of the ping.

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Auth.loginTime.testHasValue())
// Does the timer have the expected value
XCTAssert(try Auth.loginTime.testGetValue() > 0)
// Was the timing recorded incorrectly?
XCTAssertEqual(1, Auth.loginTime.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

def on_show_login():
    metrics.auth.login_time.start()
    # ...

def on_login():
    metrics.auth.login_time.stop()
    # ...

def on_login_cancel():
    metrics.auth.login_time.cancel()
    # ...

The Python bindings also have a context manager for measuring time:

with metrics.auth.login_time.measure():
    # ... Do the login ...

The time reported in the telemetry ping will be timespan recorded during the lifetime of the ping.

There are test APIs available too:

# Was anything recorded?
assert metrics.auth.login_time.test_has_value()
# Does the timer have the expected value
assert metrics.auth.login_time.test_get_value() > 0
# Was the timing recorded incorrectly?
assert 1 == metrics.auth.local_time.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Auth;

void OnShowLogin()
{
    Auth.loginTime.Start();
    // ...
}

void OnLogin()
{
    Auth.loginTime.Stop();
    // ...
}

void OnLoginCancel()
{
    Auth.loginTime.Cancel();
    // ...
}

The time reported in the telemetry ping will be timespan recorded during the lifetime of the ping.

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Auth;

// Was anything recorded?
Assert.True(Auth.loginTime.TestHasValue());
// Does the timer have the expected value
Assert.True(Auth.loginTime.TestGetValue() > 0);
// Was the timing recorded incorrectly?
Assert.Equals(1, Auth.loginTime.TestGetNumRecordedErrors(ErrorType.InvalidValue));

Raw API

Note: The raw API was designed to support a specific set of use-cases. Please consider using the higher level APIs listed above.

It's possible to explicitly set the timespan value, in nanoseconds. This API should only be used if your library or application requires recording times in a way that can not make use of start/stop/cancel.

The raw API will not overwrite a running timer or existing timespan value.

import org.mozilla.yourApplication.GleanMetrics.HistorySync

val duration = SyncResult.status.syncs.took.toLong()
HistorySync.setRawNanos(duration)
let duration = SyncResult.status.syncs.took.toLong()
HistorySync.setRawNanos(duration)
import org.mozilla.yourApplication.GleanMetrics.HistorySync

val duration = SyncResult.status.syncs.took.toLong()
HistorySync.setRawNanos(duration)

TODO. To be implemented in bug 1648442.

Limits

  • Timings are recorded in nanoseconds.

    • On Android, the SystemClock.elapsedRealtimeNanos() function is used, so it is limited by the accuracy and performance of that timer. The time measurement includes time spent in sleep.

    • On iOS, the mach_absolute_time function is used, so it is limited by the accuracy and performance of that timer. The time measurement does not include time spent in sleep.

    • On Python 3.7 and later, time.monotonic_ns() is used. On earlier versions of Python, time.monotonics() is used, which is not guaranteed to have nanosecond resolution.

Examples

  • How much time is spent rendering the UI?

Recorded errors

  • invalid_value
    • If recording a negative timespan.
  • invalid_state
    • If starting a timer while a previous timer is running.
    • If stopping a timer while it is not running.
    • If trying to set a raw timespan while a timer is running.

Reference

Timing Distribution

Timing distributions are used to accumulate and store time measurement, for analyzing distributions of the timing data.

To measure the distribution of single timespans, see Timespans. To record absolute times, see Datetimes.

Timing distributions are recorded in a histogram where the buckets have an exponential distribution, specifically with 8 buckets for every power of 2. That is, the function from a value \( x \) to a bucket index is:

\[ \lfloor 8 \log_2(x) \rfloor \]

This makes them suitable for measuring timings on a number of time scales without any configuration.

Note Check out how this bucketing algorithm would behave on the Simulator

Timings always span the full length between start and stopAndAccumulate. If the Glean upload is disabled when calling start, the timer is still started. If the Glean upload is disabled at the time stopAndAccumulate is called, nothing is recorded.

Multiple concurrent timespans in different threads may be measured at the same time.

Timings are always stored and sent in the payload as nanoseconds. However, the time_unit parameter controls the minimum and maximum values that will recorded:

  • nanosecond: 1ns <= x <= 10 minutes
  • microsecond: 1μs <= x <= ~6.94 days
  • millisecond: 1ms <= x <= ~19 years

Overflowing this range is considered an error and is reported through the error reporting mechanism. Underflowing this range is not an error and the value is silently truncated to the minimum value.

Additionally, when a metric comes from GeckoView (the geckoview_datapoint parameter is present), the time_unit parameter specifies the unit that the samples are in when passed to Glean. Glean will convert all of the incoming samples to nanoseconds internally.

Configuration

If you wanted to create a timing distribution to measure page load times, first you need to add an entry for it to the metrics.yaml file:

pages:
  page_load:
    type: timing_distribution
    description: >
      Counts how long each page takes to load
    ...

API

Now you can use the timing distribution from the application's code. Starting a timer returns a timer ID that needs to be used to stop or cancel the timer at a later point. Multiple intervals can be measured concurrently. For example, to measure page load time on a number of tabs that are loading at the same time, each tab object needs to store the running timer ID.

import mozilla.components.service.glean.GleanTimerId
import org.mozilla.yourApplication.GleanMetrics.Pages

val timerId : GleanTimerId

fun onPageStart(e: Event) {
    timerId = Pages.pageLoad.start()
}

fun onPageLoaded(e: Event) {
    Pages.pageLoad.stopAndAccumulate(timerId)
}

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the pageLoad example above, at this point the metric should have a sum == 11 and a count == 2:

import org.mozilla.yourApplication.GleanMetrics.Pages

// Was anything recorded?
assertTrue(pages.pageLoad.testHasValue())

// Get snapshot.
val snapshot = pages.pageLoad.testGetValue()

// Does the sum have the expected value?
assertEquals(11, snapshot.sum)

// Usually you don't know the exact timing values, but how many should have been recorded.
assertEquals(2L, snapshot.count)

// Was an error recorded?
assertEquals(1, pages.pageLoad.testGetNumRecordedErrors(ErrorType.InvalidValue))
import mozilla.components.service.glean.GleanTimerId;
import org.mozilla.yourApplication.GleanMetrics.Pages;

GleanTimerId timerId;

void onPageStart(Event e) {
    timerId = Pages.INSTANCE.pageLoad.start();
}

void onPageLoaded(Event e) {
    Pages.INSTANCE.pageLoad.stopAndAccumulate(timerId);
}

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the pageLoad example above, at this point the metric should have a sum == 11 and a count == 2:

import org.mozilla.yourApplication.GleanMetrics.Pages;

// Was anything recorded?
assertTrue(pages.INSTANCE.pageLoad.testHasValue());

// Get snapshot.
DistributionData snapshot = pages.INSTANCE.pageLoad.testGetValue();

// Does the sum have the expected value?
assertEquals(11, snapshot.getSum);

// Usually you don't know the exact timing values, but how many should have been recorded.
assertEquals(2L, snapshot.getCount);

// Was an error recorded?
assertEquals(
    1,
    pages.INSTANCE.pageLoad.testGetNumRecordedErrors(
        ErrorType.InvalidValue
    )
);
import Glean

var timerId : GleanTimerId

func onPageStart() {
    timerId = Pages.pageLoad.start()
}

func onPageLoaded() {
    Pages.pageLoad.stopAndAccumulate(timerId)
}

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the pageLoad example above, at this point the metric should have a sum == 11 and a count == 2:

@testable import Glean

// Was anything recorded?
XCTAssert(pages.pageLoad.testHasValue())

// Get snapshot.
let snapshot = try! pages.pageLoad.testGetValue()

// Does the sum have the expected value?
XCTAssertEqual(11, snapshot.sum)

// Usually you don't know the exact timing values, but how many should have been recorded.
XCTAssertEqual(2, snapshot.count)

// Was an error recorded?
XCTAssertEqual(1, pages.pageLoad.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

class PageHandler:
    def __init__(self):
        self.timer_id = None

    def on_page_start(self, event):
        # ...
        self.timer_id = metrics.pages.page_load.start()

    def on_page_loaded(self, event):
        # ...
        metrics.pages.page_load.stop_and_accumulate(self.timer_id)

The Python bindings also have a context manager for measuring time:

with metrics.pages.page_load.measure():
    # Load a page ...

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the page_load example above, at this point the metric should have a sum == 11 and a count == 2:

# Was anything recorded?
assert metrics.pages.page_load.test_has_value()

# Get snapshot.
snapshot = metrics.pages.page_load.test_get_value()

# Does the sum have the expected value?
assert 11 == snapshot.sum

# Usually you don't know the exact timing values, but how many should have been recorded.
assert 2 == snapshot.count

# Was an error recorded?
assert 1 == metrics.pages.page_load.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Pages;

var timerId;

void onPageStart(Event e) {
    timerId = Pages.pageLoad.Start();
}

void onPageLoaded(Event e) {
    Pages.pageLoad.StopAndAccumulate(timerId);
}

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the pageLoad example above, at this point the metric should have a sum == 11 and a count == 2:

using static Mozilla.YourApplication.GleanMetrics.Pages;

// Was anything recorded?
Assert.True(Pages.pageLoad.TestHasValue());

// Get snapshot.
var snapshot = Pages.pageLoad.TestGetValue();

// Does the sum have the expected value?
Assert.Equal(11, snapshot.Sum);

// Usually you don't know the exact timing values, but how many should have been recorded.
Assert.Equal(2L, snapshot.Values.Count);

// Was an error recorded?
Assert.Equal(1, Pages.pageLoad.TestGetNumRecordedErrors(ErrorType.InvalidValue));

Limits

  • Timings are recorded in nanoseconds.

    • On Android, the SystemClock.elapsedRealtimeNanos() function is used, so it is limited by the accuracy and performance of that timer. The time measurement includes time spent in sleep.

    • On iOS, the mach_absolute_time function is used, so it is limited by the accuracy and performance of that timer. The time measurement does not include time spent in sleep.

    • On Python 3.7 and later, time.monotonic_ns() is used. On earlier versions of Python, time.monotonics() is used, which is not guaranteed to have nanosecond resolution.

  • The maximum timing value that will be recorded depends on the time_unit parameter:

    • nanosecond: 1ns <= x <= 10 minutes
    • microsecond: 1μs <= x <= ~6.94 days
    • millisecond: 1ms <= x <= ~19 years Longer times will be truncated to the maximum value and an error will be recorded.

Examples

  • How long does it take a page to load?

Recorded errors

  • invalid_value: If recording a negative timespan.
  • invalid_state: If a non-existing/stopped timer is stopped again.
  • invalid_overflow: If recording a time longer than the maximum for the given unit.

Reference

Simulator

Please, insert your custom data below as a JSON array.

Data options

Properties

Note The data provided, is assumed to be in the configured time unit. The data recorded, on the other hand, is always in nanoseconds. This means that, if the configured time unit is not nanoseconds, the data will be transformed before being recorded. Notice this, by using the select field above to change the time unit and see the mean of the data recorded changing.

Memory Distribution

Memory distributions are used to accumulate and store memory sizes.

Memory distributions are recorded in a histogram where the buckets have an exponential distribution, specifically with 16 buckets for every power of 2. That is, the function from a value \( x \) to a bucket index is:

\[ \lfloor 16 \log_2(x) \rfloor \]

This makes them suitable for measuring memory sizes on a number of different scales without any configuration.

Note Check out how this bucketing algorithm would behave on the Simulator

Configuration

If you wanted to create a memory distribution to measure the amount of heap memory allocated, first you need to add an entry for it to the metrics.yaml file:

memory:
  heap_allocated:
    type: memory_distribution
    description: >
      The heap memory allocated
    memory_unit: kilobyte
    ...

API

Now you can use the memory distribution from the application's code.

For example, to measure the distribution of heap allocations:

import org.mozilla.yourApplication.GleanMetrics.Memory

fun allocateMemory(nbytes: Int) {
    // ...
    Memory.heapAllocated.accumulate(nbytes / 1024)
}

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the heapAllocated example above, at this point the metric should have a sum == 11 and a count == 2:

import org.mozilla.yourApplication.GleanMetrics.Memory

// Was anything recorded?
assertTrue(Memory.heapAllocated.testHasValue())

// Get snapshot
val snapshot = Memory.heapAllocated.testGetValue()

// Does the sum have the expected value?
assertEquals(11, snapshot.sum)

// Usually you don't know the exact memory values, but how many should have been recorded.
assertEquals(2L, snapshot.count)

// Did this record a negative value?
assertEquals(1, Memory.heapAllocated.testGetNumRecordedErrors(ErrorType.InvalidValue))
func allocateMemory(nbytes: UInt64) {
    // ...
    Memory.heapAllocated.accumulate(nbytes / 1024)
}

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the heapAllocated example above, at this point the metric should have a sum == 11 and a count == 2:

@testable import Glean

// Was anything recorded?
XCTAssert(Memory.heapAllocated.testHasValue())

// Get snapshot
let snapshot = try! Memory.heapAllocated.testGetValue()

// Does the sum have the expected value?
XCTAssertEqual(11, snapshot.sum)

// Usually you don't know the exact memory values, but how many should have been recorded.
XCTAssertEqual(2, snapshot.count)

// Did this record a negative value?
XCTAssertEqual(1, Memory.heapAllocated.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

def allocate_memory(nbytes):
    # ...
    metrics.memory.heap_allocated.accumulate(nbytes / 1024)

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

Continuing the heapAllocated example above, at this point the metric should have a sum == 11 and a count == 2:

# Was anything recorded?
assert metrics.memory.head_allocated.test_has_value()

# Get snapshot
snapshot = metrics.memory.heap_allocated.test_get_value()

# Does the sum have the expected value?
assert 11 == snapshot.sum

# Usually you don't know the exact memory values, but how many should have been recorded.
assert 2 == snapshot.count

# Did this record a negative value?
assert 1 == metrics.memory.heap_allocated.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Memory;

fun allocateMemory(ulong nbytes) {
    // ...
    Memory.heapAllocated.Accumulate(nbytes / 1024);
}

There are test APIs available too. For convenience, properties Sum and Count are exposed to facilitate validating that data was recorded correctly.

Continuing the heapAllocated example above, at this point the metric should have a Sum == 11 and a Count == 2:

using static Mozilla.YourApplication.GleanMetrics.Memory;

// Was anything recorded?
Assert.True(Memory.heapAllocated.TestHasValue());

// Get snapshot
var snapshot = Memory.heapAllocated.TestGetValue();

// Does the sum have the expected value?
Assert.Equal(11, snapshot.Sum);

// Usually you don't know the exact memory values, but how many should have been recorded.
Assert.Equal(2L, snapshot.Count);

// Did this record a negative value?
Assert.Equal(1, Memory.heapAllocated.TestGetNumRecordedErrors(ErrorType.InvalidValue));

Limits

  • The maximum memory size that can be recorded is 1 Terabyte (240 bytes). Larger sizes will be truncated to 1 Terabyte.

Examples

  • What is the distribution of the size of heap allocations?

Recorded errors

  • invalid_value: If recording a negative memory size.
  • invalid_value: If recording a size larger than 1TB.

Reference

Simulator

Please, insert your custom data below as a JSON array.

Data options

Properties

Note The data provided, is assumed to be in the configured memory unit. The data recorded, on the other hand, is always in bytes. This means that, if the configured memory unit is not byte, the data will be transformed before being recorded. Notice this, by using the select field above to change the memory unit and see the mean of the data recorded changing.

UUID

UUIDs are used to record values that uniquely identify some entity, such as a client id.

Configuration

You first need to add an entry for it to the metrics.yaml file:

user:
  client_id:
    type: uuid
    description: >
      A unique identifier for the client's profile
    lifetime: user
    ...

API

Now that the UUID is defined in metrics.yaml, you can use the metric to record values in the application's code.

import org.mozilla.yourApplication.GleanMetrics.User

User.clientId.generateAndSet() // Generate a new UUID and record it
User.clientId.set(UUID.randomUUID())  // Set a UUID explicitly

There are test APIs available too.

import org.mozilla.yourApplication.GleanMetrics.User

// Was anything recorded?
assertTrue(User.clientId.testHasValue())
// Was it the expected value?
assertEquals(uuid, User.clientId.testGetValue())
import org.mozilla.yourApplication.GleanMetrics.User;

User.INSTANCE.clientId.generateAndSet(); // Generate a new UUID and record it
User.INSTANCE.clientId.set(UUID.randomUUID());  // Set a UUID explicitly

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.User;

// Was anything recorded?
assertTrue(User.INSTANCE.clientId.testHasValue());
// Was it the expected value?
assertEquals(uuid, User.INSTANCE.clientId.testGetValue());
User.clientId.generateAndSet() // Generate a new UUID and record it
User.clientId.set(UUID())  // Set a UUID explicitly

There are test APIs available too.

@testable import Glean

// Was anything recorded?
XCTAssert(User.clientId.testHasValue())
// Was it the expected value?
XCTAssertEqual(uuid, try User.clientId.testGetValue())
import uuid

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Generate a new UUID and record it
metrics.user.client_id.generate_and_set()
# Set a UUID explicitly
metrics.user.client_id.set(uuid.uuid4())

There are test APIs available too.

# Was anything recorded?
assert metrics.user.client_id.test_has_value()
# Was it the expected value?
assert uuid == metrics.user.client_id.test_get_value()
using static Mozilla.YourApplication.GleanMetrics.User;

User.clientId.GenerateAndSet(); // Generate a new UUID and record it
User.clientId.Set(System.Guid.NewGuid()); // Set a UUID explicitly

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.User;

// Was anything recorded?
Assert.True(User.clientId.TestHasValue());
// Was it the expected value?
Assert.Equal(uuid, User.clientId.TestGetValue());

Limits

  • None.

Examples

  • A unique identifier for the client.

Recorded errors

  • None.

Reference

Datetime

Datetimes are used to record an absolute date and time, for example the date and time that the application was first run.

The device's offset from UTC is recorded and sent with the Datetime value in the ping.

To record a single elapsed time, see Timespan. To measure the distribution of multiple timespans, see Timing Distributions.

Configuration

Datetimes have a required time_unit parameter to specify the smallest unit of resolution that the timespan will record. The allowed values for time_unit are:

  • nanosecond
  • microsecond
  • millisecond
  • second
  • minute
  • hour
  • day

Carefully consider the required resolution for recording your metric, and choose the coarsest resolution possible.

You first need to add an entry for it to the metrics.yaml file:

install:
  first_run:
    type: datetime
    time_unit: day
    description: >
      Records the date when the application was first run
    lifetime: user
    ...

API

import org.mozilla.yourApplication.GleanMetrics.Install

Install.firstRun.set() // Records "now"
Install.firstRun.set(Calendar(2019, 3, 25)) // Records a custom datetime

There are test APIs available too.

import org.mozilla.yourApplication.GleanMetrics.Install

// Was anything recorded?
assertTrue(Install.firstRun.testHasValue())
// Was it the expected value?
// NOTE: Datetimes always include a timezone offset from UTC, hence the
// "-05:00" suffix.
assertEquals("2019-03-25-05:00", Install.firstRun.testGetValueAsString())
// Was the value invalid?
assertEquals(1, Install.firstRun.testGetNumRecordedErrors(ErrorType.InvalidValue))
import org.mozilla.yourApplication.GleanMetrics.Install;

Install.INSTANCE.firstRun.set(); // Records "now"
Install.INSTANCE.firstRun.set(Calendar(2019, 3, 25)); // Records a custom datetime

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Install;

// Was anything recorded?
assertTrue(Install.INSTANCE.firstRun.testHasValue());
// Was it the expected value?
// NOTE: Datetimes always include a timezone offset from UTC, hence the
// "-05:00" suffix.
assertEquals("2019-03-25-05:00", Install.INSTANCE.firstRun.testGetValueAsString());
// Was the value invalid?
assertEquals(1, Install.INSTANCE.firstRun.testGetNumRecordedErrors(ErrorType.InvalidValue));
Install.firstRun.set() // Records "now"

let dateComponents = DateComponents(
                        calendar: Calendar.current,
                        year: 2004, month: 12, day: 9, hour: 8, minute: 3, second: 29
                     )
Install.firstRun.set(dateComponents.date!) // Records a custom datetime

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Install.firstRun.testHasValue())
// Does the datetime have the expected value?
XCTAssertEqual(6, try Install.firstRun.testGetValue())
// Was the value invalid?
XCTAssertEqual(1, Install.firstRun.getNumRecordedErrors(.invalidValue))
import datetime

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

# Records "now"
metrics.install.first_run.set()
# Records a custom datetime
metrics.install.first_run.set(datetime.datetime(2019, 3, 25))

There are test APIs available too.

# Was anything recorded?
assert metrics.install.first_run.test_has_value()

# Was it the expected value?
# NOTE: Datetimes always include a timezone offset from UTC, hence the
# "-05:00" suffix.
assert "2019-03-25-05:00" == metrics.install.first_run.test_get_value_as_str()
# Was the value invalid?
assert 1 == metrics.install.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Install;

// Records "now"
Install.firstRun.Set();
// Records a custom datetime
Install.firstRun.Set(new DateTimeOffset(2018, 2, 25, 11, 10, 0, TimeZone.CurrentTimeZone.BaseUtcOffset));

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Install;

// Was anything recorded?
Assert.True(Install.firstRun.TestHasValue());
// Was it the expected value?
// NOTE: Datetimes always include a timezone offset from UTC, hence the
// "-05:00" suffix.
Assert.Equal("2019-03-25-05:00", Install.firstRun.TestGetValueAsString());
// Was the value invalid?
Assert.Equal(1, Install.firstRun.TestGetNumRecordedErrors(ErrorType.InvalidValue));

Limits

  • None.

Examples

  • When did the user first run the application?

Recorded errors

  • invalid_value: Setting the date time to an invalid value.

Reference

Events

Events allow recording of e.g. individual occurrences of user actions, say every time a view was open and from where.

Each event contains the following data:

  • A timestamp, in milliseconds. The first event in any ping always has a value of 0, and subsequent event timestamps are relative to it.
  • The name of the event.
  • A set of key-value pairs, where the keys are predefined in the extra_keys metric parameter, and the values are strings.

Configuration

Say you're adding a new event for when a view is shown. First you need to add an entry for the event to the metrics.yaml file:

views:
  login_opened:
    type: event
    description: >
      Recorded when the login view is opened.
    ...
    extra_keys:
      source_of_login:
        description: The source from which the login view was opened, e.g. "toolbar".

API

Note that an enum has been generated for handling the extra_keys: it has the same name as the event metric, with Keys added.

import org.mozilla.yourApplication.GleanMetrics.Views

Views.loginOpened.record(mapOf(Views.loginOpenedKeys.sourceOfLogin to "toolbar"))

There are test APIs available too, for example:

import org.mozilla.yourApplication.GleanMetrics.Views

// Was any event recorded?
assertTrue(Views.loginOpened.testHasValue())
// Get a List of the recorded events.
val snapshot = Views.loginOpened.testGetValue()
// Check that two events were recorded.
assertEquals(2, snapshot.size)
val first = snapshot.single()
assertEquals("login_opened", first.name)
// Check that no errors were recorded
assertEquals(0, Views.loginOpened.testGetNumRecordedErrors(ErrorType.InvalidValue))

Note that an enum has been generated for handling the extra_keys: it has the same name as the event metric, with Keys added.

Views.loginOpened.record(extra: [.sourceOfLogin: "toolbar"])

There are test APIs available too, for example:

@testable import Glean

// Was any event recorded?
XCTAssert(Views.loginOpened.testHasValue())
// Get a List of the recorded events.
val snapshot = try! Views.loginOpened.testGetValue()
// Check that two events were recorded.
XCTAssertEqual(2, snapshot.size)
val first = snapshot[0]
XCTAssertEqual("login_opened", first.name)
// Check that no errors were recorded
XCTAssertEqual(0, Views.loginOpened.testGetNumRecordedErrors(.invalidValue))

Note that an enum has been generated for handling the extra_keys: it has the same name as the event metric, with _keys added.

from glean import load_metrics
metrics = load_metrics("metrics.yaml")

metrics.views.login_opened.record(
    {
        metrics.views.login_opened_keys.SOURCE_OF_LOGIN: "toolbar"
    }
)

There are test APIs available too, for example:

# Was any event recorded?
assert metrics.views.login_opened.test_has_value()
# Get a List of the recorded events.
snapshot = metrics.views.login_opened.test_get_value()
# Check that two events were recorded.
assert 2 == len(snapshot)
first = snapshot[0]
assert "login_opened" == first.name
# Check that no errors were recorded
assert 0 == metrics.views.login_opened.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)

Note that an enum has been generated for handling the extra_keys: it has the same name as the event metric, with Keys added.

using static Mozilla.YourApplication.GleanMetrics.Views;

Views.loginOpened.Record(new Dictionary<clickKeys, string> {
    { Views.loginOpenedKeys.sourceOfLogin, "toolbar" }
});

There are test APIs available too, for example:

using static Mozilla.YourApplication.GleanMetrics.Views;

// Was any event recorded?
Assert.True(Views.loginOpened.TestHasValue());
// Get a List of the recorded events.
var snapshot = Views.loginOpened.TestGetValue();
// Check that two events were recorded.
Assert.Equal(2, snapshot.Length);
var first = snapshot.First();
Assert.Equal("login_opened", first.Name);
// Check that no errors were recorded
Assert.Equal(0, Views.loginOpened.TestGetNumRecordedErrors(ErrorType.InvalidValue));

Limits

  • When 500 events are queued on the client an events ping is immediately sent.

  • The extra_keys allows for a maximum of 10 keys.

  • The keys in the extra_keys list must be in dotted snake case, with a maximum length of 40 bytes in UTF-8.

  • The values in the extras object have a maximum length of 50 in UTF-8.

Examples

  • Every time a new tab is opened.

Recorded errors

  • invalid_overflow: if any of the values in the extras object are greater than 50 bytes in length. (Prior to Glean 31.5.0, this recorded an invalid_value).

Reference

Custom Distribution

Custom distributions are used to record the distribution of arbitrary values.

It should be used only when direct control over how the histogram buckets are computed is required. Otherwise, look at the standard distribution metric types:

Note: Custom distributions are currently only allowed for GeckoView metrics (the gecko_datapoint parameter is present) and thus have only a Kotlin API.

Configuration

Custom distributions have the following required parameters:

  • range_min: (Integer) The minimum value of the first bucket
  • range_max: (Integer) The minimum value of the last bucket
  • bucket_count: (Integer) The number of buckets
  • histogram_type:
    • linear: The buckets are evenly spaced
    • exponential: The buckets follow a natural logarithmic distribution

Note Check out how these bucketing algorithms would behave on the Custom distribution simulator

In addition, the metric should specify:

  • unit: (String) The unit of the values in the metric. For documentation purposes only -- does not affect data collection.

If you wanted to create a custom distribution of the peak number of pixels used during a checkerboard event, first you need to add an entry for it to the metrics.yaml file:

graphics:
  checkerboard_peak:
    type: custom_distribution
    description: >
      Peak number of CSS pixels checkerboarded during a checkerboard event.
    range_min: 1
    range_max: 66355200
    bucket_count: 50
    histogram_type: exponential
    unit: pixels
    gecko_datapoint: CHECKERBOARD_PEAK
    ...

API

Now you can use the custom distribution from the application's code.

import org.mozilla.yourApplication.GleanMetrics.Graphics

Graphics.checkerboardPeak.accumulateSamples([23])

There are test APIs available too. For convenience, properties sum and count are exposed to facilitate validating that data was recorded correctly.

import org.mozilla.yourApplication.GleanMetrics.Graphics

// Was anything recorded?
assertTrue(Graphics.checkerboardPeak.testHasValue())

// Get snapshot
val snapshot = Graphics.checkerboardPeak.testGetValue()

// Does the sum have the expected value?
assertEquals(11, snapshot.sum)

// Usually you don't know the exact timing values, but how many should have been recorded.
assertEquals(2L, snapshot.count())

/// Did the metric receive a negative value?
assertEquals(1, Graphics.checkerboardPeak.testGetNumRecordedErrors(ErrorType.InvalidValue))

Limits

  • The maximum value of bucket_count is 100.

  • Only non-negative values may be recorded.

Recorded errors

  • invalid_value: If recording a negative value.

Reference

Simulator

Please, insert your custom data below as a JSON array.

Data options

Properties

Quantity

Used to record a single non-negative integer value or 0. For example, the width of the display in pixels.

IMPORTANT If you need to count something (e.g. number of tabs open or number of times a button is pressed) prefer using the Counter metric type, which has a specific API for counting things and also takes care of resetting the count at the correct time.

Configuration

Say you're adding a new quantity for the width of the display in pixels. First you need to add an entry for the quantity to the metrics.yaml file:

display:
  width:
    type: quantity
    description: >
      The width of the display, in pixels.
    unit: pixels
    ...

Note that quantities have a required unit parameter, which is a free-form string for documentation purposes.

API

import org.mozilla.yourApplication.GleanMetrics.Display

Display.width.set(width)

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Display

// Was anything recorded?
assertTrue(Display.width.testHasValue())
// Does the quantity have the expected value?
assertEquals(6, Display.width.testGetValue())
// Did it record an error due to a negative value?
assertEquals(1, Display.width.testGetNumRecordedErrors(ErrorType.InvalidValue))
import org.mozilla.yourApplication.GleanMetrics.Display;

Display.INSTANCE.width.set(width);

There are test APIs available too:

import org.mozilla.yourApplication.GleanMetrics.Display;

// Was anything recorded?
assertTrue(Display.INSTANCE.width.testHasValue());
// Does the quantity have the expected value?
assertEquals(6, Display.INSTANCE.width.testGetValue());
// Did the quantity record a negative value?
assertEquals(
    1, Display.INSTANCE.width.testGetNumRecordedErrors(ErrorType.InvalidValue)
);
Display.width.set(width)

There are test APIs available too:

@testable import Glean

// Was anything recorded?
XCTAssert(Display.width.testHasValue())
// Does the quantity have the expected value?
XCTAssertEqual(6, try Display.width.testGetValue())
// Did the quantity record a negative value?
XCTAssertEqual(1, Display.width.testGetNumRecordedErrors(.invalidValue))
from glean import load_metrics
metrics = load_metrics("metrics.yaml")

metrics.display.width.set(width)

There are test APIs available too:

# Was anything recorded?
assert metrics.display.width.test_has_value()
# Does the quantity have the expected value?
assert 6 == metrics.display.width.test_get_value()
# Did the quantity record an negative value?
from glean.testing import ErrorType
assert 1 == metrics.display.width.test_get_num_recorded_errors(
    ErrorType.INVALID_VALUE
)
using static Mozilla.YourApplication.GleanMetrics.Display;

Display.width.Set(width);

There are test APIs available too:

using static Mozilla.YourApplication.GleanMetrics.Display;

// Was anything recorded?
Assert.True(Display.width.TestHasValue());
// Does the counter have the expected value?
Assert.Equal(6, Display.width.TestGetValue());
// Did the counter record an negative value?
Assert.Equal(
    1, Display.width.TestGetNumRecordedErrors(ErrorType.InvalidValue)
);

Limits

Examples

Recorded errors

Reference

Glean Pings

Every Glean ping is in JSON format and contains one or more of the common sections with shared information data.

If data collection is enabled, the Glean SDK provides a set of built-in pings that are assembled out of the box without any developer intervention. The following is a list of these built-in pings:

Applications can also define and send their own custom pings when the schedules of these pings is not suitable.

There is also a high-level overview of how the metrics and baseline pings relate and the timings they record.

Ping sections

There are two standard metadata sections that are added to most pings, in addition to their core metrics and events content (which are described in Adding new metrics).

The ping_info section

The following fields are included in the ping_info section, for every ping. Optional fields are marked accordingly.

Field nameTypeDescription
seqCounterA running counter of the number of times pings of this type have been sent
experimentsObjectOptional. A dictionary of active experiments
start_timeDatetimeThe time of the start of collection of the data in the ping, in local time and with minute precision, including timezone information.
end_timeDatetimeThe time of the end of collection of the data in the ping, in local time and with minute precision, including timezone information. This is also the time this ping was generated and is likely well before ping transmission time.
reasonStringOptional. The reason the ping was submitted. The specific set of values and their meanings are defined for each metric type in the reasons field in the pings.yaml file.

All the metrics surviving application restarts (e.g. seq, ...) are removed once the application using the Glean SDK is uninstalled.

The client_info section

The following fields are included in the client_info section. Optional fields are marked accordingly.

Field nameTypeDescription
app_buildStringThe build identifier generated by the CI system (e.g. "1234/A"). For language bindings that provide automatic detection for this value, (e.g. Android/Kotlin), in the unlikely event that the build identifier can not be retrieved from the OS, it is set to inaccessible. For other language bindings, if the value was not provided through configuration, this metric gets set to Unknown.
app_channelStringOptional The product-provided release channel (e.g. "beta")
app_display_versionStringThe user-visible version string (e.g. "1.0.3"). The meaning of the string (e.g. whether semver or a git hash) is application-specific. In the unlikely event this value can not be obtained from the OS, it is set to "inaccessible". If it is accessible, but not set by the application, it is set to "Unknown".
architectureStringThe architecture of the device (e.g. "arm", "x86")
client_idUUIDOptional A UUID identifying a profile and allowing user-oriented correlation of data
device_manufacturerStringOptional The manufacturer of the device
device_modelStringOptional The model name of the device. On Android, this is Build.MODEL, the user-visible name of the device.
first_run_dateDatetimeThe date of the first run of the application, in local time and with day precision, including timezone information.
osStringThe name of the operating system (e.g. "linux", "Android", "ios")
os_versionStringThe user-visible version of the operating system (e.g. "1.2.3")
android_sdk_versionStringOptional. The Android specific SDK version of the software running on this hardware device (e.g. "23")
telemetry_sdk_buildStringThe version of the Glean SDK
localeStringOptional. The locale of the application during initialization (e.g. "es-ES"). If the locale can't be determined on the system, the value is "und", to indicate "undetermined".

All the metrics surviving application restarts (e.g. client_id, ...) are removed once the application using the Glean SDK is uninstalled.

The experiments object

This object (included in the ping_info section) contains experiments keyed by the experiment id. Each listed experiment contains the branch the client is enrolled in and may contain a string to string map with additional data in the extra key. Both the id and branch are truncated to 30 characters. See Using the Experiments API on how to record experiments data.

{
  "<id>": {
    "branch": "branch-id",
    "extra": {
      "some-key": "a-value"
    }
  }
}

Ping submission

The pings that the Glean SDK generates are submitted to the Mozilla servers at specific paths, in order to provide additional metadata without the need to unpack the ping payload.

A typical submission URL looks like

"<server-address>/submit/<application-id>/<doc-type>/<glean-schema-version>/<document-id>"

where:

Limitations

To keep resource usage in check, the Glean SDK enforces some limitations on ping uploading and ping storage.

Submitted headers

A pre-defined set of headers is additionally sent along with the submitted ping:

HeaderValueDescription
Content-Typeapplication/json; charset=utf-8Describes the data sent to the server
User-AgentDefaults to e.g. Glean/0.40.0 (Kotlin on Android), where 0.40.0 is the Glean SDK version number and Kotlin on Android is the name of the language used by the binding that sent the request plus the name of the platform it is running on.Describes the application sending the ping using the Glean SDK
Datee.g. Mon, 23 Jan 2019 10:10:10 GMT+00:00Submission date/time in GMT/UTC+0 offset
X-Client-TypeGleanCustom header to support handling of Glean pings in the legacy pipeline
X-Client-Versione.g. 0.40.0The Glean SDK version, sent as a custom header to support handling of Glean pings in the legacy pipeline
X-Debug-IDOptional, e.g. test-tagDebug header attached to Glean pings when using the debug tools
X-Source-TagsOptional, e.g. automation, perfA list of tags to associate with the ping, useful for clustering pings at analysis time, for example to tell data generated from CI from other data.

Defining foreground and background state

These docs refer to application 'foreground' and 'background' state in several places.

Foreground

For Android, this specifically means the activity becomes visible to the user, it has entered the Started state, and the system invokes the onStart() callback.

Background

This specifically means when the activity is no longer visible to the user, it has entered the Stopped state, and the system invokes the onStop() callback.

This may occur, if the user uses Overview button to change to another app, the user presses the Back button and navigates to a previous application or the home screen, or if the user presses the Home button to return to the home screen. This can also occur if the user navigates away from the application through some notification or other means.

The system may also call onStop() when the activity has finished running, and is about to be terminated.

Foreground

For iOS, the Glean SDK attaches to the willEnterForegroundNotification. This notification is posted by the OS shortly before an app leaves the background state on its way to becoming the active app.

Background

For iOS, this specifically means when the app is no longer visible to the user, or when the UIApplicationDelegate receives the applicationDidEnterBackground event.

This may occur if the user opens the task switcher to change to another app, or if the user presses the Home button to show the home screen. This can also occur if the user navigates away from the app through a notification or other means.

Note: Glean does not currently support Scene based lifecycle events that were introduced in iOS 13.

Ping schedules and timings overview

Full reference details about the metrics and baseline ping schedules are detailed elsewhere.

The following diagram shows a typical timeline of a mobile application, when pings are sent and what timing-related information is included.

ping timeline diagram

There are two distinct runs of the application, where the OS shutdown the application at the end of Run 1, and the user started it up again at the beginning of Run 2.

There are three distinct foreground sessions, where the application was visible on the screen and the user was able to interact with it.

The rectangles for the baseline and metrics pings represent the measurement windows of those pings, which always start exactly at the end of the preceding ping. The ping_info.start_time and ping_info.end_time metrics included in these pings correspond to these beginning and the end of their measurement windows.

The baseline.duration metric (included only in baseline pings) corresponds to amount of time the application spent on the foreground, which, since measurement window always extend to the next ping, is not always the same thing as the baseline ping's measurement window.

The submission_timestamp is the time the ping was received at the telemetry endpoint, added by the ingestion pipeline. It is not exactly the same as ping_info.end_time, since there may be various networking and system latencies both on the client and in the ingestion pipeline (represented by the dotted horizontal line, not to scale). Also of note is that start_time/end_time are measured using the client's real-time clock in its local timezone, which is not a fully reliable source of time.

The "Baseline 4" ping illustrates an important corner case. When "Session 2" ended, the OS also shut down the entire process, and the Glean SDK did not have an opportunity to send a baseline ping immediately. In this case, it is sent at the next available opportunity when the application starts up again in "Run 2". This baseline ping is annotated with the reason code dirty_startup.

The "Metrics 2" ping likewise illustrates another important corner case. "Metrics 1" was able to be sent at the target time of 04:00 (local device time) because the application was currently running. However, the next time 04:00 came around, the application was not active, so the Glean SDK was unable to send a metrics ping. It is sent at the next available opportunity, when the application starts up again in "Run 2". This metrics ping is annotated with the reason code overdue.

The baseline ping

Description

This ping is intended to provide metrics that are managed by the Glean SDK itself, and not explicitly set by the application or included in the application's metrics.yaml file.

Note: As the baseline ping was specifically designed for mobile operating systems, it is not sent when using the Glean Python bindings.

Scheduling

The baseline ping is automatically submitted with a reason: foreground when the application is moved to the foreground. These baseline pings do not contain duration.

The baseline ping is automatically submitted with a reason: background when the application is moved to the background. Occasionally, the baseline ping may fail to send when going to background (e.g. the process is killed quickly). In that case, it will be submitted at startup with a reason: dirty_startup, if the previous session was not cleanly closed. This only happens from the second start onward.

See also the ping schedules and timing overview.

Contents

The baseline ping includes the following fields:

Field nameTypeDescription
durationTimespanThe duration, in seconds, of the last foreground session. Only available if reason: background. 1
1

See also the ping schedules and timing overview for how the duration metric relates to other sources of timing in the baseline ping.

The baseline ping also includes the common ping sections found in all pings.

Querying ping contents

A quick note about querying ping contents (i.e. for sql.telemetry.mozilla.org): Each metric in the baseline ping is organized by its metric type, and uses a namespace of glean.baseline. For instance, in order to select duration you would use metrics.timespan['glean.baseline.duration']. If you were trying to select a String based metric such as os, then you would use metrics.string['glean.baseline.os']

Example baseline ping

{
  "ping_info": {
    "experiments": {
      "third_party_library": {
        "branch": "enabled"
      }
    },
    "seq": 0,
    "start_time": "2019-03-29T09:50-04:00",
    "end_time": "2019-03-29T09:53-04:00",
    "reason": "foreground"
  },
  "client_info": {
    "telemetry_sdk_build": "0.49.0",
    "first_run_date": "2019-03-29-04:00",
    "os": "Android",
    "android_sdk_version": "27",
    "os_version": "8.1.0",
    "device_manufacturer": "Google",
    "device_model": "Android SDK built for x86",
    "architecture": "x86",
    "app_build": "1",
    "app_display_version": "1.0",
    "client_id": "35dab852-74db-43f4-8aa0-88884211e545"
  },
  "metrics": {
    "timespan": {
      "glean.baseline.duration": {
        "value": 52,
        "time_unit": "second"
      }
    }
  }
}

The deletion-request ping

Description

This ping is submitted when a user opts out of sending technical and interaction data.

This ping contains the client id.

This ping is intended to communicate to the Data Pipeline that the user wishes to have their reported Telemetry data deleted. As such it attempts to send itself at the moment the user opts out of data collection, and continues to try and send itself.

Note: It is possible to send secondary ids in the deletion request ping. For instance, if the application is migrating from legacy telemetry to Glean, the legacy client ids can be added to the deletion request ping by creating a metrics.yaml entry for the id to be added with a send_in_pings value of deletion_request.

An example metrics.yaml entry might look like this:

legacy_client_id:
   type: uuid
   description:
     A UUID uniquely identifying the legacy client.
   send_in_pings:
     - deletion_request
   ...

Scheduling

The deletion-request ping is automatically submitted when upload is disabled in Glean. If upload fails, it is retried after Glean is initialized.

Contents

The deletion-request does not contain additional metrics aside from secondary ids that have been added.

Example deletion-request ping

{
  "ping_info": {
    "seq": 0,
    "start_time": "2019-12-06T09:50-04:00",
    "end_time": "2019-12-06T09:53-04:00"
  },
  "client_info": {
    "telemetry_sdk_build": "22.0.0",
    "first_run_date": "2019-03-29-04:00",
    "os": "Android",
    "android_sdk_version": "28",
    "os_version": "9",
    "device_manufacturer": "Google",
    "device_model": "Android SDK built for x86",
    "architecture": "x86",
    "app_build": "1",
    "app_display_version": "1.0",
    "client_id": "35dab852-74db-43f4-8aa0-88884211e545"
  },
  "metrics": {
    "uuid": {
      "legacy_client_id": "5faffa6d-6147-4d22-a93e-c1dbd6e06171"
    }
  }
}

The metrics ping

Description

The metrics ping is intended for all of the metrics that are explicitly set by the application or are included in the application's metrics.yaml file (except events). The reported data is tied to the ping's measurement window, which is the time between the collection of two metrics pings. Ideally, this window is expected to be about 24 hours, given that the collection is scheduled daily at 04:00. However, the metrics ping is only submitted while the application is actually running, so in practice, it may not meet the 04:00 target very frequently. Data in the ping_info section of the ping can be used to infer the length of this window and the reason that triggered the ping to be submitted. If the application crashes, unsent recorded metrics are sent along with the next metrics ping.

Additionally, it is undesirable to mix metric recording from different versions of the application. Therefore, if a version upgrade is detected, the metrics ping is collected immediately before further metrics from the new version are recorded.

Note: As the metrics ping was specifically designed for mobile operating systems, it is not sent when using the Glean Python bindings.

Scheduling

The desired behavior is to collect the ping at the first available opportunity after 04:00 local time on a new calendar day, but given constraints of the platform, it can only be submitted while the application is running. This breaks down into three scenarios:

  1. the application was just installed;
  2. the application was just upgraded (the version of the app is different from the last time the app was run);
  3. the application was just started (after a crash or a long inactivity period);
  4. the application was running at 04:00.

In the first case, since the application was just installed, if the due time for the current calendar day has passed, a metrics ping is immediately generated and scheduled for sending (reason code overdue). Otherwise, if the due time for the current calendar day has not passed, a ping collection is scheduled for that time (reason code today).

In the second case, if a version change is detected at startup, the metrics ping is immediately submitted so that metrics from one version are not aggregated with metrics from another version (reason code upgrade).

In the third case, if the metrics ping was not already collected on the current calendar day, and it is before 04:00, a collection is scheduled for 04:00 on the current calendar day (reason code today). If it is after 04:00, a new collection is scheduled immediately (reason code overdue). Lastly, if a ping was already collected on the current calendar day, the next one is scheduled for collecting at 04:00 on the next calendar day (reason code tomorrow).

In the fourth and last case, the application is running during a scheduled ping collection time. The next ping is scheduled for 04:00 the next calendar day (reason code reschedule).

More scheduling examples are included below.

See also the ping schedules and timing overview.

Contents

The metrics ping contains all of the metrics defined in metrics.yaml (except events) that don't specify a ping or where default is specified in their send in pings property.

Additionally, error metrics in the glean.error category are included in the metrics ping.

The metrics ping shall also include the common ping_info and 'client_info' sections.

Querying ping contents

Information about query ping contents is available in Accessing Glean data in the Firefox data docs.

Scheduling Examples

Crossing due time with the application closed

  1. The application is opened on Feb 7 on 15:00, closed on 15:05.

    • Glean records one metric A (say startup time in ms) during this measurement window MW1.
  2. The application is opened again on Feb 8 on 17:00.

Crossing due time and changing timezones

  1. The application is opened on Feb 7 on 15:00 in timezone UTC, closed on 15:05.

    • Glean records one metric A (say startup time in ms) during this measurement window MW1.
  2. The application is opened again on Feb 8 on 17:00 in timezone UTC+1.

    • Glean notes that we passed local 04:00 UTC+1 since MW1.

    • Glean closes MW1, with:

      • start_time=Feb7/15:00/UTC;
      • end_time=Feb8/17:00/UTC+1.
    • Glean records metric A again, into MW2.

The application doesn’t run in a week

  1. The application is opened on Feb 7 on 15:00 in timezone UTC, closed on 15:05.

    • Glean records one metric A (say startup time in ms) during this measurement window MW1.
  2. The application is opened again on Feb 16 on 17:00 in timezone UTC.

    • Glean notes that we passed local 04:00 UTC since MW1.

    • Glean closes MW1, with:

      • start_time=Feb7/15:00/UTC;
      • end_time=Feb16/17:00/UTC.
    • Glean records metric A again, into MW2.

The application doesn’t run for a week, and when it’s finally re-opened the timezone has changed

  1. The application is opened on Feb 7 on 15:00 in timezone UTC, closed on 15:05.

    • Glean records one metric A (say startup time in ms) during this measurement window MW1.
  2. The application is opened again on Feb 16 on 17:00 in timezone UTC+1.

    • Glean notes that we passed local 04:00 UTC+1 since MW1.

    • Glean closes MW1, with:

      • start_time=Feb7/15:00/UTC
      • end_time=Feb16/17:00/UTC+1.
    • Glean records metric A again, into MW2.

The user changes timezone in an extreme enough fashion that they cross 04:00 twice on the same date

  1. The application is opened on Feb 7 at 15:00 in timezone UTC+11, closed at 15:05.

    • Glean records one metric A (say startup time in ms) during this measurement window MW1.
  2. The application is opened again on Feb 8 at 04:30 in timezone UTC+11.

    • Glean notes that we passed local 04:00 UTC+11.

    • Glean closes MW1, with:

      • start_time=Feb7/15:00/UTC+11;
      • end_time=Feb8/04:30/UTC+11.
    • Glean records metric A again, into MW2.

  3. The user changes to timezone UTC-10 and opens the application at Feb 7 at 22:00 in timezone UTC-10

    • Glean records metric A again, into MW2 (not MW1, which was already sent).
  4. The user opens the application at Feb 8 05:00 in timezone UTC-10

    • Glean notes that we have not yet passed local 04:00 on Feb 9
    • Measurement window MW2 remains the current measurement window
  5. The user opens the application at Feb 9 07:00 in timezone UTC-10

    • Glean notes that we have passed local 04:00 on Feb 9

    • Glean closes MW2 with:

      • start_time=Feb8/04:30/UTC+11;
      • end_time=Feb9/19:00/UTC-10.
    • Glean records metric A again, into MW3.

The events ping

Description

The events ping's purpose is to transport all of the event metric information. If the application crashes, an events ping is generated next time the application starts with events that were not sent before the crash.

Scheduling

The events ping is collected under the following circumstances:

  1. Normally, it is collected when the application goes into the background, if there are any recorded events to send.

  2. When the queue of events exceeds Glean.configuration.maxEvents (default 500).

  3. If there are any unsent events found on disk when starting the application. It would be impossible to coordinate the timestamps across a reboot, so it's best to just collect all events from the previous run into their own ping, and start over.

All of these cases are handled automatically, with no intervention or configuration required by the application.

Note: Since the Python bindings don't have a concept of "going to background", case (1) above does not apply.

Contents

At the top-level, this ping contains the following keys:

Each entry in the events array is an object with the following properties:

Example event JSON

{
  "ping_info": {
    "experiments": {
      "third_party_library": {
        "branch": "enabled"
      }
    },
    "seq": 0,
    "start_time": "2019-03-29T09:50-04:00",
    "end_time": "2019-03-29T10:02-04:00"
  },
  "client_info": {
    "telemetry_sdk_build": "0.49.0",
    "first_run_date": "2019-03-29-04:00",
    "os": "Android",
    "android_sdk_version": "27",
    "os_version": "8.1.0",
    "device_manufacturer": "Google",
    "device_model": "Android SDK built for x86",
    "architecture": "x86",
    "app_build": "1",
    "app_display_version": "1.0",
    "client_id": "35dab852-74db-43f4-8aa0-88884211e545"
  },
  "events": [
    {
      "timestamp": 123456789,
      "category": "examples",
      "name": "event_example",
      "extra": {
        "metadata1": "extra",
        "metadata2": "more_extra"
      }
    },
    {
      "timestamp": 123456791,
      "category": "examples",
      "name": "event_example"
    }
  ]
}

Custom pings

Applications can define metrics that are sent in custom pings. Unlike the built-in pings, custom pings are sent explicitly by the application.

This is useful when the scheduling of the built-in pings (metrics, baseline and events) are not appropriate for your data. Since the timing of the submission of custom pings is handled by the application, the measurement window is under the application's control.

This is especially useful when metrics need to be tightly related to one another, for example when you need to measure the distribution of frame paint times when a particular rendering backend is in use. If these metrics were in different pings, with different measurement windows, it is much harder to do that kind of reasoning with much certainty.

Defining a custom ping

Custom pings must be defined in a pings.yaml file, which is in the same directory alongside your app's metrics.yaml file.

Ping names are limited to lowercase letters from the ISO basic Latin alphabet and hyphens and a maximum of 30 characters.

Each ping has the following parameters:

In addition to these parameters, pings also support the parameters related to data review and expiration defined in common metric parameters: description, notification_emails, bugs, and data_reviews.

For example, to define a custom ping called search specifically for search information:

# Required to indicate this is a `pings.yaml` file
$schema: moz://mozilla.org/schemas/glean/pings/1-0-0

search:
  description: >
    A ping to record search data.
  include_client_id: false
  notification_emails:
    - CHANGE-ME@example.com
  bugs:
    - http://bugzilla.mozilla.org/123456789/
  data_reviews:
    - http://example.com/path/to/data-review

Note: the names baseline, metrics, events, deletion-request and all-pings are reserved and may not be used as the name of a custom ping.

Loading custom ping metadata into your application or library

The Glean SDK build generates code from pings.yaml in a Pings object, which must be instantiated so Glean can send pings by name.

In Kotlin, this object must be registered with the Glean SDK from your startup code (such as in your application's onCreate method or a function called from that method).

import org.mozilla.yourApplication.GleanMetrics.Pings

...

override fun onCreate() {
    ...
    Glean.registerPings(Pings)
    ...
}

In Swift, this object must be registered with the Glean SDK from your startup code (such as in your application's UIApplicationDelegate application(_:didFinishLaunchingWithOptions:) method or a function called from that method).

import Glean

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {
func application(_: UIApplication, didFinishLaunchingWithOptions _: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
    // ...
    Glean.shared.registerPings(GleanMetrics.Pings)
    // ...
}
}

For Python, the pings.yaml file must be available and loaded at runtime.

If your project is a script (i.e. just Python files in a directory), you can load the pings.yaml using:

from glean import load_pings

pings = load_pings("pings.yaml")

If your project is a distributable Python package, you need to include the pings.yaml file using one of the myriad ways to include data in a Python package and then use pkg_resources.resource_filename() to get the filename at runtime.

from glean import load_pings
from pkg_resources import resource_filename

pings = load_pings(resource_filename(__name__, "pings.yaml"))

In C#, this object must be registered with the Glean SDK from your startup code (such as in your application's Main method or a function called from that method).

using static Mozilla.YourApplication.GleanMetrics.Pings;

...

class Program
{
    static void Main(string[] args)
    {
        ...
        Glean.RegisterPings(Pings);
        ...
    }
}

Sending metrics in a custom ping

To send a metric on a custom ping, you add the custom ping's name to the send_in_pings parameter in the metrics.yaml file.

For example, to define a new metric to record the default search engine, which is sent in a custom ping called search, put search in the send_in_pings parameter. Note that it is an error to specify a ping in send_in_pings that does not also have an entry in pings.yaml.

search.default:
  name:
    type: string
    description: >
      The name of the default search engine.
    send_in_pings:
      - search

If this metric should also be sent in the default ping for the given metric type, you can add the special value default to send_in_pings:

    send_in_pings:
      - search
      - default

Submitting a custom ping

To collect and queue a custom ping for eventual uploading, call the submit method on the PingType object that the Glean SDK generated for your ping.

By default, if the ping doesn't currently have any events or metrics set, submit will do nothing. However, if the send_if_empty flag is set to true in the ping definition, it will always be submitted.

For example, to submit the custom ping defined above:

import org.mozilla.yourApplication.GleanMetrics.Pings
Pings.search.submit(
    GleanMetrics.Pings.searchReasonCodes.performed
)
import Glean

GleanMetrics.Pings.shared.search.submit(
    reason: .performed
)
from glean import load_pings

pings = load_pings("pings.yaml")

pings.search.submit(pings.search_reason_codes.PERFORMED)
using static Mozilla.YourApplication.GleanMetrics.Pings;

Pings.search.Submit(
    GleanMetrics.Pings.searchReasonCodes.performed
);

If none of the metrics for the ping contain data the ping is not sent (unless send_if_empty is set to true in the definition file)

Unit testing Glean custom pings for Android

Applications defining custom pings can use use the strategy defined in this document to test these pings in unit tests.

General testing strategy

The schedule of custom pings depends on the specific application implementation, since it is up to the SDK user to define the ping semantics. This makes writing unit tests for custom pings a bit more involved.

One possible strategy could be to wrap the Glean SDK API call to send the ping in a function that can be mocked in the unit test. This would allow for checking the status and the values of the metrics contained in the ping at the time in which the application would have sent it.

Example testing of a custom ping

Let us start by defining a custom ping with a sample metric in it. Here is the pings.yaml file:

$schema: moz://mozilla.org/schemas/glean/pings/1-0-0

my-custom-ping:
  description: >
    This ping is intended to showcase the recommended testing strategy for
    custom pings.
  include_client_id: false
  bugs:
    - https://bugzilla.mozilla.org/1556985/
  data_reviews:
    - https://bugzilla.mozilla.org/show_bug.cgi?id=1556985
  notification_emails:
    - custom-ping-owner@example.com

And here is the metrics.yaml

$schema: moz://mozilla.org/schemas/glean/metrics/1-0-0

custom_ping_data:
  sample_string:
    type: string
    lifetime: ping
    description: >
      A sample string metric for demonstrating unit tests for custom pings.
    send_in_pings:
      - my-custom-ping
    bugs:
      - https://bugzilla.mozilla.org/1556985/
    data_reviews:
      - https://bugzilla.mozilla.org/show_bug.cgi?id=1556985
    notification_emails:
      - custom-ping-owner@example.com
    expires: "2019-10-01"

A potential usage of the Glean SDK generated API could be the following:

import my.component.GleanMetrics.Pings
import my.component.GleanMetrics.CustomPingData

class MyCustomPingScheduler {
  /**
   * HERE ONLY TO KEEP THE EXAMPLE SIMPLE.
   *
   * A function that consumes the Glean SDK generated metrics API to
   * record some data. It doesn't really need to be in a function, nor
   * in this class. The Glean SDK API can be called when the data is
   * generated.
   */
  fun addSomeData() {
    // Record some sample data.
    CustomPingData.sampleString.set("test-data")
  }

  /**
   * Called to implement the ping scheduling logic for 'my_custom_ping'.
   */
  fun schedulePing() {
    // ... some scheduling logic that will end up calling the function below.
    submitPing()
  }

  /**
   * Internal function to only be overridden in tests. This
   * calls the Glean SDK API to send custom pings.
   */
  @VisibleForTesting(otherwise = VisibleForTesting.NONE)
  internal fun submitPing() {
    Pings.MyCustomPing.submit()
  }
}

The following unit test intercepts the MyCustomPingScheduler.submitPing() call in order to perform the validation on the data. This specific example uses Mockito, but any other framework would work.

// Metrics and pings definitions.
import my.component.GleanMetrics.Pings
import my.component.GleanMetrics.CustomPingData

// Mockito imports for using spies.
import org.mockito.Mockito.spy
import org.mockito.Mockito.`when`

@RunWith(AndroidJUnit4::class)
class MyCustomPingSchedulerTest {
    // Apply the GleanTestRule to set up a disposable Glean instance.
    // Please note that this clears the Glean data across tests.
    @get:Rule
    val gleanRule = GleanTestRule(ApplicationProvider.getApplicationContext())

    @Test
    fun `verify custom ping metrics`() {
      val scheduler = spy(MyCustomPingScheduler())
      doAnswer {
        // Here we validate the content that goes into the ping.
        assertTrue(CustomPingData.sampleString.testHasValue())
        assertEquals("test-data", CustomPingData.sampleString.testGetValue())

        // We want to intercept this call, but we also want to make sure the
        // real Glean API is called in order to clear the ping store and to provide
        // consistent behaviour with respect to the application.
        it.callRealMethod()
      }.`when`(scheduler).submitPing()

      scheduler.addSomeData()
      scheduler.schedulePing()
    }
}
import Foundation
import Glean

// Use typealiases to simplify usage.
// This can be placed anywhere in your code to be available in all files.
typealias CustomPingData = GleanMetrics.CustomPingData
typealias Pings = GleanMetrics.Pings

class MyCustomPingScheduler {
    /**
     * HERE ONLY TO KEEP THE EXAMPLE SIMPLE.
     *
     * A function that consumes the Glean SDK generated metrics API to
     * record some data. It doesn't really need to be in a function, nor
     * in this class. The Glean SDK API can be called when the data is
     * generated.
     */
    func addSomeData() {
       // Record some sample data.
       CustomPingData.sampleString.set("test-data")
    }

    /**
     * Called to implement the ping scheduling logic for 'my_custom_ping'.
     */
    func schedulePing() {
        // ... some scheduling logic that will end up calling the function below.
        submitPing()
    }

    /**
     * Internal function to only be overridden in tests. This
     * calls the Glean SDK API to send custom pings.
     */
    internal func submitPing() {
        Pings.shared.myCustomPing.submit()
    }
}

The following unit test intercepts the MyCustomPingScheduler.submitPing() call in order to perform the validation on the data. This example uses a manual mock implementation, but you could use a framework for that.

@testable import YourApplication
import Glean
import XCTest

class MyCustomPingSchedulerMock: MyCustomPingScheduler {
    var submitWasCalled = false

    deinit {
        XCTAssertTrue(submitWasCalled, "submitPing should have been called once")
    }

    override func submitPing() {
        submitWasCalled = true

        XCTAssertTrue(CustomPingData.os.testHasValue())
        XCTAssertEqual("test-data", try! CustomPingData.os.testGetValue())

        super.submitPing()
    }
}

class MyCustomPingSchedulerTests: XCTestCase {
    override func setUp() {
        Glean.shared.resetGlean(clearStores: true)
    }

    func testCustomPingMetrics() {
        let scheduler = MyCustomPingSchedulerMock()
        scheduler.addSomeData()
        scheduler.schedulePing()
    }
}
import glean

metrics = glean.load_metrics("metrics.yaml")
pings = glean.load_pings("pings.yaml")


class MyCustomPingScheduler:
    def add_some_data(self):
        """
        HERE ONLY TO KEEP THE EXAMPLE SIMPLE.

        A function that consumes the Glean SDK generated metrics API to
        record some data. It doesn't really need to be in a function, nor
        in this class. The Glean SDK API can be called when the data is
        generated.
        """
        # Record some sample data.
        metrics.custom_ping_data.sample_string.set("test-data")

    def schedule_ping(self):
        """
        Called to implement the ping scheduling logic for 'my_custom_ping'.
        """
        # ... some scheduling logic that will end up calling the function below.
        self._submit_ping()

    def _submit_ping(self):
        """
        Internal function to only be overridden in tests.
        """
        pings.my_custom_ping.submit()

The following unit test intercepts the MyCustomPingScheduler._submit_ping() call in order to perform the validation on the data.

from unittest.mock import MagicMock

from glean import testing

import custom_ping_scheduler


# This will be run before every test in the entire test suite
def pytest_runtest_setup(item):
    testing.reset_glean(application_id="my-app", application_version="0.0.1")


def test_verify_custom_ping_metrics():
    scheduler = custom_ping_scheduler.MyCustomPingScheduler()

    original_submit_ping = scheduler._submit_ping

    def _submit_ping(self):
        # Here we validate the content that goes into the ping.
        assert (
            custom_ping_scheduler.metrics.custom_ping_data.sample_string.test_has_value()
        )
        assert (
            "test-data"
            == custom_ping_scheduler.metrics.custom_ping_data.sample_string.test_get_value()
        )

        # We want to intercept this call, but we also want to make sure the
        # real Glean API is called in order to clear the ping strong and to
        # provide consistent behavior with respect to the application.
        original_submit_ping(self)

    scheduler._submit_ping = MagicMock(_submit_ping)

    scheduler.add_some_data()
    scheduler.schedule_ping()

TODO. To be implemented in bug 1648446.

Android-specific information

Android build script configuration options

This chapter describes build configuration options that control the behavior of the Glean SDK's Gradle plugin. These options are not usually required for normal use.

Options can be turned on by setting a variable on the Gradle ext object before applying the Glean Gradle plugin.

allowMetricsFromAAR

Normally, the Glean SDK looks for metrics.yaml and pings.yaml files in the root directory of the Glean-using project. However, in some cases, these files may need to ship inside the dependencies of the project. For example, this is used in the engine-gecko component to grab the metrics.yaml from the geckoview AAR.

ext.allowMetricsFromAAR = true

When this flag is set, every direct dependency of your library will be searched for a metrics.yaml file, and those metrics will be treated as the metrics as if they were defined by your library. That is, API wrappers accessible from your library will be generated for those metrics.

The metrics.yaml can be added to the dependency itself by calling this on each relevant build variant:

variant.packageLibraryProvider.get().from("${topsrcdir}/path/metrics.yaml")

gleanGenerateMarkdownDocs

The Glean SDK can automatically generate Markdown documentation for metrics and pings defined in the registry files, in addition to the metrics API code.

ext.gleanGenerateMarkdownDocs = true

Flipping the feature to true will generate a metrics.md file in $projectDir/docs at build-time.

gleanDocsDirectory

The gleanDocsDirectory can be used to customize the path of the documentation output directory. If gleanGenerateMarkdownDocs is disabled, it does nothing. Please note that only the metrics.md will be overwritten: any other file available in the target directory will be preserved.

ext.gleanDocsDirectory = "$rootDir/docs/user/telemetry"

gleanYamlFiles

By default, the Glean Gradle plugin will look for metrics.yaml and pings.yaml files in the same directory that the plugin is included from in your application or library. To override this, ext.gleanYamlFiles may be set to a list of explicit paths.

ext.gleanYamlFiles = ["$rootDir/glean-core/metrics.yaml", "$rootDir/glean-core/pings.yaml"]

Offline builds of Android applications that use Glean

The Glean SDK has basic support for building Android applications that use Glean in offline mode.

The Glean SDK uses a Python script, glean_parser to generate code for metrics from the metrics.yaml and pings.yaml files when Glean-using applications are built. When online, the pieces necessary to run this script are installed automatically.

For offline builds, the Python environment, and packages of glean_parser and its dependencies must be provided prior to building the Glean-using application.

To build a Glean-using application in offline mode, do the following:

There are a couple of environment variables that control offline building:

Focused Use Cases

Here is a list of specific examples of using Glean to instrument different situations.

Instrumenting Android Crashes With Glean

This is a simple example illustrating a strategy or method for instrumenting crashes in Android applications using Glean.

Instrumenting Android crashes with the Glean SDK

One of the things that might be useful to collect data on in an Android application is crashes. This guide will walk through a basic strategy for instrumenting an Android application with crash telemetry using a custom ping.

Note: This is a very simple example of instrumenting crashes using the Glean SDK. There will be challenges to using this approach in a production application that should be considered. For instance, when an app crashes it can be in an unknown state and may not be able to do things like upload data to a server. The recommended way of instrumenting crashes with Android Components is called lib-crash, which takes into consideration things like multiple processes and persistence.

Before You Start

There are a few things that need to be installed in order to proceed, mainly Android Studio. If you include the Android SDK, Android Studio can take a little while to download and get installed. This walk-through assumes some knowledge of Android application development. Knowing where to go to create a new project and how to add dependencies to a Gradle file will be helpful in following this guide.

Setup Build Configuration

Please follow the instruction in the "Adding Glean to your project" chapter in order to set up Glean in an Android project.

Add A Custom Metric

Since crashes will be instrumented with some custom metrics, the next step will be to add a metrics.yaml file to define the metrics used to record the crash information and a pings.yaml file to define a custom ping which will give some control over the scheduling of the uploading. See "Adding new metrics" for more information about adding metrics.

What metric type should be used to represent the crash data? While this could be implemented several ways, an event is an excellent choice, simply because events capture information in a nice concise way and they have a built-in way of passing additional information using the extras field. If it is necessary to pass along the cause of the exception or a few lines of description, events let us do that easily (with some limitations).

Now that a metric type has been chosen to represent the metric, the next step is creating the metrics.yaml. Inside of the root application folder of the Android Studio project create a new file named metrics.yaml. After adding the schema definition and event metric definition, the metrics.yaml should look like this:

# Required to indicate this is a `metrics.yaml` file
$schema: moz://mozilla.org/schemas/glean/metrics/1-0-0

crash:
  exception:
    type: event
    description: |
      Event to record crashes caused by unhandled exceptions
    notification_emails:
      - crashes@example.com
    bugs:
      - https://bugzilla.mozilla.org/show_bug.cgi?id=1582479
    data_reviews:
      - https://bugzilla.mozilla.org/show_bug.cgi?id=1582479
    expires:
      2021-01-01
    send_in_pings:
      - crash
    extra_keys:
      cause:
        description: The cause of the crash
      message:
        description: The exception message

As a brief explanation, this creates a metric called exception within a metric category called crash. There is a text description and the required notification_emails, bugs, data_reviews, and expires fields. The send_in_pings field is important to note here that it has a value of - crash. This means that the crash event metric will be sent via a custom ping named crash (which hasn't been created yet). Finally, note the extra_keys field which has two keys defined, cause and message. This allows for sending additional information along with the event to be associated with these keys.

Note: For Mozilla applications, a mandatory data review is required in order to collect information with the Glean SDK.

Add A Custom Ping

Define the custom ping that will help control the upload scheduling by creating a pings.yaml file in the same directory as the metrics.yaml file. For more information about adding custom pings, see the section on custom pings.
The name of the ping will be crash, so the pings.yaml file should look like this:

# Required to indicate this is a `pings.yaml` file
$schema: moz://mozilla.org/schemas/glean/pings/1-0-0

crash:
  description: >
    A ping to transport crash data
  include_client_id: true
  notification_emails:
    - crash@example.com
  bugs:
    - https://bugzilla.mozilla.org/show_bug.cgi?id=1582479
  data_reviews:
    - https://bugzilla.mozilla.org/show_bug.cgi?id=1582479

Before the newly defined metric or ping can be used, the application must first be built. This will cause the glean_parser to execute and generate the API files that represent the metric and ping that were newly defined.

Note: If changes to the YAML files aren't showing up in the project, try running the clean task on the project before building any time one of the Glean YAML files has been modified.

It is recommended that Glean be initialized as early in the application startup as possible, which is why it's good to use a custom Application, like the Glean Sample App GleanApplication.kt.

Initializing Glean in the Application.onCreate() is ideal for this purpose. Start by adding the import statement to allow the usage of the custom ping that was created, adding the following to the top of the file:

import org.mozilla.gleancrashexample.GleanMetrics.Pings

Next, register the custom ping by calling Glean.registerPings(Pings) in the onCreate() function, preferably before calling Glean.initialize(). The completed function should look something like this:

override fun onCreate() {
  super.onCreate()

  // Register the application's custom pings.
  Glean.registerPings(Pings)

  // Initialize the Glean library
  Glean.initialize(applicationContext)
}

This completes the registration of the custom ping with the Glean SDK so that it knows about it and can manage the storage and other important details of it like sending it when send() is called.

Instrument The App To Record The Event

In order to make the custom Application class handle uncaught exceptions, extend the class definition by adding Thread.UncaughtExceptionHandler as an inherited class like this:

class MainActivity : AppCompatActivity(), Thread.UncaughtExceptionHandler {
    ...
}

As part of implementing the Thread.UncaughtExceptionHandler interface, the custom Application needs to implement the override of the uncaughtException() function. An example of this override that records data and sends the ping could look something like this:

override fun uncaughtException(thread: Thread, exception: Throwable) {
    Crash.exception.record(
        mapOf(
            Crash.exceptionKeys.cause to exception.cause!!.toString(),
            Crash.exceptionKeys.message to exception.message!!
        )
    )
    Pings.crash.submit()
}

This records data to the Crash.exception metric from the metrics.yaml. The category of the metric is crash and the name is exception so it is accessed it by calling record() on the Crash.exception object. The extra information for the cause and the message is set as well. Finally, calling Pings.crash.submit() forces the crash ping to be scheduled to be sent.

The final step is to register the custom Application as the default uncaught exception handler by adding the following to the onCreate() function after Glean.initialize(this):

Thread.setDefaultUncaughtExceptionHandler(this)

Next Steps

This information didn't really get recorded by anything, as it would be rejected by the telemetry pipeline unless the application was already known. In order to collect telemetry from a new application, there is additional work that is necessary that is beyond the scope of this example. In order for data to be collected from your project, metadata must be added to the probe_scraper. The instructions for accomplishing this can be found in the probe_scraper documentation.

Metrics

This document enumerates the metrics collected by this project using the Glean SDK. This project may depend on other projects which also collect metrics. This means you might have to go searching through the dependency tree to get a full picture of everything collected by this project.

Pings

all-pings

These metrics are sent in every ping.

The following metrics are added to the ping:

NameTypeDescriptionData reviewsExtrasExpirationData Sensitivity
glean.error.invalid_labellabeled_counterCounts the number of times a metric was set with an invalid label. The labels are the category.name identifier of the metric.1never1
glean.error.invalid_overflowlabeled_counterCounts the number of times a metric was set a value that overflowed. The labels are the category.name identifier of the metric.1never1
glean.error.invalid_statelabeled_counterCounts the number of times a timing metric was used incorrectly. The labels are the category.name identifier of the metric.1never1
glean.error.invalid_valuelabeled_counterCounts the number of times a metric was set to an invalid value. The labels are the category.name identifier of the metric.1never1

baseline

This is a built-in ping that is assembled out of the box by the Glean SDK.

See the Glean SDK documentation for the baseline ping.

This ping is sent if empty.

This ping includes the client id.

Data reviews for this ping:

Bugs related to this ping:

Reasons this ping may be sent:

The following metrics are added to the ping:

NameTypeDescriptionData reviewsExtrasExpirationData Sensitivity
glean.baseline.durationtimespanThe duration of the last foreground session.1never1, 2

deletion-request

This is a built-in ping that is assembled out of the box by the Glean SDK.

See the Glean SDK documentation for the deletion-request ping.

This ping is sent if empty.

This ping includes the client id.

Data reviews for this ping:

Bugs related to this ping:

This ping contains no metrics.

metrics

This is a built-in ping that is assembled out of the box by the Glean SDK.

See the Glean SDK documentation for the metrics ping.

This ping includes the client id.

Data reviews for this ping:

Bugs related to this ping:

Reasons this ping may be sent:

The following metrics are added to the ping:

NameTypeDescriptionData reviewsExtrasExpirationData Sensitivity
glean.database.sizememory_distributionThe size of the database file at startup.1never1
glean.error.preinit_tasks_overflowcounterThe number of tasks queued in the pre-initialization buffer. Only sent if the buffer overflows.1never1
glean.upload.deleted_pings_after_quota_hitcounterThe number of pings deleted after the quota for the size of the pending pings directory or number of files is hit. Since quota is only calculated for the pending pings directory, and deletion request ping live in a different directory, deletion request pings are never deleted.1never1
glean.upload.discarded_exceeding_pings_sizememory_distributionThe size of pings that exceeded the maximum ping size allowed for upload.1never1
glean.upload.pending_pingscounterThe total number of pending pings at startup. This does not include deletion-request pings.1never1
glean.upload.pending_pings_directory_sizememory_distributionThe size of the pending pings directory upon initialization of Glean. This does not include the size of the deletion request pings directory.1never1
glean.upload.ping_upload_failurelabeled_counterCounts the number of ping upload failures, by type of failure. This includes failures for all ping types, though the counts appear in the next successfully sent metrics ping.1
  • status_code_4xx
  • status_code_5xx
  • status_code_unknown
  • unrecoverable
  • recoverable
never1

Data categories are defined here.

Developing the Glean SDK

In this chapter we describe how to develop the Glean SDK.

Running the tests

Running all tests

The tests for all languages may be run from the command line:

make test

Windows Note: On Windows, make is not available by default. While not required, installing make will allow you to use the convenience features in the Makefile.

Running the Rust tests

The Rust tests may be run with the following command:

cargo test --all

Log output can be controlled via the environment variable RUST_LOG for the glean_core crate:

export RUST_LOG=glean_core=debug

When running tests with logging you need to tell cargo to not suppress output:

cargo test -- --nocapture

Tests run in parallel by default, leading to interleaving log lines. This makes it harder to understand what's going on. For debugging you can force single-threaded tests:

cargo test -- --nocapture --test-threads=1

Running the Kotlin/Android tests

From the command line

The full Android test suite may be run from the command line with:

./gradlew test

From Android Studio

To run the full Android test suite, in the "Gradle" pane, navigate to glean-core -> Tasks -> verification and double-click either testDebugUnitTest or testReleaseUnitTest (depending on whether you want to run in Debug or Release mode). You can save this task permanently by opening the task dropdown in the toolbar and selecting "Save glean.rs:glean:android [testDebugUnitTest] Configuration".

To run a single Android test, navigate to the file containing the test, and right click on the green arrow in the left margin next to the test. There you have a choice of running or debugging the test.

Running the Swift/iOS tests

From the command line

The full iOS test suite may be run from the command line with:

make test-swift

From Xcode

To run the full iOS test suite, run tests in Xcode (Product -> Test). To run a single Swift test, navigate to the file containing the test, and click on the arrow in the left margin next to the test.

Testing in CI

See Continuous Integration for details.

How the Glean CI works

Current build status: Build Status

The mozilla/glean repository uses both CircleCI and TaskCluster to run tests on each Pull Request, each merge to the main branch and on releases.

Which tasks we run

Pull Request - testing

For each opened pull request we run a number of checks to ensure consistent formatting, no lint failures and that tests across all language bindings pass. These checks are required to pass before a Pull Request is merged. This includes by default the following tasks:

On CircleCI:

On TaskCluster:

For all tasks you can always find the full log and generated artifacts by following the "Details" links on each task.

iOS tests on Pull Request

Due to the long runtime and costs iOS tests are not run by default on Pull Requests. Admins of the mozilla/glean repository can run them on demand.

  1. On the pull request scroll to the list of all checks running on that PR on the bottom.
  2. Find the job titled "ci/circleci: iOS/hold".
  3. Click "Details" to get to Circle CI.
  4. Click on the "hold" task, then approve it.
  5. The iOS tasks will now run.

Merge to main

Current build status: Build Status

When pull requests are merged to the main branch we run the same checks as we do on pull requests. Additionally we run the following tasks:

If you notice that they fail please take a look and open a bug to investigate build failures.

Releases

When cutting a new release of the Glean SDK our CI setup is responsible for building, packaging and uploading release builds. We don't run the full test suite again.

We run the following tasks:

On CircleCI:

On TaskCluster:

On release the full TaskCluster suite takes up to 30 minutes to run.

How to find TaskCluster tasks

  1. Go to GitHub releases and find the version
  2. Go to the release commit, indicated by its commit id on the left
  3. Click the green tick or red cross left of the commit title to open the list of all checks.
  4. Find the "Decision task" and click "Details"
  5. From the list of tasks pick the one you want to look at and follow the "View task in Taskcluster" link.

Special keywords

Documentation-only changes

Documentation is deployed from CI, we therefore need it to run on documentation changes. However, some of the long-running code tests can be skipped. For that add the following literal string to the last commit message of your pull request:

[doc only]

Skipping CI completely

It is possible to completely skip running CI checks on a pull request.

To skip tasks on CircleCI include the following literal string in the commit message.
To skip tasks on TaskCluster add the following to the title of your pull request.

[ci skip]

This should only be used for metadata files, such as those in .github, LICENSE or CODE_OF_CONDUCT.md.

Glean release process

The Glean SDK consists of multiple libraries for different platforms and targets. The main supported libraries are released as one. Development happens on the main repository https://github.com/mozilla/glean. See Contributing for how to contribute changes to the Glean SDK.

The development & release process roughly follows the GitFlow model.

Note: The rest of this section assumes that upstream points to the https://github.com/mozilla/glean repository, while origin points to the developer fork. For some developer workflows, upstream can be the same as origin.

Table of Contents:

Published artifacts

Standard Release

Releases can only be done by one of the Glean maintainers.

Create a release branch

  1. Create a release branch from the main branch:
    git checkout -b release-v25.0.0 main
    
  2. Update the changelog .
    1. Add any missing important changes under the Unreleased changes headline.
    2. Commit any changes to the changelog file due to the previous step.
  3. Run bin/prepare-release.sh <new version> to bump the version number.
    1. The new version should be the next patch, minor or major version of what is currently released.
    2. Let it create a commit for you.
  4. Push the new release branch:
    git push upstream release-v25.0.0
    
  5. Wait for CI to finish on that branch and ensure it's green:
  6. Apply additional commits for bug fixes to this branch.
    • Adding large new features here is strictly prohibited. They need to go to the main branch and wait for the next release.

Finish a release branch

When CI has finished and is green for your specific release branch, you are ready to cut a release.

  1. Check out the main release branch:
    git checkout release
    
  2. Merge the specific release branch:
    git merge --no-ff release-v25.0.0
    
  3. Push the main release branch:
    git push upstream release
    
  4. Tag the release on GitHub:
    1. Draft a New Release in the GitHub UI (Releases > Draft a New Release).
    2. Enter v<myversion> as the tag. It's important this is the same as the version you specified to the prepare_release.sh script, with the v prefix added.
    3. Select the release branch as the target.
    4. Under the description, paste the contents of the release notes from CHANGELOG.md.
  5. Wait for the CI build to complete for the tag.
  6. Release the Rust crates:
    cd glean-core
    cargo publish --verbose
    cd ffi
    cargo publish --verbose
    
  7. Send a pull request to merge back the specific release branch to the development branch: https://github.com/mozilla/glean/compare/main...release-v25.0.0?expand=1
    • This is important so that no changes are lost.
    • This might have merge conflicts with the main branch, which you need to fix before it is merged.
  8. Once the above pull request lands, delete the specific release branch.

Hotfix release for latest version

If the latest released version requires a bug fix, a hotfix branch is used.

Create a hotfix branch

  1. Create a hotfix branch from the main release branch:
    git checkout -b hotfix-v25.0.1 release
    
  2. Run bin/prepare-release.sh <new version> to bump the version number.
    1. The new version should be the next patch version of what is currently released.
    2. Let it create a commit for you.
  3. Push the hotfix branch:
    git push upstream hotfix-v25.0.1
    
  4. Create a local hotfix branch for bugfixes:
    git checkout -b bugfix hotfix-v25.0.1
    
  5. Fix the bug and commit the fix in one or more separate commits.
  6. Push your bug fixes and create a pull request against the hotfix branch: https://github.com/mozilla/glean/compare/hotfix-v25.0.1...your-name:bugfix?expand=1
  7. When that pull request lands, wait for CI to finish on that branch and ensure it's green:

Finish a hotfix branch

When CI has finished and is green for your hotfix branch, you are ready to cut a release, similar to a normal release:

  1. Check out the main release branch:
    git checkout release
    
  2. Merge the hotfix branch:
    git merge --no-ff hotfix-v25.0.1
    
  3. Push the main release branch:
    git push upstream release
    
  4. Tag the release on GitHub:
    1. Draft a New Release in the GitHub UI (Releases > Draft a New Release).
    2. Enter v<myversion> as the tag. It's important this is the same as the version you specified to the prepare_release.sh script, with the v prefix added.
    3. Select the release branch as the target.
    4. Under the description, paste the contents of the release notes from CHANGELOG.md.
  5. Wait for the CI build to complete for the tag.
  6. Release the Rust crates:
    cd glean-core
    cargo publish --verbose
    cd ffi
    cargo publish --verbose
    
  7. Send a pull request to merge back the hotfix branch to the development branch: https://github.com/mozilla/glean/compare/main...hotfix-v25.0.1?expand=1
    • This is important so that no changes are lost.
    • This might have merge conflicts with the main branch, which you need to fix before it is merged.
  8. Once the above pull request lands, delete the hotfix branch.

Hotfix release for previous version

If you need to release a hotfix for a previously released version (that is: not the latest released version), you need a support branch.

Note: This should rarely happen. We generally support only the latest released version of Glean.

Create a support and hotfix branch

  1. Create a support branch from the version tag and push it:
    git checkout -b support/v24.0 v24.0.0
    git push upstream support/v24.0
    
  2. Create a hotfix branch for this support branch:
    git checkout -b hotfix-v24.0.1 support/v24.0
    
  3. Fix the bug and commit the fix in one or more separate commits into your hotfix branch.
  4. Push your bug fixes and create a pull request against the support branch: https://github.com/mozilla/glean/compare/support/v24.0...your-name:hotfix-v24.0.1?expand=1
  5. When that pull request lands, wait for CI to finish on that branch and ensure it's green:

Finish a support branch

  1. Check out the support branch:
    git checkout support/v24.0
    
  2. Update the changelog .
    1. Add any missing important changes under the Unreleased changes headline.
    2. Commit any changes to the changelog file due to the previous step.
  3. Run bin/prepare-release.sh <new version> to bump the version number.
    1. The new version should be the next patch version of the support branch.
    2. Let it create a commit for you.
  4. Push the support branch:
    git push upstream support/v24.0
    
  5. Tag the release on GitHub:
    1. Draft a New Release in the GitHub UI (Releases > Draft a New Release).
    2. Enter v<myversion> as the tag. It's important this is the same as the version you specified to the prepare_release.sh script, with the v prefix added.
    3. Select the support branch (e.g. support/v24.0) as the target.
    4. Under the description, paste the contents of the release notes from CHANGELOG.md.
  6. Wait for the CI build to complete for the tag.
  7. Release the Rust crates:
    cd glean-core
    cargo publish --verbose
    cd ffi
    cargo publish --verbose
    
  8. Send a pull request to merge back any bug fixes to the development branch: https://github.com/mozilla/glean/compare/main...support/v24.0?expand=1
    • This is important so that no changes are lost.
    • This might have merge conflicts with the main branch, which you need to fix before it is merged.
  9. Once the above pull request lands, delete the support branch.

Upgrading android-components to a new version of Glean

On Android, Mozilla products consume the Glean SDK through its wrapper in android-components. Therefore, when a new Glean SDK release is made, android-components must also be updated.

After following one of the above instructions to make a Glean SDK release:

  1. Ensure that CI has completed and the artifacts are published on Mozilla's Maven repository.

  2. Create a pull request against android-components to update the Glean version with the following changes:

    • The Glean version is updated in the mozilla_glean variable in the buildSrc/src/main/java/Dependencies.kt file.

    • The relevant parts of the Glean changelog copied into the top part of the android-components changelog. This involves copying the Android-specific changes and the general changes to Glean, but can omit other platform-specific changes.

IMPORTANT: Until the Glean Gradle plugin work is complete, all downstream consumers of android-components will also need to update their version of Glean to match the version used in android-components so that their unit tests can run correctly.

In Fenix, for example, the Glean version is specified here.

Contributing to the Glean SDK

Anyone is welcome to help with the Glean SDK project. Feel free to get in touch with other community members on chat.mozilla.org or through issues on GitHub or Bugzilla.

Participation in this project is governed by the Mozilla Community Participation Guidelines.

Bug Reports

To report issues or request changes, file a bug in Bugzilla in Data Platform & Tools :: Glean: SDK.

If you don't have a Bugzilla account, we also accept issues on GitHub.

Making Code Changes

To work on the code in this repository you will need to be familiar with the Rust programming language. You can get a working rust compiler and toolchain via rustup.

You can check that everything compiles by running the following from the root of your checkout:

make test-rust

If you plan to work on the Android component bindings, you should also review the instructions for setting up an Android build environment.

To run all Kotlin tests:

make test-kotlin

or run tests in Android Studio.

To run all Swift tests:

make test-swift

or run tests in Xcode.

Sending Pull Requests

Patches should be submitted as pull requests (PRs).

Before submitting a PR:

When submitting a PR:

Code Review

This project is production Mozilla code and subject to our engineering practices and quality standards. Every patch must be peer reviewed by a member of the Glean core team.

Reviewers are defined in the CODEOWNERS file and are automatically added for every pull request. Every pull request needs to be approved by at least one of these people before landing.

The submitter needs to decide on their own discretion whether the changes require a look from more than a single reviewer or any outside developer. Reviewers can also ask for additional approval from other reviewers.

Release

See the Release process on how to release a new version of the Glean SDK.

Code Coverage

In computer science, test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage. (Wikipedia)

Generating Kotlin reports locally

Locally you can generate a coverage report with the following command:

./gradlew -Pcoverage :glean:build

After that you'll find an HTML report at the following location:

glean-core/android/build/reports/jacoco/jacocoTestReport/jacocoTestReport/html/index.html

Generating Rust reports locally

Generating the Rust coverage report requires a significant amount of RAM during the build.

We use grcov to collect and aggregate code coverage information. Releases can be found on the grcov Release page.

The build process requires a Rust Nightly version. Install it using rustup:

rustup toolchain add nightly

To generate an HTML report, genhtml from the lcov package is required. Install it through your system's package manager.

After installation you can build the Rust code and generate a report:

export CARGO_INCREMENTAL=0
export RUSTFLAGS='-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads'
cargo +nightly test

zip -0 ccov.zip `find . \( -name "glean*.gc*" \) -print`
grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o lcov.info
genhtml -o report/ --show-details --highlight --ignore-errors source --legend lcov.info

After that you'll find an HTML report at the following location:

report/index.html

Generating Swift reports locally

Xcode automatically generates code coverage when running tests. You can find the report in the Report Navigator (View -> Navigators -> Show Report Navigator -> Coverage).

Generating Python reports locally

Python code coverage is determined using the coverage.py library.

Run

make python-coverage

to generate code coverage reports in the Glean virtual environment.

After running, the report will be in htmlcov.

Developing documentation

The documentation in this repository pertains to the Glean SDK. That is, the client-side code for Glean telemetry. Documentation for Glean in general, and the Glean-specific parts of the data pipeline and analysis is documented elsewhere in the firefox-data-docs repository.

The main narrative documentation is written in Markdown and converted to static HTML using mdbook.

API docs are also generated from docstrings for Rust, Kotlin, Swift and Python.

Building documentation

Building the narrative (book) documentation

The mdbook crate is required in order to build the narrative documentation:

cargo install mdbook mdbook-mermaid mdbook-open-on-gh

Then both the narrative and Rust API docs can be built with:

make docs
# ...or...
bin/build-rust-docs.sh
# ...or on Windows...
bin/build-rust-docs.bat

The built narrative documentation is saved in build/docs/book, and the Rust API documentation is saved in build/docs/docs.

Building API documentation

Kotlin API documentation is generated using dokka. It is automatically installed by Gradle.

To build the Kotlin API documentation:

make kotlin-docs

The generated documentation is saved in build/docs/javadoc.

Swift API documentation is generated using jazzy. It can be installed using:

  1. Install the latest Ruby: brew install ruby
  2. Make the installed Ruby available: export PATH=/usr/local/opt/ruby/bin:$PATH (and add that line to your .bashrc)
  3. Install the documentation tool: gem install jazzy

To build the Swift API documentation:

make swift-docs

The generated documentation is saved in build/docs/swift.

The Python API docs are generated using pdoc3. It is installed as part of creating the virtual environment for Python development.

To build the Python API documentation:

make python-docs

The generated documentation is saved in build/docs/python.

TODO. To be implemented in bug 1648410.

Checking links

Internal links within the documentation can be checked using the link-checker tool. External links are currently not checked, since this takes considerable time and frequently fails in CI due to networking restrictions or issues.

Link checking requires building the narrative documentation as well as all of the API documentation for all languages. It is rare to build all of these locally (and in particular, the Swift API documentation can only be built on macOS), therefore it is reasonable to let CI catch broken link errors for you.

If you do want to run the link-checker locally, it can be installed using npm or your system's package manager. Then, run link-checker with:

make linkcheck

Spell checking

The narrative documentation (but not the API documentation) is spell checked using aspell.

On Unix-like platforms, it can be installed using your system's package manager:

sudo apt install aspell-en
# ...or...
sudo dnf install aspell-en
# ...or...
brew install aspell

Note that aspell 0.60.8 or later is required, as that is the first release with markdown support.

You can the spell check the narrative documentation using the following:

make spellcheck
# ...or...
bin/spellcheck.sh

This will bring up an interactive spell-checking environment in the console. Pressing a to add a word will add it to the project's local .dictionary file, and the changes should be committed to git.

Upgrading glean_parser

To upgrade the version of glean_parser used by the Glean SDK, run the bin/update-glean-parser-version.sh script, providing the version as a command line parameter:

bin/update-glean-parser-version.sh 1.28.3

This will update the version in all of the required places. Commit those changes to git and submit a pull request.

No further steps are required to use the new version of glean_parser for code generation: all of the build integrations automatically update glean_parser to the correct version.

For testing the Glean Python bindings, the virtual environment needs to be deleted to force an upgrade of glean_parser:

rm -rf glean-core/python/.venv*

Android bindings

The Glean SDK contains the Kotlin bindings for use by Android applications. It makes use of the underlying Rust component with a Kotlin-specific API on top. It also includes integrations into the Android platform.

Setup the Android Build Environment

Doing a local build of the Glean SDK:

This document describes how to make local builds of the Android bindings in this repository. Most consumers of these bindings do not need to follow this process, but will instead use pre-built bindings.

Prepare your build environment

Typically, this process only needs to be run once, although periodically you may need to repeat some steps (e.g., Rust updates should be done periodically).

Setting up Android dependencies

With Android Studio

The easiest way to install all the dependencies (and automatically handle updates), is by using Android Studio. Once this is installed, start Android Studio and open the Glean project. If Android Studio asks you to upgrade the version of Gradle, decline.

The following dependencies can be installed in Android Studio through Tools > SDK Manager > SDK Tools:

You should be set to build Glean from within Android Studio now.

Manually

Set JAVA_HOME to be the location of Android Studio's JDK which can be found in Android Studio's "Project Structure" menu (you may need to escape spaces in the path).

Note down the location of your SDK. If installed through Android Studio you will find the Android SDK Location in Tools > SDK Manager.

Set the ANDROID_HOME environment variable to that path. Alternatively add the following line to the local.properties file in the root of the Glean checkout (create the file if it does not exist):

sdk.dir=/path/to/sdk

For the Android NDK:

  1. Download NDK r21 from https://developer.android.com/ndk/downloads.
  2. Extract it and put it somewhere ($HOME/.android-ndk-r21 is a reasonable choice, but it doesn't matter).
  3. Add the following line to the local.properties file in the root of the Glean checkout (create the file if it does not exist):
    ndk.dir=/path/to/.android-ndk-r21
    

Setting up Rust

Rust can be installed using rustup, with the following commands:

curl https://sh.rustup.rs -sSf | sh
rustup update

Platform specific toolchains need to be installed for Rust. This can be done using the rustup target command. In order to enable building for real devices and Android emulators, the following targets need to be installed:

rustup target add aarch64-linux-android
rustup target add armv7-linux-androideabi
rustup target add i686-linux-android
rustup target add x86_64-linux-android

Building

Before building:

With Android Studio

After importing the Glean project into Android Studio it should be all set up and you can build the project using Build > Make Project

Manually

The builds are all performed by ./gradlew and the general syntax used is ./gradlew project:task

Build the whole project and run tests:

./gradlew build

You can see a list of projects by executing ./gradlew projects and a list of tasks by executing ./gradlew tasks.

FAQ

Android SDK / NDK versions

The Glean SDK implementation is currently build against the following versions:

For the full setup see Setup the Android Build Environment.

The versions are defined in the following files. All locations need to be updated on upgrades:

Working on unreleased Glean code in android-components

This is a companion to the equivalent instructions for the android-components repository.

Modern Gradle supports composite builds, which allows to substitute on-disk projects for binary publications. Composite builds transparently accomplish what is usually a frustrating loop of:

  1. change library
  2. publish library snapshot to the local Maven repository
  3. consume library snapshot in application

Preparation

Clone the Glean SDK and android-components repositories:

git clone https://github.com/mozilla/glean
git clone https://github.com/mozilla-mobile/android-components

Cargo build targets

By default when building Android Components using gradle, the gradle-cargo plugin will compile Glean for every possible platform, which might end up in failures. You can customize which targets are built in glean/local.properties (more information):

# For physical devices:
rust.targets=arm

# For unit tests:
# rust.targets=darwin # Or linux-*, windows-* (* = x86/x64)

# For emulator only:
rust.targets=x86

Substituting projects

android-components has custom build logic for dealing with composite builds, so you should be able to configure it by simply adding the path to the Glean repository in the correct local.properties file:

In android-components/local.properties:

substitutions.glean.dir=../glean

If this doesn't seem to work, or if you need to configure composite builds for a project that does not contain this custom logic, add the following to settings.gradle:

In android-components/settings.gradle:

includeBuild('../glean') {
  dependencySubstitution {
    substitute module('org.mozilla.telemetry:glean') with project(':glean')
    substitute module('org.mozilla.telemetry:glean-forUnitTests') with project(':glean')
  }
}

Composite builds will ensure the Glean SDK is build as part of tasks inside android-components.

Caveat

There's a big gotcha with library substitutions: the Gradle build computes lazily, and AARs don't include their transitive dependencies' JNI libraries. This means that in android-components, ./gradlew :service-glean:assembleDebug does not invoke :glean:cargoBuild, even though :service-glean depends on the substitution for :glean and even if the inputs to Cargo have changed! It's the final consumer of the :service-glean project (or publication) that will incorporate the JNI libraries.

In practice that means you should always be targeting something that produces an APK: a test or a sample module. Then you should find that the cargoBuild tasks are invoked as you expect.

Inside the android-components repository ./gradlew :samples-glean:connectedAndroidTest should work. Other tests like :service-glean:testDebugUnitTest or :support-sync-telemetry:testDebugUnitTest will currently fail, because the JNI libraries are not included.

Notes

This document is based on the equivalent documentation for application-services: Development with the Reference Browser

  1. Transitive substitutions (as shown above) work but require newer Gradle versions (4.10+).

Using locally-published Glean in Fenix

Note: This is a bit tedious, and you might like to try the substitution-based > approach documented in Working on unreleased Glean code in android-components. That approach is still fairly new, and the local-publishing approach in this document is necessary if it fails.

Note: This is Fenix-specific only in that some links on the page go to the mozilla-mobile/fenix repository, however these steps should work for e.g. reference-browser, as well. (Same goes for Lockwise, or any other consumer of Glean, but they may use a different structure -- Lockwise has no Dependencies.kt, for example)

Preparation

Clone the Glean SDK, android-components and Fenix repositories:

git clone https://github.com/mozilla/glean
git clone https://github.com/mozilla-mobile/android-components
git clone https://github.com/mozilla-mobile/fenix/

Local publishing

  1. Inside the glean repository root:

    1. In .buildconfig.yml, change libraryVersion to end in -TESTING$N1, where $N is some number that you haven't used for this before.

      Example: libraryVersion: 22.0.0-TESTING1

    2. Check your local.properties file, and add rust.targets=x86 if you're testing on the emulator, rust.targets=arm if you're testing on 32-bit arm (arm64 for 64-bit arm, etc). This will make the build that's done in the next step much faster.

    3. Run ./gradlew publishToMavenLocal. This may take a few minutes.

  2. Inside the android-components repository root:

    1. In .buildconfig.yml, change componentsVersion to end in -TESTING$N1, where $N is some number that you haven't used for this before.

      Example: componentsVersion: 24.0.0-TESTING1

    2. Inside buildSrc/src/main/java/Dependencies.kt, change mozilla_glean to reference the libraryVersion you published in step 2 part 1.

      Example: const val mozilla_glean = "22.0.0-TESTING1"

    3. Inside build.gradle, add mavenLocal() inside allprojects { repositories { <here> } }.

    4. Add the following block in the settings.gradle file, before any other statement in the script, in order to have the Glean Gradle plugin loaded from the local Maven repository.

    pluginManagement {
      repositories {
          mavenLocal()
          gradlePluginPortal()
      }
    }
    
    1. Inside the android-component's local.properties file, ensure substitutions.glean.dir is NOT set.

    2. Run ./gradlew publishToMavenLocal.

  3. Inside the fenix repository root:

    1. Inside build.gradle, add mavenLocal() inside allprojects { repositories { <here> } }.

    2. Inside buildSrc/src/main/java/Dependencies.kt, change mozilla_android_components to the version you defined in step 3 part 1.

      Example: const val mozilla_android_components = "24.0.0-TESTING1"

      In the same file change mozilla_glean to the version you defined in step 1 part 1.

      Example: const val mozilla_glean = "22.0.0-TESTING1"

    3. Change the Versions.mozilla_android_components.endsWith('-SNAPSHOT') line in app/build.gradle to Versions.mozilla_android_components.endsWith('-TESTING$N') where $N is the number to reference the version you published in part 2.

You should now be able to build and run Fenix (assuming you could before all this).

Caveats

  1. This assumes you have followed the Android/Rust build setup
  2. Make sure you're fully up to date in all repositories, unless you know you need to not be.
  3. This omits the steps if changes needed because, e.g. Glean made a breaking change to an API used in android-components. These should be understandable to fix, you usually should be able to find a PR with the fixes somewhere in the android-component's list of pending PRs (or, failing that, a description of what to do in the Glean changelog).
  4. Ask in the #glean channel on chat.mozilla.org.

Notes

This document is based on the equivalent documentation for application-services: Using locally-published components in Fenix


1

It doesn't have to end with -TESTING$N, it only needs to have the format -someidentifier. -SNAPSHOT$N is also very common to use, however without the numeric suffix, this has specific meaning to gradle, so we avoid it. Additionally, while the $N we have used in our running example has matched (e.g. all of the identifiers ended in -TESTING1, this is not required, so long as you match everything up correctly at the end).

iOS bindings

The Glean SDK contains the Swift bindings for use by iOS applications. It makes use of the underlying Rust component with a Swift-specific API on top. It also includes integrations into the iOS platform.

Setup the iOS Build Environment

Prepare your build environment

  1. Install Xcode 11.0
  2. Install Carthage: brew install carthage
  3. Ensure you have Python 3 installed: brew install python
  4. Install linting and formatting tools: brew install swiftlint
  5. Run bin/bootstrap.sh to download dependencies.
  6. (Optional, only required for building on the CLI) Install xcpretty: gem install xcpretty

Setting up Rust

Rust can be installed using rustup, with the following commands:

Platform specific toolchains need to be installed for Rust. This can be done using the rustup target command. In order to enable building to real devices and iOS emulators, the following targets need to be installed:

rustup target add aarch64-apple-ios x86_64-apple-ios

Building

This should be relatively straightforward and painless:

  1. Ensure your repository is up-to-date.

  2. Ensure Rust is up-to-date by running rustup update.

  3. Run a build using the command make build-swift

    • To run tests use make test-swift

The above can be skipped if using Xcode. The project directory can be imported and all the build details can be left to the IDE.

Debug an iOS application against different builds of Glean

At times it may be necessary to debug against a local build of Glean or another git fork or branch in order to test new features or specific versions of Glean.

Since Glean is consumed through Carthage, this can be as simple as modifying the Cartfile for the consuming application.

Building against the latest Glean

For consuming the latest version of Glean, the Cartfile contents would include the following line:

github "mozilla/glean" "main"

This will fetch and compile Glean from the mozilla/glean GitHub repository from the "main" branch.

Building against a specific release of Glean

For consuming a specific version of Glean, you can specify a branch name, tag name, or commit ID, like this:

github "mozilla/glean" "v0.0.1"

Where v0.0.1 is a tagged release name, but could also be a branch name or a specific commit ID like 832b222

If the custom Glean you wish to build from is a different fork on GitHub, you could simply modify the Cartfile to point at your fork like this:

github "myGitHubHandle/glean" "myBranchName"

Replace the myGitHubHandle and myBranchName with your GitHub handle and branch name, as appropriate.

Build from a locally cloned Glean

You can also use Carthage to build from a local clone by replacing the Cartfile line with the following:

git "file:///Users/yourname/path/to/glean" "localBranchName"

Notice that the initial Carthage command is git now instead of github, and we need to use a file URL of the path to the locally cloned Glean

Perform the Carthage update

One last thing not to forget is to run the carthage update command in the directory your Cartfile resides in order to fetch and build Glean.

Once that is done, your local application should be building against the version of Glean that you specified in the Cartfile.

Python bindings

The Glean SDK contains Python bindings for use in Python applications and test frameworks. It makes use of the underlying Rust component with a Python-specific API on top.

Setup the Python Build Environment

This document describes how to set up an environment for the development of the Glean Python bindings.

Instructions for installing a copy of the Glean Python bindings into your own environment for use in your project are described in adding Glean to your project.

Prerequisites

Glean requires Python 3.6 or later.

Make sure it is installed on your system and accessible on the PATH as python3.

Setting up Rust

If you've already set up Rust for building Glean for Android or iOS, you already have everything you need and can skip this section.

Rust can be installed using rustup, with the following commands:

Create a virtual environment

It is recommended to do all Python development inside a virtual environment to make sure there are no interactions with other Python libraries installed on your system.

You may want to manage your own virtual environment if, for example, you are developing a library that is using Glean and you want to install Glean into it. If you are just developing Glean itself, however, Glean's Makefile will handle the creation and use of a virtual environment automatically (though the Makefile unfortunately does not work on Microsoft Windows).

The instructions below all have both the manual instructions and the Makefile shortcuts.

Manual method

The following instructions use the basic virtual environment functionality that comes in the Python standard library. Other higher level tools exist to manage virtual environments, such as pipenv and conda. These will work just as well, but are not documented here. Using an alternative tool would replace the instructions in this section only, but all other instructions below would remain the same.

To create a virtual environment, enter the following command, replacing <venv> with the path to the virtual environment you want to create. You will need to do this only once.

  $ python3 -m venv <venv>

Then activate the environment. You will need to do this for each new shell session when you want to develop Glean. The command to use depends on your shell environment.

PlatformShellCommand to activate virtual environment
POSIXbash/zshsource <venv>/bin/activate
fish. <venv>/bin/activate.fish
csh/tcshsource <venv>bin/activate.csh
PowerShell Core<venv>/bin/Activate.ps1
Windowscmd.exe<venv>\Scripts\activate.bat
PowerShell<venv>\Scripts\Activate.ps1

Lastly, install Glean's Python development dependencies into the virtual environment.

cd glean-core/python
pip install -r requirements_dev.txt

Makefile method

  $ make python-setup

The default location of the virtual environment used by the make file is glean-core/python/.venvX.Y, where X.Y is the version of Python in use. This makes it possible to build and test for multiple versions of Python in the same checkout.

Note: If you wish to change the location of the virtual environment that the Makefile uses, pass the GLEAN_PYENV environment variable: make python-setup GLEAN_PYENV=mypyenv.

By default, the Makefile installs the latest version available of each of Glean's dependencies. If you wish to install the minimum required versions instead, (useful primarily to ensure that Glean doesn't unintentionally use newer APIs in its dependencies) pass the GLEAN_PYDEPS=min environment variable: make python-setup GLEAN_PYDEPS=min.

Build the Python bindings

Manual method

Building the Python bindings also builds the Rust shared object for the Glean SDK core.

  $ cd glean-core/python
  $ python setup.py build install

Makefile method

This will implicitly setup the Python environment, rebuild the Rust core and then build the Python bindings.

  $ make build-python

Running the Python unit tests

Manual method

Make sure the Python bindings are built, then:

  $ cd glean-core/python
  $ py.test

Makefile method

The following will run the Python unit tests using py.test:

  $ make test-python

You can send extra parameters to the py.test command by setting the PYTEST_ARGS variable:

  $ make test-python PYTEST_ARGS="-s --pdb"

Viewing logging output

Log messages (whether originating in Python or Rust) are emitted using the Python standard library's logging module. This module provides a lot of possibilities for customization, but the easiest way to control the log level globally is with logging.basicConfig:

import logging
logging.basicConfig(level=logging.DEBUG)

Linting, formatting and type checking

The Glean Python bindings use the following tools:

Manual method

  $ cd glean-core/python
  $ flake8 glean tests
  $ black glean tests
  $ mypy glean

Makefile method

To just check the lints:

  $ make pythonlint

To reformat the Python files in-place:

  $ make pythonfmt

Building the Python API docs

The Python API docs are built using pdoc3.

Manual method

  $ python -m pdoc --html glean --force -o build/docs/python

Makefile method

  $ make python-docs

Building wheels for Linux

Building on Linux using the above instructions will create Linux binaries that dynamically link against the version of libc installed on your machine. This generally will not be portable to other Linux distributions, and PyPI will not even allow it to be uploaded. In order to create wheels that can be installed on the broadest range of Linux distributions, the Python Packaging Authority's manylinux project maintains a Docker image for building compatible Linux wheels.

The CircleCI configuration handles making these wheels from tagged releases. If you need to reproduce this locally, see the CircleCI job pypi-linux-release for an example of how this Docker image is used in practice.

Building wheels for Windows

The official wheels for Windows are produced on a Linux virtual machine using the Mingw toolchain.

The CircleCI configuration handles making these wheels from tagged releases. If you need to reproduce this locally, see the CircleCI job pypi-windows-release for an example of how this is done in practice.

Rust component

The majority of the Glean SDK is implemented as a Rust component, to be usable across all platforms.

This includes:

Rust documentation guidelines

The documentation for the Glean SDK Rust crates is automatically generated by the rustdoc tool. That means the documentation for our Rust code is written as doc comments throughout the code.

Here we outline some guidelines to be followed when writing these doc comments.

Module level documentation

Every module's index file must have a module level doc comment.

The module level documentation must start with a one-liner summary of the purpose of the module, the rest of the text is free form and optional.

Function and constants level documentation

Every public function / constant must have a dedicated doc comment.

Non public functions / constants are not required to, but may have doc comments. These comments should follow the same structure as public functions.

Note Private functions are not added to the generated documentation by default, but may be by using the --document-private-items option.

Structure

Below is an example of the structure of a doc comment in one of the Glean SDK's public functions.


#![allow(unused_variables)]
fn main() {
/// Collects and submits a ping for eventual uploading.
///
/// The ping content is assembled as soon as possible, but upload is not
/// guaranteed to happen immediately, as that depends on the upload policies.
///
/// If the ping currently contains no content, it will not be sent,
/// unless it is configured to be sent if empty.
///
/// # Arguments
///
/// * `ping` - The ping to submit
/// * `reason` - A reason code to include in the ping
///
/// # Returns
///
/// Whether the ping was succesfully assembled and queued.
///
/// # Errors
///
/// If collecting or writing the ping to disk failed.
pub fn submit_ping(&self, ping: &PingType, reason: Option<&str>) -> Result<bool> {
    ...
}
}

Every doc comment must start with a one-liner summary of what that function is about

The text of this section should always be in the third person and describe an action.

If any additional information is necessary, that can be added following a new line after the summary.

Depending on the signature and behavior of the function, additional sections can be added to its doc comment

These sections are:

  1. # Arguments - A list of the functions arguments and a short description of what they are. Note that there is no need to point out the type of the given argument on the doc comment since it is already outlined in the code.
  2. # Returns - A short summary of what this function returns. Note that there is no need to start this summary with "Returns..." since that is already the title of the section.
  3. # Errors - The possible paths that will or won't lead this function to return an error. It is at the discretion of the programmer to decide which cases are worth pointing out here.
  4. # Panics - The possible paths that will or won't lead this function to panic!. It is at the discretion of the programmer to decide which cases are worth pointing out here.
  5. # Safety - In case there are any safety concerns or guarantees, this is the section to point them out. It is at the discretion of the programmer to decide which cases are worth pointing out here.
  6. # Examples - If necessary, this section holds examples of usage of the function.

These sections should only be added to a doc comment when necessary, e.g. we only add an # Arguments section when a function has arguments.

Note The titles of these sections must be level-1 headers. Additional sections not listed here can be level-2 or below.

Link to all the things

If the doc comment refers to some external structure, either from the Glean SDK's code base or external, make sure to link to it.

Example:


#![allow(unused_variables)]
fn main() {
/// Gets a [`PingType`] by name.
///
/// # Returns
///
/// The [`PingType`] of a ping if the given name was registered before, `None` otherwise.
///
/// [`PingType`]: metrics/struct.PingType.html
pub fn get_ping_by_name(&self, ping_name: &str) -> Option<&PingType> {
    ...
}
}

Note To avoid polluting the text with multiple links, prefer using reference links which allow for adding the actual links at the end of the text block.

Warnings

Important warnings about a function should be added, when necessary, before the one-liner summary in bold text.

Example:


#![allow(unused_variables)]
fn main() {
/// **Test-only API (exported for FFI purposes).**
///
/// Deletes all stored metrics.
///
/// Note that this also includes the ping sequence numbers, so it has
/// the effect of resetting those to their initial values.
pub fn test_clear_all_stores(&self) {
    ...
}
}

Semantic line breaks

Prefer using semantic line breaks when breaking lines in any doc or non-doc comment.

References

These guidelines are largely based on the Rust API documentation guidelines.

Dependency Management Guidelines

This repository uses third-party code from a variety of sources, so we need to be mindful of how these dependencies will affect our consumers. Considerations include:

We're still evolving our policies in this area, but these are the guidelines we've developed so far.

Rust Code

Unlike Firefox, we do not vendor third-party source code directly into the repository. Instead we rely on Cargo.lock and its hash validation to ensure that each build uses an identical copy of all third-party crates. These are the measures we use for ongoing maintenance of our existing dependencies:

Adding a new dependency, whether we like it or not, is a big deal - that dependency and everything it brings with it will become part of Firefox-branded products that we ship to end users. We try to balance this responsibility against the many benefits of using existing code, as follows:

Updating to new versions of existing dependencies is a normal part of software development and is not accompanied by any particular ceremony.

Dependency updates

We use Dependabot to automatically open pull requests when dependencies release a new version. All CI tasks run on those pull requests to ensure updates don't break anything.

As glean-core is now also vendored into mozilla-central, including all of its Rust dependencies, we need to be a bit careful about these updates. Landing upgrades of the Glean SDK itself in mozilla-central is a separate process. See Updating the Glean SDK in the Firefox Source Docs. Following are some guidelines to ensure compatibility with mozilla-central:

In case of uncertainty defer a decision to :janerik or :chutten.

Manual dependency updates

You can manually check for outdated dependencies using cargo-outdated. Individual crate updates for version compatible with the requirement in Cargo.toml can be done using:

cargo update -p $cratename

To update all transitive dependencies to the latest semver-compatible version use:

cargo update

Adding a new metric type

Data in the Glean SDK is stored in so-called metrics. You can find the full list of implemented metric types in the user overview.

Adding a new metric type involves defining the metric type's API, its persisted and in-memory storage as well as its serialization into the ping payload.

The metric type's API

A metric type implementation is defined in its own file under glean-core/src/metrics/, e.g. glean-core/src/metrics/counter.rs for a Counter.

Start by defining a structure to hold the metric's metadata:

#[derive(Clone, Debug)]
pub struct CounterMetric {
    meta: CommonMetricData
}

Implement the MetricType trait to create a metric from the meta data as well as expose the meta data. This also gives you a should_record method on the metric type.

impl MetricType for CounterMetric {
    fn meta(&self) -> &CommonMetricData {
        &self.meta
    }

    fn meta_mut(&mut self) -> &mut CommonMetricData {
        &mut self.meta
    }
}

Its implementation should have a way to create a new metric from the common metric data. It should be the same for all metric types.

impl CounterMetric {
    pub fn new(meta: CommonMetricData) -> Self {
        Self { meta }
    }
}

Implement each method for the type. The first argument to accept should always be glean: &Glean, that is: a reference to the Glean object, used to access the storage:

impl CounterMetric { // same block as above
    pub fn add(&self, glean: &Glean, amount: i32) {
        // Always include this check!
        if !self.should_record() {
            return;
        }

        // Do error handling here

        glean
            .storage()
            .record_with(&self.meta, |old_value| match old_value {
                Some(Metric::Counter(old_value)) => Metric::Counter(old_value + amount),
                _ => Metric::Counter(amount),
            })
    }
}

Use glean.storage().record() to record a fixed value or glean.storage.record_with() to construct a new value from the currently stored one.

The storage operation makes use of the metric's variant of the Metric enumeration.

The Metric enumeration

Persistence and in-memory serialization as well as ping payload serialization are handled through the Metric enumeration. This is defined in glean-core/src/metrics/mod.rs. Variants of this enumeration are used in the storage implementation of the metric type.

To add a new metric type, include the metric module and declare its use, then add a new variant to the Metric enum:


mod counter;

// ...

pub use self::counter::CounterMetric;

#[derive(Serialize, Deserialize, Debug, Clone)]
pub enum Metric {
    // ...
    Counter(i32),
}

Then modify the below implementation and define the right ping section name for the new type. This will be used in the ping payload:

impl Metric {
    pub fn ping_section(&self) -> &'static str {
        match self {
            // ...
            Metric::Counter(_) => "counter",
        }
    }
}

Finally, define the ping payload serialization (as JSON). In the simple cases where the in-memory representation maps to its JSON representation it is enough to call the json! macro.

impl Metric { // same block as above
    pub fn as_json(&self) -> JsonValue {
        match self {
            // ...
            Metric::Counter(c) => json!(c),
        }
    }
}

For more complex serialization consider implementing serialization logic as a function returning a serde_json::Value or another object that can be serialized.

For example, the DateTime serializer has the following entry, where get_iso_time_string is a function to convert from the DateTime metric representation to a string:

Metric::Datetime(d, time_unit) => json!(get_iso_time_string(*d, *time_unit)),

In the next step we will create the FFI wrapper and platform-specific wrappers.

Adding a new metric type - FFI layer

In order to use a new metric type over the FFI layer, it needs implementations in the FFI component.

FFI component

The FFI component implementation can be found in glean-core/ffi/src. Each metric type is implemented in its own module.

Add a new file named after your metric, e.g. glean-core/ffi/src/counter.rs, and declare it in glean-core/ffi/src/lib.rs with mod counter;.

In the metric type module define your metric type using the define_metric macro. This allows referencing the metric name and defines the global map as well as some common functions such as the constructor and destructor. Simple operations can be also defined in the same macro invocation:

use crate::{define_metric, handlemap_ext::HandleMapExtension, GLEAN};

define_metric!(CounterMetric => COUNTER_METRICS {
    new           -> glean_new_counter_metric(),
    destroy       -> glean_destroy_counter_metric,

    add -> glean_counter_add(amount: i32),
});

More complex operations need to be defined as plain functions. For example the test helper function for a counter metric can be defined as:

#[no_mangle]
pub extern "C" fn glean_counter_test_has_value(
    glean_handle: u64,
    metric_id: u64,
    storage_name: FfiStr,
) -> u8 {
    GLEAN.call_infallible(glean_handle, |glean| {
        COUNTER_METRICS.call_infallible(metric_id, |metric| {
            metric
                .test_get_value(glean, storage_name.as_str())
                .is_some()
        })
    })
}

Adding a new metric type - Kotlin

FFI

The platform-specific FFI wrapper needs the definitions of these new functions. For Kotlin this is in glean-core/android/src/main/java/mozilla/telemetry/glean/rust/LibGleanFFI.kt:

fun glean_new_counter_metric(category: String, name: String, send_in_pings: StringArray, send_in_pings_len: Int, lifetime: Int, disabled: Byte): Long
fun glean_destroy_counter_metric(handle: Long)
fun glean_counter_add(glean_handle: Long, metric_id: Long, amount: Int)

Kotlin API

Finally, create a platform-specific metric type wrapper. For Kotlin this would be glean-core/android/src/main/java/mozilla/telemetry/glean/private/CounterMetricType.kt:

class CounterMetricType(
    private var handle: Long,
    private val disabled: Boolean,
    private val sendInPings: List<String>
) {
    /**
     * The public constructor used by automatically generated metrics.
     */
    constructor(
        disabled: Boolean,
        category: String,
        lifetime: Lifetime,
        name: String,
        sendInPings: List<String>
    ) : this(handle = 0, disabled = disabled, sendInPings = sendInPings) {
        val ffiPingsList = StringArray(sendInPings.toTypedArray(), "utf-8")
        this.handle = LibGleanFFI.INSTANCE.glean_new_counter_metric(
                category = category,
                name = name,
                send_in_pings = ffiPingsList,
                send_in_pings_len = sendInPings.size,
                lifetime = lifetime.ordinal,
                disabled = disabled.toByte())
    }

    fun add(amount: Int = 1) {
        if (disabled) {
            return
        }

        @Suppress("EXPERIMENTAL_API_USAGE")
        Dispatchers.API.launch {
            LibGleanFFI.INSTANCE.glean_counter_add(
                Glean.handle,
                this@CounterMetricType.handle,
                amount)
        }
    }

    @VisibleForTesting(otherwise = VisibleForTesting.NONE)
    fun testHasValue(pingName: String = sendInPings.first()): Boolean {
        @Suppress("EXPERIMENTAL_API_USAGE")
        Dispatchers.API.assertInTestingMode()

        val res = LibGleanFFI.INSTANCE.glean_counter_test_has_value(Glean.handle, this.handle, pingName)
        return res.toBoolean()
    }
}

Adding a new metric type - Swift

FFI

Swift can re-use the generated C header file. Re-generate it with

make cbindgen

Swift API

Finally, create a platform-specific metric type wrapper. For Swift this would be glean-core/ios/Glean/Metrics/CounterMetric.swift:

public class CounterMetricType {
    let handle: UInt64
    let disabled: Bool
    let sendInPings: [String]

    /// The public constructor used by automatically generated metrics.
    public init(category: String, name: String, sendInPings: [String], lifetime: Lifetime, disabled: Bool) {
        self.disabled = disabled
        self.sendInPings = sendInPings
        self.handle = withArrayOfCStrings(sendInPings) { pingArray in
            glean_new_counter_metric(
                category,
                name,
                pingArray,
                Int32(sendInPings.count),
                lifetime.rawValue,
                disabled.toByte()
            )
        }
    }

    public func add(amount: Int32 = 1) {
        guard !self.disabled else { return }

        _ = Dispatchers.shared.launch {
            glean_counter_add(Glean.shared.handle, self.handle, amount)
        }
    }

    func testHasValue(_ pingName: String? = nil) -> Bool {
        let pingName = pingName ?? self.sendInPings[0]
        return glean_counter_test_has_value(Glean.shared.handle, self.handle, pingName) != 0
    }

    func testGetValue(_ pingName: String? = nil) throws -> Int32 {
        let pingName = pingName ?? self.sendInPings[0]

        if !testHasValue(pingName) {
            throw "Missing value"
        }

        return glean_counter_test_get_value(Glean.shared.handle, self.handle, pingName)
    }
}

Adding a new metric type - Python

FFI

Python can use the generated C header file directly, through cffi. Re-generate it with

make cbindgen

Python API

Finally, create a platform-specific metric type wrapper. For Python this would be glean-core/python/glean/metrics/counter.py:

class CounterMetricType:
    """
    This implements the developer facing API for recording counter metrics.

    Instances of this class type are automatically generated by
    `glean.load_metrics`, allowing developers to record values that were
    previously registered in the metrics.yaml file.

    The counter API only exposes the `CounterMetricType.add` method, which
    takes care of validating the input data and making sure that limits are
    enforced.
    """

    def __init__(
        self,
        disabled: bool,
        category: str,
        lifetime: Lifetime,
        name: str,
        send_in_pings: List[str],
    ):
        self._disabled = disabled
        self._send_in_pings = send_in_pings

        self._handle = _ffi.lib.glean_new_counter_metric(
            _ffi.ffi_encode_string(category),
            _ffi.ffi_encode_string(name),
            _ffi.ffi_encode_vec_string(send_in_pings),
            len(send_in_pings),
            lifetime.value,
            disabled,
        )

    def __del__(self):
        if getattr(self, "_handle", 0) != 0:
            _ffi.lib.glean_destroy_counter_metric(self._handle)

    def add(self, amount: int = 1):
        """
        Add to counter value.

        Args:
            amount (int): (default: 1) This is the amount to increment the
                counter by.
        """
        if self._disabled:
            return

        @Dispatcher.launch
        def add():
            _ffi.lib.glean_counter_add(self._handle, amount)

    def test_has_value(self, ping_name: Optional[str] = None) -> bool:
        """
        Tests whether a value is stored for the metric for testing purposes
        only.

        Args:
            ping_name (str): (default: first value in send_in_pings) The name
                of the ping to retrieve the metric for.

        Returns:
            has_value (bool): True if the metric value exists.
        """
        if ping_name is None:
            ping_name = self._send_in_pings[0]

        return bool(
            _ffi.lib.glean_counter_test_has_value(
                self._handle, _ffi.ffi_encode_string(ping_name)
            )
        )

    def test_get_value(self, ping_name: Optional[str] = None) -> int:
        """
        Returns the stored value for testing purposes only.

        Args:
            ping_name (str): (default: first value in send_in_pings) The name
                of the ping to retrieve the metric for.

        Returns:
            value (int): value of the stored metric.
        """
        if ping_name is None:
            ping_name = self._send_in_pings[0]

        if not self.test_has_value(ping_name):
            raise ValueError("metric has no value")

        return _ffi.lib.glean_counter_test_get_value(
            self._handle, _ffi.ffi_encode_string(ping_name)
        )

    def test_get_num_recorded_errors(
        self, error_type: ErrorType, ping_name: Optional[str] = None
    ) -> int:
        """
        Returns the number of errors recorded for the given metric.

        Args:
            error_type (ErrorType): The type of error recorded.
            ping_name (str): (default: first value in send_in_pings) The name
                of the ping to retrieve the metric for.

        Returns:
            num_errors (int): The number of errors recorded for the metric for
                the given error type.
        """
        if ping_name is None:
            ping_name = self._send_in_pings[0]

        return _ffi.lib.glean_counter_test_get_num_recorded_errors(
            self._handle, error_type.value, _ffi.ffi_encode_string(ping_name),
        )

The new metric type also needs to be imported from glean-core/python/glean/metrics/__init__.py:

from .counter import CounterMetricType


__all__ = [
    "CounterMetricType",
    # ...
]

It also must be added to the _TYPE_MAPPING in glean-core/python/glean/_loader.py:

_TYPE_MAPPING = {
    "counter": metrics.CounterMetricType,
    # ...
}

FFI layer

The core part of the Glean SDK provides an FFI layer for its API. This FFI layer can be consumed by platform-specific bindings, such as the Kotlin bindings for Android.

When to use what method of passing data between Rust and Java/Swift

There are a bunch of options here. For the purposes of our discussion, there are two kinds of values you may want to pass over the FFI.

  1. Types with identity (includes stateful types, resource types, or anything that isn't really serializable).
  2. Plain ol' data.

Types with identity

Examples of this are all metric type implementations, e.g. StringMetric. These types are complex, implemented in Rust and make use of the global Glean singleton on the Rust side, They have an equivalent on the wrapper side (Kotlin/Swift/Python), that forwards calls through FFI.

They all follow the same pattern:

A ConcurrentHandleMap stores all instances of the same type, A handle is passed back and forth as a u64 from Rust, Long from Kotlin, UInt64 from Swift or other equivalent type in other languages.

This is recommended for most cases, as it's the hardest to mess up. Additionally, for types T such that &T: Sync + Send, or that you need to call &mut self method, this is the safest choice.

Additionally, this will ensure panic-safety, as it will poison the internal Mutex, making further access impossible.

The ffi_support::handle_map docs are good, and under ConcurrentHandleMap include an example of how to set this up.

Plain Old Data

This includes both primitive values, strings, arrays, or arbitrarily nested structures containing them.

Primitives

Numeric primitives are the easiest to pass between languages. The main recommendation is: use the equivalent and same-sized type as the one provided by Rust.

There are a couple of exceptions/caveats, especially for Kotlin. All of them are caused by JNA/Android issues. Swift has very good support for calling over the FFI.

  1. bool: Don't use it. JNA doesn't handle it well. Instead, use a numeric type (like u8) and represent 0 for false and 1 for true for interchange over the FFI, converting back to Kotlin's Boolean or Swift's Bool after (as to not expose this somewhat annoying limitation in our public API). All wrappers already include utility functions to turn 8-bit integers (u8) back to booleans (toBool() or equivalent methods).

  2. usize/isize: These cause the structure size to be different based on the platform. JNA does handle this if you use NativeSize, but it's awkward. Use the size-defined integers instead, such as i64/i32 and their language-equivalents (Kotlin: Long/Int, Swift:UInt64/UInt32). Caution: In Kotlin integers are signed by default. You can use u64/u32 for Long/Int if you ensure the values are non-negative through asserts or error reporting code.

  3. char: Avoid these if possible. If you really have a use case consider u32 instead.

    If you do this, you should probably be aware of the fact that Java chars are 16 bit, and Swift Characters are actually strings (they represent Extended Grapheme Clusters, not codepoints).

Strings

These we pass as null-terminated UTF-8 C-strings.

For return values, used *mut c_char, and for input, use ffi_support::FfiStr

  1. If the string is returned from Rust to Kotlin/Swift, you need to expose a string destructor from your ffi crate. See ffi_support::define_string_destructor!).

    For converting to a *mut c_char, use either rust_string_to_c if you have a String, or opt_rust_string_to_c for Option<String> (None becomes std::ptr::null_mut()).

    Important: In Kotlin, the type returned by a function that produces this must be Pointer, and not String, and the parameter that the destructor takes as input must also be Pointer.

    Using String will almost work. JNA will convert the return value to String automatically, leaking the value Rust provides. Then, when passing to the destructor, it will allocate a temporary buffer, pass it to Rust, which we'll free, corrupting both heaps 💥. Oops!

  2. If the string is passed into Rust from Kotlin/Swift, the Rust code should declare the parameter as a FfiStr<'_>. and things should then work more or less automatically. The FfiStr has methods for extracting it's data as &str, Option<&str>, String, and Option<String>.

Aggregates

This is any type that's more complex than a primitive or a string (arrays, structures, and combinations there-in). There are two options we recommend for these cases:

  1. Passing data as JSON. This is very easy and useful for prototyping, but is much slower and requires a great deal of copying and redundant encode/decode steps (in general, the data will be copied at least 4 times to make this work, and almost certainly more in practice). It can be done relatively easily by derive(Serialize, Deserialize), and converting to a JSON string using serde_json::to_string.

    This is a viable option for test-only functions, where the overhead is not important.

  2. Use repr(C) structures and copy the data across the boundary, carefully replicating the same structure on the wrapper side. In Kotlin this will require @Structure.FieldOrder annotations. Swift can directly handle C types.

    Caution: This is error prone! Structures, enumerations and unions need to be kept the same across all layers (Rust, generated C header, Kotlin, Swift, ...). Be extra careful, avoid adding references to structures and ensure the structures are correctly freed inside Rust. Copy out the data and convert into language-appropriate types (e.g. convert *mut c_char into Swift/Kotlin strings) as soon as possible.

Internal documentation

This chapter describes aspects of Glean that are internal implementation details.

This includes:

Reserved ping names

The Glean SDK reserves all ping names in send_in_pings starting with glean_.

This currently includes, but is not limited to:

Additionally, only Glean may specify all-pings. This special value has no effect in the client, but indicates to the backend infrastructure that a metric may appear in any ping.

Clearing metrics when disabling/enabling Glean

When disabling upload (Glean.setUploadEnabled(false)), metrics are also cleared to prevent their storage on the local device, and lessen the likelihood of accidentally sending them. There are a couple of exceptions to this:

When re-enabling metrics:

Payload format

The main sections of a Glean ping are described in Ping Sections. This Payload format chapter describes details of the ping payload that are relevant for decoding Glean pings in the pipeline. This is less relevant for end users of the Glean SDK.

JSON Schema

Glean's ping payloads have a formal JSON schema defined in the mozilla-pipeline-schemas project. It is written as a set of templates that are expanded by the mozilla-pipeline-schemas build infrastructure into a fully expanded schema.

Metric types

Boolean

A Boolean is represented by its boolean value.

Example

true

Counter

A Counter is represented by its integer value.

Example

17

Quantity

A Quantity is represented by its integer value.

Example

42

String

A String is represented by its string value.

Example

"sample string"

JWE

A JWE is represented by its compact representation.

Example

"eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ..48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ"

String list

A String List is represented as an array of strings.

["sample string", "another one"]

Timespan

A Timespan is represented as an object of their duration as an integer and the time unit.

Field nameTypeDescription
valueIntegerThe value in the marked time unit.
time_unitStringThe time unit, see the timespan's configuration for valid values.

Example

{
    "time_unit": "milliseconds",
    "value": 10
}

Timing Distribution

A Timing distribution is represented as an object with the following fields.

Field nameTypeDescription
sumIntegerThe sum of all recorded values.
valuesMap<String, Integer>The values in each bucket. The key is the minimum value for the range of that bucket.

A contiguous range of buckets is always sent, so that the server can aggregate and visualize distributions, without knowing anything about the specific bucketing function used. This range starts with the first bucket with a non-zero accumulation, and ends at one bucket beyond the last bucket with a non-zero accumulation (so that the upper bound on the last bucket is retained).

For example, the following shows the recorded values vs. what is sent in the payload.

recorded:  1024: 2, 1116: 1,                   1448: 1,
sent:      1024: 2, 1116: 1, 1217: 0, 1327: 0, 1448: 1, 1579: 0

Example:

{
    "sum": 4612,
    "values": {
        "1024": 2,
        "1116": 1,
        "1217": 0,
        "1327": 0,
        "1448": 1,
        "1579": 0
    }
}

Memory Distribution

A Memory distribution is represented as an object with the following fields.

Field nameTypeDescription
sumIntegerThe sum of all recorded values.
valuesMap<String, Integer>The values in each bucket. The key is the minimum value for the range of that bucket.

A contiguous range of buckets is always sent. See timing distribution for more details.

Example:

{
    "sum": 3,
    "values": {
        "0": 1,
        "1": 3,
    }
}

UUID

A UUID is represented by the string representation of the UUID.

Example

"29711dc8-a954-11e9-898a-eb4ea7e8fd3f"

Datetime

A Datetime is represented by its ISO8601 string representation, truncated to the metric's time unit. It always includes the timezone offset.

Example

"2019-07-18T14:06:00.000+02:00"

Event

Events are represented as an array of objects, with one object for each event. Each event object has the following keys:

Field nameTypeDescription
timestampIntegerA monotonically increasing timestamp value, in milliseconds. To avoid leaking absolute times, the first timestamp in the array is always zero, and subsequent timestamps in the array are relative to that reference point.
categoryStringThe event's category. This comes directly from the category under which the metric was defined in the metrics.yaml file.
nameStringThe event's name, as defined in the metrics.yaml file.
extraObject (optional)Extra data associated with the event. Both the keys and values of this object are strings. The keys must be from the set defined for this event in the metrics.yaml file. The values have a maximum length of 50 bytes, when encoded as UTF-8.

Example

[
  {
    "timestamp": 0,
    "category": "app",
    "name": "ss_menu_opened"
  },
  {
    "timestamp": 124,
    "category": "search",
    "name": "performed_search",
    "extra": {
      "source": "default.action"
    }
  }
]

Also see the JSON schema for events.

To avoid losing events when the application is killed by the operating system, events are queued on disk as they are recorded. When the application starts up again, there is no good way to determine if the device has rebooted since the last run and therefore any timestamps recorded in the new run could not be guaranteed to be consistent with those recorded in the previous run. To get around this, on application startup, any queued events are immediately collected into pings and then cleared. These "startup-triggered pings" are likely to have a very short duration, as recorded in ping_info.start_time and ping_info.end_time (see the ping_info section). The maximum timestamp of the events in these pings are quite likely to exceed the duration of the ping, but this is to be expected.

Custom Distribution

A Custom distribution is represented as an object with the following fields.

Field nameTypeDescription
sumIntegerThe sum of all recorded values.
valuesMap<String, Integer>The values in each bucket. The key is the minimum value for the range of that bucket.

A contiguous range of buckets is always sent, so that the server can aggregate and visualize distributions, without knowing anything about the specific bucketing function used. This range starts with the first bucket (as specified in the range_min parameter), and ends at one bucket beyond the last bucket with a non-zero accumulation (so that the upper bound on the last bucket is retained).

For example, suppose you had a custom distribution defined by the following parameters:

The following shows the recorded values vs. what is sent in the payload.

recorded:        12: 2,                      22: 1
sent:     10: 0, 12: 2, 14: 0, 17: 0, 19: 0, 22: 1, 24: 0

Example:

{
    "sum": 3,
    "values": {
        "10": 0,
        "12": 2,
        "14": 0,
        "17": 0,
        "19": 0,
        "22": 1,
        "24": 0
    }
}

Labeled metrics

Currently several labeled metrics are supported:

All are on the top-level represented in the same way, as an object mapping the label to the metric's value. See the individual metric types for details on the value payload:

Example for Labeled Counters

{
    "label1": 2,
    "label2": 17
}

Directory structure

This page describes the contents of the directories where Glean stores its data.

All Glean data is inside a single root directory with the name glean_data.

On Android, this directory lives inside the ApplicationInfo.dataDir directory associated with the application.

On iOS, this directory lives inside the Documents directory associated with the application.

For the Python bindings, if no directory is specified, it is stored in a temporary directory and cleared at exit.

Within the glean_data directory are the following contents:

Debug Pings

For debugging and testing purposes Glean allows to tag pings, which are then available in the Debug Ping Viewer1.

Pings are sent to the same endpoint as all pings, with the addition of one HTTP header:

X-Debug-ID: <tag>

<tag> is a alphanumeric string with a maximum length of 20 characters, used to identify pings in the Debug Ping Viewer.

See Debugging products using the Glean SDK for detailed information how to use this mechanism in applications.


1

Requires a Mozilla login.

Upload mechanism

The glean-core Rust crate does not handle the ping upload directly. Network stacks vastly differ between platforms, applications and operating systems. The Glean SDK leverages the available platform capabilities to implement any network communication.

Glean core controls all upload and coordinates the platform side with its own internals. All language bindings implement ping uploading around a common API and protocol.

The upload module in the language bindings

classDiagram
    class UploadResult {
        ~ToFFI()* int
    }

    class HttpResponse {
        int statusCode
        ~ToFFI() int
    }

    class UnrecoverableFailure {
        ~ToFFI() int
    }

    class RecoverableFailure {
        ~ToFFI() int
    }

    class PingUploader {
        <<interface>>
        +Upload() UploadResult
    }

    class BaseUploader {
        +BaseUploader(PingUploader)
        ~TriggerUploads()
        ~CancelUploads()
    }

    class HttpUploader {
        +Upload() UploadResult
    }

    UploadResult <|-- HttpResponse
    UploadResult <|-- UnrecoverableFailure
    UploadResult <|-- RecoverableFailure
    PingUploader <|-- HttpUploader
    PingUploader o-- BaseUploader

PingUploader

The PingUploader interface describes the contract between the BaseUploader and the SDK or user-provided upload modules.

BaseUploader

The BaseUploader component is responsible for interfacing with the lower level get_upload_task calls and dealing with the logic in a platform-coherent way.

HttpClientUploader

The HttpClientUploader is the default SDK-provided HTTP uploader. It acts as an adapter between the platform-specific upload library and the Glean upload APIs.

Note that most of the languages have now diverged, due to the many iterations, from this design. For example, in Kotlin, the BaseUploader is mostly empty and its functionalities are spread in the PingUploadWorker.

Upload task API

The following diagram visualizes the communication between Glean core (the Rust crate), a Glean language binding (e.g. the Kotlin or Swift implementation) and a Glean end point server.

sequenceDiagram
    participant Glean core
    participant Glean wrapper
    participant Server

    Glean wrapper->>Glean core: get_upload_task()
    Glean core->>Glean wrapper: Task::Upload(PingRequest)
    Glean wrapper-->>Server: POST /submit/{task.id}
    Server-->>Glean wrapper: 200 OK
    Glean wrapper->>Glean core: upload_response(200)
    Glean wrapper->>Glean core: get_upload_task()
    Glean core->>Glean wrapper: Task::Done

Glean core will take care of file management, cleanup, rescheduling and rate limiting1.

1

Rate limiting is achieved by limiting the amount of times a language binding is allowed to get a Task::Upload(PingRequest) from get_upload_task in a given time interval. Currently, the default limit is for a maximum of 15 upload tasks every 60 seconds and there are no exposed methods that allow changing this default (follow Bug 1647630 for updates). If the caller has reached the maximum tasks for the current interval, they will get a Task::Wait regardless if there are other Task::Upload(PingRequest)s queued.

Available APIs

For direct Rust consumers the global Glean object provides these methods:


#![allow(unused_variables)]
fn main() {
/// Gets the next task for an uploader.
fn get_upload_task(&self) -> PingUploadTask

/// Processes the response from an attempt to upload a ping.
fn process_ping_upload_response(&self, uuid: &str, status: UploadResult)
}

See the documentation for further usage and explanation of the additional types:

For FFI consumers (e.g. Kotlin/Swift/Python implementations) these functions are available:


#![allow(unused_variables)]
fn main() {
/// Gets the next task for an uploader. Which can be either:
extern "C" fn glean_get_upload_task(result: *mut FfiPingUploadTask)

/// Processes the response from an attempt to upload a ping.
extern "C" fn glean_process_ping_upload_response(task: *mut FfiPingUploadTask, status: u32)
}

See the documentation for additional information about the types:

Implementations

Project NameLanguage BindingsOperating SystemApp Lifecycle TypeEnvironment Data source
glean-coreRustallallnone
glean-ffiCallallnone
glean-preview1RustWindows/Mac/LinuxDesktop applicationOS info build-time autodetected, app info passed in
Glean AndroidKotlin, JavaAndroidMobile appAutodetected from the Android environment
Glean iOSSwiftiOSMobile appAutodetected from the iOS environment
Glean.pyPythonWindows/Mac/LinuxallAutodetected at runtime
FOG2Rust/C++/JavaScriptas Firefox supportsDesktop applicationOS info build-time autodetected, app info passed in

Features matrix

Feature/BindingsKotlinSwiftPythonC#Rust
Core metric typesX
Metrics Testing APIX
baseline pingXXX
metricsXXX
eventsX
deletion-request ping
Custom pingsX
Custom pings testing APIXX
Debug Ping View support

1

glean-preview is an experimental crate for prototyping integration into Firefox. It it not recommended for general use. See Project FOG.

2

Firefox on Glean (FOG) is the name of the layer that integrates the Glean SDK into Firefox Desktop. It is currently being designed and implemented. It is being used as a test bed for how best to write a generic Rust language binding layer and so temporarily ties directly to glean-core instead of an API crate. Prospective non-mozilla-central Rust consumers of the Glean SDK should not follow its example and should instead follow bug 1631768 for updates on when the proper language binding crate will be available.

The following language-specific API docs are available:

Appendix

The following sections contain other material related to Glean.

Glossary

A glossary with explanations and background for wording used in the Glean project.

Glean

According to the dictionary the word “glean” means:

to gather information or material bit by bit

Glean is the combination of the Glean SDK, the Glean pipeline & Glean tools.

See also: Glean - product analytics & telemetry.

Glean Pipeline

The general data pipeline is the infrastructure that collects, stores, and analyzes telemetry data from our products and logs from various services. See An overview of Mozilla’s Data Pipeline.

The Glean pipeline additionally consists of

Glean SDK

The Glean SDK is the bundle of libraries with support for different platforms. The source code is available at https://github.com/mozilla/glean.

Glean SDK book

This documentation.

Glean tools

Glean provides additional tools for its usage:

Metric

Metrics are the individual things being measured using Glean. They are defined in metrics.yaml files, also known as registry files.

Glean itself provides some metrics out of the box.

Ping

A ping is an entity used to bundle related metrics. The Glean SDK provides default pings and allows for custom ping, see Glean Pings.

Submission

"To submit" means to collect & to enqueue a ping for uploading.

The Glean SDK stores locally all the metrics set by it or by its clients. Each ping has its own schedule to gather all its locally saved metrics and create a JSON payload with them. This is called "collection".

Upon successful collection, the payload is queued for upload, which may not happen immediately or at all (in case network connectivity is not available).

Unless the user has defined their own custom pings, they don’t need to worry too much about submitting pings.

All the default pings have their scheduling and submission handled by the SDK.

Measurement window

The measurement window of a ping is the time frame in which metrics are being actively gathered for it.

The measurement window start time is the moment the previous ping is submitted. In the absence of a previous ping, this time will be the time the application process started.

The measurement window end time is the moment the current ping gets submitted. Any new metric recorded after submission will be part of the next ping, so this pings measurement window is over.

This Week in Glean (TWiG)

This Week in Glean is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work.

Unreleased changes

Full changelog

v32.3.2 (2020-09-11)

Full changelog

v32.3.1 (2020-09-09)

Full changelog

v32.3.0 (2020-08-27)

Full changelog

v32.2.0 (2020-08-25)

Full changelog

v32.1.1 (2020-08-24)

Full changelog

v32.1.0 (2020-08-17)

Full changelog

v32.0.0 (2020-08-03)

Full changelog

v31.6.0 (2020-07-24)

Full changelog

v31.5.0 (2020-07-22)

Full changelog

v31.4.1 (2020-07-20)

Full changelog

v31.4.0 (2020-07-16)

Full changelog

v31.3.0 (2020-07-10)

Full changelog

v31.2.3 (2020-06-29)

Full changelog

v31.2.2 (2020-06-26)

Full changelog

v31.2.1 (2020-06-25)

Full changelog

v31.2.0 (2020-06-24)

Full changelog

v31.1.2 (2020-06-23)

Full changelog

v31.1.1 (2020-06-12)

Full changelog

v31.1.0 (2020-06-11)

Full changelog

v31.0.2 (2020-05-29)

Full changelog

v31.0.1 (2020-05-29)

Full changelog

v31.0.0 (2020-05-28)

Full changelog

v30.1.0 (2020-05-22)

Full changelog

v30.0.0 (2020-05-13)

Full changelog

v29.1.1 (2020-05-22)

Full changelog

v29.1.0 (2020-05-11)

Full changelog

v29.0.0 (2020-05-05)

Full changelog

v28.0.0 (2020-04-23)

Full changelog

v27.1.0 (2020-04-09)

Full changelog

v27.0.0 (2020-04-08)

Full changelog

v26.0.0 (2020-03-27)

Full changelog

v25.1.0 (2020-02-26)

Full changelog

v25.0.0 (2020-02-17)

Full changelog

v24.2.0 (2020-02-11)

Full changelog

v24.1.0 (2020-01-16)

Full changelog

v24.0.0 (2020-01-14)

Full changelog

v23.0.1 (2020-01-08)

Full changelog

v23.0.0 (2020-01-07)

Full changelog

v22.1.0 (2019-12-17)

Full changelog

v22.0.0 (2019-12-05)

Full changelog

v21.3.0 (2019-12-03)

Full changelog

v21.2.0 (2019-11-21)

Full changelog

v21.1.1 (2019-11-20)

Full changelog

v21.1.0 (2019-11-20)

Full changelog

v21.0.0 (2019-11-18)

Full changelog

v20.2.0 (2019-11-11)

Full changelog

v20.1.0 (2019-11-11)

Full changelog

v20.0.0 (2019-11-11)

Full changelog

v19.1.0 (2019-10-29)

Full changelog

v19.0.0 (2019-10-22)

Full changelog

First stable release of Glean in Rust (aka glean-core). This is a major milestone in using a cross-platform implementation of Glean on the Android platform.

v0.0.1-TESTING6 (2019-10-18)

Full changelog

v0.0.1-TESTING5 (2019-10-10)

Full changelog

v0.0.1-TESTING4 (2019-10-09)

Full changelog

v0.0.1-TESTING3 (2019-10-08)

Full changelog

v0.0.1-TESTING2 (2019-10-07)

Full changelog

v0.0.1-TESTING1 (2019-10-02)

Full changelog

General

First testing release.

This Week in Glean (TWiG)

“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean.

Blog posts