Unit testing Glean custom pings for Android

Applications defining custom pings can use use the strategy defined in this document to test these pings in unit tests.

General testing strategy

The schedule of custom pings depends on the specific application implementation, since it is up to the SDK user to define the ping semantics. This makes writing unit tests for custom pings a bit more involved.

One possible strategy could be to wrap the Glean SDK API call to send the ping in a function that can be mocked in the unit test. This would allow for checking the status and the values of the metrics contained in the ping at the time in which the application would have sent it.

Example testing of a custom ping

Let us start by defining a custom ping with a sample metric in it. Here is the pings.yaml file:

$schema: moz://mozilla.org/schemas/glean/pings/1-0-0

my-custom-ping:
  description: >
    This ping is intended to showcase the recommended testing strategy for
    custom pings.
  include_client_id: false
  bugs:
    - https://bugzilla.mozilla.org/1556985/
  data_reviews:
    - https://bugzilla.mozilla.org/show_bug.cgi?id=1556985
  notification_emails:
    - custom-ping-owner@example.com

And here is the metrics.yaml

$schema: moz://mozilla.org/schemas/glean/metrics/1-0-0

custom_ping_data:
  sample_string:
    type: string
    lifetime: ping
    description: >
      A sample string metric for demonstrating unit tests for custom pings.
    send_in_pings:
      - my-custom-ping
    bugs:
      - https://bugzilla.mozilla.org/1556985/
    data_reviews:
      - https://bugzilla.mozilla.org/show_bug.cgi?id=1556985
    notification_emails:
      - custom-ping-owner@example.com
    expires: "2019-10-01"

A potential usage of the Glean SDK generated API could be the following:

import my.component.GleanMetrics.Pings
import my.component.GleanMetrics.CustomPingData

class MyCustomPingScheduler {
  /**
   * HERE ONLY TO KEEP THE EXAMPLE SIMPLE.
   *
   * A function that consumes the Glean SDK generated metrics API to
   * record some data. It doesn't really need to be in a function, nor
   * in this class. The Glean SDK API can be called when the data is
   * generated.
   */
  fun addSomeData() {
    // Record some sample data.
    CustomPingData.sampleString.set("test-data")
  }

  /**
   * Called to implement the ping scheduling logic for 'my_custom_ping'.
   */
  fun schedulePing() {
    // ... some scheduling logic that will end up calling the function below.
    submitPing()
  }

  /**
   * Internal function to only be overridden in tests. This
   * calls the Glean SDK API to send custom pings.
   */
  @VisibleForTesting(otherwise = VisibleForTesting.NONE)
  internal fun submitPing() {
    Pings.MyCustomPing.submit()
  }
}

The following unit test intercepts the MyCustomPingScheduler.submitPing() call in order to perform the validation on the data. This specific example uses Mockito, but any other framework would work.

// Metrics and pings definitions.
import my.component.GleanMetrics.Pings
import my.component.GleanMetrics.CustomPingData

// Mockito imports for using spies.
import org.mockito.Mockito.spy
import org.mockito.Mockito.`when`

@RunWith(AndroidJUnit4::class)
class MyCustomPingSchedulerTest {
    // Apply the GleanTestRule to set up a disposable Glean instance.
    // Please note that this clears the Glean data across tests.
    @get:Rule
    val gleanRule = GleanTestRule(ApplicationProvider.getApplicationContext())

    @Test
    fun `verify custom ping metrics`() {
      val scheduler = spy(MyCustomPingScheduler())
      doAnswer {
        // Here we validate the content that goes into the ping.
        assertTrue(CustomPingData.sampleString.testHasValue())
        assertEquals("test-data", CustomPingData.sampleString.testGetValue())

        // We want to intercept this call, but we also want to make sure the
        // real Glean API is called in order to clear the ping store and to provide
        // consistent behaviour with respect to the application.
        it.callRealMethod()
      }.`when`(scheduler).submitPing()

      scheduler.addSomeData()
      scheduler.schedulePing()
    }
}
import Foundation
import Glean

// Use typealiases to simplify usage.
// This can be placed anywhere in your code to be available in all files.
typealias CustomPingData = GleanMetrics.CustomPingData
typealias Pings = GleanMetrics.Pings

class MyCustomPingScheduler {
    /**
     * HERE ONLY TO KEEP THE EXAMPLE SIMPLE.
     *
     * A function that consumes the Glean SDK generated metrics API to
     * record some data. It doesn't really need to be in a function, nor
     * in this class. The Glean SDK API can be called when the data is
     * generated.
     */
    func addSomeData() {
       // Record some sample data.
       CustomPingData.sampleString.set("test-data")
    }

    /**
     * Called to implement the ping scheduling logic for 'my_custom_ping'.
     */
    func schedulePing() {
        // ... some scheduling logic that will end up calling the function below.
        submitPing()
    }

    /**
     * Internal function to only be overridden in tests. This
     * calls the Glean SDK API to send custom pings.
     */
    internal func submitPing() {
        Pings.shared.myCustomPing.submit()
    }
}

The following unit test intercepts the MyCustomPingScheduler.submitPing() call in order to perform the validation on the data. This example uses a manual mock implementation, but you could use a framework for that.

@testable import YourApplication
import Glean
import XCTest

class MyCustomPingSchedulerMock: MyCustomPingScheduler {
    var submitWasCalled = false

    deinit {
        XCTAssertTrue(submitWasCalled, "submitPing should have been called once")
    }

    override func submitPing() {
        submitWasCalled = true

        XCTAssertTrue(CustomPingData.os.testHasValue())
        XCTAssertEqual("test-data", try! CustomPingData.os.testGetValue())

        super.submitPing()
    }
}

class MyCustomPingSchedulerTests: XCTestCase {
    override func setUp() {
        Glean.shared.resetGlean(clearStores: true)
    }

    func testCustomPingMetrics() {
        let scheduler = MyCustomPingSchedulerMock()
        scheduler.addSomeData()
        scheduler.schedulePing()
    }
}
import glean

metrics = glean.load_metrics("metrics.yaml")
pings = glean.load_pings("pings.yaml")


class MyCustomPingScheduler:
    def add_some_data(self):
        """
        HERE ONLY TO KEEP THE EXAMPLE SIMPLE.

        A function that consumes the Glean SDK generated metrics API to
        record some data. It doesn't really need to be in a function, nor
        in this class. The Glean SDK API can be called when the data is
        generated.
        """
        # Record some sample data.
        metrics.custom_ping_data.sample_string.set("test-data")

    def schedule_ping(self):
        """
        Called to implement the ping scheduling logic for 'my_custom_ping'.
        """
        # ... some scheduling logic that will end up calling the function below.
        self._submit_ping()

    def _submit_ping(self):
        """
        Internal function to only be overridden in tests.
        """
        pings.my_custom_ping.submit()

The following unit test intercepts the MyCustomPingScheduler._submit_ping() call in order to perform the validation on the data.

from unittest.mock import MagicMock

from glean import testing

import custom_ping_scheduler


# This will be run before every test in the entire test suite
def pytest_runtest_setup(item):
    testing.reset_glean(application_id="my-app", application_version="0.0.1")


def test_verify_custom_ping_metrics():
    scheduler = custom_ping_scheduler.MyCustomPingScheduler()

    original_submit_ping = scheduler._submit_ping

    def _submit_ping(self):
        # Here we validate the content that goes into the ping.
        assert (
            custom_ping_scheduler.metrics.custom_ping_data.sample_string.test_has_value()
        )
        assert (
            "test-data"
            == custom_ping_scheduler.metrics.custom_ping_data.sample_string.test_get_value()
        )

        # We want to intercept this call, but we also want to make sure the
        # real Glean API is called in order to clear the ping strong and to
        # provide consistent behavior with respect to the application.
        original_submit_ping(self)

    scheduler._submit_ping = MagicMock(_submit_ping)

    scheduler.add_some_data()
    scheduler.schedule_ping()