Guide to Testing a Rust Component

This document gives a high-level overview of how we test components in application-services. It will be useful to you if you're adding a new component, or working on increasing the test coverage of an existing component.

If you are only interested in running the existing test suite, please consult the contributor docs and the tests.py script.

Unit and Functional Tests

Rust code

Since the core implementations of our components live in rust, so does the core of our testing strategy.

Each rust component should be accompanied by a suite of unit tests, following the guidelines for writing tests from the Rust Book. Some additional tips:

  • Where possible, it's better use use the Rust typesystem to make bugs impossible than to write tests to assert that they don't occur in practice. But given that the ultimate consumers of our code are not in Rust, that's sometimes not possible. The best idiomatic Rust API for a feature is not necessarily the best API for consuming it over an FFI boundary.

  • Rust's builtin assertion macros are sparse; we use the more_asserts for some additional helpers.

  • Rust's strict typing can make test mocks difficult. If there's something you need to mock out in tests, make it a Trait and use the mockiato crate to mock it.

The Rust tests for a component should be runnable via cargo test.

FFI Layer code

We are currently using uniffi to generate most ((and soon all!) of our FFI code and thus the FFI code itself does not need to be extensively tested.

Kotlin code

The Kotlin wrapper code for a component should have its own test suite, which should follow the general guidelines for testing Android code in Mozilla projects. In practice that means we use JUnit as the test framework and Robolectric to provide implementations of Android-specific APIs.

The Kotlin tests for a component should be runnable via ./gradlew <component>:test.

The tests at this layer are designed to ensure that the API binding code is working as intended, and should not repeat tests for functionality that is already well tested at the Rust level. But given that the Kotlin bindings involve a non-trivial amount of hand-written boilerplate code, it's important to exercise that code throughly.

One complication with running Kotlin tests is that the code needs to run on your local development machine, but the Kotlin code's native dependencies are typically compiled and packaged for Android devices. The tests need to ensure that an appropriate version of JNA and of the compiled Rust code is available in their library search path at runtime. Our build.gradle files contain a collection of hackery that ensures this, which should be copied into any new components.

The majority of our Kotlin bindings are autogenerated using uniffi and do not need extensive testing.

Swift code

The Swift wrapper code for a component should have its own test suite, using Apple's Xcode unittest framework.

Due to the way that all rust components need to be compiled together into a single "megazord" framework, this entire repository is a single Xcode project. The Swift tests for each component thus need to live under megazords/ios-rust/MozillaTestServicesTests/ rather than in the directory for the corresponding component. (XXX TODO: is this true? it would be nice to find a way to avoid having them live separately because it makes them easy to overlook).

The tests at this layer are designed to ensure that the API binding code is working as intended, and should not repeat tests for functionality that is already well tested at the Rust level. But given that the Swift bindings involve a non-trivial amount of hand-written boilerplate code, it's important to exercise that code thoroughly.

The majority of our Swift bindings are autogenerated using uniffi and do not need extensive testing.

Integration tests

End-to-end Sync Tests

⚠️ Those tests were disabled because of how flakey the stage server was. See #3909 ⚠️

The testing/sync-test directory contains a test harness for running sync-related Rust components against a live Firefox Sync infrastructure, so that we can verifying the functionality end-to-end.

Each component that implements a sync engine should have a corresponding suite of tests in this directory.

  • XXX TODO: places doesn't.
  • XXX TODO: send-tab doesn't (not technically a sync engine, but still, it's related)
  • XXX TODO: sync-manager doesn't

Android Components Test Suite

It's important that changes in application-services are tested against upstream consumer code in the android-components repo. This is currently a manual process involving:

  • Configuring your local checkout of android-components to use your local application-services build.
  • Running the android-components test suite via ./gradle test.
  • Manually building and running the android-components sample apps to verify that they're still working.

Ideally some or all of this would be automated and run in CI, but we have not yet invested in such automation.

Test Coverage

We currently have code coverage reporting on Github using codecov. However, our code coverage does not tell us how much more coverage is caused by our consumers' tests.

Ideas for Improvement

  • ASan, Memsan, and maybe other sanitizer checks, especially around the points where we cross FFI boundaries.
  • General-purpose fuzzing, such as via https://github.com/jakubadamw/arbitrary-model-tests
  • We could consider making a mocking backend for viaduct, which would also be mockable from Kotlin/Swift.
  • Add more end-to-end integration tests!
  • Live device tests, e.g. actual Fenixes running in an emulator and syncing to each other.
  • Run consumer integration tests in CI against main.