This document gives a high-level overview of how we test components in
It will be useful to you if you're adding a new component, or working on increasing the test
coverage of an existing component.
Since the core implementations of our components live in rust, so does the core of our testing strategy.
Where possible, it's better use use the Rust typesystem to make bugs impossible than to write tests to assert that they don't occur in practice. But given that the ultimate consumers of our code are not in Rust, that's sometimes not possible. The best idiomatic Rust API for a feature is not necessarily the best API for consuming it over an FFI boundary.
Rust's builtin assertion macros are sparse; we use the more_asserts for some additional helpers.
Rust's strict typing can make test mocks difficult. If there's something you need to mock out in tests, make it a Trait and use the mockiato crate to mock it.
The Rust tests for a component should be runnable via
We are currently using
uniffi to generate most ((and soon all!) of our FFI code and thus the FFI code itself does not need to be extensively tested.
The Kotlin wrapper code for a component should have its own test suite, which should follow the general guidelines for testing Android code in Mozilla projects. In practice that means we use JUnit as the test framework and Robolectric to provide implementations of Android-specific APIs.
The Kotlin tests for a component should be runnable via
The tests at this layer are designed to ensure that the API binding code is working as intended, and should not repeat tests for functionality that is already well tested at the Rust level. But given that the Kotlin bindings involve a non-trivial amount of hand-written boilerplate code, it's important to exercise that code throughly.
One complication with running Kotlin tests is that the code needs to run on your local development machine,
but the Kotlin code's native dependencies are typically compiled and packaged for Android devices. The
tests need to ensure that an appropriate version of JNA and of the compiled Rust code is available in
their library search path at runtime. Our
build.gradle files contain a collection of hackery that ensures
this, which should be copied into any new components.
The majority of our Kotlin bindings are autogenerated using
uniffi and do not need extensive testing.
The Swift wrapper code for a component should have its own test suite, using Apple's Xcode unittest framework.
Due to the way that all rust components need to be compiled together into a single "megazord"
framework, this entire repository is a single Xcode project. The Swift tests for each component
thus need to live under
megazords/ios-rust/MozillaTestServicesTests/ rather than in the directory
for the corresponding component. (XXX TODO: is this true? it would be nice to find a way to avoid having
them live separately because it makes them easy to overlook).
The tests at this layer are designed to ensure that the API binding code is working as intended, and should not repeat tests for functionality that is already well tested at the Rust level. But given that the Swift bindings involve a non-trivial amount of hand-written boilerplate code, it's important to exercise that code thoroughly.
The majority of our Swift bindings are autogenerated using
uniffi and do not need extensive testing.
testing/sync-test directory contains a test harness for running sync-related
Rust components against a live Firefox Sync infrastructure, so that we can verifying the functionality
Each component that implements a sync engine should have a corresponding suite of tests in this directory.
- XXX TODO: places doesn't.
- XXX TODO: send-tab doesn't (not technically a sync engine, but still, it's related)
- XXX TODO: sync-manager doesn't
It's important that changes in
application-services are tested against upstream consumer code in the
android-components repo. This is currently
a manual process involving:
- Configuring your local checkout of android-components to use your local application-services build.
- Running the android-components test suite via
- Manually building and running the android-components sample apps to verify that they're still working.
Ideally some or all of this would be automated and run in CI, but we have not yet invested in such automation.
We currently have code coverage reporting on Github using codecov. However, our code coverage does not tell us how much more coverage is caused by our consumers' tests.
- ASan, Memsan, and maybe other sanitizer checks, especially around the points where we cross FFI boundaries.
- General-purpose fuzzing, such as via https://github.com/jakubadamw/arbitrary-model-tests
- We could consider making a mocking backend for viaduct, which would also be mockable from Kotlin/Swift.
- Add more end-to-end integration tests!
- Live device tests, e.g. actual Fenixes running in an emulator and syncing to each other.
- Run consumer integration tests in CI against main.