Converting an existing Component to use UniFFI
When we started building the components in this repo, exposing Rust code to Kotlin and Swift was a manual process and each component had its own hand-written FFI layer and foreign-language bindings.
As we've gained more experience with building components in this way, we've started to automate bindings generation and capture best practices in a tool called UniFFI, which is the currently recommended approach when adding a new component from scratch.
We expect that existing components will gradually be ported over to use UniFFI, and this document is a guide to doing that port.
First, get familiar with UniFFI
First, make sure you've perused the UniFFI guide to understand the overall architecture of a UniFFI component, and take a look at the guide to adding a new component to understand how such components fit in to this repo. The aim of porting will be to have a component that looks like it was added by the process described therein.
Next, get familiar with the target component
Pre-UniFFI components typically consist of four main parts:
- A Rust crate implementing the core functionality of the component
- A separate Rust crate that exposes the core functionality over a C-style FFI.
- An Android package that imports the C-style FFI into idiomatic Kotlin.
- A Swift module that imports the C-style FFI into idiomatic Swift.
The code for these parts will be laid out something like this:
components/<component_name>/
Cargo.toml
src/
- Rust code for the core functionality of the component goes here.
ffi/
Cargo.toml
src/
- Rust code specifically for exposing the C-style FFI goes here.
android/
build.gradle
src/
main/
AndroidManifest.xml
java/mozilla/appservices/<component_name>/
Lib<ComponentName>FFI.kt
(low-level bindings to the C-style FFI)- Higher-level hand-written Kotlin that wraps the FFI.
ios/
<component_name>/
Rust<ComponentName>API.h
(low-level bindings to the C-style FFI)- Higher-level hand-written Swift that wraps the FFI.
The goal here is to replace much of the hand-written wrapper layers with autogenerated code:
- The
./ffi/
crate will disappear entirely, its work is automated by UniFFI- If you still need some hand-written
pub extern "C"
functions, perhaps to implement features not currently supported by UniFFI, then they should move intolib.rs
of the main component crate.
- If you still need some hand-written
- The low-level
Lib<ComponentName>FFI.kt
file will disappear entirely, as will some of the code that converts it back into nice high-level Kotlin classes and interfaces.- Some of the hand-written Kotlin code may remain, if it provides functionality that cannot be implemented in Rust.
- The low-level
Rust<ComponentName>API.h
file will disappear entirely, as will some of the code that converts it back into nice high-level Swift classes and interfaces.- Some of the hand-written Swift code may remain, if it provides functionality that cannot be implemented in Rust.
You'll aim to end up with a simplified file structure that looks like this:
components/<component_name>/
Cargo.toml
uniffi.toml
src/
<component_name>.udl
(abstract interface definition)- Rust code here.
android/
build.gradle
src/
main/
AndroidManifest.xml
java/mozilla/appservices/<component_name>/
- Optional hand-written Kotlin code here.
ios/
<component_name>/
- Optional hand-written Swift code here.
Write a first draft of the .udl
file for the component's interface
Make sure you've got the uniffi-bindgen
command available; cargo install uniffi_bindgen
will
ensure you have the latest version.
Create ./src/<component_name>.udl
and try to describe the intended interface for the component
using UniFFI's interface definition language.
You'll probably need to reverse-engineer it a little bit from the existing hand-written Kotlin and/or
Swift code.
Don't spend too much time on trying to match every minute detail of the existing hand-written API. There are likely to be small differences between how UniFFI likes to do things and how the hand-written APIs were structured, and it's in everyone's best long-term interests to just push ahead and update consumers to accommodate any breaking API changes, rathern than e.g. trying to convince UniFFI to capitalize enum variant names in the same style that the hand-written code was using.
To check whether the .udl
file is syntactically valid, you can use uniffi-bindgen
to generate
the Rust FFI scaffolding like so:
uniffi-bindgen scaffolding ./src/<component_name>.udl
If this succeeds, it will generate a file ./src/<component_name>.uniffi.rs
with a bunch of
thorny auto-generated Rust code. If it fails, it will likely fail with an inscrutable error message.
Unfortunately the error reporting in UniFFI is currently a known pain point, and it can take a
bit of trial-and-error to identify what part of the file is causing the issue. Sorry :-(
The aim at this point is to ensure that the intended interface of the component can be expressed in terms that UniFFI understands. Most cases should be supported, but you may find some aspect of the existing component that is hard to express in UniFFI, perhaps even uncovering new functionality that needs to be added to UniFFI itself!
The .udl
file is definitely a first draft at this point. It is normal and expected to need
to iterate on this file as you port over the underlying Rust code.
Restructure the Rust code to introduce UniFFI
You will now restructure the existing Rust crate so that its public API surface
and overall "shape" match what you defined in the .udl
file.
Start by deleting the ./ffi
sub-crate, because you're going to use UniFFI to generate
all of that code. You'll also need to remove it from the workspace in the top-level
Cargo.toml
file, as well as change the crates under /megazords
to import the core
Rust crate for the component rather than importing the FFI sub-crate.
Add UniFFI to the crate's dependencies and configure its build.rs
script to invoke the
UniFFI scaffolding generator, as described in "adding a new component".
Now, edit ./lib.rs
so that it matches the interface defined in the .udl
file as closely
as possible. If the .udl
has an interface Example
then lib.rs
should contain a
pub struct Example
, if the .udl
contains an enum ExampleItem
then lib.rs
should
contain a pub enum ExampleItem
, and so-on.
The details of this step will depend heavily on the specific crate, but some tips include:
-
You may find it useful to move all of the existing code into a sub-module named
internal
, and then make a brand newlib.rs
that imports or re-defines just the pieces it needs in order to implement the interface from the.udl
file. Thefxa-client
crate is an example of a case where this worked out well, though of course your mileage may vary. -
If the existing crate contains a file named like
<component_name>_msg_types.proto
, then it was using Protocol Buffers to serialize data to pass over the FFI. The message types defined in the.proto
file will need to be converted intodictionary
orenum
definitions in your.udl
file. See the section below for more details.
As noted above, don't be afraid to accept some API churn during the conversion process. We're willing to accept some breaking API changes as the cost of getting bindings generated for free, as long as the core functionality and mental model of the component remain intact.
At this point, in theory the crate should be buildable with UniFFI, although it's likely
to require some iteration to get it all working! Run cargo check
to check for any
compilation errors without having to do a full build.
Removing Protobuf Messages
Passing rich structured data over the FFI is the most complex part of our hand-written bindings, and was previously done by serializing data via Protocol Buffers. This is something that UniFFI tries to make as simple as possible.
Start by locating the <component_name>_msg_types.proto
file for the component. This file defines
the structured messages that can be passed over the FFI, and you should see that they correspond
to various types of structured data that the component wants to receive from, or return to,
the foreign-language code.
Find the places in your .udl
interface that correspond to these message types and make sure
that you've got a similarly-shaped dictionary
or enum
for each one. You should find that
representing this structured data in UDL is simpler than protobuf in many cases - for example
many of our .protobuf
files need to use a separate ExampleStructs
message in order to
pass a list of ExampleStruct
messages over the FFI, but in UniFFI this is represented
directly as sequence<ExampleStruct>
.
Find the places in the Rust code that are using these message types to return structured data.
In simple cases, you may be able to directly replace uses of msg_types::ExampleStruct
with
the corresponding crate::ExampleStruct
from your public API.
For more complex cases, you may find it helpful to define an Into
mapping between the
UniFFI dictionary/enum in the crate's public interface, and a more complex struct designed
for internal use.
As noted above, don't be afraid to accept some API churn during this conversion process.
Once you have replaced all uses of the msg_types
structs in the Rust code:
- Delete
./src/<component_name>_msg_types.proto
. - Delete
./src/mozilla.appservices.<component_name>.protobuf.rs
, which is generated from the.proto
file. - Remote
prost
andprost-derive
from the crate's dependencies. - Delete the crate from the list in
/tools/protobuf_files.toml
.
If you happen to find that you've deleted the last crate from the list in protobuf_files.toml
,
congratulations! You've successfully removed protocol buffers from this repo entirely, and should
file a bug to track the complete removal of protobuf from our tooling and dependency chain.
Document the Public API in the Rust code
Write consumer-facing documentation on the public API in lib.rs
using Rust's standard
rustdoc conventions
and tools. The fxa-client
crate may serve as a good example.
You can view the generated documentation by running:
cargo doc --no-deps --open
In future, we intend to automatically extract documentation from the Rust code and make it easily available to consumers of the generated bindings.
(In fact there is some work-in-progress code in uniffi-rs#416
that can read docs from the Rust code and write them back into the .udl
file, which you're
welcome to try out if you're feeling adventurous. But it's just a very hacky prototype.)
Set up the Kotlin wrapper
It's easiest to start by removing all of the hand-written Kotlin code under android/src/main/java
and then restoring parts of it later if necessary. Leave the AndroidManifest.xml
file and any tests
in place.
Delete the android/build.gradle
file and then follow the instructions for adding Kotlin bindings
for a new component to create a new build.gradle
file and a corresponding uniffi.toml
.
This should be all that's required to set up UniFFI to build the Kotlin bindings. Try building the Android package to confirm:
./gradlew <component_name>:assembleDebug
The UniFFI-generated Kotlin code will be under ./android/build/generated/source/uniffi/
and
may be useful for debugging.
If there are existing Kotlin tests for the component, the next step is to get those passing:
./gradlew <component_name>:test
As noted above, it is normal and expected for the autogenerated bindings to be subtly different from the previous hand-written ones. For example, UniFFI insists on using SHOUTY_SNAKE_CASE variant names in Kotlin enums while the hand-written code may have used CamelCase. Some components also have small naming differences between the Rust code and the hand-written Kotlin bindings, which UniFFI will not allow.
If the component had functionality in its Kotlin layer that was not part of the Rust API,
then you'll need to add some hand-written Kotlin code under android/src/main/java
to
implement it. The fxa-client
component may be a good example here: its Rust layer exposes
a FirefoxAccount
struct that the Kotlin code wraps into a PersistedFirefoxAccount
class,
adding the ability to set a persistence callback.
Finally, you will need to try out the new bindings with a consuming app. For Kotlin code you should make a local build of android-components and Fenix, updating them to accomodate any changes in the component's public API.
Set up the Swift wrapper
It's easiest to start by removing all of the hand-written Swift code under ./ios
and then
restoring parts of it later if necessary.
Edit /megazords/ios-rust/MozillaTestServices.h
to remove any references to Rust<ComponentName>API.h
,
replacing them with the UniFFI-generated header file name <component_name>FFI.h
.
Open /megazords/ios-rust/MozillaTestServices.xcodeproj
in Xcode and follow the instructions for
adding Swift bindings for a new component to
configure Xcode to build your UniFFI-generated bindings.
While you are in the Xcode Project Navigator, you should also delete any references to
Rust<ComponentName>API.h
or to the old hand-written Swift wrappers. (They should be highlighted
in red in the Project Navigator, because the files will be missing from disk after you
deleted them above).
This should be all that's required to set up UniFFI to build the Swift bindings. Try building the project in Xcode to confirm.
The UniFFI-generated Swift code will be under ios/Generated
and may be useful for debugging.
If there are existing Swift tests for the component, the next step is to get those passing:
./automation/run_ios_tests.sh
- (or run them from the Xcode GUI)
As noted above, it is normal and expected for the autogenerated bindings to be subtly different from the previous hand-written ones. Many existing components have small naming differences between the Rust code and the hand-written Swift bindings, which UniFFI will not allow.
If the component had functionality in its Swift layer that was not part of the Rust API,
then you'll need to add some hand-written Swift code under ./ios/<ComponentName>
to
implement it. The fxa-client
component may be a good example here: its Rust layer exposes
a FirefoxAccount
struct that the Swift code wraps into a PersistedFirefoxAccount
class,
adding the ability to set a persistence callback.
You will need to add any such file to the "Compile Sources" list in Xcode, in the same way
that you added the .udl
file.
Finally, you will need to try out the new bindings with a consuming app. For Swift code you should make a local build of Firefox iOS, you can do that by following the steps in this document