Converting an existing Component to use UniFFI

When we started building the components in this repo, exposing Rust code to Kotlin and Swift was a manual process and each component had its own hand-written FFI layer and foreign-language bindings.

As we've gained more experience with building components in this way, we've started to automate bindings generation and capture best practices in a tool called UniFFI, which is the currently recommended approach when adding a new component from scratch.

We expect that existing components will gradually be ported over to use UniFFI, and this document is a guide to doing that port.

First, get familiar with UniFFI

First, make sure you've perused the UniFFI guide to understand the overall architecture of a UniFFI component, and take a look at the guide to adding a new component to understand how such components fit in to this repo. The aim of porting will be to have a component that looks like it was added by the process described therein.

Next, get familiar with the target component

Pre-UniFFI components typically consist of four main parts:

  • A Rust crate implementing the core functionality of the component
  • A separate Rust crate that exposes the core functionality over a C-style FFI.
  • An Android package that imports the C-style FFI into idiomatic Kotlin.
  • A Swift module that imports the C-style FFI into idiomatic Swift.

The code for these parts will be laid out something like this:

  • components/<component_name>/
    • Cargo.toml
    • src/
      • Rust code for the core functionality of the component goes here.
    • ffi/
      • Cargo.toml
      • src/
        • Rust code specifically for exposing the C-style FFI goes here.
    • android/
      • build.gradle
      • src/
        • main/
          • AndroidManifest.xml
          • java/mozilla/appservices/<component_name>/
            • Lib<ComponentName>FFI.kt (low-level bindings to the C-style FFI)
            • Higher-level hand-written Kotlin that wraps the FFI.
    • ios/
      • <component_name>/
        • Rust<ComponentName>API.h (low-level bindings to the C-style FFI)
        • Higher-level hand-written Swift that wraps the FFI.

The goal here is to replace much of the hand-written wrapper layers with autogenerated code:

  • The ./ffi/ crate will disappear entirely, its work is automated by UniFFI
    • If you still need some hand-written pub extern "C" functions, perhaps to implement features not currently supported by UniFFI, then they should move into lib.rs of the main component crate.
  • The low-level Lib<ComponentName>FFI.kt file will disappear entirely, as will some of the code that converts it back into nice high-level Kotlin classes and interfaces.
    • Some of the hand-written Kotlin code may remain, if it provides functionality that cannot be implemented in Rust.
  • The low-level Rust<ComponentName>API.h file will disappear entirely, as will some of the code that converts it back into nice high-level Swift classes and interfaces.
    • Some of the hand-written Swift code may remain, if it provides functionality that cannot be implemented in Rust.

You'll aim to end up with a simplified file structure that looks like this:

  • components/<component_name>/
    • Cargo.toml
    • uniffi.toml
    • src/
      • <component_name>.udl (abstract interface definition)
      • Rust code here.
    • android/
      • build.gradle
      • src/
        • main/
          • AndroidManifest.xml
          • java/mozilla/appservices/<component_name>/
            • Optional hand-written Kotlin code here.
    • ios/
      • <component_name>/
        • Optional hand-written Swift code here.

Write a first draft of the .udl file for the component's interface

Make sure you've got the uniffi-bindgen command available; cargo install uniffi_bindgen will ensure you have the latest version.

Create ./src/<component_name>.udl and try to describe the intended interface for the component using UniFFI's interface definition language. You'll probably need to reverse-engineer it a little bit from the existing hand-written Kotlin and/or Swift code.

Don't spend too much time on trying to match every minute detail of the existing hand-written API. There are likely to be small differences between how UniFFI likes to do things and how the hand-written APIs were structured, and it's in everyone's best long-term interests to just push ahead and update consumers to accommodate any breaking API changes, rathern than e.g. trying to convince UniFFI to capitalize enum variant names in the same style that the hand-written code was using.

To check whether the .udl file is syntactically valid, you can use uniffi-bindgen to generate the Rust FFI scaffolding like so:

uniffi-bindgen scaffolding ./src/<component_name>.udl

If this succeeds, it will generate a file ./src/<component_name>.uniffi.rs with a bunch of thorny auto-generated Rust code. If it fails, it will likely fail with an inscrutable error message. Unfortunately the error reporting in UniFFI is currently a known pain point, and it can take a bit of trial-and-error to identify what part of the file is causing the issue. Sorry :-(

The aim at this point is to ensure that the intended interface of the component can be expressed in terms that UniFFI understands. Most cases should be supported, but you may find some aspect of the existing component that is hard to express in UniFFI, perhaps even uncovering new functionality that needs to be added to UniFFI itself!

The .udl file is definitely a first draft at this point. It is normal and expected to need to iterate on this file as you port over the underlying Rust code.

Restructure the Rust code to introduce UniFFI

You will now restructure the existing Rust crate so that its public API surface and overall "shape" match what you defined in the .udl file.

Start by deleting the ./ffi sub-crate, because you're going to use UniFFI to generate all of that code. You'll also need to remove it from the workspace in the top-level Cargo.toml file, as well as change the crates under /megazords to import the core Rust crate for the component rather than importing the FFI sub-crate.

Add UniFFI to the crate's dependencies and configure its build.rs script to invoke the UniFFI scaffolding generator, as described in "adding a new component".

Now, edit ./lib.rs so that it matches the interface defined in the .udl file as closely as possible. If the .udl has an interface Example then lib.rs should contain a pub struct Example, if the .udl contains an enum ExampleItem then lib.rs should contain a pub enum ExampleItem, and so-on.

The details of this step will depend heavily on the specific crate, but some tips include:

  • You may find it useful to move all of the existing code into a sub-module named internal, and then make a brand new lib.rs that imports or re-defines just the pieces it needs in order to implement the interface from the .udl file. The fxa-client crate is an example of a case where this worked out well, though of course your mileage may vary.

  • If the existing crate contains a file named like <component_name>_msg_types.proto, then it was using Protocol Buffers to serialize data to pass over the FFI. The message types defined in the .proto file will need to be converted into dictionary or enum definitions in your .udl file. See the section below for more details.

As noted above, don't be afraid to accept some API churn during the conversion process. We're willing to accept some breaking API changes as the cost of getting bindings generated for free, as long as the core functionality and mental model of the component remain intact.

At this point, in theory the crate should be buildable with UniFFI, although it's likely to require some iteration to get it all working! Run cargo check to check for any compilation errors without having to do a full build.

Removing Protobuf Messages

Passing rich structured data over the FFI is the most complex part of our hand-written bindings, and was previously done by serializing data via Protocol Buffers. This is something that UniFFI tries to make as simple as possible.

Start by locating the <component_name>_msg_types.proto file for the component. This file defines the structured messages that can be passed over the FFI, and you should see that they correspond to various types of structured data that the component wants to receive from, or return to, the foreign-language code.

Find the places in your .udl interface that correspond to these message types and make sure that you've got a similarly-shaped dictionary or enum for each one. You should find that representing this structured data in UDL is simpler than protobuf in many cases - for example many of our .protobuf files need to use a separate ExampleStructs message in order to pass a list of ExampleStruct messages over the FFI, but in UniFFI this is represented directly as sequence<ExampleStruct>.

Find the places in the Rust code that are using these message types to return structured data. In simple cases, you may be able to directly replace uses of msg_types::ExampleStruct with the corresponding crate::ExampleStruct from your public API. For more complex cases, you may find it helpful to define an Into mapping between the UniFFI dictionary/enum in the crate's public interface, and a more complex struct designed for internal use.

As noted above, don't be afraid to accept some API churn during this conversion process.

Once you have replaced all uses of the msg_types structs in the Rust code:

  • Delete ./src/<component_name>_msg_types.proto.
  • Delete ./src/mozilla.appservices.<component_name>.protobuf.rs, which is generated from the .proto file.
  • Remote prost and prost-derive from the crate's dependencies.
  • Delete the crate from the list in /tools/protobuf_files.toml.

If you happen to find that you've deleted the last crate from the list in protobuf_files.toml, congratulations! You've successfully removed protocol buffers from this repo entirely, and should file a bug to track the complete removal of protobuf from our tooling and dependency chain.

Document the Public API in the Rust code

Write consumer-facing documentation on the public API in lib.rs using Rust's standard rustdoc conventions and tools. The fxa-client crate may serve as a good example.

You can view the generated documentation by running:

cargo doc --no-deps --open

In future, we intend to automatically extract documentation from the Rust code and make it easily available to consumers of the generated bindings.

(In fact there is some work-in-progress code in uniffi-rs#416 that can read docs from the Rust code and write them back into the .udl file, which you're welcome to try out if you're feeling adventurous. But it's just a very hacky prototype.)

Set up the Kotlin wrapper

It's easiest to start by removing all of the hand-written Kotlin code under android/src/main/java and then restoring parts of it later if necessary. Leave the AndroidManifest.xml file and any tests in place.

Delete the android/build.gradle file and then follow the instructions for adding Kotlin bindings for a new component to create a new build.gradle file and a corresponding uniffi.toml.

This should be all that's required to set up UniFFI to build the Kotlin bindings. Try building the Android package to confirm:

  • ./gradlew <component_name>:assembleDebug

The UniFFI-generated Kotlin code will be under ./android/build/generated/source/uniffi/ and may be useful for debugging.

If there are existing Kotlin tests for the component, the next step is to get those passing:

  • ./gradlew <component_name>:test

As noted above, it is normal and expected for the autogenerated bindings to be subtly different from the previous hand-written ones. For example, UniFFI insists on using SHOUTY_SNAKE_CASE variant names in Kotlin enums while the hand-written code may have used CamelCase. Some components also have small naming differences between the Rust code and the hand-written Kotlin bindings, which UniFFI will not allow.

If the component had functionality in its Kotlin layer that was not part of the Rust API, then you'll need to add some hand-written Kotlin code under android/src/main/java to implement it. The fxa-client component may be a good example here: its Rust layer exposes a FirefoxAccount struct that the Kotlin code wraps into a PersistedFirefoxAccount class, adding the ability to set a persistence callback.

Finally, you will need to try out the new bindings with a consuming app. For Kotlin code you should make a local build of android-components and Fenix, updating them to accomodate any changes in the component's public API.

Set up the Swift wrapper

It's easiest to start by removing all of the hand-written Swift code under ./ios and then restoring parts of it later if necessary.

Edit /megazords/ios-rust/MozillaTestServices.h to remove any references to Rust<ComponentName>API.h, replacing them with the UniFFI-generated header file name <component_name>FFI.h.

Open /megazords/ios-rust/MozillaTestServices.xcodeproj in Xcode and follow the instructions for adding Swift bindings for a new component to configure Xcode to build your UniFFI-generated bindings.

While you are in the Xcode Project Navigator, you should also delete any references to Rust<ComponentName>API.h or to the old hand-written Swift wrappers. (They should be highlighted in red in the Project Navigator, because the files will be missing from disk after you deleted them above).

This should be all that's required to set up UniFFI to build the Swift bindings. Try building the project in Xcode to confirm.

The UniFFI-generated Swift code will be under ios/Generated and may be useful for debugging.

If there are existing Swift tests for the component, the next step is to get those passing:

  • ./automation/run_ios_tests.sh
  • (or run them from the Xcode GUI)

As noted above, it is normal and expected for the autogenerated bindings to be subtly different from the previous hand-written ones. Many existing components have small naming differences between the Rust code and the hand-written Swift bindings, which UniFFI will not allow.

If the component had functionality in its Swift layer that was not part of the Rust API, then you'll need to add some hand-written Swift code under ./ios/<ComponentName> to implement it. The fxa-client component may be a good example here: its Rust layer exposes a FirefoxAccount struct that the Swift code wraps into a PersistedFirefoxAccount class, adding the ability to set a persistence callback.

You will need to add any such file to the "Compile Sources" list in Xcode, in the same way that you added the .udl file.

Finally, you will need to try out the new bindings with a consuming app. For Swift code you should make a local build of Firefox iOS, you can do that by following the steps in this document