Trait SyncEngine

pub trait SyncEngine {
    // Required methods
    fn collection_name(&self) -> CollectionName;
    fn stage_incoming(
        &self,
        inbound: Vec<IncomingBso>,
        telem: &mut Engine,
    ) -> Result<()>;
    fn apply(
        &self,
        timestamp: ServerTimestamp,
        telem: &mut Engine,
    ) -> Result<Vec<OutgoingBso>>;
    fn set_uploaded(
        &self,
        new_timestamp: ServerTimestamp,
        ids: Vec<Guid>,
    ) -> Result<()>;
    fn get_collection_request(
        &self,
        server_timestamp: ServerTimestamp,
    ) -> Result<Option<CollectionRequest>>;
    fn get_sync_assoc(&self) -> Result<EngineSyncAssociation>;
    fn reset(&self, assoc: &EngineSyncAssociation) -> Result<()>;

    // Provided methods
    fn prepare_for_sync(
        &self,
        _get_client_data: &dyn Fn() -> ClientData,
    ) -> Result<()> { ... }
    fn set_local_encryption_key(&mut self, _key: &str) -> Result<()> { ... }
    fn sync_finished(&self) -> Result<()> { ... }
    fn wipe(&self) -> Result<()> { ... }
}

A “sync engine” is a thing that knows how to sync. It’s often implemented by a “store” (which is the generic term responsible for all storage associated with a component, including storage required for sync.)

The model described by this trait is that engines first “stage” sets of incoming records, then apply them returning outgoing records, then handle the success (or otherwise) of each batch as it’s uploaded.

Staging incoming records is (or should be ;) done in batches - eg, 1000 record chunks. Some engines will “stage” these into a database temp table, while ones expecting less records might just store them in memory.

For outgoing records, a single vec is supplied by the engine. The sync client will use the batch facilities of the server to make multiple POST requests and commit them. Sadly it’s not truly atomic (there’s a batch size limit) - so the model reflects that in that the engine gets told each time a batch is committed, which might happen more than once for the supplied vec. We should upgrade this model so the engine can avoid reading every outgoing record into memory at once (ie, we should try and better reflect the upload batch model at this level)

Sync Engines should not assume they live for exactly one sync, so prepare_for_sync() should clean up any state, including staged records, from previous syncs.

Different engines will produce errors of different types. To accommodate this, we force them all to return anyhow::Error.

Required Methods

fn collection_name(&self) -> CollectionName

fn stage_incoming( &self, inbound: Vec<IncomingBso>, telem: &mut Engine, ) -> Result<()>

Stage some incoming records. This might be called multiple times in the same sync if we fetch the incoming records in batches.

Note there is no timestamp provided here, because the procedure for fetching in batches means that the timestamp advancing during a batch means we must abort and start again. The final collection timestamp after staging all records is supplied to apply()

fn apply( &self, timestamp: ServerTimestamp, telem: &mut Engine, ) -> Result<Vec<OutgoingBso>>

Apply the staged records, returning outgoing records. Ideally we would adjust this model to better support batching of outgoing records without needing to keep them all in memory (ie, an iterator or similar?)

fn set_uploaded( &self, new_timestamp: ServerTimestamp, ids: Vec<Guid>, ) -> Result<()>

Indicates that the given record IDs were uploaded successfully to the server. This may be called multiple times per sync, once for each batch. Batching is determined dynamically based on payload sizes and counts via the server’s advertised limits.

fn get_collection_request( &self, server_timestamp: ServerTimestamp, ) -> Result<Option<CollectionRequest>>

The engine is responsible for building a single collection request. Engines typically will store a lastModified timestamp and use that to build a request saying “give me full records since that date” - however, other engines might do something fancier. It can return None if the server timestamp has not advanced since the last sync. This could even later be extended to handle “backfills”, and we might end up wanting one engine to use multiple collections (eg, as a “foreign key” via guid), etc.

fn get_sync_assoc(&self) -> Result<EngineSyncAssociation>

Get persisted sync IDs. If they don’t match the global state we’ll be reset() with the new IDs.

fn reset(&self, assoc: &EngineSyncAssociation) -> Result<()>

Reset the engine (and associated store) without wiping local data, ready for a “first sync”. assoc defines how this store is to be associated with sync.

Provided Methods

fn prepare_for_sync( &self, _get_client_data: &dyn Fn() -> ClientData, ) -> Result<()>

Prepares the engine for syncing. The tabs engine currently uses this to store the current list of clients, which it uses to look up device names and types.

Note that this method is only called by sync_multiple, and only if a command processor is registered. In particular, prepare_for_sync will not be called if the store is synced using sync::synchronize or sync_multiple::sync_multiple. It will be called if the store is synced via the Sync Manager.

TODO(issue #2590): This is pretty cludgey and will be hard to extend for any case other than the tabs case. We should find another way to support tabs…

fn set_local_encryption_key(&mut self, _key: &str) -> Result<()>

Tells the engine what the local encryption key is for the data managed by the engine. This is only used by collections that store data encrypted locally and is unrelated to the encryption used by Sync. The intent is that for such collections, this key can be used to decrypt local data before it is re-encrypted by Sync and sent to the storage servers, and similarly, data from the storage servers will be decrypted by Sync, then encrypted by the local encryption key before being added to the local database.

The expectation is that the key value is being maintained by the embedding application in some secure way suitable for the environment in which the app is running - eg, the OS “keychain”. The value of the key is implementation dependent - it is expected that the engine and embedding application already have some external agreement about how to generate keys and in what form they are exchanged. Finally, there’s an assumption that sync engines are short-lived and only live for a single sync - this means that sync doesn’t hold on to the key for an extended period. In practice, all sync engines which aren’t a “bridged engine” are short lived - we might need to rethink this later if we need engines with local encryption keys to be used on desktop.

This will panic if called by an engine that doesn’t have explicit support for local encryption keys as that implies a degree of confusion which shouldn’t be possible to ignore.

fn sync_finished(&self) -> Result<()>

Called once the sync is finished. Not currently called if uploads fail (which seems sad, but the other batching confusion there needs sorting out first). Many engines will have nothing to do here, as most “post upload” work should be done in set_uploaded()

fn wipe(&self) -> Result<()>

Wipes the engine’s data This is typically triggered by a client command, which at the time of writing, only supported wiping bookmarks.

This panics if triggered on a sync engine that does not explicitly implement wipe, because that implies a confustion that shouldn’t occur.

Implementors