Sync Overview

This document provides a high-level overview of how syncing works. Note: each component has its own quirks and will handle sync slightly differently than the general process described here.

General flow and architecture

• Crates involved:
  • The sync15 and support/sync15-traits handle the general syncing logic and define the SyncEngine trait
  • Individual component crates (logins, places, autofill, etc). These implement SyncEngine.
  • sync_manager manages the overall syncing process.
• High level sync flow:
  • Sync is initiated by the application that embeds application-services.
  • The application calls SyncManager.sync() to start the sync process.
  • SyncManager creates SyncEngine instances to sync the individual components. Each SyncEngine corresponds to a collection on the sync server.

Sync manager

SyncManager is responsible for performing the high-level parts of the sync process:

• The consumer code calls it's sync() function to start the sync, passing in a SyncParams object in, which describes what should be synced.
• SyncManager performs all network operations on behalf of the individual engines. It's also responsible for tracking the general authentication state (primarily by inspecting the responses from these network requests) and fetching tokens from the token server.
• SyncManager checks if we are currently in a backoff period and should wait before contacting the server again.
• Before syncing any engines, the sync manager checks the state of the meta/global collection and compares it with the enabled engines specified in the SyncParams. This handles the cases when the user has requested an engine be enabled or disabled on this device, or when it was requested on a different device. (Note that engines enabled and disabled states are state on the account itself and not a per-device setting). Part of this process is comparing the collection's GUID on the server with the GUID known locally - if they are different, it implies some other device has "reset" the collection, so the engine drops all metadata and attempts to reconcile with every record on the server (ie, acts as though this is the very first sync this engine has ever done).
• SyncManager instantiates a SyncEngine for each enabled component. We currently use 2 different methods for this:
  • The older method is for the SyncManager to hold a weakref to a Store use that to create the SyncEngine (tabs and places). The SyncEngine uses the Store for database access, see the TabsStore for an example.
  • The newer method is for the components to provide a function to create the SyncEngine, hiding the details of how that engine gets created (autofill/logins). These components also define a Store instance for the SyncEngine to use, but it's all transparent to the SyncManager. (See autofill::get_registered_sync_engine() and autofill::db::store::Store)
• For components that use local encryption, SyncManager passes the local encryption key to their SyncEngine
• Finally, calls sync_multiple() function from the sync15 crate, sending it the SyncEngine instances. sync_multiple() then calls the sync() function for each individual SyncEngine

Sync engines

• SyncEngine is defined in the support/sync15-traits crate and defines the interface for syncing a component.
• A new SyncEngine instance is created for each sync
• SyncEngine.apply_incoming() does the main work. It is responsible for processing incoming records from the server in order to update the local records and calculating which local records should be synced back.

The apply_incoming pattern

SyncEngine instances are free to implement apply_incoming() any way they want, but the most components follow a general pattern.

Database Tables

• The local table stores records for the local application
• The mirror table stores the last known record from the server
• The staging temporary table stores the incoming records that we're currently processing
• The local/mirror/staging tables contains a guid as its primary key. A record will share the same guid for the local/mirror/staging table.
• The metadata table stores the GUID for the collection as a whole and the the last-known server timestamp of the collection.

apply_incoming stages

• stage incoming: write out each incoming server record to the staging table
• fetch states: take the rows from all 3 tables and combine them into a single struct containing Options for the local/mirror/staging records.
• iterate states: loop through each state, decide how to do change the local records, then execute that plan.
  • reconcile/plan: For each state we create an action plan for it. The action plan is a low-level description of what to change (add this record, delete this one, modify this field, etc). Here are some common situations:
    • A record only appears in the staging table. It's a new record from the server and should be added to the local DB
    • A record only appears in the local table. It's a new record on the local instance and should be synced back to the serve
    • Identical records appear in the local/mirror tables and a changed record is in the staging table. The record was updated remotely and the changes should be propagated to the local DB.
    • A record appears in the mirror table and changed records appear in both the local and staging tables. The record was updated both locally and remotely and we should perform a 3-way merge.
  • apply plan: After we create the action plan, then we execute it.
• fetch outgoing:
  • Calculate which records need to be sent back to the server
  • Update the mirror table
  • Return those records back to the sync15 code so that it can upload them to the server.
  • The sync15 code returns the timestamp reported by the server in the POST response and hands it back to the engine. The engine persists this timestamp in the metadata table - the next sync will then use this timestamp to only fetch records that have since been changed by other devices

syncChangeCounter

The local table has an integer column syncChangeCounter which is incremented every time the embedding app makes a change to a local record (eg, updating a field). Thus, any local record with a non-zero change counter will need to be updated on the server (with either the local record being used, or after it being merged if the record also changed remotely). At the start of the sync, when we are determining what action to take, we take a copy of the change counter, typically in a temp staging table. After we have uploaded the record to the server, we decrement the counter by whatever it was when the sync started. This means that if a record is changed in between staging the record and uploading it, the change counter will not drop to zero, and so it will correctly be seen as locally modified on the next sync