[Back to top]

Process Isolation in Firefox

Randell Jesup, Mozilla Browser Architecture team

NOTE: this document is a Work in Progress!

Overview

We’ve recently moved a number of parts of Firefox into separate processes (e10s, multi-e10s, GPU process, compositor, WebExtensions process, GMP). This has produced some serious improvements in both security (sandboxing content, codecs) and stability (restarting GPU process when a GPU driver crashes, etc). This project is to evaluate how much further we can push process isolation to further improve these things.

Problems:

There are some secondary benefits we hope to achieve by doing this, such as decoupling parts of the system and providing more-stable interfaces to them, as well as easing some aspects of unit testing. There may be some advantages in build times or cost of maintenance; we’ll see.

There are costs: development time, memory use, and performance all can or will be negatively impacted. Part of this project is to quantify these impacts (and hopefully reduce them) in order to guide the decisions on how far to push this process.

Chrome/Chromium has been doing similar work recently on “Servicification”. This is related to their slowly replacing classic ipc/chromium-based IPC with “Mojo”; a new IPC implementation (and ipdl-like layer) that has improved performance compared to classic Chromium IPC. Note that some ways Mozilla uses IPC might avoid some of the performance costs in Chrome (multiple channels, PBackground and the like, for example) - but we haven’t yet assessed how much overlap there is between Mojo and our additions to Chromium IPC. It may be that with some smaller modifications our use of Chromium IPC will be as efficient (or nearly so) as Mojo.

Chrome is also working on Site Isolation, to avoid a compromised Renderer from breaking cross-origin isolation (more details below).

As part of this work, since it affects the performance impact of Process Isolation, we plan to explore the potential of adopting Mojo for Mozilla IPC (either wholesale, or progressively as Chrome has been doing).

Alternatively, and maybe more interestingly, we could look at using a Rust IPC crate and some added interface logic. <Need some more concrete suggestions/pointers here>

Note: we don’t want to just “follow” Chrome’s lead, but to do what’s smart for the users and Firefox, whether it’s similar to Chrome or not. Leapfrogging them in some manner would be great, but is not a requirement. If we can leverage work or research they’ve done to reduce our cost or risks, great.

GOAL

Tentative Work Plan

  1. Measure the overhead of using Process Isolation:

    1. Memory cost for various scenarios (which largely depend on what part of firefox code need to be initialized/used in the process - XPCOM, etc)

      1. Memory overhead for a Process on Win10 x64 appears to be circa 7-10 MB (Private) for a process with minimal XPCOM inited, IPC, etc. The GPU process (which also loads GPU drivers, etc) is on the order to 20MB, and anecdotally the GMP process uses on the order of 10MB, which is in line.

    2. Performance cost - latency and throughput of calls through IPC (with classic IPC and Mojo, and perhaps a Rust IPC crate)

      1. Still to be measured

    3. Evaluate if we can make Chromium IPC as efficient as Mojo

      1. I suspect we can; the main perf advantage Mojo has on classic IPC is one less thread-hop per message (2 less per round-trip - 4 vs 6). The overall code is probably a lot “cleaner”, but it would be a fair bit of work to convert over, though probably much of it could be hidden by ipdl.

      2. We believe that Mojo’s shutdown code may be more robust/better engineered than IPC’s; shutdown has been a common source of crash/security bugs.

      3. We think it might be possible in some special(?) cases to avoid a thread hop on the receiving side as well as the sending. Mojo does not do this.

    4. Startup cost - cost to browser startup time

      1. Unknown. Need to measure the cost in time to start a minimal process, and see how many we need at startup. We may be able to partly overlap creation and startup of service processes. Expectation is that this will not be a blocker.

    5. Service startup time cost - cost on first use of a service which requires spawning a Process

      1. Evaluate using “embryo” processes to allow the forked processes to be smaller but not pay new relocation/etc costs (i.e. don’t fork the Master process and all that entails when we spin up a process/service). (B2G did something like this with Nuwa.) Note that this doesn’t help Windows apparently, and there was some sort of issue with this on Mac - these need to be examined/verified.

  2. Analysis of Process Isolation

    1. Analysis of the code maintenance impact

    2. Analysis of the stability impact

    3. Analysis of the security impact

    4. Analysis of embryo processes

      1. Per-OS

    5. Analysis of IPC options

      1. Update/improve current Chromium IPC

      2. Mojo

      3. Rust IPC

    6. Android analysis - android will likely require different tradeoffs

  3. Develop a preliminary list of potential subsystems/features to consider for Isolation

    1. Necko is already planning to Isolate network socket access and protocol code (and some crypto code) after 57 or 58 – Bug 1322426

      1. They expect to land code in 61 behind a pref, and enable it in release in 63.

        1. This will not be happening in the near term, due to staffing changes. It’s not yet clear when or if this will occur.

    2. Video camera capture code (and screensharing) is another prime target, as it’s already remoted over IPC even when running non-e10s. The way this works is very similar in principle to Chrome’s remote-services-over-Mojo approach.

    3. Places in particular, eventually profile data access in general. This pushes storage asynchrony to the process boundary and decouples lifetimes (background storage and syncing, faster ‘startup’ and relaunch, with Places possibly living longer than a crashed browser). Future technology refactors could make this standalone process reusable outside of desktop Firefox. No bugs filed yet.

    4. Font and/or Image code to avoid or reduce duplication of data between Content processes and the Compositor.

    5. Printing

    6. PDF display/PDFium

  4. Look at the Content Process state and model (mostly this has been done in the e10s team)

    1. How far do we want to push the model towards Chrome’s 1-per-origin/iframe model?

      1. Probably not as far… note however that Chrome has closed the gap we created with them on memory use. [reference?]

      2. Even Chrome can’t get away with their stated goal (yet?)

        1. While site isolation with a small number of sites is in the 15-20% range, this is largely because of the base overhead of Chrome (Master process, GPU processes, zygotes, etc). WIth 17 tabs (a mixture of simple and complex sites), the measured overhead was about 50%.

    2. How much does servicification help reduce Chrome process overhead (avoiding N instances of things)

      1. It may not help a lot

    3. How much can this work help sandbox hardening?

      1. Main Process will still need file access probably

  5. Very speculative: examine if the current Master Process could be moved to be a child process of a thin Master Process, allowing restarts on Master Process crash without reloading all the running Content Processes.

    1. Very likely this is too hard and/or too much work. There’s lots of state stored in the Master on behalf of the Content processes; the reconnect operation between Content and restarted Master would be considerably complex.

Previous work

Chrome’s Servicification and Mojo

Chrome has been discussing Servicification since roughly early 2016, and major work on it has begun this year (2017). This is the primary document: Chrome Service Model, and this is the primary root of the Servicification work: Servicification.

An example of one item currently being moved to a Service is Video Capture: Design Doc and detailed plans and measurements. Another which I think has been completed is the Prefs Service.

Mojo consists of a set of common IPC primitives, a message IDL format, and a bindings library (with generation for a number of languages; undoubtedly we’d need to add Rust binding generation – C++ bindings are here). Mojo has been measured in Chrome as being about ⅓ faster than classic IPC, and produces ⅓ less context switches. (Probably these two facts are related, and their performance analysis indicates that not having to “hop to the IO thread” is part of why it’s faster, which makes sense.)

One thing we plan to experiment with is seeing if (leveraging our IPDL compiler) we can fairly transparently replace (some?) existing Chromium IPC channels/messages with Mojo.

Chrome has docs on how they move legacy IPC code to Mojo. This is a (somewhat dated) cheat sheet on moving code from IPC and Mojo.

One interesting tidbit from that cheatsheet:

IPC

IPCs can be sent and received from any threads. If the sending/receiving thread is not the IO thread, there is always a hop and memory copy to/from the IO thread.

Mojo

A binding or interface pointer can only be used on one thread at a time since they’re not thread-safe. However the message pipe can be unbound using either Binding::Unbind or InterfacePtr::PassInterface and then it can be bound again on a different thread. Sending an IPC message in Mojo doesn’t involve a thread hop, but receiving it on a thread other than the IO thread does involve a thread hop.

Mojo has extensive support for message validation:

Regardless of target language, all interface messages are validated during deserialization before they are dispatched to a receiving implementation of the interface. This helps to ensure consistent validation across interfaces without leaving the burden to developers and security reviewers every time a new message is added.

Overhead Details

1. Memory use.  Content Process overhead is tracked in [Bug 1436250](https://bugzilla.mozilla.org/show_bug.cgi?id=1436250).  It’s measured both with a small patch to dump the ASAN heap state on command, and using a DMD build with specific environment options.

    1. Minimal process (with IPC)

    2. With XPCOM

        1. 7-10MB

    3. Full Content process

        2. 25-30MB (varies by OS, memory model (32 vs 64), and system details (fonts, etc))

        3. Content Process overhead is critical for Site Isolation

2. Performance -- measure each with small messages, large messages, and shared-memory payloads (anything else?)

    4. Latency

    5. Messages/second

Security implications

Moving services and other code into sandboxed processes should generally increase the resilience of the system to security bugs. In particular, compromising the system in one sandboxed process will require a sandbox escape of some sort to leverage that into full or increased control, and generally will only expose data already flowing through the Service to the attacker - and for many sandboxed processes, exfiltrating compromised data would be much harder.

How hard exfiltration would be depends on what sort of data flows through the Service, how locked-down we can make the process, and if the output of the process normally is in some way visible to content - for example if the process did image decoding, then data could be exfiltrated by using it to break cross-domain restrictions (such as taking the content of image A in domain X, and outputting it in place of image B from domain Y (the attacker’s domain), allowing the attacker to render it in a canvas).

Another way that isolation will help security is by separating memory pools - memory errors such as UAFs can become much harder to exploit if you can’t put the memory system under load (forcing reallocation of the memory with content you control, for example). In a Content process with JS running (and tons of other stuff), this is often not too hard; in an isolated Service it might be very hard indeed.

Once a Process is compromised, leveraging that into an exploit requires escaping into other Processes (using further bugs), or leveraging an OS bug. How hard that is depends on the OS and how locked-down we can make these Processes. “Small” processes may have much smaller OS attack surfaces, though this might be tougher to do on (say) Windows due to granularity of permissions.

What Chrome does

Chrome doesn’t actually use a Process-per-tab (or origin), though many people believe it does: see Peter Kasting’s post from June. (That was partially in response to some of our announcements around E10S.) The number of Render processes they use depends on the available memory - though it sounds like they have bugs there, and that may cause them to overrun and slow down the user’s system.

Chrome is working on Site Isolation. Part of this is putting iframes OutOfProcess from the main page renderer, but more generally it’s about not using a renderer (Content process) for more than one origin. This has some serious downsides if taken to an extreme, and currently they’re planning to do this only for “high value” sites. (It’s unclear what “high value” means here; one presumes banks, paypal, and other especially juicy targets.)

As mentioned above, Chrome is increasing the number of non-Render processes they use as part of Servicification.

The Chrome Process Model document is useful, but very out of date with current details - for example, the Site Isolation work they’ve done. Some of the code for all these decisions is here.

What Edge does

In “Browser security beyond sandboxing” Microsoft goes into detail on a Chrome vulnerability, but also highlights some of what they’ve done - in particular Arbitrary Code Guard: “ACG, which was introduced in Windows 10 Creators Update, enforces strict Data Execution Prevention (DEP) and moves the JIT compiler to an external process. This creates a strong guarantee that attackers cannot overwrite executable code without first somehow compromising the JIT process, which would require the discovery and exploitation of additional vulnerabilities.” Their post introducing ACG is here.

In more detail: “To support this, we moved the JIT functionality of Chakra into a separate process that runs in its own isolated sandbox. The JIT process is responsible for compiling JavaScript to native code and mapping it into the requesting content process. In this way, the content process itself is never allowed to directly map or modify its own JIT code pages.”

This suggests that we should consider moving the JIT itself out of the main process space, and just share the result of the JIT back with the Content process requesting it. Since we already do JIT on a non-MainThread, the main issue here is probably shared-memory management. There will be some perf tests to do on this to validate if this is feasible within our current overall architecture, but it seems possible. This is being tracked in Bug 1348341, and Tom Ritter has been investigating in this doc.

Note that if the JIT process is compromised, anything running through it is as well, and any content processes it provides code for. This would imply that JIT processes may need to be tied to a single requesting Content Process to avoid stepping backwards in security here (and this also increases the potential memory cost).

[Back to top]