Mozilla Firefox: Development Process

Status: Draft

This document describes the process by which changes to Firefox source code become a general Firefox release. Firefox has hundreds of millions of users, so some care is in order.

Firefox uses a schedule-driven process, where releases take place at regular intervals. That means each release happens regardless of whether a given feature is ready, and releases are not delayed to wait for a feature to stabilize. The goal of the process is to provide regular improvements to users without disrupting longer term work.

Powers of Ten

All changes to Firefox source code are initially integrated in the mozilla-central Mercurial repository. At scheduled intervals, the changes are imported from mozilla-central to release channels with wider audiences, such as the beta channel. Eventually, the changes appear in the general Firefox release. To get your patch into the mozilla-central repository, see How to Submit a Patch. Larger features and projects are usually initially developed in other repositories which track mozilla-central.

The four channels available are: mozilla-central (or "nightly"), mozilla-aurora, mozilla-beta, and Firefox (release) [these names are placeholders]. The nightly channel gets new features as soon as they are ready, but it has the lowest stability of the four channels. The UI might change each day, and websites might not work at times. The mozilla-aurora channel gets new features at regular intervals, but some of them might be disabled if it looks like they need more work. The beta channel receives only new features that are slated for the next Firefox release.

Each release channel is backed by a Mercurial repository containing a distinct copy of the Firefox source code.As a set of changes progresses through the repositories, features that aren't quite ready are disabled. New features are never directly added to the mozilla-aurora or mozilla-beta channels. Features that end up disabled or miss the scheduled transition to the experimental channel can be pulled again the next time the schedule permits. This policy does allow for features that take a long time to develop. It's just that they'll be present only on mozilla-central until they're ready. The number of users on each channel grows by roughly 10x as the changes make their way through the release process, and features on each channel require an accompanying improvement in stability.

Again, new features are only added to the release in the first phase, with 100K users, because the rest of the time in the schedule is spent stabilizing those changes. Additionally, some features might only make it part of the way through the process for some period of time. For example, one could imagine a feature with a server-side component being available only on the experimental channel in order to test the scalability of companion server-side infrastructure.

Releases Over Time

Since the four stages of development each require some time, they can be staggered. This arrangement allows for continuous new feature development on mozilla-central, while the other channels are devoted to stabilizing features ready for a wider audience.

Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
mozilla-central        
mozilla-exp        
mozilla-beta      
Firefox    
      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

In the diagram above, the blue bars represent one set of changes progressing through each channel. Simultaneously, work continues on the mozilla-central repository. Some features might need three cycles (18 weeks) on mozilla-central before they're ready. Since mozilla-central will be pulled into the mozilla-aurora channel at week 6 and week 12, such features will need to have a mechanism allowing them to be easily disabled. Essentially, it needs to be easy to get the code on mozilla-central into a shippable state by disabling new code.

The mozilla-central, mozilla-aurora, and mozilla-beta channels receive frequent updates so changes to the program can be validated before appearing in the general release. The release channel receives updates from the beta channel periodically.

Under this system, there is a choice to ship a general Firefox release at week 16 and every six weeks thereafter. That doesn't mean a release will happen every six weeks, but the option will be available.

Moving work from one channel to another

New code migrates from one repository to another on a fixed schedule. For example, in the diagram above, new code from mozilla-central arrives in week 6, 12, and 18. It might seem expedient to land new features from mozilla-central as soon as they are "ready", but such a pattern would introduce too many variables into stability measurements. The one exception to this schedule is that unscheduled releases may be required for critical security and stability issues.

The first few days after code arrives on a channel are spent turning features off, backing out patches that cause stability problems, etc. There will be many judgement calls concerning whether it's possible to save a feature with a few spot fixes. As Mozilla gains experience with this process, more concrete policies will emerge. The reason this work happens after new code appears on a channel is so that work can continue uninterrupted in the other repositories.

If a feature needs to be turned off on a release channel, it's the developer's responsibility to make a patch to do so. If a feature can't be disabled without turning off or backing out other things, then those other things will also get turned off or backed out.

The Lifecycle of One Release

A single release takes about 16 weeks of time. This section outlines some key release management dates in that timeline. The diagram below describes where quality assurance, localization, security, extension compatibility, and security updates fit in.

Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
mozilla-central        
mozilla-aurora        
mozilla-beta      
Firefox    
 
l10n-central        
l10n-exp        
l10n-beta      
Fx l10n    
 
Extensions          
Security      
QA      
Go / No Go          
Repo Pull              
      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Week 1-5

Feature development on mozilla-central.

The feature development is tracked by a core of the localization community, giving feedback about l12y early in the development process.

Week 6

The code on mozilla-central is pulled to the mozilla-aurora channel.

Extension developers can rely on the APIs they see here, with the exception of new APIs that might end up disabled. The front end XUL might change for stability reasons, but not in major ways. Again, XUL related to new features may not be available if those features end up getting disabled.

en-US strings are frozen.

Week 7-10

Release management, QA, and engineering work to stabilize or disable new features.

The majority of localization teams finish their work on the product.

Week 11

Release management makes a Go/No Go decision on whether the mozilla-aurora channel can be pulled.

If the decision is made to Go, a repository pull from mozilla-aurora to mozilla-beta is done.

Only minor changes to localizations are still taken.

Week 12-13

Release management, QA, and engineering work to stabilize any issues that arise as the code is exposed to the mozilla-beta channel's wider audience.

Security patches from the shadow security repository are landed. n.b.: exact timing and mechanics undecided.

Week 14

Release management makes a Go/No Go to make release the contents of the mozilla-beta channel to the general audience.

Week 16

If there's a release to do, it happens during this week.

Localizations that are in beta are not pulled into the release channel.

Transition and Bootstrapping

This section describes how Mozilla will transition to the process described above. Firefox 4 took about a year to develop, and the proposed schedule is quite a bit faster.

For Mozilla to make a clean break from long, feature-driven release schedules, the project will need find a way to transition to schedule-driven releases. The accumulation of patches on mozilla-central during the initial rollout of Firefox 4 is a primary concern. The way we're going to handle this problem is by taking an extremely early cut at Firefox 5. In the table below, there's not yet a calendar date associated with "Week 1", but it will come very soon after the general release of Firefox 4.

Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
mozilla-central        
mozilla-aurora          
mozilla-beta        
Firefox      
      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

The goal of this proposed initial release pattern is to limit patch build up in mozilla-central, as well as establish a regular six week cadence for release go/no-go decisions thereafter.

Channel Update System

For this plan to work at all, we need establish the mozilla-aurora and mozilla-beta channels. The plan is to seed them with roughly 500K users on the experimental channel, and about 1.5 million people on the mozilla-beta channel. We'll use the current Firefox 4 beta audience for this. Once the channels are available as a choice in the general Firefox release, their audiences should grow.

There's also a dependency on the updater code and UI needed for this system in the first place, and it will be difficult to get it done in time.

Security Releases

This proposal makes security updates occur along with Firefox releases, meaning we'll no longer be maintaining old branches. Having security branches for each major update is untenable if we release as often as we aim to.

Silent Updates

This proposal also requires changes to our software update behavior to make them happen more automatically in the background and interrupt the user less often. Otherwise, we will disrupt mozilla-beta users too much. There will also need to be an option for users to completely disable automatic updates, so that they can manage their own upgrade process.

Extension Compatibility

Extension compatibility is the trickiest part of the transition. In particular, it's not what the policy should be when a user has extensions that are incompatible with a new Firefox release. Each release will have at least 12 weeks to identify extensions that are no longer working, but this issue will be complicated.

What about Firefox 4.0.x?

We'll be using existing branch maintenance processes for Firefox 4.0.x and Firefox 3.6.x security and stability updates.

 

Acknowledgements: Thanks to the following contributors for their input and ideas. Listed in alphabetical order by first name:

Alexander Limi, Anthony LaForge, Asa Dotzler, Axel Hecht, Benjamin Smedberg, Brendan Eich, Cheng Wang, Christian Legnitto, Damon Sicore, Daniel Veditz, David Anderson, David Ascher, David Mandelin, Ehsan Akhgari, Erica Jostedt, Jay Sullivan, John O'Duinn, Johnathan Nightingale, Johnny Stenback, Juan Becerra, Justin Wood, L. David Baron, Laura Mesa, Lucas Adamski, Mark Finkle, Matt Evans, Mayumi Matsuno, Michael Morgan, Mike Beltzner, Mike Shaver, Mitchell Baker, Nick Thomas, Rob Sayre, Robert Kaiser, Robert O'Callahan, Robert Strong, Sheila Mooney, Stuart Parmenter