Experiments & A/B Testing

Current as of December 3rd, 2019

This document outlines how to add an experiment, A/B test, or phased rollout of a feature in the content server.

Experiment grouping rules

Every new experiment, A/B test, or phased rollout needs an experiment grouping rule. These rules decide whether a user is part of an experiment, and if so, which group.

An experiment grouping rule needs some metadata about the user, within the rule the metadata is called subject.

Creating a new rule

1. Copy TEMPLATE.js to a new grouping rule file.
2. Change ChangeMeGroupingRule class name to another name.
3. Change this.name from CHANGE_ME in the constructor.
4. Fill in the choose function. See choose recipes for guidance on different experiment types.
5. Include the new grouping rule file in index.js.
6. Add experiment name to one of MANUAL_EXPERIMENTS or STARTUP_EXPERIMENTS to ensure test name/group are reported to Amplitude.
7. Access in views, see View recipes.

Determining choice within a view

Once a grouping rule has been created, a view can make choices depending on whether the user is in the experiment, and if so which group. ExperimentMixin must be mixed in to the view for the view to have experiment capabilities.

See also: View recipes.

Functional and manual testing

To avoid random breakage of functional tests due to experiments, by default, all experiments are disabled during test runs. It is however possible to force an experiment using the forceExperiment and forceExperimentGroup query parameters.

Both of these examples force the user into the treatment group of the my-new-experiment experiment.

Manual test

In the URL bar, open:

https://accounts.firefox.com/settings?forceExperiment=my-new-experiment&forceExperimentGroup=treatment

Functional test

  ...
  'test new feature here': function () {
    this.remote
      .then(openPage('/settings', {
        query: {
          forceExperiment: 'my-new-experiment',
          forceExperimentGroup: 'treatment'
        }
      }))
      .then(testTreatmentGroupFunctionality());
  },
  ...

Amplitude Metrics

warning

We no longer use Amplitude. This section is out of date.

As long as the experiment name is added to one of MANUAL_EXPERIMENTS or STARTUP_EXPERIMENTS, the experiment name and group are reported to Amplitude and added to the user's experiments user property. The Amplitude experiment name is in the form:

amplitudeExperimentName = ${snake_case(experimentName)}_${snake_case(groupName)}`

choose recipes

The subject parameter contains data used to determine the value returned from choose.

subject automatically contains the following fields:

Field	Description
account	Account model for the user.
experimentGroupingRules	A reference to all grouping rules. Used to recursively choose experiments
forceExperiment	the value of the forceExperiment query parameter. Used in functional/manual tests to force a particular experiment.
forceExperimentGroup	the value of the forceExperimentGroup query parameter. Used in functional/manual tests to force a particular experiment group.
isMetricsEnabledValue	A legacy of when metrics were sampled at 10%. Does not apply to Amplitude events.
uniqueUserId	A stable UUID that is distinct from the user's uid. The same uniqueUserId is used across multiple users of the same device to ensure multiple users of a single device receive the same experience.

Additional fields can be passed to choose via getExperimentGroup.

uniqueUserId is used to ensure multiple users receive the same experience on the same device. Conversely, a single user will have distinct uniqueUserIds on multiple devices. If the goal is for a single user to have the same experience across multiple devices, in the examples below replace subject.uniqueUserId with subject.account.get('email') or subject.account.get('uid').

feature flag

Return a boolean value.

choose (subject = {}) {
  return subject.firefoxVersion >= MIN_FIREFOX_VERSION;
}

Phased rollout feature flag

Use bernoulliTrial to return a Boolean value.

const ROLLOUT_RATE = 0.3;
choose (subject = {}) {
  return this.bernoulliTrial(ROLLOUT_RATE, subject.uniqueUserId);
}

A/B test

Use uniformChoice to return the bucket choice.

choose (subject = {}) {
  const GROUPS = ['control', 'treatment'];
  return this.uniformChoice(GROUPS, subject.uniqueUserId);
}

Phased rollout A/B test

Combine bernoulliTrial and uniformChoice. First, use bernoulliTrial to determine if the user is part of the experiment. If the user is part of the experiment, use uniformChoice to determine which bucket.

choose (subject = {}) {
  const ROLLOUT_RATE = 0.3;
  // first, determine if the user is part of the experiment.
  if (this.bernoulliTrial(ROLLOUT_RATE, subject.uniqueUserId)) {
    // User is part of the experiment, determine which bucket.
    // 15% of users will be in `control`, 15% in `treatment`
    const GROUPS = ['control', 'treatment'];
    return this.uniformChoice(GROUPS, subject.uniqueUserId);
  }
  return false;
}

Recursive calls to other rules

subject will contain a reference to experimentGroupingRules which can be used to recursively call other tests.

choose (subject = {}) {
  const choice = this.uniformChoice(['recursive-rule-1', 'recursive-rule-2'], subject.uniqueUserId);
  return subject.experimentGroupingRules.choose(choice, subject);
}

Mutually exclusive grouping rules

Recursive rules can be used to implement mutual exclusion amongst two or more grouping rules.

group_chooser.js

constructor () {
  super();
  this.name = 'experiment-chooser';
}

choose (subject = {}) {
  return this.uniformChoice(['experiment-1', 'experiment-2'], subject.uniqueUserId);
}

experiment_1.js

constructor () {
  super();
  this.name = 'experiment-1';
}

choose (subject = {}) {
  if (subject.experimentGroupingRules.choose('experiment-chooser') === this.name) {
    // user is part of the experiment-1, determine which bucket.
    const GROUPS = ['control', 'treatment'];
    return this.uniformChoice(GROUPS, subject.uniqueUserId);
  }
}

experiment_2.js

constructor () {
  super();
  this.name = 'experiment-2';
}

choose (subject = {}) {
  if (subject.experimentGroupingRules.choose('experiment-chooser') === this.name) {
    // user is part of the experiment-2, determine which bucket.
    const GROUPS = ['control', 'treatment'];
    return this.uniformChoice(GROUPS, subject.uniqueUserId);
  }
}

View recipes

Add the ExperimentMixin

import BaseView from './base';
import Cocktail from '../lib/cocktail';
import ExperimentMixin from './mixins/experiment-mixin'

class MyView extends BaseView {
  ...
}

Cocktail.extend(
  MyView,
  ExperimentMixin
);

Get the experiment group, report the choice to Amplitude

This is the most common form as well as the most flexible because it does everything necessary to report to Amplitude.

const experimentGroup = this.getAndReportExperimentGroup('experiment-2');
if (experimentGroup === 'treatment') {
    // do something awesome here.
}

Get the experiment group without reporting choice to Amplitude

const experimentGroup = this.getExperimentGroup('experiment-2');
if (experimentGroup === 'treatment') {
    // do something awesome here.
}

Is user in an experiment?

if (this.isInExperiment('experiment-2')) {
    // do something awesome here.
}

Is user in a group?

if (this.isInExperimentGroup('experiment-2', 'treatment')) {
    // do something awesome here.
}

Pass additional data to choose

const country = getCountrySomehow();
const experimentGroup = this.getExperimentGroup('experiment-2', {
  // can be used by `choose` to gate a feature on the user's country
  country
});
if (experimentGroup === 'treatment') {
    // do something awesome here.
}

Ensuring experiment metrics are reported to Amplitude

As noted in Creating a new rule, the experiment name must be added to one of MANUAL_EXPERIMENTS or STARTUP_EXPERIMENTS to ensure experiment metrics are reported to Amplitude.

Within a view, if the user is part of an experiment, this.createExperiment(experimentName, groupName) must be called.