mozanalysis.experiment

class mozanalysis.experiment.EnrollmentsQueryType(value)[source]

An enumeration.

class mozanalysis.experiment.Experiment(experiment_slug, start_date, num_dates_enrollment=None, app_id=None, app_name=None)[source]

Query experiment data; store experiment metadata.

The methods here query data in a way compatible with the following principles, which are important for experiment analysis:

  • The population of clients in each branch must have the same properties, aside from the intervention itself and its consequences; i.e. there must be no underlying bias in the branch populations.

  • We must measure the same thing for each client, to minimize the variance associated with our measurement.

So that our analyses follow these abstract principles, we follow these rules:

  • Start with a list of all clients who enrolled.

  • We can filter this list of clients only based on information known to us at or before the time that they enrolled, because later information might be causally connected to the intervention.

  • For any given metric, every client gets a non-null value; we don’t implicitly ignore anyone, even if they churned and stopped sending data.

  • Typically if an enrolled client no longer qualifies for enrollment, we’ll still want to include their data in the analysis, unless we’re explicitly using stats methods that handle censored data.

  • We define a “analysis window” with respect to clients’ enrollment dates. Each metric only uses data collected inside this analysis window. We can only analyze data for a client if we have data covering their entire analysis window.

Example usage (in a colab notebook):

from google.colab import auth
auth.authenticate_user()
print('Authenticated')

from mozanalysis.experiment import Experiment
from mozanalysis.bq import BigQueryContext
from mozanalysis.metrics.desktop import active_hours, uri_count

bq_context = BigQueryContext(
    dataset_id='your-dataset-id',  # e.g. mine's flawrence
    project_id='moz-fx-data-bq-data-science'  # this is the default anyway
)

experiment = Experiment(
    experiment_slug='pref-fingerprinting-protections-retention-study-release-70',
    start_date='2019-10-29',
    num_dates_enrollment=8
)

# Run the query and get the results as a DataFrame
res = experiment.get_single_window_data(
    bq_context,
    [
        active_hours,
        uri_count
    ],
    last_date_full_data='2019-12-01',
    analysis_start_days=0,
    analysis_length_days=7
)
Parameters:

  • experiment_slug (str) – Name of the study, used to identify the enrollment events specific to this study.

  • start_date (str) – e.g. ‘2019-01-01’. First date on which enrollment events were received.

  • num_dates_enrollment (int, optional) – Only include this many dates of enrollments. If None then use the maximum number of dates as determined by the metric’s analysis window and last_date_full_data. Typically 7n+1, e.g. 8. The factor ‘7’ removes weekly seasonality, and the +1 accounts for the fact that enrollment typically starts a few hours before UTC midnight.

  • app_id (str, optional) – For a Glean app, the name of the BigQuery dataset derived from its app ID, like org_mozilla_firefox.

  • app_name (str, optional) – The Glean app name, like fenix.

experiment_slug

Name of the study, used to identify the enrollment events specific to this study.

Type:

str

start_date

e.g. ‘2019-01-01’. First date on which enrollment events were received.

Type:

str

num_dates_enrollment

Only include this many days of enrollments. If None then use the maximum number of days as determined by the metric’s analysis window and last_date_full_data. Typically 7n+1, e.g. 8. The factor ‘7’ removes weekly seasonality, and the +1 accounts for the fact that enrollment typically starts a few hours before UTC midnight.

Type:

int, optional

get_app_name()[source]

Determine the correct app name.

If no explicit app name has been passed into Experiment, lookup app name from a pre-defined list. (this is deprecated)

get_single_window_data(bq_context: BigQueryContext, metric_list: list, last_date_full_data: str, analysis_start_days: int, analysis_length_days: int, enrollments_query_type: EnrollmentsQueryType = EnrollmentsQueryType.NORMANDY, custom_enrollments_query: str | None = None, custom_exposure_query: str | None = None, exposure_signal=None, segment_list=None) → DataFrame[source]

Return a DataFrame containing per-client metric values.

Also store them in a permanent table in BigQuery. The name of this table will be printed. Subsequent calls to this function will simply read the results from this table.

Parameters:

  • bq_context (BigQueryContext) – BigQuery configuration and client.

  • metric_list (list of mozanalysis.metric.Metric or str) – The metrics to analyze.

  • last_date_full_data (str) – The most recent date for which we have complete data, e.g. ‘2019-03-22’. If you want to ignore all data collected after a certain date (e.g. when the experiment recipe was deactivated), then do that here.

  • analysis_start_days (int) – the start of the analysis window, measured in ‘days since the client enrolled’. We ignore data collected outside this analysis window.

  • analysis_length_days (int) – the length of the analysis window, measured in days.

  • enrollments_query_type (EnrollmentsQueryType) – (‘normandy’, ‘glean-event’, ‘cirrus’, or ‘fenix-fallback’) Specifies the query type to use to get the experiment’s enrollments, unless overridden by custom_enrollments_query.

  • custom_enrollments_query (str) –

    A full SQL query that will generate the enrollments common table expression used in the main query. The query must produce the columns client_id, branch, enrollment_date, and num_enrolled_events.

    WARNING: this query’s results must be uniquely keyed by (client_id, branch), or else your results will be subtly wrong.

  • custom_exposure_query (str) –

    A full SQL query that will generate the exposures common table expression used in the main query. The query must produce the columns client_id, branch, enrollment_date, and num_exposure_events.

    If not provided, the exposure will be determined based on exposure_signal, if provided, or Normandy and Nimbus exposure events. custom_exposure_query takes precedence over exposure_signal.

  • exposure_signal (ExposureSignal) – Optional signal definition of when a client has been exposed to the experiment. If not provided, the exposure will be determined based on Normandy exposure events for desktop and Nimbus exposure events for Fenix and iOS.

  • segment_list (list of mozanalysis.segment.Segment or str) – The user segments to study.

Returns:

A pandas DataFrame of experiment data. One row per client_id. Some metadata columns, then one column per metric in metric_list, and one column per sanity-check metric. Columns (not necessarily in order):

  • client_id (str): Not necessary for “happy path” analyses.

  • branch (str): The client’s branch

  • other columns of enrollments.

  • [metric 1]: The client’s value for the first metric in metric_list.

  • …

  • [metric n]: The client’s value for the nth (final) metric in metric_list.

  • [sanity check 1]: The client’s value for the first sanity check metric for the first data source that supports sanity checks.

  • …

  • [sanity check n]: The client’s value for the last sanity check metric for the last data source that supports sanity checks.

This format - the schema plus there being one row per enrolled client, regardless of whether the client has data in data_source - was agreed upon by the DS team, and is the standard format for queried experimental data.

get_time_series_data(bq_context: BigQueryContext, metric_list: list, last_date_full_data: str, time_series_period: str = 'weekly', enrollments_query_type: EnrollmentsQueryType = EnrollmentsQueryType.NORMANDY, custom_enrollments_query: str | None = None, custom_exposure_query: str | None = None, exposure_signal=None, segment_list=None) → TimeSeriesResult[source]

Return a TimeSeriesResult with per-client metric values.

Roughly equivalent to looping over get_single_window_data() with different analysis windows, and reorganising the results.

Parameters:

  • bq_context (BigQueryContext) – BigQuery configuration and client.

  • metric_list (list of mozanalysis.metric.Metric) – The metrics to analyze.

  • last_date_full_data (str) – The most recent date for which we have complete data, e.g. ‘2019-03-22’. If you want to ignore all data collected after a certain date (e.g. when the experiment recipe was deactivated), then do that here.

  • time_series_period ('daily' or 'weekly') – How long each analysis window should be.

  • enrollments_query_type (EnrollmentsQueryType) – (‘normandy’, ‘glean-event’, ‘cirrus’, or ‘fenix-fallback’) Specifies the query type to use to get the experiment’s enrollments, unless overridden by custom_enrollments_query.

  • custom_enrollments_query (str) –

    A full SQL query that will generate the enrollments common table expression used in the main query. The query must produce the columns client_id, branch, enrollment_date, and num_enrolled_events.

    WARNING: this query’s results must be uniquely keyed by (client_id, branch), or else your results will be subtly wrong.

  • custom_exposure_query (str) –

    A full SQL query that will generate the exposures common table expression used in the main query. The query must produce the columns client_id, branch, enrollment_date, and num_exposure_events.

    If not provided, the exposure will be determined based on exposure_signal, if provided, or Normandy and Nimbus exposure events. custom_exposure_query takes precedence over exposure_signal.

  • exposure_signal (ExposureSignal) – Optional signal definition of when a client has been exposed to the experiment. If not provided, the exposure will be determined based on Normandy exposure events for desktop and Nimbus exposure events for Fenix and iOS.

  • segment_list (list of mozanalysis.segment.Segment) – The user segments to study.

Returns:

A mozanalysis.experiment.TimeSeriesResult object, which may be used to obtain a pandas DataFrame of per-client metric data, for each analysis window. Each DataFrame is a pandas DataFrame in “the standard format”: one row per client, some metadata columns, plus one column per metric and sanity-check metric. Its columns (not necessarily in order):

  • branch (str): The client’s branch

  • other columns of enrollments.

  • [metric 1]: The client’s value for the first metric in metric_list.

  • …

  • [metric n]: The client’s value for the nth (final) metric in metric_list.

  • [sanity check 1]: The client’s value for the first sanity check metric for the first data source that supports sanity checks.

  • …

  • [sanity check n]: The client’s value for the last sanity check metric for the last data source that supports sanity checks.

build_enrollments_query(time_limits: TimeLimits, enrollments_query_type: EnrollmentsQueryType = EnrollmentsQueryType.NORMANDY, custom_enrollments_query: str | None = None, custom_exposure_query: str | None = None, exposure_signal=None, segment_list=None, sample_size: int = 100) → str[source]

Return a SQL query for querying enrollment and exposure data.

Parameters:

  • time_limits (TimeLimits) – An object describing the interval(s) to query

  • enrollments_query_type (EnrollmentsQueryType) – (‘normandy’, ‘glean-event’, ‘cirrus’, or ‘fenix-fallback’) Specifies the query type to use to get the experiment’s enrollments, unless overridden by custom_enrollments_query.

  • custom_enrollments_query (str) –

    A full SQL query that will generate the enrollments common table expression used in the main query. The query must produce the columns client_id, branch, enrollment_date, and num_enrolled_events.

    WARNING: this query’s results must be uniquely keyed by (client_id, branch), or else your results will be subtly wrong.

  • custom_exposure_query (str) – A full SQL query that will generate the exposures common table expression used in the main query. The query must produce the columns client_id, branch, enrollment_date, and num_exposure_events.

  • exposure_signal (ExposureSignal) – Optional signal definition of when a client has been exposed to the experiment

  • segment_list (list of mozanalysis.segment.Segment or str) – The user segments to study.

  • sample_size (int) – Optional integer percentage of clients, used for downsampling enrollments. Default 100.

Returns:

A string containing a BigQuery SQL expression.

build_metrics_query(metric_list: list, time_limits: TimeLimits, enrollments_table: str, analysis_basis=AnalysisBasis.ENROLLMENTS, exposure_signal=None) → str[source]

Return a SQL query for querying metric data.

For interactive use, prefer get_time_series_data() or get_single_window_data(), according to your use case, which will run the query for you and return a materialized dataframe.

The optional exposure_signal parameter allows to check if clients have received the exposure signal during enrollment or after. When using the exposures analysis basis, metrics will be computed for these clients.

Parameters:

  • metric_list (list of mozanalysis.metric.Metric or str) – The metrics to analyze.

  • time_limits (TimeLimits) – An object describing the interval(s) to query

  • enrollments_table (str) – The name of the enrollments table

  • basis (AnalysisBasis) – Use exposures as basis for calculating metrics if True, otherwise use enrollments.

  • exposure_signal (Optional[ExposureSignal]) – Optional exposure signal parameter that will be used for computing metrics for certain analysis bases (such as exposures).

Returns:

A string containing a BigQuery SQL expression.

Building this query is the main goal of this module.

class mozanalysis.experiment.TimeLimits(first_enrollment_date: str, last_enrollment_date: str, first_date_data_required: str, last_date_data_required: str, analysis_windows)[source]

Expresses time limits for different kinds of analysis windows.

Instantiated and used by the Experiment class; end users should not need to interact with it.

Do not directly instantiate: use the constructors provided.

There are several time constraints needed to specify a valid query for experiment data:

  • When did enrollments start?

  • When did enrollments stop?

  • How long after enrollment does the analysis window start?

  • How long is the analysis window?

Even if these four quantities are specified directly, it is important to check that they are consistent with the available data - i.e. that we have data for the entire analysis window for every enrollment.

Furthermore, there are some extra quantities that are useful for writing efficient queries:

  • What is the first date for which we need data from our data source?

  • What is the last date for which we need data from our data source?

Instances of this class store all these quantities and do validation to make sure that they’re consistent. The “store lots of overlapping state and validate” strategy was chosen over “store minimal state and compute on the fly” because different state is supplied in different contexts.

classmethod for_single_analysis_window(first_enrollment_date: str, last_date_full_data: str, analysis_start_days: int, analysis_length_dates: int, num_dates_enrollment: int | None = None) → TimeLimits[source]

Return a TimeLimits instance with the following parameters

Parameters:

  • first_enrollment_date (str) – First date on which enrollment events were received; the start date of the experiment.

  • last_date_full_data (str) – The most recent date for which we have complete data, e.g. ‘2019-03-22’. If you want to ignore all data collected after a certain date (e.g. when the experiment recipe was deactivated), then do that here.

  • analysis_start_days (int) – the start of the analysis window, measured in ‘days since the client enrolled’. We ignore data collected outside this analysis window.

  • analysis_length_days (int) – the length of the analysis window, measured in days.

  • num_dates_enrollment (int, optional) – Only include this many days of enrollments. If None then use the maximum number of days as determined by the metric’s analysis window and last_date_full_data. Typically 7n+1, e.g. 8. The factor 7 removes weekly seasonality, and the +1 accounts for the fact that enrollment typically starts a few hours before UTC midnight.

classmethod for_ts(first_enrollment_date: str, last_date_full_data: str, time_series_period: str, num_dates_enrollment: int) → TimeLimits[source]

Return a TimeLimits instance for a time series.

Parameters:

  • first_enrollment_date (str) – First date on which enrollment events were received; the start date of the experiment.

  • last_date_full_data (str) – The most recent date for which we have complete data, e.g. ‘2019-03-22’. If you want to ignore all data collected after a certain date (e.g. when the experiment recipe was deactivated), then do that here.

  • time_series_period – ‘daily’ or ‘weekly’.

  • num_dates_enrollment (int) – Take this many days of client enrollments. This is a mandatory argument because it determines the number of points in the time series.

class mozanalysis.experiment.AnalysisWindow(start: int, end: int)[source]

Represents the range of days in which to measure a metric.

The range is measured in “days relative enrollment”, and is inclusive.

For example, AnalysisWindow(0, 6) is the first week after enrollment and AnalysisWindow(-8,-1) is the week before enrollment

Parameters:

  • start (int) – First day of the analysis window, in days relative to enrollment start. 0 indicates the date of enrollment. Positive numbers are after enrollment, negative are before. Must be the same sign as end (zero counts as positive)

  • end (int) – Final day of the analysis window, in days relative to enrollment start. 0 indicates the date of enrollment. Positive numbers are after enrollment, negative are before. Must be the same sign as start (zero counts as positive).

class mozanalysis.experiment.TimeSeriesResult(fully_qualified_table_name: str, analysis_windows: list)[source]

Result from a time series query.

For each analysis window, this object lets us get a dataframe in “the standard format” (one row per client).

Example usage:

result_dict = dict(time_series_result.items(bq_context))
window_0 = result_dict[0]

window_0 would then be a pandas DataFrame of results for the analysis window starting at day 0. result_dict would be a dictionary of all such DataFrames, keyed by the start days of their analysis windows.

Or, to load only one analysis window into RAM:

window_0 = time_series_result.get(bq_context, 0)
get(bq_context: BigQueryContext, analysis_window) → DataFrame[source]

Get the DataFrame for a specific analysis window.

N.B. this makes a BigQuery query each time it is run; caching results is your responsibility.

Parameters:
get_full_data(bq_context: BigQueryContext) → DataFrame[source]

Get the full DataFrame from TimeSeriesResult.

This DataFrame has a row for each client for each period of the time series and may be very large. A warning will print the size of data to be downloaded.

Parameters:

bq_context (BigQueryContext) –

get_aggregated_data(bq_context: BigQueryContext, metric_list: list, aggregate_function: str = 'AVG') → tuple[DataFrame, int][source]

Results from a time series query, aggregated over analysis windows by a SQL aggregate function.

This DataFrame has a row for each analysis window, with a column for each metric in the supplied metric_list.

Parameters: