mozanalysis.bayesian_stats

mozanalysis.bayesian_stats.compare_samples(samples, ref_branch_label, individual_summary_quantiles=(0.005, 0.025, 0.5, 0.975, 0.995), comparative_summary_quantiles=(0.005, 0.025, 0.5, 0.975, 0.995))

Return descriptive statistics for branch stats and uplifts.

Given per-branch samples for some quantity, return summary statistics (percentiles and the mean) for the quantity for each individual branch. Also return comparative summary statistics for the uplift of this quantity for each branch with respect to the reference branch.

Args:

samples (dict of pandas.Series or pandas.DataFrame): Each key

is the label for a branch. Each value is the corresponding sample set.

ref_branch_label (str): Label for the reference branch

(typically the control).

individual_summary_quantiles (list of float): Quantiles that

define the summary stats for the individual branches’ samples.

comparative_summary_quantiles (list of float): Quantiles that

define the summary stats for the comparative stats.

Returns:

A dictionary. When the values of samples are Series, then this function returns a dictionary with the following keys and values:

‘individual’: dictionary mapping each branch name to a

pandas Series that holds the per-branch sample means and quantiles.

‘comparative’: dictionary mapping each branch name to a

pandas Series of summary statistics for the possible uplifts of the sampled quantity relative to the reference branch.

Otherwise, when the values of samples are DataFrames, then this function returns a similar dictionary, except the Series are replaced with DataFrames. The index for each DataFrame is the columns of a value of samples.

mozanalysis.bayesian_stats.summarize_one_branch_samples(samples, quantiles=(0.005, 0.025, 0.5, 0.975, 0.995))

Return descriptive statistics for sampled population-level stats.

Given samples from one or more distributions, calculate some quantiles and the mean.

The intended primary use-case is for calculating credible intervals for stats when bootstrapping, or credible intervals around Bayesian model parameters; in both cases samples are from a posterior concerning one branch of an experiment.

Args:

samples (pandas.Series or pandas.DataFrame): Samples over which

to compute the mean and quantiles.

quantiles (list, optional): The quantiles to compute - a good

reason to override the defaults would be when Bonferroni corrections are required.

Returns:

If samples is a Series, then returns a pandas Series; the index contains the stringified quantiles plus 'mean'.

If samples is a DataFrame, then returns a pandas DataFrame; the columns contain the stringified quantiles plus 'mean'. The index matches the columns of samples.

mozanalysis.bayesian_stats.summarize_joint_samples(focus, reference, quantiles=(0.005, 0.025, 0.5, 0.975, 0.995))

Return descriptive statistics for uplifts.

The intended use case of this function is to compare a ‘focus’ experiment branch to a ‘reference’ experiment branch (e.g. the control). Samples from each branch are combined pairwise; these pairs are considered to be samples from the joint probability distribution (JPD). We compute various quantities from the JPD:

• We compute summary statistics for the distribution over relative uplifts focus / reference - 1

• We compute summary statistics for the distribution over absolute uplifts focus - reference

• We compute a summary statistic for the distribution over the L1 norm of absolute uplifts abs(focus - reference)

• We compute the fraction of probability mass in the region focus > reference, which in a Bayesian context may be interpreted as the probability that the ground truth model parameter is larger for the focus branch than the reference branch.

focus and reference are samples from distributions; each is the same format that would be supplied to summarize_one_branch_samples when analyzing the branches independently.

Can be used to analyse a single metric (supply Series as arguments) or in batch mode (supply DataFrames as arguments).

Args:

focus (pandas.Series or pandas.DataFrame): Bootstrapped samples

or samples of a model parameter for a branch of an experiment. If a DataFrame, each column represents a different quantity.

reference (pandas.Series or pandas.DataFrame): The same

quantity, calculated for a different branch (typically the control).

quantiles (list, optional): The quantiles to compute - a good

reason to override the defaults would be when Bonferroni corrections are required.

Returns:

A pandas Series or DataFrame containing a MultiIndex with the following labels on the higher level and stringified floats on the inner level

• rel_uplift: Expectation value and quantiles over the relative uplift.

• abs_uplift: Expectation value and quantiles over the absolute uplift.

• max_abs_diff: Quantile 0.95 on the L1 norm of differences/ absolute uplifts. In a Bayesian context, there is a 95% probability that the absolute difference is less than this in either direction.

• prob_win: In a Bayesian context, the probability that the ground truth model parameter is larger for the focus than the reference branch.

If returning a DataFrame, this MultiIndex is for the columns, and the index matches the columns of focus.