presc.evaluations package
presc.evaluations.conditional_distribution module
- class presc.evaluations.conditional_distribution.ConditionalDistribution(model, test_dataset, settings=None, config=None)[source]
Bases:
objectComputation of data distributions conditional on prediction results.
- model
The ClassificationModel to run the evaluation for.
- test_dataset
A Dataset to use for evaluation.
- settings
An optional dict specifying option values under evaluations.conditional_distribution, eg. {“computation.binning”: 5}, These are restricted to the class instance and do not change the global config.
- Type
dict
- config
An optional PrescConfig instance to read options from. This will be overridden by settings values.
- class presc.evaluations.conditional_distribution.ConditionalDistributionResult(vals, bins, categorical, binning, common_bins)[source]
Bases:
objectResult of the conditional distribution computation for a single column of data.
- vals
A Series listing the bin counts for each group, indexed by (<true_label>, <predicted_label>, <bin_label>).
- Type
Series
- bins
A Series listing the bin endpoints. If common_bins is False, this should be indexed by (<true_label>, <predicted_label>) and list the endpoints for each group. If the data was treated as numeric, this will have length len(vals)+1 (within each group), otherwise len(vals).
- Type
Series
- categorical
Was the feature treated as categorical?
- Type
bool
- binning
The binning scheme used
- Type
str
- common_bins
Were common bins used across all groups?
- Type
bool
- presc.evaluations.conditional_distribution.compute_conditional_distribution(data_col, true_labs, pred_labs, as_categorical=False, binning='fd', common_bins=True)[source]
Compute a distributional summary.
The metric is computed within unique values of the grouping column (categorical) or within bins partitioning its range (continuous).
- Parameters
data_col – A column of data from a test dataset.
true_labs (Series) – A series of true labels for the test dataset.
pred_labs (Series) – A series of labels predicted by a model for the test dataset.
as_categorical (bool) – Should the data column be treated as categorical, ie. binned on its unique values? If it is not numeric, this param is ignored.
binning (str) – Binning scheme to use for a numerical column, passed to numpy.histogram. Can be a fixed number of bins or a string indicating a binning scheme
common_bins (bool) – Should the bins be computed over the entire column and shared across groups (True) or computed within each group (False)
- Return type
presc.evaluations.conditional_metric module
- class presc.evaluations.conditional_metric.ConditionalMetric(model, test_dataset, settings=None, config=None)[source]
Bases:
objectComputation of confusion-based metrics across subsets of a test dataset.
- model
The ClassificationModel to run the evaluation for.
- test_dataset
A Dataset to use for evaluation.
- settings
An optional dict specifying option values under evaluations.conditional_metric, eg. {“computation.num_bins”: 5} These are restricted to the class instance and do not change the global config.
- Type
dict
- config
An optional PrescConfig instance to read options from. This will be overridden by settings values.
- compute_for_column(colname, metric, **kwargs)[source]
Compute the evaluation for the given dataset column.
The metric is computed within unique values of the specified column (if categorical) or within bins partitioning its range (if continuous).
- colnamestr
A column in the dataset to partition on.
- metricfunction
The evaluation metric to compute across the partitions. This should be a function f(y_true, y_pred) which accepts Series of true and predicted labels.
- kwargs :
On-the-fly overrides to the config option values for the computation.
- Return type
- display(colnames=None)[source]
Computes and displays the conditional metric result for each specified column.
- Parameters
colnames (list of str) – A list of column names to run the evaluation over, creating a plot for each. If not supplied, defaults to columns specifed in the config.
metric_name (str) – Display name identifying the metric to show on the plot
- class presc.evaluations.conditional_metric.ConditionalMetricResult(vals, bins, categorical, num_bins, quantile)[source]
Bases:
objectResult of the conditional metric evaluation for a single grouping.
- vals
A Series listing the computation result for each bin.
- Type
Series
- bins
A Series listing the bin endpoints. If the feature was treated as numeric, this will have length len(vals)+1, otherwise len(vals).
- Type
Series
- categorical
Aas the feature treated as categorical?
- Type
bool
- num_bins
Number of bins used for grouping.
- Type
int
- quantile
Was grouping quantile-based?
- Type
bool
- presc.evaluations.conditional_metric.compute_conditional_metric(grouping_col, true_labs, pred_labs, metric, as_categorical=False, num_bins=10, quantile=False)[source]
Compute metric values conditional on the grouping column.
The metric is computed within unique values of the grouping column (categorical) or within bins partitioning its range (continuous).
- Parameters
grouping_col (Series) – Series defining a grouping for the metric computation.
true_labs (Series) – Series of true labels for a test dataset.
pred_labs (Series) – Series of labels predicted by a model for a test dataset.
metric (function) – The evaluation metric to compute across the groupings. This should be a function f(y_true, y_pred) which accepts Series of true and predicted labels.
as_categorical (bool) – Should the grouping column be treated as categorical, ie. binned on its unique values? If it is not numeric, this param is ignored.
num_bins (int) – Number of bins to use for grouping a numeric column.
quantile (bool) – Should the bin widths correspond to quantiles of a numerical column’s distribution (True) or be equally-spaced over its range (False).
- Return type
presc.evaluations.spatial_distribution module
- class presc.evaluations.spatial_distribution.SpatialDistribution(model, test_dataset, train_dataset, settings=None, config=None)[source]
Bases:
objectComputation of distributions of data in feature space.
- model
The ClassificationModel to run the evaluation for.
- test_dataset
A Dataset to use for evaluation.
- train_dataset
A Dataset to use as the baseline for distance measures (eg. the training data).
- settings
An optional dict specifying option values under evaluations.spatial_distribution, eg. {“summary_agg”: “median”} These are restricted to the class instance and do not change the global config.
- Type
dict
- config
An optional PrescConfig instance to read options from. This will be overridden by settings values.
- class presc.evaluations.spatial_distribution.SpatialDistributionResult(vals, dist_metrics_num, dist_metrics_categ, summary)[source]
Bases:
objectResult of the spatial distribution computation.
- vals
A DataFrame listing the summary values for each test datapoint, indexed by (<true_label>, <predicted_label>, <datapoint_index>).
- Type
DataFrame
- dist_metrics_num
Mapping of numerical column names to the metric that was used.
- Type
dict
- dist_metrics_categ
Mapping of categorical column names to the metric that was used.
- Type
dict
- summary
The summary used to aggregate distances for each class
- Type
str
- presc.evaluations.spatial_distribution.compute_spatial_distribution(test_features, test_labs_true, test_labs_pred, base_features, base_labs, numerical_dist_metric=None, categorical_dist_metric=None, summary='mean')[source]
Compute a summary of the pairwise distances between points.
This computes pairwise distances between the test points and base points in feature space (ie. how far each test point is from each base point), and returns a summary of the distance for each test point relative to each base class.
- Parameters
test_features (DataFrame) – Feature values for the test dataset.
test_labs_true (Series) – True labels for the test dataset.
test_labs_pred (Series) – Labels predicted by a model for the test dataset.
base_features (DataFrame) – Feature values for the base dataset.
base_labs (Series) – True labels for the base dataset.
numerical_dist_metric (dict) – The metrics to use to measure distance between numerical (continuous)-valued columns. This should be a dict mapping column names to strings, each a named metric as accepted by sklearn.metrics.pairwise_distances appropriate for continuous data
categorical_dist_metric (dict) – The metrics to use to measure distance between categorical (discrete)-valued columns. This should be a dict mapping column names to strings, each a named metric as accepted by sklearn.metrics.pairwise_distances appropriate for discrete data
summary (str) – An aggregation function to apply to a Pandas Grouped object.
the (Only columns listed in the distance metric dists will be included in) –
computation. (distance) –
- Return type
presc.evaluations.train_test_splits module
- class presc.evaluations.train_test_splits.TrainTestSplits(model, train_dataset, settings=None, config=None)[source]
Bases:
objectSimulation of performance across different train-test split ratios.
- model
The ClassificationModel to run the evaluation for.
- train_dataset
A Dataset to use for evaluation.
- settings
An optional dict specifying option values under evaluations.conditional_distribution, eg. {“computation.num_replicates”: 5}, These are restricted to the class instance and do not change the global config.
- Type
dict
- config
An optional PrescConfig instance to read options from. This will be overridden by settings values.
- compute(metric, **kwargs)[source]
Compute the evaluation for the given dataset column.
- Parameters
metric (str) – The evaluation metric to compute for each split. This should be the name of a sklearn.metrics scorer function.
kwargs – On-the-fly overrides to the config option values for the computation.
- Return type
- class presc.evaluations.train_test_splits.TrainTestSplitsResult(vals, metric, split_size_increment, num_replicates, random_state)[source]
Bases:
objectResult of the train-test splits evaluation.
- vals
A DataFrame listing the performance score results for each split size. Rows are indexed by split size, with replicated results across columns.
- Type
DataFrame
- metric
The evaluation metric used to score performance.
- Type
str
- split_size_increment
Increment between linearly-space train-test split ratios.
- Type
float
- num_replicates
Number of randomized replicates for each split.
- Type
int
- random_state
Random state setting.
- Type
int
- presc.evaluations.train_test_splits.compute_train_test_splits(dataset, classifier, metric, split_size_increment=0.1, num_replicates=10, random_state=543)[source]
Evaluate classifier performance across different train-test split ratios.
This traverses a grid of train-test splits, retraining the model on each training subset and computing a performance metric on the corresponding test set.
- Parameters
dataset (presc.dataset.Dataset) – The Dataset to use for the evaluation.
classifier (sklearn Classifier) – The classifier to evaluate.
metric (str) – The evaluation metric to compute for each split. This should be the name of a sklearn.metrics scorer function.
split_size_increment (float) – Increment between linearly-space train-test split ratios.
num_replicates (int) – Number of randomized replicates to run for each split.
random_state (int) – Set the random state for reproducibility.
- Return type
presc.evaluations.utils module
- presc.evaluations.utils.get_bins(s, num_bins, quantile=False)[source]
Split a Series into discrete bins.
- Parameters
s (pandas Series) –
num_bins (int) – The number of bins to split the range of s into.
quantile (bool) – If True, bin edges will correspond to quantiles for equally-spaced probabilities. Otherwise, bins are equally spaced on the original scale.
- Returns
Series of the same length as s indicating the bin for each value, as well as an array of bin edges of length num_bins+1.
- Return type
Series