Statistics API

Paired comparison of evaluations produced by compresso_recsys.evaluation. For an introduction, how to read the output and how to report it, see Comparing Models Statistically; this page is the reference.

Every function here works on the paired per-user difference between two models evaluated on identical users, and requires results carrying per-user values. After repeated sample_ids are grouped, at least two independent users must remain; repeated evaluation draws from one user do not provide a variance estimate.

Comparison Functions

compresso_recsys.stats.compare_models(results, *, metrics, reference=None, confidence_level=0.95, n_resamples=9999, alternative='two-sided', correction='holm', test_method='randomization', random_state=0, resample_batch_size=64, show_progress=False)[source]

Compare several models across one or more metrics in a single family.

With reference set, every other model is compared against it. Without it, every unordered pair is compared in mapping insertion order, with the earlier model as baseline.

The correction spans every pair and metric produced by the call, so calling this once with three metrics is not the same as calling it three times: the family is what the call generates.

show_progress draws a bar when tqdm is installed. Cost is linear in units and in the number of hypotheses – roughly a second per hypothesis per 25,000 units at the default resample count – so a large evaluation compared across several metrics and models can run for minutes. The bar counts hypotheses but advances within each one, so it still moves when a call produces only a single very slow comparison.

Holm is the default because these hypotheses are dependent: they are computed over overlapping users, and several metrics on one pair of models measure closely related things. Holm controls the family-wise error rate under arbitrary dependence. Procedures that assume independence or positive dependence are not offered for that reason.

Each hypothesis draws its own resamples, seeded from its metric and its pair of model names. Adding a metric, reordering metrics, or reordering results therefore cannot change a raw comparison that was already in the report. Adjusted p-values still move, because the family changed.

Return type:

ComparisonReport

Parameters:

  • results (Mapping[str, EvaluationResult])

  • metrics (str | Sequence[str])

  • reference (str | None)

  • confidence_level (float)

  • n_resamples (int)

  • alternative (Literal['two-sided', 'greater', 'less'])

  • correction (Literal['holm', 'bonferroni'] | None)

  • test_method (Literal['randomization', 'bootstrap', 't'])

  • random_state (int | None)

  • resample_batch_size (int)

  • show_progress (bool)

compresso_recsys.stats.compare_pair(baseline, candidate, *, metric, baseline_name='baseline', candidate_name='candidate', confidence_level=0.95, n_resamples=9999, alternative='two-sided', test_method='randomization', random_state=0, resample_batch_size=64, show_progress=False)[source]

Compare one candidate against one baseline on one metric.

The difference is candidate - baseline, so positive values favour the candidate. No multiplicity correction is applied: a single comparison is a single hypothesis, and adjusted_p_value equals p_value. Use compare_models() when testing more than one hypothesis together.

Return type:

PairwiseComparison

Parameters:

  • baseline (EvaluationResult)

  • candidate (EvaluationResult)

  • metric (str)

  • baseline_name (str)

  • candidate_name (str)

  • confidence_level (float)

  • n_resamples (int)

  • alternative (Literal['two-sided', 'greater', 'less'])

  • test_method (Literal['randomization', 'bootstrap', 't'])

  • random_state (int | None)

  • resample_batch_size (int)

  • show_progress (bool)

Results

class compresso_recsys.stats.PairwiseComparison(metric, baseline, candidate, n_samples, n_units, n_nonzero, baseline_mean, candidate_mean, difference, relative_difference, bootstrap_standard_error, ci_low, ci_high, confidence_level, p_value, adjusted_p_value, significant, alternative, test_method, interval_method, n_resamples, random_state)[source]

One model-versus-model hypothesis for one metric.

difference is always candidate - baseline, so positive values favour the candidate.

Parameters:

  • metric (str)

  • baseline (str)

  • candidate (str)

  • n_samples (int)

  • n_units (int)

  • n_nonzero (int)

  • baseline_mean (float)

  • candidate_mean (float)

  • difference (float)

  • relative_difference (float | None)

  • bootstrap_standard_error (float)

  • ci_low (float)

  • ci_high (float)

  • confidence_level (float)

  • p_value (float)

  • adjusted_p_value (float)

  • significant (bool)

  • alternative (Literal['two-sided', 'greater', 'less'])

  • test_method (Literal['randomization', 'bootstrap', 't'])

  • interval_method (str)

  • n_resamples (int)

  • random_state (int | None)

property tie_rate: float

Fraction of units whose mean paired difference is exactly zero.

With one row per user that means the two models scored them identically. With several, it means those rows cancelled: a user who gained on one draw and lost the same amount on another is tied here even though no single row was.

High tie rates are normal for ranking metrics at small cutoffs and are not a defect. They say the two models come out level for that share of the population, which is itself a finding, and they are why n_nonzero is reported: it bounds how discrete the randomization test’s null distribution can be, since flipping the sign of a zero changes nothing. The estimate still rests on every unit.

property direction: str

'better', 'worse' or 'inconclusive'.

to_dict()[source]

Return this comparison as a flat dictionary, including direction.

Return type:

dict[str, Any]

class compresso_recsys.stats.ComparisonReport(comparisons, metrics, model_names, reference, correction, confidence_level, alternative, test_method, n_resamples, random_state)[source]

Every hypothesis produced by one compare_models() call.

The multiple-testing correction applies across the whole report, so a report is the unit of analysis rather than any single comparison in it.

Parameters:

  • comparisons (tuple[PairwiseComparison, ...])

  • metrics (tuple[str, ...])

  • model_names (tuple[str, ...])

  • reference (str | None)

  • correction (Literal['holm', 'bonferroni'] | None)

  • confidence_level (float)

  • alternative (Literal['two-sided', 'greater', 'less'])

  • test_method (Literal['randomization', 'bootstrap', 't'])

  • n_resamples (int)

  • random_state (int | None)

to_frame()[source]

Return one row per hypothesis with a fixed column order.

Return type:

DataFrame

Parameter Values

alternative

"two-sided", "greater" or "less". Selects the question being asked and the orientation of the reported interval.

correction

"holm", "bonferroni" or None. Applied across every pair and metric produced by one compare_models() call.

test_method

"randomization" for the paired sign-flip test, or "bootstrap" for the null-centred bootstrap.

n_resamples

Number of resampling replicates. The smallest achievable p-value is 1 / (n_resamples + 1); with J hypotheses the smallest achievable adjusted p-value is J / (n_resamples + 1).

confidence_level

Interval level, and through alpha = 1 - confidence_level the significance threshold as well.