Ranking Evaluation API

Predictions and Targets

Ranked predictions use compresso.SRPTensor: each row stores exactly the top K item indices and their scores in descending score order. Targets use a SciPy CSR matrix so each row can contain a different number of relevant items. Target values are treated as binary relevance; only nonzero locations matter.

The evaluator creates one boolean hit tensor of shape (batch_size, K) and shares it across all configured metrics. It does not densify model scores or the complete target matrix. Rows without target items are excluded from metric means and from n_scored_rows.

n_scored_rows counts rows, not users. They differ whenever a protocol gives one user several rows – eval_draws above 1 – and n_units reports the distinct identifiers behind them. Paired comparison resamples units rather than rows, so it is n_units that bounds how much independent evidence an evaluation carries.

Prediction validation is enabled by default for both evaluation entry points. It checks item bounds, duplicate recommendations, NaN scores, and score order.

Evaluation Results

Both entry points return an EvaluationResult rather than a plain dictionary. It behaves as a mapping over the aggregate metrics plus n_scored_rows, so result["ndcg@20"], iteration and dict(result) continue to work, and to_dict() is available where an actual dict is required.

Beyond the aggregates it carries the per-user value behind every metric, together with the sample_ids that identify the rows those values came from:

result = evaluate_recommender(
    model, source=source, targets=targets,
    metrics=[NDCG(20)], sample_ids=user_ids,
)

result["ndcg@20"]            # aggregate, as before
result.per_user["ndcg@20"]   # one float32 per evaluable row
result.sample_ids            # aligned identifiers
result.n_rows                # rows supplied
result.n_scored_rows         # rows with at least one target
result.n_units               # distinct identifiers among those rows

Those per-user values are what make paired statistical comparison possible; see Comparing Models Statistically. Retaining them costs roughly 4 * n_users * n_metrics bytes, small beside model parameters, so collection is enabled by default. Pass collect_per_user=False for deployment-style monitoring that only needs aggregates, at the cost of being unable to compare the result against another.

Identifiers default to global input row indices. Supply sample_ids when the rows have stable identities of their own, so that comparison across evaluations fails loudly rather than silently pairing different users.

class compresso_recsys.evaluation.EvaluationResult(metrics, per_user, sample_ids, n_rows, n_scored_rows, required_k, metadata=<factory>, debug_rows=None, target_fingerprint=None)[source]

Aggregate metrics plus the per-user observations behind them.

The mapping view carries the aggregates and n_scored_rows, so existing code that treats an evaluation as a dictionary keeps working:

result["ndcg@20"]
dict(result)

Per-user values, sample identifiers and metadata are attributes rather than mapping keys, because they are large and because a caller reaching for them is doing something other than reading a headline number.

per_user and sample_ids are what make paired statistical comparison possible: two evaluations can only be compared when they refer to the same evaluation units in the same order.

Parameters:

  • metrics (dict[str, float])

  • per_user (dict[str, ndarray] | None)

  • sample_ids (ndarray | None)

  • n_rows (int)

  • n_scored_rows (int)

  • required_k (int)

  • metadata (dict[str, Any])

  • debug_rows (tuple[dict[str, Any], ...] | None)

  • target_fingerprint (str | None)

property n_units: int

Independent evaluation units behind the scored rows.

Distinct sample_ids, which is smaller than n_scored_rows when a protocol gives one user several rows – eval_draws above 1, say. It matches compresso_recsys.stats.PairwiseComparison.n_units, the count paired comparison actually resamples.

Without identifiers there is nothing to group by, and every row is its own unit, which is also what comparison assumes when it numbers rows positionally.

property has_per_user: bool

Whether per-user observations were collected.

to_dict(*, include_debug=True)[source]

Return the mapping view as a plain dict.

Use this where an actual dict is required, such as JSON serialization. Per-user values are deliberately excluded.

Return type:

dict[str, Any]

Parameters:

include_debug (bool)

Custom Metrics

compresso_recsys.metrics.RankingMetric.update() returns the per-row values it computed, with shape (rows, len(result_keys)), rather than only folding them into a running sum. Without that the evaluator would have to compute every metric twice to retain per-user observations.

Third-party metric implementations must therefore return their values. When collection is enabled the evaluator validates the returned tensor: two dimensions, one row per prediction row, one column per result key, floating-point dtype, and finite values for rows with targets.

Values must be produced for every row, including rows with no targets. Those rows are excluded from the aggregate but must still occupy a position so that values stay aligned with sample_ids before filtering.

compresso_recsys.evaluation.evaluate_recommender(model, *, source, targets, metrics, sample_ids=None, collect_per_user=True, metadata=None, batch_size=1024, match_backend='auto', max_dense_cells=20000000, validate_predictions=True, debug=False, debug_users=5, show_progress=False, logger=None, log_every_n_steps=1000)[source]

Evaluate a recommender without retaining predictions between batches.

Overloads:

  • model (Recommender), source (csr_matrix), targets (csr_matrix), metrics (Sequence[RankingMetric]), sample_ids (Sequence[Any] | np.ndarray | None), collect_per_user (bool), metadata (Mapping[str, Any] | None), batch_size (int), match_backend (MatchBackend), max_dense_cells (int), validate_predictions (bool), debug (bool), debug_users (int), show_progress (bool), logger (Any | None), log_every_n_steps (int) → EvaluationResult

  • model (SequentialRecommender), source (ItemSequences), targets (csr_matrix), metrics (Sequence[RankingMetric]), sample_ids (Sequence[Any] | np.ndarray | None), collect_per_user (bool), metadata (Mapping[str, Any] | None), batch_size (int), match_backend (MatchBackend), max_dense_cells (int), validate_predictions (bool), debug (bool), debug_users (int), show_progress (bool), logger (Any | None), log_every_n_steps (int) → EvaluationResult

Parameters:

  • model (Recommender | SequentialRecommender)

  • source (csr_matrix | ItemSequences)

  • targets (csr_matrix)

  • metrics (Sequence[RankingMetric])

  • sample_ids (Sequence[Any] | ndarray | None)

  • collect_per_user (bool)

  • metadata (Mapping[str, Any] | None)

  • batch_size (int)

  • match_backend (Literal['auto', 'dense', 'searchsorted'])

  • max_dense_cells (int)

  • validate_predictions (bool)

  • debug (bool)

  • debug_users (int)

  • show_progress (bool)

  • logger (Any | None)

  • log_every_n_steps (int)

Return type:

EvaluationResult

The largest metric cutoff determines the k passed to the model’s predict_on_batch method. Source and target rows are sliced together, and each prediction batch is immediately sent to RankingEvaluator. Source and target column counts may differ: source columns describe the model’s history vocabulary, while target columns describe its candidates.

source may be a csr_matrix of interactions or an ItemSequences of chronological histories, matching whichever the model reads. Only the row count has to agree with targets; nothing downstream of predict_on_batch knows or cares which was given, which is why sequential and matrix models can be compared against each other with no statistics-side changes.

evaluate_recommender requires matching source and target row counts, but their column counts may differ. This supports a fixed history vocabulary with a separately managed candidate catalog. Every prediction batch must use the same column space as its corresponding target matrix.

For a terminal or notebook, pass show_progress=True to display its tqdm bar. For a service, pass any object with an info(str) method as logger; the logger takes precedence over the bar and receives start, finish, and intra-evaluation lines at log_every_n_steps batches. A failing logger is disabled after one warning without aborting evaluation.

compresso_recsys.evaluation.evaluate_ranked_predictions(*, predictions, targets, metrics=None, sample_ids=None, collect_per_user=True, metadata=None, batch_size=4096, match_backend='auto', max_dense_cells=20000000, validate_predictions=True, debug=False, debug_users=5)[source]

Evaluate ranked top-k SRP predictions against binary CSR targets.

Prediction columns must be unique within each row and ordered by descending prediction score. Target values are interpreted as binary relevance. Rows without nonzero targets are excluded from metric means.

When metrics is omitted, calibrated recall and nDCG are calculated at the full prediction width. Use RankingEvaluator directly when predictions are generated one batch at a time.

Return type:

EvaluationResult

Parameters:

  • predictions (SRPTensor)

  • targets (csr_matrix)

  • metrics (Sequence[RankingMetric] | None)

  • sample_ids (Sequence[Any] | ndarray | None)

  • collect_per_user (bool)

  • metadata (Mapping[str, Any] | None)

  • batch_size (int)

  • match_backend (Literal['auto', 'dense', 'searchsorted'])

  • max_dense_cells (int)

  • validate_predictions (bool)

  • debug (bool)

  • debug_users (int)

class compresso_recsys.evaluation.RankingEvaluator(metrics, *, match_backend='auto', max_dense_cells=20000000, validate_predictions=True, collect_per_user=True, metadata=None, debug=False, debug_users=5)[source]

Stream ranked SRP predictions against variable-length CSR targets.

The evaluator matches each prediction batch to its target CSR rows once and sends the resulting RankingBatch to every metric. auto matching uses a dense boolean target mask for small batches and composite-key torch.searchsorted matching for larger item spaces.

Parameters:

  • metrics (Sequence[RankingMetric])

  • match_backend (MatchBackend)

  • max_dense_cells (int)

  • validate_predictions (bool)

  • collect_per_user (bool)

  • metadata (Mapping[str, Any] | None)

  • debug (bool)

  • debug_users (int)

Metrics

Every built-in metric accepts one cutoff or a sequence of cutoffs. Rows without target items are excluded from all metric means.

Default Metrics

When metrics are not supplied to prediction or embedding evaluation, the defaults remain:

  • CalibratedRecall, reported as calibrated_recall@K. It divides hits by min(K, number_of_targets).

  • NDCG, reported as ndcg@K with binary relevance.

Optional Metrics

The following metrics are available only when explicitly included in the metrics argument:

  • Recall, reported as recall@K. It divides hits by the total number of target items, which is the usual definition and the one to compare against published numbers. It cannot exceed K / number_of_targets, so users with many targets cap below one; calibrated_recall@K truncates the denominator instead and is greater than or equal to it for every user.

  • Precision, reported as precision@K. It divides hits by K.

  • HitRate, reported as hit_rate@K. It is one when at least one target occurs in the top K, otherwise zero.

  • MRR, reported as mrr@K. It is the reciprocal rank of the first hit, or zero when no hit occurs by K.

  • MAP, reported as map@K. Average precision sums precision at each hit and divides by min(K, number_of_targets).

For example:

from compresso_recsys.metrics import HitRate, MAP, MRR, Precision, Recall

optional_metrics = [
    Recall([20, 50, 100]),
    Precision([20, 50, 100]),
    HitRate([20, 50, 100]),
    MRR([20, 50, 100]),
    MAP([20, 50, 100]),
]
class compresso_recsys.metrics.RankingMetric[source]

Abstract streaming metric over ranked recommendation batches.

abstract property required_k: int

Largest recommendation rank needed by this metric.

abstract property result_keys: tuple[str, ...]

Metric keys returned by compute().

abstractmethod reset()[source]

Clear accumulated metric state.

Return type:

None

abstractmethod update(batch)[source]

Accumulate one ranking batch and return its per-row values.

Returns:

Floating-point tensor of shape (batch.predictions.rows, len(result_keys)), with column i holding the per-row value behind result_keys[i]. Rows whose target_counts is zero are excluded from the aggregate but must still occupy a row here, so the evaluator can align values with sample identifiers before filtering.

Returning the values rather than only accumulating them lets RankingEvaluator retain per-user observations without computing every metric twice.

Return type:

Tensor

Parameters:

batch (RankingBatch)

abstractmethod compute()[source]

Return aggregated metric values.

Return type:

dict[str, float]

class compresso_recsys.metrics.RankingBatch(predictions, hits, target_counts)[source]

Shared vectorized inputs for ranking metrics.

hits[row, rank] records whether the item at that prediction rank is relevant. target_counts contains the number of relevant items per row. Metric implementations can therefore stay independent of CSR matching.

Parameters:

  • predictions (SRPTensor)

  • hits (Tensor)

  • target_counts (Tensor)

class compresso_recsys.metrics.CalibratedRecall(cutoffs)[source]

Recall normalized by min(k, number of relevant targets).

Reported as calibrated_recall@k. The truncated denominator mirrors the ideal ranking used by NDCG, so a user with more relevant items than k can still reach 1.0. It is greater than or equal to Recall for every user, with equality exactly when a user has at most k relevant items, so the two are not interchangeable in a results table.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)

class compresso_recsys.metrics.NDCG(cutoffs)[source]

Binary-relevance normalized discounted cumulative gain.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)

class compresso_recsys.metrics.Recall(cutoffs)[source]

Recall normalized by the total number of relevant targets.

Reported as recall@k. This is the usual definition, so it is the one to use when comparing against published numbers unless that work states it truncates the denominator. It cannot exceed k / (number of relevant targets), so users with many relevant items cap below 1.0.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)

class compresso_recsys.metrics.Precision(cutoffs)[source]

Fraction of the top-k predictions that are relevant.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)

class compresso_recsys.metrics.HitRate(cutoffs)[source]

Whether at least one relevant item occurs in the top-k predictions.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)

class compresso_recsys.metrics.MRR(cutoffs)[source]

Mean reciprocal rank of the first relevant prediction up to each cutoff.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)

class compresso_recsys.metrics.MAP(cutoffs)[source]

Mean average precision with binary relevance at each cutoff.

Parameters:

cutoffs (int | Sequence[int])

batch_values(batch)[source]

Return per-row values with shape (rows, len(cutoffs)).

Return type:

Tensor

Parameters:

batch (RankingBatch)