Dataset and model validation matrix

This is a validation workbench, not a claim of reproduced results. Published scores below are reference targets; no full-data training runs have been measured for this matrix yet. Small numerical differences are expected, but different architectures, candidate sets, or split protocols are not rounding differences.

For an all-dataset statistics and baseline run, see Dataset statistics and baseline sweep.

Runnable code

Download benchmark.py or run examples/validation/benchmark.py from the repository root after installing the package and its model dependencies. Its build function builds the checkpoint; train fits candidates, selects by validation NDCG@100, and evaluates the selected model on test only once. JSON output records configurations, validation/test scores, matrix dimensions, package version, seed, and checkpoint SHA-256. It does not save trained model weights.

python examples/validation/benchmark.py build --dataset ml20m --checkpoint artifacts/validation/ml20m.zip
python examples/validation/benchmark.py train --checkpoint artifacts/validation/ml20m.zip --model ease --l2 200 500 1000 --output artifacts/validation/ml20m-ease.json
python examples/validation/benchmark.py train --checkpoint artifacts/validation/ml20m.zip --model multvae --epochs 20 50 100 --output artifacts/validation/ml20m-multvae.json
python examples/validation/benchmark.py train --checkpoint artifacts/validation/ml20m.zip --model multdae --epochs 20 50 100 --output artifacts/validation/ml20m-multdae.json

Replace ml20m in the build command and artifact paths with netflix or taste-profile for the other rows. Epoch candidates are independent fits from the same seed, not resumed runs. These grids are starting points, not the papers’ complete hyperparameter searches. Extend them using validation results only. Use --device cuda when available for neural models; CPU is the default.

Published reference results

All nine rows use full-catalog ranking with observed source items excluded. R20/R50 mean the papers’ Recall@20/50: hits divided by min(cutoff, number of relevant items). The runner therefore uses CalibratedRecall, not Recall. N100 is binary-relevance NDCG@100. Every row uses the builder and training code linked above.

Reference targets, not locally measured results
Paper / model	Build dataset	R20	R50	N100	Train model / status
EASE, Table 1	ml20m	0.391	0.521	0.420	ease / adapted split
EASE, Table 1	netflix	0.362	0.445	0.393	ease / adapted split
EASE, Table 1	taste-profile	0.333	0.428	0.389	ease / adapted split
MultVAE, Table 2	ml20m	0.395	0.537	0.426	multvae / adapted split
MultVAE, Table 2	netflix	0.351	0.444	0.386	multvae / adapted split
MultVAE, Table 2	taste-profile	0.266	0.364	0.316	multvae / adapted split
MultVAE (MultDAE), Table 2	ml20m	0.387	0.524	0.419	multdae / architecture differs
MultVAE (MultDAE), Table 2	netflix	0.344	0.438	0.380	multdae / architecture differs
MultVAE (MultDAE), Table 2	taste-profile	0.266	0.363	0.313	multdae / architecture differs

For original preprocessing and neural training code, see the authors’ vae_cf repository. EASE’s paper includes the closed-form training algorithm. Both papers report standard errors around 0.002 for ML-20M and 0.001 for Netflix/MSD; these are not acceptance tolerances for a changed protocol.

Checkpoint protocol and limitations

Explicit builder settings
Dataset	Positive interaction	Min user / item support	Validation / test users
ML-20M	Rating >= 4, then binary	5 / 1	10,000 / 10,000
Netflix	Rating >= 4, then binary	5 / 1	40,000 / 40,000
MSD Taste Profile	Any play count, then binary	20 / 200	50,000 / 50,000

The recipe fixes seed 98765, disjoint held-out users, an 80/20 source/target split, one evaluation draw, and no minimum metadata-text length. This overrides the normal ML-20M builder defaults. Netflix requires the source files to be available locally; consult Datasets for download availability and terms.

The current builder is paper-inspired, not the authors’ preprocessing program: filter ordering, iterative support pruning, random-number generation, catalog construction, and holdout rounding can change membership. Compare generated counts and saved manifests before interpreting score gaps. For strict replication, import the authors’ exact split through Bringing Your Own Dataset and preserve their train-derived item vocabulary and evaluation eligibility rules.

MultVAE uses a 600-hidden/200-latent architecture. Our MultDAE is items -> 200 -> items, whereas the main Table 2 reference uses a deeper network. Its scores are contextual references, not valid pass/fail targets for our shallower model. Training schedules and validation selection also need alignment before claiming replication.

EASE forms dense item-by-item matrices. One float64 matrix at 41,140 items is about 13.5 GB; inversion, retained models during validation selection, and workspace require substantially more. Do not start the MSD EASE grid on a memory-constrained machine. Neural recipes stream sparse training minibatches.

Sequential, cold-start, and other models

For ML-1M/DBbook/Last.fm precomputed features, see Datasets. Those adapters preserve incomplete feature coverage and do not reproduce the authors’ all-modalities MMRec benchmark; no matching score target is assigned yet.

Additional coverage and honest comparison boundaries
Model / dataset	Paper / upstream code	Local builder and training code	Published target / status
SASRec / ML-1M	SASRec paper; authors’ code	Reproducing SASRec Results on ML1M contains both	HR@10 0.8245, NDCG@10 0.5905 (Table III). Sampled-negative protocol only; see notebook caveats.
EASE / Steam	EASE; no Steam target in that paper	Same script: build `--dataset steam`, train `--model ease`	Runnable warm-start baseline; establish a new measured baseline
Steam cold-start / temporal / LLO	Datasets	Checkpoint CLI Reference and Datasets describe checkpoint construction	No matched published target established; CF script is not a cold-start trainer
ELSA	Authors’ reproduction code; paper in Citing Compresso Recsys	`ELSATrainer` in Models API; dedicated recipe pending	Not yet protocol-audited; no acceptance target assigned
TEASER / TEASERGD	Authors’ code; paper in Citing Compresso Recsys	Models API; metadata/protocol recipe pending	Not yet protocol-audited; do not substitute CF-only targets
KNN, simple sequence models, compressed ELSA	Citing Compresso Recsys	Models API	No matched validation recipe in this first matrix

Validation checklist

Record the repository commit and dirty diff with each run, plus data provenance.
Build once and reuse the identical checkpoint for competing models.
Check users/items/interactions, binary values, source/target separation, candidate vocabulary, and full-catalog versus sampled ranking.
Tune on validation only. Keep the test set sealed until configuration selection.
Compare the JSON test metrics with the matching row, recording absolute gaps. Repeat with several training seeds before diagnosing small differences.
Label outcomes adapted benchmark until preprocessing, architecture, training, and evaluation all match. Do not treat an arbitrary numerical tolerance as evidence of reproduction.