Public benchmarks have a coverage problem
Most public benchmarks for spatial transcriptomics models over-index on lung, breast, and colorectal — the cancers with the most curated data. Models that score well on those benchmarks routinely fall apart on the cancers that have less public data, which are exactly the cancers most BD partners ask us about.
Our internal eval suite
We track three orthogonal metrics for every model release:
- Cell-type AUROC across 7 cancers — NSCLC, SCLC, BRCA, COAD, PRAD, RCC, ESCA
- Cross-cohort robustness — score variance when we hold out an entire patient cohort, not just samples
- Out-of-distribution detection — performance on rare cancers the model wasn’t trained on, expressed as flagging accuracy
PortraiTME™ currently sits at 0.96 epithelial AUROC on NSCLC and 0.88 on the most-distant cancer in our held-out set.
Where this is going
Public benchmarks will catch up eventually. Until they do, internal benchmarks are how we keep model decisions honest with BD partners.
