Why a multimodal model
Cancer cells leave fingerprints across modalities — image texture in H&E, gene expression in spatial transcriptomics, clinical context in pathology reports. Models trained on one modality keep missing what the other two would have caught.
VGL (Vision · Gene · Language) is our attempt at a single model that sees all three.
Architecture
We use modality-specific encoders that project into a shared latent, with cross-attention layers stitching them together. The whole thing trains end-to-end on aligned tissue regions where image, expression, and reporting are all available.
What it gives us
- One embedding space for image-only, gene-only, and image+gene queries
- Strong zero-shot performance on cell-type classification across 7 cancer types
- Drop-in features for downstream models (target prioritization, drug-distribution prediction)
VGL is one of the engines feeding both PortraiTARGET and PortraiTME™. The next iteration extends to mass-cytometry and single-cell modalities.
