PRISM & PRISMS
Incomplete Multimodal Medical Image Segmentation
Problem
In clinical MRI, multiple sequences (T1, T1ce, T2, FLAIR) provide complementary tissue contrasts for brain tumor segmentation. However, modalities are frequently missing or degraded due to patient motion, scanner limitations, or shortened protocols. Existing methods either require all modalities during training, or suffer from modality imbalance when certain sequences are rarely available.
PRISM — Self-Distillation Module
- Multi-Uni Self-Distillation: A shared fusion decoder acts as an implicit teacher, guiding individual modality-specific encoders (students) via pixel-level KL-divergence and prototype-level L2 alignment — no external teacher model needed.
- Preference-Aware Regularization: A dynamic coefficient based on each modality's relative preference score adjusts learning rates, actively boosting rare modalities while tempering dominant ones.
- Plug-and-Play Design: PRISM integrates into existing architectures (U-Net, mmFormer, RFNet) without architectural changes, consistently improving performance across all tested backbones.
PRISMS — Transformer Extension
- Adaptive Fusion Transformer (AFT): Uses learnable fusion tokens and Modality-Masked Attention (MMA) to robustly fuse features even when modalities are missing.
- Spatial Relevance Attention (SRA): Generates spatial attention masks that emphasize tumor-relevant regions across each modality.
- Channel-wise Fusion Transformer (KFT): Adaptively reweights feature channels to reduce redundancy and amplify discriminative information.
Experiments & Results
- Evaluated on BraTS2020 (369 cases, 4 MRI sequences, brain tumors) and MyoPS2020 (25 cases, 3 sequences, cardiac pathology)
- Tested under realistic Incomplete Training Data (UTD) scenarios with heterogeneous missing rates (FR = 0.2–0.8)
- Consistent Dice improvements of 5%+ over baselines in UTD settings
- Significant Hausdorff Distance reductions, indicating better boundary precision
- PRISMS outperforms SOTA methods (HeMIS, U-HVED, RobustMSeg, RFNet, mmFormer) across all modality configurations