Minh To All publications

Original Article · IJCARS · 2026

Shift happens

A fairness-oriented framework for medical classification under hidden bias

Minh Nguyen Nhat To1,4,*, Diane Kim1,†, Mohamed Harmanani2,†, Paul F. R. Wilson2,4, Fahimeh Fooladgar1,4, Samira Sojoudi1, Amoon Jamzad2,4, Sherif Abdalla1,3, Teresa Tsang1,3, Christina Luong1,3, Silvia Chang1,3, Peter Black1,3, Robert Siemens2, Michael Leveridge2, Rahul G. Krishnan4, Parvin Mousavi2,4,†, Purang Abolmaesumi1,†

  • 1 University of British Columbia, Vancouver
  • 2 Queen's University, Kingston
  • 3 Vancouver General Hospital, Vancouver
  • 4 University of Toronto, Toronto

Corresponding author: tnnhatminh@gmail.com. Diane Kim, Mohamed Harmanani, Parvin Mousavi and Purang Abolmaesumi contributed equally to this work.

International Journal of Computer Assisted Radiology and Surgery
Volume 21, Issue 5, pages 1025–1032 · Published online 24 April 2026

Read the paper (opens in a new tab) Code (opens in a new tab) Cite

Abstract

Purpose
Many medical AI models perform unevenly across patient groups because they learn shortcuts from biased data. These hidden biases make models less reliable and less fair in real-world use. This work aims to develop a system that remains accurate and fair across different patient subpopulations, even when those groups are not explicitly labeled.
Methods
We introduce DPE-Former, a model that combines prototype-based learning with transformer attention. The system trains several complementary classifiers on balanced subsets of data, each capturing different aspects of the population. A transformer module then learns how to combine its outputs in an adaptive way, helping the model make more balanced decisions across unseen or minority groups.
Results
Across diverse datasets, including prostate ultrasound, skin lesion images, and cardiac patient records, DPE-Former achieved higher accuracy on underrepresented groups and more consistent performance overall compared to standard training methods.
Conclusion
DPE-Former offers a simple yet effective approach to reduce hidden bias in medical AI. By improving fairness and reliability across both image and tabular data, it supports more equitable decision-making in clinical applications such as cancer diagnosis and cardiac care.

Framework overview

Three-stage DPE-Former diagram. Stage 1, representation learning: class-imbalanced prostate ultrasound and skin lesion training samples pass through a feature extractor, and per-group and per-class accuracy bar charts show uneven first-stage performance. Stage 2, prototype learning: class-balanced validation subsets train multiple prototypes per class, shown as separate feature-space embeddings. Stage 3, ensemble aggregation: a transformer encoder combines prototype predictions over the entire validation set, producing more even per-group and per-class accuracy.
Figure 1. Overview of DPE-Former. Three stages: supervised representation learning, diverse prototype discovery on class-balanced subsets, and attention-based aggregation over the full validation set. Subgroup annotations such as acquisition site are used for evaluation only, and are not required to train the prototype ensemble. View full size ↗(opens in a new tab)

Subpopulation shift in medical classification

Deep learning has achieved expert-level accuracy in many areas of medical image analysis, yet models frequently fail to perform consistently across patient groups and acquisition settings. In this work, fairness denotes the ability of a model to maintain comparable performance across subpopulations, and bias denotes systematic performance gaps that arise when predictions depend on confounding or non-causal cues.

These disparities are often driven by subpopulation shift, in which the relationship between image features and diagnostic labels changes across hidden or unobserved groups. If most malignant cases in a training set originate from a single clinical site, a model may associate site-specific characteristics such as probe orientation, contrast settings, or patient demographics with malignancy. Chest X-ray models have been shown to associate laterality markers or portable machine artifacts with disease status, and skin lesion classifiers may depend on skin tone or background colour rather than on the lesion itself. When such a model is deployed in a different environment these associations may no longer hold, and the resulting degradation is visible not in overall accuracy but in a large gap between the best and worst performing subgroups.

Standard training by Empirical Risk Minimization (ERM) optimises average performance over the training distribution and therefore tends to exploit whichever correlations dominate the data. Existing corrections either reweight or resample the training data to minimise the worst-group loss, or learn representations that remain stable across domains. Both families generally require explicit subgroup annotations, which are rarely available in clinical practice because of privacy, governance, and cost constraints. DPE-Former is developed for that setting: it represents latent variation through multiple prototypes per class, and learns how those prototypes should contribute to each prediction without access to subgroup labels during training.

Diverse prototypes with transformer aggregation

Diverse Prototypical Ensembles (DPE) learn several prototypes per class and encourage them to specialise in different regions of the feature space, but combine prototype outputs by simple averaging, which limits the ability to model relationships among prototypes. DPE-Former replaces that averaging step with a transformer aggregator, which weights prototype predictions according to their interactions for a given input.

  1. Representation learning

    A base encoder is trained under standard supervised learning on data carrying no subgroup annotations. The classifier head is then discarded and the feature embedding is retained.

  2. Prototype diversification

    Multiple learnable prototypes per class are trained on class-balanced validation subsets. A diversity regulariser penalises alignment between prototypes of the same class, so that each prototype occupies a distinct region of the feature space.

  3. Transformer aggregation

    Prototype logits are treated as tokens. A transformer encoder with a classification token attends across them, so that the contribution of each prototype is determined by the input rather than fixed in advance.

Datasets and evaluation protocol

Three datasets were selected to reflect distinct modalities and distinct sources of subpopulation shift. All splits are made by patient identifier and preserve class proportions, with no patient overlap between the training, validation, and test sets.

  • Ultrasound

    Prostate ultrasound

    Interventional imaging with strong acquisition-site variability. Benign and cancerous samples are unequally distributed between two hospitals, which creates a spurious correlation between site and label.

    311 patients · 2 centres

  • Dermoscopy

    HAM10000

    Dermoscopic images exhibiting both visual and demographic imbalance, with substantial intra-class variation in acquisition conditions and colour tone.

    10,015 images · 7 classes

  • Tabular

    Cardiac tabular

    Structured clinical and echocardiographic variables for treatment prediction in acute coronary syndrome, in which the intervention class contains two latent subgroups of unequal size.

    1,571 patients · 2009–2019

Clinical-use note

This is a research framework, not a medical device. Any clinical application would require independent external validation, governance review, and appropriate regulatory oversight.

Results

  • ACC overall accuracy
  • BA balanced accuracy, averaged over classes
  • WGA worst-group accuracy, the lowest-scoring subgroup
  • all values in per cent, averaged over three random seeds
Accuracy, balanced accuracy, and worst-group accuracy for six methods on the Prostate Ultrasound, HAM10000, and Cardiac Tabular datasets, in percent.
Method Prostate ultrasound HAM10000 Cardiac tabular
Method ACCBAWGA ACCBAWGA ACCBAWGA
ERM —not reported—not reported—not reported 78.480.166.7 62.060.735.7
MedSAM 75.466.138.1 —not applicable—not applicable—not applicable —not applicable—not applicable—not applicable
Cinepro 74.564.338.1 —not applicable—not applicable—not applicable —not applicable—not applicable—not applicable
DFR 66.763.850.3 81.084.072.5 67.971.638.1
DPE 66.365.758.7 80.483.575.3 67.568.842.9
DPE-Former 66.166.360.2 82.983.675.6 68.673.162.5

Table 1. The best value in each column is shown in colour. Em dashes mark configurations that are not reported: MedSAM and Cinepro are ultrasound-specific, and ERM is not listed separately for prostate ultrasound because the MedSAM and Cinepro rows are themselves ERM classifiers trained on features from their respective encoders. On prostate ultrasound DPE-Former obtains a worst-group accuracy of 60.2%, compared with 38.1% for MedSAM and Cinepro and 50.3% for DFR, while its overall accuracy of 66.1% is below the 75.4% of MedSAM. On the cardiac tabular data it obtains the highest balanced accuracy at 73.1% and the highest worst-group accuracy at 62.5%. All three datasets are class-imbalanced, so balanced and worst-group accuracy should be read alongside overall accuracy rather than in place of it.

Two-panel ablation figure on the prostate ultrasound dataset. Panel a: line plots of balanced accuracy and worst-group accuracy against the number of prototypes, both rising steeply up to around six prototypes and then plateauing. Panel b: bar charts comparing Voting, Bagging, Stacking, and the proposed aggregation, with the proposed method highest on both balanced and worst-group accuracy.
Figure 2. Ablations on prostate ultrasound. (a) Effect of the number of prototypes. (b) Comparison of aggregation strategies. Each experiment is repeated three times. View full size ↗(opens in a new tab)

  • Effect of the number of prototypes

    On prostate ultrasound, balanced accuracy increases from approximately 64% with two prototypes to approximately 66% at around six prototypes, and remains at that level as the number increases to fifty. Worst-group accuracy follows a comparable trend, increasing from approximately 53% to slightly above 60%. These results suggest that a moderate number of prototypes is sufficient to represent the useful variation, and that further prototypes yield diminishing returns.

  • Effect of the aggregation strategy

    Voting, bagging, and stacking obtain balanced accuracies of approximately 58%, 60%, and 63% respectively, with worst-group accuracy below 50% in each case. The transformer aggregator obtains approximately 67% balanced accuracy and 61% worst-group accuracy in the same setting, which suggests that weighting prototype outputs adaptively offers an advantage over uniform or heuristic aggregation.

  • Comparison with group-aware training

    GroupDRO, which has access to subgroup labels during training, obtains 66.5 ± 2.1% accuracy, 65.4 ± 2.5% balanced accuracy, and 47.6 ± 6.1% worst-group accuracy on the cardiac dataset. It improves worst-group accuracy over the baselines but does not reach DPE-Former on any of the three metrics, which indicates that access to subgroup labels alone does not guarantee improved robustness.

Limitations

  • Balanced and worst-group accuracy indicate whether accuracy is even across groups, but not which types of error occur within each group, and false positives and false negatives carry different clinical consequences.
  • Although the transformer models prototype interactions effectively, its interpretability remains limited, and visualising attention patterns or aligning them with known subgroup features is left to future work.
  • The evaluation covers datasets with known subgroup variability, and larger or more heterogeneous cohorts, particularly those in which subgroup structure is highly overlapping, may present additional difficulties.
  • The aggregator configuration was selected on the basis of stable optimisation and consistent validation behaviour rather than an exhaustive search over layers, heads, or token dimensions, so the gain attributable to the adaptive aggregation principle is not separated from that of this particular transformer instantiation.

Institutions and support

Funding

Supported in part by the Canadian Institutes of Health Research (CIHR) and the Natural Sciences and Engineering Research Council of Canada (NSERC). Parvin Mousavi is supported by the CIFAR AI Chair and the Vector Institute.

Data & ethics

Approved by the Institutional Research Ethics Board at both participating sites; all patients provided informed consent. Data were provided by the Provincial Health Services Authority (PHSA) with appropriate ethical permissions.

Declarations

The authors declare no conflict of interest. The authors are responsible for all analyses, interpretations, and conclusions presented, which do not necessarily reflect the views of PHSA.

Citation

To, M. N. N., Kim, D., Harmanani, M. et al. Shift happens: a fairness-oriented framework for medical classification under hidden bias. Int J CARS 21, 1025–1032 (2026). https://doi.org/10.1007/s11548-026-03624-0 (opens in a new tab)

BibTeX 
@article{to2026shifthappens,
  title   = {Shift happens: a fairness-oriented framework for
             medical classification under hidden bias},
  author  = {To, Minh Nguyen Nhat and Kim, Diane and
             Harmanani, Mohamed and Wilson, Paul F. R. and
             Fooladgar, Fahimeh and Sojoudi, Samira and
             Jamzad, Amoon and Abdalla, Sherif and
             Tsang, Teresa and Luong, Christina and
             Chang, Silvia and Black, Peter and
             Siemens, Robert and Leveridge, Michael and
             Krishnan, Rahul G. and Mousavi, Parvin and
             Abolmaesumi, Purang},
  journal = {International Journal of Computer Assisted
             Radiology and Surgery},
  volume  = {21},
  number  = {5},
  pages   = {1025--1032},
  year    = {2026},
  doi     = {10.1007/s11548-026-03624-0}
}

Institutional wordmarks are reproduced without modification from UBC, Queen's, Vancouver Hospital, and University of Toronto. These marks remain the property of their respective organisations and appear here for identification only; their use does not imply endorsement.