AI uncertainty flags unfamiliar head and neck scans before auto-segmentation errors occur

Uncertainty scoring detected metal-related distribution shifts with AUC up to 0.89 and identified cases with reduced segmentation accuracy.

KEY POINTS

  • This retrospective study used planning CT scans from 1,016 patients with head and neck cancer to investigate whether AI-generated uncertainty scores could identify inputs outside an auto-segmentation model's training distribution. The evaluated nnU-Net models segmented 14 organs of interest, including salivary glands, oral cavity, mandible, spinal cord, laryngeal structures, thyroid and cervical esophagus.
  • Uncertainty was quantified using Monte Carlo dropout with 20 inference passes per patient, producing voxel-level predictive entropy maps and aggregate case-level scores. Two complementary experiments examined a known distribution shift caused by dental metal artifacts and an unknown distribution shift simulated through selective fine-tuning of segmentation models.
  • In the controlled experiment, a baseline model trained on 620 scans, including 44% with metal artifacts, was compared with a second model trained on 620 artifact-free scans. Testing included 10 scans with metal artifacts and 10 without; metal-containing images were unfamiliar to the artifact-free model, creating a defined out-of-distribution challenge.
  • The artifact-free model showed stronger uncertainty separation between metal and nonmetal cases, with mean counts of high-uncertainty voxels of approximately 110,000 versus 60,000 (p=0.004). Its out-of-distribution detection performance reached AUC 0.89 (95% CI 0.70–1.00) and 90% classification accuracy, compared with AUC 0.87 and 80% accuracy for the baseline model.
  • Dental artifacts also degraded anatomical segmentation. In the artifact-free model, mean mandible surface Dice decreased from 0.98 to 0.93, while oral cavity surface Dice decreased from 0.81 to 0.74 in artifact-containing cases. The baseline model, which had encountered metal artifacts during training, showed smaller deterioration, supporting the relationship between training-data coverage and model reliability.
  • In the second experiment, models were fine-tuned using either 21 low-uncertainty or 21 high-uncertainty scans and evaluated on 76 held-out patients. The narrowly trained model performed better on low-uncertainty than high-uncertainty cases (mean surface Dice 0.80 vs 0.77; p=0.04), whereas the broadly trained model showed no significant difference (0.77 vs 0.76; p=0.34). Interestingly, the narrow model still had higher overall geometric accuracy for several structures despite reduced robustness to unfamiliar inputs.
  • A complete set of uncertainty calculations required approximately 60 seconds per patient, introducing additional computational overhead. The authors emphasize that high uncertainty does not necessarily mean incorrect segmentation, and low uncertainty does not guarantee correctness; scores are model-specific, affected by training distribution and require local calibration before clinical use.

CLINICAL TAKEAWAY

Uncertainty quantification may help radiotherapy departments identify patients whose scans fall outside the validated operating range of an auto-segmentation model, enabling more targeted contour review or routing to specialized models. The findings support automated QA development but do not justify omitting physician verification, particularly because uncertainty measures model familiarity rather than clinical contour correctness.

SOURCE

Radiotherapy and Oncology

Browse more research Suggest a correction