KEY POINTS
- PRECISE-PC, developed at Mayo Clinic, evaluated automated extraction of prostate cancer risk factors from unstructured biopsy and MRI reports. The retrospective cohort included 702 patients treated with definitive ultrahypofractionated RT between 2015 and 2024, with median age 70 years and median pretreatment PSA 7.1 ng/mL. 501 patients (71.4%) received IMPT and 201 (28.6%) VMAT, with a median radiation dose of 38 Gy (IQR 38–40 Gy), 53% receiving ADT, and median follow-up of 29 months.
- The system used Gemini 2.5 Pro with TextGrad-based automated prompt optimization to extract primary and secondary Gleason scores, percentage of positive biopsy cores, and MRI findings of extraprostatic extension (EPE) and seminal vesicle invasion (SVI). These features, together with pretreatment PSA, were used to determine NCCN risk categories. Prompt development used an independent 200-report dataset divided into training (n=100), validation (n=50), and testing (n=50), with final optimized prompts subsequently frozen for evaluation in the separate 702-patient cohort.
- Prompt optimization substantially improved extraction accuracy. On the optimization test sets, overall accuracy increased from 91% to 95% for MRI reports and from 82% to 96% for biopsy reports. Particularly important improvements included extraction of ≥50% positive biopsy cores (66% to 90%), primary Gleason score (90% to 100%), secondary Gleason score (90% to 98%), MRI-defined EPE (86% to 92%), and SVI (96% to 98%). The accepted prompt versions were reached after approximately 11 minutes for pathology and 15 minutes for MRI optimization.
- In the independent 702-patient clinical evaluation cohort, automated extraction achieved exact agreement with manual annotation for both primary and secondary Gleason scores (100%). Performance was less reliable for high-volume biopsy involvement, with 92.3% overall accuracy, 79.3% sensitivity, and Krippendorff's α of 0.807. EPE extraction achieved overall accuracy of 95.3% when indeterminate findings were retained as a distinct category, while SVI extraction achieved 98.9%, demonstrating that ambiguous or less consistently documented variables remained more challenging than numerical Gleason grades.
- NCCN risk classification agreed with manual review in 675/702 patients (96.2%), leaving 27 misclassified cases. Of these, 24 errors involved favorable versus unfavorable intermediate-risk disease, including 22 patients incorrectly downgraded from unfavorable to favorable intermediate risk. Two favorable intermediate-risk patients were classified as high risk, and one high-risk patient as very high risk. Misclassification frequently involved the 50% positive-core threshold, emphasizing that high overall accuracy can conceal treatment-relevant errors in individual patients.
- Risk groups generated from LLM-derived features produced three-year biochemical recurrence-free and distant metastasis-free survival curves broadly similar to manual classification. In the high-risk group, however, three-year biochemical recurrence-free survival was 84.7% using LLM classifications versus 87.5% manually, and distant metastasis-free survival was 92.7% versus 95.9%. These analyses demonstrate preservation of broad prognostic stratification within the existing dataset, rather than independent prediction of outcomes or evidence that automated classification improves clinical decisions.
- The study did not measure actual manual abstraction time savings, prospectively deploy the system, or validate it across fully independent institutions. Manual reference labels were created through sequential review rather than independent dual annotation, and indeterminate EPE or SVI findings were treated as negative for risk classification. The cohort was also dominated by intermediate-risk disease, with relatively few very-high-risk patients. Consequently, model performance for uncommon high-risk presentations and its robustness to different reporting conventions remain uncertain.
CLINICAL TAKEAWAY
Prompt-optimized LLMs can extract prostate cancer risk factors from routine clinical documentation with high accuracy and may support automated registries, clinical trial screening, and risk-stratification workflows. However, errors involving positive-core thresholds and clinically relevant risk categories demonstrate that current performance is insufficient for unsupervised treatment decision-making, and prospective external validation with physician review remains necessary.