Human-validated LLM-assisted assessment of a large-scale online precision medicine course in diabetes

Precision medicine increasingly relies on multimodal data and artificial intelligence, creating a need for scalable education and assessment. We developed an online diabetes precision medicine course and conducted an observational single-group pre–post evaluation of knowledge, confidence, and rubric-scored open-ended responses among 435 learners who completed the course. Because manual scoring of open-ended responses is time-consuming and subject to variability and human judgment, we developed a human-validated, large language model (LLM)-assisted scoring workflow. Knowledge, confidence, and open-ended response scores increased by 30.0 percentage points, 1.32 points, and 0.39 points, respectively (all p < 0.001). Across 4350 responses evaluated by the LLM-assisted workflow, only 12 (0.3%) required human review. In a stratified 300-response validation sample, the final LLM consensus score achieved 75.3% exact agreement, 99.3% adjacent agreement, and 0.7% major disagreements (LLM unanimity, 74.0%; human unanimity, 65.7%) with the final human score. Course completion was associated with improvements in knowledge, confidence, and open-ended response scores, while the findings also support human-validated LLM-assisted assessment as a scalable approach for evaluating open-ended responses.