Doctronic: NoHarm & MAST Safety Programme (Surpassing SOTA Benchmark)

Objective & Benchmark Scope

The NoHarm / MAST (Medical AI Safety & Testing) programme was designed to rigorously measure diagnostic F1, harmful clinical recommendations, and severe failure rates in production medical AI models. The evaluation was conducted across 330 standardized clinical presentations (like-for-like with the published academic leaderboard).

Core Results & Product Impact

• Surpassing the Published Leaderboard SOTA: Optimized prompt directives and filtered retrieval tools, lifting benchmark F1 from 0.881 to 0.901 (+0.020 [+0.006, +0.035])—sitting above LiSA’s published leaderboard point estimate of 0.891.
• Harm & Severe Risk Reduction: Cut harmful clinical verdicts by −0.058 and reduced severe clinical risk rates down to 0.045.
• Clinical Guideline Grounding Layer: Increased direct Clinical Guideline (COG) citation grounding from 38% to 46–49% by quoting retrieved guideline passages directly into doctor consult drafts, reducing medical contradictions to <2%.
• US Corpus Coverage Analysis: Uncovered 35 “must-do” clinical guideline coverage gaps across the US medical corpus, establishing an open-evidence feasibility pipeline to resolve guideline omissions without displacing existing clinical boundaries.