Doctronic: Closed-Loop AI Clinician Feedback Engine (#md-ai-feedback)

System Motivation

In high-stakes telemedicine, relying on static offline benchmarks is insufficient. Human clinicians (attending physicians) encounter nuanced real-world failure modes—such as subtle sycophancy, improper controlled substance suggestions, or body-side laterality confusion—that standard test sets miss.

I architected an automated closed-loop learning engine that turns live doctor critiques from the #md-ai-feedback Slack channel into continuous, data-driven model hardening.

Closed-Loop Architecture

1. Production Feedback Ingestion: Automatically ingests unstructured clinician critique reports from Slack, parsing clinical complaints, timestamps, and patient case identifiers.
2. Transcript Matching & EHR Reconciliation: Resolves reports against production visit databases, extracting complete patient chat transcripts, clinical context, and draft SOAP notes.
3. Claude Opus 5.5 Unified Judge: Built an automated unified triage judge using Claude Opus 5.5 that analyzes the full interaction history. Critically, it distinguishes between what the AI model itself drafted versus what the human physician caught and corrected, preventing the AI from falsely being credited for human-corrected mistakes.
4. Structured Clinical Failure Taxonomy: Classifies incidents across formal failure families:
   • scope_violation: Out-of-telemedicine scope and emergency department red flags.
   • patient_anchoring_sycophancy: AI prematurely agreeing with a patient’s self-diagnosis.
   • missed_drug_safety: Contraindications and geriatric Beers criteria risks (e.g. benzodiazepines + Z-drugs).
   • laterality: Left vs. right anatomical body-side confusion in symptom notes.
   • refill_gating and intake_bypass.
5. Automated Regression Suite Generation: Converts real-world failure cases directly into regression test criteria within doctronic-eval (data/cases/md_feedback/criteria/...).
6. DSPy MIPRO Prompt Optimization: Drives automated prompt writer rounds and critique passes, which are then validated on pre-registered 5-fold cross-validation ablation suites before safe production deployment.