Overview
Iterative Refinement
Doctors edit SOAP notes and add feedback → system re-runs analysis
Max 1 Re-Debate
Prevents infinite loops — only 1 re-analysis per case allowed
Full Context Preservation
Original patient data + doctor feedback merged for re-analysis
Audit Trail
All edits and feedback logged for compliance and learning
Workflow
Doctor Reviews Output
Clinician reads the SOAP note and identifies:
- Missing differentials
- Incorrect assessments
- Overlooked safety concerns
- Desired plan modifications
Doctor Provides Feedback
Two input methods:
- Edited SOAP
- Structured Feedback
Doctor directly edits the SOAP text:
System Re-Analyzes
ClinicalPilot merges original case + feedback:The full debate pipeline runs again:
backend/main.py:278
- Clinical Agent sees doctor’s corrections
- Literature Agent searches based on feedback keywords
- Safety Agent re-checks with new considerations
- Critic Agent reconciles AI + human input
API Reference
Endpoint
Submit doctor feedback and trigger re-analysis
Request Body
The clinician’s manually edited SOAP note. If provided, the re-analysis will prioritize this version.
The original clinical input text from the initial
/api/analyze request. This ensures the system has full patient context.Structured feedback from the clinician (e.g., “Missing differential: aortic dissection. Add CT angio to plan.”). This is appended to the prompt for re-analysis.
Response
Code Implementation
backend/main.py:262
Use Cases
Missed Differential
Missed Differential
Scenario: AI outputs differential for chest pain but misses aortic dissection.Doctor Action:Re-Analysis Output:
- Aortic dissection added to differentials
- Plan updated: “CT angio chest to r/o dissection before heparin”
- Safety flag: “Hold anticoagulation until dissection ruled out”
Incorrect Risk Stratification
Incorrect Risk Stratification
Scenario: AI classifies NSTEMI as low-risk (HEART score 3), but patient has 3-vessel CAD.Doctor Action:Re-Analysis Output:
- Risk stratification corrected to high-risk
- Plan changed from “Outpatient stress test” to “Admit, cardiology consult, likely cath within 24h”
Drug Safety Concern
Drug Safety Concern
Scenario: AI plan includes metformin, but patient has acute kidney injury (Cr 3.2).Doctor Action:Re-Analysis Output:
- Safety Agent flags metformin contraindication
- Plan updated: “Hold metformin. Start insulin SSI. Recheck Cr in 24h.”
Guideline Update
Guideline Update
Scenario: AI recommends older treatment protocol (e.g., 2019 sepsis guidelines), but institution uses updated 2023 protocol.Doctor Action:Re-Analysis Output:
- Plan updated to reflect 2023 guidelines
- Citations updated to reference 2023 Surviving Sepsis Campaign
Design Rationale
Why Max 1 Re-Debate?
The system limits re-analysis to 1 iteration to:- Prevent Infinite Loops: Without a cap, doctors could repeatedly re-submit feedback, causing exponential LLM calls.
- Encourage Finalization: After 1 re-analysis, the doctor should finalize the SOAP manually if still unsatisfied.
- Cost Control: Each full pipeline run costs ~1.00 in LLM API calls. Unlimited iterations would be prohibitively expensive.
Future Enhancement: Allow configurable
max_iterations per user role (e.g., attending = 2, resident = 1).How Feedback Is Merged
The system appends feedback to the original text:- Full Context: Agents see original patient data + doctor’s corrections
- Clear Delineation: Separator (
---) marks where human input begins - Prompt Engineering: Agents are trained to weigh human feedback heavily in debate rounds
Frontend Integration
The ClinicalPilot frontend provides a SOAP Editor for HITL:frontend/index.html (excerpt)
UI Flow
Doctor Clicks Edit
SOAP text becomes editable in a
<textarea>. A separate “Feedback” field appears for commentary.Audit Trail
For compliance (HIPAA, medico-legal), all HITL interactions should be logged:Performance Considerations
Latency
| Step | Duration | Notes |
|---|---|---|
| Submit feedback | ~50ms | API call |
| Re-run full pipeline | ~100s | Same as initial analysis |
| Update frontend | ~200ms | Render new SOAP |
| Total | ~100s | Same cost as initial run |
Cost: Each HITL re-analysis costs the same as the initial analysis (~1.00 in LLM API fees).
Optimization Ideas
Delta Re-Analysis
Only re-run agents affected by feedback (e.g., if feedback is about differentials, skip Safety Agent)
Caching
Cache Literature Agent PubMed results if feedback doesn’t change search queries
Async Notification
Return immediately, send email/SMS when re-analysis completes (for long cases)
Partial SOAP Update
Allow doctors to flag specific sections for re-generation (“Re-generate Plan only”)
Best Practices
Be Specific in Feedback
✅ Good: “Add CT angio to rule out aortic dissection — patient has back pain radiating to shoulders”❌ Bad: “Plan is incomplete”
Reference Clinical Data
Cite labs, vitals, or exam findings:✅ “Troponin is 1.2 (not 0.12) — this is NSTEMI, not unstable angina”
Suggest Evidence-Based Changes
Reference guidelines when correcting:✅ “Per 2023 AHA STEMI guidelines, door-to-balloon should be <90 min, not <120 min”
Limitations
Future Enhancements
Structured Feedback Forms
Guided UI: “Add differential”, “Flag safety issue”, “Correct lab value”
Version History
Track all SOAP versions (v1 = AI, v2 = after feedback, v3 = manual edits)
Multi-User Review
Allow attending + resident to both provide feedback → system reconciles
Reinforcement Learning
Use doctor corrections to fine-tune Clinical Agent (RLHF)
Next Steps
Full Analysis API
Learn how the multi-agent debate pipeline works
Emergency Mode
Fast-path triage for time-critical cases