Who's accountable when healthcare AI makes a mistake?
Originally drafted 23 January 2026. Republished under Aqta May 2026.
Consider this scenario: a radiologist reviews 200 chest X-rays daily with AI assistance. On scan #147, the AI misses a small tumour. The doctor, trusting the AI's recommendation, moves on. Three months later, the patient returns with stage 3 cancer.
It's hypothetical, but the question it raises is not. Who's liable?
According to Ireland's Medical Council, the answer is clear: the doctor is1.
The hospital's quarterly audit won't catch it for months. The AI vendor's logs show "functioning normally." Yet the Medical Council's October 2025 position paper states doctors "ultimately remain responsible for their clinical decisions"1.
The problem: How can doctors be confident in AI tools they don't fully understand? How can they maintain accountability when AI systems fail invisibly?
The accountability gap
AI is increasingly deployed across Irish healthcare. An Irish public-healthcare provider established a centre for AI in digital health2. Radiology departments use AI to flag abnormalities. Pathology labs employ AI for tissue analysis. Emergency departments use AI triage.
The Medical Council says AI should "augment, rather than replace, clinical decision-making"1. But this creates a gap:
•Doctors are responsible for AI-assisted decisions
•But AI systems are black boxes with opaque decision-making processes
•And traditional monitoring only catches failures after harm occurs
As Jantze Cotter, Executive Director of Regulatory Policy at the Medical Council, noted: "AI advancements hold great potential for the medical field but also introduce significant ethical, legal, regulatory and professional challenges"1.
Why traditional monitoring fails
Healthcare organisations monitor AI through periodic audits and retrospective reviews. These approaches miss critical failures:
Consider these failure modes that traditional monitoring misses:
| Failure Mode | Clinical Impact | Why Traditional Monitoring Misses It | How Aqta Catches It |
|---|---|---|---|
| Loop drift | AI gets stuck in repetitive diagnostic patterns | Only visible in aggregate trends over time | Every decision is signed and hash-chained into a receipt, so repeated patterns are traceable |
| Bias amplification | Underdiagnosis in underrepresented populations | Requires demographic analysis across cases | Every decision is signed and hash-chained, so outcomes can be reviewed across cases |
| Model drift | Accuracy degrades over time without warning | Performance changes are gradual and subtle | Model version signed on every receipt, so drift is traceable |
| Integration failures | AI receives incomplete or corrupted patient data | Data quality issues aren't logged systematically | Each decision's inputs are signed into a verifiable receipt you can export and audit |
The Medical Council acknowledges: "Doctors must have confidence in the standard of the tool they are utilising"1. Confidence requires visibility. Most healthcare AI systems operate as black boxes.
EU AI Act requirements
Healthcare AI falls under the EU AI Act's "high-risk" category. Requirements take effect 2 December 2027 under the Digital Omnibus3:
•Risk management systems with continuous monitoring
•Data governance ensuring training data quality and representativeness
•Technical documentation proving system reliability
•Human oversight with meaningful intervention capabilities
•Transparency enabling users to interpret outputs
•Accuracy, robustness and cybersecurity throughout the lifecycle
Non-compliance: up to €15 million or 3% of global annual turnover3.
Beyond compliance: How do you give doctors confidence while maintaining efficiency?
Healthcare AI accountability requirements
The Medical Council's principles map to technical requirements:
1. Transparency and auditability
"Patients must be informed when AI tools are used"1. Healthcare organisations need:
•Complete audit trails showing when AI was used, what data it processed, and what recommendations it made
•Explainability that clinicians can understand and communicate to patients
•Version tracking to identify which model version was used for each decision
2. Human-in-the-loop
"AI should augment, not replace, clinical decision-making"1:
•Mandatory review points where clinicians must actively confirm AI recommendations
•Override capabilities that preserve clinical judgment
•Escalation protocols when AI confidence is low or results are ambiguous
3. Bias detection
AI "could reinforce bias, particularly affecting vulnerable groups"1. A widely used US healthcare risk algorithm assigned Black patients lower risk scores than equally sick white patients, because it used past cost as a proxy for need4. Requirements:
•Demographic monitoring to detect performance disparities
•Regular bias audits across patient populations
•Diverse training data requirements
4. Real-time safeguards
Healthcare AI needs real-time protection, not just retrospective review:
•Loop detection to catch AI systems stuck in repetitive patterns
•Anomaly detection flagging unusual behaviour before harm occurs
•Automatic circuit breakers that pause AI systems when safety thresholds are breached
Ireland's position
Ireland is uniquely positioned: EU member state, thriving tech sector, English-speaking healthcare. Irish hospitals are early AI adopters.
Opportunity
Irish healthcare organisations that implement strong AI enforcement now can become exemplars for EU AI Act compliance.
Risk
Organisations that deploy AI without visibility into how it behaves face accountability gaps that are hard to close after the fact.
Closing the accountability gap
If doctors are accountable for AI-assisted decisions, they need real visibility into what those systems are doing, not a quarterly audit report. The requirements that follow from the Medical Council's principles and the EU AI Act are concrete. They resolve to four technical capabilities, each of which the regulation already names.
Real-time loop detection
Catches AI systems stuck in repetitive diagnostic patterns before they impact patient care. Automatic circuit breakers pause systems when safety thresholds are breached.
Complete audit trails
Every AI interaction is logged with timestamps, model versions, input data, and recommendations, meeting both Medical Council transparency requirements and EU AI Act documentation standards.
Human-in-the-loop workflows
Configurable review points ensure clinicians maintain authority over AI-assisted decisions. Override capabilities preserve clinical judgment while maintaining audit trails.
Bias and fairness monitoring
Track AI performance across demographic groups to detect and address disparities. Automated alerts when performance varies significantly by population.
Compliance reporting
Exportable, signed audit trails you can hand to a regulator.
Data sovereignty
Self-hosted deployment option keeps patient data within your infrastructure. EU-based hosting available for organisations requiring data residency.
The accountability gap in healthcare AI is real and it is not going to close by itself. The question is whether you build visibility in from the start or try to reconstruct it after something goes wrong.
Apply to the AqtaCore early access programme to see how we help healthcare organisations meet Medical Council guidance and EU AI Act requirements while maintaining clinical efficiency.
References
- Medical Council of Ireland. "Position Paper: Use of Artificial Intelligence in Medical Practice". 21 October 2025. Source
- An Irish public-healthcare provider. "Centre for AI and Digital Health Launch". Dublin, Ireland. 2025.
- European Parliament and Council. "Regulation (EU) 2024/1689 on Artificial Intelligence (AI Act)". Official Journal of the European Union, 12 July 2024. Source
- Obermeyer, Z., et al. "Dissecting racial bias in an algorithm used to manage the health of populations". Science, Vol. 366, Issue 6464, pp. 447-453, 2019. Source
About Aqta
We build the trust layer for AI in regulated industries: a signed, offline-verifiable receipt for every AI decision, across any model, that anyone can check without trusting us. Based in Dublin and Switzerland. More at about / research / manifesto.
Aqta on LinkedInRelated Articles
The human supply chain behind AI
As models become commodities, the real risk lives in the software and human supply chains beneath them.
Eight days before the WHO. AqtaBio's first prospective Ebola signal.
The W19 commit listed DR Congo Ebola at rank 4 of the Congo Basin tiles, eight days before WHO declared a Bundibugyo PHEIC.