Trading off accuracy and explainability in AI decision-making: findings from 2 citizens' juries | JoVE Visualize

Trading off accuracy and explainability in AI decision-making: findings from 2 citizens' juries

Sabine N van der Veer¹, Lisa Riste^2,3, Sudeh Cheraghi-Sohi^2,4

¹Centre for Health Informatics, Division of Informatics, Imaging and Data Science, Manchester Academic Health Science Centre, The University of Manchester, Manchester, UK.

Journal of the American Medical Informatics Association : JAMIA|August 1, 2021

View abstract on PubMed

Summary

The public may prioritize AI accuracy over explainability in healthcare, unlike in non-healthcare settings. This finding highlights the need for public input in developing AI explainability policies.

Area of Science:

Artificial Intelligence (AI)
Public Policy
Human-Computer Interaction

Background:

Public perception of artificial intelligence (AI) systems is crucial for their ethical development and deployment.
Understanding public trade-offs between AI accuracy and explainability is essential, particularly in sensitive domains like healthcare.
Existing research often assumes a high public demand for AI explainability, which may not hold true across different contexts.

Purpose of the Study:

To investigate public preferences regarding the trade-off between accuracy and explainability in artificial intelligence (AI) systems.
To determine if these preferences differ between healthcare and non-healthcare applications.
To inform policy development concerning AI explainability.

Main Methods:

Two 5-day citizens' juries were conducted in the UK, involving 18 jurors each.
Jurors evaluated three AI systems with varying accuracy and explainability levels across two healthcare and two non-healthcare scenarios.
Quantitative voting data and qualitative thematic analysis of juror deliberations were used to assess preferences.

Main Results:

In healthcare scenarios, jurors prioritized AI system accuracy over explainability.
In non-healthcare scenarios, jurors valued explainability equally to or more than accuracy.
Reasons for favoring accuracy included impact on individuals/society and service efficiency; reasons for valuing explainability included learning, improvement, and bias detection.

Conclusions:

Public valuation of AI explainability may be lower in healthcare than in non-healthcare contexts, contrary to professional assumptions.
These findings underscore the importance of actively consulting the public when formulating policies on AI explainability.
Engaging the public ensures AI systems align with societal values and expectations across diverse application domains.