Explainability
Also called: interpretability, XAI
The degree to which a model's reasoning can be understood and evaluated by a person, and a live regulatory and clinical concern.
Explainability covers techniques for making model behavior inspectable: saliency maps, feature importance, example-based explanations, and inherently interpretable model classes. It matters clinically because a clinician cannot appropriately weigh a recommendation they cannot evaluate.
Where This Gets Misread
Most post-hoc explanation methods explain the explanation method as much as the model. Saliency maps in particular have been shown to be unstable and sometimes independent of what the model actually used. An explanation that looks reasonable is not evidence the reasoning was, and it can manufacture unwarranted confidence.