Causal insights:
adaptive explanations for interpretable probabilistic models
adaptive explanations for interpretable probabilistic models
This project is defined within the scope of the Hybrid Intelligence: Augmenting Human Intellect project. It addresses the Trust and Trustworthiness challenge and the Diabetes use case (see this matrix). The project is a collaboration between partners from Utrecht University and Leiden University.
What is our aim?This project aims to incorporate causality into explanations of probabilistic models with the purpose of providing meaningful explanations in a hybrid intelligence context where humans interact with an AI system that helps them understand the effects of their actions and how to improve upon them. Explanations are required in order for the human to understand and accept the output and trust the model in general.
Probabilistic models are powerful tools to represent and reason about uncertainty in complex high-dimensional domains. Employed both for data analysis and as tools for prediction and decision support, probabilistic models are widely used in scientific and application contexts. Probabilistic models, including their predictions, can however be challenging to understand for humans, limiting their adoption in high-stake domains such as medicine. Thus, a need exists for good explanation methods that provide understandable information for users.
Many probabilistic models can be cast as (probabilistic) graphical models (PGM), which enhances their interpretability by representing the statistical dependencies among their variables via a graph. Examples include Bayesian Networks (BNs) and Influence Diagrams (IDs). The latter extends Bayesian networks by explicitly incorporating decision and utility variables for computing optimal decisions under uncertainty. Although PGMs exist that build on undirected graphs, many use a directed acyclic graph to capture their dependency structure. These graphs are claimed to be quite intuitive for humans. The arcs in the graph, however, represent (symmetric) statistical associations and not necessarily (asymmetric) cause-effect relations, whereas humans tend to interpret them as the latter.
PGMs can be constructed in different ways: with the help of domain experts, through knowledge graphs and directly from data. The direction of arcs is often chosen by employing the notion of causality, both in communication with domain experts and upon using the so-called causal discovery algorithms which learn the network structure from data. As such, even though the PGMs might not be true causal models, to some extent, we can exploit the causal knowledge or hypothesised causal relations, in their computations and explanations.
In this project, we will introduce the concept of causality into the explanation of PGMs, by exploiting knowledge about causality (or lack thereof) taken into account in the construction of the networks, as well as in the computations performed for determining the explanations. Since the existence of true causal relations can often not be determined with certainty, we will design methods for establishing and explaining how robust the network predictions are to varying causal assumptions. We aim at explanations that have desirable properties, e.g., being contrastive, selective and interactive and adapted to the needs of different types of users. In addition, we aim to develop metrics for assessing the quality and robustness of explanations, especially in relation to the users' needs.
Why is this important?PGMs can be a valuable tool for decision support in complex domains. Yet, their adoption in high-stake domains such as medicine is still quite limited, partly due to a lack of good explanation methods that provide understandable information for users. Probabilities are hard to explain and even harder to interpret for users with no technical background, but causality is a concept closer to human intuition. Integrating causality into explanations will make it easier for humans to interact and collaborate with probabilistic systems and, as such, profit more from their decision-support capabilities in complex domains in which uncertainty is necessarily omnipresent.
In high-stake domains, decisions supported through AI systems can have a significant impact, and users therefore need to be able to contest and understand how the decisions are made. Such understanding is necessary to build trust. The demand for trustworthy AI is linked to the desire for explainable AI (XAI). Whereas XAI initially focussed on 'opening the black-box', people have by now realised that AI models that are considered interpretable require explanation as well. Designing and combining explanation techniques to provide a more comprehensive understanding of interpretable probabilistic models is therefore an emerging trend that aims to improve the transparency, interpretability and trustworthiness of such models, making them more accessible and understandable for users.
How will we approach this?In recent years, researchers have already leveraged causal concepts from higher-level models to design causal versions of measures such as Information Gain and Entropy used in lower-level models. We can directly employ these concepts in generating explanations as well as leverage additional causal concepts using similar approaches. The information measures can in addition be used to quantify the effect of possibly unjustly assuming the existence of causal variables and relations on the predictions of PGMs. We will then explore suitable ways of considering this in the explanations, and evaluate their suitability.
To study the effects of violating assumptions on the ability of causal discovery algorithms to reveal actual causal relations, we will do controlled experiments with data. These will likely be synthetic data sets generated for this purpose, but real or realistic datasets will be preferred.
We will review research into the use of causal concepts in explanations interpretable by humans with varying levels of expertise in a given domain. Based on this, we aim to define a quality measure for adaptive explanations for probabilistic models in a shared decision-making setting. In the context of the Diabetes use case, we can investigate what preferences potential users have about the type of causal knowledge they expect to see in an explanation. In addition, we would like to investigate to what extent the fact that some causal relations may be hypothesized rather than proven, influences the acceptability of the advice/prediction and its associated explanation.
Connection with matrix elementsThe project aligns with the vision of the HI challenge 'Trust and Trustworthiness' which advocates the creation of HI systems that are trustworthy on multiple levels: technical, social-interaction and socio-technical. This project contributes to at least the first two levels, and possibly the third, via evaluating to what extent we can trust relations to be causal and by providing explanations designed for humans.
The Diabetes use case is a clear motivation for the relevance of our project: a patient and physician ultimately have to decide together, in a well-informed fashion, on a lifestyle change that best fits the patient and benefits their health (also see: Viewpoint: Hybrid Intelligence Supports Application Development for Diabetes Lifestyle Management). The decision-support system should be able to provide the patient with explanations in terms of individual values and preferences of the patient, whereas the physician would be more interested in explanations involving medical factors. A third type of explanation is possibly required to support the shared (cooperative) decision-making. The current demo-system does not include an explanation module. If this is added, we can experiment with causality-based explanations that adapt to the user type. The demo currently also does not include probabilistic models, even if the application domain involves a lot of uncertainty. We see a possibility for using the currently used knowledge graphs as a basis for constructing the structure of a BN, hypothesising causal relations.