The Need for Explainability in Artificial Intelligence (AI)

“Why did the chicken cross the road?”

“Because AI predicted it would….”

In the above example, while it is undoubtedly impressive that AI correctly predicted the chicken would cross the road, the prediction alone provides no details about why or how the AI system arrived at its conclusion.

At first glance, information explaining the prediction might seem superfluous. After all, the AI system predicted the road crossing in advance of the event, providing us with an opportunity to prepare for and potentially prevent the event.

However, in order to react to the predicted event, it is important to understand how to act. We need to understand from the universe of actions that could be considered, what is an appropriate action that could prevent the event. In many instances, without this guidance the chosen action may, at best, have no impact, and, at worst, could even help precipitate or exacerbate the event. In a word, we need explainability. This explainability builds trust with the predictions and provides conviction to act.

While some simple AI models are intrinsically explainable, more complex models -- for example deep neural nets -- are frequently referred to as ‘black box’ models. In these models, we have limited insights into what feature(s) in the input caused the model to make the prediction it did. Nobody wants to make important decisions based on predictions that came from “magic”.

However, all is not lost, and there’s a growing list of freely available libraries that allow users to gain some understanding of why arbitrarily complex black box models are making the predictions they do.

In many of these approaches, the model is probed by taking the input associated with the prediction of interest, perturbing it and examining the impact on the prediction. By undertaking this perturbation numerous times, we can begin to understand what aspects of the input are driving the prediction. For example, returning to our initial example, if the state of the chicken coop door (open/closed) is an input feature, by probing the model once with the door closed and once with door open, we might find that the door being open is what drove the prediction of the road crossing event. With this insight, we can now formulate a plan of action, and move to rapidly close the gate.

By understanding what is driving the predictions, we can not only react appropriately to each event, but even start to formulate plans about how to react to broad classes of problems and prevent the likelihood of future escapes e.g. by undertaking employee training about the importance of gate closure, placing warning and reminder signs in front of the coop, and even installing an alarm on the door.

And while I have chosen to highlight the importance of explainability in a simple example, the necessity is there whether we build a chicken escape predictor, or a customer churn model, an employee attrition model, or even a credit card fraud predictor. Without explainability, all we have is a prediction of an event. With explainability, we have an understanding of the underlying causes, and can rapidly develop a targeted plan of action to effectively tackle the event.

At RSquared, we leverage AI to allow companies to understand the morale, engagement, collaboration and inclusion of their users and groups. And, unsurprisingly, model explainability is a first-order consideration. Every time a prediction (i.e. an “actionable insight”) is presented to the user, it is accompanied by a wealth of explanatory information allowing users to understand the driving factors and respond in an informed and targeted way.

Model explainability can’t be covered in a blog as short as this, but I presented on this topic in more detail last year at AI Dev World. The slides can be found here for folks that are interested in more details, along with links to useful explainability libraries on the last slide.