The part of the image or text most ‘responsible’ for a given classification outcome is referred to by the term ‘salience’ or ‘saliency’. (TODO: this definition sucks.) A saliency map for an image classifier will show you which parts of the image were most ‘important’. This can be done either by looking at activations or gradients.

If you want to explain ‘why’ a certain outcome was chosen, salience maps can help.

Useful posts & references:

https://deepai.org/publication/efficient-saliency-maps-for-explainable-ai

https://medium.com/@j.jacobsen/deep-classifiers-ignore-almost-everything-they-see-and-how-we-may-be-able-to-fix-it-a6888012516f

Adebayo et al (incl. Goodfellow): “Sanity Checks for Salience Maps”. https://arxiv.org/abs/1810.03292

fastai does grad-cam