Classification Concept-based Explanation Tutorial¶
Welcome to this tutorial, our will be to obtain concept-based explanations starting from the beginning.
For any precision, please refer to the Interpreto documentation.
There are five key steps for concepts based explanations:
- ➗ Split your model in two parts
- 🚦 Compute a dataset of activations
- 🏋️♂️ Fit a concept model on activations
- 🏷️ Interpret the concept dimensions
- 🌍 Find the globally important concepts
On which we add three bonus steps:
- 📚 Class-wise concepts and LLM label
- 📍 Locally important concepts
- ⚖️ Evaluate concept-based explanations
Author: Antonin Poché
1. ➗ Split your model in two parts ¶
We choose a DistilBERT fine-tuned on the AG-News dataset and split it just before the classification head.
To split the model, we use the interpreto.SplitSequenceClassification which wraps around the transformers model.
It splits the model at the [CLS] token, thus the first part is the encoder and the second the classification head. Which is the setup we highly recommend for interpretability.
For other settings, we invite the user to use the interpreto.ModelWithSplitPoints instead (parent of the above class).
2. 🚦 Compute a datasets of activations ¶
We load the AG-News train set.
Then we extract the activations of the [CLS] token of each document.
from datasets import load_dataset
# load the AG-News dataset
dataset = load_dataset("fancyzhx/ag_news")
inputs = list(dataset["train"]["text"])
classes_names = dataset["train"].features["label"].names
# Compute the [CLS] token activations
activations, predictions = splitter.get_activations(inputs, tqdm_bar=True)
3. 🏋️♂️ Fit a concept model on activations ¶
With activations, we can train a concept model to find patterns (concepts).
The concept_model is an attribute of our concept explainer, similarly to the model_with_split_points. With these these two elements, we can go from inputs to concepts and from concepts to outputs.
In this tutorial, we use interpreto.concepts.ICAConcepts built upon the ICA (Independent Component Analysis) dimension reduction algorithm.
There are at least 15 others concept model available in interpreto. do not hesitate to explore them.
> 🔥 Tip
>
> ICAConcepts is a good first candidate for classification. It has no requirements, is fast, and provide correct first results on most datasets.
>
> Well the SemiNMFConcepts used in the better concepts section is too.
4. 🏷️ Interpret the concept dimensions ¶
We have our concepts and the link between concepts and classes. But now, we need to make sense of these concepts.
In this case, we will use the interpreto.concepts.interpretations.TopKInputs to find the 8 words which activates the most our concepts.
from interpreto.concepts.interpretations import TopKInputs
# instantiate the interpretation method with the concept explainer
topk_inputs_method = TopKInputs(
concept_explainer=concept_explainer,
k=5,
use_unique_words=3, # for classification we are force to use unique words or ngrams
unique_words_kwargs={
"count_min_threshold": round(
len(inputs) * 0.002
), # appear in at least 0.2% of the samples | increase if random words appear and decrease if some words appear too often
"lemmatize": True,
"words_to_ignore": [], # include noise words and punctuation
},
)
5. 🌍 Find the globally important concepts ¶
We have concept directions, it means that our model has access to them, but not that it uses them.
It is the same when you train a model on tabular data, not all features are used.
In this step, we use the ConceptAutoEncoderExplainer.concept_output_gradients to evaluate the importance of each concept with respect to the predicted classes.
> ➡️ Note
>
> All unsupervised concept-based explainers in Interpreto inherit from ConceptAutoEncoderExplainer.
> ➡️ Note 2
>
> This step can be done prior to the interpretation, as the interpretation step can be compute heavy. Then specify using the concept_indices parameter.
> Only interpreting the important concepts can be wise. (Here we only have 50 concepts, so it does not matter.)
import torch
# estimate the importance of concepts for each class using the gradient
gradients = concept_explainer.concept_output_gradient(
inputs=activations,
targets=None, # None means all classes
tqdm_bar=True,
)
# stack gradients on samples and average them over samples
mean_gradients = torch.stack(gradients).abs().squeeze().mean(0) # (num_classes, num_concepts)
# for each class, sort the importance scores
order = torch.argsort(mean_gradients, descending=True)
# visualize the top 5 concepts for each class
for target in range(order.shape[0]):
print(f"\nClass: {classes_names[target]}:")
for i in range(5):
concept_id = order[target, i].item()
importance = mean_gradients[target, concept_id].item()
words = list(topk_words.get(concept_id, None).keys())
print(f"\tconcept id: {concept_id},\timportance: {round(importance, 3)},\ttopk words: {words}")
> ❓ The concepts are not interpretable, what do I do?
>
> - Try to improve the concept-space:
> - Increases the number of samples. You can artificially do so by splitting then by sentences (not included)
> - Try different concept-models and parameters
> - Try to compute concepts class-wise see next section
>
> - Improve the interpretation of concepts:
> - Play with the parameters
> - Try LLMLabels see next section
>
> - Try to evaluate the concepts, to automatically find the best methods. Check this other tutorial: TODO
>
> - Never forget the faithfulness-plausibility trade-off of explanations
6. 📚 Better concepts with class-wise concepts and LLM labels ¶
This section aims at improving the concepts learned by the model. We try three different approaches:
- Training class-wise concepts
- Using another concept model:
SemiNMFConcepts - Using
interpreto.concepts.LLMLabelsto interpret the concepts
When a single concept-space is defined for all classes, concepts tend to correspond to the classes themselves. In particular, when the concept-space is built upon on the latent space just before the classification head.
In this section, we will learn a concept space for each class separately. Thus, the class-wise concept explainers will only see examples from a single class (based on the predictions).
> ⚠️ Warning
>
> The following cell and several others will not work with an OpenAI API key. As LLMLabels requires a LLMInterface, and we chose the OpenAILLM one.
>
> What you can do to make it work:
> - Get an OpenAI API key and set it as an environment variable OPENAI_API_KEY.
> - Use TopKInputs to replace LLMLabels.
> - Branch your own LLMInterface and use it instead of OpenAILLM. See last section for an example.
import os
from interpreto.commons.llm_interface import OpenAILLM
from interpreto.concepts import LLMLabels, SemiNMFConcepts
# Load API key from environment variable
api_key = os.getenv("OPENAI_API_KEY")
if api_key is None:
raise ValueError(
"An API key is required to use `OpenAILLM` interface. ",
"Cannot use LLMLabels without an LLM interface. ",
"See last section for an example of how to branch one.",
)
# set the LLM interface used to generate labels based on the constructed prompts
llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-nano")
concept_explainers = {}
concept_interpretations = {}
concept_importances = {}
# iterate over classes
for target, class_name in enumerate(classes_names):
# ----------------------------------------------------------------------------------------------
# 2. construct the dataset of activations (extract the ones related to the class)
indices = (predictions == target).nonzero(as_tuple=True)[0]
if not len(indices):
continue
class_wise_inputs = [inputs[i] for i in indices]
class_wise_activations = activations[indices]
# ----------------------------------------------------------------------------------------------
# 3. train concept model
concept_explainers[target] = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda")
concept_explainers[target].fit(class_wise_activations)
# ----------------------------------------------------------------------------------------------
# 5. compute concepts importance (before interpretations to limit the number of concepts interpreted)
gradients = concept_explainers[target].concept_output_gradient(
inputs=class_wise_activations,
targets=[target],
concepts_x_gradients=True,
batch_size=64,
)
# stack gradients on samples and average them over samples
concept_importances[target] = torch.stack(gradients, axis=0).squeeze().abs().mean(dim=0) # (num_concepts,)
# for each class, sort the importance scores
important_concept_indices = torch.argsort(concept_importances[target], descending=True).tolist()
# ----------------------------------------------------------------------------------------------
# 4. interpret the important concepts concepts
llm_labels_method = LLMLabels(
concept_explainer=concept_explainers[target],
llm_interface=llm_interface,
k_examples=20,
)
concept_interpretations[target] = llm_labels_method.interpret(
inputs=class_wise_inputs,
concepts_indices=important_concept_indices,
)
print(f"\nClass: {class_name}")
for concept_id in important_concept_indices[:5]:
label = concept_interpretations[target].get(concept_id, None)
importance = concept_importances[target][concept_id].item()
if label is not None:
print(f"\timportance: {round(importance, 3)},\t{label}")
6. 📍 Locally important concepts ¶
6.1 Local concepts-to-ouputs attributions¶
We got which concept are important for the classes globally. However, the concepts are not all present in each sample and the model might rely on a specific concept for a specific sample. Let's look at locally important concepts, meaning, the concepts the model used in a specific sample.
> ➡️ Note > > We cannot look at which word activates which concept without doing a forward pass for each word individually, because we use the [CLS] token. You could do the following cell, iterate on words and replace the example by the word.
example = "Bio-engineered shoes will revolutionized running throughout the world, as they cost only 50 dollars."
local_activations, local_predictions = splitter.get_activations([example])
pred = local_predictions.item()
concepts_activations = concept_explainers[pred].activations_to_concepts(local_activations)
print(f"Example: {example}")
print(f"Predicted class: {classes_names[pred]}")
# compute local concepts importance for the class
# we use the class-wise
local_importance = concept_explainers[pred].concept_output_gradient(
inputs=[example],
concepts_x_gradients=True,
)[0] # there is only one sample
plot_concepts(
# sample=[example],
classes_names=classes_names,
# concepts_activations=concepts_activations,
concepts_importances=local_importance.squeeze(), # importance of shape (t, g, c) -> (t, c)
concepts_labels=concept_interpretations,
)
6.2 Inputs-to-concepts attributions¶
This can be seen as an interpretation, but it works even better when concepts have labels as defined earlier. (Note that these concepts are not absolute.)
To obtain inputs-to-concepts attributions, we use the get_inputs_to_concepts_model() method from concepts explainers and give it to perturbation-based attribution methods.
from interpreto import KernelShap
attributions = {}
# Iterate on explainers because we did class-wise explanations
for class_id, cpt_explainer in concept_explainers.items():
attribution_explainer = KernelShap(
model=cpt_explainer.get_inputs_to_concepts_model(),
tokenizer=splitter.tokenizer,
)
# Compute the attributions
attribution_outputs = attribution_explainer.explain(
example,
targets=None, # we want to explain all concepts
)[0]
attributions[class_id] = attribution_outputs.attributions.T
# Visualization
plot_concepts(
sample=attribution_outputs.elements,
classes_names=classes_names, # TODO WARNING THE CLASS IS NOT CORRECT
concepts_activations=attributions, # argument name is miss-leading because we use concepts activations for generation
concepts_importances=local_importance.squeeze(), # importance of shape (t, g, c) -> (t, c)
concepts_labels=concept_interpretations,
)
7. ⚖️ Evaluate concept-based explanations ¶
We take back the ICAConcepts explainer and evaluate it on new samples.
7.1 🌐 Evaluate the concept-space from the third part¶
> ⚠️ Warning: > > These metrics should only be used to compare the concept-space trained in similar contexts, same model, split point, activation dataset...
Reconstruction error¶
> ➡️ Note > > Alone these values are useless, they should be compared between several concept explainers.
Dictionary metrics¶
The Stability metric requires two concept explainer, hence our first step step will be to train a new ICAConcepts with the same model, split, dataset, and hyper-parameters as the original ICAConcepts. However, to get a statistically robust metric score, one should compare more than just two instances of the same explainer.
from interpreto.concepts.metrics import Stability
# instantiate and train a second concept explainer
second_explainer = ICAConcepts(splitter, nb_concepts=50, device="cuda")
second_explainer.fit(activations)
stability = Stability(concept_explainer, second_explainer).compute()
del second_explainer
print(f"Stability: {round(stability, 3)}")
7.2 💭 Evaluate the concepts-interpretations from the fourth step¶
7.3 ↔️ Evaluate the whole concept-based explanations with ConSim¶
ConSim is a metric evaluating the whole concept-based explanations in an end-to-end manner. Indeed, this metric, evaluates to which extend the provided concept-based explanations help a meta-predictor to predict what the studied model would have predicted. The idea is that is a meta-predictor understands the model, it is able to predict what the model would have predicted on new samples.
> ➡️ Note > > For significant scores, we iterate on 10 different seeds. (5 were used in the paper).
from interpreto.concepts.metrics.consim import ConSim, PromptTypes
# convert step 5 global concept importances to a dictionary
global_importances = {
class_name: dict(enumerate(importances))
for class_name, importances in zip(classes_names, mean_gradients, strict=True)
}
# Load API key from environment variable
api_key = os.getenv("OPENAI_API_KEY")
if api_key is None:
raise ValueError(
"An API key is required to use `OpenAILLM` interface. ",
"Cannot use LLMLabels without an LLM interface. ",
"See last section for an example of how to branch one.",
)
# set the LLM interface used to generate labels based on the constructed prompts
llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-nano")
# Initialize the ConSim with the split model and the user LLM
# Therefore, a given ConSim metric can be used on different explainers for cleaner comparison
con_sim = ConSim(
splitter,
llm_interface,
classes=classes_names,
)
baseline_list = []
ica_score_list = []
for seed in range(10):
# Select examples for evaluation
samples, labels, predictions = con_sim._extract_interesting_elements(
inputs=test_inputs,
labels=test_labels,
predictions=test_predictions,
seed=seed,
)
# Compute a baseline and ConSim score to give sense to the explainer ConSim score
baseline = con_sim.evaluate(
interesting_samples=samples, predictions=predictions, prompt_type=PromptTypes.L2_baseline_with_lp
)
if baseline is None:
continue
# Compute the ConSim score for an explainer
ica_score = con_sim.evaluate(
interesting_samples=samples,
predictions=predictions,
concept_explainer=concept_explainer,
concepts_interpretation=topk_words,
global_importances=global_importances,
prompt_type=PromptTypes.E2_global_concepts_with_lp,
)
if ica_score is None:
continue
baseline_list.append(baseline)
ica_score_list.append(ica_score)
print(f"Baseline: {round(sum(baseline_list) / 10, 2)}, ICA: {round(sum(ica_score_list) / 10, 2)}")
> ➡️ Note > > We evaluated the first ICA explainer here. Concepts where not really interpretable, ConSim agrees.
8. Using your own LLM interface ¶
from interpreto.commons.llm_interface import LLMInterface, Role
class GeminiLLM(LLMInterface):
def __init__(self, api_key: str, model: str = "gemini-1.5-flash", num_try: int = 5):
try:
import google.generativeai as genai # noqa: PLC0415 # ruff: disable=import-outside-toplevel
except ImportError as e:
raise ImportError("Install google-generativeai to use Google Gemini API.") from e
self.genai = genai
self.genai.configure(api_key=api_key)
self.model = model
self.num_try = num_try
def generate(self, prompt: list[tuple[Role, str]]) -> str | None:
# Build system instruction and chat history for Gemini
system_messages: list[str] = []
contents: list[dict] = []
for role, content in prompt:
if role == Role.SYSTEM:
system_messages.append(content)
elif role == Role.USER:
contents.append(
{
"role": "user",
"parts": [{"text": content}],
}
)
elif role == Role.ASSISTANT:
contents.append(
{
"role": "model",
"parts": [{"text": content}],
}
)
else:
raise ValueError(f"Unknown role for google gemini api: {role}")
system_instruction: str | None = "\n".join(system_messages) if system_messages else None
label: str | None = None
for _ in range(self.num_try):
try:
model = self.genai.GenerativeModel(
model_name=self.model,
system_instruction=system_instruction,
)
response = model.generate_content(contents) # type: ignore[arg-type]
# google-generativeai exposes the main text as .text
label = response.text
break
except Exception as e: # noqa: BLE001
print(e)
return label