{ style="display:block; max-width:100%; height:auto; margin:0 auto;" }

Classification Concept-based Explanation Tutorial¶

Welcome to this tutorial, our will be to obtain concept-based explanations starting from the beginning.

For any precision, please refer to the Interpreto documentation.

There are five key steps for concepts based explanations:

1. ➗ Split your model in two parts
2. 🚦 Compute a dataset of activations
3. 🏋️‍♂️ Fit a concept model on activations
4. 🏷️ Interpret the concept dimensions
5. 🌍 Find the globally important concepts

On which we add three bonus steps:

6. 📚 Class-wise concepts and LLM label
7. 📍 Locally important concepts
8. ⚖️ Evaluate concept-based explanations

Author: Antonin Poché

In [1]:

import torch

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

import torch DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

1. ➗ Split your model in two parts ¶

We choose a DistilBERT fine-tuned on the AG-News dataset and split it just before the classification head.

To split the model, we use the interpreto.SplitSequenceClassification which wraps around the transformers model.

It splits the model at the [CLS] token, thus the first part is the encoder and the second the classification head. Which is the setup we highly recommend for interpretability.

For other settings, we invite the user to use the interpreto.ModelWithSplitPoints instead (parent of the above class).

In [2]:

from interpreto import SplitterForClassification

splitter = SplitterForClassification(
    model_or_repo_id="textattack/distilbert-base-uncased-ag-news",
    device_map="cuda",
    batch_size=64,
)

from interpreto import SplitterForClassification splitter = SplitterForClassification( model_or_repo_id="textattack/distilbert-base-uncased-ag-news", device_map="cuda", batch_size=64, )

2. 🚦 Compute a datasets of activations ¶

We load the AG-News train set.

Then we extract the activations of the [CLS] token of each document.

interpreto.SplitSequenceClassification.get_activations()

In [3]:

from datasets import load_dataset

# load the AG-News dataset
dataset = load_dataset("fancyzhx/ag_news")
inputs = list(dataset["train"]["text"])
classes_names = dataset["train"].features["label"].names

# Compute the [CLS] token activations
activations, predictions = splitter.get_activations(inputs, tqdm_bar=True)

from datasets import load_dataset # load the AG-News dataset dataset = load_dataset("fancyzhx/ag_news") inputs = list(dataset["train"]["text"]) classes_names = dataset["train"].features["label"].names # Compute the [CLS] token activations activations, predictions = splitter.get_activations(inputs, tqdm_bar=True)

  0%|          | 0/1875 [00:00<?, ?it/s]

Loading weights:   0%|          | 0/104 [00:00<?, ?it/s]

100%|██████████| 1875/1875 [01:09<00:00, 26.80it/s]

3. 🏋️‍♂️ Fit a concept model on activations ¶

With activations, we can train a concept model to find patterns (concepts).

The concept_model is an attribute of our concept explainer, similarly to the model_with_split_points. With these these two elements, we can go from inputs to concepts and from concepts to outputs.

In this tutorial, we use interpreto.concepts.ICAConcepts built upon the ICA (Independent Component Analysis) dimension reduction algorithm.

There are at least 15 others concept model available in interpreto. do not hesitate to explore them.

🔥 Tip

ICAConcepts is a good first candidate for classification. It has no requirements, is fast, and provide correct first results on most datasets.

Well the SemiNMFConcepts used in the better concepts section is too.

In [4]:

from interpreto.concepts import ICAConcepts

# instantiate the concept explainer
concept_explainer = ICAConcepts(splitter, nb_concepts=50, device="cuda")

# fit the concept explainer on activations
concept_explainer.fit(activations)

from interpreto.concepts import ICAConcepts # instantiate the concept explainer concept_explainer = ICAConcepts(splitter, nb_concepts=50, device="cuda") # fit the concept explainer on activations concept_explainer.fit(activations)

4. 🏷️ Interpret the concept dimensions ¶

We have our concepts and the link between concepts and classes. But now, we need to make sense of these concepts.

In this case, we will use the interpreto.concepts.interpretations.TopKInputs to find the 8 words which activates the most our concepts.

In [5]:

from interpreto.concepts.interpretations import TopKInputs

# instantiate the interpretation method with the concept explainer
topk_inputs_method = TopKInputs(
    concept_explainer=concept_explainer,
    k=5,
    use_unique_words=3,  # for classification we are force to use unique words or ngrams
    unique_words_kwargs={
        "count_min_threshold": round(
            len(inputs) * 0.002
        ),  # appear in at least 0.2% of the samples | increase if random words appear and decrease if some words appear too often
        "lemmatize": True,
        "words_to_ignore": [],  # include noise words and punctuation
    },
)

from interpreto.concepts.interpretations import TopKInputs # instantiate the interpretation method with the concept explainer topk_inputs_method = TopKInputs( concept_explainer=concept_explainer, k=5, use_unique_words=3, # for classification we are force to use unique words or ngrams unique_words_kwargs={ "count_min_threshold": round( len(inputs) * 0.002 ), # appear in at least 0.2% of the samples | increase if random words appear and decrease if some words appear too often "lemmatize": True, "words_to_ignore": [], # include noise words and punctuation }, )

In [6]:

# call the interpretation methods on the inputs
# we cannot give the previously computed activations because `use_unique_words=True` creates samples with a single word inside
topk_words = topk_inputs_method.interpret(
    inputs=inputs,
    concepts_indices="all",
)

# call the interpretation methods on the inputs # we cannot give the previously computed activations because `use_unique_words=True` creates samples with a single word inside topk_words = topk_inputs_method.interpret( inputs=inputs, concepts_indices="all", )

5. 🌍 Find the globally important concepts ¶

We have concept directions, it means that our model has access to them, but not that it uses them.

It is the same when you train a model on tabular data, not all features are used.

In this step, we use the ConceptAutoEncoderExplainer.concept_output_gradients to evaluate the importance of each concept with respect to the predicted classes.

➡️ Note

All unsupervised concept-based explainers in Interpreto inherit from ConceptAutoEncoderExplainer.

➡️ Note 2

This step can be done prior to the interpretation, as the interpretation step can be compute heavy. Then specify using the concept_indices parameter. Only interpreting the important concepts can be wise. (Here we only have 50 concepts, so it does not matter.)

In [7]:

import torch

# estimate the importance of concepts for each class using the gradient
gradients = concept_explainer.concept_output_gradient(
    inputs=activations,
    targets=None,  # None means all classes
    tqdm_bar=True,
)

# stack gradients on samples and average them over samples
mean_gradients = torch.stack(gradients).abs().squeeze().mean(0)  # (num_classes, num_concepts)

# for each class, sort the importance scores
order = torch.argsort(mean_gradients, descending=True)

# visualize the top 5 concepts for each class
for target in range(order.shape[0]):
    print(f"\nClass: {classes_names[target]}:")
    for i in range(5):
        concept_id = order[target, i].item()
        importance = mean_gradients[target, concept_id].item()
        words = list(topk_words.get(concept_id, None).keys())
        print(f"\tconcept id: {concept_id},\timportance: {round(importance, 3)},\ttopk words: {words}")

import torch # estimate the importance of concepts for each class using the gradient gradients = concept_explainer.concept_output_gradient( inputs=activations, targets=None, # None means all classes tqdm_bar=True, ) # stack gradients on samples and average them over samples mean_gradients = torch.stack(gradients).abs().squeeze().mean(0) # (num_classes, num_concepts) # for each class, sort the importance scores order = torch.argsort(mean_gradients, descending=True) # visualize the top 5 concepts for each class for target in range(order.shape[0]): print(f"\nClass: {classes_names[target]}:") for i in range(5): concept_id = order[target, i].item() importance = mean_gradients[target, concept_id].item() words = list(topk_words.get(concept_id, None).keys()) print(f"\tconcept id: {concept_id},\timportance: {round(importance, 3)},\ttopk words: {words}")

100%|██████████| 1875/1875 [00:02<00:00, 710.88it/s]

Class: World:
	concept id: 4,	importance: 0.085,	topk words: ['baghdad ,', 'hamid karzai', 'al-sadr', 'of baghdad', 'baghdad']
	concept id: 42,	importance: 0.049,	topk words: ['foreign minister', 'west bank', 'iraq and', 'darfur', 'in darfur']
	concept id: 12,	importance: 0.049,	topk words: ['security council', 'the un', 'european union', 'european commission', 'china # 39']
	concept id: 11,	importance: 0.045,	topk words: ['al-sadr', 'cbs.mw ) --', 'of iraq', 'iraq', 'iraq -']
	concept id: 31,	importance: 0.045,	topk words: ['vodafone', 'verizon', 'oracle corp.', '( nasdaq :', 'google inc.']

Class: Sports:
	concept id: 8,	importance: 0.087,	topk words: [', fla.', 'gt ; ...', 'west bank', 'on wednesday .', 'on saturday .']
	concept id: 41,	importance: 0.084,	topk words: ['nl', 'ap ) ap', '; a href=', 'pacer', 'real madrid']
	concept id: 21,	importance: 0.072,	topk words: [', fla.', 'west bank', 'cbs.mw ) --', ', quot ;', '. quot ;']
	concept id: 19,	importance: 0.067,	topk words: ['real madrid', 'manchester united', 'arsenal', 'chelsea', ', quot ;']
	concept id: 20,	importance: 0.066,	topk words: ['. quot ;', ', quot ;', 'third-quarter', 'on tuesday ,', 'the third quarter']

Class: Business:
	concept id: 33,	importance: 0.053,	topk words: ['shopper', 'wal-mart', 'oracle corp.', 'store inc.', 'prime minister tony']
	concept id: 16,	importance: 0.052,	topk words: ['# 151 ;', 'pfizer', 'energy', 'gt ; ,', '; ,']
	concept id: 31,	importance: 0.052,	topk words: ['vodafone', 'verizon', 'oracle corp.', '( nasdaq :', 'google inc.']
	concept id: 8,	importance: 0.045,	topk words: [', fla.', 'gt ; ...', 'west bank', 'on wednesday .', 'on saturday .']
	concept id: 20,	importance: 0.042,	topk words: ['. quot ;', ', quot ;', 'third-quarter', 'on tuesday ,', 'the third quarter']

Class: Sci/Tech:
	concept id: 8,	importance: 0.054,	topk words: [', fla.', 'gt ; ...', 'west bank', 'on wednesday .', 'on saturday .']
	concept id: 41,	importance: 0.048,	topk words: ['nl', 'ap ) ap', '; a href=', 'pacer', 'real madrid']
	concept id: 21,	importance: 0.043,	topk words: [', fla.', 'west bank', 'cbs.mw ) --', ', quot ;', '. quot ;']
	concept id: 19,	importance: 0.041,	topk words: ['real madrid', 'manchester united', 'arsenal', 'chelsea', ', quot ;']
	concept id: 39,	importance: 0.041,	topk words: ['peoplesoft inc.', 'saturn', 'microsoft corp.', 'hewlett-packard', 'oracle corp.']

In [8]:

from interpreto import plot_concepts

labels = {k: list(v.keys()) for k, v in topk_words.items()}

plot_concepts(
    classes_names=classes_names,
    concepts_importances=mean_gradients,
    concepts_labels=labels,
)

from interpreto import plot_concepts labels = {k: list(v.keys()) for k, v in topk_words.items()} plot_concepts( classes_names=classes_names, concepts_importances=mean_gradients, concepts_labels=labels, )

Classes

Concepts

❓ The concepts are not interpretable, what do I do?

• Try to improve the concept-space:

  • Increases the number of samples. You can artificially do so by splitting then by sentences (not included)
  • Try different concept-models and parameters
  • Try to compute concepts class-wise see next section

• Improve the interpretation of concepts:

  • Play with the parameters
  • Try LLMLabels see next section

• Try to evaluate the concepts to automatically find the best methods.

• Never forget the faithfulness-plausibility trade-off of explanations

6. 📚 Better concepts with class-wise concepts and LLM labels ¶

This section aims at improving the concepts learned by the model. We try three different approaches:

• Training class-wise concepts
• Using another concept model: SemiNMFConcepts
• Using interpreto.concepts.LLMLabels to interpret the concepts

When a single concept-space is defined for all classes, concepts tend to correspond to the classes themselves. In particular, when the concept-space is built upon on the latent space just before the classification head.

In this section, we will learn a concept space for each class separately. Thus, the class-wise concept explainers will only see examples from a single class (based on the predictions).

⚠️ Warning

The following cell and several others will not work with an OpenAI API key. As LLMLabels requires a LLMInterface, and we chose the OpenAILLM one.

What you can do to make it work:

• Get an OpenAI API key and set it as an environment variable OPENAI_API_KEY.
• Use TopKInputs to replace LLMLabels.
• Branch your own LLMInterface and use it instead of OpenAILLM. See last section for an example.

In [19]:

import os

from interpreto.commons.llm_interface import OpenAILLM
from interpreto.concepts import LLMLabels, SemiNMFConcepts

# Load API key from environment variable
api_key = os.getenv("OPENAI_API_KEY")
if api_key is None:
    raise ValueError(
        "An API key is required to use `OpenAILLM` interface. ",
        "Cannot use LLMLabels without an LLM interface. ",
        "See last section for an example of how to branch one.",
    )

# set the LLM interface used to generate labels based on the constructed prompts
llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-nano")

concept_explainers = {}
concept_interpretations = {}
concept_importances = {}

# iterate over classes
for target, class_name in enumerate(classes_names):
    # ----------------------------------------------------------------------------------------------
    # 2. construct the dataset of activations (extract the ones related to the class)
    indices = (predictions == target).nonzero(as_tuple=True)[0]
    if not len(indices):
        continue
    class_wise_inputs = [inputs[i] for i in indices]
    class_wise_activations = activations[indices]

    # ----------------------------------------------------------------------------------------------
    # 3. train concept model
    concept_explainers[target] = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda")
    concept_explainers[target].fit(class_wise_activations)

    # ----------------------------------------------------------------------------------------------
    # 5. compute concepts importance (before interpretations to limit the number of concepts interpreted)
    gradients = concept_explainers[target].concept_output_gradient(
        inputs=class_wise_activations,
        targets=[target],
        concepts_x_gradients=True,
        batch_size=64,
    )

    # stack gradients on samples and average them over samples
    concept_importances[target] = torch.stack(gradients, axis=0).squeeze().abs().mean(dim=0)  # (num_concepts,)

    # for each class, sort the importance scores
    important_concept_indices = torch.argsort(concept_importances[target], descending=True).tolist()

    # ----------------------------------------------------------------------------------------------
    # 4. interpret the important concepts concepts
    llm_labels_method = LLMLabels(
        concept_explainer=concept_explainers[target],
        llm_interface=llm_interface,
        k_examples=20,
    )

    concept_interpretations[target] = llm_labels_method.interpret(
        inputs=class_wise_inputs,
        concepts_indices=important_concept_indices,
    )

    print(f"\nClass: {class_name}")
    for concept_id in important_concept_indices[:5]:
        label = concept_interpretations[target].get(concept_id, None)
        importance = concept_importances[target][concept_id].item()
        if label is not None:
            print(f"\timportance: {round(importance, 3)},\t{label}")

import os from interpreto.commons.llm_interface import OpenAILLM from interpreto.concepts import LLMLabels, SemiNMFConcepts # Load API key from environment variable api_key = os.getenv("OPENAI_API_KEY") if api_key is None: raise ValueError( "An API key is required to use `OpenAILLM` interface. ", "Cannot use LLMLabels without an LLM interface. ", "See last section for an example of how to branch one.", ) # set the LLM interface used to generate labels based on the constructed prompts llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-nano") concept_explainers = {} concept_interpretations = {} concept_importances = {} # iterate over classes for target, class_name in enumerate(classes_names): # ---------------------------------------------------------------------------------------------- # 2. construct the dataset of activations (extract the ones related to the class) indices = (predictions == target).nonzero(as_tuple=True)[0] if not len(indices): continue class_wise_inputs = [inputs[i] for i in indices] class_wise_activations = activations[indices] # ---------------------------------------------------------------------------------------------- # 3. train concept model concept_explainers[target] = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda") concept_explainers[target].fit(class_wise_activations) # ---------------------------------------------------------------------------------------------- # 5. compute concepts importance (before interpretations to limit the number of concepts interpreted) gradients = concept_explainers[target].concept_output_gradient( inputs=class_wise_activations, targets=[target], concepts_x_gradients=True, batch_size=64, ) # stack gradients on samples and average them over samples concept_importances[target] = torch.stack(gradients, axis=0).squeeze().abs().mean(dim=0) # (num_concepts,) # for each class, sort the importance scores important_concept_indices = torch.argsort(concept_importances[target], descending=True).tolist() # ---------------------------------------------------------------------------------------------- # 4. interpret the important concepts concepts llm_labels_method = LLMLabels( concept_explainer=concept_explainers[target], llm_interface=llm_interface, k_examples=20, ) concept_interpretations[target] = llm_labels_method.interpret( inputs=class_wise_inputs, concepts_indices=important_concept_indices, ) print(f"\nClass: {class_name}") for concept_id in important_concept_indices[:5]: label = concept_interpretations[target].get(concept_id, None) importance = concept_importances[target][concept_id].item() if label is not None: print(f"\timportance: {round(importance, 3)},\t{label}")

Class: World
	importance: 0.081,	Event-focused, third-person news reporting.
	importance: 0.076,	Military reporting patterns.
	importance: 0.07,	Aggressive, crime-related, global or political themes, and focus on terrorism or security.
	importance: 0.069,	Event emphasis on hostage releases and geopolitical tensions.
	importance: 0.066,	Data reflects political, social, and ethnic themes with patterns of elections, protests, and minority issues.

Class: Sports
	importance: 0.075,	Repetitive sports and news reporting language, with frequent mention of names, scores, and events.
	importance: 0.073,	Sports reporting focuses on events, scores, and player movements.
	importance: 0.072,	Structured sports and news reports with embedded factual details.
	importance: 0.07,	Scores and events.
	importance: 0.069,	Concise race and sports event descriptions, utilizing brief summaries and key details.

Class: Business
	importance: 0.094,	Financial and corporate events center on numerical data, specific terminology, and formal reporting.
	importance: 0.084,	Structured, financial news summaries with embedded hyperlinks.
	importance: 0.078,	Consistent use of non-standard character sequences, especially "#39;", replacing apostrophes, and irregular formatting.
	importance: 0.076,	Consistent mentions of acquisitions, bids, and mergers involving corporations and assets, often linked to specific countries and financial figures.
	importance: 0.073,	Global commodity and energy market shifts; industry and corporate financials.

Class: Sci/Tech
	importance: 0.098,	Technical innovation focus
	importance: 0.085,	Technological, informative language with frequent references to innovation, products, and research.
	importance: 0.075,	Consistent focus on handheld gaming devices; mentions of release dates, prices, sales, and market rivalry.
	importance: 0.067,	Concise style, technical vocabulary, objectivity
	importance: 0.062,	Fact-based, formal reporting style; frequent use of specific details and proper nouns; consistent IMF-style sentence structure; focus on significant events and official statements.

In [20]:

plot_concepts(
    classes_names=classes_names,
    concepts_importances=concept_importances,
    concepts_labels=concept_interpretations,
)

plot_concepts( classes_names=classes_names, concepts_importances=concept_importances, concepts_labels=concept_interpretations, )

Classes

Concepts

6. 📍 Locally important concepts ¶

6.1 Local concepts-to-ouputs attributions¶

We got which concept are important for the classes globally. However, the concepts are not all present in each sample and the model might rely on a specific concept for a specific sample. Let's look at locally important concepts, meaning, the concepts the model used in a specific sample.

➡️ Note

We cannot look at which word activates which concept without doing a forward pass for each word individually, because we use the [CLS] token. You could do the following cell, iterate on words and replace the example by the word.

In [21]:

example = "Bio-engineered shoes will revolutionized running throughout the world, as they cost only 50 dollars."

local_activations, local_predictions = splitter.get_activations([example])

pred = local_predictions.item()
concepts_activations = concept_explainers[pred].activations_to_concepts(local_activations)

print(f"Example: {example}")
print(f"Predicted class: {classes_names[pred]}")

# compute local concepts importance for the class
# we use the class-wise
local_importance = concept_explainers[pred].concept_output_gradient(
    inputs=[example],
    concepts_x_gradients=True,
)[0]  # there is only one sample

plot_concepts(
    # sample=[example],
    classes_names=classes_names,
    # concepts_activations=concepts_activations,
    concepts_importances=local_importance.squeeze(),  # importance of shape (t, g, c) -> (t, c)
    concepts_labels=concept_interpretations,
)

example = "Bio-engineered shoes will revolutionized running throughout the world, as they cost only 50 dollars." local_activations, local_predictions = splitter.get_activations([example]) pred = local_predictions.item() concepts_activations = concept_explainers[pred].activations_to_concepts(local_activations) print(f"Example: {example}") print(f"Predicted class: {classes_names[pred]}") # compute local concepts importance for the class # we use the class-wise local_importance = concept_explainers[pred].concept_output_gradient( inputs=[example], concepts_x_gradients=True, )[0] # there is only one sample plot_concepts( # sample=[example], classes_names=classes_names, # concepts_activations=concepts_activations, concepts_importances=local_importance.squeeze(), # importance of shape (t, g, c) -> (t, c) concepts_labels=concept_interpretations, )

Example: Bio-engineered shoes will revolutionized running throughout the world, as they cost only 50 dollars.
Predicted class: Sci/Tech

Classes

Concepts

6.2 Inputs-to-concepts attributions¶

This can be seen as an interpretation, but it works even better when concepts have labels as defined earlier. (Note that these concepts are not absolute.)

To obtain inputs-to-concepts attributions, we use the get_inputs_to_concepts_model() method from concepts explainers and give it to perturbation-based attribution methods.

In [22]:

from interpreto import KernelShap

attributions = {}
# Iterate on explainers because we did class-wise explanations
for class_id, cpt_explainer in concept_explainers.items():
    attribution_explainer = KernelShap(
        model=cpt_explainer.get_inputs_to_concepts_model(),
        tokenizer=splitter.tokenizer,
    )

    # Compute the attributions
    attribution_outputs = attribution_explainer.explain(
        example,
        targets=None,  # we want to explain all concepts
    )[0]
    attributions[class_id] = attribution_outputs.attributions.T

# Visualization
plot_concepts(
    sample=attribution_outputs.elements,
    classes_names=classes_names,  # TODO WARNING THE CLASS IS NOT CORRECT
    concepts_activations=attributions,  # argument name is miss-leading because we use concepts activations for generation
    concepts_importances=local_importance.squeeze(),  # importance of shape (t, g, c) -> (t, c)
    concepts_labels=concept_interpretations,
)

from interpreto import KernelShap attributions = {} # Iterate on explainers because we did class-wise explanations for class_id, cpt_explainer in concept_explainers.items(): attribution_explainer = KernelShap( model=cpt_explainer.get_inputs_to_concepts_model(), tokenizer=splitter.tokenizer, ) # Compute the attributions attribution_outputs = attribution_explainer.explain( example, targets=None, # we want to explain all concepts )[0] attributions[class_id] = attribution_outputs.attributions.T # Visualization plot_concepts( sample=attribution_outputs.elements, classes_names=classes_names, # TODO WARNING THE CLASS IS NOT CORRECT concepts_activations=attributions, # argument name is miss-leading because we use concepts activations for generation concepts_importances=local_importance.squeeze(), # importance of shape (t, g, c) -> (t, c) concepts_labels=concept_interpretations, )

Classes

Concepts

Sample

7. ⚖️ Evaluate concept-based explanations ¶

We take back the ICAConcepts explainer and evaluate it on new samples.

In [23]:

test_inputs = dataset["test"]["text"][:1000]  # let's take one thousand test samples
test_labels = torch.tensor(dataset["test"]["label"][:1000])

# Compute the [CLS] token activations
test_activations, test_predictions = splitter.get_activations(inputs=test_inputs)

test_inputs = dataset["test"]["text"][:1000] # let's take one thousand test samples test_labels = torch.tensor(dataset["test"]["label"][:1000]) # Compute the [CLS] token activations test_activations, test_predictions = splitter.get_activations(inputs=test_inputs)

7.1 🌐 Evaluate the concept-space from the third part¶

⚠️ Warning:

These metrics should only be used to compare the concept-space trained in similar contexts, same model, split point, activation dataset...

Reconstruction error¶

• interpreto.concepts.metrics.MSE
• interpreto.concepts.metrics.FID

In [24]:

from interpreto.concepts.metrics import FID, MSE

mse = MSE(concept_explainer).compute(test_activations)
fid = FID(concept_explainer).compute(test_activations)

print(f"MSE: {round(mse, 3)}, FID: {round(fid, 3)}")

from interpreto.concepts.metrics import FID, MSE mse = MSE(concept_explainer).compute(test_activations) fid = FID(concept_explainer).compute(test_activations) print(f"MSE: {round(mse, 3)}, FID: {round(fid, 3)}")

MSE: 71.467, FID: 0.026

➡️ Note

Alone these values are useless, they should be compared between several concept explainers.

Sparsity¶

• interpreto.concepts.metrics.Sparsity
• interpreto.concepts.metrics.SparsityRatio

In [25]:

from interpreto.concepts.metrics import Sparsity, SparsityRatio

sparsity = Sparsity(concept_explainer).compute(test_activations)
ratio = SparsityRatio(concept_explainer).compute(test_activations)

print(f"Sparsity: {round(sparsity, 3)}, Sparsity ratio: {round(ratio, 3)}")

from interpreto.concepts.metrics import Sparsity, SparsityRatio sparsity = Sparsity(concept_explainer).compute(test_activations) ratio = SparsityRatio(concept_explainer).compute(test_activations) print(f"Sparsity: {round(sparsity, 3)}, Sparsity ratio: {round(ratio, 3)}")

Sparsity: 1.0, Sparsity ratio: 0.02

Dictionary metrics¶

• interpreto.concepts.metrics.Stability

The Stability metric requires two concept explainer, hence our first step step will be to train a new ICAConcepts with the same model, split, dataset, and hyper-parameters as the original ICAConcepts. However, to get a statistically robust metric score, one should compare more than just two instances of the same explainer.

In [26]:

from interpreto.concepts.metrics import Stability

# instantiate and train a second concept explainer
second_explainer = ICAConcepts(splitter, nb_concepts=50, device="cuda")
second_explainer.fit(activations)

stability = Stability(concept_explainer, second_explainer).compute()
del second_explainer

print(f"Stability: {round(stability, 3)}")

from interpreto.concepts.metrics import Stability # instantiate and train a second concept explainer second_explainer = ICAConcepts(splitter, nb_concepts=50, device="cuda") second_explainer.fit(activations) stability = Stability(concept_explainer, second_explainer).compute() del second_explainer print(f"Stability: {round(stability, 3)}")

Stability: 1.0

7.2 💭 Evaluate the concepts-interpretations from the fourth step¶

In [17]:

# Work in progress, coming soon

# Work in progress, coming soon

7.3 ↔️ Evaluate the whole concept-based explanations with ConSim¶

ConSim is a metric evaluating the whole concept-based explanations in an end-to-end manner. Indeed, this metric, evaluates to which extend the provided concept-based explanations help a meta-predictor to predict what the studied model would have predicted. The idea is that is a meta-predictor understands the model, it is able to predict what the model would have predicted on new samples.

• interpreto.concepts.metrics.ConSim

➡️ Note

For significant scores, we iterate on 10 different seeds. (5 were used in the paper).

In [27]:

from interpreto.concepts.metrics.consim import ConSim, PromptTypes

# convert step 5 global concept importances to a dictionary
global_importances = {
    class_name: dict(enumerate(importances))
    for class_name, importances in zip(classes_names, mean_gradients, strict=True)
}

# Load API key from environment variable
api_key = os.getenv("OPENAI_API_KEY")
if api_key is None:
    raise ValueError(
        "An API key is required to use `OpenAILLM` interface. ",
        "Cannot use LLMLabels without an LLM interface. ",
        "See last section for an example of how to branch one.",
    )

# set the LLM interface used to generate labels based on the constructed prompts
llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-nano")

# Initialize the ConSim with the split model and the user LLM
# Therefore, a given ConSim metric can be used on different explainers for cleaner comparison
con_sim = ConSim(
    splitter,
    llm_interface,
    classes=classes_names,
)

baseline_list = []
ica_score_list = []
for seed in range(10):
    # Select examples for evaluation
    samples, labels, predictions = con_sim._extract_interesting_elements(
        inputs=test_inputs,
        labels=test_labels,
        predictions=test_predictions,
        seed=seed,
    )

    # Compute a baseline and ConSim score to give sense to the explainer ConSim score
    baseline = con_sim.evaluate(
        interesting_samples=samples, predictions=predictions, prompt_type=PromptTypes.L2_baseline_with_lp
    )

    if baseline is None:
        continue

    # Compute the ConSim score for an explainer
    ica_score = con_sim.evaluate(
        interesting_samples=samples,
        predictions=predictions,
        concept_explainer=concept_explainer,
        concepts_interpretation=topk_words,
        global_importances=global_importances,
        prompt_type=PromptTypes.E2_global_concepts_with_lp,
    )

    if ica_score is None:
        continue

    baseline_list.append(baseline)
    ica_score_list.append(ica_score)

print(f"Baseline: {round(sum(baseline_list) / 10, 2)}, ICA: {round(sum(ica_score_list) / 10, 2)}")

from interpreto.concepts.metrics.consim import ConSim, PromptTypes # convert step 5 global concept importances to a dictionary global_importances = { class_name: dict(enumerate(importances)) for class_name, importances in zip(classes_names, mean_gradients, strict=True) } # Load API key from environment variable api_key = os.getenv("OPENAI_API_KEY") if api_key is None: raise ValueError( "An API key is required to use `OpenAILLM` interface. ", "Cannot use LLMLabels without an LLM interface. ", "See last section for an example of how to branch one.", ) # set the LLM interface used to generate labels based on the constructed prompts llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-nano") # Initialize the ConSim with the split model and the user LLM # Therefore, a given ConSim metric can be used on different explainers for cleaner comparison con_sim = ConSim( splitter, llm_interface, classes=classes_names, ) baseline_list = [] ica_score_list = [] for seed in range(10): # Select examples for evaluation samples, labels, predictions = con_sim._extract_interesting_elements( inputs=test_inputs, labels=test_labels, predictions=test_predictions, seed=seed, ) # Compute a baseline and ConSim score to give sense to the explainer ConSim score baseline = con_sim.evaluate( interesting_samples=samples, predictions=predictions, prompt_type=PromptTypes.L2_baseline_with_lp ) if baseline is None: continue # Compute the ConSim score for an explainer ica_score = con_sim.evaluate( interesting_samples=samples, predictions=predictions, concept_explainer=concept_explainer, concepts_interpretation=topk_words, global_importances=global_importances, prompt_type=PromptTypes.E2_global_concepts_with_lp, ) if ica_score is None: continue baseline_list.append(baseline) ica_score_list.append(ica_score) print(f"Baseline: {round(sum(baseline_list) / 10, 2)}, ICA: {round(sum(ica_score_list) / 10, 2)}")

/home/antonin.poche/interpreto/interpreto/concepts/metrics/consim.py:603: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at /pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:836.)
  importances = torch.abs(torch.Tensor(list(concepts_importance.values())))
/home/antonin.poche/interpreto/interpreto/concepts/metrics/consim.py:1288: UserWarning: The user-llm responses are empty or the format is not respected. Returning None. The response was: 'Sample_20: Sports'
  return self._compute_score(

Baseline: 0.22, ICA: 0.23

➡️ Note

We evaluated the first ICA explainer here. Concepts where not really interpretable, ConSim agrees.

8. Using your own LLM interface ¶

In [ ]:

from interpreto.commons.llm_interface import LLMInterface, Role


class GeminiLLM(LLMInterface):
    def __init__(self, api_key: str, model: str = "gemini-1.5-flash", num_try: int = 5):
        try:
            import google.generativeai as genai  # noqa: PLC0415  # ruff: disable=import-outside-toplevel
        except ImportError as e:
            raise ImportError("Install google-generativeai to use Google Gemini API.") from e

        self.genai = genai
        self.genai.configure(api_key=api_key)
        self.model = model
        self.num_try = num_try

    def generate(self, prompt: list[tuple[Role, str]]) -> str | None:
        # Build system instruction and chat history for Gemini
        system_messages: list[str] = []
        contents: list[dict] = []

        for role, content in prompt:
            if role == Role.SYSTEM:
                system_messages.append(content)
            elif role == Role.USER:
                contents.append(
                    {
                        "role": "user",
                        "parts": [{"text": content}],
                    }
                )
            elif role == Role.ASSISTANT:
                contents.append(
                    {
                        "role": "model",
                        "parts": [{"text": content}],
                    }
                )
            else:
                raise ValueError(f"Unknown role for google gemini api: {role}")

        system_instruction: str | None = "\n".join(system_messages) if system_messages else None

        label: str | None = None
        for _ in range(self.num_try):
            try:
                model = self.genai.GenerativeModel(
                    model_name=self.model,
                    system_instruction=system_instruction,
                )
                response = model.generate_content(contents)  # type: ignore[arg-type]
                # google-generativeai exposes the main text as .text
                label = response.text
                break
            except Exception as e:  # noqa: BLE001
                print(e)
        return label

from interpreto.commons.llm_interface import LLMInterface, Role class GeminiLLM(LLMInterface): def __init__(self, api_key: str, model: str = "gemini-1.5-flash", num_try: int = 5): try: import google.generativeai as genai # noqa: PLC0415 # ruff: disable=import-outside-toplevel except ImportError as e: raise ImportError("Install google-generativeai to use Google Gemini API.") from e self.genai = genai self.genai.configure(api_key=api_key) self.model = model self.num_try = num_try def generate(self, prompt: list[tuple[Role, str]]) -> str | None: # Build system instruction and chat history for Gemini system_messages: list[str] = [] contents: list[dict] = [] for role, content in prompt: if role == Role.SYSTEM: system_messages.append(content) elif role == Role.USER: contents.append( { "role": "user", "parts": [{"text": content}], } ) elif role == Role.ASSISTANT: contents.append( { "role": "model", "parts": [{"text": content}], } ) else: raise ValueError(f"Unknown role for google gemini api: {role}") system_instruction: str | None = "\n".join(system_messages) if system_messages else None label: str | None = None for _ in range(self.num_try): try: model = self.genai.GenerativeModel( model_name=self.model, system_instruction=system_instruction, ) response = model.generate_content(contents) # type: ignore[arg-type] # google-generativeai exposes the main text as .text label = response.text break except Exception as e: # noqa: BLE001 print(e) return label