{ style="display:block; max-width:100%; height:auto; margin:0 auto;" }

Classification Demonstration¶

This notebook show case what can be done with interpreto for classification.

author: Antonin Poché

0. Imports and model loading¶

In [1]:

import os

import datasets
import torch
import transformers

import interpreto
from interpreto import Lime, plot_attributions, plot_concepts
from interpreto.attributions.metrics import Insertion
from interpreto.commons.llm_interface import OpenAILLM
from interpreto.concepts import LLMLabels, SemiNMFConcepts
from interpreto.concepts.interpretations import TopKInputs
from interpreto.concepts.metrics import FID, SparsityRatio

import os import datasets import torch import transformers import interpreto from interpreto import Lime, plot_attributions, plot_concepts from interpreto.attributions.metrics import Insertion from interpreto.commons.llm_interface import OpenAILLM from interpreto.concepts import LLMLabels, SemiNMFConcepts from interpreto.concepts.interpretations import TopKInputs from interpreto.concepts.metrics import FID, SparsityRatio

In [2]:

# load dataset examples
dataset = datasets.load_dataset("dair-ai/emotion", "split")["train"]["text"][:1000]

classes_names = ["sadness", "joy", "love", "anger", "fear", "surprise"]
# class_names={0: "sadness", 1: "joy", 2: "love", 3: "anger", 4: "fear", 5: "surprise"}

# load dataset examples dataset = datasets.load_dataset("dair-ai/emotion", "split")["train"]["text"][:1000] classes_names = ["sadness", "joy", "love", "anger", "fear", "surprise"] # class_names={0: "sadness", 1: "joy", 2: "love", 3: "anger", 4: "fear", 5: "surprise"}

In [3]:

# load the model and tokenizer
tokenizer = transformers.AutoTokenizer.from_pretrained("nateraw/bert-base-uncased-emotion", use_fast=True)
model = transformers.AutoModelForSequenceClassification.from_pretrained("nateraw/bert-base-uncased-emotion").cuda()

# load the model and tokenizer tokenizer = transformers.AutoTokenizer.from_pretrained("nateraw/bert-base-uncased-emotion", use_fast=True) model = transformers.AutoModelForSequenceClassification.from_pretrained("nateraw/bert-base-uncased-emotion").cuda()

Loading weights:   0%|          | 0/201 [00:00<?, ?it/s]

BertForSequenceClassification LOAD REPORT from: nateraw/bert-base-uncased-emotion
Key                          | Status     |  | 
-----------------------------+------------+--+-
bert.embeddings.position_ids | UNEXPECTED |  | 

Notes:
- UNEXPECTED	:can be ignored when loading from different task/architecture; not ok if you expect identical arch.

1. Attribution Demonstration¶

1.1 Obtain the attributions¶

In [4]:

# instantiate Lime
attribution_explainer = Lime(model, tokenizer)

# compute attributions
attributions = attribution_explainer(
    model_inputs="Love and hate are two sides of the same coin.",
    targets=torch.tensor([[0, 1, 2, 3, 4, 5]]),
)

# visualize attributions
plot_attributions(attributions[0], classes_names=classes_names)

# instantiate Lime attribution_explainer = Lime(model, tokenizer) # compute attributions attributions = attribution_explainer( model_inputs="Love and hate are two sides of the same coin.", targets=torch.tensor([[0, 1, 2, 3, 4, 5]]), ) # visualize attributions plot_attributions(attributions[0], classes_names=classes_names)

Classes

Inputs

1.2 Evaluate the attributions¶

In [5]:

# use the same model as for the explainer
metric = Insertion(model, tokenizer)

# compute scores on attributions
auc, detailed_scores = metric.evaluate(attributions)

print(f"Insertion AUC: {round(auc, 3)} (lower is better)")

# use the same model as for the explainer metric = Insertion(model, tokenizer) # compute scores on attributions auc, detailed_scores = metric.evaluate(attributions) print(f"Insertion AUC: {round(auc, 3)} (lower is better)")

Insertion AUC: 0.295 (lower is better)

2. Concepts Demonstration¶

2.1 Split model and get a dataset of activations¶

In [6]:

# split the model
splitter = interpreto.SplitterForClassification(model, tokenizer=tokenizer)

# compute the [CLS] token activations
activations, predictions = splitter.get_activations(dataset)

# split the model splitter = interpreto.SplitterForClassification(model, tokenizer=tokenizer) # compute the [CLS] token activations activations, predictions = splitter.get_activations(dataset)

2.2 Learn concepts as patterns in the activations¶

In [7]:

# create an explainer around the split model
concept_explainer = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda")

# train the concept model of the explainer
concept_explainer.fit(activations)

# create an explainer around the split model concept_explainer = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda") # train the concept model of the explainer concept_explainer.fit(activations)

2.3 Interpret the concepts¶

In [8]:

# instantiate the interpretation method with the concept explainer
interpretation_method = TopKInputs(
    concept_explainer=concept_explainer,
    k=5,
    use_unique_words=True,
    unique_words_kwargs={"count_min_threshold": 20, "lemmatize": True},
)

# interpret the concepts via top-k words
topk_words = interpretation_method.interpret(
    inputs=dataset,
    concepts_indices="all",
)

# instantiate the interpretation method with the concept explainer interpretation_method = TopKInputs( concept_explainer=concept_explainer, k=5, use_unique_words=True, unique_words_kwargs={"count_min_threshold": 20, "lemmatize": True}, ) # interpret the concepts via top-k words topk_words = interpretation_method.interpret( inputs=dataset, concepts_indices="all", )

2.4 Estimate concepts importance to classes¶

In [9]:

# estimate the importance of concepts for each class using the gradient
gradients = concept_explainer.concept_output_gradient(
    inputs=activations,
    batch_size=32,
)

# stack gradients on samples and average them over samples
mean_gradients = torch.stack(gradients).abs().squeeze().mean(0)  # (num_classes, num_concepts)

# for each class, sort the importance scores
order = torch.argsort(mean_gradients, descending=True)

# estimate the importance of concepts for each class using the gradient gradients = concept_explainer.concept_output_gradient( inputs=activations, batch_size=32, ) # stack gradients on samples and average them over samples mean_gradients = torch.stack(gradients).abs().squeeze().mean(0) # (num_classes, num_concepts) # for each class, sort the importance scores order = torch.argsort(mean_gradients, descending=True)

In [10]:

activations.device

activations.device

Out[10]:

device(type='cpu')

2.5 Visualization¶

2.5.1 global concepts importance¶

In [11]:

plot_concepts(
    classes_names=classes_names,
    concepts_importances=mean_gradients,
    concepts_labels={k: list(v.keys()) for k, v in topk_words.items()},
)

plot_concepts( classes_names=classes_names, concepts_importances=mean_gradients, concepts_labels={k: list(v.keys()) for k, v in topk_words.items()}, )

Classes

Concepts

Tip: The above visualization is interactive, you should click on the classes.

2.5.2 Inputs-to-concepts attributions¶

In [12]:

sample = "Love and hate are two sides of the same coin."

# Use the same class as attributions
concepts_attribution = Lime(concept_explainer.get_inputs_to_concepts_model(), tokenizer)

outputs = concepts_attribution(sample)[0]

plot_concepts(
    sample=outputs.elements,
    classes_names=classes_names,
    concepts_activations=outputs.attributions.T,
    concepts_importances=mean_gradients,
    concepts_labels={k: list(v.keys()) for k, v in topk_words.items()},
)

sample = "Love and hate are two sides of the same coin." # Use the same class as attributions concepts_attribution = Lime(concept_explainer.get_inputs_to_concepts_model(), tokenizer) outputs = concepts_attribution(sample)[0] plot_concepts( sample=outputs.elements, classes_names=classes_names, concepts_activations=outputs.attributions.T, concepts_importances=mean_gradients, concepts_labels={k: list(v.keys()) for k, v in topk_words.items()}, )

Classes

Concepts

Sample

2.6 Evaluate concepts¶

In [13]:

test_activations = activations  # in practice, these should be different

fid = FID(concept_explainer).compute(test_activations)
ratio = SparsityRatio(concept_explainer).compute(test_activations)

print(f"FID: {round(fid, 3)}, Sparsity ratio: {round(ratio, 3)}")

test_activations = activations # in practice, these should be different fid = FID(concept_explainer).compute(test_activations) ratio = SparsityRatio(concept_explainer).compute(test_activations) print(f"FID: {round(fid, 3)}, Sparsity ratio: {round(ratio, 3)}")

FID: 0.022, Sparsity ratio: 0.05

2.7 Class-wise and LLM-labels¶

In [14]:

# set the LLM interface used to generate labels based on the constructed prompts
llm_interface = OpenAILLM(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4.1-nano")

# iterate over classes
concepts_interpretations = {}
concepts_importances = {}
for target, class_name in enumerate(classes_names):
    # ----------------------------------------------------------------------------------------------
    # 1. construct the dataset of activations (extract the ones related to the class)
    indices = (predictions == target).nonzero(as_tuple=True)[0]
    class_wise_inputs = [dataset[i] for i in indices]
    class_wise_activations = activations[indices]

    # ----------------------------------------------------------------------------------------------
    # 2. train concept model
    concept_explainer = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda")
    concept_explainer.fit(class_wise_activations)

    # ----------------------------------------------------------------------------------------------
    # 4. compute concepts importance (before interpretations to limit the number of concepts interpreted)
    gradients = concept_explainer.concept_output_gradient(
        inputs=class_wise_activations,
        targets=[target],
        concepts_x_gradients=True,
        batch_size=32,
    )

    # stack gradients on samples and average them over samples
    concepts_importances[target] = torch.stack(gradients, axis=0).squeeze().abs().mean(dim=0)  # (num_concepts,)

    # for each class, sort the importance scores
    important_concept_indices = torch.argsort(concepts_importances[target], descending=True).tolist()

    # ----------------------------------------------------------------------------------------------
    # 3. interpret the important concepts concepts
    llm_labels_method = LLMLabels(
        concept_explainer=concept_explainer,
        llm_interface=llm_interface,
        k_examples=20,
    )

    concepts_interpretations[target] = llm_labels_method.interpret(
        inputs=class_wise_inputs,
        concepts_indices=important_concept_indices,  # only the top 5 concepts for this class
    )

    print(f"\nClass: {class_name}")
    for concept_id in important_concept_indices[:5]:
        label = concepts_interpretations[target].get(concept_id, None)
        importance = concepts_importances[target][concept_id].item()
        if label is not None:
            print(f"\timportance: {round(importance, 3)},\t{label}")

# set the LLM interface used to generate labels based on the constructed prompts llm_interface = OpenAILLM(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4.1-nano") # iterate over classes concepts_interpretations = {} concepts_importances = {} for target, class_name in enumerate(classes_names): # ---------------------------------------------------------------------------------------------- # 1. construct the dataset of activations (extract the ones related to the class) indices = (predictions == target).nonzero(as_tuple=True)[0] class_wise_inputs = [dataset[i] for i in indices] class_wise_activations = activations[indices] # ---------------------------------------------------------------------------------------------- # 2. train concept model concept_explainer = SemiNMFConcepts(splitter, nb_concepts=20, device="cuda") concept_explainer.fit(class_wise_activations) # ---------------------------------------------------------------------------------------------- # 4. compute concepts importance (before interpretations to limit the number of concepts interpreted) gradients = concept_explainer.concept_output_gradient( inputs=class_wise_activations, targets=[target], concepts_x_gradients=True, batch_size=32, ) # stack gradients on samples and average them over samples concepts_importances[target] = torch.stack(gradients, axis=0).squeeze().abs().mean(dim=0) # (num_concepts,) # for each class, sort the importance scores important_concept_indices = torch.argsort(concepts_importances[target], descending=True).tolist() # ---------------------------------------------------------------------------------------------- # 3. interpret the important concepts concepts llm_labels_method = LLMLabels( concept_explainer=concept_explainer, llm_interface=llm_interface, k_examples=20, ) concepts_interpretations[target] = llm_labels_method.interpret( inputs=class_wise_inputs, concepts_indices=important_concept_indices, # only the top 5 concepts for this class ) print(f"\nClass: {class_name}") for concept_id in important_concept_indices[:5]: label = concepts_interpretations[target].get(concept_id, None) importance = concepts_importances[target][concept_id].item() if label is not None: print(f"\timportance: {round(importance, 3)},\t{label}")

Class: sadness
	importance: 0.114,	Emotive self-referencing expressions
	importance: 0.103,	Expressions of emotional and physical sluggishness
	importance: 0.087,	Expressive emotions and self-perceptions.
	importance: 0.085,	Expressive self-reference and emotional states.
	importance: 0.063,	Expressive self-references with emotion words.

Class: joy
	importance: 0.082,	Expressing subjective emotional states with "feel" plus adjectives or nouns.
	importance: 0.075,	Expressive self-affirmation and emotional safety.
	importance: 0.073,	Subjective emotional states and sensations expressed through nuanced, sensory, or evaluative language.
	importance: 0.071,	Subjective emotion expression tied to context.
	importance: 0.07,	Subjective self-perception and emotional states

Class: love
	importance: 0.139,	Personal emotional state and sensory perception
	importance: 0.114,	Subjective emotional awareness and varied sentiments.
	importance: 0.097,	Expressive emotion-linked self-reference.
	importance: 0.096,	Subjective emotional states linked to intimacy and desire.
	importance: 0.092,	Expressions of personal emotion or perception often introduced by "I feel" or "I think," with variability in emotional tone and explicitness.

Class: anger
	importance: 0.081,	Expressive self-reference and emotional states.
	importance: 0.074,	Expressive anger and frustration structural patterns.
	importance: 0.074,	Subjective emotional state expression with explicit "I feel" or similar phrases.
	importance: 0.071,	Expressive emotional states indicated by "feel" or "feeling."
	importance: 0.071,	Frequent expression of personal emotions and sensations.

Class: fear
	importance: 0.142,	Expressive of anxiety, uncertainty, and vulnerability
	importance: 0.099,	Expressive uncertainty and vulnerability.
	importance: 0.098,	Expressive uncertainty and internal emotional conflict
	importance: 0.074,	Subjective emotional states expressed through direct self-reference; emphasis on feelings of confusion, overwhelm, vulnerability, and introspection.
	importance: 0.069,	Expressive emotional and sensory states.

Class: surprise
	importance: 0.167,	Expresses subjective sensations and emotional states consistently using "I feel" phrasing.
	importance: 0.104,	Emotional and physical sensation descriptions
	importance: 0.085,	Subjective emotion and sensory states expressed explicitly.
	importance: 0.082,	Expressive self-reference and emotional states.
	importance: 0.078,	Subjective sensation expressions with varied emotional intensity.

In [15]:

plot_concepts(
    classes_names=classes_names,
    concepts_importances=concepts_importances,
    concepts_labels=concepts_interpretations,
)

plot_concepts( classes_names=classes_names, concepts_importances=concepts_importances, concepts_labels=concepts_interpretations, )

Classes

Concepts