Generation Concept-based Explanation Tutorial¶
Welcome to this tutorial, our will be to obtain concept-based explanations starting from the beginning.
We will start with a minimal example in section 1 and then go through the key steps of the concept-based:
- β Split your model in two parts
- β© Minimal example: Top-k tokens for neurons
- π¦ Compute a dataset of activations
- ποΈββοΈ Fit a concept model on activations
- π·οΈ Interpret the concept dimensions
On which we add two bonus steps present in most papers:
Author: Antonin PochΓ©
0. β Split your model in two parts ¶
Let's take a Qwen3-0.6B for both the minimal example and the detailed pipeline. But you can naturally use larger models.
Here we split at the 6 / 28 layers. But you can specify the module path to split at.
To split the model, we use the interpreto.SplitterForGeneration which wraps around the transformers causal language model and allows the computation of token activations at the specified split_point.
> β‘οΈ Note
>
> Interpreto's splitting based on nnsight, so depending on your use case, you might want to use nnsight directly.
1. β© Minimal example: Top-k tokens for neurons ¶
from interpreto.concepts import NeuronsAsConcepts, TopKInputs
# 2. No dataset of activation is needed as we consider the latent space as the concept space
# 3. No training neither
# Use `NeuronsAsConcepts` to use the concept-based pipeline with neurons
concept_explainer = NeuronsAsConcepts(splitter)
# 4. Use `TopKInputs` to get the top-k tokens that maximally activate each neuron
method = TopKInputs(
concept_explainer=concept_explainer,
use_vocab=True, # use the vocabulary of the model and test all tokens (~150k with Qwen-0.6B)
k=10, # get the top 10 tokens for each neuron
)
topk_tokens = method.interpret(
concepts_indices=list(range(5)), # interpret the five first neurons
)
# show some neurons' interpretations
for concept_idx, tokens in topk_tokens.items():
print(f"Concept {concept_idx}: {list(tokens.keys())}")
del concept_explainer, method, topk_tokens
> β The concepts are not interpretable? > > Well, this is surely due to superposition, this is why we use dictionary learning. > > Let's explore and solve this problem in the next sections.
2. π¦ Compute a datasets of activations ¶
We will use the IMDB dataset to build a dataset of activations.
> β οΈ Warning > > The dataset used has a considerable impact on the concept-space obtained. Hence, in practice, we recommend to use a subset of the model training set. > > The concept model we train be it SAEs or others, find patterns in the dataset of activations, which explains the dependence of the concepts found on the activations dataset.
> π₯ Tip > > The larger the dataset the better. But at the cost of computation time to get activations. > > Hence, the best dataset for SAEs would be one where each token is only seen once. But this is harder in practice, and such pipeline is not covered in this tutorial.
interpreto.SplitterForGeneration.get_activations() returns flattened non-special token activations by default.
from datasets import load_dataset
# take the whole dataset, more samples leads to better results, but some methods do not support too big datasets
imdb = load_dataset("stanfordnlp/imdb")["train"]["text"][:10000]
# compute the non-special-token activations of the whole IMDB dataset
# activations are flattened between the n_sample and seq_len dimensions
# which leads us to more then 6 million tokens
# (n * l, d)
splitter.batch_size = 8
activations, _ = splitter.get_activations(
inputs=imdb,
tqdm_bar=True,
)
splitter.to("cpu")
print(f"{activations.shape = }")
3. ποΈββοΈ Fit a concept model on activations ¶
Now we can fit a concept model on the activations. They exist more or less complex concept models. Here we use an SAE, so it is quite complex and has a lot more parameters than a simple concept model.
In particular, we use interpreto.concepts.BatchTopK.
The concept_explainer wraps around both splitter, the model wrapper, and the concept_model.
> β‘οΈ Note > > Most of our concept models and all the SAEs implementations depend on the Overcomplete library. It is a library that provides a lot of concept models and optimization methods for concept extraction. So if you want to go deeper on these, we suggest digging there.
import torch
from interpreto.concepts.methods.overcomplete import BatchTopKSAEConcepts, DeadNeuronsReanimationLoss
top_k_individual = 10
concept_model_batch_size = 512
epochs = 5
# instantiate the concept explainer with the splitted model
concept_explainer = BatchTopKSAEConcepts(
splitter,
nb_concepts=1000,
device="cuda",
top_k=top_k_individual * concept_model_batch_size,
)
# train the SAE on the activations
log = concept_explainer.fit(
activations=activations,
criterion=DeadNeuronsReanimationLoss, # set an MSE loss with dead neurons reanimation
optimizer_class=torch.optim.Adam,
scheduler_class=torch.optim.lr_scheduler.CosineAnnealingLR,
scheduler_kwargs={"T_max": epochs, "eta_min": 1e-6},
lr=1e-3,
nb_epochs=epochs,
batch_size=concept_model_batch_size,
monitoring=1,
)
4. π·οΈ Interpret the concept dimensions ¶
Once the concept directions are set, it is time to interpret them. We explore two possibilities here, as the second one requires an OpenAI api key.
4.1 Interpret concepts via top-k tokens¶
Via interpreto.concepts.TopKInputs, it is possible to extract the top-k tokens that activate each concept the most.
from interpreto.concepts import TopKInputs
interpretation_method = TopKInputs(
concept_explainer=concept_explainer,
concept_encoding_batch_size=512 * concept_model_batch_size,
k=10, # get the top 10 tokens for each concept
)
interpretations = interpretation_method.interpret(
inputs=imdb,
latent_activations=activations,
concepts_indices="all", # interpret all concepts
)
# Clean Δ for Qwen specifically
interpretations = {
concept_idx: [token.lstrip("Δ ") for token in tokens.keys()] if tokens else None
for concept_idx, tokens in interpretations.items()
}
for concept_idx, tokens in list(interpretations.items())[:10]:
print(f"Concept {concept_idx}: {tokens if tokens else None}")
> β‘οΈ Note > > While not the most interpretable concepts, this is still better than the minimal example. > > Let's improve interpretations and see what happens.
4.2 Interpret concepts with an LLM labels¶
Here we use interpreto.concepts.LLMLabels, it uses LLM to label the concepts based on examples activating the concept.
> β οΈ Warning
>
> The following cell is skipped if you do not set an OpenAI API key. As LLMLabels requires a LLMInterface, and we chose the OpenAILLM one.
>
> What you can do to make it work:
> - Get an OpenAI API key and set it as an environment variable OPENAI_API_KEY.
> - Branch your own LLMInterface and use it instead of OpenAILLM. See last section for an example.
import os
from interpreto.commons.llm_interface import OpenAILLM
from interpreto.concepts import LLMLabels
api_key = os.getenv("OPENAI_API_KEY")
if api_key is not None:
# set the LLM interface used to generate labels based on the constructed prompts
llm_interface = OpenAILLM(api_key=api_key, model="gpt-4.1-mini")
splitter.to("cpu")
interpretation_method = LLMLabels(
concept_explainer=concept_explainer,
llm_interface=llm_interface,
k_examples=20, # number of examples in each concept
k_context=5, # number of tokens before and after the maximally activating one to give context
concept_encoding_batch_size=concept_model_batch_size,
)
# compute the labels for an arbitrary subset of the concepts
interpretations = interpretation_method.interpret(
inputs=imdb,
latent_activations=activations,
concepts_indices=list(
range(10)
), # we could put `"all"` but that would take lon and cost a bit through the API
)
for concept_id, label in interpretations.items():
print(f"Concept {concept_id}: {label if label is not None else 'None'}")
> β‘οΈ Note 1
>
> The labels highly depend on the system prompt provided to the LLM interface.
>
> For labels formulation to better align with what you expect, you can set the system_prompt argument of the LLMLabels.
>
> Here is the default system prompt:
SYSTEM_PROMPT_WITH_CONTEXT = """Your role is to label the concepts/patterns present in the different examples.
You will be given a list of text examples on which special tokens are selected and between delimiters like <<this>>.
How important each token is for the behavior is listed after each example in parentheses, with importance from 0 to 10.
Hard rules:
- The label should summarize the concept linking the examples together. Give a single label describing all examples highlighted tokens.
- The label should be between 1 and 5 words long.
- The shorter the label the better. The best is a word.
- Do not make a sentence.
- The label should be the most precise possible. The goal is to be able to differentiate between concepts.
- The label should encompass most examples. But you can ignore the non-informative ones.
- Do not mention the marker tokens (<< >>) in your explanation. Nor refer to the importance.
- Only focus on the content and the label.
- Never ever give labels with more than 5 words, they would be cut out.
Some examples: 'blue', 'positive sentiment and enthusiasm', 'legal entities', 'medical places', 'hate', 'noun phrase', 'ion or iou sounds', 'questions' final words'...
"""
> β‘οΈ Note 2 > > Having our concepts and their interpretation is great. But what is really useful is to know: > - When are each concept used? (see next section) > - If the concepts are pertinent. (see final section)
5. π Local concept analysis ¶
To see the important concepts in a sample, there are two complementary steps: - Find the most important concepts for the predicted tokens. - Highlight the input tokens activating this concept.
5.0 Create a random sample and decompose it into tokens¶
# create a sample
sample = ["Interpreto has awesome visualizations."]
# split text between tokens
sample_tokens = splitter.tokenizer.tokenize(sample[0])
# to include special tokens:
# encoded = splitter.tokenizer(sample[0], add_special_tokens=True)
# sample_tokens = splitter.tokenizer.convert_ids_to_tokens(encoded["input_ids"])
# specific to the model
sample_tokens = [tok.replace("Δ ", " ") for tok in sample_tokens]
print(f"The sample is: {sample[0]}\n\nIt is decomposed in {len(sample_tokens)} tokens:\n{sample_tokens}")
5.1 Important concepts for text (with respect to the output)¶
Here we use the gradient of the concept-to-output function to estimate the importance of each concept for the outputs of the model.
interpreto.concepts.ConceptAutoEncoderExplainer.concept_output_gradient()
splitter.to("cuda")
# (seq_len (out), seq_len (in), nb_concepts)
local_importances = concept_explainer.concept_output_gradient(
inputs=sample,
targets=None, # all predicted tokens
)[0] # only one sample
splitter.to("cpu")
# we take the sum over the input sequence dimension, has we focus on the concept-output relationship
# (seq_len (out), nb_concepts)
local_importances = local_importances.abs().sum(dim=1)
print(f"{local_importances.shape = }")
5.2 Input tokens activating the concepts¶
This step just looks at the concepts activation on the input tokens. It is a simple way to check if the concepts are activated on the input tokens.
5.3 Visualization¶
We did not interpret all concepts previously, so we need to interpret the locally most important concepts.
# select the most important concepts for each output token
to_interpret = []
for importance in local_importances:
top10 = torch.argsort(importance, descending=True)[:10]
to_interpret.append(top10)
to_interpret = torch.concat(to_interpret, dim=0).unique()
to_interpret = [c.item() for c in to_interpret if c.item() not in interpretations.keys()]
# interpret these concepts
new_interpretations = interpretation_method.interpret(
inputs=imdb,
latent_activations=activations,
concepts_indices=to_interpret,
)
interpretations.update(new_interpretations)
> π₯ Tip > > By default, the most important concepts for the whole sample are shown. > > But you can click on an output token to select it and look at concepts for that token only.
6. βοΈ Evaluate concept-based explanations ¶
Let's compare the minimal example and the Batch TopK SAE explainers.
6.1 π Evaluate the concept-space from the third part¶
> β οΈ Warning: > > These metrics should only be used to compare the concept-space trained in similar contexts, same model, split point, activation dataset...
Reconstruction error
Sparsity
Dictionary metrics
> β‘οΈ Note > > We do not apply the stability metric here because it requires to retrain the concept space several times and compare them. It would be too long for the tutorial. Nonetheless, SAEs in general are not really stable.
from interpreto.concepts.metrics import FID, MSE, Sparsity, SparsityRatio
# BatchTopKSAEConcepts
mse = MSE(concept_explainer).compute(test_activations)
fid = FID(concept_explainer).compute(test_activations)
sparsity = Sparsity(concept_explainer).compute(test_activations)
ratio = SparsityRatio(concept_explainer).compute(test_activations)
print(f"MSE: {round(mse, 3)}, FID: {round(fid, 3)}, Sparsity: {round(sparsity, 3)}, Sparsity ratio: {round(ratio, 3)}")
# NeuronsAsConcepts
identity_explainer = NeuronsAsConcepts(splitter)
mse = MSE(identity_explainer).compute(test_activations)
fid = FID(identity_explainer).compute(test_activations)
sparsity = Sparsity(identity_explainer).compute(test_activations)
ratio = SparsityRatio(identity_explainer).compute(test_activations)
print(f"MSE: {round(mse, 3)}, FID: {round(fid, 3)}, Sparsity: {round(sparsity, 3)}, Sparsity ratio: {round(ratio, 3)}")
6.2 π Evaluate the concepts-interpretations from the fourth step¶
7. Using your own LLM interface ¶
In this case, will use the same model we are studying to label the concepts. We do not specifically recommend it over using another model. Though some work argue, model's are better to label their own concepts. Here we mainly use the same to reduce the memory cost.
from interpreto.commons.llm_interface import LLMInterface, Role
class HuggingFaceLLM(LLMInterface):
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
self.device = model.device
def preprocess(self, prompt: list(tuple[Role, str])) -> str:
assert prompt[0][0] == Role.SYSTEM
assert prompt[1][0] == Role.USER
if getattr(self.tokenizer, "chat_template", None):
messages = [
{"role": "system", "content": prompt[0][1]},
{"role": "user", "content": prompt[1][1]},
]
return self.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=False,
)
f"System:\n{prompt[0][1]}\n\nUser:\n{prompt[1][1]}\n\nAssistant:\n"
def generate(self, prompt: list[tuple[Role, str]]) -> list[str]:
processed = self.preprocess(prompt)
inputs = self.tokenizer(
processed,
return_tensors="pt",
padding=True,
truncation=True,
# max_length=tokenizer_max_length,
).to(self.device)
with torch.no_grad():
generated_ids = self.model.generate(
**inputs,
pad_token_id=self.tokenizer.pad_token_id,
max_new_tokens=32,
)
input_lengths = inputs["attention_mask"].sum(dim=1)
outputs = []
for j, output_ids in enumerate(generated_ids):
new_tokens = output_ids[input_lengths[j] :]
text = self.tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
outputs.append(text)
return outputs
splitter.to("cuda")
my_llm = HuggingFaceLLM(splitter._model, splitter.tokenizer)
interpretation_method = LLMLabels(
concept_explainer=concept_explainer,
llm_interface=my_llm,
k_examples=20, # number of examples in each concept
k_context=5, # number of tokens before and after the maximally activating one to give context
concept_encoding_batch_size=concept_model_batch_size,
)
# compute the labels for an arbitrary subset of the concepts
interpretations = interpretation_method.interpret(
inputs=imdb,
latent_activations=activations,
concepts_indices=list(range(10)), # we could put `"all"` but that would take lon and cost a bit through the API
)
for concept_id, label in interpretations.items():
print(f"Concept {concept_id}: {label if label is not None else 'None'}")