Base Classes

interpreto.concepts.ConceptEncoderExplainer

ConceptEncoderExplainer(model_with_split_points, concept_model, split_point=None)

Bases: ABC, Generic[ConceptModel]

Code: concepts/base.py

Abstract class defining an interface for concept explanation. Child classes should implement the fit and encode_activations methods, and only assume the presence of an encoding step using the concept_model to convert activations to latent concepts.

Attributes:

Name	Type	Description
model_with_split_points	ModelWithSplitPoints	The model to apply the explanation on. It should have at least one split point on which concept_model can be fitted.
split_point	str	The split point used to train the concept_model.
concept_model	ConceptModelProtocol	The model used to extract concepts from the activations of model_with_split_points. The only assumption for classes inheriting from this class is that the concept_model can encode activations into concepts with encode_activations. The ConceptModelProtocol is defined in interpreto.typing. It is basically a torch.nn.Module with an encode method.
is_fitted	bool	Whether the concept_model was fit on model activations.
has_differentiable_concept_encoder	bool	Whether the encode_activations operation is differentiable.

Parameters:

Name	Type	Description	Default
model_with_split_points	ModelWithSplitPoints	The model to apply the explanation on. It should have at least one split point on which a concept explainer can be trained.	required
concept_model	ConceptModelProtocol	The model used to extract concepts from the activations of model_with_split_points. The ConceptModelProtocol is defined in interpreto.typing. It is basically a torch.nn.Module with an encode method.	required
split_point	str | None	The split point used to train the concept_model. If None, tries to use the split point of model_with_split_points if a single one is defined.	None

Source code in interpreto/concepts/base.py

def __init__(
    self,
    model_with_split_points: ModelWithSplitPoints,
    concept_model: ConceptModelProtocol,
    split_point: str | None = None,
):
    """Initializes the concept explainer with a given splitted model.

    Args:
        model_with_split_points (ModelWithSplitPoints): The model to apply the explanation on.
            It should have at least one split point on which a concept explainer can be trained.
        concept_model (ConceptModelProtocol): The model used to extract concepts from
            the activations of `model_with_split_points`.
            The `ConceptModelProtocol` is defined in `interpreto.typing`. It is basically a `torch.nn.Module` with an `encode` method.
        split_point (str | None): The split point used to train the `concept_model`. If None, tries to use the
            split point of `model_with_split_points` if a single one is defined.
    """
    if not isinstance(model_with_split_points, ModelWithSplitPoints):
        raise TypeError(
            f"The given model should be a ModelWithSplitPoints, but {type(model_with_split_points)} was given."
        )
    self.model_with_split_points: ModelWithSplitPoints = model_with_split_points
    self._concept_model = concept_model
    self.split_point = split_point  # Verified by `split_point.setter`
    self.__is_fitted: bool = False

fit abstractmethod

fit(activations, *args, **kwargs)

Fits concept_model on the given activations.

Parameters:

Name	Type	Description	Default
activations	Tensor | dict[str, Tensor]	A dictionary with model paths as keys and the corresponding tensors as values.	required

Returns:

Type	Description
Any	None, concept_model is fitted in-place, is_fitted is set to True and split_point is set.

Source code in interpreto/concepts/base.py

@abstractmethod
def fit(self, activations: LatentActivations | dict[str, LatentActivations], *args, **kwargs) -> Any:
    """Fits `concept_model` on the given activations.

    Args:
        activations (torch.Tensor | dict[str, torch.Tensor]): A dictionary with model paths as keys and the corresponding
            tensors as values.

    Returns:
        `None`, `concept_model` is fitted in-place, `is_fitted` is set to `True` and `split_point` is set.
    """
    pass

interpret

interpret(*args, **kwargs)

Deprecated API for concept interpretation.

Interpretation methods should now be instantiated directly with the fitted concept explainer. For example:

TopKInputs(concept_explainer).interpret(inputs, latent_activations)

This method is kept only for backwards compatibility and will always raise a :class:NotImplementedError.

Source code in interpreto/concepts/base.py

@check_fitted
def interpret(self, *args, **kwargs) -> Mapping[int, Any]:  # TODO: 0.5.0 remove
    """Deprecated API for concept interpretation.

    Interpretation methods should now be instantiated directly with the
    fitted concept explainer. For example:

    ``TopKInputs(concept_explainer).interpret(inputs, latent_activations)``

    This method is kept only for backwards compatibility and will always
    raise a :class:`NotImplementedError`.
    """
    raise NotImplementedError("Use the new API: TopKInputs(concept_explainer).interpret(...).")

input_concept_attribution

input_concept_attribution(inputs, concept, attribution_method, **attribution_kwargs)

Attributes model inputs for a selected concept.

Parameters:

Name	Type	Description	Default
inputs	ModelInputs	The input data, which can be a string, a list of tokens/words/clauses/sentences or a dataset.	required
concept	int	Index identifying the position of the concept of interest (score in the ConceptsActivations tensor) for which relevant input elements should be retrieved.	required
attribution_method	type[AttributionExplainer]	The attribution method to obtain importance scores for input elements.	required

Returns:

Type	Description
list[float]	A list of attribution scores for each input.

Source code in interpreto/concepts/base.py

@check_fitted
def input_concept_attribution(
    self,
    inputs: ModelInputs,
    concept: int,
    attribution_method: type[AttributionExplainer],
    **attribution_kwargs,
) -> list[float]:
    """Attributes model inputs for a selected concept.

    Args:
        inputs (ModelInputs): The input data, which can be a string, a list of tokens/words/clauses/sentences
            or a dataset.
        concept (int): Index identifying the position of the concept of interest (score in the
            `ConceptsActivations` tensor) for which relevant input elements should be retrieved.
        attribution_method: The attribution method to obtain importance scores for input elements.

    Returns:
        A list of attribution scores for each input.
    """
    raise NotImplementedError("Input-to-concept attribution method is not implemented yet.")

interpreto.concepts.ConceptAutoEncoderExplainer

ConceptAutoEncoderExplainer(model_with_split_points, concept_model, split_point=None)

Bases: ConceptEncoderExplainer[BaseDictionaryLearning], Generic[BDL]

Code: concepts/base.py

A concept bottleneck explainer wraps a concept_model that should be able to encode activations into concepts and decode concepts into activations.

We use the term "concept bottleneck" loosely, as the latent space can be overcomplete compared to activation space, as in the case of sparse autoencoders.

We assume that the concept model follows the structure of an overcomplete.BaseDictionaryLearning model, which defines the encode and decode methods for encoding and decoding activations into concepts.

Attributes:

Name	Type	Description
model_with_split_points	ModelWithSplitPoints	The model to apply the explanation on. It should have at least one split point on which concept_model can be fitted.
split_point	str	The split point used to train the concept_model.
concept_model	[BaseDictionaryLearning](https	//github.com/KempnerInstitute/overcomplete/blob/24568ba5736cbefca4b78a12246d92a1be04a1f4/overcomplete/base.py#L10)): The model used to extract concepts from the activations of model_with_split_points. The only assumption for classes inheriting from this class is that the concept_model can encode activations into concepts with encode_activations.
is_fitted	bool	Whether the concept_model was fit on model activations.
has_differentiable_concept_encoder	bool	Whether the encode_activations operation is differentiable.
has_differentiable_concept_decoder	bool	Whether the decode_concepts operation is differentiable.

Parameters:

Name	Type	Description	Default
model_with_split_points	ModelWithSplitPoints	The model to apply the explanation on. It should have at least one split point on which a concept explainer can be trained.	required
concept_model ([BaseDictionaryLearning](https		//github.com/KempnerInstitute/overcomplete/blob/24568ba5736cbefca4b78a12246d92a1be04a1f4/overcomplete/base.py#L10)): The model used to extract concepts from the activations of model_with_split_points.	required
split_point	str | None	The split point used to train the concept_model. If None, tries to use the split point of model_with_split_points if a single one is defined.	None

Source code in interpreto/concepts/base.py

def __init__(
    self,
    model_with_split_points: ModelWithSplitPoints,
    concept_model: BaseDictionaryLearning,
    split_point: str | None = None,
):
    """Initializes the concept explainer with a given splitted model.

    Args:
        model_with_split_points (ModelWithSplitPoints): The model to apply the explanation on.
            It should have at least one split point on which a concept explainer can be trained.
        concept_model ([BaseDictionaryLearning](https://github.com/KempnerInstitute/overcomplete/blob/24568ba5736cbefca4b78a12246d92a1be04a1f4/overcomplete/base.py#L10)): The model used to extract concepts from
            the activations of `model_with_split_points`.
        split_point (str | None): The split point used to train the `concept_model`. If None, tries to use the
            split point of `model_with_split_points` if a single one is defined.
    """
    self.concept_model: BaseDictionaryLearning
    super().__init__(model_with_split_points, concept_model, split_point)

fit abstractmethod

fit(activations, *args, **kwargs)

Fits concept_model on the given activations.

Parameters:

Name	Type	Description	Default
activations	Tensor | dict[str, Tensor]	A dictionary with model paths as keys and the corresponding tensors as values.	required

Returns:

Type	Description
Any	None, concept_model is fitted in-place, is_fitted is set to True and split_point is set.

Source code in interpreto/concepts/base.py

@abstractmethod
def fit(self, activations: LatentActivations | dict[str, LatentActivations], *args, **kwargs) -> Any:
    """Fits `concept_model` on the given activations.

    Args:
        activations (torch.Tensor | dict[str, torch.Tensor]): A dictionary with model paths as keys and the corresponding
            tensors as values.

    Returns:
        `None`, `concept_model` is fitted in-place, `is_fitted` is set to `True` and `split_point` is set.
    """
    pass

encode_activations

encode_activations(activations)

Encode the given activations using the concept_model encoder.

Parameters:

Name	Type	Description	Default
activations	LatentActivations	The activations to encode.	required

Returns:

Type	Description
Tensor	The encoded concept activations.

Source code in interpreto/concepts/base.py

@check_fitted
def encode_activations(self, activations: LatentActivations) -> torch.Tensor:  # ConceptsActivations
    """Encode the given activations using the `concept_model` encoder.

    Args:
        activations (LatentActivations): The activations to encode.

    Returns:
        The encoded concept activations.
    """
    if hasattr(self.concept_model, "device") and self.concept_model.device != activations.device:
        activations = activations.to(self.concept_model.device, non_blocking=True)
        self.concept_model.to(activations.device)
    return self.concept_model.encode(activations)  # type: ignore

decode_concepts

decode_concepts(concepts)

Decode the given concepts using the concept_model decoder.

Parameters:

Name	Type	Description	Default
concepts	ConceptsActivations	The concepts to decode.	required

Returns:

Type	Description
Tensor	The decoded model activations.

Source code in interpreto/concepts/base.py

@check_fitted
def decode_concepts(self, concepts: ConceptsActivations) -> torch.Tensor:  # LatentActivations
    """Decode the given concepts using the `concept_model` decoder.

    Args:
        concepts (ConceptsActivations): The concepts to decode.

    Returns:
        The decoded model activations.
    """
    if hasattr(self.concept_model, "device") and self.concept_model.device != concepts.device:
        concepts = concepts.to(self.concept_model.device, non_blocking=True)
        self.concept_model.to(concepts.device)
    return self.concept_model.decode(concepts)  # type: ignore

get_dictionary

get_dictionary()

Get the dictionary learned by the fitted concept_model.

Returns:

Type	Description
Tensor	torch.Tensor: A torch.Tensor containing the learned dictionary.

Source code in interpreto/concepts/base.py

@check_fitted
def get_dictionary(self) -> torch.Tensor:  # TODO: add this to tests
    """Get the dictionary learned by the fitted `concept_model`.

    Returns:
        torch.Tensor: A `torch.Tensor` containing the learned dictionary.
    """
    return self.concept_model.get_dictionary()  # type: ignore

interpret

interpret(*args, **kwargs)

Deprecated API for concept interpretation.

Interpretation methods should now be instantiated directly with the fitted concept explainer. For example:

TopKInputs(concept_explainer).interpret(inputs, latent_activations)

This method is kept only for backwards compatibility and will always raise a :class:NotImplementedError.

Source code in interpreto/concepts/base.py

@check_fitted
def interpret(self, *args, **kwargs) -> Mapping[int, Any]:  # TODO: 0.5.0 remove
    """Deprecated API for concept interpretation.

    Interpretation methods should now be instantiated directly with the
    fitted concept explainer. For example:

    ``TopKInputs(concept_explainer).interpret(inputs, latent_activations)``

    This method is kept only for backwards compatibility and will always
    raise a :class:`NotImplementedError`.
    """
    raise NotImplementedError("Use the new API: TopKInputs(concept_explainer).interpret(...).")

concept_output_gradient

concept_output_gradient(inputs, targets=None, split_point=None, activation_granularity=TOKEN, aggregation_strategy=MEAN, concepts_x_gradients=True, normalization=True, tqdm_bar=False, batch_size=None)

Compute the gradients of the predictions with respect to the concepts.

To clarify what this function does, lets detail some notations. Suppose the initial model was splitted such that \(f = g \circ h\). Hence the concept model was fitted on \(A = h(X)\) with \(X\) a dataset of samples. The resulting concept model encoders and decoders are noted \(t\) and \(t^{-1}\). \(t\) can be seen as projections from the latent space to the concept space. Hence, the function going from the inputs to the concepts is \(f_{ic} = t \circ h\) and the function going from the concepts to the outputs is \(f_{co} = g \circ t^-1\).

Given a set of samples \(X\), and the functions \((h, t, t^{-1}, g)\) This function first compute \(C = t(A) = t \circ h(X)\), then returns \(\nabla{f_{co}}(C)\).

In practice all computations are done by ModelWithSplitPoints._get_concept_output_gradients, which relies on NNsight. The current method only forwards the \(t\) and \(t^{-1}\), respectively self.encode_activations and self.decode_concepts methods.

Parameters:

Name	Type	Description	Default
inputs	list[str] | Tensor | BatchEncoding	The input data, either a list of samples, the tokenized input or a batch of samples.	required
targets	list[int] | None	Specify which outputs of the model should be used to compute the gradients. Note that \(f_{co}\) often has several outputs, by default gradients are computed for each output. The t dimension of the returned tensor is equal to the number of selected targets. (For classification, those are the classes logits and for generation, those are the most probable tokens probabilities).	None
split_point	str | None	The split point used to train the concept_model. If None, tries to use the split point of model_with_split_points if a single one is defined.	None
activation_granularity	ActivationGranularity	The granularity of the activations to use for the attribution. It is highly recommended to to use the same granularity as the one used in the fit method. Possibles values are: ModelWithSplitPoints.activation_granularities.CLS_TOKEN: only the first token (e.g. [CLS]) activation is returned (batch, d_model). ModelWithSplitPoints.activation_granularities.ALL_TOKENS: every token activation is treated as a separate element (batch x seq_len, d_model). ModelWithSplitPoints.activation_granularities.TOKEN: remove special tokens. ModelWithSplitPoints.activation_granularities.WORD: aggregate by words following the split defined by :class:~interpreto.commons.granularity.Granularity.WORD. ModelWithSplitPoints.activation_granularities.SENTENCE: aggregate by sentences following the split defined by :class:~interpreto.commons.granularity.Granularity.SENTENCE. Requires spacy to be installed.	TOKEN
aggregation_strategy	GranularityAggregationStrategy	Strategy to aggregate token activations into larger inputs granularities. Applied for WORD and SENTENCE activation strategies. Token activations of shape n * (l, d) are aggregated on the sequence length dimension. The concatenated into (ng, d) tensors. Existing strategies are: ModelWithSplitPoints.aggregation_strategies.SUM: Tokens activations are summed along the sequence length dimension. ModelWithSplitPoints.aggregation_strategies.MEAN: Tokens activations are averaged along the sequence length dimension. ModelWithSplitPoints.aggregation_strategies.MAX: The maximum of the token activations along the sequence length dimension is selected. ModelWithSplitPoints.aggregation_strategies.SIGNED_MAX: The maximum of the absolute value of the activations multiplied by its initial sign. signed_max([[-1, 0, 1, 2], [-3, 1, -2, 0]]) = [-3, 1, -2, 2]	MEAN
concepts_x_gradients	bool	If the resulting gradients should be multiplied by the concepts activations. True by default (similarly to attributions), because of mathematical properties. Therefore the out put is \(C * \nabla{f_{co}}(C)\).	True
normalization	bool	Whether to normalize the gradients. Gradients will be normalized on the concept (c) and sequence length (g) dimensions. Such that for a given sample-target-granular pair, the sum of the absolute values of the gradients is equal to 1. (The granular elements depend on the :arg:activation_granularity).	True
tqdm_bar	bool	Whether to display a progress bar.	False
batch_size	int | None	Batch size for the model. It might be different from the one used in ModelWithSplitPoints.get_activations because gradients have a much larger impact on the memory.	None

Returns:

Type	Description
list[Float[Tensor, 't g c']]	list[Float[torch.Tensor, "t g c"]]: The gradients of the model output with respect to the concept activations. List length: correspond to the number of inputs. Tensor shape: (t, g, c) with t the target dimension, g the number of granularity elements in one input, and c the number of concepts.

Source code in interpreto/concepts/base.py

@check_fitted
def concept_output_gradient(
    self,
    inputs: torch.Tensor | list[str] | BatchEncoding,
    targets: list[int] | None = None,
    split_point: str | None = None,
    activation_granularity: ActivationGranularity = ActivationGranularity.TOKEN,
    aggregation_strategy: GranularityAggregationStrategy = GranularityAggregationStrategy.MEAN,
    concepts_x_gradients: bool = True,
    normalization: bool = True,
    tqdm_bar: bool = False,
    batch_size: int | None = None,
) -> list[Float[torch.Tensor, "t g c"]]:
    """
    Compute the gradients of the predictions with respect to the concepts.

    To clarify what this function does, lets detail some notations.
    Suppose the initial model was splitted such that $f = g \\circ h$.
    Hence the concept model was fitted on $A = h(X)$ with $X$ a dataset of samples.
    The resulting concept model encoders and decoders are noted $t$ and $t^{-1}$.
    $t$ can be seen as projections from the latent space to the concept space.
    Hence, the function going from the inputs to the concepts is $f_{ic} = t \\circ h$
    and the function going from the concepts to the outputs is $f_{co} = g \\circ t^-1$.

    Given a set of samples $X$, and the functions $(h, t, t^{-1}, g)$
    This function first compute $C = t(A) = t \\circ h(X)$, then returns $\\nabla{f_{co}}(C)$.

    In practice all computations are done by `ModelWithSplitPoints._get_concept_output_gradients`,
    which relies on NNsight. The current method only forwards the $t$ and $t^{-1}$,
    respectively `self.encode_activations` and `self.decode_concepts` methods.

    Args:
        inputs (list[str] | torch.Tensor | BatchEncoding):
            The input data, either a list of samples, the tokenized input or a batch of samples.

        targets (list[int] | None):
            Specify which outputs of the model should be used to compute the gradients.
            Note that $f_{co}$ often has several outputs, by default gradients are computed for each output.
            The `t` dimension of the returned tensor is equal to the number of selected targets.
            (For classification, those are the classes logits and for generation, those are the most probable tokens probabilities).

        split_point (str | None):
            The split point used to train the `concept_model`.
            If None, tries to use the split point of `model_with_split_points` if a single one is defined.

        activation_granularity (ActivationGranularity):
            The granularity of the activations to use for the attribution.
            It is highly recommended to to use the same granularity as the one used in the `fit` method.
            Possibles values are:

            - ``ModelWithSplitPoints.activation_granularities.CLS_TOKEN``:
                only the first token (e.g. ``[CLS]``) activation is returned ``(batch, d_model)``.

            - ``ModelWithSplitPoints.activation_granularities.ALL_TOKENS``:
                every token activation is treated as a separate element ``(batch x seq_len, d_model)``.

            - ``ModelWithSplitPoints.activation_granularities.TOKEN``: remove special tokens.

            - ``ModelWithSplitPoints.activation_granularities.WORD``:
                aggregate by words following the split defined by
                :class:`~interpreto.commons.granularity.Granularity.WORD`.

            - ``ModelWithSplitPoints.activation_granularities.SENTENCE``:
                aggregate by sentences following the split defined by
                :class:`~interpreto.commons.granularity.Granularity.SENTENCE`.
                Requires `spacy` to be installed.

        aggregation_strategy:
            Strategy to aggregate token activations into larger inputs granularities.
            Applied for `WORD` and `SENTENCE` activation strategies.
            Token activations of shape  n * (l, d) are aggregated on the sequence length dimension.
            The concatenated into (ng, d) tensors.
            Existing strategies are:

            - ``ModelWithSplitPoints.aggregation_strategies.SUM``:
                Tokens activations are summed along the sequence length dimension.

            - ``ModelWithSplitPoints.aggregation_strategies.MEAN``:
                Tokens activations are averaged along the sequence length dimension.

            - ``ModelWithSplitPoints.aggregation_strategies.MAX``:
                The maximum of the token activations along the sequence length dimension is selected.

            - ``ModelWithSplitPoints.aggregation_strategies.SIGNED_MAX``:
                The maximum of the absolute value of the activations multiplied by its initial sign.
                signed_max([[-1, 0, 1, 2], [-3, 1, -2, 0]]) = [-3, 1, -2, 2]

        concepts_x_gradients (bool):
            If the resulting gradients should be multiplied by the concepts activations.
            True by default (similarly to attributions), because of mathematical properties.
            Therefore the out put is $C * \\nabla{f_{co}}(C)$.

        normalization (bool):
            Whether to normalize the gradients.
            Gradients will be normalized on the concept (c) and sequence length (g) dimensions.
            Such that for a given sample-target-granular pair,
            the sum of the absolute values of the gradients is equal to 1.
            (The granular elements depend on the :arg:`activation_granularity`).

        tqdm_bar (bool):
            Whether to display a progress bar.

        batch_size (int | None):
            Batch size for the model.
            It might be different from the one used in `ModelWithSplitPoints.get_activations`
            because gradients have a much larger impact on the memory.

    Returns:
        list[Float[torch.Tensor, "t g c"]]:
            The gradients of the model output with respect to the concept activations.
            List length: correspond to the number of inputs.
                Tensor shape: (t, g, c) with t the target dimension, g the number of granularity elements in one input, and c the number of
                concepts.
    """
    if not self.has_differentiable_concept_decoder:
        raise ValueError(
            "The concept decoder of this explainer is not differentiable. This is required to compute concept-to-output gradients. "
            f"Current explainer class: {self.__class__.__name__}."
        )

    # put everything on device
    self.concept_model.to(self.model_with_split_points.device)

    # forward all computations to
    gradients = self.model_with_split_points._get_concept_output_gradients(
        inputs=inputs,
        targets=targets,
        encode_activations=self.encode_activations,
        decode_concepts=self.decode_concepts,
        split_point=split_point,
        activation_granularity=activation_granularity,
        aggregation_strategy=aggregation_strategy,
        concepts_x_gradients=concepts_x_gradients,
        tqdm_bar=tqdm_bar,
        batch_size=batch_size,
    )

    # normalize the gradients if required
    if normalization:
        gradients = [self.__normalize_gradients(g) for g in gradients]
    return gradients