Skip to content

interpreto_banner

Classification Concept-based N-gram Explanation Tutorial

This tutorial shows how to extract and interpret concepts for a classification task using n-gram words, in two settings: globally and per class.

  • Globally: we decompose the activations of the entire validation set to obtain k concepts in total.
  • Per class: we decompose the activations of data from each class separately, obtaining k concepts per class.

Several decomposition methods are available (e.g. Semi-NMF, PCA, etc.). For more details, please refer to the Interpreto documentation.

How it works

Extracting concepts: we use CLS_TOKEN granularity, meaning only the CLS token activation of each input is kept. The decomposition (fit) is then performed on these CLS token activations, producing a concept space of k dimensions.

Interpreting concepts with n-grams: to find which words or groups of words best represent each concept, we extract all unique n-grams (groups of up to n consecutive words) from the dataset and pass each n-gram individually to the model. We take its CLS token activation and project it into the concept space obtained during the fit. The n-grams whose projections activate a concept the most are selected to represent the concept.

Using n-grams (2, 3, or more words) rather than single words seems to give more meaningful results: a single word passed alone to the model often lacks context, producing a less informative CLS activation.

1. Global n-gram concept interpretation

Extract k concepts from all activations, then interpret them with n-grams.

  1. โž— Split your model in two parts
  2. ๐Ÿšฆ Compute CLS token activations on the whole test set
  3. ๐Ÿ‹๏ธโ€โ™‚๏ธ Decompose the CLS activation space to obtain k concepts
  4. ๐Ÿท๏ธ Interpret each concept by projecting n-gram activations into the concept space
  5. ๐Ÿ“Š Compute the importance of each concept for each class
  6. ๐Ÿ” Visualize concept importance and n-gram labels

2. Class-wise n-gram concept interpretation

Extract k concepts per class by decomposing only the CLS activations of data from that class, then interpret them with n-grams.

  1. ๐Ÿšฆ Compute CLS token activations for each class separately
  2. ๐Ÿ‹๏ธโ€โ™‚๏ธ Decompose the CLS activation space of each class to obtain k concepts per class
  3. ๐Ÿท๏ธ Interpret each concept by projecting n-gram activations into the concept space
  4. ๐Ÿ“Š Compute the importance of each concept for its class
  5. ๐Ÿ” Visualize concept importance and n-gram labels per class
import torch

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# utils
import json

from IPython.display import HTML, display


def display_topngram_dropdown(topngram_words):
    data_json = json.dumps(
        {str(k): [{"ngram": ng, "score": f"{s:.4f}"} for ng, s in v.items()] for k, v in topngram_words.items()}
    )

    options_html = "".join(f'<option value="{k}">{k}</option>' for k in sorted(topngram_words.keys()))

    html = f"""
    <select id="concept-select" onchange="updateTable()">
    {options_html}
    </select>
<table border="1" id="concept-table" style="margin-top:8px"></table>
<script>
    var data = {data_json};
    function updateTable() {{
        var key = document.getElementById('concept-select').value;
        var rows = data[key];
        var html = '<tr><th>ngram</th><th>score</th></tr>';
        rows.forEach(function(r) {{ html += '<tr><td>'+r.ngram+'</td><td>'+r.score+'</td></tr>'; }});
        document.getElementById('concept-table').innerHTML = html;
    }}
    updateTable();
    </script>
    """
    display(HTML(html))

1. Global n-gram concept interpretation

1. โž— Split your model in two parts

We choose a bert fine-tuned on the go-emotions dataset and split it just before the classification head.

To split the model, we use the interpreto.ModelWithSplitPoints which wraps around the transformers model and allows the computation of activations at the specified split_point.

from transformers import AutoModelForSequenceClassification

from interpreto import ModelWithSplitPoints

model_with_split_points = ModelWithSplitPoints(
    model_or_repo_id="SchuylerH/bert-multilingual-go-emtions",
    automodel=AutoModelForSequenceClassification,
    split_point=5,  # split at the sixth layer
    device_map="cuda",
    batch_size=32,
)

2. ๐Ÿšฆ Compute a datasets of activations

We load the first 10000 documents of the go_emotions validation set.

Then we extract the activations of the [CLS] token of each document.

> โžก๏ธ Common practice > > In the literature, to train concepts for classification it is common to use the [CLS] just before the classification head. > > In fact, at this layer, it makes no sense to use other elements.

> โš ๏ธ Warning > > In this notebook, many things are specific to the use of the [CLS] token.

interpreto.ModelWithSplitPoints.get_activations()

from datasets import load_dataset

# load the AG-News dataset
dataset = load_dataset("google-research-datasets/go_emotions", "simplified")
inputs = list(dataset["validation"]["text"])
classes_names = dataset["validation"].features["labels"].feature.names

# Compute the [CLS] token activations
granularity = ModelWithSplitPoints.activation_granularities.CLS_TOKEN  # required for ngram interpretation.
activations, predictions = model_with_split_points.get_activations(
    inputs=inputs,
    activation_granularity=granularity,
    include_predicted_classes=True,
)
Generating train split:   0%|          | 0/43410 [00:00<?, ? examples/s]
Generating validation split:   0%|          | 0/5426 [00:00<?, ? examples/s]
Generating test split:   0%|          | 0/5427 [00:00<?, ? examples/s]
Loading weights:   0%|          | 0/201 [00:00<?, ?it/s]

3. ๐Ÿ‹๏ธโ€โ™‚๏ธ Fit a concept model on activations

With activations, we can train a concept model to find patterns (concepts).

The concept_model is an attribute of our concept explainer, similarly to the model_with_split_points. With these these two elements, we can go from inputs to concepts and from concepts to outputs.

In this tutorial, we use interpreto.concepts.SemiNMFConcepts. There are at least 15 others concept model available in interpreto. do not hesitate to explore them.

from interpreto.concepts import SemiNMFConcepts

# instantiate the concept explainer
concept_explainer = SemiNMFConcepts(
    model_with_split_points, nb_concepts=50, device="cuda"
)  # nb: nb_concept must be bigger than the number of classes, since there should be at least 1 concept = 1 class

# fit the concept explainer on activations
concept_explainer.fit(activations)

4. ๐Ÿท๏ธ Interpret the concept dimensions

We need to make sense of these concepts. In this case, we will use the interpreto.concepts.interpretations.TopKInputs to find the k (here, 5) ngram words (here 6 grams) which activates the most our concepts.

The unique_words_kwargs parameter controls how n-grams are extracted from the dataset before being passed to the model:

  • count_min_threshold: minimum number of occurrences for an n-gram to be kept. Increase this value if irrelevant or noisy n-grams appear in the results; decrease it if meaningful n-grams are filtered out.
  • lemmatize: if True, words are reduced to their base form before forming n-grams (e.g. "running" โ†’ "run", "cats" โ†’ "cat"). This avoids duplicates like "run" and "running" being treated as separate n-grams.
  • words_to_ignore: list of words to exclude before forming n-grams (e.g. punctuation, stopwords, or domain-specific noise).
from interpreto.concepts.interpretations import TopKInputs

# instantiate the interpretation method with the concept explainer
topngram_inputs_method = TopKInputs(
    concept_explainer=concept_explainer,
    k=5,
    activation_granularity=granularity,
    use_unique_words=6,  # 6-grams words. Put 1 to get unique words only. With the [CLS] token granularity, we are forced to use unique words
    unique_words_kwargs={
        "count_min_threshold": round(
            len(inputs) * 0.002
        ),  # appear in at least 0.2% of the samples | increase if random ngrams appear and decrease if some ngram appear too often
        "lemmatize": True,
        "words_to_ignore": [],  # include noise words and punctuation
    },
)
topngram_words = topngram_inputs_method.interpret(
    inputs=inputs,
    concepts_indices="all",
)
display_topngram_dropdown(topngram_words)
import torch

# estimate the importance of concepts for each class using the gradient
gradients = concept_explainer.concept_output_gradient(
    inputs=inputs,
    targets=None,  # None means all classes
    activation_granularity=granularity,
    concepts_x_gradients=True,  # the concept to output gradients are multiplied by the concepts values, this is common practice in the literature
    batch_size=32,
)

# stack gradients on samples and average them over samples
mean_gradients = torch.stack(gradients).abs().squeeze().mean(0)  # (num_classes, num_concepts)

# for each class, sort the importance scores
order = torch.argsort(mean_gradients, descending=True)

# visualize the top 5 concepts for each class
for target in range(order.shape[0]):
    print(f"\nClass: {classes_names[target]}:")
    for i in range(5):
        concept_id = order[target, i].item()
        importance = mean_gradients[target, concept_id].item()
        words = list(topngram_words.get(concept_id, None).keys())
        print(f"\tconcept id: {concept_id},\timportance: {round(importance, 3)},\ttopk words: {words}")

Class: admiration:
    concept id: 19, importance: 0.037,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 20, importance: 0.034,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 33, importance: 0.033,  topk words: ['lol', 'lol .', 'glad', 'glad to', 'haha']
    concept id: 48, importance: 0.028,  topk words: ['to your', 'for your', 'in your', 'to you', 'from the']
    concept id: 5,  importance: 0.027,  topk words: ['working', 'very', 'quite', 'care about', 'talking']

Class: amusement:
    concept id: 37, importance: 0.027,  topk words: ["n't believe", '* *', "n't think", 'relationship', '*']
    concept id: 49, importance: 0.026,  topk words: ['have been', 'made', 'it out', 'leave', 'become']
    concept id: 42, importance: 0.026,  topk words: ['i love this', 'i love it', 'i love the', 'love it', 'love this']
    concept id: 20, importance: 0.026,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 15, importance: 0.025,  topk words: ['lol', 'lol ,', 'haha', 'lol .', 'oh']

Class: anger:
    concept id: 38, importance: 0.03,   topk words: ['one .', 'sweet', 'i love the', 'one is', 'i don โ€™']
    concept id: 9,  importance: 0.028,  topk words: ['surprised', 'terrible', 'dumb', 'wrong', 'true']
    concept id: 44, importance: 0.027,  topk words: ['too !', 'live', 'lost', 'that โ€™ s what', '...']
    concept id: 40, importance: 0.026,  topk words: ['sorry for', 'so sorry', 'sorry', '. thank you', 'i agree']
    concept id: 39, importance: 0.025,  topk words: ['disgusting', 'killed', 'stupid', 'the worst', 'the fuck']

Class: annoyance:
    concept id: 48, importance: 0.032,  topk words: ['to your', 'for your', 'in your', 'to you', 'from the']
    concept id: 32, importance: 0.03,   topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 38, importance: 0.028,  topk words: ['one .', 'sweet', 'i love the', 'one is', 'i don โ€™']
    concept id: 16, importance: 0.028,  topk words: ['my life', 'so happy', 'the same', 'i hope', 'year old']
    concept id: 21, importance: 0.028,  topk words: ['laugh', 'difference', 'lol', 'lol ,', 'joke']

Class: approval:
    concept id: 20, importance: 0.033,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 15, importance: 0.033,  topk words: ['lol', 'lol ,', 'haha', 'lol .', 'oh']
    concept id: 38, importance: 0.031,  topk words: ['one .', 'sweet', 'i love the', 'one is', 'i don โ€™']
    concept id: 23, importance: 0.029,  topk words: ['on your', 'part', "i 'm", 'in your', 'of your']
    concept id: 46, importance: 0.027,  topk words: ['i hope you', 'hope you', 'care about', 'hope', 'i hope']

Class: caring:
    concept id: 17, importance: 0.034,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 32, importance: 0.031,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 19, importance: 0.03,   topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 46, importance: 0.03,   topk words: ['i hope you', 'hope you', 'care about', 'hope', 'i hope']
    concept id: 31, importance: 0.029,  topk words: ['you know', 'you think', 'they were', 'he is', 'they are']

Class: confusion:
    concept id: 9,  importance: 0.037,  topk words: ['surprised', 'terrible', 'dumb', 'wrong', 'true']
    concept id: 40, importance: 0.036,  topk words: ['sorry for', 'so sorry', 'sorry', '. thank you', 'i agree']
    concept id: 32, importance: 0.032,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 11, importance: 0.031,  topk words: ['i hope you', 'hope you', 'hope', 'i want to', 'hopefully']
    concept id: 16, importance: 0.03,   topk words: ['my life', 'so happy', 'the same', 'i hope', 'year old']

Class: curiosity:
    concept id: 32, importance: 0.045,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 31, importance: 0.037,  topk words: ['you know', 'you think', 'they were', 'he is', 'they are']
    concept id: 9,  importance: 0.034,  topk words: ['surprised', 'terrible', 'dumb', 'wrong', 'true']
    concept id: 6,  importance: 0.033,  topk words: ['hate', 'everyone', 'support', 'everything', 'anyone']
    concept id: 11, importance: 0.032,  topk words: ['i hope you', 'hope you', 'hope', 'i want to', 'hopefully']

Class: desire:
    concept id: 31, importance: 0.036,  topk words: ['you know', 'you think', 'they were', 'he is', 'they are']
    concept id: 32, importance: 0.036,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 11, importance: 0.032,  topk words: ['i hope you', 'hope you', 'hope', 'i want to', 'hopefully']
    concept id: 46, importance: 0.032,  topk words: ['i hope you', 'hope you', 'care about', 'hope', 'i hope']
    concept id: 17, importance: 0.032,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']

Class: disappointment:
    concept id: 19, importance: 0.032,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 16, importance: 0.03,   topk words: ['my life', 'so happy', 'the same', 'i hope', 'year old']
    concept id: 29, importance: 0.03,   topk words: ["if you do n't", 'character', 'thank you .', 'pretty', 'favourite']
    concept id: 17, importance: 0.028,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 25, importance: 0.027,  topk words: ['guess', "ca n't", 'idea', 'i thought', 'dumb']

Class: disapproval:
    concept id: 21, importance: 0.04,   topk words: ['laugh', 'difference', 'lol', 'lol ,', 'joke']
    concept id: 48, importance: 0.033,  topk words: ['to your', 'for your', 'in your', 'to you', 'from the']
    concept id: 49, importance: 0.029,  topk words: ['have been', 'made', 'it out', 'leave', 'become']
    concept id: 16, importance: 0.028,  topk words: ['my life', 'so happy', 'the same', 'i hope', 'year old']
    concept id: 2,  importance: 0.028,  topk words: ['disgusting', 'i would', 'i will', 'a fucking', 'i appreciate']

Class: disgust:
    concept id: 19, importance: 0.03,   topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 47, importance: 0.028,  topk words: ['i should', 'happens', 'change', 'broke', 'made me']
    concept id: 38, importance: 0.028,  topk words: ['one .', 'sweet', 'i love the', 'one is', 'i don โ€™']
    concept id: 9,  importance: 0.027,  topk words: ['surprised', 'terrible', 'dumb', 'wrong', 'true']
    concept id: 20, importance: 0.027,  topk words: ['>', 'else', '/s', ': (', ')']

Class: embarrassment:
    concept id: 19, importance: 0.034,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 20, importance: 0.032,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 9,  importance: 0.031,  topk words: ['surprised', 'terrible', 'dumb', 'wrong', 'true']
    concept id: 17, importance: 0.03,   topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 29, importance: 0.03,   topk words: ["if you do n't", 'character', 'thank you .', 'pretty', 'favourite']

Class: excitement:
    concept id: 48, importance: 0.039,  topk words: ['to your', 'for your', 'in your', 'to you', 'from the']
    concept id: 32, importance: 0.034,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 33, importance: 0.031,  topk words: ['lol', 'lol .', 'glad', 'glad to', 'haha']
    concept id: 20, importance: 0.029,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 31, importance: 0.029,  topk words: ['you know', 'you think', 'they were', 'he is', 'they are']

Class: fear:
    concept id: 31, importance: 0.033,  topk words: ['you know', 'you think', 'they were', 'he is', 'they are']
    concept id: 17, importance: 0.032,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 19, importance: 0.031,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 20, importance: 0.03,   topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 32, importance: 0.028,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']

Class: gratitude:
    concept id: 43, importance: 0.038,  topk words: ['[', 'favorite', '] , [', '[ name ] ,', ', [']
    concept id: 19, importance: 0.035,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 45, importance: 0.03,   topk words: ['now .', 'why', 'him .', 'and then', 'then']
    concept id: 23, importance: 0.029,  topk words: ['on your', 'part', "i 'm", 'in your', 'of your']
    concept id: 7,  importance: 0.029,  topk words: ['even if', 'surprised', 'because it', "if you do n't", 'because you']

Class: grief:
    concept id: 17, importance: 0.036,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 19, importance: 0.033,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 32, importance: 0.03,   topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 20, importance: 0.029,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 31, importance: 0.029,  topk words: ['you know', 'you think', 'they were', 'he is', 'they are']

Class: joy:
    concept id: 48, importance: 0.034,  topk words: ['to your', 'for your', 'in your', 'to you', 'from the']
    concept id: 32, importance: 0.033,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 33, importance: 0.031,  topk words: ['lol', 'lol .', 'glad', 'glad to', 'haha']
    concept id: 45, importance: 0.029,  topk words: ['now .', 'why', 'him .', 'and then', 'then']
    concept id: 21, importance: 0.027,  topk words: ['laugh', 'difference', 'lol', 'lol ,', 'joke']

Class: love:
    concept id: 32, importance: 0.034,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 42, importance: 0.033,  topk words: ['i love this', 'i love it', 'i love the', 'love it', 'love this']
    concept id: 48, importance: 0.031,  topk words: ['to your', 'for your', 'in your', 'to you', 'from the']
    concept id: 17, importance: 0.03,   topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 20, importance: 0.027,  topk words: ['>', 'else', '/s', ': (', ')']

Class: nervousness:
    concept id: 17, importance: 0.034,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 19, importance: 0.033,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 20, importance: 0.03,   topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 25, importance: 0.03,   topk words: ['guess', "ca n't", 'idea', 'i thought', 'dumb']
    concept id: 29, importance: 0.029,  topk words: ["if you do n't", 'character', 'thank you .', 'pretty', 'favourite']

Class: optimism:
    concept id: 46, importance: 0.041,  topk words: ['i hope you', 'hope you', 'care about', 'hope', 'i hope']
    concept id: 15, importance: 0.033,  topk words: ['lol', 'lol ,', 'haha', 'lol .', 'oh']
    concept id: 19, importance: 0.032,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 17, importance: 0.03,   topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 11, importance: 0.029,  topk words: ['i hope you', 'hope you', 'hope', 'i want to', 'hopefully']

Class: pride:
    concept id: 19, importance: 0.037,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 20, importance: 0.035,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 17, importance: 0.032,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 32, importance: 0.031,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 33, importance: 0.028,  topk words: ['lol', 'lol .', 'glad', 'glad to', 'haha']

Class: realization:
    concept id: 15, importance: 0.036,  topk words: ['lol', 'lol ,', 'haha', 'lol .', 'oh']
    concept id: 19, importance: 0.029,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 29, importance: 0.028,  topk words: ["if you do n't", 'character', 'thank you .', 'pretty', 'favourite']
    concept id: 17, importance: 0.028,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 20, importance: 0.027,  topk words: ['>', 'else', '/s', ': (', ')']

Class: relief:
    concept id: 19, importance: 0.037,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 17, importance: 0.033,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 20, importance: 0.033,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 45, importance: 0.027,  topk words: ['now .', 'why', 'him .', 'and then', 'then']
    concept id: 33, importance: 0.026,  topk words: ['lol', 'lol .', 'glad', 'glad to', 'haha']

Class: remorse:
    concept id: 19, importance: 0.035,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 17, importance: 0.031,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 32, importance: 0.028,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 40, importance: 0.028,  topk words: ['sorry for', 'so sorry', 'sorry', '. thank you', 'i agree']
    concept id: 45, importance: 0.027,  topk words: ['now .', 'why', 'him .', 'and then', 'then']

Class: sadness:
    concept id: 17, importance: 0.035,  topk words: ['wasn', 'didn', 'and [', 'for [', 'doesn']
    concept id: 19, importance: 0.031,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 29, importance: 0.029,  topk words: ["if you do n't", 'character', 'thank you .', 'pretty', 'favourite']
    concept id: 2,  importance: 0.027,  topk words: ['disgusting', 'i would', 'i will', 'a fucking', 'i appreciate']
    concept id: 41, importance: 0.026,  topk words: ['sorry for', 'sorry', 'so sorry', 'i will', 'miss']

Class: surprise:
    concept id: 9,  importance: 0.033,  topk words: ['surprised', 'terrible', 'dumb', 'wrong', 'true']
    concept id: 29, importance: 0.029,  topk words: ["if you do n't", 'character', 'thank you .', 'pretty', 'favourite']
    concept id: 32, importance: 0.029,  topk words: ['curious', 'i agree', 'i think', 'welcome', 'think']
    concept id: 20, importance: 0.028,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 25, importance: 0.027,  topk words: ['guess', "ca n't", 'idea', 'i thought', 'dumb']

Class: neutral:
    concept id: 19, importance: 0.038,  topk words: ['>', 'happen', 'why', 'out', 'i appreciate']
    concept id: 20, importance: 0.033,  topk words: ['>', 'else', '/s', ': (', ')']
    concept id: 37, importance: 0.032,  topk words: ["n't believe", '* *', "n't think", 'relationship', '*']
    concept id: 25, importance: 0.031,  topk words: ['guess', "ca n't", 'idea', 'i thought', 'dumb']
    concept id: 33, importance: 0.03,   topk words: ['lol', 'lol .', 'glad', 'glad to', 'haha']

from interpreto import plot_concepts

labels = {k: list(v.keys()) for k, v in topngram_words.items()}

plot_concepts(
    classes_names=classes_names,
    concepts_importances=mean_gradients,
    concepts_labels=labels,
)

Classes

2. Class-wise n-gram concept interpretation

In the global approach above, we decompose the activations of the entire dataset, producing k concepts shared across all classes. Here, we take a different approach: for each class, we select only the data points predicted as that class, and decompose their activations separately.

This yields k concepts per class, each capturing patterns specific to that class. For example, in a sentiment analysis task, a "joy" class might have concepts related to celebration, gratitude, or humor, while a "sadness" class might have concepts related to loss, loneliness, or disappointment.

The steps are the same as before (decompose, interpret with n-grams, compute importance), but repeated independently for each class. The model split and CLS token activations computed earlier are reused โ€” only the decomposition and interpretation are class-specific.

concept_explainers = {}
concept_interpretations = {}
concept_importances = {}

# iterate over classes
for target, class_name in enumerate(classes_names):
    # ----------------------------------------------------------------------------------------------
    # 2. construct the dataset of activations (extract the ones related to the class)
    indices = (predictions == target).nonzero(as_tuple=True)[0]
    class_wise_inputs = [inputs[i] for i in indices]
    class_wise_activations = activations[indices]

    if len(indices) == 0:
        print(f"\nClass: {class_name} โ€” no samples predicted, skipping.")
        continue

    # ----------------------------------------------------------------------------------------------
    # 3. train concept model
    concept_explainers[target] = SemiNMFConcepts(model_with_split_points, nb_concepts=20, device="cuda")
    concept_explainers[target].fit(class_wise_activations)

    # ----------------------------------------------------------------------------------------------
    # 4. get the k-ngram-words activating the most the concept
    topngram_inputs_method = TopKInputs(
        concept_explainer=concept_explainers[target],
        k=15,
        activation_granularity=granularity,
        use_unique_words=5,  # 5-grams words to illustrate each concept
        unique_words_kwargs={
            "count_min_threshold": round(len(inputs) * 0.002),
            "lemmatize": True,
            "words_to_ignore": [],
        },
    )
    concept_interpretations[target] = topngram_inputs_method.interpret(
        inputs=inputs,
        concepts_indices="all",
    )

    # ----------------------------------------------------------------------------------------------
    # 5. compute concepts importance
    gradients = concept_explainers[target].concept_output_gradient(
        inputs=class_wise_inputs,
        targets=[target],
        activation_granularity=granularity,
        concepts_x_gradients=True,
        batch_size=8,
    )

    concept_importances[target] = torch.stack(gradients, axis=0).squeeze().abs().mean(dim=0)

    important_concept_indices = torch.argsort(concept_importances[target], descending=True).tolist()

    print(f"\nClass: {class_name}")
    for concept_id in important_concept_indices[:5]:
        ngrams = concept_interpretations[target].get(concept_id)
        importance = concept_importances[target][concept_id].item()
        words = list(ngrams.keys()) if ngrams is not None else []
        print(f"\timportance: {round(importance, 3)},\tconcept {concept_id},\ttopk ngrams: {words}")

Class: admiration
    importance: 0.089,  concept 4,  topk ngrams: ['i can โ€™ t', "i 'm not", 'i don โ€™ t', "'re not", 'it โ€™ s not', 'i โ€™ m not', "it 's not", "i did n't", "'s not", 'favorite', "n't have", 'good luck', 'you can', "i do n't", 'cute']
    importance: 0.086,  concept 9,  topk ngrams: ['i thought it', 'i used', 'wrong .', 'โ€™ t think', "i did n't", 'i wish', 'i thought', 'i hate', 'i don โ€™ t', 'not .', 'it โ€™ s not', 'i can โ€™ t', 'wish', 'lol', 'why']
    importance: 0.082,  concept 6,  topk ngrams: ['than the', 'that [', 'if you', 'now', '] and [', 'of those', 'now i', 'about [', '] would', "if you 're", 'if we', 'to [', 'of them', 'out of', 'edit :']
    importance: 0.082,  concept 18, topk ngrams: ['cute', 'great', 'good luck', 'awesome', 'pretty', 'awesome .', 'good .', 'good', 'nothing', 'is awesome', 'the best', 'cut', 'amazing', 'beautiful', 'huge']
    importance: 0.068,  concept 0,  topk ngrams: ["what 's", 'my favorite', 'favorite', '. what', 'is what', 'favourite', 'โ€™ s what', 'didn โ€™ t', 's what', 'watching', ', what', 'what', 'amazing', 'isn โ€™ t', 'everything']

Class: amusement
    importance: 0.129,  concept 9,  topk ngrams: ['is awesome', 'doesn โ€™ t', "n't want", 'doesn', 'good luck', 'few', 'awesome', "have n't", "did n't", 'without', 'cute', 'sense', 'awesome .', 'doesn โ€™', "i did n't"]
    importance: 0.102,  concept 1,  topk ngrams: ['? !', 'what ?', '? ! ?', '! ? !', '? ?', '! ?', '?', '? i', '? that', 'you ?', 'it ?', 'lol .', 'surprised', '? it', 'this ?']
    importance: 0.079,  concept 5,  topk ngrams: ['working', 'live', 'mean', 'issue', 'feel', 'look at', 'example', 'look', 'know ,', 'take', 'talking about', 'i mean', 'such', 'work', 'seem']
    importance: 0.056,  concept 15, topk ngrams: ['[ name ] !', '] , [ name ]', '[ name ] would', '[ name ] is', 'dude', '[ name ]', '[ name ] ,', '[ name ] .', 'my [ name ]', '] , [ name', 'me .', '] , [', 'name ] is', ', [ name ]', 'name ] would']
    importance: 0.05,   concept 10, topk ngrams: [', thank', '. thank', 'thank you .', ', thank you', '. thank you', 'thank you', 'thank you !', 'thank you so much', 'thank you so', 'i will', 'thank you for', '. thanks', 'proud', 'i had', 'thank']

Class: anger
    importance: 0.11,   concept 15, topk ngrams: ['fucking', 'fuck', 'the fuck', 'stupid', 'disgusting', 'to you .', 'lost', 'i hate', 'horrible', 'but it', 'too .', 'i think it', 'the worst', 'a fucking', "n't"]
    importance: 0.097,  concept 16, topk ngrams: ['surprised', 'reading', 'talking', 'talking about', 'thinking', 'dead', 'feeling', 'think', 'afraid', 'having', 'sorry for', 'taking', 'getting', 'how', 'disgusting']
    importance: 0.092,  concept 2,  topk ngrams: ['surprised', 'this is', 'work .', 'this .', 'is this', 'the game', 'him .', 'it just', 'think it', 'it !', 'work', 'i think it', 'is it', 'it', 'it is']
    importance: 0.078,  concept 4,  topk ngrams: ['living', 'kinda', 'likely', 'super', 'โ€™ ve', 'kind', 'starting', 'hoping', 'worried', 'behind', "'ve been", 'i had', 'like [ name', 'face', "i 've"]
    importance: 0.067,  concept 14, topk ngrams: ['? ! ?', '? ?', '? !', 'disgusting', '?', 'what ?', '? it', '? i', 'you ?', '! ?', '[ name ] ?', '! ? !', '....', 'it ?', 'i hate']

Class: annoyance
    importance: 0.083,  concept 7,  topk ngrams: ['think', 'think it', 'think that', 'talking about', 'thought it', 'think you', 'happened', 'thinking', 'talking', 'feeling', 'happens', 'you mean', 'believe', 'i think it', 'think i']
    importance: 0.075,  concept 3,  topk ngrams: ['not even', 'not a', 'not the', 'and not', 'not', 'not .', 'โ€™ s not', "'s not", ', not', "'re not", "what 's", '. not', 'it โ€™ s not', 'only', "it 's not"]
    importance: 0.073,  concept 8,  topk ngrams: ['i hate', 'โ€™ t think', 'sorry for', "i did n't", 'so sorry', "n't have", 'hate', 'sorry', 'not be', 'not sure', 'it โ€™ s not', 'โ€™ s not', "i do n't", 't think', "you ca n't"]
    importance: 0.065,  concept 1,  topk ngrams: ['[ name ] .', 'shit', '[ name ] is', 'so .', '[ name ] ha', '[ name ] !', 'isn โ€™ t', 'is not', 'isn', 'source', 'she', 'damn', "'d be", 'do not', 'they are']
    importance: 0.063,  concept 17, topk ngrams: ['disgusting', 'surprised', 'watching', 'stupid', 'worried', 'problem .', 'the worst', 'terrible', 'shit', 'weird', 'horrible', 'talking', 'fuck', 'experience', 'the fuck']

Class: approval
    importance: 0.102,  concept 1,  topk ngrams: ['yesterday', 'cold', 'play', 'situation', 'stay', 'horrible', 'out', 'the worst', 'night', 'away', 'area', 'alone', 'stupid', 'scene', 'trying']
    importance: 0.074,  concept 14, topk ngrams: ['i hope you', 'i hope', 'i thought it', '. i hope', 'โ€™ t think', 'hope you', 'i don โ€™ t', 'i can โ€™ t', "i did n't", 'hope', "it 's not", 'not .', "i do n't", 'it โ€™ s not', 'i hate']
    importance: 0.073,  concept 3,  topk ngrams: ['sorry for', 'sorry', '. thanks', 'welcome', 'great', 'thanks .', 'huge', 'luck', 'good', 'i will', 'well', 'nothing', 'well .', 'whole', 'baby']
    importance: 0.068,  concept 19, topk ngrams: ['good .', 'haha', 'omg', 'lmao', 'like it', 'sweet', 'a fucking', 'lol', 'i appreciate', 'like you', 'awesome .', 'i thought it', 'i thought', 'care about', 'made me']
    importance: 0.065,  concept 18, topk ngrams: ['sorry', 'sorry for', 'so sorry', "is n't", "i did n't", "'re not", 'do not', "i ca n't", 'i don โ€™ t', "i do n't", 'i can โ€™ t', 'kid', 'is not', "it 's not", "i 'm not"]

Class: caring
    importance: 0.084,  concept 0,  topk ngrams: ['worried', 'amazing', 'starting', 'so happy', 'i appreciate', 'behind', 'the only one', 'i thought it', 'seen', 'a great', 'helped', 'a good', 'especially', 'like he', 'disgusting']
    importance: 0.083,  concept 19, topk ngrams: ['so sorry', 'disgusting', 'sorry for', 'sorry', 'thank you for', '. thank you', 'worried', 'thank you so much', 'thank you so', ', thank you', 'thank you', 'i hope you', 'thank you .', ', thank', 'thank you !']
    importance: 0.079,  concept 2,  topk ngrams: ['disgusting', 'experience', 'worried', 'problem .', 'awful', 'horrible', 'amazing', 'fuck', 'worst', 'fucking', 'so sorry', 'the fuck', 'she โ€™', '! ! ! ! !', 'my favorite']
    importance: 0.074,  concept 15, topk ngrams: ['i wish', 'i hope', 'hope', 'so sorry', '. i hope', 'wish', 'hope you', 'i want', 'you want', 'sorry', 'i hope you', 'sorry for', 'wish i', 'i โ€™ m not', 'hopefully']
    importance: 0.066,  concept 16, topk ngrams: ['help', 'i wish', 'i used', 'helped', "what 's", 'what', 'hopefully', 'why', 'enjoy', 's what', 'i want', 'away', '. what', 'what i', 'experience']

Class: confusion
    importance: 0.085,  concept 5,  topk ngrams: ['. thank you', 'thank you', 'thank you .', 'thank you so', 'thank you for', 'thank you !', ', thank you', 'thank you so much', '. thank', 'thank', 'i appreciate', 'appreciate', ', thank', '. thanks', 'thanks .']
    importance: 0.075,  concept 6,  topk ngrams: ['it out', 'idea .', 'surprised', 'fucking', 'awful', 'dumb', 'i thought it', 'shit', 'fuck', 'horrible', 'now .', 'expect', 'funny', 'yeah i', 'the worst']
    importance: 0.073,  concept 8,  topk ngrams: ['so happy', 'awful', 'confused', 'terrible', 'happy', 'why', 'worse', 'either', 'enough', 'worst', 'the worst', 'ever', 'worried', 'shame', 'better']
    importance: 0.06,   concept 11, topk ngrams: ['confused', ', what', 'what i', 'why', 'is what', '. what', 'i will', 's what', '. thank', 'm not', 'how to', ', thank', 'what', '] wa', '. thank you']
    importance: 0.059,  concept 1,  topk ngrams: ["wa n't", "is n't", "ca n't", 'โ€™ s what', "n't be", "i ca n't", "that 's a", "you 've", "i do n't", 'the most', "i 'm not", "that 's", "'s the", "'m not", 'that would be']

Class: curiosity
    importance: 0.093,  concept 14, topk ngrams: ['curious', 'work .', 'wow ,', 'worried', 'problem .', 'wow', 'disgusting', 'group', 'yeah ,', 'them .', 'cat', 'work', 'horrible', 'man .', 'rude']
    importance: 0.082,  concept 0,  topk ngrams: ['. thank', 'thank you so much', ', thank', 'thank you for', ', thank you', 'thank you so', '. thank you', 'thank you', 'thank you .', 'thank', 'thank you !', '. thanks', ', thanks', 'thanks ,', 'thanks']
    importance: 0.069,  concept 9,  topk ngrams: ['thank you for', ', thank you', '. thank you', 'thank you', 'thank you .', ', thank', 'thank you !', 'thank', 'thank you so', '. thank', 'thank you so much', 'thanks .', '. thanks', '. thanks for', 'thanks ,']
    importance: 0.064,  concept 10, topk ngrams: ['playing', 'trying', 'trying to', 'between', 'using', 'saying', 'coming', 'played', 'if they', 'from a', "they 're", 'new year', 'doing', 'response', 'if he']
    importance: 0.064,  concept 15, topk ngrams: ['will be', 'i can โ€™ t', 'will', 'โ€™ t think', "it 's not", 'it โ€™ s not', 'not .', "i did n't", 'i don โ€™ t', 'i want to', 'i want', 'is not', 'and not', 'would be', 'will get']

Class: desire
    importance: 0.082,  concept 12, topk ngrams: ["you 'll", "i 'll", "'ll be", 'phone', "'ll", 'you !', 'i โ€™ ll', 'yes !', 'game .', 'man .', 'pay', 'you .', 'comment', 'posting', 'posted']
    importance: 0.076,  concept 3,  topk ngrams: ['! ?', '? that', '? i', '? !', 'you ?', 'right ?', 'this ?', 'what ?', '! ? !', '?', '? it', '] ?', '? ?', '? ! ?', 'name ] ?']
    importance: 0.073,  concept 14, topk ngrams: ['would have', "would n't", "could n't", 'will get', 'would be', 'could have', 'i would', 'could be', '] would', 'would', 'might be', 'lose', 'can โ€™ t', 'myself', 'must be']
    importance: 0.07,   concept 5,  topk ngrams: ['hope you', 'surprised', 'horrible', 'thinking', 'they were', 'thought it', 'they are', 'think you', 'watching', 'they do', 'wish i', 'they have', 'hopefully', 'they did', 'thought']
    importance: 0.069,  concept 16, topk ngrams: ['i wish', 'i would', 'think', 'i hope', '. i hope', 'i will', 'guess', 'wish', 'i hate', 'i agree', 'i should', 'i could', 'you will', 'post', 'to hear']

Class: disappointment
    importance: 0.105,  concept 3,  topk ngrams: ['. that โ€™ s', 'you did', ". that 's", 'that โ€™ s a', 'food', 'that โ€™ s', 'that would', '. just', "that 's a", "! ''", "you 'll", 'hair', 'yes !', 'that they', '! it']
    importance: 0.065,  concept 1,  topk ngrams: ['kid', 'will get', 'i hope', 'will', 'guess', 'i would', 'the kid', 'i should', 'idk', 'i wish', 'will be', 'i will', 'sorry', '. i hope', 'would be']
    importance: 0.062,  concept 0,  topk ngrams: ['experience', "'ve been", 'live', 'amazing', 'i feel', 'work .', 'deep', 'time .', "it 's", 'โ€™ m so', 'ha a', "i 'm a", 'have been', 'beautiful', "you 've"]
    importance: 0.061,  concept 13, topk ngrams: ['enjoy', 'awful', 'wow ,', 'wow', 'happy', 'worried', 'my [', 'living', 'super', 'oh ,', 'so happy', '(', 'class', 'glad to', 'huge']
    importance: 0.058,  concept 6,  topk ngrams: ['living', 'care about', 'matter', 'live', 'relationship', 'a couple', 'especially', 'i feel', 'experience', 'work .', 'completely', 'couple', 'unfortunately', 'working', 'least']

Class: disapproval
    importance: 0.151,  concept 3,  topk ngrams: ['haha', 'oh ,', 'to use', 'baby', 'ha', 'to make', 'oh', 'used to', 'to play', 'to me .', 'save', 'important', 'edit :', ';', 'found']
    importance: 0.064,  concept 13, topk ngrams: ['it ?', 'you ?', 'i agree', 'right ?', 'what ?', 'this ?', 'name ] ?', '? it', 'i like', 'i love', '? i', '? that', '? ?', '] ?', '[ name ] ?']
    importance: 0.064,  concept 1,  topk ngrams: ['i see', '. i just', 'anyone', 'it look', 'i really', 'i thought it', 'i think', 'i did', 'i could', 'it โ€™ s not', 'that โ€™', ', not', 'i had', 'but i', 'i just']
    importance: 0.063,  concept 18, topk ngrams: ['original', 'beautiful', 'change', 'cute', 'biggest', 'amazing', 'dead', 'absolute', 'awesome .', 'good', 'control', 'good .', 'entire', 'awesome', 'figure']
    importance: 0.055,  concept 15, topk ngrams: ['it just', 'i think it', 'it !', 'it is', 'he is', "i 'm just", 'him .', "you 'll", ', but it', 'i don โ€™ t', 'it out', 'it look', 'i think i', ', this is', 'everyone']

Class: disgust
    importance: 0.092,  concept 1,  topk ngrams: ['i hope you', 'hope', 'hope you', '. i hope', 'i hope', 'hopefully', 'awesome .', 'i wish', 'awesome', 'care about', 'worry', 'i appreciate', 'wish', 'appreciate', 'that make']
    importance: 0.087,  concept 18, topk ngrams: ['talking', 'difference', '] ?', '? that', 'right ?', 'watching', 'sharing', 'agree', '? i', 'talking about', 'what ?', '? it', 'getting', '. if', 'comment']
    importance: 0.085,  concept 13, topk ngrams: ['what', 'want', 'โ€™ s what', "what 's", 'what you', 's what', 'only', ', what', 'didn โ€™', 'already', '. what', 'so happy', 'than', 'that โ€™ s what', 'enjoy']
    importance: 0.085,  concept 6,  topk ngrams: ['i hope you', 'hope', 'hope you', 'i hope', '. i hope', 'wish', 'hopefully', 'hoping', 'i wish', 'guess', 'wish i', 'kinda', 'i guess', 'think', 'i think i']
    importance: 0.081,  concept 11, topk ngrams: ['sorry for', 'sorry', 'so sorry', 'not sure', 'not be', 'not .', 'not to', "'s not", "'re not", 'nothing', 't think', 'terrible', 'not', 'i can โ€™ t', 'โ€™ t think']

Class: embarrassment
    importance: 0.097,  concept 1,  topk ngrams: ['though .', 'see', 'yes !', 'so you', ':', ';', "''", "'' .", "! ''", 'so', 'they did', '] and [ name', '] would', 'on', 'is it']
    importance: 0.075,  concept 7,  topk ngrams: ['live in', 'i feel', ', it โ€™', 'i used', 'so happy', 'so sorry', 'i ca', 'see', 'ha', '! ! ! !', 'water', 'like you', 'i โ€™ m so', 'life .', 'i am']
    importance: 0.066,  concept 10, topk ngrams: ['my [ name ]', 'enjoy', "''", 'a fucking', 'so happy', 'whole', 'wow', '] and [ name ]', 'it would', '] and [ name', 'save', 'the fuck', '. it โ€™', 'stupid', 'my life']
    importance: 0.063,  concept 13, topk ngrams: ['thought it', 'i thought it', 'think it', 'remember', 'think', 'i think i', 'i should', 'will be', 'think that', 'maybe', 'could be', 'you should', 'one .', 'i agree', 'not be']
    importance: 0.061,  concept 8,  topk ngrams: ['at least', "is n't", 'i don โ€™ t', 'honestly', "n't even", 'i still', "i did n't", "i ca n't", "wa n't", 'already', 'totally', 'than the', 'though .', 'it โ€™ s not', 'work .']

Class: excitement
    importance: 0.093,  concept 15, topk ngrams: ['would love', 'i love this', 'i love it', 'i love the', 'i love', 'love', '] ?', 'love this', '? i', 'love it .', 'i would', 'love it', 'what ?', '? that', 'love to']
    importance: 0.079,  concept 17, topk ngrams: ['fuck', 'shit', 'worst', 'i hate', 'horrible', 'the fuck', 'suck', 'the worst', 'stupid', 'the kid', 'worried', "n't want", 'i wish', 'disgusting', 'luck']
    importance: 0.078,  concept 12, topk ngrams: ['talking', 'right ?', 'saw', 'seeing', 'seen', 'heard', 'sure', 'out', 'killed', '[ name ] ?', '! ? !', 'why', 'thinking', 'it out', 'confused']
    importance: 0.075,  concept 13, topk ngrams: ['you ?', 'right ?', 'it ?', 'what ?', 'favorite', 'this ?', '! ?', '? ! ?', '! ? !', '? ?', '?', '? !', '? that', 'name ] ?', '? it']
    importance: 0.073,  concept 2,  topk ngrams: ['i like', 'love it .', 'like it', 'i love', 'like you', 'love it', 'i don โ€™', 'i love the', 'i love it', 'i love this', 'love', 'love the', 'love this', '. and', ', and i']

Class: fear
    importance: 0.096,  concept 6,  topk ngrams: ['why', 'how it', 'how', 'what is', 'thought it', 'know how', "what 's", 'idea .', 'what', 'think', '. what', 'think you', 'confused', 'what ?', 'how to']
    importance: 0.08,   concept 1,  topk ngrams: ['why', 'surprised', 'โ€™ s what', 'i appreciate', 'confused', 'that โ€™ s what', 'what', 'this ?', 'worse', 'didn โ€™', 'especially', "what 's", 'terrible', 'too .', 'watching']
    importance: 0.074,  concept 10, topk ngrams: ['disgusting', 'the fuck', 'fuck', 'a fucking', 'something', 'nothing', 'isn โ€™ t', "i ca n't", 'not a', 'it โ€™ s not', 'not even', 'nobody', 'suck', "n't", 'stupid']
    importance: 0.068,  concept 19, topk ngrams: ['other', 'deserve', 'parent', 'friend', 'a few', 'support', 'problem', 'government', 'mother', 'the other', 'wife', 'a couple', 'state', '* *', 'relationship']
    importance: 0.067,  concept 0,  topk ngrams: ['worry', 'wow ,', 'haha', 'oh', 'oh ,', 'wow', 'working', 'kid', 'ha been', 'it look', 'so happy', 'it โ€™ s a', 'worried', 'he ha', 'lmao']

Class: gratitude
    importance: 0.108,  concept 11, topk ngrams: ['my favorite', 'favorite', 'favourite', 'the best', 'cute', 'awful', 'good', 'best', 'worried', 'amazing', 'welcome', 'good .', 'love it .', 'the fuck', 'a good']
    importance: 0.066,  concept 10, topk ngrams: ['cut', 'so happy', '. thank', 'strong', 'my favorite', 'happy', 'huge', 'a great', 'proud', 'thank', 'than i', 'a good', 'genuinely', 'worth', 'good']
    importance: 0.064,  concept 3,  topk ngrams: ['behind', 'away from', "n't believe", "n't think", 'wrong .', 'not even', 'away', 'not the', 'not to', 'i hate', 'killed', 'died', 'โ€™ t think', 'left', "it 's not"]
    importance: 0.063,  concept 17, topk ngrams: ['i love it', 'i love this', 'i love the', 'love it .', 'love it', 'i love', 'love this', 'love', 'love the', 'i like', 'would love', 'like you', 'love to', 'my favorite', 'favorite']
    importance: 0.062,  concept 2,  topk ngrams: ['appreciate', 'is awesome', 'awesome .', 'i appreciate', 'awesome', 'wow ,', 'great', 'cute', 'good luck', 'good .', 'wow', 'i hope you', 'i can โ€™ t', 'so happy', 'wasn โ€™ t']

Class: grief โ€” no samples predicted, skipping.

Class: joy
    importance: 0.081,  concept 17, topk ngrams: ['i love it', 'i love the', 'love it .', 'i love this', 'i like', 'like you', 'i love', 'love it', 'love this', 'so sorry', 'the fuck', 'sorry for', 'class', 'like it', 'fucking']
    importance: 0.074,  concept 13, topk ngrams: ['you will', 'i would', 'i can โ€™ t', 'you can', 'we can', "i did n't", 'it would', 'sorry for', 'will', 'sorry', 'so sorry', "n't have", "would n't", 'i don โ€™ t', 'it โ€™ s not']
    importance: 0.066,  concept 12, topk ngrams: ['i hope you', 'i hope', '. i hope', 'i thought it', 'hope', 'i think it', 'i used', 'the kid', 'i think i', 'i wish', 'it would', 'hope you', 'i agree', 'kid', 'you will']
    importance: 0.065,  concept 11, topk ngrams: ['i hope', 'i hope you', 'hope', '. i hope', 'i wish', 'the kid', 'wish', 'so happy', 'hope you', 'hilarious', 'hoping', 'worry', 'so sorry', 'i thought', 'expect']
    importance: 0.063,  concept 18, topk ngrams: ['so happy', 'happy', 'loved', 'amazing', 'pretty', 'whole', 'the best', 'entire', 'beautiful', 'i appreciate', 'awesome', 'cute', 'a good', 'kind', 'nothing']

Class: love
    importance: 0.105,  concept 10, topk ngrams: ['favourite', 'my favorite', 'like this', 'like it', 'favorite', 'like you', 'i like', 'sweet', 'loved', 'love it .', 'like the', 'enjoy', 'like that', 'it just', 'like he']
    importance: 0.081,  concept 17, topk ngrams: ["i 'm not", 'i can โ€™ t', 'i โ€™ m not', 'is not', "'m not", 'it โ€™ s not', "'re not", 'thanks for the', "it 's not", "'s not", 'thanks', "i do n't", ', thanks for', 'โ€™ s not', 'thanks for']
    importance: 0.073,  concept 19, topk ngrams: ['appreciate', 'awesome .', 'awesome', 'sorry', 'sorry for', 'surprised', 'i appreciate', 'worry', 'cut', 'ever', ', just', 'welcome', 'started', 'wow', 'reply']
    importance: 0.07,   concept 7,  topk ngrams: ['favourite', 'favorite', 'do not', 'no', 'doesn โ€™ t', "have n't", 'have no', "ca n't", "if you do n't", "did n't", 'did not', 'the fuck', 'when they', 'out of', 'no .']
    importance: 0.068,  concept 16, topk ngrams: ['thanks for the', 'thanks for', 'thanks', 'thanks !', 'thanks ,', ', thanks for', '. thanks for', 'thanks .', ', thanks', 'thank you so much', ', thank you', 'thank you for', '. thank you', 'thank you so', '. thanks']

Class: nervousness
    importance: 0.094,  concept 4,  topk ngrams: ['! ?', 'it ?', '? that', 'you ?', 'what ?', 'this ?', '? i', '?', '? !', '? ?', 'amazing', 'wow', 'name ] ?', '] ?', '[ name ] ?']
    importance: 0.068,  concept 18, topk ngrams: ['curious', 'i did', '. what', 'oh ,', 'posting', 'man .', ". ''", '. i wa', 'make', "'d be", 'so .', 'worried', 'have to', 'i don', "it 's a"]
    importance: 0.066,  concept 15, topk ngrams: ['honestly', 'โ€™ s a', 'luck', 'anything', 'and', 'unfortunately', 'hit', "i did n't", 'when he', 'why', 'though .', 'sound', 'i did', 'to think', 'worried']
    importance: 0.065,  concept 8,  topk ngrams: ['to have', 'back to', 'didn โ€™', 'without', 'doesn โ€™ t', "if you 're", 'right now', 'away from', 'meant', 'out of', 'saying', 'hurt', 'feel like', "have n't", 'rather']
    importance: 0.061,  concept 13, topk ngrams: ['working', 'curious', 'upvote', ': )', 'do .', 'they โ€™', 'i have a', 'okay', '. i', 'stop', '] , [', 'so i', 'change', 'effort', 'man .']

Class: optimism
    importance: 0.078,  concept 9,  topk ngrams: [': (', 'should', 'should have', 'can โ€™', 'that [ name', 'would', 'had', 'could', ', we', ', [ name', 'read', 'might', '] , [ name ]', '/s', 'wait']
    importance: 0.065,  concept 14, topk ngrams: ['. good', 'my [ name', 'a [', '. [', 'a [ name', 'is awesome', 'try to', 'the [', 'the [ name', 'single', 'cute', 'to [', 'original', '. [ name', '[ name']
    importance: 0.063,  concept 5,  topk ngrams: ['โ€™ s what', 'right ?', 'what ?', ', what', '? ?', '?', '? ! ?', 'that โ€™ s what', 'how', '] ?', 'this ?', 'i can โ€™ t', 'a great', 'kinda', 'you ?']
    importance: 0.063,  concept 6,  topk ngrams: ['idea', 'okay', 'know ,', 'hope', 'wa', 'idea .', 'kinda', 'first', 'think', 'that wa', 'fan', 'remember', 'especially', '. i hope', 'sure']
    importance: 0.059,  concept 8,  topk ngrams: ['i wish', 'wish', 'loved', 'wish i', 'you want', '. i hope', 'the kid', 'i hope', 'guess', 'the best', 'i want', 'genuinely', 'hope', 'idea', 'i want to']

Class: pride โ€” no samples predicted, skipping.

Class: realization
    importance: 0.082,  concept 4,  topk ngrams: ['>', '(', 'sorry', ': (', ')', ': )', 'wait', 'example', '/s', 'dont', '-', 'can โ€™', 'class', 'dog', 'my [']
    importance: 0.076,  concept 16, topk ngrams: ['name ] ?', '? ?', 'right ?', '?', '? ! ?', '? it', '? !', '[ name ] ?', '? that', 'it ?', '! ? !', '! ?', 'you ?', '? i', 'what ?']
    importance: 0.068,  concept 17, topk ngrams: ['disgusting', 'living', 'stupid', 'unfortunately', 'i agree', 'worried', 'the worst', 'problem .', 'happen', 'experience', 'wrong .', 'worse', 'confused', 'least', 'terrible']
    importance: 0.064,  concept 7,  topk ngrams: ['rude', 'okay', 'i thought it', 'money', 'hope you', 'that โ€™ s what', 'hair', 'i hate', 'ever', 'wish i', 'omg', 'genuinely', 'god', 'why', 'original']
    importance: 0.063,  concept 6,  topk ngrams: ['โ€™ s what', 's what', '] ?', 'what', 'is what', ', what', 'trust', ', we', '] , [', '] ,', ', you are', '. what', 'also ,', ', [ name', "'ve been"]

Class: relief
    importance: 0.085,  concept 12, topk ngrams: ['amazing', 'awesome', 'awesome .', 'i appreciate', 'pretty', 'appreciate', 'good .', 'cute', 'lmao', 'great', 'a great', 'a good', 'is awesome', 'good', "i did n't"]
    importance: 0.084,  concept 4,  topk ngrams: ['i never', 'โ€™ t think', 'it would', 'loved', 'please', 'totally', 'used', 'not .', 'well .', "could n't", 'able to', 'strong', 'made', 'left', 'the only one']
    importance: 0.078,  concept 19, topk ngrams: ['i hate', 't', 'so sorry', "i do n't", 'lost', "i ca n't", 'i never', 'sorry', 'wow', "n't get", 'not .', 'though .', 'i โ€™ m not', 'laugh', 'didn โ€™']
    importance: 0.066,  concept 3,  topk ngrams: ['left', 'expect', 'different', 'i could', 'โ€™ t think', 'except', 'problem', 'curious', 'effort', 'support', 'buddy', '. i hope', 'sharing', 'guess', 'and not']
    importance: 0.059,  concept 1,  topk ngrams: ['expect', 'the only', 'long', 'different', 'used', 'left', 'afraid', 'wish', '. i hope', 'trust', 'experience', 'the only one', 'kind', 'worried', 'i need']

Class: remorse
    importance: 0.091,  concept 13, topk ngrams: [', thanks', 'thanks', ', thanks for', '. thanks for', 'thanks for the', 'thanks ,', 'thanks for', 'thanks .', '. thanks', 'thanks !', '. thank', 'great', ', thank you', 'good luck', 'the best']
    importance: 0.085,  concept 15, topk ngrams: ['thank you !', 'thank you for', 'thank you so much', 'thank you so', 'thank you', 'thank you .', ', thank you', 'thank', '. thank', ', thank', '] ?', '. thank you', 'what ?', 'it ?', 'this ?']
    importance: 0.073,  concept 2,  topk ngrams: ['so happy', 'like you', 'would love', 'take a', 'happy', 'i agree', 'need a', 'i hope you', 'the game', 'i love this', 'happens', 'wish i', 'hoping', 'i love it', 'watched']
    importance: 0.073,  concept 17, topk ngrams: ['i can โ€™ t', 'i appreciate', 'appreciate', 'lmao', 'i don โ€™ t', 'awesome .', 'i will', 'a good', 'i see', "i do n't", "is n't", 'awesome', "i did n't", 'cute', 'surprised']
    importance: 0.072,  concept 8,  topk ngrams: ['thanks ,', 'thanks .', '. thanks for', '. thanks', 'thanks', ', thanks', 'thanks for the', ', thanks for', 'thanks !', 'thanks for', 'i would', 'it would', 'the game', 'the same thing', 'thank you so much']

Class: sadness
    importance: 0.08,   concept 3,  topk ngrams: ['. thanks', 'thanks .', 'hope', 'i hope you', ', thanks', '. thanks for', 'hope you', 'thank you for', 'thanks ,', 'thanks', ', thank you', 'i hope', ', thanks for', 'thank you so much', 'thanks for the']
    importance: 0.075,  concept 16, topk ngrams: ['not even', "n't even", 'instead', 'even if', 'doesn โ€™ t', 'stay', 'too .', 'don โ€™ t think', 'without', 'outside', 'feeling', 'didn', "'s not", 'trying', 'not .']
    importance: 0.072,  concept 4,  topk ngrams: ['experience', 'i wish', 'unfortunately', 'i thought', 'i used', 'hope', 'the kid', 'i can โ€™ t', 'hopefully', 'nothing', 'hoping', 'help', 'heart', '. i hope', 'wish']
    importance: 0.07,   concept 0,  topk ngrams: ['. thanks', 'welcome', 'care about', 'to use', ', thanks', 'name ] and', 'thanks', 'name ] and [ name', '. thanks for', 'name ] and [', 'name ] , [', '[ name ] , [', 'thanks .', '[ name ] and', ', thanks for']
    importance: 0.069,  concept 12, topk ngrams: ['yourself .', '``', 'โ€', 'โ€™', 'โ€œ', 'myself', 'like it', '. it โ€™', ', it โ€™', 'yourself', 'him .', "i 'm just", ', i โ€™', 'it just', 'live']

Class: surprise
    importance: 0.087,  concept 16, topk ngrams: ['enjoy', 'away', 'fucking', 'expect', 'fun', 'out', 'out .', 'seriously', "n't have", 'nothing', 'happy', 'away from', 'better .', 'awful', 'cool .']
    importance: 0.08,   concept 10, topk ngrams: [', i โ€™', '. it โ€™', 'hold', 'damn ,', 'else', "he 's", ', it โ€™', 'her', 'date', 'i โ€™', ')', 'damn', 'leave', 'hey', 'guy']
    importance: 0.071,  concept 17, topk ngrams: ['why', 'i used', 'i thought', '``', 'fucking', 'confused', 'something', ', but it', ', what', 'problem .', 'stupid', 'i thought it', 'horrible', 'unfortunately', 'terrible']
    importance: 0.071,  concept 1,  topk ngrams: ['haha', 'oh', 'joke', 'lmao', 'lol', 'omg', 'i hope', 'i hope you', 'worry', 'kid', 'wow', 'a joke', 'so happy', 'lol ,', 'i see']
    importance: 0.067,  concept 2,  topk ngrams: ['expect', 'anything', 'is not', '! ? !', 'do not', 'so .', '. not', 'ta', 'attention', 'area', 'surprised', '. he', 'tbh', 'everyone', 'time']

Class: neutral
    importance: 0.102,  concept 2,  topk ngrams: ['happened', 'name ] ,', 'name ] .', 'name ] and', 'him .', 'name ]', 'him', 'name ] ha', 'name ] is', 'name ] and [', 'her .', 'name', 'his', 'one is', 'name ] and [ name']
    importance: 0.076,  concept 3,  topk ngrams: ['a fucking', 's a', '(', 's', ': (', 'fucking', 'a', 'stupid', 'life .', 'really', 'โ€™ s a', 'be a', 'cool .', '>', 'just']
    importance: 0.07,   concept 8,  topk ngrams: ["if you 're", 'even if', 'feel like', 'if we', 'when they', 'by [ name', 'if this', 'if they', 'pick', 'because they', 'because you', 'when it', 'if the', 'posted', 'if you']
    importance: 0.067,  concept 17, topk ngrams: ['so sorry', 'sorry for', 'sorry', "is n't", "i 'm not", "do n't", 'i can โ€™ t', "n't be", "ca n't", "i ca n't", 'i โ€™ m not', "'m not", "n't", "n't get", "i do n't"]
    importance: 0.064,  concept 7,  topk ngrams: ['that you', 'that i', 'that we', "that 's the", 'that would be', 'that it', '. that โ€™', 'today', 'that โ€™', 'that they', 'when i', 'when you', 'that is', 'it would', 'that make']

plot_concepts(
    classes_names=classes_names,
    concepts_importances={classes_names[t]: concept_importances[t] for t in concept_importances},
    concepts_labels={
        classes_names[t]: {
            cid: list(ngrams.keys()) for cid, ngrams in concept_interpretations[t].items() if ngrams is not None
        }
        for t in concept_interpretations
    },
    top_k=5,
)

Classes