The easiest way to use textattack is from the command-line. Installing textattack
will provide you with the handy
textattack command which will allow you to do
just about anything TextAttack offers in a single bash command.
Tip: If you are for some reason unable to use the
textattackcommand, you can access all the same functionality by prepending
python -mto the command (
python -m textattack ...).
examples/folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file.
The documentation website contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
To see all available commands, type
textattack --help. This page explains
some of the most important functionalities of textattack: NLP data augmentation,
adversarial attacks, and training and evaluating models.
Data Augmentation with
The easiest way to use our data augmentation tools is with
textattack augment <args>.
takes an input CSV file and text column to augment, along with the percentage of words to change per augmentation
and the number of augmentations per input example. It outputs a CSV in the same format with all the augmentation
examples corresponding to the proper columns.
For example, given the following as
"text",label "the rock is destined to be the 21st century's new conan and that he's going to make a splash even greater than arnold schwarzenegger , jean- claud van damme or steven segal.", 1 "the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .", 1 "take care of my cat offers a refreshingly different slice of asian cinema .", 1 "a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish line proves simply too discouraging to let slide .", 0 "it's a mystery how the movie could be released in this condition .", 0
textattack augment --csv examples.csv --input-column text --recipe eda --pct-words-to-swap .1 \ --transformations-per-example 2 --exclude-original
will augment the
text column with 10% of words edited per augmentation, twice as many augmentations as original inputs, and exclude the original inputs from the
output CSV. (All of this will be saved to
augment.csv by default.)
After augmentation, here are the contents of
text,label "the rock is destined to be the 21st century's new conan and that he's to make splash even greater arnold schwarzenegger , jean- claud van damme or steven segal.",1 "the Arnold rock is destined to be the 21st vanguard century's new specify conan and that he's going to make a splash even greater than arnold schwarzenegger , jean- claud van damme or steven segal.",1 the gorgeously continuation of 'the lord of the rings' trilogy is so huge that a column of cannot adequately describe co-writer/ peter jackson's expanded vision of j . r . r . tolkien's middle-earth .,1 the splendidly elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of parole cannot adequately describe co-writer/director peter jackson's expanded vision of J . r . r . tolkien's middle-earth .,1 take care of my cat offers a refreshingly slice different of asian cinema .,1 take care of my cast offers a refreshingly different slice of asian cinema .,1 a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish line proves simply too discouraging to rush let IT slide .,0 a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish line proves just too discouraging to let chute .,0 it's a mystery how the movie could this released in be condition .,0 it's a whodunit how the movie could be released in this condition .,0
The ‘eda’ augmentation uses a combination of word swaps, insertions, and substitutions to generate new examples.
Adversarial Attacks with
The heart of textattack is running adversarial attacks on NLP models with
textattack attack. You can build an attack from the command-line in several ways:
Use an attack recipe to launch an attack from the literature:
textattack attack --recipe deepwordbug
Build your attack from components:
textattack attack --model lstm-mr --num-examples 20 --search-method beam-search^beam_width=4 \ --transformation word-swap-embedding \ --constraints repeat stopword max-words-perturbed^max_num_words=2 embedding^min_cos_sim=0.8 part-of-speech \ --goal-function untargeted-classification
Create a python file that builds your attack and load it:
textattack attack --attack-from-file my_file.py^my_attack_name
Training Models with
With textattack, you can train models on any classification or regression task
datasets using a single line.
TextAttack has two build-in model types, a 1-layer bidirectional LSTM with a hidden
state size of 150 (
lstm), and a WordCNN with 3 window sizes
(3, 4, 5) and 100 filters for the window size (
cnn). Both models set dropout
to 0.3 and use a base of the 200-dimensional GLoVE embeddings.
Along with the
cnn, you can theoretically fine-tune any model based
in the huggingface transformers
repo. Just type the model name (like
bert-base-cased) and it will be automatically
Here are some models from transformers that have worked well for us:
Evaluating Models with
Any TextAttack-compatible model can be evaluated using
textattack eval-model. TextAttack-trained models can be evaluated using
textattack eval --num-examples <num-examples> --model /path/to/trained/model/
Some attacks can take a very long time. Sometimes this is because they’re using a very slow search method (like beam search with a high beam width) or sometimes they’re just attacking a large number of samples. In these cases, it can be useful to save attack checkpoints throughout the course of the attack. Then, if the attack crashes for some reason, you can resume without restarting from scratch.
To save checkpoints while running an attack, add the argument
--checkpoint-interval X, where X is the number of attacks you want to run between checkpoints (for example
textattack attack <args> --checkpoint-interval 5).
To load an attack from a checkpoint, use
textattack attack-resume --checkpoint-file <checkpoint-file>.
Listing features with
TextAttack has a lot of built-in features (models, search methods, constraints, etc.)
and it can get overwhelming to keep track of all the options. To list all of the
options within a given category, use
list all the built-in models:
textattack list models
list all constraints:
textattack list constraints
list all search methods:
textattack list search-methods
Examining datasets with
It can be useful to take a cursory look at and compute some basic statistics of
whatever dataset you’re working with. Whether you’re loading a dataset of your
own from a file, or one from NLP, you can use
textattack peek-dataset to
see some basic information about the dataset.
For example, use
textattack peek-dataset --dataset-from-huggingface glue^mrpc to see
information about the MRPC dataset (from the GLUE set of datasets). This will
print statistics like the number of labels, average number of words, etc.