Attack Recipes API

We provide a number of pre-built attack recipes, which correspond to attacks from the literature. To run an attack recipe from the command line, run:

textattack attack --recipe [recipe_name]

To initialize an attack in Python script, use:

<recipe name>.build(model_wrapper)

For example, attack = InputReductionFeng2018.build(model) creates attack, an object of type Attack with the goal function, transformation, constraints, and search method specified in that paper. This object can then be used just like any other attack; for example, by calling attack.attack_dataset.

TextAttack supports the following attack recipes (each recipe’s documentation contains a link to the corresponding paper):

Attacks on classification models

0. A2T (Towards Improving Adversarial Training of NLP Models” (Yoo et al., 2021))

1. Alzantot Genetic Algorithm (Generating Natural Language Adversarial Examples)

2. Faster Alzantot Genetic Algorithm (Certified Robustness to Adversarial Word Substitutions)

3. BAE (BAE: BERT-Based Adversarial Examples)

4. BERT-Attack: (BERT-Attack: Adversarial Attack Against BERT Using BERT)

5. CheckList: (Beyond Accuracy: Behavioral Testing of NLP models with CheckList)

6. DeepWordBug (Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers)

7. HotFlip (HotFlip: White-Box Adversarial Examples for Text Classification)

8. Improved Genetic Algorithm (Natural Language Adversarial Attacks and Defenses in Word Level)

9. Input Reduction (Pathologies of Neural Models Make Interpretations Difficult)

10. Kuleshov (Adversarial Examples for Natural Language Classification Problems)

11. Particle Swarm Optimization (Word-level Textual Adversarial Attacking as Combinatorial Optimization)

12. PWWS (Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency)

13. TextFooler (Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment)

14. TextBugger (TextBugger: Generating Adversarial Text Against Real-world Applications)

15. Pruthi (Combating Adversarial Misspellings with Robust Word Recognition 2019)

16. CLARE (Contextualized Perturbation for Textual Adversarial Attack 2020)

17. LEAP (LEAP: Efficient and Automated Test Method for NLP Software 2023) – a Levy-flight/adaptive-inertia variant of the Particle Swarm Optimization search used by recipe 11 above

A2T (A2T: Attack for Adversarial Training Recipe)

class textattack.attack_recipes.a2t_yoo_2021.A2TYoo2021(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Towards Improving Adversarial Training of NLP Models.

(Yoo et al., 2021)

https://arxiv.org/abs/2109.00544

static build(model_wrapper, mlm=False)

Build attack recipe.

Parameters:

• model_wrapper (ModelWrapper) – Model wrapper containing both the model and the tokenizer.

• mlm (bool, optional, defaults to False) – If True, load A2T-MLM attack. Otherwise, load regular A2T attack.

Returns:

A2T attack.

Return type:

Attack

Alzantot Genetic Algorithm

(Generating Natural Language Adversarial Examples)

Warning

This attack uses a very slow language model. Consider using the faster-alzantot recipe instead.

class textattack.attack_recipes.genetic_algorithm_alzantot_2018.GeneticAlgorithmAlzantot2018(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Alzantot, M., Sharma, Y., Elgohary, A., Ho, B., Srivastava, M.B., & Chang, K. (2018).

Generating Natural Language Adversarial Examples.

https://arxiv.org/abs/1804.07998

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Faster Alzantot Genetic Algorithm

(Certified Robustness to Adversarial Word Substitutions)

class textattack.attack_recipes.faster_genetic_algorithm_jia_2019.FasterGeneticAlgorithmJia2019(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Certified Robustness to Adversarial Word Substitutions.

Robin Jia, Aditi Raghunathan, Kerem Göksel, Percy Liang (2019).

https://arxiv.org/pdf/1909.00986.pdf

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

BAE (BAE: BERT-Based Adversarial Examples)

class textattack.attack_recipes.bae_garg_2019.BAEGarg2019(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Siddhant Garg and Goutham Ramakrishnan, 2019.

BAE: BERT-based Adversarial Examples for Text Classification.

https://arxiv.org/pdf/2004.01970

This is “attack mode” 1 from the paper, BAE-R, word replacement.

We present 4 attack modes for BAE based on the

R and I operations, where for each token t in S: • BAE-R: Replace token t (See Algorithm 1) • BAE-I: Insert a token to the left or right of t • BAE-R/I: Either replace token t or insert a token to the left or right of t • BAE-R+I: First replace token t, then insert a token to the left or right of t

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

BERT-Attack:

(BERT-Attack: Adversarial Attack Against BERT Using BERT)

Note

Candidate size max_candidates defaults to 8 (down from the paper’s 48) to keep runtime tractable; see https://github.com/QData/TextAttack/issues/586. Pass a larger max_candidates to WordSwapMaskedLM for closer fidelity to the original paper at the cost of much slower attacks.

class textattack.attack_recipes.bert_attack_li_2020.BERTAttackLi2020(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Li, L.., Ma, R., Guo, Q., Xiangyang, X., Xipeng, Q. (2020).

BERT-ATTACK: Adversarial Attack Against BERT Using BERT

https://arxiv.org/abs/2004.09984

This is “attack mode” 1 from the paper, BAE-R, word replacement.

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

CheckList:

(Beyond Accuracy: Behavioral Testing of NLP models with CheckList)

class textattack.attack_recipes.checklist_ribeiro_2020.CheckList2020(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

An implementation of the attack used in “Beyond Accuracy: Behavioral Testing of NLP models with CheckList”, Ribeiro et al., 2020.

This attack focuses on a number of attacks used in the Invariance Testing Method: Contraction, Extension, Changing Names, Number, Location

https://arxiv.org/abs/2005.04118

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

DeepWordBug

(Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers)

class textattack.attack_recipes.deepwordbug_gao_2018.DeepWordBugGao2018(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Gao, Lanchantin, Soffa, Qi.

Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers.

https://arxiv.org/abs/1801.04354

static build(model_wrapper, use_all_transformations=True)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

HotFlip

(HotFlip: White-Box Adversarial Examples for Text Classification)

class textattack.attack_recipes.hotflip_ebrahimi_2017.HotFlipEbrahimi2017(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Ebrahimi, J. et al. (2017)

HotFlip: White-Box Adversarial Examples for Text Classification

https://arxiv.org/abs/1712.06751

This is a reproduction of the HotFlip word-level attack (section 5 of the paper).

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Improved Genetic Algorithm

(Natural Language Adversarial Attacks and Defenses in Word Level)

class textattack.attack_recipes.iga_wang_2019.IGAWang2019(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Xiaosen Wang, Hao Jin, Kun He (2019).

Natural Language Adversarial Attack and Defense in Word Level.

http://arxiv.org/abs/1909.06723

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Input Reduction

(Pathologies of Neural Models Make Interpretations Difficult)

class textattack.attack_recipes.input_reduction_feng_2018.InputReductionFeng2018(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Feng, Wallace, Grissom, Iyyer, Rodriguez, Boyd-Graber. (2018).

Pathologies of Neural Models Make Interpretations Difficult.

https://arxiv.org/abs/1804.07781

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Kuleshov2017

(Adversarial Examples for Natural Language Classification Problems)

class textattack.attack_recipes.kuleshov_2017.Kuleshov2017(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Kuleshov, V. et al.

Generating Natural Language Adversarial Examples.

https://openreview.net/pdf?id=r1QZ3zbAZ.

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Particle Swarm Optimization

(Word-level Textual Adversarial Attacking as Combinatorial Optimization)

class textattack.attack_recipes.pso_zang_2020.PSOZang2020(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Zang, Y., Yang, C., Qi, F., Liu, Z., Zhang, M., Liu, Q., & Sun, M. (2019).

Word-level Textual Adversarial Attacking as Combinatorial Optimization.

https://www.aclweb.org/anthology/2020.acl-main.540.pdf

Methodology description quoted from the paper:

“We propose a novel word substitution-based textual attack model, which reforms both the aforementioned two steps. In the first step, we adopt a sememe-based word substitution strategy, which can generate more candidate adversarial examples with better semantic preservation. In the second step, we utilize particle swarm optimization (Eberhart and Kennedy, 1995) as the adversarial example searching algorithm.”

And “Following the settings in Alzantot et al. (2018), we set the max iteration time G to 20.”

See also LEAP2023, a Levy-flight/ adaptive-inertia variant of this recipe’s Particle Swarm Optimization search.

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

PWWS

(Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency)

class textattack.attack_recipes.pwws_ren_2019.PWWSRen2019(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

An implementation of Probability Weighted Word Saliency from “Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency”, Ren et al., 2019.

Words are prioritized for a synonym-swap transformation based on a combination of their saliency score and maximum word-swap effectiveness. Note that this implementation does not include the Named Entity adversarial swap from the original paper, because it requires access to the full dataset and ground truth labels in advance.

https://www.aclweb.org/anthology/P19-1103/

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

TextFooler (Is BERT Really Robust?)

A Strong Baseline for Natural Language Attack on Text Classification and Entailment)

class textattack.attack_recipes.textfooler_jin_2019.TextFoolerJin2019(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Jin, D., Jin, Z., Zhou, J.T., & Szolovits, P. (2019).

Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment.

https://arxiv.org/abs/1907.11932

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

TextBugger

(TextBugger: Generating Adversarial Text Against Real-world Applications)

class textattack.attack_recipes.textbugger_li_2018.TextBuggerLi2018(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Li, J., Ji, S., Du, T., Li, B., and Wang, T. (2018).

TextBugger: Generating Adversarial Text Against Real-world Applications.

https://arxiv.org/abs/1812.05271

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

CLARE Recipe

(Contextualized Perturbation for Textual Adversarial Attack)

class textattack.attack_recipes.clare_li_2020.CLARE2020(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Li, Zhang, Peng, Chen, Brockett, Sun, Dolan.

“Contextualized Perturbation for Textual Adversarial Attack” (Li et al., 2020)

https://arxiv.org/abs/2009.07502

This method uses greedy search with replace, merge, and insertion transformations that leverage a pretrained language model. It also uses USE similarity constraint.

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Pruthi2019: Combating with Robust Word Recognition

class textattack.attack_recipes.pruthi_2019.Pruthi2019(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

An implementation of the attack used in “Combating Adversarial Misspellings with Robust Word Recognition”, Pruthi et al., 2019.

This attack focuses on a small number of character-level changes that simulate common typos. It combines:

• Swapping neighboring characters

• Deleting characters

• Inserting characters

• Swapping characters for adjacent keys on a QWERTY keyboard.

https://arxiv.org/abs/1905.11268

Parameters:

• model – Model to attack.

• max_num_word_swaps – Maximum number of modifications to allow.

static build(model_wrapper, max_num_word_swaps=1)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

LEAP

(LEAP: Efficient and Automated Test Method for NLP Software)

class textattack.attack_recipes.leap_2023.LEAP2023(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

LEAP: Efficient and Automated Test Method for NLP Software.

https://arxiv.org/abs/2308.11284

LEAP is a Levy-flight/adaptive-inertia variant of the Particle Swarm Optimization search used by PSOZang2020 (see ParticleSwarmOptimizationLEAP, which subclasses ParticleSwarmOptimization). Where PSOZang2020 swaps HowNet sememes and uses a fixed, globally-shared inertia weight, LEAP swaps WordNet synonyms, caps the total fraction of words modified via MaxModificationRate, and computes a per-particle inertia weight (Levy-flight sampled for above-average particles, fitness- interpolated otherwise) intended to better balance exploration and exploitation than PSOZang2020’s shared decaying weight.

Benchmark vs. PSOZang2020’s search method (2026-08-12)

Comparing search methods directly requires holding everything else fixed, since PSOZang2020 normally swaps HowNet sememes rather than WordNet synonyms – see tests/benchmark_leap_vs_pso.py for the script (not run in CI) that reproduces this: a vanilla ParticleSwarmOptimization attack built with LEAP2023’s own transformation/constraints/hyperparameters, so only the search method’s internals differ. Run against cnn-ag-news on the AG News test set (first N examples, seed 765):

	Unrestricted budget (n=20)	2000-query budget (n=100)
LEAP success rate PSO(WordNet) success rate	95.0% 95.0%	24.0% 23.0%
LEAP avg. queries PSO(WordNet) avg. queries	44,071.8 46,753.4	1,540.8 1,560.1
LEAP avg. % words changed PSO(WordNet) avg. changed	9.28% 9.87%	4.51% 5.04%
LEAP wall time PSO(WordNet) wall time	157.3s 353.6s	43.3s 46.5s

LEAP matches or modestly beats this apples-to-apples PSO baseline on every metric (success rate, query count, perturbation size, wall time), but the margin is small, not dramatic – and it narrows further under a realistic query budget, where both methods’ success rate collapses well below their unrestricted-budget numbers since neither search gets far enough in to fully exploit its strategy. Caveats: single model/dataset, a specific random seed, and this size of sample – treat as directional, not definitive, evidence for LEAP’s claimed advantage over vanilla PSO.

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Attacks on sequence-to-sequence models

18. MORPHEUS (It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations)

19. Seq2Sick (Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples)

MORPHEUS2020

(It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations)

class textattack.attack_recipes.morpheus_tan_2020.MorpheusTan2020(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Samson Tan, Shafiq Joty, Min-Yen Kan, Richard Socher.

It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations

https://www.aclweb.org/anthology/2020.acl-main.263/

Like Seq2SickCheng2018BlackBox, this works against any encoder-decoder generation model loaded via HuggingFaceModelWrapper, so it can attack machine-translation checkpoints (e.g. MarianMT, mBART, a BART checkpoint fine-tuned for translation) the same way it attacks the text-classification models used elsewhere in this codebase’s examples. MinimizeBleu’s ground_truth_output is the reference translation (not the model’s own unperturbed output, unlike NonOverlappingOutput/seq2sick), so pass the target-language reference sentence as the second argument to .attack(). See https://github.com/QData/TextAttack/issues/725.

Example, attacking an English-to-German translation model (any encoder-decoder checkpoint works the same way, including BART/MarianMT/ mBART translation checkpoints – swap in the model/tokenizer below for one of those):

import transformers
from textattack.attack_recipes import MorpheusTan2020
from textattack.models.wrappers import HuggingFaceModelWrapper

model = transformers.AutoModelForSeq2SeqLM.from_pretrained("t5-small")
tokenizer = transformers.AutoTokenizer.from_pretrained("t5-small")
# `max_length` avoids `transformers`' 20-token generation default,
# which truncates sentence-length translations.
model_wrapper = HuggingFaceModelWrapper(model, tokenizer, max_length=200)

attack = MorpheusTan2020.build(model_wrapper)
# t5-small needs the task prefix; other translation checkpoints
# (BART/MarianMT/mBART) typically don't.
input_text = "translate English to German: The quick brown fox jumps over the lazy dog."
reference_translation = "Der schnelle braune Fuchs springt über den faulen Hund."
result = attack.attack(input_text, reference_translation)
print(result.__str__(color_method="ansi"))

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Seq2Sick

(Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples)

class textattack.attack_recipes.seq2sick_cheng_2018_blackbox.Seq2SickCheng2018BlackBox(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Cheng, Minhao, et al.

Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

https://arxiv.org/abs/1803.01128

This is a greedy re-implementation of the seq2sick attack method. It does not use gradient descent.

Works against any encoder-decoder generation model loaded via HuggingFaceModelWrapper (e.g. a raw transformers.BartForConditionalGeneration or transformers.T5ForConditionalGeneration), not just TextAttack’s own T5ForTextToText helper. See https://github.com/QData/TextAttack/issues/771 and https://github.com/QData/TextAttack/issues/772.

Example, attacking a BART summarization model directly from transformers:

import transformers
from textattack.attack_recipes import Seq2SickCheng2018BlackBox
from textattack.models.wrappers import HuggingFaceModelWrapper

model = transformers.AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-large-cnn")
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
model_wrapper = HuggingFaceModelWrapper(model, tokenizer)

attack = Seq2SickCheng2018BlackBox.build(model_wrapper)
result = attack.attack(input_text, original_summary)
print(result.__str__(color_method="ansi"))

This also works against translation models (e.g. attacking BART/MarianMT on an en-de translation task): pass the model’s own unperturbed translation as the second argument to .attack(), since NonOverlappingOutput (unlike MinimizeBleu, used by MorpheusTan2020) compares against that rather than a ground-truth reference translation. See MorpheusTan2020 for a worked translation example and https://github.com/QData/TextAttack/issues/725.

static build(model_wrapper, goal_function='non_overlapping')

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

General

Unlike the recipes above, which are each locked to one task and one goal function, BadCharacters2021 takes a goal_function_type argument and can be configured to attack classification models, NER models, or seq2seq/translation models. Since it spans both of the sections above (and NER, which is neither), it’s listed here instead of being duplicated or assigned to one arbitrarily.

20. BadCharacters (Bad Characters: Imperceptible NLP Attacks)

Imperceptible Perturbations Algorithm

class textattack.attack_recipes.bad_characters_2021.BadCharacters2021(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

Imperceptible Perturbations Attack Recipe

Implements imperceptible adversarial attacks on NLP models as outlined in the Bad Characters paper.

This recipe combines imperceptible transformations with the Differential Evolution search method. It supports a variety of goal functions (targeted, untargeted, NER, translation) and several types of character-level perturbations.

Transformations supported:

• WordSwapInvisibleCharacters: injects invisible Unicode characters

• WordSwapHomoglyphSwap: replaces characters with homoglyphs

• WordSwapDeletions: inserts deletion control characters

• WordSwapReorderings: inserts reordering control characters

Goal functions supported:

• TargetedClassification

• TargetedStrict

• TargetedBonus

• LogitSum (for logits-based classifiers like toxic comment detection)

• MinimizeBleu (translation BLEU score minimization)

• MaximizeLevenshtein (translation Levenshtein distance maximization)

All transformations are compatible with all goal functions.

Note: This recipe assumes the model wrapper is compatible with the goal function chosen. For example, a NamedEntityRecognition goal function expects a model wrapper that outputs a list of dictionaries per input, while LogitSum expects an array of logits.

static build(model_wrapper, goal_function_type: str, perturbation_type: str = None, allow_skip: bool = False, perturbs=1, popsize=32, maxiter=10, **goal_function_kwargs)

Builds an imperceptible attack instance.

Parameters:

• model_wrapper (ModelWrapper) – A TextAttack model wrapper compatible with the selected goal function.

• goal_function_type (str, optional) –

  Goal function type. One of:

  • "targeted_classification": targeted attack on a classification model (default).

  • "targeted_strict": stricter targeted attack.

  • "targeted_bonus": bonus if prediction for target class is highest.

  • "named_entity_recognition": token-level NER attack.

  • "logit_sum": untargeted attack minimizing total logits.

  • "minimize_bleu": translation attack minimizing BLEU.

  • "maximize_levenshtein": translation attack maximizing Levenshtein distance.

• perturbation_type (str, optional) –

  Type of character-level perturbation. One of:

  • "homoglyphs" (default)

  • "invisible"

  • "deletions"

  • "reorderings"

• allow_skip (bool) – If False, the attack will continue even if the goal is already satisfied.

• perturbs (int) – Maximum number of perturbations allowed per input string.

• popsize (int) – Population size for differential evolution. Typically 32.

• maxiter (int) – Maximum number of generations for differential evolution. Typically 10.

• **goal_function_kwargs (dict) – Additional arguments passed to the goal function.

Returns:

Configured Attack instance.

Return type:

textattack.Attack

Multi-lingual attacks

TextAttack also includes recipes for non-English classification models, each a language-specific adaptation of a contextualized-perturbation attack (see https://github.com/QData/TextAttack/issues/423):

21. FrenchRecipe – Attack French Recipe (Contextualized Perturbation for French NLP Adversarial Attack)

22. SpanishRecipe – Attack Spanish Recipe (Contextualized Perturbation for Spanish NLP Adversarial Attack)

23. ChineseRecipe – Attack Chinese Recipe (Contextualized Perturbation for Chinese NLP Adversarial Attack)

Attack French Recipe

(Contextualized Perturbation for French NLP Adversarial Attack)

class textattack.attack_recipes.french_recipe.FrenchRecipe(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Attack Spanish Recipe

(Contextualized Perturbation for Spanish NLP Adversarial Attack)

class textattack.attack_recipes.spanish_recipe.SpanishRecipe(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack

Attack Chinese Recipe

(Contextualized Perturbation for Chinese NLP Adversarial Attack)

class textattack.attack_recipes.chinese_recipe.ChineseRecipe(goal_function: GoalFunction, constraints: List[Constraint | PreTransformationConstraint], transformation: Transformation, search_method: SearchMethod, transformation_cache_size=32768, constraint_cache_size=32768)

An implementation of the attack used in “Beyond Accuracy: Behavioral Testing of NLP models with CheckList”, Ribeiro et al., 2020.

This attack focuses on a number of attacks used in the Invariance Testing Method: Contraction, Extension, Changing Names, Number, Location

https://arxiv.org/abs/2005.04118

static build(model_wrapper)

Creates pre-built Attack that correspond to attacks from the literature.

Parameters:

• model_wrapper (ModelWrapper) – ModelWrapper that contains the victim model and tokenizer. This is passed to GoalFunction when constructing the attack.

• kwargs – Additional keyword arguments.

Returns:

Attack