TextAttack

Get Started

  • Basic-Introduction
    • What is TextAttack?
    • Where should I start?
    • NLP Attacks
    • Data Augmentation
    • Features
  • Installation
    • Install with pip
    • Install from Source
    • Optional Dependencies
    • FAQ on installation
  • Command-Line Usage
    • Data Augmentation with textattack augment
    • Adversarial Attacks with textattack attack
    • Training Models with textattack train
      • Available Models
        • TextAttack Models
        • transformers Models
    • Evaluating Models with textattack eval-model
    • Other Commands
      • Checkpoints and textattack attack-resume
      • Listing features with textattack list
      • Examining datasets with textattack peek-dataset
  • Quick API Usage
    • Attacking a BERT model
  • FAQ
    • Via Slack: Where to Ask Questions:
    • Via CLI: --help
    • Via our papers: More details on results
    • Via readthedocs: More details on APIs
    • More Concrete Questions:
      • 0. For many of the dependent library issues, the following command is the first you could try:
      • 1. How to Train
      • 2. Use Custom Models
        • Model from a file
      • 3. Use Custom Datasets
        • From a file
        • Dataset loading via other mechanism, see: more details at here
        • Custom Dataset via AttackedText class
      • 4. Benchmarking Attacks
      • 5. Create Custom or New Attacks
      • 6. The attacking is too slow

Recipes

  • Attack Recipes CommandLine Use
    • Help: textattack --help
    • Running Attacks: textattack attack --help
    • Attacks and Papers Implemented (”Attack Recipes”): textattack attack --recipe [recipe_name]
    • Recipe Usage Examples
  • Attack Recipes API
    • Attacks on classification models
      • A2T (A2T: Attack for Adversarial Training Recipe)
      • Alzantot Genetic Algorithm
      • Faster Alzantot Genetic Algorithm
      • BAE (BAE: BERT-Based Adversarial Examples)
      • BERT-Attack:
      • CheckList:
      • DeepWordBug
      • HotFlip
      • Improved Genetic Algorithm
      • Input Reduction
      • Kuleshov2017
      • Particle Swarm Optimization
      • PWWS
      • TextFooler (Is BERT Really Robust?)
      • TextBugger
      • CLARE Recipe
      • Pruthi2019: Combating with Robust Word Recognition
    • Attacks on sequence-to-sequence models
      • MORPHEUS2020
      • Seq2Sick
    • General
      • Imperceptible Perturbations Algorithm
  • Augmenter Recipes CommandLine Use
    • Augmenting Text: textattack augment
    • Augmentation Command-Line Interface
  • Augmenter Recipes API
    • Augmenter Recipes:
  • TextAttack Model Zoo
    • Available Models
      • TextAttack Models
      • transformers Models
    • Evaluation Results of Available Models
      • LSTM
      • wordCNN
      • albert-base-v2
      • bert-base-uncased
      • distilbert-base-cased
      • distilbert-base-uncased
      • roberta-base
      • xlnet-base-cased
    • How we have trained the TextAttack Models
    • Training details for each TextAttack Model
    • More details on TextAttack fine-tuned NLP models (details on target NLP task, input type, output type, SOTA results on paperswithcode; model card on huggingface):

Using TextAttack

  • What is an adversarial attack in NLP?
    • Terminology
    • Adversarial Examples in NLP
    • Generating adversarial examples with TextAttack
    • The future of adversarial attacks in NLP
  • How to Cite TextAttack
    • Main Paper: TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
      • Our Github on TextAttack: https://github.com/QData/TextAttack
    • Our Analysis paper: Reevaluating Adversarial Examples in Natural Language
      • Our Github on Reevaluation: Reevaluating-NLP-Adversarial-Examples Github
    • Our Analysis paper: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
      • Our Github on benchmarking: TextAttack-Search-Benchmark Github
    • Our defense paper: Title: “Towards Improving Adversarial Training of NLP Models”
      • Code is available
      • Citations:
    • Our extended use case paper: “Expanding Scope: Adapting English Adversarial Attacks to Chinese”
      • Abstract:
      • Venue:
      • Tutorial code:
      • Citations:
  • Four Components of TextAttack Attacks
    • Goal Functions
    • Constraints
    • Transformations
    • Search Methods
    • On Benchmarking Attack Recipes
    • Four components in Attack Recipes we have implemented
  • Benchmarking Search Algorithms for Generating NLP Adversarial Examples
    • Title: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
    • Our search benchmarking result Github
    • Our benchmarking results on comparing search methods used in the past attacks.
    • Benchmarking Attack Recipes
  • On Quality of Generated Adversarial Examples and How to Set Attack Contraints
    • Title: Reevaluating Adversarial Examples in Natural Language
    • Our Github on Reevaluation: Reevaluating-NLP-Adversarial-Examples Github
    • Some of our evaluation results on quality of two SOTA attack recipes
    • Some of our evaluation results on how to set constraints to evaluate NLP model’s adversarial robustness
  • Making Vanilla Adversarial Training of NLP Models Feasible!
    • Title: Towards Improving Adversarial Training of NLP Models
      • Video recording of this talk: https://underline.io/events/192/sessions/7928/lecture/38377-towards-improving-adversarial-training-of-nlp-models
    • Code is available
    • Citations:
    • A2T Attack Recipe
  • Lessons learned in designing TextAttack
    • Presentations on TextAttack
      • 2020: Jack Morris’ summary tutorial talk on TextAttack
      • 2021: Dr. Qi’s summary tutorial talk on TextAttack
    • Challenges in Design
    • Our design tips
    • TextAttack flowchart
    • More Details in Reference
  • TextAttack Extended Functions (Multilingual)
    • Textattack Supports Multiple Model Types besides huggingface models and our textattack models:
    • Multilingual Supports
    • User defined custom inputs and models
      • Custom Datasets: Dataset from a file
        • Custom Model: from a file
    • User defined Custom attack components
    • Visulizing TextAttack generated Examples;
  • How can I contribute to TextAttack?
    • Slack Channel
    • Ways to contribute
    • Submitting a new issue or feature request
      • Found a bug?
      • Do you want to add your model?
      • Do you want a new feature: a component, a recipe, or something else?
    • Start contributing! (Pull Requests)
      • Checklist
      • Tests
        • This guide was heavily inspired by the awesome transformers guide to contributing

Notebook Tutorials

  • Tutorial 0: TextAttack End-To-End (Train, Eval, Attack)
    • Training
    • Evaluation
    • Attack
    • Conclusion
    • Bonus
  • Tutorial 1: Transformations
    • Goal function
    • Search method
    • Transformation
    • Constraints
    • A custom transformation
    • Banana word swap
    • Using our transformation
    • Creating the goal function, model, and dataset
    • Creating the attack
    • Using the attack
    • Visualizing attack results
    • Conclusion
    • Bonus: Attacking Custom Samples
  • Tutorial 2: Constraints
    • Classes of constraints
    • A new constraint
    • A custom constraint
    • NLTK and Named Entity Recognition
    • NLTK NER Example
    • Caching with @functools.lru_cache
    • Putting it all together: getting a list of Named Entity Labels from a sentence
    • Creating our NamedEntityConstraint
    • Testing our constraint
    • Conclusion
  • Tutorial 3: Augmentation
    • Creating an Augmenter
    • Pre-built Augmentation Recipes
    • Conclusion
  • Tutorial 4: Custom Word Embeddings
    • Importing the Model
    • Creating A Custom Dataset
    • Creating An Attack
    • Attack Results With Custom Dataset
    • Creating A Custom Word Embedding
    • Attack Results With Custom Dataset and Word Embedding
  • Tutorial 5: Attacking TensorFlow models
    • Run textattack on a trained tensorflow model:
      • First: Training
      • Attacking
    • Conclusion
  • Tutorial 6: Attacking scikit-learn models
    • Training
    • Attacking
    • Conclusion
  • Tutorial 7: Attacking AllenNLP models
  • Tutorial 8: Attacking Keras models
    • This notebook runs textattack on a trained keras model:
    • Training
    • Attacking
    • Conclusion
  • Tutorial 9: Attacking multilingual models
  • Tutorial10: Explaining Attacking BERT model using Captum
  • Tutorial11: Attacking multilingual - Chinese NLP model using Textattack

API User Guide

  • Attack
    • Attack
      • Attack
        • Attack.attack()
        • Attack.cpu_()
        • Attack.cuda_()
        • Attack.filter_transformations()
        • Attack.get_indices_to_order()
        • Attack.get_transformations()
    • AttackRecipe
      • AttackRecipe
        • AttackRecipe.build()
  • Attacker
    • Attacker
      • Attacker
        • Attacker.attack_dataset()
        • Attacker.from_checkpoint()
        • Attacker.update_attack_args()
    • AttackArgs
      • AttackArgs
        • AttackArgs.create_loggers_from_args()
  • AttackResult
    • AttackResult
      • AttackResult
        • AttackResult.diff_color()
        • AttackResult.goal_function_result_str()
        • AttackResult.original_text()
        • AttackResult.perturbed_text()
        • AttackResult.str_lines()
    • SuccessfulAttackResult
      • SuccessfulAttackResult
    • FailedAttackResult
      • FailedAttackResult
        • FailedAttackResult.goal_function_result_str()
        • FailedAttackResult.str_lines()
    • SkippedAttackResult
      • SkippedAttackResult
        • SkippedAttackResult.goal_function_result_str()
        • SkippedAttackResult.str_lines()
    • MaximizedAttackResult
      • MaximizedAttackResult
  • Trainer
    • Trainer
      • Trainer
        • Trainer.evaluate()
        • Trainer.evaluate_step()
        • Trainer.get_eval_dataloader()
        • Trainer.get_optimizer_and_scheduler()
        • Trainer.get_train_dataloader()
        • Trainer.train()
        • Trainer.training_step()
    • TrainingArgs
      • TrainingArgs
  • Datasets
    • Dataset
      • Dataset
        • Dataset.__getitem__()
        • Dataset.__len__()
    • HuggingFaceDataset
      • HuggingFaceDataset
        • HuggingFaceDataset.__getitem__()
        • HuggingFaceDataset.__len__()
  • GoalFunction
    • GoalFunction
      • GoalFunction
        • GoalFunction.extra_repr_keys()
        • GoalFunction.get_output()
        • GoalFunction.get_result()
        • GoalFunction.get_results()
        • GoalFunction.init_attack_example()
    • LogitSum
      • LogitSum
    • NamedEntityRecognition
      • NamedEntityRecognition
    • TargetedStrict
      • TargetedStrict
    • TargetedBonus
      • TargetedBonus
    • ClassificationGoalFunction
      • ClassificationGoalFunction
        • ClassificationGoalFunction.extra_repr_keys()
    • TargetedClassification
      • TargetedClassification
        • TargetedClassification.extra_repr_keys()
    • UntargetedClassification
      • UntargetedClassification
    • InputReduction
      • InputReduction
        • InputReduction.extra_repr_keys()
    • TextToTextGoalFunction
      • TextToTextGoalFunction
    • MinimizeBleu
      • MinimizeBleu
        • MinimizeBleu.extra_repr_keys()
    • NonOverlappingOutput
      • NonOverlappingOutput
    • MaximizeLevenshtein
      • MaximizeLevenshtein
        • MaximizeLevenshtein.extra_repr_keys()
  • Constraints
    • Constraint
      • Constraint
        • Constraint.call_many()
        • Constraint.check_compatibility()
        • Constraint.extra_repr_keys()
    • PreTransformationConstraint
      • PreTransformationConstraint
        • PreTransformationConstraint.check_compatibility()
        • PreTransformationConstraint.extra_repr_keys()
  • Transformations
    • Transformation
      • Transformation
    • Composite Transformation
      • CompositeTransformation
  • SearchMethod
    • SearchMethod
      • SearchMethod
        • SearchMethod.check_transformation_compatibility()
        • SearchMethod.perform_search()
        • SearchMethod.is_black_box
    • BeamSearch
      • BeamSearch
        • BeamSearch.extra_repr_keys()
        • BeamSearch.perform_search()
        • BeamSearch.is_black_box
    • GreedySearch
      • GreedySearch
        • GreedySearch.extra_repr_keys()
    • GreedyWordSwapWIR
      • GreedyWordSwapWIR
        • GreedyWordSwapWIR.check_transformation_compatibility()
        • GreedyWordSwapWIR.extra_repr_keys()
        • GreedyWordSwapWIR.perform_search()
        • GreedyWordSwapWIR.is_black_box
    • AlzantotGeneticAlgorithm
      • AlzantotGeneticAlgorithm
    • ImprovedGeneticAlgorithm
      • ImprovedGeneticAlgorithm
        • ImprovedGeneticAlgorithm.extra_repr_keys()
    • ParticleSwarmOptimization
      • ParticleSwarmOptimization
        • ParticleSwarmOptimization.check_transformation_compatibility()
        • ParticleSwarmOptimization.extra_repr_keys()
        • ParticleSwarmOptimization.perform_search()
        • ParticleSwarmOptimization.is_black_box
    • DifferentialEvolution
      • DifferentialEvolution
        • DifferentialEvolution.check_transformation_compatibility()
        • DifferentialEvolution.extra_repr_keys()
        • DifferentialEvolution.perform_search()
        • DifferentialEvolution.is_black_box

Full Reference

  • textattack package
    • Subpackages
      • textattack.attack_recipes package
        • Attack Recipes Package:
        • Submodules
      • textattack.attack_results package
        • TextAttack attack_results Package
        • Submodules
      • textattack.augmentation package
        • TextAttack augmentation package:
        • Submodules
      • textattack.commands package
        • TextAttack commands Package
        • Submodules
      • textattack.constraints package
        • Constraints
        • Subpackages
        • Submodules
      • textattack.datasets package
        • datasets package:
        • Subpackages
        • Submodules
      • textattack.goal_function_results package
        • Goal Function Result package:
        • Subpackages
        • Submodules
      • textattack.goal_functions package
        • Goal Functions
        • Subpackages
        • Submodules
      • textattack.llms package
        • Large Language Models
        • Submodules
      • textattack.loggers package
        • Misc Loggers: Loggers track, visualize, and export attack results.
        • Submodules
      • textattack.metrics package
        • metrics package: to calculate advanced metrics for evaluting attacks and augmented text
        • Subpackages
        • Submodules
      • textattack.models package
        • Models
        • Subpackages
      • textattack.prompt_augmentation package
        • Prompt Augmentation
        • Submodules
      • textattack.search_methods package
        • Search Methods
        • Submodules
      • textattack.shared package
        • Shared TextAttack Functions
        • Subpackages
        • Submodules
      • textattack.transformations package
        • Transformations
        • Subpackages
        • Submodules
    • Submodules
      • Attack Class
      • Attack
        • Attack.attack()
        • Attack.clear_cache()
        • Attack.cpu_()
        • Attack.cuda_()
        • Attack.filter_transformations()
        • Attack.get_indices_to_order()
        • Attack.get_transformations()
      • AttackArgs Class
      • AttackArgs
        • AttackArgs.create_loggers_from_args()
        • AttackArgs.attack_n
        • AttackArgs.checkpoint_dir
        • AttackArgs.checkpoint_interval
        • AttackArgs.csv_coloring_style
        • AttackArgs.disable_stdout
        • AttackArgs.enable_advance_metrics
        • AttackArgs.log_summary_to_json
        • AttackArgs.log_to_csv
        • AttackArgs.log_to_txt
        • AttackArgs.log_to_visdom
        • AttackArgs.log_to_wandb
        • AttackArgs.metrics
        • AttackArgs.num_examples
        • AttackArgs.num_examples_offset
        • AttackArgs.num_successful_examples
        • AttackArgs.num_workers_per_device
        • AttackArgs.parallel
        • AttackArgs.query_budget
        • AttackArgs.random_seed
        • AttackArgs.shuffle
        • AttackArgs.silent
      • CommandLineAttackArgs
      • Attacker Class
      • Attacker
        • Attacker.attack_dataset()
        • Attacker.attack_interactive()
        • Attacker.from_checkpoint()
        • Attacker.update_attack_args()
      • attack_from_queue()
      • pytorch_multiprocessing_workaround()
      • set_env_variables()
      • AugmenterArgs Class
      • AugmenterArgs
        • AugmenterArgs.enable_advanced_metrics
        • AugmenterArgs.exclude_original
        • AugmenterArgs.fast_augment
        • AugmenterArgs.high_yield
        • AugmenterArgs.input_column
        • AugmenterArgs.input_csv
        • AugmenterArgs.interactive
        • AugmenterArgs.output_csv
        • AugmenterArgs.overwrite
        • AugmenterArgs.pct_words_to_swap
        • AugmenterArgs.random_seed
        • AugmenterArgs.recipe
        • AugmenterArgs.transformations_per_example
      • DatasetArgs Class
      • DatasetArgs
        • DatasetArgs.dataset_by_model
        • DatasetArgs.dataset_from_file
        • DatasetArgs.dataset_from_huggingface
        • DatasetArgs.dataset_split
        • DatasetArgs.filter_by_labels
      • ModelArgs Class
      • ModelArgs
        • ModelArgs.model
        • ModelArgs.model_from_file
        • ModelArgs.model_from_huggingface
      • Trainer Class
      • Trainer
        • Trainer.evaluate()
        • Trainer.evaluate_step()
        • Trainer.get_eval_dataloader()
        • Trainer.get_optimizer_and_scheduler()
        • Trainer.get_train_dataloader()
        • Trainer.train()
        • Trainer.training_step()
      • TrainingArgs Class
      • CommandLineTrainingArgs
        • CommandLineTrainingArgs.output_dir
      • TrainingArgs
        • TrainingArgs.alpha
        • TrainingArgs.attack_epoch_interval
        • TrainingArgs.attack_num_workers_per_device
        • TrainingArgs.checkpoint_interval_epochs
        • TrainingArgs.checkpoint_interval_steps
        • TrainingArgs.early_stopping_epochs
        • TrainingArgs.gradient_accumulation_steps
        • TrainingArgs.learning_rate
        • TrainingArgs.load_best_model_at_end
        • TrainingArgs.log_to_tb
        • TrainingArgs.log_to_wandb
        • TrainingArgs.logging_interval_step
        • TrainingArgs.num_clean_epochs
        • TrainingArgs.num_epochs
        • TrainingArgs.num_train_adv_examples
        • TrainingArgs.num_warmup_steps
        • TrainingArgs.output_dir
        • TrainingArgs.parallel
        • TrainingArgs.per_device_eval_batch_size
        • TrainingArgs.per_device_train_batch_size
        • TrainingArgs.query_budget_train
        • TrainingArgs.random_seed
        • TrainingArgs.save_last
        • TrainingArgs.tb_log_dir
        • TrainingArgs.wandb_project
        • TrainingArgs.weight_decay
      • default_output_dir()
TextAttack
  • Search

© Copyright 2021-24, UVA QData Lab.

Built with Sphinx using a theme provided by Read the Docs.