Attacker API Reference

Attacker

While Attack is the main class used to carry out the adversarial attack, it is only useful for attacking one example at a time. It lacks features that support attacking multiple samples in parallel (i.e. multi-GPU), saving checkpoints, or logging results to text file, CSV file, or wandb. Attacker provides these features in an easy-to-use API.

class textattack.Attacker(attack, dataset, attack_args=None)

Class for running attacks on a dataset with specified parameters. This class uses the Attack to actually run the attacks, while also providing useful features such as parallel processing, saving/resuming from a checkpint, logging to files and stdout.

Parameters:

• attack (Attack) – Attack used to actually carry out the attack.
• dataset (Dataset) – Dataset to attack.
• attack_args (AttackArgs) – Arguments for attacking the dataset. For default settings, look at the AttackArgs class.

Example:

>>> import textattack
>>> import transformers

>>> model = transformers.AutoModelForSequenceClassification.from_pretrained("textattack/bert-base-uncased-imdb")
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("textattack/bert-base-uncased-imdb")
>>> model_wrapper = textattack.models.wrappers.HuggingFaceModelWrapper(model, tokenizer)

>>> attack = textattack.attack_recipes.TextFoolerJin2019.build(model_wrapper)
>>> dataset = textattack.datasets.HuggingFaceDataset("imdb", split="test")

>>> # Attack 20 samples with CSV logging and checkpoint saved every 5 interval
>>> attack_args = textattack.AttackArgs(
...     num_examples=20,
...     log_to_csv="log.csv",
...     checkpoint_interval=5,
...     checkpoint_dir="checkpoints",
...     disable_stdout=True
... )

>>> attacker = textattack.Attacker(attack, dataset, attack_args)
>>> attacker.attack_dataset()

attack_dataset()

Attack the dataset.

Returns:list[AttackResult] - List of AttackResult obtained after attacking the given dataset..

classmethod from_checkpoint(attack, dataset, checkpoint)

Resume attacking from a saved checkpoint. Attacker and dataset must be recovered by the user again, while attack args are loaded from the saved checkpoint.

Parameters:

• attack (Attack) – Attack object for carrying out the attack.
• dataset (Dataset) – Dataset to attack.
• checkpoint (Union[str, :class:`~textattack.shared.AttackChecpoint]`) – Path of saved checkpoint or the actual saved checkpoint.

update_attack_args(**kwargs)

To update any attack args, pass the new argument as keyword argument to this function.

Examples:

>>> attacker = #some instance of Attacker
>>> # To switch to parallel mode and increase checkpoint interval from 100 to 500
>>> attacker.update_attack_args(parallel=True, checkpoint_interval=500)

AttackArgs

AttackArgs represents arguments to be passed to Attacker, such as number of examples to attack, interval at which to save checkpoints, logging details.

class textattack.AttackArgs(num_examples: int = 10, num_successful_examples: int = None, num_examples_offset: int = 0, attack_n: bool = False, shuffle: bool = False, query_budget: int = None, checkpoint_interval: int = None, checkpoint_dir: str = 'checkpoints', random_seed: int = 765, parallel: bool = False, num_workers_per_device: int = 1, log_to_txt: str = None, log_to_csv: str = None, log_summary_to_json: str = None, csv_coloring_style: str = 'file', log_to_visdom: dict = None, log_to_wandb: dict = None, disable_stdout: bool = False, silent: bool = False, enable_advance_metrics: bool = False, metrics: Optional[Dict[KT, VT]] = None)

Attack arguments to be passed to Attacker. :param num_examples: The number of examples to attack. -1 for entire dataset. :type num_examples: int, ‘optional`, defaults to 10 :param num_successful_examples: The number of successful adversarial examples we want. This is different from num_examples

as num_examples only cares about attacking N samples while num_successful_examples aims to keep attacking until we have N successful cases. .. note:

If set, this argument overrides `num_examples` argument.

Parameters:

• ( (num_examples_offset) – obj: int, optional, defaults to 0): The offset index to start at in the dataset.
• attack_n (bool, optional, defaults to False) – Whether to run attack until total of N examples have been attacked (and not skipped).
• shuffle (bool, optional, defaults to False) – If True, we randomly shuffle the dataset before attacking. However, this avoids actually shuffling the dataset internally and opts for shuffling the list of indices of examples we want to attack. This means shuffle can now be used with checkpoint saving.
• query_budget (int, optional, defaults to None) –

  The maximum number of model queries allowed per example attacked. If not set, we use the query budget set in the GoalFunction object (which by default is float("inf")). .. note:

  Setting this overwrites the query budget set in :class:`~textattack.goal_functions.GoalFunction` object.
  

• checkpoint_interval (int, optional, defaults to None) – If set, checkpoint will be saved after attacking every N examples. If None is passed, no checkpoints will be saved.
• checkpoint_dir (str, optional, defaults to "checkpoints") – The directory to save checkpoint files.
• random_seed (int, optional, defaults to 765) – Random seed for reproducibility.
• parallel (False, optional, defaults to False) – If True, run attack using multiple CPUs/GPUs.
• num_workers_per_device (int, optional, defaults to 1) – Number of worker processes to run per device in parallel mode (i.e. parallel=True). For example, if you are using GPUs and num_workers_per_device=2, then 2 processes will be running in each GPU.
• log_to_txt (str, optional, defaults to None) – If set, save attack logs as a .txt file to the directory specified by this argument. If the last part of the provided path ends with .txt extension, it is assumed to the desired path of the log file.
• log_to_csv (str, optional, defaults to None) – If set, save attack logs as a CSV file to the directory specified by this argument. If the last part of the provided path ends with .csv extension, it is assumed to the desired path of the log file.
• csv_coloring_style (str, optional, defaults to "file") – Method for choosing how to mark perturbed parts of the text. Options are "file", "plain", and "html". "file" wraps perturbed parts with double brackets [[ <text> ]] while "plain" does not mark the text in any way.
• log_to_visdom (dict, optional, defaults to None) – If set, Visdom logger is used with the provided dictionary passed as a keyword arguments to VisdomLogger. Pass in empty dictionary to use default arguments. For custom logger, the dictionary should have the following three keys and their corresponding values: "env", "port", "hostname".
• log_to_wandb (dict, optional, defaults to None) – If set, WandB logger is used with the provided dictionary passed as a keyword arguments to WeightsAndBiasesLogger. Pass in empty dictionary to use default arguments. For custom logger, the dictionary should have the following key and its corresponding value: "project".
• disable_stdout (bool, optional, defaults to False) – Disable displaying individual attack results to stdout.
• silent (bool, optional, defaults to False) – Disable all logging (except for errors). This is stronger than disable_stdout.
• enable_advance_metrics (bool, optional, defaults to False) – Enable calculation and display of optional advance post-hoc metrics like perplexity, grammar errors, etc.

classmethod create_loggers_from_args(args)

Creates AttackLogManager from an AttackArgs object.