The TextAttack ecosystem: search, transformations, and constraints

Open In Colab

View Source on GitHub

Please remember to run pip3 install textattack[tensorflow] in your notebook enviroment before the following codes:

[ ]:
!pip3 install textattack[tensorflow]

An attack in TextAttack consists of four parts.

Goal function

The goal function determines if the attack is successful or not. One common goal function is untargeted classification, where the attack tries to perturb an input to change its classification.

Search method

The search method explores the space of potential transformations and tries to locate a successful perturbation. Greedy search, beam search, and brute-force search are all examples of search methods.

Transformation

A transformation takes a text input and transforms it, for example replacing words or phrases with similar ones, while trying not to change the meaning. Paraphrase and synonym substitution are two broad classes of transformations.

Constraints

Finally, constraints determine whether or not a given transformation is valid. Transformations don’t perfectly preserve syntax or semantics, so additional constraints can increase the probability that these qualities are preserved from the source to adversarial example. There are many types of constraints: overlap constraints that measure edit distance, syntactical constraints check part-of-speech and grammar errors, and semantic constraints like language models and sentence encoders.

A custom transformation

This lesson explains how to create a custom transformation. In TextAttack, many transformations involve word swaps: they take a word and try and find suitable substitutes. Some attacks focus on replacing characters with neighboring characters to create “typos” (these don’t intend to preserve the grammaticality of inputs). Other attacks rely on semantics: they take a word and try to replace it with semantic equivalents.

Banana word swap

As an introduction to writing transformations for TextAttack, we’re going to try a very simple transformation: one that replaces any given word with the word ‘banana’. In TextAttack, there’s an abstract WordSwap class that handles the heavy lifting of breaking sentences into words and avoiding replacement of stopwords. We can extend WordSwap and implement a single method, _get_replacement_words, to indicate to replace each word with ‘banana’. 🍌

[1]:
from textattack.transformations import WordSwap


class BananaWordSwap(WordSwap):
    """Transforms an input by replacing any word with 'banana'."""

    # We don't need a constructor, since our class doesn't require any parameters.

    def _get_replacement_words(self, word):
        """Returns 'banana', no matter what 'word' was originally.

        Returns a list with one item, since `_get_replacement_words` is intended to
            return a list of candidate replacement words.
        """
        return ["banana"]

Using our transformation

Now we have the transformation chosen, but we’re missing a few other things. To complete the attack, we need to choose the search method and constraints. And to use the attack, we need a goal function, a model and a dataset. (The goal function indicates the task our model performs – in this case, classification – and the type of attack – in this case, we’ll perform an untargeted attack.)

Creating the goal function, model, and dataset

We are performing an untargeted attack on a classification model, so we’ll use the UntargetedClassification class. For the model, let’s use BERT trained for news classification on the AG News dataset. We’ve pretrained several models and uploaded them to the HuggingFace Model Hub. TextAttack integrates with any model from HuggingFace’s Model Hub and any dataset from HuggingFace’s datasets!

[2]:
# Import the model
import transformers
from textattack.models.wrappers import HuggingFaceModelWrapper

model = transformers.AutoModelForSequenceClassification.from_pretrained(
    "textattack/bert-base-uncased-ag-news"
)
tokenizer = transformers.AutoTokenizer.from_pretrained(
    "textattack/bert-base-uncased-ag-news"
)

model_wrapper = HuggingFaceModelWrapper(model, tokenizer)

# Create the goal function using the model
from textattack.goal_functions import UntargetedClassification

goal_function = UntargetedClassification(model_wrapper)

# Import the dataset
from textattack.datasets import HuggingFaceDataset

dataset = HuggingFaceDataset("ag_news", None, "test")
textattack: Unknown if model of class <class 'transformers.models.bert.modeling_bert.BertForSequenceClassification'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.
Using custom data configuration default
Reusing dataset ag_news (/p/qdata/jy2ma/.cache/textattack/datasets/ag_news/default/0.0.0/0eeeaaa5fb6dffd81458e293dfea1adba2881ffcbdc3fb56baeb5a892566c29a)
textattack: Loading datasets dataset ag_news, split test.

Creating the attack

Let’s keep it simple: let’s use a greedy search method, and let’s not use any constraints for now.

[3]:
from textattack.search_methods import GreedySearch
from textattack.constraints.pre_transformation import (
    RepeatModification,
    StopwordModification,
)
from textattack import Attack

# We're going to use our Banana word swap class as the attack transformation.
transformation = BananaWordSwap()
# We'll constrain modification of already modified indices and stopwords
constraints = [RepeatModification(), StopwordModification()]
# We'll use the Greedy search method
search_method = GreedySearch()
# Now, let's make the attack from the 4 components:
attack = Attack(goal_function, constraints, transformation, search_method)

Let’s print our attack to see all the parameters:

[4]:
print(attack)
Attack(
  (search_method): GreedySearch
  (goal_function):  UntargetedClassification
  (transformation):  BananaWordSwap
  (constraints):
    (0): RepeatModification
    (1): StopwordModification
  (is_black_box):  True
)
[5]:
print(dataset[0])
(OrderedDict([('text', "Fears for T N pension after talks Unions representing workers at Turner   Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul.")]), 2)

Using the attack

Let’s use our attack to successfully attack 10 samples.

[6]:
from tqdm import tqdm  # tqdm provides us a nice progress bar.
from textattack.loggers import CSVLogger  # tracks a dataframe for us.
from textattack.attack_results import SuccessfulAttackResult
from textattack import Attacker
from textattack import AttackArgs
from textattack.datasets import Dataset

attack_args = AttackArgs(num_examples=10)

attacker = Attacker(attack, dataset, attack_args)

attack_results = attacker.attack_dataset()

# The following legacy tutorial code shows how the Attack API works in detail.

# logger = CSVLogger(color_method='html')

# num_successes = 0
# i = 0
# while num_successes < 10:
# result = next(results_iterable)
#    example, ground_truth_output = dataset[i]
#    i += 1
#    result = attack.attack(example, ground_truth_output)
#    if isinstance(result, SuccessfulAttackResult):
#        logger.log_attack_result(result)
#        num_successes += 1
#       print(f'{num_successes} of 10 successes complete.')
  0%|          | 0/10 [00:00<?, ?it/s]
Attack(
  (search_method): GreedySearch
  (goal_function):  UntargetedClassification
  (transformation):  BananaWordSwap
  (constraints):
    (0): RepeatModification
    (1): StopwordModification
  (is_black_box):  True
)

[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  10%|█         | 1/10 [00:01<00:14,  1.57s/it]
--------------------------------------------- Result 1 ---------------------------------------------
Business (100%) --> World (89%)

Fears for T N pension after talks Unions representing workers at Turner   Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul.

Fears for T N banana after banana banana representing banana at Turner   Newall say they are 'banana after talks with stricken parent firm Federal banana.


[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  20%|██        | 2/10 [00:13<00:53,  6.68s/it]
--------------------------------------------- Result 2 ---------------------------------------------
Sci/tech (100%) --> World (64%)

The Race is On: Second Private Team Sets Launch Date for Human Spaceflight (SPACE.com) SPACE.com - TORONTO, Canada -- A second\team of rocketeers competing for the  #36;10 million Ansari X Prize, a contest for\privately funded suborbital space flight, has officially announced the first\launch date for its manned rocket.

The Race is On: Second Private banana Sets Launch banana for banana banana (banana.banana) banana.banana - banana, banana -- banana banana\banana of rocketeers banana for the  #36;10 million Ansari X banana, a banana for\banana funded banana banana banana, has officially banana the first\banana date for its banana rocket.


[Succeeded / Failed / Skipped / Total] 3 / 0 / 0 / 3:  30%|███       | 3/10 [00:18<00:42,  6.06s/it]
--------------------------------------------- Result 3 ---------------------------------------------
Sci/tech (100%) --> Business (77%)

Ky. Company Wins Grant to Study Peptides (AP) AP - A company founded by a chemistry researcher at the University of Louisville won a grant to develop a method of producing better peptides, which are short chains of amino acids, the building blocks of proteins.

Ky. Company Wins Grant to banana banana (banana) banana - banana company banana by a banana banana at the banana of Louisville won a grant to develop a method of producing better banana, which are short chains of banana banana, the building blocks of banana.


[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4:  40%|████      | 4/10 [00:20<00:30,  5.11s/it]
--------------------------------------------- Result 4 ---------------------------------------------
Sci/tech (100%) --> World (65%)

Prediction Unit Helps Forecast Wildfires (AP) AP - It's barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.

banana Unit Helps banana Wildfires (AP) banana - It's barely dawn when Mike Fitzpatrick banana his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.


[Succeeded / Failed / Skipped / Total] 5 / 0 / 0 / 5:  50%|█████     | 5/10 [00:22<00:22,  4.42s/it]
--------------------------------------------- Result 5 ---------------------------------------------
Sci/tech (100%) --> World (62%)

Calif. Aims to Limit Farm-Related Smog (AP) AP - Southern California's smog-fighting agency went after emissions of the bovine variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.

Calif. Aims to Limit Farm-Related banana (AP) AP - Southern California's banana agency went after banana of the banana variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.


[Succeeded / Failed / Skipped / Total] 6 / 0 / 0 / 6:  60%|██████    | 6/10 [00:54<00:36,  9.07s/it]
--------------------------------------------- Result 6 ---------------------------------------------
Sci/tech (100%) --> World (53%)

Open Letter Against British Copyright Indoctrination in Schools The British Department for Education and Skills (DfES) recently launched a "Music Manifesto" campaign, with the ostensible intention of educating the next generation of British musicians. Unfortunately, they also teamed up with the music industry (EMI, and various artists) to make this popular. EMI has apparently negotiated their end well, so that children in our schools will now be indoctrinated about the illegality of downloading music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.

Open banana Against banana banana Indoctrination in Schools The banana Department for Education and Skills (DfES) banana banana a "banana banana" campaign, with the ostensible banana of banana the banana banana of banana banana. banana, they also teamed up with the banana industry (banana, and banana banana) to make this popular. banana has banana banana their end well, so that banana in our schools will now be indoctrinated about the illegality of banana music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.


[Succeeded / Failed / Skipped / Total] 6 / 1 / 0 / 7:  70%|███████   | 7/10 [01:47<00:46, 15.36s/it]
--------------------------------------------- Result 7 ---------------------------------------------
Sci/tech (100%) --> [FAILED]

Loosing the War on Terrorism \\"Sven Jaschan, self-confessed author of the Netsky and Sasser viruses, is\responsible for 70 percent of virus infections in 2004, according to a six-month\virus roundup published Wednesday by antivirus company Sophos."\\"The 18-year-old Jaschan was taken into custody in Germany in May by police who\said he had admitted programming both the Netsky and Sasser worms, something\experts at Microsoft confirmed. (A Microsoft antivirus reward program led to the\teenager's arrest.) During the five months preceding Jaschan's capture, there\were at least 25 variants of Netsky and one of the port-scanning network worm\Sasser."\\"Graham Cluley, senior technology consultant at Sophos, said it was staggeri ...\\


[Succeeded / Failed / Skipped / Total] 6 / 2 / 0 / 8:  80%|████████  | 8/10 [02:55<00:43, 21.96s/it]
--------------------------------------------- Result 8 ---------------------------------------------
Sci/tech (100%) --> [FAILED]

FOAFKey: FOAF, PGP, Key Distribution, and Bloom Filters \\FOAF/LOAF  and bloom filters have a lot of interesting properties for social\network and whitelist distribution.\\I think we can go one level higher though and include GPG/OpenPGP key\fingerpring distribution in the FOAF file for simple web-of-trust based key\distribution.\\What if we used FOAF and included the PGP key fingerprint(s) for identities?\This could mean a lot.  You include the PGP key fingerprints within the FOAF\file of your direct friends and then include a bloom filter of the PGP key\fingerprints of your entire whitelist (the source FOAF file would of course need\to be encrypted ).\\Your whitelist would be populated from the social network as your client\discovered new identit ...\\


[Succeeded / Failed / Skipped / Total] 7 / 2 / 0 / 9:  90%|█████████ | 9/10 [02:56<00:19, 19.57s/it]
--------------------------------------------- Result 9 ---------------------------------------------
Sci/tech (98%) --> World (100%)

E-mail scam targets police chief Wiltshire Police warns about "phishing" after its fraud squad chief was targeted.

banana scam targets police chief Wiltshire Police warns about "banana" after its fraud squad chief was targeted.


[Succeeded / Failed / Skipped / Total] 8 / 2 / 0 / 10: 100%|██████████| 10/10 [02:56<00:00, 17.66s/it]
--------------------------------------------- Result 10 ---------------------------------------------
Sci/tech (98%) --> World (77%)

Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated card fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.

Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated banana fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.



+-------------------------------+--------+
| Attack Results                |        |
+-------------------------------+--------+
| Number of successful attacks: | 8      |
| Number of failed attacks:     | 2      |
| Number of skipped attacks:    | 0      |
| Original accuracy:            | 100.0% |
| Accuracy under attack:        | 20.0%  |
| Attack success rate:          | 80.0%  |
| Average perturbed word %:     | 18.71% |
| Average num. words per input: | 63.0   |
| Avg num queries:              | 934.0  |
+-------------------------------+--------+

Visualizing attack results

We are logging AttackResult objects using a CSVLogger. This logger stores all attack results in a dataframe, which we can easily access and display. Since we set color_method to 'html', the attack results will display their differences, in color, in HTML. Using IPython utilities and pandas

[7]:
import pandas as pd

pd.options.display.max_colwidth = (
    480  # increase colum width so we can actually read the examples
)

logger = CSVLogger(color_method="html")

for result in attack_results:
    if isinstance(result, SuccessfulAttackResult):
        logger.log_attack_result(result)

from IPython.core.display import display, HTML

results = pd.DataFrame.from_records(logger.row_list)
display(HTML(results[["original_text", "perturbed_text"]].to_html(escape=False)))
textattack: Logging to CSV at path results.csv
original_text perturbed_text
0 Fears for T N pension after talks Unions representing workers at Turner Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul. Fears for T N banana after banana banana representing banana at Turner Newall say they are 'banana after talks with stricken parent firm Federal banana.
1 The Race is On: Second Private Team Sets Launch Date for Human Spaceflight (SPACE.com) SPACE.com - TORONTO, Canada -- A second\team of rocketeers competing for the #36;10 million Ansari X Prize, a contest for\privately funded suborbital space flight, has officially announced the first\launch date for its manned rocket. The Race is On: Second Private banana Sets Launch banana for banana banana (banana.banana) banana.banana - banana, banana -- banana banana\banana of rocketeers banana for the #36;10 million Ansari X banana, a banana for\banana funded banana banana banana, has officially banana the first\banana date for its banana rocket.
2 Ky. Company Wins Grant to Study Peptides (AP) AP - A company founded by a chemistry researcher at the University of Louisville won a grant to develop a method of producing better peptides, which are short chains of amino acids, the building blocks of proteins. Ky. Company Wins Grant to banana banana (banana) banana - banana company banana by a banana banana at the banana of Louisville won a grant to develop a method of producing better banana, which are short chains of banana banana, the building blocks of banana.
3 Prediction Unit Helps Forecast Wildfires (AP) AP - It's barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar. banana Unit Helps banana Wildfires (AP) banana - It's barely dawn when Mike Fitzpatrick banana his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.
4 Calif. Aims to Limit Farm-Related Smog (AP) AP - Southern California's smog-fighting agency went after emissions of the bovine variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure. Calif. Aims to Limit Farm-Related banana (AP) AP - Southern California's banana agency went after banana of the banana variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.
5 Open Letter Against British Copyright Indoctrination in Schools The British Department for Education and Skills (DfES) recently launched a "Music Manifesto" campaign, with the ostensible intention of educating the next generation of British musicians. Unfortunately, they also teamed up with the music industry (EMI, and various artists) to make this popular. EMI has apparently negotiated their end well, so that children in our schools will now be indoctrinated about the illegality of downloading music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area. Open banana Against banana banana Indoctrination in Schools The banana Department for Education and Skills (DfES) banana banana a "banana banana" campaign, with the ostensible banana of banana the banana banana of banana banana. banana, they also teamed up with the banana industry (banana, and banana banana) to make this popular. banana has banana banana their end well, so that banana in our schools will now be indoctrinated about the illegality of banana music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.
6 Loosing the War on Terrorism \\"Sven Jaschan, self-confessed author of the Netsky and Sasser viruses, is\responsible for 70 percent of virus infections in 2004, according to a six-month\virus roundup published Wednesday by antivirus company Sophos."\\"The 18-year-old Jaschan was taken into custody in Germany in May by police who\said he had admitted programming both the Netsky and Sasser worms, something\experts at Microsoft confirmed. (A Microsoft antivirus reward program led to the\teenager's arrest.) During the five months preceding Jaschan's capture, there\were at least 25 variants of Netsky and one of the port-scanning network worm\Sasser."\\"Graham Cluley, senior technology consultant at Sophos, said it was staggeri ...\\ banana the banana on banana \\"banana banana, banana banana of the banana and banana banana, is\banana for banana banana of banana banana in banana, banana to a banana\banana banana banana banana by banana banana banana."\\"banana banana banana was banana into banana in banana in banana by banana who\banana he had banana banana both the banana and banana banana, banana\banana at banana banana. (banana banana banana banana banana banana to the\banana banana.) banana the banana banana banana banana banana, there\were at banana banana banana of banana and banana of the banana banana banana\banana."\\"banana banana, banana banana banana at banana, banana it was banana ...\\
7 FOAFKey: FOAF, PGP, Key Distribution, and Bloom Filters \\FOAF/LOAF and bloom filters have a lot of interesting properties for social\network and whitelist distribution.\\I think we can go one level higher though and include GPG/OpenPGP key\fingerpring distribution in the FOAF file for simple web-of-trust based key\distribution.\\What if we used FOAF and included the PGP key fingerprint(s) for identities?\This could mean a lot. You include the PGP key fingerprints within the FOAF\file of your direct friends and then include a bloom filter of the PGP key\fingerprints of your entire whitelist (the source FOAF file would of course need\to be encrypted ).\\Your whitelist would be populated from the social network as your client\discovered new identit ...\\ banana: banana, banana, banana banana, and banana banana \\banana/banana and banana banana have a banana of banana banana for banana\banana and banana banana.\\banana banana we can banana banana banana banana banana and banana banana/banana banana\banana banana in the banana banana for banana banana banana banana\banana.\\banana if we banana banana and banana the banana banana banana(s) for banana?\banana banana banana a banana. banana banana the banana banana banana banana the banana\banana of your banana banana and then banana a banana banana of the banana banana\banana of your banana banana (the banana banana banana banana of banana banana\to be banana ).\\banana banana banana be banana from the banana banana as your banana\banana banana banana ...\\
8 E-mail scam targets police chief Wiltshire Police warns about "phishing" after its fraud squad chief was targeted. banana scam targets police chief Wiltshire Police warns about "banana" after its fraud squad chief was targeted.
9 Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated card fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m. Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated banana fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.

Conclusion

We can examine these examples for a good idea of how many words had to be changed to “banana” to change the prediction score from the correct class to another class. The examples without perturbed words were originally misclassified, so they were skipped by the attack. Looks like some examples needed only a couple “banana”s, while others needed up to 17 “banana” substitutions to change the class score. Wow! 🍌

Bonus: Attacking Custom Samples

We can also attack custom data samples, like these ones I just made up!

[8]:
# For AG News, labels are 0: World, 1: Sports, 2: Business, 3: Sci/Tech

custom_dataset = [
    ("Malaria deaths in Africa fall by 5% from last year", 0),
    ("Washington Nationals defeat the Houston Astros to win the World Series", 1),
    ("Exxon Mobil hires a new CEO", 2),
    ("Microsoft invests $1 billion in OpenAI", 3),
]

attack_args = AttackArgs(num_examples=4)

dataset = Dataset(custom_dataset)

attacker = Attacker(attack, dataset, attack_args)

results_iterable = attacker.attack_dataset()

logger = CSVLogger(color_method="html")

for result in results_iterable:
    logger.log_attack_result(result)

from IPython.core.display import display, HTML

display(HTML(logger.df[["original_text", "perturbed_text"]].to_html(escape=False)))
  0%|          | 0/4 [00:00<?, ?it/s]
Attack(
  (search_method): GreedySearch
  (goal_function):  UntargetedClassification
  (transformation):  BananaWordSwap
  (constraints):
    (0): RepeatModification
    (1): StopwordModification
  (is_black_box):  True
)

[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  25%|██▌       | 1/4 [00:00<00:00,  7.13it/s]
--------------------------------------------- Result 1 ---------------------------------------------
0 (96%) --> 3 (80%)

Malaria deaths in Africa fall by 5% from last year

Malaria banana in Africa fall by 5% from last year


[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  50%|█████     | 2/4 [00:00<00:00,  3.79it/s]
--------------------------------------------- Result 2 ---------------------------------------------
1 (98%) --> 3 (87%)

Washington Nationals defeat the Houston Astros to win the World Series

banana banana banana the Houston Astros to win the World Series


[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4: 100%|██████████| 4/4 [00:00<00:00,  4.31it/s]
--------------------------------------------- Result 3 ---------------------------------------------
2 (99%) --> 3 (94%)

Exxon Mobil hires a new CEO

banana banana banana a new banana


--------------------------------------------- Result 4 ---------------------------------------------
3 (93%) --> 2 (100%)

Microsoft invests $1 billion in OpenAI

banana invests $1 billion in OpenAI



+-------------------------------+--------+
| Attack Results                |        |
+-------------------------------+--------+
| Number of successful attacks: | 4      |
| Number of failed attacks:     | 0      |
| Number of skipped attacks:    | 0      |
| Original accuracy:            | 100.0% |
| Accuracy under attack:        | 0.0%   |
| Attack success rate:          | 100.0% |
| Average perturbed word %:     | 30.15% |
| Average num. words per input: | 8.25   |
| Avg num queries:              | 12.75  |
+-------------------------------+--------+

textattack: Logging to CSV at path results.csv
textattack: CSVLogger exiting without calling flush().

original_text perturbed_text
0 Malaria deaths in Africa fall by 5% from last year Malaria banana in Africa fall by 5% from last year
1 Washington Nationals defeat the Houston Astros to win the World Series banana banana banana the Houston Astros to win the World Series
2 Exxon Mobil hires a new CEO banana banana banana a new banana
3 Microsoft invests $1 billion in OpenAI banana invests $1 billion in OpenAI