TextAttack End-to-End

This tutorial provides a broad end-to-end overview of training, evaluating, and attacking a model using TextAttack.

Open In Colab

View Source on GitHub

[ ]:
!pip3 install textattack[tensorflow]
Collecting textattack[tensorflow]
  Downloading textattack-0.3.3-py3-none-any.whl (361 kB)
     |████████████████████████████████| 361 kB 4.1 MB/s
Collecting language-tool-python
  Downloading language_tool_python-2.6.1-py3-none-any.whl (30 kB)
Collecting lemminflect
  Downloading lemminflect-0.2.2-py3-none-any.whl (769 kB)
     |████████████████████████████████| 769 kB 36.0 MB/s
Collecting word2number
  Downloading word2number-1.1.zip (9.7 kB)
Collecting transformers>=3.3.0
  Downloading transformers-4.11.3-py3-none-any.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 37.9 MB/s
Collecting flair
  Downloading flair-0.9-py3-none-any.whl (319 kB)
     |████████████████████████████████| 319 kB 50.4 MB/s
Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (1.4.1)
Collecting terminaltables
  Downloading terminaltables-3.1.0.tar.gz (12 kB)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (8.10.0)
Requirement already satisfied: editdistance in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (0.5.3)
Collecting bert-score>=0.3.5
  Downloading bert_score-0.3.10-py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 7.1 MB/s
Requirement already satisfied: torch!=1.8,>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (1.9.0+cu111)
Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (3.2.5)
Collecting num2words
  Downloading num2words-0.5.10-py3-none-any.whl (101 kB)
     |████████████████████████████████| 101 kB 11.5 MB/s
Collecting datasets
  Downloading datasets-1.13.0-py3-none-any.whl (285 kB)
     |████████████████████████████████| 285 kB 53.3 MB/s
Collecting lru-dict
  Downloading lru-dict-1.1.7.tar.gz (10 kB)
Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (1.1.5)
Collecting tqdm<4.50.0,>=4.27
  Downloading tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB 7.5 MB/s
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (1.7.1)
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (3.3.0)
Requirement already satisfied: numpy>=1.19.2 in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (1.19.5)
Requirement already satisfied: tensorflow>=2 in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (2.6.0)
Requirement already satisfied: tensorflow-hub in /usr/local/lib/python3.7/dist-packages (from textattack[tensorflow]) (0.12.0)
Collecting tensorboardX
  Downloading tensorboardX-2.4-py2.py3-none-any.whl (124 kB)
     |████████████████████████████████| 124 kB 45.0 MB/s
Collecting tensorflow-text>=2
  Downloading tensorflow_text-2.6.0-cp37-cp37m-manylinux1_x86_64.whl (4.4 MB)
     |████████████████████████████████| 4.4 MB 38.7 MB/s
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from bert-score>=0.3.5->textattack[tensorflow]) (3.2.2)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.7/dist-packages (from bert-score>=0.3.5->textattack[tensorflow]) (21.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from bert-score>=0.3.5->textattack[tensorflow]) (2.23.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.9->bert-score>=0.3.5->textattack[tensorflow]) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0.1->textattack[tensorflow]) (2.8.2)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0.1->textattack[tensorflow]) (2018.9)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=1.0.1->textattack[tensorflow]) (1.15.0)
Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (1.12)
Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (0.12.0)
Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (0.2.0)
Requirement already satisfied: h5py~=3.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (3.1.0)
Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (3.7.4.3)
Requirement already satisfied: tensorflow-estimator~=2.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (2.6.0)
Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (1.1.0)
Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (1.12.1)
Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (1.1.2)
Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (1.6.3)
Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (0.4.0)
Requirement already satisfied: tensorboard~=2.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (2.6.0)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (3.17.3)
Requirement already satisfied: clang~=5.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (5.0)
Requirement already satisfied: keras~=2.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (2.6.0)
Requirement already satisfied: grpcio<2.0,>=1.37.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (1.41.0)
Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (0.37.0)
Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2->textattack[tensorflow]) (3.3.0)
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py~=3.1.0->tensorflow>=2->textattack[tensorflow]) (1.5.2)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (1.35.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (3.3.4)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (0.4.6)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (57.4.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (0.6.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (1.8.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (1.0.1)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (4.7.2)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (4.2.4)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (1.3.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (4.8.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (0.4.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score>=0.3.5->textattack[tensorflow]) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score>=0.3.5->textattack[tensorflow]) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score>=0.3.5->textattack[tensorflow]) (2021.5.30)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score>=0.3.5->textattack[tensorflow]) (3.0.4)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (3.1.1)
Collecting sacremoses
  Downloading sacremoses-0.0.46-py3-none-any.whl (895 kB)
     |████████████████████████████████| 895 kB 38.8 MB/s
Collecting pyyaml>=5.1
  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
     |████████████████████████████████| 636 kB 30.6 MB/s
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.3.0->textattack[tensorflow]) (2019.12.20)
Collecting huggingface-hub>=0.0.17
  Downloading huggingface_hub-0.0.19-py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 5.2 MB/s
Collecting tokenizers<0.11,>=0.10.1
  Downloading tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3 MB)
     |████████████████████████████████| 3.3 MB 39.6 MB/s
Collecting xxhash
  Downloading xxhash-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (243 kB)
     |████████████████████████████████| 243 kB 54.2 MB/s
Collecting aiohttp
  Downloading aiohttp-3.7.4.post0-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 49.0 MB/s
Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets->textattack[tensorflow]) (0.70.12.2)
Collecting fsspec[http]>=2021.05.0
  Downloading fsspec-2021.10.0-py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 44.3 MB/s
Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets->textattack[tensorflow]) (0.3.4)
Collecting datasets
  Downloading datasets-1.12.1-py3-none-any.whl (270 kB)
     |████████████████████████████████| 270 kB 52.8 MB/s
  Downloading datasets-1.12.0-py3-none-any.whl (269 kB)
     |████████████████████████████████| 269 kB 51.8 MB/s
  Downloading datasets-1.11.0-py3-none-any.whl (264 kB)
     |████████████████████████████████| 264 kB 50.7 MB/s
Requirement already satisfied: pyarrow!=4.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from datasets->textattack[tensorflow]) (3.0.0)
Collecting sqlitedict>=1.6.0
  Downloading sqlitedict-1.7.0.tar.gz (28 kB)
Collecting more-itertools
  Downloading more_itertools-8.8.0-py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 5.4 MB/s
Collecting conllu>=4.0
  Downloading conllu-4.4.1-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from flair->textattack[tensorflow]) (0.22.2.post1)
Collecting segtok>=1.5.7
  Downloading segtok-1.5.10.tar.gz (25 kB)
Collecting langdetect
  Downloading langdetect-1.0.9.tar.gz (981 kB)
     |████████████████████████████████| 981 kB 30.2 MB/s
Requirement already satisfied: gensim<=3.8.3,>=3.4.0 in /usr/local/lib/python3.7/dist-packages (from flair->textattack[tensorflow]) (3.6.0)
Collecting sentencepiece==0.1.95
  Downloading sentencepiece-0.1.95-cp37-cp37m-manylinux2014_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 39.0 MB/s
Collecting gdown==3.12.2
  Downloading gdown-3.12.2.tar.gz (8.2 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting mpld3==0.3
  Downloading mpld3-0.3.tar.gz (788 kB)
     |████████████████████████████████| 788 kB 33.7 MB/s
Collecting bpemb>=0.3.2
  Downloading bpemb-0.3.3-py3-none-any.whl (19 kB)
Collecting janome
  Downloading Janome-0.4.1-py2.py3-none-any.whl (19.7 MB)
     |████████████████████████████████| 19.7 MB 50 kB/s
Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from flair->textattack[tensorflow]) (0.8.9)
Requirement already satisfied: hyperopt>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from flair->textattack[tensorflow]) (0.1.2)
Collecting ftfy
  Downloading ftfy-6.0.3.tar.gz (64 kB)
     |████████████████████████████████| 64 kB 2.9 MB/s
Collecting deprecated>=1.2.4
  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Collecting konoha<5.0.0,>=4.0.0
  Downloading konoha-4.6.5-py3-none-any.whl (20 kB)
Collecting wikipedia-api
  Downloading Wikipedia-API-0.5.4.tar.gz (18 kB)
Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from flair->textattack[tensorflow]) (4.2.6)
Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.7/dist-packages (from gensim<=3.8.3,>=3.4.0->flair->textattack[tensorflow]) (5.2.1)
Requirement already satisfied: pymongo in /usr/local/lib/python3.7/dist-packages (from hyperopt>=0.1.1->flair->textattack[tensorflow]) (3.12.0)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from hyperopt>=0.1.1->flair->textattack[tensorflow]) (0.16.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from hyperopt>=0.1.1->flair->textattack[tensorflow]) (2.6.3)
Collecting importlib-metadata
  Downloading importlib_metadata-3.10.1-py3-none-any.whl (14 kB)
Collecting requests
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 851 kB/s
Collecting overrides<4.0.0,>=3.0.0
  Downloading overrides-3.1.0.tar.gz (11 kB)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard~=2.6->tensorflow>=2->textattack[tensorflow]) (3.6.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bert-score>=0.3.5->textattack[tensorflow]) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bert-score>=0.3.5->textattack[tensorflow]) (1.3.2)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score>=0.3.5->textattack[tensorflow]) (2.0.6)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->flair->textattack[tensorflow]) (1.0.1)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from ftfy->flair->textattack[tensorflow]) (0.2.5)
Requirement already satisfied: docopt>=0.6.2 in /usr/local/lib/python3.7/dist-packages (from num2words->textattack[tensorflow]) (0.6.2)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=3.3.0->textattack[tensorflow]) (7.1.2)
Building wheels for collected packages: gdown, mpld3, overrides, segtok, sqlitedict, ftfy, langdetect, lru-dict, terminaltables, wikipedia-api, word2number
  Building wheel for gdown (PEP 517) ... done
  Created wheel for gdown: filename=gdown-3.12.2-py3-none-any.whl size=9704 sha256=be76d15e41eb103e46e0ba0d72cc277d90b7699bae860782279f5ee6ed86247b
  Stored in directory: /root/.cache/pip/wheels/ba/e0/7e/726e872a53f7358b4b96a9975b04e98113b005cd8609a63abc
  Building wheel for mpld3 (setup.py) ... done
  Created wheel for mpld3: filename=mpld3-0.3-py3-none-any.whl size=116702 sha256=729d5e596c97fd6b6855617ae575e2f299e4c3883055d82c82cd0e9f6dceebb2
  Stored in directory: /root/.cache/pip/wheels/26/70/6a/1c79e59951a41b4045497da187b2724f5659ca64033cf4548e
  Building wheel for overrides (setup.py) ... done
  Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10186 sha256=7dfa089a9708c2250125f86ed7e62bb443bdbf3d555ba72acf5e94c175dbdde6
  Stored in directory: /root/.cache/pip/wheels/3a/0d/38/01a9bc6e20dcfaf0a6a7b552d03137558ba1c38aea47644682
  Building wheel for segtok (setup.py) ... done
  Created wheel for segtok: filename=segtok-1.5.10-py3-none-any.whl size=25030 sha256=e77700c3b756745ae1e92f298016bf2f0b539bdfd6b5a90ee918e18139587df2
  Stored in directory: /root/.cache/pip/wheels/67/b7/d0/a121106e61339eee5ed083bc230b1c8dc422c49a5a28c2addd
  Building wheel for sqlitedict (setup.py) ... done
  Created wheel for sqlitedict: filename=sqlitedict-1.7.0-py3-none-any.whl size=14392 sha256=259e857c9f3cfc72c0c8b50d931b88b79d4f5382b8a70275d203a89427ffbb14
  Stored in directory: /root/.cache/pip/wheels/af/94/06/18c0e83e9e227da8f3582810b51f319bbfd181e508676a56c8
  Building wheel for ftfy (setup.py) ... done
  Created wheel for ftfy: filename=ftfy-6.0.3-py3-none-any.whl size=41933 sha256=c66614666a8be4942d16fab87ee11128fc03fa8424fc0a12c0f3600801fa687e
  Stored in directory: /root/.cache/pip/wheels/19/f5/38/273eb3b5e76dfd850619312f693716ac4518b498f5ffb6f56d
  Building wheel for langdetect (setup.py) ... done
  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993242 sha256=553fa791a31538b8828322bde06203789fc30bbbdf4bb0c80a7d7003632aa0ae
  Stored in directory: /root/.cache/pip/wheels/c5/96/8a/f90c59ed25d75e50a8c10a1b1c2d4c402e4dacfa87f3aff36a
  Building wheel for lru-dict (setup.py) ... done
  Created wheel for lru-dict: filename=lru_dict-1.1.7-cp37-cp37m-linux_x86_64.whl size=28412 sha256=aef5a191c8f01db62a275cc05e3befaee153aaaabd41f41aa2d724b4678b60ec
  Stored in directory: /root/.cache/pip/wheels/9d/0b/4e/aa8fec9833090cd52bcd76f92f9d95e1ee7b915c12093663b4
  Building wheel for terminaltables (setup.py) ... done
  Created wheel for terminaltables: filename=terminaltables-3.1.0-py3-none-any.whl size=15354 sha256=03639ca3ebb53caeaa19ad643c2149cb1884638f368b72fd18e53b90a7d52d1b
  Stored in directory: /root/.cache/pip/wheels/ba/ad/c8/2d98360791161cd3db6daf6b5e730f34021fc9367d5879f497
  Building wheel for wikipedia-api (setup.py) ... done
  Created wheel for wikipedia-api: filename=Wikipedia_API-0.5.4-py3-none-any.whl size=13475 sha256=a8fb9efb0f94278251a1983fbd3d07e4fff610ef8ea1205bb2433a4866b79b15
  Stored in directory: /root/.cache/pip/wheels/d3/24/56/58ba93cf78be162451144e7a9889603f437976ef1ae7013d04
  Building wheel for word2number (setup.py) ... done
  Created wheel for word2number: filename=word2number-1.1-py3-none-any.whl size=5580 sha256=cee3a246b5d687cb221e19e789bdcc409ea1f97d9c59d66a6a205c0f3c62457d
  Stored in directory: /root/.cache/pip/wheels/4b/c3/77/a5f48aeb0d3efb7cd5ad61cbd3da30bbf9ffc9662b07c9f879
Successfully built gdown mpld3 overrides segtok sqlitedict ftfy langdetect lru-dict terminaltables wikipedia-api word2number
Installing collected packages: requests, tqdm, pyyaml, importlib-metadata, tokenizers, sentencepiece, sacremoses, overrides, huggingface-hub, xxhash, wikipedia-api, transformers, sqlitedict, segtok, mpld3, more-itertools, langdetect, konoha, janome, gdown, ftfy, fsspec, deprecated, conllu, bpemb, word2number, terminaltables, num2words, lru-dict, lemminflect, language-tool-python, flair, datasets, bert-score, textattack, tensorflow-text, tensorboardX
  Attempting uninstall: requests
    Found existing installation: requests 2.23.0
    Uninstalling requests-2.23.0:
      Successfully uninstalled requests-2.23.0
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.62.3
    Uninstalling tqdm-4.62.3:
      Successfully uninstalled tqdm-4.62.3
  Attempting uninstall: pyyaml
    Found existing installation: PyYAML 3.13
    Uninstalling PyYAML-3.13:
      Successfully uninstalled PyYAML-3.13
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 4.8.1
    Uninstalling importlib-metadata-4.8.1:
      Successfully uninstalled importlib-metadata-4.8.1
  Attempting uninstall: more-itertools
    Found existing installation: more-itertools 8.10.0
    Uninstalling more-itertools-8.10.0:
      Successfully uninstalled more-itertools-8.10.0
  Attempting uninstall: gdown
    Found existing installation: gdown 3.6.4
    Uninstalling gdown-3.6.4:
      Successfully uninstalled gdown-3.6.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.26.0 which is incompatible.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
Successfully installed bert-score-0.3.10 bpemb-0.3.3 conllu-4.4.1 datasets-1.11.0 deprecated-1.2.13 flair-0.9 fsspec-2021.10.0 ftfy-6.0.3 gdown-3.12.2 huggingface-hub-0.0.19 importlib-metadata-3.10.1 janome-0.4.1 konoha-4.6.5 langdetect-1.0.9 language-tool-python-2.6.1 lemminflect-0.2.2 lru-dict-1.1.7 more-itertools-8.8.0 mpld3-0.3 num2words-0.5.10 overrides-3.1.0 pyyaml-5.4.1 requests-2.26.0 sacremoses-0.0.46 segtok-1.5.10 sentencepiece-0.1.95 sqlitedict-1.7.0 tensorboardX-2.4 tensorflow-text-2.6.0 terminaltables-3.1.0 textattack-0.3.3 tokenizers-0.10.3 tqdm-4.49.0 transformers-4.11.3 wikipedia-api-0.5.4 word2number-1.1 xxhash-2.0.2

Training

First, we’re going to train a model. TextAttack integrates directly with transformers and datasets to train any of the transformers pre-trained models on datasets from datasets.

Let’s use the Rotten Tomatoes Movie Review dataset: it’s relatively short , and showcasesthe key features of textattack train. Let’s take a look at the dataset using textattack peek-dataset:

[ ]:
!textattack peek-dataset --dataset-from-huggingface rotten_tomatoes
textattack: Updating TextAttack package dependencies.
textattack: Downloading NLTK required packages.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Downloading package omw to /root/nltk_data...
[nltk_data]   Unzipping corpora/omw.zip.
[nltk_data] Downloading package universal_tagset to /root/nltk_data...
[nltk_data]   Unzipping taggers/universal_tagset.zip.
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Unzipping corpora/wordnet.zip.
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
textattack: Downloading https://textattack.s3.amazonaws.com/word_embeddings/paragramcf.
100% 481M/481M [00:39<00:00, 12.3MB/s]
textattack: Unzipping file /root/.cache/textattack/tmpc6pdkqvf.zip to /root/.cache/textattack/word_embeddings/paragramcf.
textattack: Successfully saved word_embeddings/paragramcf to cache.
Downloading: 5.11kB [00:00, 4.99MB/s]
Downloading: 2.02kB [00:00, 2.04MB/s]
Using custom data configuration default
Downloading and preparing dataset rotten_tomatoes_movie_review/default (download: 476.34 KiB, generated: 1.28 MiB, post-processed: Unknown size, total: 1.75 MiB) to /root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5...
Downloading: 100% 488k/488k [00:00<00:00, 28.3MB/s]
Dataset rotten_tomatoes_movie_review downloaded and prepared to /root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5. Subsequent calls will reuse this data.
textattack: Loading datasets dataset rotten_tomatoes, split train.
textattack: Number of samples: 8530
textattack: Number of words per input:
textattack:         total:   157755
textattack:         mean:    18.49
textattack:         std:     8.58
textattack:         min:     1
textattack:         max:     51
textattack: Dataset lowercased: True
textattack: First sample:
the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .

textattack: Last sample:
things really get weird , though not particularly scary : the movie is all portent and no content .

textattack: Found 2 distinct outputs.
textattack: Most common outputs:
         1      (4265)
         0      (4265)

The dataset looks good! It’s lowercased already, so we’ll make sure our model is uncased. The longest input is 51 words, so we can cap our maximum sequence length (--model-max-length) at 64.

We’ll train `distilbert-base-uncased <https://huggingface.co/transformers/model_doc/distilbert.html>`__, since it’s a relatively small model, and a good example of how we integrate with transformers.

So we have our command:

textattack train                      \ # Train a model with TextAttack
    --model distilbert-base-uncased   \ # Using distilbert, uncased version, from `transformers`
    --dataset rotten_tomatoes         \ # On the Rotten Tomatoes dataset
    --model-num-labels 3              \ # That has 2 labels
    --model-max-length 64             \ # With a maximum sequence length of 64
    --per-device-train-batch-size 128 \ # And batch size of 128
    --num-epochs 3                    \ # For 3 epochs

Now let’s run it (please remember to use GPU if you have access):

[ ]:
!textattack train --model-name-or-path distilbert-base-uncased --dataset rotten_tomatoes --model-num-labels 2 --model-max-length 64 --per-device-train-batch-size 128 --num-epochs 3
textattack: Loading transformers AutoModelForSequenceClassification: distilbert-base-uncased
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.bias', 'vocab_projector.weight', 'vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias']
- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'pre_classifier.bias', 'classifier.bias', 'pre_classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using custom data configuration default
Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5)
textattack: Loading datasets dataset rotten_tomatoes, split train.
Using custom data configuration default
Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5)
Using custom data configuration default
Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5)
Using custom data configuration default
Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5)
textattack: Loading datasets dataset rotten_tomatoes, split validation.
textattack: Writing logs to ./outputs/2021-10-13-17-37-27-247436/train_log.txt.
textattack: Wrote original training args to ./outputs/2021-10-13-17-37-27-247436/training_args.json.
textattack: ***** Running training *****
textattack:   Num examples = 8530
textattack:   Num epochs = 3
textattack:   Num clean epochs = 3
textattack:   Instantaneous batch size per device = 128
textattack:   Total train batch size (w. parallel, distributed & accumulation) = 128
textattack:   Gradient accumulation steps = 1
textattack:   Total optimization steps = 201
textattack: ==========================================================
textattack: Epoch 1
textattack: Running clean epoch 1/3
Loss 0.68924: 100% 67/67 [01:16<00:00,  1.14s/it]
textattack: Train accuracy: 52.86%
textattack: Eval accuracy: 70.83%
textattack: Best score found. Saved model to ./outputs/2021-10-13-17-37-27-247436/best_model/
textattack: ==========================================================
textattack: Epoch 2
textattack: Running clean epoch 2/3
Loss 0.59931: 100% 67/67 [01:16<00:00,  1.13s/it]
textattack: Train accuracy: 77.07%
textattack: Eval accuracy: 82.83%
textattack: Best score found. Saved model to ./outputs/2021-10-13-17-37-27-247436/best_model/
textattack: ==========================================================
textattack: Epoch 3
textattack: Running clean epoch 3/3
Loss 0.51365: 100% 67/67 [01:16<00:00,  1.14s/it]
textattack: Train accuracy: 85.67%
textattack: Eval accuracy: 84.80%
textattack: Best score found. Saved model to ./outputs/2021-10-13-17-37-27-247436/best_model/
textattack: Wrote README to ./outputs/2021-10-13-17-37-27-247436/README.md.

Evaluation

We successfully fine-tuned distilbert-base-cased for 3 epochs. Now let’s evaluate it using textattack eval. This is as simple as providing the path to the pretrained model (that you just obtain from running the above command!) to --model, along with the number of evaluation samples. textattack eval will automatically load the evaluation data from training:

[ ]:
!textattack eval --num-examples 1000 --model ./outputs/2021-10-13-17-37-27-247436/best_model/ --dataset-from-huggingface rotten_tomatoes --dataset-split test
Using custom data configuration default
Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5)
textattack: Loading datasets dataset rotten_tomatoes, split test.
textattack: Got 1000 predictions.
textattack: Correct 847/1000 (84.70%)

Awesome – we were able to train a model up to 84.9% accuracy on the test dataset – with only a single command!

Attack

Finally, let’s attack our pre-trained model. We can do this the same way as before (by providing the path to the pretrained model to --model). For our attack, let’s use the “TextFooler” attack recipe, from the paper “Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment” (Jin et al, 2019). We can do this by passing --recipe textfooler to textattack attack.

Warning: We’re printing out 100 examples and, if the attack succeeds, their perturbations. The output of this command is going to be quite long!

[ ]:
!textattack attack --recipe textfooler --num-examples 100 --model ./outputs/2021-10-13-17-37-27-247436/best_model/ --dataset-from-huggingface rotten_tomatoes --dataset-split test
Using custom data configuration default
Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/e06abb624abab47e1a64608fdfe65a913f5a68c66118408032644a3285208fb5)
textattack: Loading datasets dataset rotten_tomatoes, split test.
textattack: Unknown if model of class <class 'transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.
Attack(
  (search_method): GreedyWordSwapWIR(
    (wir_method):  delete
  )
  (goal_function):  UntargetedClassification
  (transformation):  WordSwapEmbedding(
    (max_candidates):  50
    (embedding):  WordEmbedding
  )
  (constraints):
    (0): WordEmbeddingDistance(
        (embedding):  WordEmbedding
        (min_cos_sim):  0.5
        (cased):  False
        (include_unknown_words):  True
        (compare_against_original):  True
      )
    (1): PartOfSpeech(
        (tagger_type):  nltk
        (tagset):  universal
        (allow_verb_noun_swap):  True
        (compare_against_original):  True
      )
    (2): UniversalSentenceEncoder(
        (metric):  angular
        (threshold):  0.840845057
        (window_size):  15
        (skip_text_shorter_than_window):  True
        (compare_against_original):  False
      )
    (3): RepeatModification
    (4): StopwordModification
    (5): InputColumnModification(
        (matching_column_labels):  ['premise', 'hypothesis']
        (columns_to_ignore):  {'premise'}
      )
  (is_black_box):  True
)

  0% 0/100 [00:00<?, ?it/s]Using /tmp/tfhub_modules to cache modules.
Downloading TF-Hub Module 'https://tfhub.dev/google/universal-sentence-encoder/4'.
Downloaded https://tfhub.dev/google/universal-sentence-encoder/4, Total size: 987.47MB
Downloaded TF-Hub Module 'https://tfhub.dev/google/universal-sentence-encoder/4'.
2021-10-13 18:00:34.997360: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:34.998552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:34.999292: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.002729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.003549: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.004300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.012931: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.013723: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.014528: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-13 18:00:35.015277: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-10-13 18:00:35.015394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 10020 MB memory:  -> device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7
2021-10-13 18:00:36.592844: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 34133760 exceeds 10% of free system memory.
2021-10-13 18:00:36.609893: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 34133760 exceeds 10% of free system memory.
2021-10-13 18:00:36.627953: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 34133760 exceeds 10% of free system memory.
2021-10-13 18:00:36.688755: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 34133760 exceeds 10% of free system memory.
2021-10-13 18:00:36.717187: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 34133760 exceeds 10% of free system memory.
2021-10-13 18:00:38.127267: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
  1% 1/100 [00:22<36:37, 22.19s/it]--------------------------------------------- Result 1 ---------------------------------------------
Positive (95%) --> Negative (61%)

lovingly photographed in the manner of a golden book sprung to life , stuart little 2 manages sweetness largely without stickiness .

lovingly photographed in the manner of a golden book sprung to life , stuart little 2 administration honeyed largely without stickiness .


[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:   2% 2/100 [00:22<18:20, 11.23s/it]--------------------------------------------- Result 2 ---------------------------------------------
Positive (98%) --> Negative (52%)

consistently clever and suspenseful .

progressively clever and enigmatic .


[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:   2% 2/100 [00:22<18:20, 11.23s/it]--------------------------------------------- Result 3 ---------------------------------------------
Negative (85%) --> [SKIPPED]

it's like a " big chill " reunion of the baader-meinhof gang , only these guys are more harmless pranksters than political activists .


[Succeeded / Failed / Skipped / Total] 2 / 0 / 1 / 3:   4% 4/100 [00:22<09:10,  5.73s/it]--------------------------------------------- Result 4 ---------------------------------------------
Positive (98%) --> Negative (59%)

the story gives ample opportunity for large-scale action and suspense , which director shekhar kapur supplies with tremendous skill .

the story gives ample opportunity for large-scale action and suspense , which director shekhar kapur supplies with stupendous skilful .


[Succeeded / Failed / Skipped / Total] 3 / 0 / 1 / 4:   4% 4/100 [00:22<09:10,  5.73s/it]--------------------------------------------- Result 5 ---------------------------------------------
Negative (76%) --> [SKIPPED]

red dragon " never cuts corners .


[Succeeded / Failed / Skipped / Total] 3 / 0 / 2 / 5:   6% 6/100 [00:23<06:02,  3.86s/it]--------------------------------------------- Result 6 ---------------------------------------------
Positive (73%) --> Negative (61%)

fresnadillo has something serious to say about the ways in which extravagant chance can distort our perspective and throw us off the path of good sense .

fresnadillo has something serious to say about the modo in which extravagant chance can distort our perspective and throw us off the path of good sense .


[Succeeded / Failed / Skipped / Total] 4 / 0 / 2 / 6:   6% 6/100 [00:23<06:02,  3.86s/it]--------------------------------------------- Result 7 ---------------------------------------------
Positive (97%) --> Negative (70%)

throws in enough clever and unexpected twists to make the formula feel fresh .

throws in enough clever and unwanted tendrils to make the formula feel fresh .


[Succeeded / Failed / Skipped / Total] 5 / 0 / 2 / 7:   8% 8/100 [00:23<04:29,  2.93s/it]--------------------------------------------- Result 8 ---------------------------------------------
Negative (81%) --> [SKIPPED]

weighty and ponderous but every bit as filling as the treat of the title .


[Succeeded / Failed / Skipped / Total] 5 / 0 / 3 / 8:   8% 8/100 [00:23<04:29,  2.93s/it]--------------------------------------------- Result 9 ---------------------------------------------
Positive (97%) --> Negative (72%)

a real audience-pleaser that will strike a chord with anyone who's ever waited in a doctor's office , emergency room , hospital bed or insurance company office .

a actual audience-pleaser that will slugged a chords with anyone who's ever waited in a doctor's office , emergency room , hospital bed or insurance company office .


[Succeeded / Failed / Skipped / Total] 6 / 0 / 3 / 9:  10% 10/100 [00:24<03:37,  2.41s/it]--------------------------------------------- Result 10 ---------------------------------------------
Positive (98%) --> Negative (91%)

generates an enormous feeling of empathy for its characters .

generates an dreaded feeling of empathy for its characters .


[Succeeded / Failed / Skipped / Total] 7 / 0 / 3 / 10:  10% 10/100 [00:24<03:37,  2.41s/it]--------------------------------------------- Result 11 ---------------------------------------------
Positive (98%) --> Negative (53%)

exposing the ways we fool ourselves is one hour photo's real strength .

exposing the ways we fool ourselves is one hour photo's real strenght .


[Succeeded / Failed / Skipped / Total] 8 / 0 / 3 / 11:  12% 12/100 [00:24<02:58,  2.03s/it]--------------------------------------------- Result 12 ---------------------------------------------
Negative (59%) --> [SKIPPED]

it's up to you to decide whether to admire these people's dedication to their cause or be repelled by their dogmatism , manipulativeness and narrow , fearful view of american life .


[Succeeded / Failed / Skipped / Total] 8 / 0 / 4 / 12:  12% 12/100 [00:24<02:58,  2.03s/it]--------------------------------------------- Result 13 ---------------------------------------------
Negative (80%) --> [SKIPPED]

mostly , [goldbacher] just lets her complicated characters be unruly , confusing and , through it all , human .


[Succeeded / Failed / Skipped / Total] 8 / 0 / 5 / 13:  14% 14/100 [00:24<02:32,  1.77s/it]--------------------------------------------- Result 14 ---------------------------------------------
Positive (88%) --> Negative (96%)

. . . quite good at providing some good old fashioned spooks .

. . . too good at providing some guten old fashioned spooks .


[Succeeded / Failed / Skipped / Total] 9 / 0 / 5 / 14:  14% 14/100 [00:24<02:32,  1.77s/it]--------------------------------------------- Result 15 ---------------------------------------------
Negative (91%) --> [SKIPPED]

at its worst , the movie is pretty diverting ; the pity is that it rarely achieves its best .


[Succeeded / Failed / Skipped / Total] 9 / 0 / 6 / 15:  16% 16/100 [00:25<02:14,  1.60s/it]--------------------------------------------- Result 16 ---------------------------------------------
Positive (98%) --> Negative (62%)

scherfig's light-hearted profile of emotional desperation is achingly honest and delightfully cheeky .

scherfig's light-hearted characterize of psychiatric discouragement is achingly cordial and blithely cheeky .


[Succeeded / Failed / Skipped / Total] 10 / 0 / 6 / 16:  16% 16/100 [00:25<02:14,  1.60s/it]--------------------------------------------- Result 17 ---------------------------------------------
Positive (98%) --> Negative (74%)

a journey spanning nearly three decades of bittersweet camaraderie and history , in which we feel that we truly know what makes holly and marina tick , and our hearts go out to them as both continue to negotiate their imperfect , love-hate relationship .

a trekking expectancy nearly three decades of bittersweet camaraderie and history , in which we feel that we truly know what makes holly and marina tick , and our coeur go out to them as both continue to negotiate their inadequate , love-hate relationship .


[Succeeded / Failed / Skipped / Total] 11 / 0 / 6 / 17:  18% 18/100 [00:26<02:02,  1.50s/it]--------------------------------------------- Result 18 ---------------------------------------------
Positive (97%) --> Negative (51%)

the wonderfully lush morvern callar is pure punk existentialism , and ms . ramsay and her co-writer , liana dognini , have dramatized the alan warner novel , which itself felt like an answer to irvine welsh's book trainspotting .

the appallingly impeccably morvern callar is pure punk existentialism , and ms . ramsay and her co-writer , liana dognini , have dramatized the alan warner novel , which itself felt like an answer to irvine welsh's book trainspotting .


[Succeeded / Failed / Skipped / Total] 12 / 0 / 6 / 18:  18% 18/100 [00:26<02:02,  1.50s/it]--------------------------------------------- Result 19 ---------------------------------------------
Positive (64%) --> Negative (68%)

as it turns out , you can go home again .

as it pivot out , you can go homepage again .


[Succeeded / Failed / Skipped / Total] 13 / 0 / 6 / 19:  20% 20/100 [00:27<01:49,  1.37s/it]--------------------------------------------- Result 20 ---------------------------------------------
Positive (95%) --> Negative (78%)

you've already seen city by the sea under a variety of titles , but it's worth yet another visit .

you've already seen city by the sea under a variety of titles , but it's chastisement yet another visit .


[Succeeded / Failed / Skipped / Total] 14 / 0 / 6 / 20:  20% 20/100 [00:27<01:49,  1.37s/it]--------------------------------------------- Result 21 ---------------------------------------------
Positive (97%) --> Negative (59%)

this kind of hands-on storytelling is ultimately what makes shanghai ghetto move beyond a good , dry , reliable textbook and what allows it to rank with its worthy predecessors .

this kind of hands-on myth is ultimately what do shanghai ghetto move beyond a opportune , dry , reliable textbook and what allows it to rank with its reputable predecessors .


[Succeeded / Failed / Skipped / Total] 15 / 0 / 6 / 21:  22% 22/100 [00:28<01:42,  1.31s/it]--------------------------------------------- Result 22 ---------------------------------------------
Positive (88%) --> Negative (56%)

making such a tragedy the backdrop to a love story risks trivializing it , though chouraqui no doubt intended the film to affirm love's power to help people endure almost unimaginable horror .

making such a tragedy the backdrop to a love story risks trivializing it , though chouraqui no suspecting intended the movies to affirm love's power to pomoc people endure almost unimaginable horror .


[Succeeded / Failed / Skipped / Total] 16 / 0 / 6 / 22:  22% 22/100 [00:28<01:42,  1.31s/it]--------------------------------------------- Result 23 ---------------------------------------------
Negative (54%) --> [SKIPPED]

grown-up quibbles are beside the point here . the little girls understand , and mccracken knows that's all that matters .


[Succeeded / Failed / Skipped / Total] 16 / 0 / 7 / 23:  24% 24/100 [00:29<01:33,  1.22s/it]--------------------------------------------- Result 24 ---------------------------------------------
Positive (98%) --> Negative (80%)

a powerful , chilling , and affecting study of one man's dying fall .

a conclusive , macabre , and affecting scrutinized of one man's dying fall .


[Succeeded / Failed / Skipped / Total] 17 / 0 / 7 / 24:  24% 24/100 [00:29<01:33,  1.22s/it]--------------------------------------------- Result 25 ---------------------------------------------
Positive (52%) --> Negative (65%)

this is a fascinating film because there is no clear-cut hero and no all-out villain .

this is a interesting film because there is no clear-cut hero and no all-out villain .


[Succeeded / Failed / Skipped / Total] 18 / 0 / 7 / 25:  26% 26/100 [00:29<01:24,  1.15s/it]--------------------------------------------- Result 26 ---------------------------------------------
Positive (86%) --> Negative (89%)

a dreadful day in irish history is given passionate , if somewhat flawed , treatment .

a dreadful day in irish history is given voracious , if somewhat flawed , treatment .


[Succeeded / Failed / Skipped / Total] 19 / 0 / 7 / 26:  26% 26/100 [00:29<01:24,  1.15s/it]--------------------------------------------- Result 27 ---------------------------------------------
Positive (94%) --> Negative (89%)

. . . a good film that must have baffled the folks in the marketing department .

. . . a advisable film that must have baffled the folks in the marketing department .


[Succeeded / Failed / Skipped / Total] 20 / 0 / 7 / 27:  28% 28/100 [00:30<01:18,  1.10s/it]--------------------------------------------- Result 28 ---------------------------------------------
Positive (98%) --> Negative (64%)

. . . is funny in the way that makes you ache with sadness ( the way chekhov is funny ) , profound without ever being self-important , warm without ever succumbing to sentimentality .

. . . is outlandish in the itineraries that makes you ache with sadness ( the way chekhov is funny ) , shum without ever being self-important , warmest without ever fending to sentimentality .


[Succeeded / Failed / Skipped / Total] 21 / 0 / 7 / 28:  28% 28/100 [00:30<01:18,  1.10s/it]--------------------------------------------- Result 29 ---------------------------------------------
Negative (94%) --> [SKIPPED]

devotees of star trek ii : the wrath of khan will feel a nagging sense of deja vu , and the grandeur of the best next generation episodes is lacking .


[Succeeded / Failed / Skipped / Total] 21 / 0 / 8 / 29:  30% 30/100 [00:32<01:15,  1.07s/it]--------------------------------------------- Result 30 ---------------------------------------------
Positive (98%) --> [FAILED]

a soul-stirring documentary about the israeli/palestinian conflict as revealed through the eyes of some children who remain curious about each other against all odds .


[Succeeded / Failed / Skipped / Total] 21 / 1 / 8 / 30:  30% 30/100 [00:32<01:15,  1.07s/it]--------------------------------------------- Result 31 ---------------------------------------------
Positive (64%) --> Negative (87%)

what's so striking about jolie's performance is that she never lets her character become a caricature -- not even with that radioactive hair .

what's so staggering about jolie's performance is that she never lets her character become a caricature -- not even with that radioactive hair .


[Succeeded / Failed / Skipped / Total] 22 / 1 / 8 / 31:  32% 32/100 [00:32<01:08,  1.01s/it]--------------------------------------------- Result 32 ---------------------------------------------
Negative (67%) --> [SKIPPED]

the main story . . . is compelling enough , but it's difficult to shrug off the annoyance of that chatty fish .


[Succeeded / Failed / Skipped / Total] 22 / 1 / 9 / 32:  32% 32/100 [00:32<01:08,  1.01s/it]--------------------------------------------- Result 33 ---------------------------------------------
Positive (96%) --> Negative (90%)

the performances are immaculate , with roussillon providing comic relief .

the performances are faultless , with roussillon providing comic relief .


[Succeeded / Failed / Skipped / Total] 23 / 1 / 9 / 33:  34% 34/100 [00:33<01:04,  1.03it/s]--------------------------------------------- Result 34 ---------------------------------------------
Positive (97%) --> Negative (55%)

kinnear . . . gives his best screen performance with an oddly winning portrayal of one of life's ultimate losers .

kinnear . . . stipulates his best screen performance with an oddly wons sketch of one of life's ultimate losers .


[Succeeded / Failed / Skipped / Total] 24 / 1 / 9 / 34:  34% 34/100 [00:33<01:04,  1.03it/s]--------------------------------------------- Result 35 ---------------------------------------------
Negative (59%) --> [SKIPPED]

hugh grant , who has a good line in charm , has never been more charming than in about a boy .


[Succeeded / Failed / Skipped / Total] 24 / 1 / 10 / 35:  36% 36/100 [00:33<00:59,  1.08it/s]--------------------------------------------- Result 36 ---------------------------------------------
Positive (89%) --> Negative (60%)

there's a lot of tooth in roger dodger . but what's nice is that there's a casual intelligence that permeates the script .

there's a lot of tooth in roger dodger . but what's gentil is that there's a casual intelligence that permeates the script .


[Succeeded / Failed / Skipped / Total] 25 / 1 / 10 / 36:  36% 36/100 [00:33<00:59,  1.08it/s]--------------------------------------------- Result 37 ---------------------------------------------
Negative (78%) --> [SKIPPED]

reminiscent of alfred hitchcock's thrillers , most of the scary parts in 'signs' occur while waiting for things to happen .


[Succeeded / Failed / Skipped / Total] 25 / 1 / 11 / 37:  38% 38/100 [00:33<00:55,  1.13it/s]--------------------------------------------- Result 38 ---------------------------------------------
Positive (98%) --> Negative (58%)

one of the best looking and stylish animated movies in quite a while . . .

one of the strictest looking and trendy animated movies in quite a while . . .


[Succeeded / Failed / Skipped / Total] 26 / 1 / 11 / 38:  38% 38/100 [00:33<00:55,  1.13it/s]--------------------------------------------- Result 39 ---------------------------------------------
Positive (98%) --> Negative (70%)

its use of the thriller form to examine the labyrinthine ways in which people's lives cross and change , buffeted by events seemingly out of their control , is intriguing , provocative stuff .

its use of the thriller form to examine the labyrinthine ways in which people's lives cross and change , buffeted by events seemingly out of their control , is disconcerting , incite stuff .


[Succeeded / Failed / Skipped / Total] 27 / 1 / 11 / 39:  40% 40/100 [00:34<00:51,  1.16it/s]--------------------------------------------- Result 40 ---------------------------------------------
Positive (91%) --> Negative (54%)

denver should not get the first and last look at one of the most triumphant performances of vanessa redgrave's career . it deserves to be seen everywhere .

denver woud not get the first and last look at one of the most triumphant performances of vanessa redgrave's career . it deserves to be seen everywhere .


[Succeeded / Failed / Skipped / Total] 28 / 1 / 11 / 40:  40% 40/100 [00:34<00:51,  1.16it/s]--------------------------------------------- Result 41 ---------------------------------------------
Positive (60%) --> Negative (64%)

you needn't be steeped in '50s sociology , pop culture or movie lore to appreciate the emotional depth of haynes' work . though haynes' style apes films from the period . . . its message is not rooted in that decade .

you needn't be steeped in '50s sociology , pop culture or movie lore to appreciate the emotional depth of haynes' work . albeit haynes' style apes films from the period . . . its message is not rooted in that decade .


[Succeeded / Failed / Skipped / Total] 29 / 1 / 11 / 41:  42% 42/100 [00:34<00:48,  1.21it/s]--------------------------------------------- Result 42 ---------------------------------------------
Positive (93%) --> Negative (59%)

waiting for godard can be fruitful : 'in praise of love' is the director's epitaph for himself .

waiting for godard can be propitious : 'in praise of love' is the director's epitaph for himself .


[Succeeded / Failed / Skipped / Total] 30 / 1 / 11 / 42:  42% 42/100 [00:34<00:48,  1.20it/s]--------------------------------------------- Result 43 ---------------------------------------------
Positive (94%) --> Negative (94%)

a gangster movie with the capacity to surprise .

a gangster movie with the capacity to flabbergasted .


[Succeeded / Failed / Skipped / Total] 31 / 1 / 11 / 43:  44% 44/100 [00:35<00:44,  1.25it/s]--------------------------------------------- Result 44 ---------------------------------------------
Positive (78%) --> Negative (82%)

the film has a laundry list of minor shortcomings , but the numerous scenes of gory mayhem are worth the price of admission . . . if " gory mayhem " is your idea of a good time .

the film has a laundry list of minor shortcomings , but the numerous scenes of gory mayhem are priceless the price of admission . . . if " gory mayhem " is your idea of a good time .


[Succeeded / Failed / Skipped / Total] 32 / 1 / 11 / 44:  44% 44/100 [00:35<00:44,  1.25it/s]--------------------------------------------- Result 45 ---------------------------------------------
Negative (52%) --> [SKIPPED]

if not a home run , then at least a solid base hit .


[Succeeded / Failed / Skipped / Total] 32 / 1 / 12 / 45:  46% 46/100 [00:35<00:41,  1.30it/s]--------------------------------------------- Result 46 ---------------------------------------------
Positive (88%) --> Negative (79%)

goldmember is funny enough to justify the embarrassment of bringing a barf bag to the moviehouse .

goldmember is comical enough to justify the embarrassment of bringing a barf bag to the moviehouse .


[Succeeded / Failed / Skipped / Total] 33 / 1 / 12 / 46:  46% 46/100 [00:35<00:41,  1.30it/s]--------------------------------------------- Result 47 ---------------------------------------------
Positive (72%) --> Negative (93%)

. . . a fairly disposable yet still entertaining b picture .

. . . a fairly disposable yet still droll b picture .


[Succeeded / Failed / Skipped / Total] 34 / 1 / 12 / 47:  48% 48/100 [00:36<00:39,  1.32it/s]--------------------------------------------- Result 48 ---------------------------------------------
Positive (98%) --> Negative (52%)

it may not be particularly innovative , but the film's crisp , unaffected style and air of gentle longing make it unexpectedly rewarding .

it may not be particularly unpublished , but the film's crisp , unaffected style and air of soft vacuuming make it unexpectedly bounties .


[Succeeded / Failed / Skipped / Total] 35 / 1 / 12 / 48:  48% 48/100 [00:36<00:39,  1.32it/s]--------------------------------------------- Result 49 ---------------------------------------------
Positive (94%) --> Negative (80%)

the film truly does rescue [the funk brothers] from motown's shadows . it's about time .

the film awfully does rescue [the funk brothers] from motown's shadows . it's about time .


[Succeeded / Failed / Skipped / Total] 36 / 1 / 12 / 49:  50% 50/100 [00:37<00:37,  1.34it/s]--------------------------------------------- Result 50 ---------------------------------------------
Positive (98%) --> Negative (52%)

drawing on an irresistible , languid romanticism , byler reveals the ways in which a sultry evening or a beer-fueled afternoon in the sun can inspire even the most retiring heart to venture forth .

drawing on an stupendous , languid romanticism , byler betrays the methodology in which a sultry evening or a beer-fueled afternoon in the sun can inspire even the most retiring heart to venture forth .


[Succeeded / Failed / Skipped / Total] 37 / 1 / 12 / 50:  50% 50/100 [00:37<00:37,  1.34it/s]--------------------------------------------- Result 51 ---------------------------------------------
Negative (92%) --> [SKIPPED]

works because we're never sure if ohlinger's on the level or merely a dying , delusional man trying to get into the history books before he croaks .


[Succeeded / Failed / Skipped / Total] 37 / 1 / 13 / 51:  52% 52/100 [00:37<00:34,  1.38it/s]--------------------------------------------- Result 52 ---------------------------------------------
Positive (64%) --> Negative (68%)

[scherfig] has made a movie that will leave you wondering about the characters' lives after the clever credits roll .

[scherfig] is made a movie that will leave you wondering about the characters' lives after the cleverer credits roll .


[Succeeded / Failed / Skipped / Total] 38 / 1 / 13 / 52:  52% 52/100 [00:37<00:34,  1.38it/s]--------------------------------------------- Result 53 ---------------------------------------------
Positive (98%) --> Negative (51%)

a heady , biting , be-bop ride through nighttime manhattan , a loquacious videologue of the modern male and the lengths to which he'll go to weave a protective cocoon around his own ego .

a hectic , gnawing , be-bop ride through nighttime manhattan , a loquacious videologue of the upgraded male and the lengths to which he'll go to weave a protective cocoon around his own ego .


[Succeeded / Failed / Skipped / Total] 39 / 1 / 13 / 53:  54% 54/100 [00:38<00:32,  1.41it/s]--------------------------------------------- Result 54 ---------------------------------------------
Positive (97%) --> Negative (54%)

skin of man gets a few cheap shocks from its kids-in-peril theatrics , but it also taps into the primal fears of young people trying to cope with the mysterious and brutal nature of adults .

skin of man gets a few cheap shocks from its kids-in-peril theatrics , but it also faucets into the primordial fears of young people trying to cope with the mysterious and brutal nature of adults .


[Succeeded / Failed / Skipped / Total] 40 / 1 / 13 / 54:  54% 54/100 [00:38<00:32,  1.41it/s]--------------------------------------------- Result 55 ---------------------------------------------
Positive (86%) --> Negative (55%)

the piano teacher is not an easy film . it forces you to watch people doing unpleasant things to each other and themselves , and it maintains a cool distance from its material that is deliberately unsettling .

the piano teacher is not an easy film . it forces you to watch people doing unpleasant things to each other and themselves , and it maintains a copacetic distance from its material that is deliberately unsettling .


[Succeeded / Failed / Skipped / Total] 41 / 1 / 13 / 55:  56% 56/100 [00:38<00:30,  1.45it/s]--------------------------------------------- Result 56 ---------------------------------------------
Positive (95%) --> Negative (91%)

as refreshing as a drink from a woodland stream .

as retrofit as a drink from a woodland stream .


[Succeeded / Failed / Skipped / Total] 42 / 1 / 13 / 56:  56% 56/100 [00:38<00:30,  1.45it/s]--------------------------------------------- Result 57 ---------------------------------------------
Positive (55%) --> Negative (95%)

williams absolutely nails sy's queasy infatuation and overall strangeness .

williams absolutely nails sy's queasy infatuation and overall ennui .


[Succeeded / Failed / Skipped / Total] 43 / 1 / 13 / 57:  58% 58/100 [00:39<00:28,  1.48it/s]--------------------------------------------- Result 58 ---------------------------------------------
Positive (67%) --> Negative (52%)

can i admit xxx is as deep as a petri dish and as well-characterized as a telephone book but still say it was a guilty pleasure ?

can i admit xxx is as deep as a petri dish and as well-characterized as a telephone book but still say it was a guilty amusement ?


[Succeeded / Failed / Skipped / Total] 44 / 1 / 13 / 58:  58% 58/100 [00:39<00:28,  1.48it/s]--------------------------------------------- Result 59 ---------------------------------------------
Positive (85%) --> Negative (55%)

while it's nothing we haven't seen before from murphy , i spy is still fun and enjoyable and so aggressively silly that it's more than a worthwhile effort .

while it's nothing we haven't seen before from murphy , i spy is still fun and cosy and so aggressively silly that it's more than a worthwhile effort .


[Succeeded / Failed / Skipped / Total] 45 / 1 / 13 / 59:  60% 60/100 [00:39<00:26,  1.52it/s]--------------------------------------------- Result 60 ---------------------------------------------
Negative (73%) --> [SKIPPED]

by the time it ends in a rush of sequins , flashbulbs , blaring brass and back-stabbing babes , it has said plenty about how show business has infiltrated every corner of society -- and not always for the better .


[Succeeded / Failed / Skipped / Total] 45 / 1 / 14 / 60:  60% 60/100 [00:39<00:26,  1.52it/s]--------------------------------------------- Result 61 ---------------------------------------------
Positive (98%) --> Negative (75%)

an intimate contemplation of two marvelously messy lives .

an squeamish contemplation of two marvelously messy lives .


[Succeeded / Failed / Skipped / Total] 46 / 1 / 14 / 61:  62% 62/100 [00:39<00:24,  1.56it/s]--------------------------------------------- Result 62 ---------------------------------------------
Positive (54%) --> Negative (58%)

rarely has skin looked as beautiful , desirable , even delectable , as it does in trouble every day .

rarely has skin looked as nice , desirable , even delectable , as it does in trouble every day .


[Succeeded / Failed / Skipped / Total] 47 / 1 / 14 / 62:  62% 62/100 [00:39<00:24,  1.56it/s]--------------------------------------------- Result 63 ---------------------------------------------
Positive (97%) --> Negative (60%)

this is one of those rare docs that paints a grand picture of an era and makes the journey feel like a party .

this is one of those rare docs that paints a hefty picture of an eras and ai the trip feel like a portion .


[Succeeded / Failed / Skipped / Total] 48 / 1 / 14 / 63:  64% 64/100 [00:40<00:22,  1.57it/s]--------------------------------------------- Result 64 ---------------------------------------------
Positive (98%) --> Negative (94%)

poignant if familiar story of a young person suspended between two cultures .

disquieting if familiar story of a young person suspended between two cultures .


[Succeeded / Failed / Skipped / Total] 49 / 1 / 14 / 64:  64% 64/100 [00:40<00:22,  1.57it/s]--------------------------------------------- Result 65 ---------------------------------------------
Positive (94%) --> Negative (93%)

a metaphor for a modern-day urban china searching for its identity .

a cliché for a modern-day urban china searching for its identity .


[Succeeded / Failed / Skipped / Total] 50 / 1 / 14 / 65:  66% 66/100 [00:41<00:21,  1.60it/s]--------------------------------------------- Result 66 ---------------------------------------------
Positive (98%) --> Negative (69%)

for all its brooding quality , ash wednesday is suspenseful and ultimately unpredictable , with a sterling ensemble cast .

for all its brooding quality , ash wednesday is upsetting and ultimately unpredictable , with a stirling ensemble cast .


[Succeeded / Failed / Skipped / Total] 51 / 1 / 14 / 66:  66% 66/100 [00:41<00:21,  1.60it/s]--------------------------------------------- Result 67 ---------------------------------------------
Positive (90%) --> Negative (52%)

an odd drama set in the world of lingerie models and bar dancers in the midwest that held my interest precisely because it didn't try to .

an odd cinematographic set in the globo of lingerie models and bar dancers in the midwest that held my interest precisely because it didn't try to .


[Succeeded / Failed / Skipped / Total] 52 / 1 / 14 / 67:  68% 68/100 [00:41<00:19,  1.63it/s]--------------------------------------------- Result 68 ---------------------------------------------
Positive (85%) --> Negative (67%)

the film feels uncomfortably real , its language and locations bearing the unmistakable stamp of authority .

the film feels uncomfortably actual , its language and locations bearing the unmistakable stamp of authority .


[Succeeded / Failed / Skipped / Total] 53 / 1 / 14 / 68:  68% 68/100 [00:41<00:19,  1.63it/s]--------------------------------------------- Result 69 ---------------------------------------------
Positive (98%) --> Negative (90%)

despite its faults , gangs excels in spectacle and pacing .

despite its faults , gangs overwhelms in spectacle and pacing .


[Succeeded / Failed / Skipped / Total] 54 / 1 / 14 / 69:  70% 70/100 [00:42<00:18,  1.66it/s]--------------------------------------------- Result 70 ---------------------------------------------
Positive (80%) --> Negative (51%)

entertaining despite its one-joke premise with the thesis that women from venus and men from mars can indeed get together .

amusing despite its one-joke premise with the thesis that women from venus and men from mars can indeed get together .


[Succeeded / Failed / Skipped / Total] 55 / 1 / 14 / 70:  70% 70/100 [00:42<00:18,  1.66it/s]--------------------------------------------- Result 71 ---------------------------------------------
Positive (97%) --> Negative (91%)

a tightly directed , highly professional film that's old-fashioned in all the best possible ways .

a tightly directed , excessively professional film that's old-fashioned in all the best possible ways .


[Succeeded / Failed / Skipped / Total] 56 / 1 / 14 / 71:  72% 72/100 [00:43<00:16,  1.66it/s]--------------------------------------------- Result 72 ---------------------------------------------
Positive (98%) --> Negative (60%)

it's dark but has wonderfully funny moments ; you care about the characters ; and the action and special effects are first-rate .

it's ghoulish but has unspeakably joke mins ; you zorg about the characters ; and the operating and special effects are first-rate .


[Succeeded / Failed / Skipped / Total] 57 / 1 / 14 / 72:  72% 72/100 [00:43<00:16,  1.66it/s]--------------------------------------------- Result 73 ---------------------------------------------
Positive (89%) --> Negative (52%)

in visual fertility treasure planet rivals the top japanese animations of recent vintage .

in visual fertility coffer planet rivals the supremo japanese animations of recent vintage .


[Succeeded / Failed / Skipped / Total] 58 / 1 / 14 / 73:  74% 74/100 [00:44<00:15,  1.68it/s]--------------------------------------------- Result 74 ---------------------------------------------
Positive (98%) --> Negative (96%)

enormously enjoyable , high-adrenaline documentary .

enormously droll , high-adrenaline paperwork .


[Succeeded / Failed / Skipped / Total] 59 / 1 / 14 / 74:  74% 74/100 [00:44<00:15,  1.68it/s]--------------------------------------------- Result 75 ---------------------------------------------
Positive (97%) --> Negative (50%)

buy is an accomplished actress , and this is a big , juicy role .

buy is an accomplished actress , and this is a big , crusty role .


[Succeeded / Failed / Skipped / Total] 60 / 1 / 14 / 75:  76% 76/100 [00:44<00:14,  1.69it/s]--------------------------------------------- Result 76 ---------------------------------------------
Positive (98%) --> Negative (58%)

it works its magic with such exuberance and passion that the film's length becomes a part of its fun .

it functioned its potions with such elation and passion that the film's length becomes a part of its banter .


[Succeeded / Failed / Skipped / Total] 61 / 1 / 14 / 76:  76% 76/100 [00:44<00:14,  1.69it/s]--------------------------------------------- Result 77 ---------------------------------------------
Positive (98%) --> Negative (58%)

beautifully crafted and brutally honest , promises offers an unexpected window into the complexities of the middle east struggle and into the humanity of its people .

impossibly crafted and hastily veritable , promises offers an unforeseen fibreglass into the complexities of the middle east tussle and into the humans of its people .


[Succeeded / Failed / Skipped / Total] 62 / 1 / 14 / 77:  78% 78/100 [00:46<00:13,  1.68it/s]--------------------------------------------- Result 78 ---------------------------------------------
Positive (98%) --> Negative (67%)

an old-fashioned but emotionally stirring adventure tale of the kind they rarely make anymore .

an old-fashioned but emotionally wavering adventure tale of the kind they rarely make anymore .


[Succeeded / Failed / Skipped / Total] 63 / 1 / 14 / 78:  78% 78/100 [00:46<00:13,  1.68it/s]--------------------------------------------- Result 79 ---------------------------------------------
Positive (98%) --> Negative (93%)

charlotte sometimes is a gem . it's always enthralling .

charlotte sometimes is a bling . it's always hallucinatory .


[Succeeded / Failed / Skipped / Total] 64 / 1 / 14 / 79:  80% 80/100 [00:47<00:11,  1.70it/s]--------------------------------------------- Result 80 ---------------------------------------------
Positive (92%) --> Negative (50%)

in my opinion , analyze that is not as funny or entertaining as analyze this , but it is a respectable sequel .

in my opinion , analyze that is not as funny or entertaining as discusses this , but it is a reputable sequel .


[Succeeded / Failed / Skipped / Total] 65 / 1 / 14 / 80:  80% 80/100 [00:47<00:11,  1.70it/s]--------------------------------------------- Result 81 ---------------------------------------------
Positive (98%) --> Negative (94%)

a remarkable film by bernard rose .

a whopping film by bernard rose .


[Succeeded / Failed / Skipped / Total] 66 / 1 / 14 / 81:  82% 82/100 [00:47<00:10,  1.72it/s]--------------------------------------------- Result 82 ---------------------------------------------
Positive (80%) --> Negative (58%)

zhuangzhuang creates delicate balance of style , text , and subtext that's so simple and precise that anything discordant would topple the balance , but against all odds , nothing does .

zhuangzhuang creates delicate balance of style , text , and subtext that's so simple and specify that anything discordant would topple the balance , but against all odds , nothing does .


[Succeeded / Failed / Skipped / Total] 67 / 1 / 14 / 82:  82% 82/100 [00:47<00:10,  1.72it/s]--------------------------------------------- Result 83 ---------------------------------------------
Positive (84%) --> Negative (71%)

a much more successful translation than its most famous previous film adaptation , writer-director anthony friedman's similarly updated 1970 british production .

a much more propitious translation than its most famous previous film adaptation , writer-director anthony friedman's similarly updated 1970 british production .


[Succeeded / Failed / Skipped / Total] 68 / 1 / 14 / 83:  84% 84/100 [00:47<00:09,  1.75it/s]--------------------------------------------- Result 84 ---------------------------------------------
Positive (98%) --> Negative (56%)

an original and highly cerebral examination of the psychopathic mind

an rudimentary and highly cerebral examination of the psychopathic mind


[Succeeded / Failed / Skipped / Total] 69 / 1 / 14 / 84:  84% 84/100 [00:47<00:09,  1.75it/s]--------------------------------------------- Result 85 ---------------------------------------------
Positive (93%) --> Negative (83%)

michel piccoli's moving performance is this films reason for being .

michel piccoli's resettled performance is this films reason for being .


[Succeeded / Failed / Skipped / Total] 70 / 1 / 14 / 85:  86% 86/100 [00:48<00:07,  1.76it/s]--------------------------------------------- Result 86 ---------------------------------------------
Positive (99%) --> Negative (95%)

a captivating and intimate study about dying and loving . . .

a hallucinatory and cosy scrutinized about decedent and loving . . .


[Succeeded / Failed / Skipped / Total] 71 / 1 / 14 / 86:  86% 86/100 [00:48<00:07,  1.76it/s]--------------------------------------------- Result 87 ---------------------------------------------
Positive (96%) --> Negative (75%)

this is an elegantly balanced movie -- every member of the ensemble has something fascinating to do -- that doesn't reveal even a hint of artifice .

this is an prettily balancing movie -- every member of the ensemble has something fascinating to do -- that doesn't reveal even a hint of artifice .


[Succeeded / Failed / Skipped / Total] 72 / 1 / 14 / 87:  88% 88/100 [00:49<00:06,  1.78it/s]--------------------------------------------- Result 88 ---------------------------------------------
Positive (98%) --> Negative (61%)

[grant] goes beyond his usual fluttering and stammering and captures the soul of a man in pain who gradually comes to recognize it and deal with it .

[grant] goes beyond his usual fluttering and stammering and incarcerate the wits of a man in pain who gradually comes to recognize it and deal with it .


[Succeeded / Failed / Skipped / Total] 73 / 1 / 14 / 88:  88% 88/100 [00:49<00:06,  1.78it/s]--------------------------------------------- Result 89 ---------------------------------------------
Positive (96%) --> Negative (58%)

a high-spirited buddy movie about the reunion of berlin anarchists who face arrest 15 years after their crime .

a high-spirited buddy video about the pooled of germania anarchist who facial intercepted 15 olds after their penal .


[Succeeded / Failed / Skipped / Total] 74 / 1 / 14 / 89:  90% 90/100 [00:50<00:05,  1.77it/s]--------------------------------------------- Result 90 ---------------------------------------------
Negative (84%) --> [SKIPPED]

about the best thing you could say about narc is that it's a rock-solid little genre picture . whether you like it or not is basically a matter of taste .


[Succeeded / Failed / Skipped / Total] 74 / 1 / 15 / 90:  90% 90/100 [00:50<00:05,  1.77it/s]--------------------------------------------- Result 91 ---------------------------------------------
Positive (97%) --> Negative (80%)

an involving , inspirational drama that sometimes falls prey to its sob-story trappings .

an involving , incentive cataclysmic that sometimes falls prey to its sob-story trappings .


[Succeeded / Failed / Skipped / Total] 75 / 1 / 15 / 91:  92% 92/100 [00:51<00:04,  1.79it/s]--------------------------------------------- Result 92 ---------------------------------------------
Positive (96%) --> Negative (75%)

some of the most inventive silliness you are likely to witness in a movie theatre for some time .

some of the most contrivance silliness you are likely to witness in a movie theatre for some time .


[Succeeded / Failed / Skipped / Total] 76 / 1 / 15 / 92:  92% 92/100 [00:51<00:04,  1.79it/s]--------------------------------------------- Result 93 ---------------------------------------------
Positive (98%) --> Negative (57%)

canadian filmmaker gary burns' inventive and mordantly humorous take on the soullessness of work in the city .

canadian scriptwriter gary burns' inventor and mordantly prank takes on the soullessness of work in the shing .


[Succeeded / Failed / Skipped / Total] 77 / 1 / 15 / 93:  94% 94/100 [00:52<00:03,  1.78it/s]--------------------------------------------- Result 94 ---------------------------------------------
Positive (98%) --> Negative (65%)

a rollicking ride , with jaw-dropping action sequences , striking villains , a gorgeous color palette , astounding technology , stirring music and a boffo last hour that leads up to a strangely sinister happy ending .

a rollicking wrinkle , with jaw-dropping action sequences , striking villains , a leggy color palette , astounding technology , agitation music and a boffo last hour that leads up to a strangely sinister happy ending .


[Succeeded / Failed / Skipped / Total] 78 / 1 / 15 / 94:  94% 94/100 [00:52<00:03,  1.78it/s]--------------------------------------------- Result 95 ---------------------------------------------
Positive (99%) --> Negative (51%)

everyone's insecure in lovely and amazing , a poignant and wryly amusing film about mothers , daughters and their relationships .

everyone's insecure in lovely and whopping , a disquieting and wryly amusing film about mothers , daughters and their relationships .


[Succeeded / Failed / Skipped / Total] 79 / 1 / 15 / 95:  96% 96/100 [00:53<00:02,  1.80it/s]--------------------------------------------- Result 96 ---------------------------------------------
Positive (52%) --> Negative (75%)

the closest thing to the experience of space travel

the closest thing to the pilot of space travel


[Succeeded / Failed / Skipped / Total] 80 / 1 / 15 / 96:  96% 96/100 [00:53<00:02,  1.80it/s]--------------------------------------------- Result 97 ---------------------------------------------
Positive (96%) --> Negative (96%)

full of surprises .

full of stumped .


[Succeeded / Failed / Skipped / Total] 81 / 1 / 15 / 97:  98% 98/100 [00:54<00:01,  1.81it/s]--------------------------------------------- Result 98 ---------------------------------------------
Positive (90%) --> Negative (52%)

connoisseurs of chinese film will be pleased to discover that tian's meticulous talent has not withered during his enforced hiatus .

connoisseurs of cantonese film will be pleased to discover that tian's meticulous staffing is not bloomed during his enforced harford .


[Succeeded / Failed / Skipped / Total] 82 / 1 / 15 / 98:  98% 98/100 [00:54<00:01,  1.81it/s]--------------------------------------------- Result 99 ---------------------------------------------
Positive (95%) --> Negative (69%)

if you can push on through the slow spots , you'll be rewarded with some fine acting .

if you can push on through the slow spots , you'll be recompense with some wondrous behaving .


[Succeeded / Failed / Skipped / Total] 83 / 1 / 15 / 99: 100% 100/100 [00:54<00:00,  1.84it/s]--------------------------------------------- Result 100 ---------------------------------------------
Negative (50%) --> [SKIPPED]

an unusually dry-eyed , even analytical approach to material that is generally played for maximum moisture .


[Succeeded / Failed / Skipped / Total] 83 / 1 / 16 / 100: 100% 100/100 [00:54<00:00,  1.84it/s]

+-------------------------------+--------+
| Attack Results                |        |
+-------------------------------+--------+
| Number of successful attacks: | 83     |
| Number of failed attacks:     | 1      |
| Number of skipped attacks:    | 16     |
| Original accuracy:            | 84.0%  |
| Accuracy under attack:        | 1.0%   |
| Attack success rate:          | 98.81% |
| Average perturbed word %:     | 13.68% |
| Average num. words per input: | 18.45  |
| Avg num queries:              | 80.13  |
+-------------------------------+--------+

Looks like our model was 84% successful (makes sense - same evaluation set as textattack eval!), meaning that TextAttack attacked the model with 84 examples (since the attack won’t run if an example is originally mispredicted). The attack success rate was 98.8%, meaning that TextFooler failed to find an adversarial example only 1.2% (1 out of 84) of the time.

Conclusion

That’s all, folks! We’ve learned how to train, evaluate, and attack a model with TextAttack, using only three commands! 😀

Bonus

There are many powerful functions in TextAttack, we can use through command lines. Here is a list of examples as bonus for your learning.

[ ]:
!textattack attack --recipe deepwordbug --model lstm-mr --num-examples 2 --log-summary-to-json attack_summary.json
[ ]:
!textattack attack --model cnn-yelp --num-examples 3 --search-method greedy-word-wir --transformation word-swap-wordnet --constraints cola^max_diff=0.1 bert-score^min_bert_score=0.7 --enable-advance-metrics
[ ]:
!textattack attack --model lstm-mr --recipe deepwordbug --num-examples 2 --attack-n --enable-advance-metrics
[ ]:
!textattack attack --model lstm-mr --recipe hotflip --num-examples 4 --num-examples-offset 3 --enable-advance-metrics
[ ]:
!textattack attack --model-from-huggingface distilbert-base-uncased-finetuned-sst-2-english --dataset-from-huggingface glue^sst2^train --recipe deepwordbug --num-examples 3 --enable-advance-metrics
[ ]:
! textattack attack --model cnn-imdb --attack-from-file tests/sample_inputs/attack_from_file.py^Attack --num-examples 2  --num-examples-offset 18 --attack-n