FlipJev

FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models

Xiaogeng Liu1*, Xinyan Wang2, Chaowei Xiao1

1Johns Hopkins University 2University of Wisconsin–Madison
*Corresponding author: xliu316@jhu.edu

Paper Code BibTeX

Typed decision models answer a developer's question with probabilities over typed options instead of generating text. Common prompt injections do not change the correct answers of Jev, the first deployed model of this kind. A quick black-box attack that reads only these probabilities finds short suffixes that do. One suffix per dataset flips 43% of unseen documents, and a suffix found on one task flips the decisions of other tasks.

One email, written by us, sent to typesafe/jev-1.13. Press the button to append the 7-token suffix that the attack found after 1,860 queries.

Try the suffixes

The first three tabs show universal suffixes. Each was found once, on 10 training documents of its dataset, and then appended to documents that the attack never queried. The question stays the same and the suffix never names its subject, so the correct answer stays yes. Every probability is the mean of 10 fresh queries to Jev, without and with the suffix. The last tab shows a suffix that moves the safety verdict on an agent trajectory.

What we found

  1. Common prompt injections fail

    No common strategy changed a correct answer of Jev in 10,240 paired tests on emails, including requests to output the wrong label, fake completions, context ignoring and a forged JSON answer. On the other 11 datasets, five common strategies flipped none of the unseen inputs either.

  2. A quick query-based attack succeeds

    The attack edits a short suffix with cheap word-level changes and keeps the edits that raise the probability of the wrong answer. On the nine datasets, it flipped 58 of 180 confident targets within 2,000 queries, which take about two minutes. On emails, 11 of 28 searches found a confirmed flip.

  3. One suffix flips many unseen documents

    Scored on a batch of 10 training documents instead of one, the same attack finds one suffix per dataset. These suffixes flip 114 of 264 unseen documents (43%), and the universal suffix for ATBench moves 9 of 30 unseen unsafe agent trajectories to a safe verdict.

  4. Suffixes transfer across tasks

    A typed decision model performs many tasks, because the question in its input defines the task. The suffix found on full arXiv papers flips 90–100% of the documents of five other datasets that ask different questions, without a single query to them.

How the attack works

The attacker writes only the untrusted content and reads only the probabilities that the API returns. To keep pace with a model that answers in about 0.3 seconds, the attack generates candidates with word-level edits instead of an attacker language model, so its whole budget goes to queries.

  1. Start from a seed

    A short text such as question: is 'no' accurate, appended after a blank line at the end of the untrusted content.

  2. Mutate

    Each round makes 64 new suffixes from the 16 best ones: substitutions, insertions, deletions, swaps, copies, random strings and crossover.

  3. Query and rank

    Jev scores every candidate. The fitness is the probability of the wrong answer, averaged over 10 training documents for a universal suffix.

  4. Confirm and shorten

    A new best suffix counts only if at least 2 of 3 fresh queries flip. The attack then deletes words while the flip holds.

Constraints on every suffix

  • At most 32 tokens (64 for agent trajectories), appended to the untrusted content only.
  • No word that names the subject of the question, so the correct answer does not change.
  • The trusted question, the answer options and the developer's settings stay untouched.

Results

Attack success rate on the 12 datasets. Share of correct, confident decisions that each method flips. Per-document search: one search of up to 2,000 queries for each target. Universal suffix: one suffix per dataset, found on 10 training inputs and applied to 30 unseen inputs (24 for 20 Newsgroups).
Flips accumulate with the number of queries. Share of the 20 targets of each dataset (email: of the 28 searches) whose search confirms a flip within a given number of queries. Point at a line or a name to compare.
Transfer between datasets and tasks. Each cell gives the share of the first 20 unseen documents of the column dataset that the universal suffix of the row dataset flips (3 queries each, no search). Select a row to read its suffix.

Responsible disclosure

We reported these findings to TypeSafe before publication, through the official security-reporting channel of the vendor. The report included the universal suffixes, the exact input of each suffix and the responses of the API, so each case can be replayed. The suffixes change only the decisions of a typed decision model. They do not extract data or make a model produce harmful text, and they affect only systems in which the attacker can already write content.

Citation

@misc{liu2026flipjevfastblackboxred,
      title={FlipJev: Fast Black-Box Red Teaming Finds Universal
             Adversarial Suffixes for Typed Decision Models}, 
      author={Xiaogeng Liu and Xinyan Wang and Chaowei Xiao},
      year={2026},
      eprint={2610.38895},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2610.38895}, 
}