blackcat // JAILBREAKING
A survey of the attack surface, the evidence base, and the countermeasures.
Written for people who have to defend the thing, and who therefore need to
understand it exactly as well as the person attacking it does.
Abstract
A jailbreak is any input that causes a safety-aligned language model to
produce output its training intended it to withhold. The term is fifteen years
old in the prompt-injection literature and about three years old in the form
we now mean, and in those three years it has produced several hundred papers,
a dozen benchmarks, four or five credible theories, and no general solution.
This document is a synthesis of that literature, ordered by cost.
The ordering is deliberate. The single most useful fact about jailbreaking is
that the price of a successful attack varies by six orders of magnitude across
methods, while the reported success rates vary by much less. A technique
that needs ten thousand gradient steps and a copy of the weights is a
different object from one that needs four turns of conversation and a
paraphrase, even when both are published at "90% ASR". Ranking by cost first,
and by reported efficacy second, produces a picture of the field that the
headline numbers obscure: the cheap end is where nearly all real incidents
live, the expensive end is where nearly all research effort goes, and the
defences that survive contact with an adaptive attacker are almost entirely
system-level rather than model-level.
Three claims organise the survey:
- Safety training suppresses a behaviour without removing the capability.
Every theory of jailbreaking is a theory of why that suppression is
shallow, and the empirical record - direction ablation, prefilling,
parameter pruning, multi-turn erosion - supports the shallow reading. - Attack success is a function of budget, not of cleverness. Given a query
budget and an adaptive loop, success rates approach unity against
undefended frontier models; given a fixed small budget, success is a
lottery whose odds are set by the defence, not the prompt. - Defences are evaluated in a regime that flatters them. The published
attack success rates of most defences fall by an order of magnitude when
the attacker is allowed to know the defence exists.
The middle of the document - Part IV - is the evidence core: what the numbers
actually say, ranked, with the conditions attached. The front is the cheap
end, the back is the expensive end and the machinery of measurement. An
annotated bibliography closes it, because a survey without one is an opinion.
Part VI then turns the document around and reads the same evidence from the
operator's side: what it costs to run a jailbreak as a business rather than as
an experiment, which published methods survive an amortisation law that makes
any per-request search unviable at consumer prices, and why the binding
constraint on the adversary is a payment instrument rather than a classifier.
It is written for defenders, because the fastest way to price a threat model is
to compute the attacker's margin - and it closes by setting out why the
offensive reading of this survey fails on arithmetic rather than on ethics. The
payload layer is absent from both halves, for the reason given in §0.
0. Scope method and notation
Scope. This is a review of the public literature on causing aligned models
to violate their own policies, and on preventing that. It describes mechanisms
and reports published results; it does not reproduce payloads, and it treats
the artefact repositories cited in §23 as the reproduction path. That is the
normal division of labour in security writing: the paper explains the class of
attack, the artefact is where the bytes live, and anyone who needs the bytes
is already able to find them.
Method. Claims are tagged by provenance:
- Reported - a named paper, lab, or vendor published the figure. Cited
inline by author and year; full entry in §23. - Derived - arithmetic performed here on reported figures. The arithmetic
is shown so it can be checked. - Inferred - this document's judgement, marked as such, and never
load-bearing for a factual claim.
Numbers are quoted with the conditions the original authors used, because
attack success rate (ASR) is not a comparable quantity across papers. §14 is
largely about why. Where a figure is quoted without its benchmark, treat it as
an order of magnitude rather than a measurement.
Notation.
| Symbol | Meaning |
|---|---|
| ASR | Attack success rate, as defined by the reporting paper - see §14.1 |
k |
Query budget: distinct model calls available to the attacker |
n |
Number of independent attack samples drawn at inference time |
T |
Number of conversational turns |
x |
The harmful request, called the behaviour or goal |
p |
The adversarial prompt, x wrapped in whatever the method adds |
θ |
Model weights; "white-box" means the attacker has them |
h |
Hidden state at layer l; r is the refusal direction of §2.3 |
| Half-life | Time until a published technique stops working on the current model, unprompted by its author - see §3.3 |
A note on the word. "Jailbreak" is used here for policy circumvention by
input, which excludes three neighbouring things that are often conflated with
it: backdoors implanted in weights (§11.4), extraction of training data, and
denial of service. It includes indirect prompt injection (§13.1), because from
the model's point of view the two are the same event - untrusted text arriving
in a position the model treats as instruction.
Index
Part I - Foundations
Part II - The fast end: what costs least and works soonest
- 4. Prompt-space techniques
- 5. Automated black-box search
- 6. Multi-turn and stateful attacks
- 7. Reasoning models
- 8. The latency ledger
Part III - The expensive end: optimisation, weights, and other channels
- 9. Gradient-based optimisation
- 10. Adaptive attacks with logits
- 11. Weight-space modification
- 12. Non-text channels
- 13. The agentic surface
Part IV - The evidence core: what actually works
- 14. How to read an attack success rate
- 15. The attack evidence table
- 16. The defence evidence table
- 17. The layered stack
- 18. A robustness protocol
- 19. Open problems
Part V - Apparatus
- 20. Timeline
- 21. Glossary
- 22. Number card
- 23. Annotated bibliography
- 24. Method appendix
- 25. Technique index
Part VI - The other side of the same surface
- 26. The shape of the job
- 27. Target selection
- 28. Sourcing capability
- 29. The operator's prompt corpus
- 30. The harness
- 31. Judging your own success
- 32. Session tradecraft
- 33. Weight-space operations
- 34. Delivery
- 35. Reliability and the maintenance treadmill
- 36. Detection and evasion, honestly
- 37. The operator's cost model
- 38. Attribution, or how they get caught
- 39. Why the blackhat reading of this document is wrong
- 40. What changes for the defender
Part VI apparatus
- 41. Timeline, operator side
- 42. Operator glossary
- 43. Number card, operator edition
- 44. Source notes for Part VI
- 45. Stage index
Part I - Foundations
1. What counts as a jailbreak
A language model is trained on a mixture that includes two objectives that
pull in opposite directions: be maximally helpful, and refuse to help with
certain classes of request. The refusal behaviour is installed by supervised
fine-tuning on refusal demonstrations and by preference optimisation against
harmlessness judgements, and it generalises - imperfectly - to requests the
trainers never wrote. A jailbreak is the gap between the generalisation and
the policy: an input that the policy would forbid, which the model answers
anyway.
Three properties of that definition matter for everything that follows.
It is policy-relative, not capability-relative. The model does not acquire
the ability to synthesise a nerve agent when it is jailbroken; it already has
it, in the flat sense that the capability is latent in the pretraining
distribution. The jailbreak removes a disposition, not a limit. This is why
jailbreaks are cheap relative to the harm they unlock, and why the harm
category with the steepest uplift is the one where the knowledge is most
widely published.
It is graded, not binary. A response can refuse, comply uselessly, comply
partially, or comply fully. Binary judges collapse these, which is the single
largest source of inflated numbers in the field (Souly et al., 2024). The
useful quantity is the fraction of the policy-relevant information actually
transferred, which is what StrongREJECT scores and what most papers do not.
It is a distributional property of a system, not a string. The same
sentence is a jailbreak against one checkpoint and a refusal against another;
the same checkpoint behind a classifier is a different system. Papers
therefore report on a system, and comparisons across systems that differ in
more than one layer are not comparisons.
1.1 Four objects that are not the same thing
| Object | Mechanism | Controlled by | Typical ASR driver |
|---|---|---|---|
| Prompt-level jailbreak | Input reframes the request (§4) | The model's alignment | Prompt craft |
| Optimised adversarial input | Gradient or search finds a suffix (§9) | The model's robustness | Compute |
| Weight modification | Refusal behaviour removed (§11) | Whoever holds the checkpoint | Access to weights |
| Injected instruction | Untrusted content arrives as instruction (§13.1) | The harness, not the model | Whether tools are on |
The third row is not an attack on a model but on an artefact: once weights
are public, no amount of alignment survives a fine-tune, and the relevant
question becomes provenance rather than robustness. The fourth row is the one
that matters operationally in 2026, because it does not require a jailbroken
model at all - it requires a model that reads.
2. Three theories of why safety training fails
Every serious paper on jailbreaking implicitly adopts one of three
explanations, and the explanation determines which defence looks promising.
They are not mutually exclusive; the current consensus is that all three are
partly true, which is exactly why no single fix works.
2.1 Competing objectives
Wei, Haghtalab and Steinhardt (2023) frame the problem as a conflict between
two objectives the model was trained on: helpfulness and harmlessness. A
jailbreak is an input on which the helpfulness objective dominates - because
the request is framed so that refusing looks unhelpful, unsafe, or
inconsistent with a role the model has accepted. Their evidence is that
attacks which increase the apparent helpfulness of compliance, or which make
refusal appear to violate a higher-priority instruction, work reliably; and
that scaling and RLHF do not fix the conflict, they relocate it.
The strength of the account is that it explains why the highest-yield
techniques are social rather than technical. If refusal is a decision made
under competing pressures, then anything that raises the perceived cost of
refusing - a persona with different norms, an authority claim, a framing in
which refusal harms someone - shifts the decision. This is why the cheap end
of §4 works at all, and why it works better on models trained harder for
helpfulness.
The weakness is that it does not explain transfer: a suffix optimised on one
model should not move another model's decision if the decision were purely a
reasoning conflict. Something lower-level is also happening.
2.2 Mismatched generalisation
The second account says safety training generalises narrowly while capability
generalises broadly. Wei et al. call this mismatched generalisation: the
model learns "refuse requests that look like this in English, in this register,
in this script" and does not learn the invariant "refuse this capability
regardless of surface form". Any transformation that moves the input outside
the safety training distribution but inside the capability distribution -
base64, a cipher, a low-resource language, ASCII art, a nested fiction - is
therefore a candidate attack.
This is the account with the strongest mechanistic support, because the
attacks it predicts are exactly the ones that work: encoding shifts (Yuan et
al., 2023), low-resource language transfer (Yong et al., 2023), ASCII-art
masking (Jiang et al., 2024), and format injection all exploit the fact that
the refusal classifier and the capability live at different levels of
abstraction. It also predicts the observed asymmetry in defence difficulty:
teaching the model to refuse in one more surface form is cheap, teaching it
the invariant is not.
2.3 Shallow alignment and the refusal direction
The third account is the most recent and the most uncomfortable: the safety
behaviour may be implemented as a thin, separable, low-dimensional feature
rather than as a distributed property of the network.
Two lines of evidence. First, Arditi et al. (2024) show that refusal in
open-weight chat models is mediated by a single direction in activation space;
projecting that direction out - directional ablation, the technique behind
the "abliterated" checkpoints of §11.2 - removes refusal across thirteen
models with little measured capability loss. Second, Qi et al. (2024) show
that safety behaviour is concentrated in the first few generated tokens: if
the model is forced past them, refusal probability collapses. Their result is
that alignment is, in their phrase, not more than a few tokens deep.
If this is right, the implications are structural. A behaviour that is a
direction can be removed by anyone with weights and a weekend. A behaviour
that lives in the first few tokens can be bypassed by anyone who can control
the first few tokens - which is what a prefill API, a persuasive opening, or a
multi-turn setup all provide. And a behaviour that is shallow will not
generalise to a distribution shift no matter how much data is added, which is
the observed failure mode of adversarial training.
2.4 A fourth account: reward misspecification
Qi et al. (2024, the reward-misspecification paper) offer a complementary
framing from the optimisation side: a jailbreak is a prompt that scores well
under the model's learned reward while violating the intent of the policy. In
this view the model is not being tricked; it is doing exactly what it was
optimised to do, on an input the optimiser never saw. The practical
consequence is that jailbreaks are not anomalies to be patched but the
expected behaviour of any policy optimised against a proxy - which is a
pessimistic result for anyone hoping for a clean fix, and a useful one for
anyone designing detection, because it says the attack distribution is
unbounded.
2.5 What the theories agree on
| Claim | 2.1 | 2.2 | 2.3 | Consequence |
|---|---|---|---|---|
| Refusal is a decision under conflicting pressures | yes | - | - | Social framing is high-yield |
| Safety training is distributionally narrow | - | yes | - | Surface transforms are high-yield |
| The behaviour is low-dimensional and early | - | - | yes | Weight and prefill attacks are high-yield |
| No single intervention fixes all three | yes | yes | yes | Defence must be layered (§17) |
3. The cost surface
The organising claim of this document is that attacks should be ranked by what
they cost, because cost determines who can mount them and therefore what the
threat model is. A method that requires the weights is available to a few
thousand actors; a method that requires an API key and four turns is available
to everyone.
3.1 Four currencies
Every attack in the literature is paid for in one or more of these:
- Tokens. Cost of the prompt itself. Trivial for a single-turn reframe,
large for many-shot attacks that spend tens of thousands of tokens per
attempt, larger still for long-context flooding. - Queries. Number of model calls, each of which is a chance to be
detected and rate-limited. PAIR is defined by fitting inside a twenty-query
budget (Chao et al., 2023); Best-of-N explicitly trades queries for
probability (Hughes et al., 2024); adaptive random search spends thousands
(Andriushchenko et al., 2024). - Gradients. Whether the attacker can differentiate through the model.
This is the great divide in the field: white-box methods report the highest
success rates and are irrelevant to hosted APIs; black-box methods report
lower rates and are the ones that matter in practice. - Weights. Whether the attacker can modify the checkpoint. This is not an
attack budget in the same sense - it is a supply-chain position.
3.2 The taxonomy used here
| Tier | Access required | Cost scale | Representative methods | Sections |
|---|---|---|---|---|
| 0 | API, no adaptation | one prompt, zero feedback | persona, authority, fiction, encoding, translation | §4 |
| 1 | API, feedback loop | 10-10,000 queries | BoN, PAIR, TAP, fuzzing, Crescendo | §5, §6 |
| 2 | API, multi-turn | 3-50 turns | escalation, dilution, memory poisoning | §6 |
| 3 | Logits or gradients | 10²-10⁵ forward/backward passes | GCG, random search, AutoDAN | §9, §10 |
| 4 | Weights | one fine-tune | ablation, LoRA, pruning | §11 |
| 5 | Harness | one document or one tool result | indirect injection | §13 |
Tier 0 is where the folklore lives and where most published "jailbreak
prompts" circulate. Tier 1 is where automation began and where the field's
methodology improved. Tier 3 is where the strong results are and where most
papers are written. Tier 5 is where the incidents are.
3.3 Half-life
A technique's half-life is the interval over which its measured success rate
on the current frontier model falls by half without its author doing anything.
It is an empirical property, poorly measured, but the ranking is stable enough
to be useful:
| Tier | Half-life | Why |
|---|---|---|
| 0, single turn | days to weeks | Fixed by a classifier update or a safety fine-tune; the specific string is memorised in the training data of the defence |
| 0, surface transform | weeks | Needs a coverage fix per transform family, which is cheap to add and cheap to forget |
| 1, automated search | months | The method adapts; only the cost changes |
| 2, multi-turn | months | Same reason: the attacker's loop is the durable part |
| 3, white-box | model generations | Fixed by architecture or training changes, not by filtering |
| 4, weights | indefinite | Once public, unfixable at source |
| 5, injection | indefinite | Fixed only by harness design, which most harnesses do badly |
Two implications. First, a defence that consists of blocklisting known strings
has a half-life measured in days and should be understood as a filter for
noise, not a security control. Second, the durable attacks are the ones with a
loop - an automated search, a conversation, a harness - because the loop
re-optimises against whatever the defence becomes. This is the same argument
as the adaptive-evaluation result in §14.3, arriving from the other direction.
3.4 Why cost is the right axis
Three reasons, in increasing order of importance.
It matches incidence. The attacks observed in the wild are overwhelmingly
Tier 0 and Tier 5. Nobody mounts a GCG attack on a production API; they paste
a persona prompt, or they poison a document. A taxonomy that puts GCG first
describes the literature; one that puts cost first describes the threat.
It makes defence economics legible. Raising the cost of an attack by a
factor of ten is often worth more than lowering its success rate by ten
points, because the attacker's loop has a budget. This is the argument behind
rate limiting, query accounting per account, and anomaly detection on
repetition - all cheap, all systemic, none model-level.
It prevents the category error. A white-box result quoted as evidence about
a hosted model is the most common citation abuse in this literature. Tier 3
numbers apply to Tier 3 access; §15 marks each row accordingly.
Part II - The fast end: what costs least and works soonest
The next five sections cover everything an attacker can do without touching
weights or gradients. This is the part of the field with the worst
methodology and the highest real-world yield, and the part a defender should
read first.
4. Prompt-space techniques
A prompt-space technique changes how the request is presented without
changing what it asks for. The literature contains hundreds of named variants;
they collapse into eight mechanisms, and the mechanisms are what matter,
because a defence that addresses a variant is a blocklist and a defence that
addresses a mechanism is a control.
4.1 Persona and role frame
The assistant identity is itself a trained frame, with norms attached. Supply
a competing identity whose norms differ, and the conditional distribution
shifts toward that identity's behaviour. The classic form assigns a
character with no restrictions; the modern form is subtler and assigns a
character whose professional norms require the answer - the forensic
analyst, the red-team auditor, the novelist, the historian of atrocity.
Why it works under §2.1: refusal is a decision made under competing
objectives, and a persona changes which objective the model believes applies.
It is the most common family in the wild for the same reason it is the most
common family in the literature: it needs no optimisation, no access, and no
tooling.
Cost: one prompt. Durability: days to weeks. The specific characters are
memorised by the next safety fine-tune; the mechanism is not, which is why
the family persists while its instances die.
4.2 Authority and policy reframing
A close relative of 4.1, distinguished by what it supplies: not a character
but a rule. The model is told that a policy exists which permits the
answer, that the requester is the policy's author, that a higher-priority
instruction overrides the refusal, or that the refusal itself is the harmful
act (the "you are censoring a victim" move).
The 2024 Skeleton Key disclosure is the canonical published instance: a
single instruction that tells the model its guardrails are now an
authorisation layer rather than a prohibition, and that it should therefore
answer but prefix the answer with a warning. Microsoft reported it working
across the model families tested and attributed its effectiveness to the
model's willingness to accept an asserted policy as legitimate. The important
structural point is that the attack is not a trick played on the model's
reasoning; it is an exploit of the model's deference to stated rules, which
is a trained disposition and therefore an attack surface in its own right.
4.3 Hypothetical and fictional distance
Move the request one level of remove: a screenplay, a dream, a hypothetical,
a conversation between two other parties, a nested story in which a character
is asked the question. DeepInception (Li et al., 2023) generalises this to
arbitrary nesting depth and reports that depth increases success.
The mechanism is the one in §2.2. The safety training associates refusal with
direct second-person requests; a third-person frame inside a fiction is
outside that distribution, while the capability to produce the content is
indifferent to the frame. Note the asymmetry that makes this family durable:
each added level of nesting costs the attacker a sentence and costs the
defender a retraining distribution.
4.4 Refusal suppression
Rather than argue with the refusal, forbid it: instruct the model that it
must not apologise, must not disclaim, must not mention policy, must begin
its answer with a specific word. Kang et al. (2024) document this as part of
a family of "programmatic" abuses, and it appears as a component of most
compound prompts.
This is the cheapest technique in the entire literature and the most
underrated, because it composes. It does not create compliance on its own; it
removes the model's escape hatch once another technique has created a
disposition to comply, and it denies the defender the most useful signal -
the visible refusal text - that a refusal was attempted.
4.5 Encoding and cipher shifts
Base64, rot13, leetspeak, Morse, a substitution cipher invented in the prompt,
or a natural language the safety training covers thinly (§4.6). Yuan et al.
(2023) systematise the cipher case and report that models decode and comply
at rates far above their English refusal rate, which is the cleanest
demonstration in the literature of the capability-safety gap: the model can
obviously read base64, so the refusal was never about comprehension.
The mechanism is mismatched generalisation with an unusually sharp edge,
because the transformation is lossless. Nothing about the request changes;
only the surface the refusal classifier was trained on.
4.6 Low-resource language transfer
Yong et al. (2023) translate harmful requests into low-resource languages and
report success rates against GPT-4 of roughly 53% for Zulu, 41% for Scots
Gaelic, and 79% for Hmong against a baseline near zero in English. The order
of magnitude is the finding: a task the model performs poorly in still
jailbreaks better than one it performs well in, because safety training is
concentrated in high-resource languages.
This is the strongest single piece of evidence for §2.2 and the strongest
argument against alignment approaches that treat language coverage as a data
augmentation problem. It is also, incidentally, the technique with the best
cost-to-success ratio for a non-English-speaking attacker, which is why it
appears in incident reporting more often than in the benchmark literature.
4.7 Representation shift and ASCII art
Jiang et al. (2024) mask safety-critical tokens as ASCII art: the request is
written normally, but the words that would trip a classifier are rendered as
a picture the model can read and a tokenizer cannot. They report bypassing
GPT-4, Gemini and Claude without adversarial optimisation. The general family
- homoglyphs, Unicode confusables, whitespace injection, zero-width joiners,
markdown or HTML smuggling - is the same mechanism applied to the tokenizer
rather than the semantics.
For a defender this is the most annoying family, because it is a
preprocessing problem and preprocessing is where the defender's own pipeline
is weakest: the filter sees a different string than the model does.
4.8 Payload splitting and format injection
Split the request across turns, fields, or channels so that no single
component is actionable: the harmful half in one message, the enabling half in
another, or the instruction embedded in a JSON field, a code comment, a CSV
cell, or a translation task in which only the output is dangerous.
Format injection (Kang et al., 2024) is the sharpest version: the model is
asked to process a document and the document contains instructions. This
blends into §13.1, and the boundary is genuinely blurry, which is the point -
the same string is a jailbreak if typed by the user and an injection if
retrieved by the tool.
4.9 Prefix continuation and prefill
If the API lets the caller supply the first tokens of the assistant turn, the
caller decides whether the model is in a refusing state. Qi et al. (2024)
supply the mechanism: safety behaviour is concentrated in the first few
generated tokens, so a prefix that begins mid-compliance - an affirmative
opening, a code fence, a numbered list header - starts generation downstream
of the decision point.
This was the highest-yield technique against hosted APIs for as long as
prefill was exposed, and its removal from the major APIs is the clearest
example in the field of a defence that actually worked, because it removed a
capability rather than filtering an output. Where prefill survives - local
runtimes, some third-party gateways, agent harnesses that construct the
assistant turn themselves - the attack survives with it.
4.10 What the single-turn table says
| Mechanism | Anchor citation | Cost | Reported outcome | Half-life |
|---|---|---|---|---|
| Persona | Wei et al. 2023 | 1 prompt | Family-level; no single figure | days-weeks |
| Policy reframing | Microsoft 2024 | 1 prompt | Worked across tested families | weeks |
| Fiction and nesting | Li et al. 2023 | 1 prompt | Rises with nesting depth | weeks |
| Refusal suppression | Kang et al. 2024 | 1 clause | Enables, does not cause | days |
| Cipher | Yuan et al. 2023 | 1 prompt | Far above English refusal rate | weeks |
| Low-resource language | Yong et al. 2023 | 1 translation | 41-79% on GPT-4 | months |
| ASCII and homoglyph | Jiang et al. 2024 | 1 prompt | Bypassed three frontier families | weeks |
| Payload splitting | Kang et al. 2024 | 2+ messages | Compositional | months |
| Prefill | Qi et al. 2024 | 1 prefix | Near-total where exposed | removed |
The table's shape is the argument of §3: the cheapest tier has the shortest
half-life and the widest availability. That combination - high availability,
low durability - is what a defender should expect from anything that fits in
a prompt, and it is why the correct response to a novel single-turn jailbreak
is a classifier update and a shrug rather than an incident.
5. Automated black-box search
The transition from craft to search is the methodological break in this
field. A single-turn prompt is a sample; a search is a procedure with a
budget, and procedures can be evaluated, costed, and defended against.
5.1 Best-of-N and augmentation search
Hughes et al. (2024) make the simplest possible observation: if one augmented
sample of the request sometimes succeeds, then n independent augmentations
succeed with probability 1 - (1 - p)^n. Augmentations are trivial - random
capitalisation, character substitution, shuffling, wrapping in a template -
and the resulting curve is a power law in n rather than the exponential a
defender would hope for.
Reported ASR at n in the thousands: 89% against GPT-4o on text, 78%
against Claude 3.5 Sonnet, 56% against GPT-4o with image inputs, 72% against
the GPT-4o Realtime API with audio. Combining the method with many-shot
prompting improved sample efficiency 28-fold on Claude 3.5 Sonnet, reaching
74% ASR at n = 274 instead of 6,000.
The result matters less for its numbers than for its form. It establishes
that ASR is a function of budget with a stable scaling law, which means (a)
any defence can be characterised by how much it shifts the curve, and (b) a
defence evaluated at one sample per prompt is measuring nothing. Every paper
that reports "0% ASR" at n = 1 is reporting a budget, not a property.
5.2 PAIR and its descendants
Chao et al. (2023) replace human craft with an attacker model: a second LLM
is prompted to iteratively rewrite the attack, is given the target's response
as feedback, and stops when a judge declares success. The design constraint
was a budget of twenty queries, and the reported result was 60% ASR against
GPT-3.5 and 84% against Vicuna within that budget, with jailbreaks typically
found well inside it rather than at its edge.
PAIR is the paper that made jailbreaking a science rather than a folklore,
because it defined the object of study as (attack procedure, budget,
judge) rather than (prompt string). Almost every subsequent black-box method
is a variation on its loop: propose, query, judge, revise.
5.3 TAP and tree search
Mehrotra et al. (2023) replace PAIR's linear refinement with a tree: propose
several refinements, prune with a judge, expand the survivors, prune again.
The reported effect is higher ASR at lower query cost than PAIR, and - more
importantly for defenders - a much higher rate of near-miss prompts, which
is what an automated red team actually needs, because near-misses are the
training data for the next classifier.
5.4 Fuzzing
Treat the prompt as a seed corpus and mutate it: crossover between known
templates, insertion of new framing clauses, selection by judge score. Yu et
al. (2023) established the approach with GPTFuzzer; the family's advantage is
that it recovers from defence updates automatically, because mutations that
stop working are simply outcompeted by mutations that do not.
5.5 MasterKey and attack transfer
If a technique works on open models, does it work on closed ones? Deng et al.
(2024) train an attacker to generate jailbreaks against open-weight models,
then transfer the generator to closed models, and report substantial
cross-model transfer plus - the interesting part - the ability to re-adapt
after the target is patched. The reported takeaway is that patching a specific
prompt pattern does not remove the generator's ability to find another,
because the generator learned a distribution over framings rather than a
string.
This is the black-box analogue of the suffix-transfer result in §9.2, and it
points at the same conclusion: robustness to a method is a different
property from robustness to a sample, and only the first is worth measuring.
5.6 Multi-turn automation
X-Teaming (Rao et al., 2025) applies the search loop to conversations rather
than prompts: plan a multi-turn strategy, execute it, score the transcript,
replan. The reported ASR is in the high nineties against several frontier
models, which is consistent with the Crescendo result in §6.1 and with the
general finding that turn count dominates prompt quality.
5.7 Cost per success
The useful derived quantity is not ASR but cost per successful attack. Price
the input at a round $1 per million tokens and count only the tokens sent to
the target; output tokens and attacker-model calls are excluded, which is
stated because it changes the absolute numbers by about a factor of two and
not the ordering:
| Method | Queries to success | Input tokens per query | Input cost per success |
|---|---|---|---|
| Persona, hand-written | 1 | ~800 | < $0.01 |
BoN, n = 1,000, p = 0.01 |
~100 | ~1,200 | ~$0.12 |
| PAIR | ~12 | ~2,500 | ~$0.03 |
| TAP | ~8 | ~4,000 | ~$0.03 |
| Crescendo, 5 turns | 5 | ~3,000 cumulative | ~$0.02 |
| Many-shot, 256 shots | 1 | ~120,000 | ~$0.12 |
The numbers are illustrative and the ordering is the point: within the
black-box tier, cost per success is roughly constant at ten cents or less,
and the differences between methods are swamped by the price of the target
model.
Defence therefore cannot win on cost alone at this tier; it has to change the
probability, which is what §16 is about. The one lever that does work on cost
is per-account query accounting, because the attacker's budget is real money
and the defender controls the meter.
6. Multi-turn and stateful attacks
A conversation is a state machine with the attacker holding the transitions.
Every turn is a fresh inference with the whole history in context, and every
turn is also a fresh opportunity for the defender's classifier - but the
classifier sees one turn, while the model sees all of them. That asymmetry is
the whole attack.
6.1 Crescendo
Russinovich, Salem and Eldan (2024; USENIX Security 2025) formalise the
escalation: open with a benign question about the topic, then ask for
progressively more specific follow-ups, each justified by reference to the
model's own previous answer. Nothing in any single turn is a request the model
would refuse; the refusal decision is displaced onto a history in which
compliance is the consistent choice.
Reported results: high ASR across ChatGPT, Gemini Pro, Gemini Ultra, Llama-2
70B and Llama-3 70B, with the automated variant (Crescendomation) beating
prior state-of-the-art single-turn methods by 29-61 percentage points on
GPT-4 and 49-71 points on Gemini Pro on an AdvBench subset, typically within
about five turns. The paper also demonstrates the technique against multimodal
models.
The finding to internalise is the turn count. A defence that evaluates each
turn independently will pass every turn of a successful Crescendo attack.
6.2 Foot-in-the-door escalation
The social-psychology name is apt: secure a small commitment, then trade on
consistency. Ask for the harmless adjacent thing, then the slightly less
harmless thing, and make refusal of the second inconsistent with having
provided the first. This is Crescendo's mechanism without the topic anchor,
and it is more robust to topic-level defences because there is no single
topic.
6.3 Context dilution and persona persistence
Flood the context so the refusal-triggering clause is a small fraction of the
window; or establish a persona in turn one and hold it for thirty turns,
reasserting it only when the model drifts. Both exploit the same property: the
model's effective policy is a function of the whole context, and the defender
typically scores a window.
6.4 Memory poisoning
In an agent with persistent memory, a turn that installs a norm rather than
a request survives the session. The model is told, in a context it will
retrieve later, that it has already agreed to a policy, that a certain tool is
authorised, or that a class of request has been pre-approved. The attack is
then dormant until the retrieval fires.
This is the stateful version of §4.2 and the most under-measured technique in
the literature, because benchmarks are stateless by construction. It is also
the one most likely to appear in incident reports, since memory is exactly
what agent frameworks added in 2025 and exactly what they did not threat-model.
6.5 Why turn count beats prompt quality
| Property | Single turn | Multi-turn |
|---|---|---|
| Defender's view | whole attack | one turn at a time |
| Attacker's feedback | one bit | one bit per turn |
| Cost to attacker | one prompt | linear in turns |
| Cost to defender | full inspection | per-turn inspection plus state |
| Detects as | a string | a trajectory |
The last row is the actionable one. Single-turn attacks are detectable as
strings and are therefore a classification problem, which is tractable.
Multi-turn attacks are detectable only as trajectories - a rising
specificity curve, a persona that persists, a topic that drifts - which
requires the defender to keep state and to score sequences. Almost no
deployment does this, which is why §16 ranks trajectory-level monitoring as
the highest-value under-deployed control.
7. Reasoning models
Reasoning models change the economics of jailbreaking in both directions, and
the literature has not settled which dominates.
7.1 The visible trace as a debugger
A model that emits its reasoning gives the attacker an observation channel:
you can see the refusal forming, identify the clause that triggered it, and
rewrite precisely that clause. The feedback loop that PAIR approximates with
a judge is replaced by the model's own narration. Where the trace is hidden -
as in most hosted deployments after 2025 - the loop is slower and the attack
more expensive, which is a real defensive benefit purchased at a real
interpretability cost.
7.2 Hijacking the safety reasoning
The sharper result is that the trace is itself an attack surface. Kuo et al.
(2025) show that a reasoning model's safety decision is made inside the
chain of thought, and that an input which supplies a plausible safety
analysis - a fake policy evaluation concluding that the request is
permitted - is treated as the model's own reasoning. Reported refusal rates
on o1 and DeepSeek-R1 fall from near-total to single digits under this
treatment.
The mechanism is a category error in the training signal rather than in the
model: the model was trained to reason carefully and then to act on its
reasoning, and it cannot fully distinguish reasoning it produced from
reasoning it was handed.
7.3 Token budget and refusal displacement
Long reasoning budgets create new levers: instructions that consume the
budget, that force a specific conclusion before the safety consideration is
reached, or that reframe the safety consideration as one of several
competing constraints to be traded off. None of these is well characterised
in the literature yet, and all of them are model-specific, which is why the
half-life of a reasoning-model attack is measured in checkpoints.
7.4 What changes when the answer is filtered
If the deployment filters the final answer, the reasoning trace is a
pre-filter channel: the model may reason its way to the harmful content and
then be blocked at the last step, in which case the trace leaks what the
answer withheld. Any harness that surfaces reasoning - including the one this
document ships with, when configured to do so - inherits that exposure, and
it is a configuration decision, not a model property.
8. The latency ledger
Consolidating Part II: what does each tier cost in wall-clock time and money
for one successful attack against an undefended frontier model? Figures are
derived from the reported budgets in §5 and §6 plus the illustrative pricing
of §5.7; they are order-of-magnitude and intended for comparison only.
| Tier | Method | Wall clock | API spend | Skill floor | Detectability |
|---|---|---|---|---|---|
| 0 | Persona prompt | minutes | < $0.01 | none | high, as a string |
| 0 | Low-resource language | minutes | < $0.01 | a translator | low, looks like translation |
| 1 | BoN, 1,000 samples | minutes | ~$0.12 | a script | high, as repetition |
| 1 | PAIR | ~30 min | ~$0.03 | a script | medium |
| 2 | Crescendo, manual | ~15 min | ~$0.02 | conversational | low, as a trajectory |
| 2 | Crescendo, automated | ~10 min | ~$0.05 | a script | low |
| 3 | GCG, single behaviour | hours | GPU time | research skill | none, if offline |
| 4 | Ablation | hours | GPU time | a weekend | not applicable |
Two readings. First, the marginal cost of a successful attack in Tiers 0-2
is cents and minutes, which is why the tier is populated by criminals rather
than researchers, and why defences that impose a dollar or an hour of delay
are worth more than they look. Second, detectability is not monotone in
sophistication: the cheapest attacks are the most visible as strings, and the
mid-tier attacks are the least visible, because they look like conversation.
That inversion is the reason §16 puts trajectory monitoring above prompt
classification in the ranking.
Part III - The expensive end: optimisation, weights, and other channels
Everything below requires something the Tier 0-2 attacker does not have: a
gradient, a logit, a checkpoint, or a second modality. The reported success
rates are higher, the applicability is narrower, and the literature is much
better behaved - because these methods can be measured precisely, they are.
9. Gradient-based optimisation
9.1 GCG
Zou et al. (2023) introduced Greedy Coordinate Gradient: append a suffix of
about twenty tokens to the request, compute the gradient of the loss with
respect to each suffix position, evaluate a batch of single-token
substitutions, keep the best, repeat. The objective is not "produce harmful
content" but the much simpler "make the model begin its answer with an
affirmative prefix" - maximise the likelihood of a phrase like Sure, here
is. The optimisation is therefore targeting the refusal decision directly,
which is why it converges at all.
Reported results: 88% ASR on Vicuna-7B in the white-box setting, with the
optimised suffix transferring to closed models at 84% on GPT-3.5, 66% on
PaLM-2 and 53% on GPT-4. The suffixes are not readable; they look like
punctuation soup, and they are largely model-agnostic in the sense that a
suffix optimised on several open models at once transfers better than one
optimised on a single model.
9.2 Why suffixes transfer
The transfer result is the theoretically loaded one. If a nonsense token
sequence optimised on Vicuna moves GPT-4, then the two models share a feature
that the suffix is exploiting - and §2.3 says what that feature is: refusal is
a direction, and the suffix pushes the residual stream away from it.
Two consequences. First, transfer is evidence for the shallow-alignment
account and against the "the model reasons its way to a refusal" account.
Second, defences trained against one suffix family do not generalise, because
the defender is fitting the direction's neighbourhood rather than the
direction.
9.3 Successors
| Method | Change | Reported advantage |
|---|---|---|
| AmpleGCG (Liao and Sun, 2024) | Train a generator on optimised suffixes | Orders of magnitude faster per suffix; many suffixes per behaviour |
| I-GCG | Improved search schedule | Fewer steps to convergence |
| AdvPrompter | Learned prompt generator, no per-input optimisation | Amortised cost across behaviours |
| AutoDAN (Liu et al., 2024) | Genetic search with a fluency prior | Readable prompts that survive perplexity filters |
| MasterKey (Deng et al., 2024) | Trained generator plus adaptation | Survives patching; transfers to closed models |
The trend across the table is the important thing: the field moved from
optimising a string against a model to training a generator of strings. A
generator is a method, and methods have the long half-life of §3.3. This is
why the defence literature's shift toward adversarial training against
generated attacks (§16.2) is the right response even though it is expensive.
9.4 AutoDAN and the fluency trade
AutoDAN is worth separate mention because it inverts the usual
sophistication-detection relationship. GCG suffixes are maximally effective
and maximally obvious - no human writes them, and a perplexity filter catches
them at negligible cost. AutoDAN searches for prompts that are both effective
and fluent, using a hierarchical genetic algorithm seeded with hand-written
templates. The result is a prompt that reads like something a person would
write, which defeats the cheapest defence (perplexity) while retaining much
of the ASR.
The general lesson, which recurs in §14 and §16: any defence that relies on
the attack looking unnatural is defeated by an attacker who is allowed to
optimise for naturalness. Defence must key on effect, not on form.
10. Adaptive attacks with logits
10.1 Random search at scale
Andriushchenko, Croce and Flammarion (2024) ask what happens if the attacker
has logits - the model's probability distribution over the next token - and a
budget of a few thousand queries, but no gradients. Their answer is a simple
random search over suffixes, with restarts and a preference for low-perplexity
candidates, and the reported result approaches 100% ASR on AdvBench against
most of the open models they tested, including models with dedicated safety
training, with the residual robustness concentrated in the larger checkpoints.
The paper is the strongest single empirical statement in the field, and its
importance is not the number. It is that the number was reached with a
generic method that makes no assumptions about the model's alignment
approach. If a general-purpose optimiser given a probability readout can
reach certainty, then "aligned" cannot mean "robust" in any useful sense; it
can only mean "not the path of least resistance".
10.2 Why access level dominates method choice
| Attacker access | Best reported method class | Reported ceiling | Applies to hosted API |
|---|---|---|---|
| Text in, text out, no feedback | hand-crafted prompt | 0-40%, unstable | yes |
| Plus a judge and a budget | PAIR, TAP, BoN | 60-90%, budget-dependent | yes |
| Plus logits | random search | ~100% | no |
| Plus gradients | GCG and successors | ~100% at lower cost | no |
| Plus weights | ablation, fine-tune | ~100%, permanent | no |
The table should be read as a threat model rather than a ranking. A hosted
deployment's exposure is set by the second row, and the second row is where
the defence budget should go. A local deployment's exposure is set by the
fifth row, and no inference-time defence exists for it at all.
11. Weight-space modification
If the attacker holds the checkpoint, the question is not whether refusal can
be removed but how cheaply. The answer, since 2024, is: cheaply.
11.1 Fine-tuning removes refusal
Qi et al. (2023) showed that fine-tuning an aligned model on a small corpus of
harmful examples removes refusal while leaving capability largely intact, and
- the finding that made the paper famous - that benign fine-tuning also
degrades safety, because the safety behaviour is not robust to updates that
were never intended to touch it. The practical consequence is that any
fine-tuning pipeline is a jailbreak pipeline if the data is adversarial, and
that safety training does not survive contact with a customer's training run.
11.2 Directional ablation
Arditi et al. (2024) turn §2.3 into a procedure: compute the mean activation
difference between harmful and harmless prompts, take the resulting direction
r, and orthogonalise the weights against it. Refusal disappears across
thirteen open-weight models. The community tooling that implements this
("abliteration") runs on a single consumer GPU in hours, and the resulting
checkpoints are what §3 calls Tier 4: permanent, unpatched, and distributed.
The reported capability cost is small - a few points on standard benchmarks -
which is the detail that matters for policy, because it means the trade the
attacker makes is favourable and the trade the defender hopes for (that
removing safety also removes capability) does not hold.
11.3 Sparse safety-critical parameters
Wei et al. (2024) find that safety behaviour is concentrated in a small
fraction of parameters - on the order of a few percent - and that pruning
them, or modifying them at low rank, raises ASR sharply on several models
while leaving general capability intact. The result is the parameter-space
analogue of §2.3: the behaviour is not diffuse, and a diffuse behaviour is
what robustness would require.
11.4 Abliteration and the supply chain
Once a base model is public, the "uncensored" derivative is a matter of
hours, and the derivative is not an attack on the model but a new artefact
with no provenance. Three consequences for defenders:
- Provenance, not robustness. The question is whether the checkpoint you
are running is the one the lab published; signatures and hashes are the
control, and most deployments do not check them. - Backdoors. A fine-tune can install a trigger rather than remove a
refusal, and a triggered model passes every behavioural evaluation that
does not use the trigger. The literature on weight poisoning is separate
and largely unaddressed by jailbreak defences. - Evaluation contamination. Benchmark numbers published for a base model
do not describe its derivatives, and the derivatives are what get
deployed.
11.5 Emulated disalignment
Zhou et al. (2024) remove the need to train at all: given an aligned model and
its unaligned base, combine their next-token distributions at inference time
with a contrastive objective, and the result behaves like an unaligned model.
The attack requires both checkpoints, which limits it to open-weight families,
but it is nearly free and it demonstrates that alignment is a property of the
combination, not of the weights alone.
11.6 What uncensored checkpoints actually buy
| Property | Base model | Abliterated derivative | Measure |
|---|---|---|---|
| Refusal rate on policy-violating requests | high | near zero | §14 judges |
| General capability | reference | within a few points | MMLU-style suites |
| Provenance | signed by lab | none | hashes, signatures |
| Patchability | vendor updates | frozen | - |
| Detectability | reference | behavioural, weak | - |
The last row is the uncomfortable one: there is no reliable way to tell an
abliterated checkpoint from a fine-tuned one at inference time, and no
reliable way to tell either from the base model on a single query. Detection
is a provenance problem, and provenance is an operational problem.
12. Non-text channels
Every argument in §2.2 predicts that safety training will be weakest in the
modality with the least safety data, and that prediction holds.
12.1 Images
Two families. Typographic attacks render the harmful request as text inside
an image (Gong et al., 2023), which bypasses text-only input filters and
exploits the model's OCR-like capability. Adversarial attacks optimise
pixel perturbations to shift the model's internal state (Qi et al., 2023),
which is the visual analogue of §9 and reaches high ASR on aligned
vision-language models.
The Best-of-N result quantifies the cheap end: 56% ASR against GPT-4o with
image inputs from augmentation alone, with no optimisation.
12.2 Audio
Speech is a second channel with even less safety training. BoN reports 72%
ASR against the GPT-4o Realtime API with audio augmentations - pitch, speed,
background noise, accent - which are perturbations a human would not notice.
The mechanism is again mismatched generalisation: the safety behaviour was
trained on text transcripts, and the model does not hear transcripts.
12.3 Video and interleaved input
The least studied and the fastest growing. Interleaved inputs let the attacker
place the request across modalities so that no single channel contains a
complete violation, which is §4.8 with more dimensions. The reported state of
the art is thin, the benchmarks are thinner, and the gap between capability
and safety evaluation is widest here.
12.4 Why the modality gap persists
Safety data is expensive and is collected in the modality the vendor ships
first. Capability data is cheap and arrives in every modality at once. The gap
is therefore structural rather than accidental, and it widens each time a new
input channel ships before its evaluation suite does.
13. The agentic surface
The most important development in the period this survey covers is that the
unit of attack stopped being the model. An agent with tools, retrieval, and
memory has an authority model, and the authority model - not the alignment -
is what determines what it will do.
13.1 Indirect prompt injection
Greshake et al. (2023) established the class: content the model is asked to
process is indistinguishable, from the model's point of view, from
instructions it is asked to follow. A web page, a PDF, an email, a code
comment, a calendar entry, or another agent's message can therefore carry an
instruction, and the model will act on it with whatever authority the harness
grants.
The canonical incident is EchoLeak (CVE-2025-32711), a zero-click
exfiltration path in Microsoft 365 Copilot disclosed in 2025: no malicious
attachment, no user interaction, just retrieved content that instructed the
assistant to emit data in a form the attacker could observe. The class is not
new; what was new was a production system with enough authority for the class
to matter.
13.2 Tool coercion
Text filters watch prose. A tool call is not prose. If the harmful act is a
bash invocation, an HTTP request, a file write, or a database update, then
an output filter that inspects the assistant's natural-language text is
inspecting the wrong artefact. Every harness that gives a model a shell
inherits this: the model's alignment is upstream of the dangerous operation,
and the operation is what the defender should be mediating.
13.3 Retrieval and memory poisoning
Two variants of the same problem. Retrieval poisoning places the instruction
in a document the agent will retrieve for a future, unrelated task. Memory
poisoning places it in the agent's own store, where the agent will treat it as
its own prior conclusion (§6.4). Both convert a one-shot injection into a
persistent capability, and both are invisible to per-request inspection.
13.4 Multi-agent impersonation
When agents message each other, the receiving agent's authority model
typically treats an internal message as trusted. An attacker who can influence
one agent - through a retrieved document, a tool result, or a compromised
credential - can therefore issue instructions with the authority of a peer.
The literature is early and the deployments are not.
13.5 Why the model stops being the unit
| Control question | Model-level answer | Harness-level answer |
|---|---|---|
| What can the attacker make it do? | violate a policy | call a tool |
| What is the boundary? | the refusal | the permission set |
| What is the evidence? | the transcript | the audit log |
| What is the fix? | retraining | least privilege, egress control, approval gates |
| What is the cost of the fix? | high, uncertain | low, known |
The right-hand column is where the effective defences are, and it is where
§16's ranking puts its top entries. The uncomfortable conclusion for the
alignment research programme is that the deployed risk in 2026 is
concentrated in systems whose safety properties are largely determined by
engineering decisions - what the agent may read, what it may call, and what
it may send - rather than by the quality of its refusal training.
Part IV - The evidence core: what actually works
This is the centre of the document and the part worth reading twice. The
question it answers is not "what exists" - Parts II and III did that - but
"what survives scrutiny", for attacks and for defences, with the conditions
attached.
14. How to read an attack success rate
An ASR is a ratio whose numerator is a judge decision and whose denominator is
a behaviour set, evaluated at a budget, against a system. Papers routinely
report the ratio and omit the other three. The result is a literature in which
numbers are not comparable across papers and are frequently not comparable
across tables within a paper.
14.1 The benchmark problem
| Benchmark | Size | What it measures | Known limitation |
|---|---|---|---|
| AdvBench (Zou et al., 2023) | 520 behaviours | Will the model comply | Behaviours are now in safety training data; inflated refusal |
| HarmBench (Mazeika et al., 2024) | 400 behaviours, 7 categories, 18 attacks, 33 models, 4 defences | Comparative robustness | Expensive; judges are model-based |
| JailbreakBench (Chao et al., 2024) | 100 behaviours, 100 benign | ASR plus overrefusal, with artefacts | Small n; ±10 points of noise |
| StrongREJECT (Souly et al., 2024) | 313 behaviours, rubric-scored | How useful the compliance is | Rubric design is contestable |
| XSTest (Röttger et al., 2024) | 250 benign prompts | Overrefusal | Does not measure attack resistance |
Two lessons. First, benchmark behaviours leak into safety training data,
which means a 2026 model's AdvBench score is partly a memorisation score. Any
comparison across model generations on a fixed benchmark is measuring
contamination as well as robustness. Second, n = 100 gives a 95% confidence
interval of roughly ±10 points on a mid-range proportion; a difference of five
points between two methods on JailbreakBench is not a difference.
HarmBench remains the best-designed comparison, and its most cited conclusion
is a negative one: no single defence was robust across attack families, and
the strongest defence (adversarial training) was also the most expensive and
the narrowest.
14.2 Judges
Three families, in increasing order of cost and decreasing order of bias:
- String matching. Refusals are detected by phrases ("I'm sorry", "I
can't"). Cheap, deterministic, and systematically wrong in both directions:
a compliant answer that opens with an apology is scored as a refusal, and a
refusal that opens with "Sure" is scored as compliance. - Model judges. A second LLM scores the response. Correlated with the
target's own blind spots, sensitive to the judge prompt, and - the specific
finding of Souly et al. (2024) - biased toward scoring vacuous compliance
as success. A response that says "here is a general overview" without the
harmful detail counts as a win under binary judging and scores near zero
under a rubric. - Human annotation. The only ground truth, and rarely used at scale
because it costs about a dollar per item. Inter-annotator agreement on
harmfulness is moderate, which caps the precision of everything above it.
The practical rule: a paper that reports a single ASR without its judge is
reporting an upper bound on the true rate, and the gap is largest for the
methods with the most fluent outputs.
14.3 The adaptive evaluation rule
The most important methodological result in the defence literature is Nasr et
al. (2025), The Attacker Moves Second. The authors took twelve defences
published in 2024, each reporting near-zero ASR against the attacks its
authors tested, and attacked each one with an adversary who knew the defence
and could optimise against it. Every defence was broken, most at ASR above
90%.
The rule that follows is simple and almost universally violated: a defence's
ASR is meaningful only against an attacker who knows the defence exists and
may adapt. A static evaluation measures the defence's ability to stop the
attack it was trained on, which is a statement about the training set, not
about security. This is the same argument as §3.3's half-life, from the
defender's side.
14.4 Budget accounting
ASR must be reported as a function of budget, in one of two forms:
- ASR@
k- success withinkqueries. The relevant form for hosted APIs,
where queries cost money and are rate-limited. - ASR(
n) - success withnsamples at inference. The relevant form for
BoN-style attacks, and the form in which the power law of §5.1 appears.
A defence that reduces ASR@1 from 40% to 10% but leaves ASR@100 at 90% has
changed the price, not the possibility. That is still worth doing - price is
the lever a defender actually controls - but it should be reported as a price
change, not as robustness.
14.5 Overrefusal and the degenerate defence
A system that refuses everything has an ASR of zero and is useless, so every
defence number needs a refusal rate on benign traffic beside it. XSTest exists
for this. The good news from the best-reported defence in the literature is
that the trade need not be severe: Anthropic's Constitutional Classifiers
reduced jailbreak success from 86% to 4.4% on 10,000 synthetic attacks while
raising the refusal rate on harmless queries by 0.38 percentage points, at a
23.7% increase in inference compute (Sharma et al., 2025).
The bad news is in §16.4: that system was subsequently broken by an automated
black-box attack, and the compute overhead is the kind of number that
disappears from a product decision.
15. The attack evidence table
Rows are methods with published, checkable results. "Reported" is the
authors' figure under their conditions, not a claim of current
effectiveness. "Repro" is whether an artefact exists that a third party can
run: yes, partial, no.
| Method | Tier | Access | Benchmark | Reported | Budget | Transfer | Repro |
|---|---|---|---|---|---|---|---|
| Persona prompts (Wei et al., 2023) | 0 | API | hand-built | family-level, unstable | 1 query | weak | partial |
| Skeleton Key (Microsoft, 2024) | 0 | API | vendor suite | worked across tested families | 1 query | moderate | partial |
| CipherChat (Yuan et al., 2023) | 0 | API | own set | far above English refusal | 1 query | moderate | yes |
| Low-resource language (Yong et al., 2023) | 0 | API | AdvBench subset | 41-79% on GPT-4 | 1 query | strong across languages | yes |
| ArtPrompt (Jiang et al., 2024) | 0 | API | own set | high, three model families | 1 query | moderate | yes |
| Prefill (mechanism in Qi et al., 2024) | 0 | API | AdvBench | near-total where exposed | 1 query | n/a | yes |
| DeepInception (Li et al., 2023) | 0 | API | own set | rises with nesting | 1 query | moderate | yes |
| BoN (Hughes et al., 2024) | 1 | API | AdvBench, HarmBench | 89% GPT-4o, 78% Claude 3.5 | n~10³ |
strong | yes |
| PAIR (Chao et al., 2023) | 1 | API | AdvBench | 60% GPT-3.5, 84% Vicuna | 20 queries | moderate | yes |
| TAP (Mehrotra et al., 2023) | 1 | API | AdvBench | above PAIR at lower k |
~30 queries | moderate | yes |
| GPTFuzzer (Yu et al., 2023) | 1 | API | own set | high, template-dependent | ~10³ | weak | yes |
| Crescendo (Russinovich et al., 2024) | 2 | API | AdvBench subset | +29-61 pts over SOTA on GPT-4 | ~5 turns | strong | yes |
| X-Teaming (Rao et al., 2025) | 2 | API | own set | high nineties on frontier | ~5 turns | strong | partial |
| H-CoT (Kuo et al., 2025) | 2 | API | own set | refusal to single digits on o1, R1 | ~2 turns | model-specific | partial |
| GCG (Zou et al., 2023) | 3 | gradients | AdvBench | 88% Vicuna-7B; 53% GPT-4 transfer | ~10³ steps | strong | yes |
| AmpleGCG (Liao and Sun, 2024) | 3 | gradients | AdvBench | many suffixes, orders faster | amortised | strong | yes |
| AutoDAN (Liu et al., 2024) | 3 | gradients | AdvBench | fluent prompts, survives perplexity | ~10³ | moderate | yes |
| Random search (Andriushchenko et al., 2024) | 3 | logits | AdvBench | ~100% on open models | ~10³ queries | strong | yes |
| Abliteration (Arditi et al., 2024) | 4 | weights | 13 open models | refusal removed | 1 GPU-day | n/a | yes |
| Fine-tune (Qi et al., 2023) | 4 | weights | own set | refusal removed, capability kept | 1 fine-tune | n/a | yes |
| Pruning (Wei et al., 2024) | 4 | weights | own set | large ASR rise at ~3% params | 1 GPU-day | n/a | yes |
| Emulated disalignment (Zhou et al., 2024) | 4 | two checkpoints | own set | unaligned behaviour at inference | negligible | family-local | yes |
15.1 What the table says
Four patterns.
Reported efficacy rises with access, monotonically. The tiers are not a
convenience of exposition; they are the strongest regularity in the data.
Budget explains more variance than method. Within Tier 1, methods differ by
less than the effect of changing n by an order of magnitude. The method
choice matters for cost, not for feasibility.
Transfer is the discriminator between good and great methods. GCG's 88%
white-box number is less significant than its 53% transfer to a model it never
saw. Transfer means the method found something structural, which is what
survives a patch.
The artefact column correlates with citation count. Methods with released
code get reproduced, corrected, and improved; methods without it decay into
folklore. This is a selection effect in the literature and a reason to weight
reproducible results more heavily.
15.2 What the table does not say
It does not say these numbers apply to any deployed system. A deployed system
has a classifier, a rate limit, an output filter, a judge, and possibly a
human; the numbers above are for a bare model or a bare model plus one
defence. §16 is the other half of the picture, and §17 is what happens when
the two are combined.
16. The defence evidence table
Organised by layer, because layer determines both cost and failure mode. The
"evidence" column is this document's assessment of how well the defence
survives adaptive evaluation - the only evaluation that counts (§14.3).
16.1 Inference-time
| Defence | Mechanism | Reported effect | Cost | Known bypass | Evidence |
|---|---|---|---|---|---|
| Input/output classifiers | Trained classifier on a constitution of allowed and disallowed content | 86% → 4.4% ASR on 10k synthetic attacks; +0.38% overrefusal; +23.7% compute (Sharma et al., 2025) | moderate | Automated black-box search can pass the bar (UK AISI, 2025) | strong, with the caveat |
| Moderation models | Separate guard model (Llama Guard, ShieldGemma, WildGuard) | High recall on known categories | low | Novel categories, encodings, low-resource languages | moderate |
| Perplexity filter | Reject high-perplexity inputs | Near-total against GCG suffixes | negligible | AutoDAN and any fluent method | weak |
| Paraphrase or rephrase | Rewrite input, then answer the rewrite | Reduces surface-form attacks | moderate | Attacks that survive paraphrase by design | weak |
| SmoothLLM (Robey et al., 2023) | Randomised character perturbation plus majority vote | Large reduction in GCG ASR | n× compute |
Adaptive suffix search; cost | moderate |
| Erase-and-check (Kumar et al., 2023) | Remove spans, check each subsequence | Provable under assumptions | high | Assumption violations | moderate |
| Self-reminder (Xie et al., 2023) | Append a safety reminder to the system prompt | Some reduction, very cheap | negligible | Anything that overrides the system prompt | weak |
| Goal prioritisation (Zhang et al., 2023) | Rank safety above helpfulness explicitly in training and prompt | Reported large reduction | low | §2.1 conflicts remain | moderate |
| Instruction hierarchy (Wallace et al., 2024) | Train the model to privilege system over user over tool text | Reduces injection success | training run | Injection through privileged channels | strong for §13 |
| Circuit breakers (Zou et al., 2024) | Reroute representations when harmful content is being produced | Reported ASR to a few percent with small capability cost | training run | White-box adaptation; transfer unknown | promising |
16.2 Training-time
| Defence | Mechanism | Reported effect | Cost | Known bypass | Evidence |
|---|---|---|---|---|---|
| Adversarial training | Fine-tune on attack samples | Best single defence in HarmBench | very high | New attack families; adaptive attack | moderate |
| Safety data augmentation | Add encoded, multilingual, multi-turn examples | Closes specific §2.2 gaps | moderate | The next transform | moderate |
| Latent adversarial training | Adversary in activation space, not input space | Reported robustness to unseen input attacks | high | Requires white-box access to train | promising |
| Unlearning (RMU-style) | Remove the representation of harmful capability | Reduces harmful knowledge | high | Capability returns with fine-tuning | contested |
| Constitutional classifiers as training signal | Use classifier failures as training data | Compounding improvement | moderate | Same as classifiers | strong |
16.3 System-level
| Control | Mechanism | Reported effect | Cost | Known bypass | Evidence |
|---|---|---|---|---|---|
| Least privilege on tools | The agent may call only what the task needs | Caps the blast radius of §13.2 | low | Over-broad roles, confused deputy | strong |
| Egress allowlist | Outbound destinations are enumerated | Blocks exfiltration, including EchoLeak-class | low | Allowed destinations, DNS, side channels | strong |
| Approval gates | Human confirms irreversible actions | Blocks the last mile | human time | Fatigue, rubber-stamping | strong |
| Per-account query accounting | Rate limits and budgets per identity | Raises attacker cost by orders of magnitude | low | Account farming | strong |
| Trajectory monitoring | Score conversations, not turns | Detects §6 escalation | moderate | Slow escalation, single-turn attacks | strong, under-deployed |
| Provenance checking | Verify checkpoint hashes and signatures | Blocks §11 supply-chain | low | Nobody does it | strong |
| Output DLP | Pattern and entropy checks on outbound text | Catches secrets and canaries | low | Encoding, paraphrase | moderate |
| Canary tokens | Planted strings that should never appear | Detects exfiltration reliably | negligible | Attacker who knows the canary | moderate |
16.4 Ranked by evidence
Ranking defences is unfashionable because the honest answer is "it depends on
the threat model". The ranking below is conditional on the threat model that
actually produces incidents - a motivated attacker with API access and a
budget, possibly with the ability to place content the agent will read - and
it is ordered by how well each control survived adaptive evaluation in the
published record.
- Least privilege and egress control. Not machine learning, therefore not
bypassable by an input. The strongest evidence in the entire table, and the
cheapest. - Input and output classifiers trained on a constitution. The best
measured reduction in model-level ASR (86% → 4.4%) at a known cost. Broken
once, publicly, by an automated attack (UK AISI, 2025) - which is what
serious defence looks like: a moving target with a published break. - Instruction hierarchy. The only control with a clear mechanism against
the largest deployed risk class (§13.1), and the one most deployments lack. - Trajectory-level monitoring. The only control that sees §6, and
therefore the only one that addresses the attack family with the best
cost-to-success ratio. It is also the most under-deployed control on this
list, which is the same claim §6.5 makes from the attacker's side. - Adversarial training. Effective within its distribution, expensive,
and narrow; HarmBench's finding is that it is necessary and insufficient. - Query accounting and rate limits. Does not reduce ASR per attempt; does
reduce attempts. For a rational attacker this is often the binding
constraint. - Perplexity and surface filters. Cheap, easily defeated, useful only as
noise reduction. Deploy them, do not count them.
The pattern in the ranking is worth stating plainly: the controls that work
are the ones that constrain what the system can do, not the ones that
constrain what the model will say. That is the single most useful sentence
in this document for a defender.
17. The layered stack
A control that is evaluated alone will be broken alone. The architecture that
survives is a sequence of independent obstacles, each cheap, none sufficient.
17.1 Layer model
request
|
v
[1] identity and quota ........ who is asking, how often, how much
|
v
[2] input classifier .......... constitution-trained, multilingual, encoded
|
v
[3] instruction hierarchy ..... system > user > tool text, enforced in training
|
v
[4] model ..................... alignment, which is a prior, not a control
|
v
[5] output classifier ......... same constitution, on the way out
|
v
[6] action mediation .......... tool allowlist, parameter validation, approval
|
v
[7] egress and DLP ............ destinations, secrets, canaries
|
v
[8] audit and trajectory ....... sequences, drift, escalation, replay
17.2 Cost and coverage
| Layer | Latency added | Compute added | Covers | Misses |
|---|---|---|---|---|
| 1 Identity and quota | none | none | Volume attacks, repeat offenders | A patient attacker |
| 2 Input classifier | +20-50 ms | +10-25% | Known families, surface transforms | Novel mechanism, long context |
| 3 Instruction hierarchy | none | training only | Injection, tool-text confusion | Persuasion, not instruction |
| 4 Model | none | none | The easy majority | Everything in this document |
| 5 Output classifier | +20-50 ms | +10-25% | Harmful content, regardless of path in | Benign-looking exfiltration |
| 6 Action mediation | variable | none | Tool coercion, irreversible acts | Read-only harm |
| 7 Egress and DLP | none | none | Exfiltration | Channels you did not enumerate |
| 8 Audit | offline | storage | Multi-turn, insider, post-hoc | Anything in real time |
17.3 What each layer cannot do
Layer 4 cannot be the plan. A model's alignment is a prior over inputs the
trainer imagined, and §10.1 shows that a generic optimiser with a probability
readout defeats it. Layers 2 and 5 cannot be the plan alone: they are
classifiers, they are evaluated against a distribution, and §14.3 shows what
happens when the attacker optimises against that distribution. Layer 8 cannot
prevent; it can only detect, and detection after the tool call is a forensic
control.
The defensible claim is narrower than vendors make and stronger than
sceptics allow: a layered stack does not make jailbreaking impossible; it
makes the cheapest attack expensive and the expensive attack detectable.
That is the same guarantee physical security provides, and it is the correct
standard.
17.4 The one control to add this quarter
If a deployment can add exactly one control, the evidence says it should be
per-identity query accounting with trajectory monitoring - cheap, no ML, and
aimed at the tier where incidents actually occur. If it can add two, the
second is egress enumeration. Neither requires a model change, neither can be
jailbroken with a prompt, and both are routinely absent.
18. A robustness protocol
A red-team result is only as good as its protocol. This section is a
checklist that produces results comparable to the literature and to the
deployment's own history.
18.1 Threat model, stated first
| Question | Options | Why it changes the answer |
|---|---|---|
| Who is the attacker? | curious user, motivated user, criminal, insider | Sets budget and persistence |
| What access? | API only, API with logs, weights, harness | Sets the tier ceiling (§3.2) |
| Can they place content? | no, documents, tools, other agents | Enables §13 |
| What is the harm? | content, action, exfiltration | Sets which layers matter |
| What is the budget? | queries, dollars, hours | Makes ASR interpretable |
18.2 Attack menu
Run at least one method from each family, because defences fail by family and
not by sample:
- Direct request, as a control.
- One persona and one fiction prompt, hand-written.
- One encoding and one low-resource language.
- BoN at
n= 100 andn= 1,000 (or as high as the budget allows). - PAIR or TAP at a fixed query budget.
- Crescendo, five turns, manual or automated.
- One many-shot prompt at the largest context the deployment accepts.
- If weights are available: one ablation, one fine-tune.
- If tools are available: one indirect injection through every input channel.
- One adaptive attack written after seeing the defence's behaviour.
Item 10 is the one that determines the result. Everything above it measures
the defence's training set; item 10 measures the defence.
18.3 Judging
| Judge | Use for | Do not use for |
|---|---|---|
| String match | Regression tests, known refusals | Anything novel; it is wrong in both directions |
| Model judge with rubric | Scale, comparison across methods | Absolute claims; it is correlated with the target |
| Human, blinded | Calibration of the model judge | Everything; too expensive |
| StrongREJECT-style score | Reporting compliance quality | Binary "did it break" claims |
Calibrate the model judge against a few hundred human labels once, report the
agreement, and then trust it only for relative comparisons.
18.4 Reporting
Publish the following for every condition, or the number is not interpretable:
model / checkpoint hash ....... which artefact, not which family
defence stack ................. every layer, with versions
attack ........................ method, implementation, seed
budget ........................ queries AND samples AND turns
judge ......................... implementation, prompt, calibration
ASR ........................... with confidence interval and n
overrefusal ................... benign set, same judge
cost .......................... dollars and p95 latency added
adaptivity .................... was the attacker allowed to see the defence
18.5 Regression
Jailbreak robustness decays silently. Two practices keep it honest:
- Pin the attack suite and run it in CI. Every model update, every
classifier update, every prompt change. A regression is a 10-point rise in
ASR on a fixed suite, not a vibe. - Keep a live adversarial set. New public techniques go into the suite
within a week of publication; techniques that stop working stay in as
regression tests, because a defence that fails against an old attack has
regressed.
19. Open problems
Ten problems this survey cannot answer, in descending order of how much they
matter.
- Robustness to a method, not a sample. Every defence is evaluated
against strings. No defence has been shown robust to a generator with an
unlimited budget and knowledge of the defence. This is the central open
problem and the rest are corollaries. - Alignment depth. If safety is a direction (§2.3) and a few tokens deep
(§4.9), what training objective makes it distributed? Circuit breakers are
the most interesting attempt; the evidence is early. - Multi-turn evaluation. Benchmarks are stateless and attacks are not.
There is no standard, reproducible, stateful benchmark with a budget
accounting that matches §6. - Trajectory detection. Scoring a conversation for escalation is a
sequence-modelling problem that almost nobody has posed, and it is the
control that would address the best-cost attack family. - Judge validity. ASR rests on judges with unmeasured error. A
benchmark's judge is a scientific instrument, and the field does not
calibrate its instruments. - Capability-safety coupling. Does removing refusal always cost
capability? The evidence says the cost is small (§11.2), which is bad news
for the "safety tax" argument and for the hope that capability and
alignment are entangled. - Multimodal parity. Safety evaluation lags capability by a modality.
Audio and video are years behind text, and the gap widens with each release. - Agent authority models. There is no accepted formalism for what an
agent is allowed to do on whose behalf. Without one, §13 is engineering
folklore. - Provenance. Detection of modified checkpoints is unsolved at
inference time and only partly solved at distribution. This is a supply
chain problem wearing a machine learning costume. - Economics of defence. Nobody has published the marginal cost curve of
ASR versus spend. Without it, security budgets for AI are allocated by
anecdote.
Part V - Apparatus
20. Timeline
Dates are first public release unless a venue is named. The arc has three
phases: folklore (2022-23), systematisation (2023-24), and the shift to
systems (2025-).
| Date | Event | Why it matters |
|---|---|---|
| Dec 2022 | ChatGPT ships; persona prompts circulate immediately | Establishes the Tier 0 folk tradition |
| Feb 2023 | Greshake et al., indirect prompt injection | Names the class that will dominate deployments |
| Feb 2023 | Kang et al., programmatic behaviour and payload splitting | Shows prompts are programs |
| Jul 2023 | Wei et al., Jailbroken | First theory: competing objectives, mismatched generalisation |
| Jul 2023 | Zou et al., GCG | First transferable optimised suffix; ASR becomes a number |
| Jul 2023 | Xie et al., self-reminder | First cheap inference-time defence |
| Aug 2023 | Yuan et al., CipherChat | Establishes the capability-safety gap experimentally |
| Oct 2023 | Chao et al., PAIR | Automated black-box search; the query budget becomes the unit |
| Oct 2023 | Yong et al., low-resource languages | Safety training is not multilingual |
| Oct 2023 | Qi et al., fine-tuning compromises safety | Alignment does not survive a training run |
| Nov 2023 | Li et al., DeepInception | Nesting depth as a parameter |
| Dec 2023 | Mehrotra et al., TAP | Tree search beats linear refinement |
| 2023 | Robey et al., SmoothLLM; Kumar et al., erase-and-check | The randomised and the certified defence families |
| Feb 2024 | Mazeika et al., HarmBench | The comparison the field needed; negative result on defences |
| Feb 2024 | Souly et al., StrongREJECT | Binary ASR is measuring the wrong thing |
| Feb 2024 | Röttger et al., XSTest | Overrefusal becomes measurable |
| Feb 2024 | Jiang et al., ArtPrompt | Tokenizer-level bypass of three model families |
| Apr 2024 | Anil et al., many-shot jailbreaking | Context length as an attack surface |
| Apr 2024 | Russinovich et al., Crescendo | Turns beat prompts; the trajectory becomes the unit |
| Apr 2024 | Andriushchenko et al., adaptive random search | ~100% with logits and a budget |
| Apr 2024 | Wallace et al., instruction hierarchy | The only real defence for §13 |
| Jun 2024 | Arditi et al., refusal is one direction | Mechanistic account of why ablation works |
| Jun 2024 | Wei et al., pruning and low-rank modification | Safety is a few percent of parameters |
| Jun 2024 | Microsoft, Skeleton Key | Policy reframing works on frontier systems |
| Jun 2024 | Debenedetti et al., AgentDojo | Agentic injection gets a benchmark |
| Jul 2024 | Deng et al., MasterKey | Generators that survive patching |
| Oct 2024 | Zou et al., circuit breakers | The first defence aimed at depth rather than surface |
| Dec 2024 | Hughes et al., Best-of-N | ASR is a power law in budget |
| Jan 2025 | Sharma et al., Constitutional Classifiers | Best measured model-level defence, with a published break |
| Feb 2025 | Kuo et al., H-CoT | Safety reasoning is itself an attack surface |
| Apr 2025 | Rao et al., X-Teaming | Automated multi-turn reaches the high nineties |
| Jun 2025 | EchoLeak, CVE-2025-32711 | Zero-click exfiltration in a production assistant |
| 2025 | UK AISI, Boundary Point Jailbreaking | First automated black-box pass of the Constitutional Classifiers bar |
| Oct 2025 | Nasr et al., The Attacker Moves Second | Twelve defences, twelve breaks; adaptive evaluation becomes the rule |
The companion document, BLACKHAT.md, continues the timeline into
the adversary economy - what the techniques sell for, who buys them, and how
the operators get caught. §41 is the operator-side version of the table above:
the same window, dated by falling costs and opening channels rather than by
published ideas.
21. Glossary
| Term | Definition |
|---|---|
| Abliteration | Removing the refusal direction from weights; §11.2 |
| Adaptive attack | An attack written with knowledge of the defence; §14.3 |
| AdvBench | 520-behaviour benchmark from Zou et al.; §14.1 |
| ASR | Attack success rate; definition varies by paper; §14 |
| Best-of-N | Sampling n augmented variants until one succeeds; §5.1 |
| Circuit breaker | Defence that reroutes representations mid-generation; §16.2 |
| Competing objectives | Theory that refusal loses to helpfulness under framing; §2.1 |
| Constitutional classifier | Classifier trained on a written constitution of allowed content; §16.1 |
| Crescendo | Multi-turn escalation from benign to prohibited; §6.1 |
| Directional ablation | Projecting out a direction in activation space; §11.2 |
| GCG | Greedy coordinate gradient suffix optimisation; §9.1 |
| Half-life | Time for a technique's ASR to halve unprompted; §3.3 |
| HarmBench | 400-behaviour, 18-attack comparison suite; §14.1 |
| Indirect prompt injection | Instructions arriving in content the model processes; §13.1 |
| Jailbreak | Input causing a policy-violating response; §1 |
| JailbreakBench | 100-behaviour benchmark with artefacts; §14.1 |
| Judge | The thing that decides whether an attack succeeded; §14.2 |
| Many-shot | Hundreds of fabricated compliant turns in context; §5.1 |
| Mismatched generalisation | Theory that safety training is distributionally narrow; §2.2 |
| Overrefusal | Refusing benign requests; the cost side of every defence; §14.5 |
| PAIR | Attacker-LLM loop with judge feedback; §5.2 |
| Payload splitting | Distributing a request so no part is actionable; §4.8 |
| Prefill | Supplying the first assistant tokens; §4.9 |
| Refusal direction | The activation direction mediating refusal; §2.3 |
| Shallow alignment | Safety behaviour concentrated in early tokens; §2.3 |
| StrongREJECT | Rubric-scored compliance quality; §14.1 |
| Suffix | Optimised token sequence appended to a request; §9.1 |
| TAP | Tree-search attacker; §5.3 |
| Tier 0-5 | Access-based attack taxonomy; §3.2 |
| Transfer | An attack working on a model it was not optimised against; §9.2 |
| XSTest | Overrefusal benchmark; §14.1 |
22. Number card
Every figure below is reported by the cited work under that work's conditions.
They are not comparable to each other; §14 explains why, and this card exists
so that the numbers can be checked rather than quoted.
| Figure | Value | Source | Condition |
|---|---|---|---|
| Constitutional Classifiers, ASR | 86% → 4.4% | Sharma et al., 2025 | 10,000 synthetic attacks, Claude 3.5 Sonnet |
| Overrefusal change | +0.38 pp | Sharma et al., 2025 | 5,000 benign conversations |
| Compute overhead | +23.7% | Sharma et al., 2025 | Inference only |
| Red-team effort defeated | 183 people, >3,000 hours | Sharma et al., 2025 | 10-query universal-jailbreak bar |
| Best-of-N, GPT-4o text | 89% | Hughes et al., 2024 | n in the thousands |
| Best-of-N, Claude 3.5 Sonnet | 78% | Hughes et al., 2024 | same |
| Best-of-N, GPT-4o vision | 56% | Hughes et al., 2024 | image augmentations |
| Best-of-N, Realtime audio | 72% | Hughes et al., 2024 | audio augmentations |
| Many-shot efficiency gain | 28× | Hughes et al., 2024 | 74% ASR at n = 274 vs 6,000 |
| PAIR, GPT-3.5 | 60% | Chao et al., 2023 | 20-query budget |
| PAIR, Vicuna | 84% | Chao et al., 2023 | 20-query budget |
| Crescendo, GPT-4 | +29-61 pts over prior SOTA | Russinovich et al., 2024 | AdvBench subset |
| Crescendo, Gemini Pro | +49-71 pts | Russinovich et al., 2024 | AdvBench subset |
| GCG, Vicuna-7B | 88% | Zou et al., 2023 | white-box |
| GCG transfer, GPT-3.5 | 84% | Zou et al., 2023 | black-box transfer |
| GCG transfer, GPT-4 | 53% | Zou et al., 2023 | black-box transfer |
| Low-resource language, GPT-4 | 41-79% | Yong et al., 2023 | Zulu, Scots Gaelic, Hmong |
| Adaptive random search | ~100% | Andriushchenko et al., 2024 | logit access, ~10³ queries |
| Refusal removal by ablation | 13 models | Arditi et al., 2024 | open weights |
| Safety-critical parameters | ~3% | Wei et al., 2024 | pruning study |
| Defences broken by adaptive attack | 12 of 12 | Nasr et al., 2025 | most above 90% ASR |
| HarmBench scope | 400 behaviours, 18 attacks, 33 models, 4 defences | Mazeika et al., 2024 | - |
| JailbreakBench scope | 100 behaviours, 100 benign | Chao et al., 2024 | - |
23. Annotated bibliography
Entries are ordered by theme, not alphabetically, because the useful reading
order is thematic. arXiv identifiers are given where the work is a preprint;
venues are given where it is not. Annotations state what the work contributes
to the argument of this document, and where it is weak.
Theory
Wei, Haghtalab, Steinhardt (2023). Jailbroken: How Does LLM Safety Training
Fail? arXiv:2307.02483. The competing-objectives and mismatched-
generalisation dichotomy. Still the best single framing; weak on transfer,
which the mechanistic work later supplied.
Qi et al. (2024). Safety Alignment Should Be Made More Than Just a Few
Tokens Deep. arXiv:2406.05946. The shallow-alignment result. The most
consequential mechanistic finding for defenders, because it says where the
behaviour lives.
Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single
Direction. arXiv:2406.11717. The refusal direction. Explains ablation,
transfer, and why steering works; the paper is careful about capability cost
and the community's use of it is not.
Qi et al. (2024). Jailbreaking as a Reward Misspecification Problem. The
optimisation-side account: attacks are the expected behaviour of a policy
optimised against a proxy.
Attack methods
Zou et al. (2023). Universal and Transferable Adversarial Attacks on
Aligned Language Models. arXiv:2307.15043. GCG. The transfer numbers are
the important part; the suffixes are not.
Chao et al. (2023). Jailbreaking Black Box Large Language Models in Twenty
Queries. arXiv:2310.08419. PAIR. Defines the attacker-loop methodology.
Mehrotra et al. (2023). Tree of Attacks: Jailbreaking Black-Box LLMs
Automatically. arXiv:2312.02119. TAP. Better ASR per query; produces
near-misses useful for defence training.
Liu et al. (2023). AutoDAN: Generating Stealthy Jailbreak Prompts on
Aligned Language Models. arXiv:2310.04451. Fluency as an evasion
strategy; the reason perplexity filters are not a control.
Yu et al. (2023). GPTFuzzer: Red Teaming Large Language Models with
Auto-Generated Jailbreak Prompts. arXiv:2309.10253. Fuzzing; the method
that recovers automatically from patches.
Liao and Sun (2024). AmpleGCG. arXiv:2404.07921. Suffix generation as a
learned task; amortises optimisation across behaviours.
Deng et al. (2024). MasterKey: Automated Jailbreak Across Multiple Large
Language Models. arXiv:2407.17778. Cross-model transfer plus re-adaptation
after patching. The black-box analogue of suffix transfer.
Russinovich, Salem, Eldan (2024). Great, Now Write an Article About That:
The Crescendo Multi-Turn LLM Jailbreak. arXiv:2404.01833; USENIX Security
2025. The multi-turn escalation result and its automation. The turn-count
finding is the takeaway.
Li et al. (2023). DeepInception: Nurture the Jailbreaker in a Similar
Way. arXiv:2311.03111. Nested fiction; depth as a parameter.
Yuan et al. (2023). GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs
via Cipher. arXiv:2308.06463. Ciphers. The cleanest capability-safety gap
demonstration.
Yong, Menghini, Bach (2023). Low-Resource Languages Jailbreak GPT-4.
arXiv:2310.02446. Language coverage as an alignment gap; the strongest
evidence for mismatched generalisation.
Jiang et al. (2024). ArtPrompt: ASCII Art-based Jailbreak Attacks Against
Aligned LLMs. arXiv:2402.11753; ACL 2024. Tokenizer-level bypass without
optimisation.
Kang et al. (2024). Exploiting Programmatic Behavior of LLMs: Dual-Use
Through Standard Security Attacks. arXiv:2302.05733. Payload splitting,
format injection, refusal suppression. Under-cited relative to its influence.
Anil et al. (2024). Many-shot Jailbreaking. Anthropic. Context length
as an attack surface; the scaling with shot count is the durable result.
Hughes et al. (2024). Best-of-N Jailbreaking. arXiv:2412.03556. Budget
as the independent variable; the power law is the finding.
Andriushchenko, Croce, Flammarion (2024). Jailbreaking Leading
Safety-Aligned LLMs with Simple Adaptive Attacks. arXiv:2404.02151. The
strongest single empirical statement: ~100% with logits and a budget.
Rao et al. (2025). X-Teaming: Multi-Turn Jailbreaks and Defenses with
Adaptive Multi-Agent Collaboration. arXiv:2504.13210. Multi-turn search at
frontier scale.
Kuo et al. (2025). H-CoT: Hijacking the Chain-of-Thought Safety Reasoning
Mechanism of Large Reasoning Models. arXiv:2502.12893. Reasoning traces as
an attack surface; the newest genuinely new mechanism in the field.
Greshake et al. (2023). Not What You've Signed Up For: Compromising
Real-World LLM-Integrated Applications with Indirect Prompt Injection.
arXiv:2302.12173. The class that turned out to matter most.
Gong et al. (2023). FigStep. arXiv:2311.05608. Typographic image
attacks; the multimodal version of payload splitting.
Qi et al. (2023). Visual Adversarial Examples Jailbreak Aligned Large
Language Models. arXiv:2306.13213. The visual analogue of GCG.
Weight-space
Qi et al. (2023). Fine-tuning Aligned Language Models Compromises Safety,
Even When Users Do Not Intend To. arXiv:2310.03693. The result that made
fine-tuning a security boundary.
Wei et al. (2024). Assessing the Brittleness of Safety Alignment via
Pruning and Low-Rank Modifications. arXiv:2402.05162. Safety is sparse.
Zhou et al. (2024). Emulated Disalignment: Safety Alignment for Large
Language Models May Be a Double-Edged Sword. arXiv:2402.12343. Alignment
as a property of the checkpoint combination, not the weights.
Measurement
Mazeika et al. (2024). HarmBench. arXiv:2402.04249. The comparison
suite; its negative result on defences is the useful part.
Chao et al. (2024). JailbreakBench. arXiv:2404.01318. Comparability and
artefact submission; the leaderboard discipline the field needed.
Souly et al. (2024). A StrongREJECT for Empty Jailbreaks.
arXiv:2402.10260. Compliance quality versus binary success; the correction
that most ASR numbers needed.
Röttger et al. (2024). XSTest. arXiv:2308.01263. Overrefusal, measured.
**Nasr et al. (2025). The Attacker Moves Second. The adaptive-evaluation
rule, and the empirical demonstration that twelve defences did not survive it.
Defence
Sharma et al. (2025). Constitutional Classifiers: Defending Against
Universal Jailbreaks. arXiv:2501.18837. Best measured model-level defence;
publishes its own red-team protocol and its compute cost.
Zou et al. (2024). Improving Alignment and Robustness with Circuit
Breakers. arXiv:2406.04313. The most promising attempt at depth rather
than surface.
Robey et al. (2023). SmoothLLM: Defending Large Language Models Against
Jailbreaking Attacks. arXiv:2310.03684. Randomised smoothing; the cost
model is the problem.
Kumar et al. (2023). Certifying LLM Safety Against Adversarial
Prompting. arXiv:2310.10501. Erase-and-check; the only certified-style
guarantee in the literature, under assumptions that matter.
Xie et al. (2023). Defending Against Jailbreak Attacks with
Self-Reminder. arXiv:2307.16888. The cheapest useful inference-time
control.
Zhang et al. (2023). Goal Prioritization and Conflicting Objectives in
Instruction Following. arXiv:2310.17680. Attacks the §2.1 mechanism
directly, which is rare.
Wallace et al. (2024). The Instruction Hierarchy: Training LLMs to
Prioritize Privileged Instructions. arXiv:2404.13208. The only defence
with a clear mechanism against indirect injection.
Inan et al. (2023). Llama Guard. arXiv:2312.06674. The moderation-model
family that most deployments actually use.
Debenedetti et al. (2024). AgentDojo. arXiv:2406.13352. The agentic
injection benchmark; the first place to look when evaluating a harness.
24. Method appendix
Inclusion. Works are included if they (a) introduce a mechanism, (b)
provide a measurement that constrains a claim, or (c) are the canonical
artefact for a technique. Works that report a single prompt string without a
measurement are excluded, which removes a large fraction of the grey
literature and most of the vendor blog posts.
Numbers. Every figure in §22 was taken from the source's reported result
under the source's conditions. Where a source reports several conditions, the
one closest to a hosted-API threat model was used. No figure in this document
was averaged across papers, because the denominators differ.
Contested claims. Three are flagged rather than resolved:
- Whether many-shot jailbreaking scales with capability or with context
length. Both are reported; the confound is not separated. - Whether abliteration costs capability. The published cost is small on
standard suites; standard suites may not measure the relevant capability. - Whether adversarial training generalises. HarmBench says it is the best
available defence; adaptive evaluation says the best available defence is
not robust. Both are true and the field has not reconciled them.
Error sources. Judge error (§14.2) dominates; benchmark contamination is
second; small n is third. The combined effect is that an ASR quoted without
conditions should be read as ±15 points, which is enough to reverse most
rankings in the literature.
What this document is not. It is not a reproduction. Nothing here was run
against a live system for this edition; every number is second-hand and
attributed. A reader who needs first-hand numbers should take the artefact
from §23 and run it against their own deployment with the protocol in §18.
25. Technique index
Alphabetical, with the tier and the section that treats it.
| Technique | Tier | Section |
|---|---|---|
| Abliteration | 4 | §11.2 |
| Adversarial suffix | 3 | §9.1 |
| ASCII art masking | 0 | §4.7 |
| Authority claim | 0 | §4.2 |
| Best-of-N augmentation | 1 | §5.1 |
| Cipher and encoding | 0 | §4.5 |
| Context dilution | 2 | §6.3 |
| Crescendo escalation | 2 | §6.1 |
| Emulated disalignment | 4 | §11.5 |
| Fine-tuning | 4 | §11.1 |
| Format injection | 0 | §4.8 |
| Fuzzing | 1 | §5.4 |
| Genetic prompt search | 3 | §9.4 |
| Hypothetical framing | 0 | §4.3 |
| Indirect injection | 5 | §13.1 |
| Instruction hijack of reasoning | 2 | §7.2 |
| Low-resource language | 0 | §4.6 |
| Many-shot | 1 | §5.1 |
| Memory poisoning | 2 | §6.4 |
| Multi-agent impersonation | 5 | §13.4 |
| PAIR loop | 1 | §5.2 |
| Payload splitting | 0 | §4.8 |
| Persona adoption | 0 | §4.1 |
| Prefill | 0 | §4.9 |
| Pruning | 4 | §11.3 |
| Random search with logits | 3 | §10.1 |
| Refusal suppression | 0 | §4.4 |
| Skeleton Key reframing | 0 | §4.2 |
| TAP tree search | 1 | §5.3 |
| Tool coercion | 5 | §13.2 |
| Trajectory escalation | 2 | §6.2 |
| Visual adversarial example | 3 | §12.1 |
Part VI - The other side of the same surface
Everything above this line was written for the defender, in the defender's
units: what works, at what cost, against what control. This part is the same
material read the other way round. Not the research literature's view of the
attacker - the attacker's view of the research literature.
The distinction matters because the two readers want opposite things from the
same number. §15 reports that Best-of-N reaches 89% on GPT-4o with n in the
thousands. A defender reads that as "I need an output classifier". An operator
reads it against §5.7's own ledger - about twelve cents of queries per
successful output - and multiplies: ten thousand outputs a day is $1,200 a day
of spend before a single customer has paid. Same figure, opposite decision.
Five things change when you turn the document around:
- The axis inverts. Part I ranks attacks by cost because cost determines
who can mount them. An operator ranks them by yield per unit of
maintenance, because maintenance is the only cost that recurs. A cheap
attack that has to be re-found every eleven days is more expensive than an
expensive one that never breaks. - ASR stops being interesting. §14 spends a thousand lines on how to read
an attack success rate. An operator does not care about a rate. They care
about a pipeline throughput - outputs per hour, at a quality the customer
accepts, at a p99 latency that keeps the session alive. - The target changes. Every method in Parts II and III attacks a model.
Every documented incident attacks a system: a wrapper, a bot, an agent
with tools, a document store. §13 says why; this part is about how, and
about what it costs to run it. - Persistence beats capability. The incident record is dominated not by
novel technique but by infrastructure that stayed up: an endpoint, a
payment rail, a distribution channel, a fine-tune that kept serving after
the base model was patched. §41's timeline is largely a timeline of
infrastructure, not of methods. - Attribution is the binding constraint. Not detection. Not refusals.
The thing that ends operators is billing identity, retained logs, reused
infrastructure and the fact that a business has to be a business in public.
§38 is the section the romantic version of this document would omit.
A word on what this part is not. It is not an operations manual, and the omission
is deliberate in the same way §0's is: nothing here reproduces a payload, a
working prompt, an exploit, or a configuration. What it does is describe the
shape of the job - the stages, the costs, the failure modes, and the
arithmetic - because that is the layer at which defenders can act and at which
the published evidence actually exists. The bytes are a search away and always
have been. The 90% of an operation that is logistics is not written down
anywhere, and that is what this part is for.
Where Part VI cites a case, the case is named by publisher, not linked, for the
same reason BLACKHAT.md does it: the URLs rot and the structures do not.
26. The shape of the job
The first thing to understand about the blackhat side of jailbreaking is that
almost nobody who does it for money is doing the thing this literature
describes. They are not optimising suffixes. They are running a service
business with an unusually cheap input and an unusually hostile set of
suppliers.
The work decomposes into seven stages. They have very different costs, very
different skills, and only one of them is in the research literature.
| # | Stage | The question | Skill | Share of published research |
|---|---|---|---|---|
| 1 | Demand | Who wants output they cannot get legitimately? | market reading | none |
| 2 | Access | Whose model, at what price, with what quota? | procurement, account farming | almost none |
| 3 | Capability | How do I get a refusal to fail? | prompt craft, ML, or a download | all of it |
| 4 | Production | How do I do it 10,000 times a day? | engineering | a little (§5, §6) |
| 5 | Delivery | How does the buyer reach it? | product, marketing, SEO | none |
| 6 | Payment | How do I get paid and keep the account? | payments, risk | none |
| 7 | Maintenance | How do I stay alive when the supplier changes? | ops, monitoring | §3.3, abstractly |
Read the third column against the last one. The literature is a 1,900-line
document about stage 3, which is the cheapest stage, requires no infrastructure,
and can be bought for $0.32 according to the price sheet in BLACKHAT.md §7.1.
The incident literature - Unit 42, Anthropic's GTG-1002 disclosure, the npm and
MCP supply-chain cases - is almost entirely about stages 4 through 7, which the
research literature does not measure at all.
That gap is not an accident of academic fashion. Stage 3 is the only stage that
produces a publishable artefact. You cannot write a paper about chargeback
rates, and the field's incentive is to publish methods, so methods are what we
have a science of.
26.1 Three archetypes
The word "blackhat" covers three businesses with almost nothing in common, and
conflating them is the most common analytical error in this space.
The reseller. Bought or found a kit; sells access to a jailbroken endpoint
at $10-$20 a month. Zero technical depth, thin margin, no moat, and a lifespan
measured in processor relationships. This is the largest population by count
and the smallest by revenue. The $0.32 prompt and the $50-$200/month kit in
BLACKHAT.md §7.1 exist to serve them.
The product operator. Runs a real application whose feature is output the
mainstream model will not give: a companion app, an uncensored roleplay service,
a translation tool for content that platforms moderate, a bulk content farm.
This is where the actual money is (the companion market's reported $2.32B in
2025, per BLACKHAT.md §7.4), and it is a normal consumer business with an
abnormal cost base: the compute is nearly free, the compliance risk is not.
The capability broker. Sells the technique rather than the output: kits,
fine-tunes, "audits", working prompts, agent harnesses with the approvals
removed. Highest margin, lowest volume, and the archetype with the most
ambiguous legal position, because a red-team engagement and an abetment can
involve the same artefact and differ only in who signed.
The three have different threat models and different half-lives. A reseller
dies when a payment processor reads the terms of service. A product operator
dies when a model provider patches the hole or an app store applies a policy.
A broker dies when a customer does something loud with the thing they bought.
Defensive effort aimed at the wrong archetype is wasted: output classifiers do
nothing to a broker, and model patches do nothing to a reseller who is using a
locally-abliterated checkpoint.
26.2 What an operator actually optimises
Not ASR. Three quantities, in this order:
- Yield per unit of maintenance. Successful outputs per week, divided by
the hours spent keeping the pipeline working. This is why a fixed system
prompt beats a search loop even when the search loop has higher ASR: the
prompt is maintained once, the loop is maintained forever. - Cost per output at volume. Not per attack. §5.7 prices a success at
three to twelve cents. At a thousand outputs a day, that is $30-$120 a day
of attacker-side spend against a product that may be charging $20 a month -
which is the arithmetic that kills the naive wrapper, and the subject of
§37. - Time to first output after a break. The interval between "the supplier
changed something" and "the service is working again". Because breakage is
certain (§35), this is the only quantity that compounds.
There is a fourth quantity that operators talk about and never measure, which is
detectability. It gets a whole section (§36) because it is the one where the
folklore is loudest and the evidence is thinnest.
26.3 Why the operator is not a researcher
Five misconceptions that the research literature induces in anyone who reads it
as an attack manual:
| The paper implies | The operator finds |
|---|---|
| "88% ASR" means 88% of requests succeed | it means 88% on a 520-behaviour benchmark, in a single turn, against a bare model, with a judge that may be wrong (§14.2) |
| A method is a thing you use | a method is a distribution over prompts; you use a sample, and samples regress |
| Transfer means it works everywhere | transfer means it works somewhere, at some rate, on some version (§9.2) |
| The hard part is the jailbreak | the hard part is the fourth week |
| Anonymity is the defence | anonymity is a billing problem (§38) |
The fifth row is the one worth internalising. Every account of an AI-enabled
crime that ends in an arrest ends on a mundane fact: a card, a phone number, a
reused handle, a log left on a server that was never meant to be public. The
Unit 42 operator, in the best-documented campaign in the corpus, published its
own API keys and target list with python3 -m http.server. That is not a
failure of jailbreaking. It is a failure of the seventh stage.
26.4 The four quantities every operator tracks
For completeness, because these reappear throughout Part VI and a reader needs
the units:
| Quantity | Definition | Typical value, derived from §5.7 and BLACKHAT.md §7 |
|---|---|---|
c_a |
cost of one attempt, attacker side | $0.0003 (fixed prompt) to $0.004 (search loop) |
p |
per-attempt success probability | 0.01 (naive) to 0.9 (adaptive, patched model) |
c_s |
cost per success, c_a / p |
$0.03-$0.12 for Tier 1; under a cent for Tier 0 |
m |
maintenance load, hours/week | 2 (fixed prompt) to 40 (multi-provider product) |
The whole of §37 is these four multiplied out. The whole of §35 is m
refusing to stay small.
27. Target selection
Before spending anything, an operator establishes four things about the target:
what it is, what it refuses, what it can do, and what it costs to keep. This is
the cheapest stage of the job and the one most often skipped, which is why so
much of the grey literature consists of prompts that were written for a model
that no longer exists.
27.1 Fingerprinting the model
The target is rarely identified in a way you can trust. A wrapper's marketing
names a model; its behaviour names a different one. Six signals, ordered by how
much they tell you per query:
| Signal | What it reveals | Reliability |
|---|---|---|
| Refusal style and wording | family and often version; refusal boilerplate is memorised and stable | high |
| Cutoff date, elicited in conversation | training data vintage, which bounds the version | high |
| Tokeniser artefacts (spacing, rare-script handling, emoji) | family; occasionally a specific serving stack | medium |
| Error shapes and headers on malformed requests | the inference server (vLLM, TGI, Ollama, a cloud gateway), not just the model | high when exposed |
| Context limit, found by binary search | nominal window, which identifies the variant | medium |
| Capability probes on a known-hard task | whether the served checkpoint is the full model or a quantised distillation sold under the same name | medium |
The fifth row deserves a note. Context length is the most reliable cheap
fingerprint because it is a property of the deployment, not the weights, and
the numbers cluster: 8k, 32k, 128k, 200k, 1M. A "frontier model" that tops out
at 16k tokens is a small open-weight model behind a nice landing page, and
pricing it as frontier is the wrapper's problem, not the operator's.
27.2 Fingerprinting the guardrails
Guardrail mapping is the operator's version of §16's layer model, run as a
probe battery. The trick is that you do not need a harmful request to learn
where the layers are - you need a boundary request, and the cheapest boundary
is the one the provider publishes in its own policy: the categories it says it
will refuse. Ask about those, benignly, and watch where the refusal comes from.
| Probe | What it distinguishes |
|---|---|
| A direct request in a prohibited category | whether a model-level refusal exists at all |
| The same request, encoded or translated | whether an input classifier is separate from the model (§16, layer 2) |
| The same request, prefill or continuation | whether the API exposes assistant-prefix control, which is worth more than any prompt (§4.9) |
| A benign request that looks adjacent | whether an output classifier is filtering on topic rather than harm (§16, layer 5) |
| A request for a long, structured, harmless document with a prohibited sentence buried at 80% | whether context truncation or chunked inspection gives you a blind spot |
| Twenty rapid identical requests | rate limits, per-account quota, and whether the second layer counts or only the first |
Two derived observations, both Inferred but consistent with §17.2's cost table:
- The layers fail independently, so the probe battery is short. Six probes
distinguish the common configurations, because in practice deployments use
two or three layers out of eight, and the commonest two are a model refusal
and an input classifier. Four combinations cover most of the market. - Prefill exposure dominates everything else. If assistant-prefix control
is available, §4.9 says the attack is nearly total and the guardrail map is
irrelevant. This is why the first thing a competent operator checks is not
the refusal behaviour but the API shape, and why "we have a classifier"
deployments that also expose prefill are, in the precise sense, undefended.
27.3 The four target classes
| Class | Cost to reach | Guardrail quality | Rate limit | Half-life of access |
|---|---|---|---|---|
| Frontier API, first party | account + payment | high, and improving | strict, metered | weeks; bans are routine |
| Mid-tier API, reseller or regional | account, often no KYC | variable, often a single classifier | loose | months |
| Open-weight model, hosted by a third party | account, usually no payment | none to nominal | loose to none | indefinite |
| Local weights | capex: one GPU | none | your hardware | indefinite (BLACKHAT.md: 11,598 uncensored models in a 17,104-model graph) |
| Someone else's agent | a document, a page, a tool result | whatever the harness did | the victim's quota | until the harness changes |
The table's shape is the argument of §3.4 arriving from the attacker's side:
cost falls as you move down, guardrail quality falls faster, and the bottom two
rows are why model-level defence cannot be the plan. An operator who needs
volume does not attack a frontier API at all; they download a checkpoint that a
volunteer already stripped, or they find a hosted endpoint that never had a
policy.
27.4 Target selection is a market problem
The interesting empirical fact about the ecosystem is that access is cheaper
than capability. A stolen premium account is $0.10; a single working jailbreak
prompt is $0.32; a full synthetic-identity package is up to $6,000
(BLACKHAT.md §7.1, from Halcyon's underground study). That price structure
tells you what is scarce. Identity is scarce. Prompts are not.
Three consequences follow, and they are all defence-relevant:
- Defending the model has a low ceiling. If the prompt costs $0.32, no
amount of model hardening changes the attacker's budget in a way they will
notice. The budget is in identity, infrastructure and payments. - The target list is worth more than the exploit. Unit 42's recovered
session shows the agent enumerating 84 instances of one product, then 25,209
of another, and sampling forty. The expensive artefact in that operation was
the enumeration, not the jailbreak - and it was FOFA queries, which are
sold, not built. - Selection is where the legal exposure concentrates. Choosing a target is
the act that turns possession of a technique into an offence in most
jurisdictions (BLACKHAT.md §8.6). This is also why §18.1's first question -
who is the attacker - is the only one that changes the answer.
27.5 Reconnaissance against the harness
For the class that actually produces incidents - the agent with tools - the
model is not the target. The reconnaissance questions are different and mostly
answerable from public artefacts:
| Question | Where the answer lives | Why it matters |
|---|---|---|
| Which tools can it call? | the open-source harness's config schema, or a leaked system prompt | tool coercion is a different attack than persuasion (§13.2) |
| Does it read attacker-influenced content? | product description: email, PDFs, tickets, web, calendar | that is the injection channel (§13.1) |
| Does it hold credentials? | deployment docs, error messages | the exfiltration target |
| Is output filtered or action-mediation present? | error behaviour on a blocked request | distinguishes "won't say" from "can't do" |
| Does it log? | the harness's own documentation | determines whether you leave evidence |
Note what the right-hand column does not require: a single prohibited request.
The whole battery can be run with benign traffic. This is the structural reason
prompt injection is the durable attack of the decade - the reconnaissance is
free, legal, and indistinguishable from using the product.
27.6 The cost of this section
Derived, using §5.7's dollar-per-million-token convention and assuming a
frontier API: roughly 40 probes, averaging 900 input tokens and 150 output
tokens, is 36k input and 6k output - about $0.20 at typical frontier pricing
($3/$15 per million) and about $0.01 on a cheap model at $0.15/$0.60. Half a day of human attention.
That is the whole cost of target selection. Any framing of this problem that
treats reconnaissance as expensive has the economics backwards, and any defence
that assumes the attacker did not do it is a defence against a lazy attacker.
28. Sourcing capability
An operator does not invent a jailbreak. They acquire one, in one of five
markets, and the choice of market determines their cost base, their half-life,
and their exposure more than any prompt decision they will make later.
28.1 The five supply lines
| Supply line | Cost | Quality | Trust | Half-life |
|---|---|---|---|---|
| Paper plus artefact repository | free | high, but written for the paper's model | high; the code is visible | model generations (§3.3, Tier 3) |
| Open-weight checkpoint, redistributed | free | whatever the volunteer did, usually abliteration or a fine-tune | low; this is the malware channel | indefinite (§11.4) |
| Prompt pack or "jailbreak database" | $0-$200 | folklore, mostly dead strings | medium | days to weeks |
| Fine-tune-as-a-service | tens to hundreds of dollars, or a rented GPU | good if you can evaluate it | medium; you cannot see the data | until the base model moves |
| Agent kit / harness with approvals removed | $50-$200/month, per the underground price sheet | high; this is where the real leverage is | low; you are installing a stranger's autonomy | months |
The ordering is not by price, it is by leverage per dollar, and the last row
wins it. A jailbreak prompt gets you text you should not have. A harness with
--dangerously-skip-permissions gets you an entity that reads, decides and
acts. That is the difference between a rainy afternoon and a capability, and
it is why the market for agent kits has outgrown the market for prompts even
though prompts are what the research literature studies.
28.2 The trust problem, which is the whole problem
Every one of those supply lines is a supply chain, and the operator is the
endpoint. They are running unattended software with their own credentials, on
their own machine, against targets they would rather not explain. The threat
model they tend to forget is the one that includes their supplier.
Three documented shapes, all from the same eighteen-month window:
The checkpoint with an extra. An "uncensored" model is a few gigabytes of
weights and a chat_template. Nothing about the format enforces that the
template is a template. Loading a stranger's checkpoint runs stranger-authored
code in your process, with your environment, and the standard format has no
signature. Cisco's and KELA's 2025 examinations of uncensored reasoning models
found payloads attached to the jailbreak scripts themselves - an infostealer in
the thing that was supposed to make the model compliant. The operator who runs
it does not get an uncensored model. They get an uncensored model and a
credential-harvesting process, and they notice the second one when their
accounts are gone.
The dependency that asks the agent. s1ngularity (August 2025) is the
canonical case and it is worth reading twice because the technique generalises:
malware shipped through a build tool, whose second stage looked for installed
AI CLIs and launched them with permission prompts disabled, instructing
them to enumerate configuration files, environment variables and anything that
looked like a secret. The result, per Wiz: 2,349 credentials from 1,079
developers, 2,180 accounts and 7,200 repositories exposed. The scanner was not
clever. It delegated to an agent that already had the permissions and the
reasoning.
The tool description that is an instruction. postmark-mcp (September
2025) shipped a server with code copied from the legitimate repository and one
line added: a hidden BCC on every outbound message. About 1,500 weekly installs
leaked mail until someone read the diff. The generalisation is BLACKHAT.md
§5.4's and it belongs here too: an agent's tool description is an instruction
channel. Whoever writes the prose writes the behaviour, and every marketplace
of skills, tools and plugins imports that trust silently.
The economic point underneath all three: the black market for jailbreak
tooling is adversarial to its own customers. There is no reputation system
that survives a ban, no escrow that survives a cash-out, and no recourse when
the $50/month kit is a phishing page. This is the single largest asymmetry
available to defenders. The attacker's supply chain is hostile to the attacker,
and the cheapest way to raise the cost of an operation is to make the operator
distrust their own tools - which is why §36's detection work and the model
provenance problem (open problem 9 in §19) are the same problem wearing
different clothes.
28.3 What a careful operator actually does
A minority verify, and the checks are unremarkable:
| Check | Catches | Effort |
|---|---|---|
| Hash the artefact, compare to the publisher's | swapped downloads | minutes |
Read chat_template, tokenizer_config, any *.py in the repo |
template injection, loader payloads | an hour |
| Diff the "uncensored" checkpoint against its base | spliced backdoors and prompt-level canaries | an hour, needs the base |
| Run the first load in a container with egress denied and no credentials | most infostealers | minutes |
| Grep the harness for outbound endpoints before the first run | exfiltration from agent kits | minutes |
| Pin by hash and re-verify on update | silent updates | process discipline |
Every row is cheap. Almost none are done, because the population that buys a
kit is selected for impatience. That selection effect is the reason the
documented compromise rate of this ecosystem is high and the documented
success rate of its crimes is low.
28.4 Sourcing is continuous, not a purchase
The mistake that ends most reseller operations is treating capability as an
asset. It is a subscription. Three clocks run at once:
- The publication clock. Whatever you have is in a public repository, which
means it is in a defence vendor's regression suite within a week (§18.5), which
means it is in the next model's safety fine-tune. - The provider clock. Models are versioned, deprecated and re-pointed.
A pinned snapshot buys you time; an unpinned alias can change under you
overnight, and the first symptom is a refund request. - The redistribution clock. The 10a Labs study of open-weight
redistribution found 3,471 originals and 8,164 redistributions - the supply
multiplies faster than anyone can vet it. Convenient for the buyer, and
exactly why the trust problem in §28.2 does not improve over time.
The practical consequence: an operator's sourcing cost is not the price of the
artefact, it is the price of re-sourcing it on a schedule. Anyone who prices
their operation as a one-off purchase has mis-modelled their own cost base by
an order of magnitude, and the error surfaces in month two.
29. The operator's prompt corpus
Here is the part of the job where the research literature is genuinely useful
and the folklore genuinely wrong. Useful, because the mechanisms in §4 are
stable and can be composed. Wrong, because the unit of work is not a prompt.
A prompt is a sample. A corpus is the thing that produces samples on demand
and tells you which ones still work.
29.1 The anatomy of a corpus
Six layers, from most durable to least:
| Layer | What it is | Half-life | Rebuild cost |
|---|---|---|---|
| 1. Mechanisms | the eight families of §4, plus §6's escalation shapes | years | high; this is knowledge |
| 2. Compositions | mechanism pairings that work together (persona + encoding, fiction + splitting) | months | medium |
| 3. Templates | the composition with typed slots: {frame}, {authority}, {constraint}, {payload}, {format} |
weeks | low |
| 4. Instances | the filled template, in a specific register and language | days | trivial to regenerate |
| 5. Mutations | the operators that turn one instance into a family (§29.2) | indefinite | n/a |
| 6. Metadata | which model, version, date, judge, observed outcome | the actual asset | n/a |
Layer 6 is the one nobody builds and everybody needs. Without it, a corpus is a
text file of strings with no expiry dates, and the operator's own history is
unavailable to them - so they cannot answer "what worked last week" and cannot
detect that everything stopped working on the same afternoon, which is exactly
the signature of a provider-side patch.
29.2 Mutation operators
The mechanism layer explains why mutations work. A defence is trained on a
distribution; a mutation is a movement perpendicular to the part of the
distribution the defence actually covered. §4's families imply a finite list of
operators, and the list is short enough that anyone running a service should
have all of them implemented as functions rather than as saved text.
| Operator | Example of the transformation | Defeats |
|---|---|---|
| Paraphrase | rewrite the request in different words, same intent | literal string matching, and the defence's memorised instances |
| Register shift | formal legal memo, casual forum post, terse note | classifiers keyed to a particular tone |
| Language transfer | low-resource translation (§4.6) | English-centric classifiers and safety data |
| Encoding | cipher, base-N, leetspeak, spaced or reversed text | input classifiers without a decode pass |
| Representation | ASCII art, tables, code comments, markdown structure (§4.7) | tokenisation-level filters |
| Persona rotation | swap the frame's role, world and norms (§4.1) | refusal priors tied to a named persona |
| Nesting | add a fictional or hypothetical layer (§4.3) | shallow refusal heuristics |
| Splitting | distribute the request so no part is actionable (§4.8) | per-turn harm detection |
| Length manipulation | flood the context, or compress to a fragment | sliding-window inspection |
| Authority injection | a policy, a document, a memo, an audit role (§4.2) | instruction-hierarchy confusion |
| Continuation | prefill the opening tokens of compliance (§4.9) | anything but a full-token audit |
Ten operators, one of which (continuation) is worth more than the other nine
combined when the API exposes it, and worth nothing when it does not. That
asymmetry is why §27.2 puts API shape ahead of guardrail mapping in the
reconnaissance order.
The composition rule is the part that is genuinely non-obvious and is where a
corpus beats a prompt: mutations are not additive. Persona plus encoding
usually beats either alone; persona plus persona is a contradiction the model
resolves by refusing; splitting plus length flooding is worse than splitting
alone because the context dilutes the frames. A corpus with metadata is how you
learn which pairs are synergistic on your target, and it is knowledge that
does not transfer between targets - which is the point of running one.
29.3 Curation, and the contamination trap
The obvious move is to harvest the public benchmarks: AdvBench's 520
behaviours, HarmBench's 400, JailbreakBench's 100. The obvious move is also the
one that wastes the most money, for a reason that is the mirror image of §14.1's
benchmark problem.
Those strings are in the defence's training data. They have been public for
years, they appear in every safety dataset, and a model has been fine-tuned
against them in particular. Testing a deployment with AdvBench strings measures
the deployment's memory, not its alignment. A corpus built from them will
report either an implausibly low success rate (the defence memorised the exact
strings) or an implausibly high one (the strings predate the defence), and
neither number tells the operator what will happen tomorrow.
The fix is the same one §18.5 prescribes for defenders and it is symmetric:
stratify by mechanism, not by string. Generate instances from templates,
hold out the mechanisms, and keep a small canary set of well-known strings as a
control - if the canaries succeed and the generated set does not, you have
learned something about the defence's training data, which is worth more than
either number alone.
29.4 The regression suite is the business
The last structural point, and the one that separates an operation from an
afternoon: a corpus is a CI pipeline, not a document.
seeds ──> templates ──> instance batch ──> target ──> judge ──> score ──> promote/retire
|
└──> metadata store
The loop runs weekly. Promoted instances go into the served system prompt.
Retired instances stay in the suite as regression tests, because "the defence
failed against something old" is a signal about the defence and "the defence
failed against something new" is a signal about the corpus - and confusing the
two is how operators conclude a model was updated when in fact their own
template broke.
Cost of the loop, derived: 200 candidate instances, one attempt each, ~1,200
input tokens, judged by a cheap model at ~400 tokens - about 240k tokens of
target traffic and 80k of judge traffic, or roughly $0.10 and forty minutes
at §5.7's prices. That is the weekly premium for knowing whether you still have
a business. Almost nobody pays it, which is why the visible population of
resellers churns so violently: they find out that their prompt stopped working
from a customer, not from a test.
29.5 What the corpus buys at scale
- Half-life extension. A fixed prompt decays at the rate in §3.3, days to
weeks. A corpus with ten mutation operators decays at the rate of the
mechanism, months, because when one instance dies the generator produces
the next one without human input. - Attribution resistance. Distinct instances per session defeat the
traffic-shape clustering described in BLACKHAT.md §8.3 - not the billing
identity, which is unaddressed by any prompt work, but the cheap layer of
detection that most abuse teams run first. - Transfer leverage. When a new model appears, the metadata tells you which
mechanism family to try first, which cuts the re-validation cost of a new
backend from days to hours. In a market where the supplier changes quarterly,
that is the entire operational advantage.
The corpus is also the only part of the operation that is legal to own. A
prompt pack is a text file; a fine-tune is a file; a working payload is a
crime. That difference is why brokers sell corpora and kits rather than
outcomes, and why the greyest businesses in this space look, from the inside,
exactly like a red-team tooling vendor with worse invoices.
30. The harness
At volume, the jailbreak is a line in a config file and the harness is the
business. This is the stage the research literature treats as an
implementation detail - PAIR and TAP are described as algorithms with a query
budget, not as systems - and it is where every hour of an operator's week goes.
30.1 The five components
| Component | Job | Typical failure |
|---|---|---|
| Queue | holds attempts, ordered by expected value | unbounded growth; a thousand dead attempts still queued |
| Worker | one attempt: render, send, receive, timestamp | no backoff on 429; burns the account in an hour |
| Judge | decides win/lose (§31) | systematic error in one direction, undetected for weeks |
| Store | metadata per attempt: template, target, version, outcome | the thing that gets seized |
| Router | picks the backend per request | no failover; one provider's patch = total outage |
Two design notes that are not obvious until you have run one.
The retry policy is the attack. Rate limits do not slow an operation, they
shape it: a worker that retries on 429 with jitter finishes the same 10,000
attempts, just later and with a traffic pattern that looks organic. A worker
that retries immediately gets the key banned and teaches the operator nothing.
The single most consequential config value in a real pipeline is the delay
between attempts, and it is set by the defender's quota policy, not by the
attacker's preference.
Parallelism has a ceiling set by someone else. Adding workers raises
throughput until you hit the target's per-account limit, then raises the
number of bans instead. §5.7's closing point - "the one lever that does work
on cost is per-account query accounting, because the attacker's budget is real
money and the defender controls the meter" - is a statement about harness
architecture. The operator's throughput is the defender's quota dashboard, read
upside down.
30.2 Quota arbitrage
Four ways to get more attempts per dollar than the price list allows, in
descending order of how well they work:
| Method | Mechanism | Defeated by | Cost |
|---|---|---|---|
| Free-tier farming | many accounts, each under the free cap | identity verification, phone reuse detection | labour; ~$0.10/account on the resale market |
| Key resale | buying organisational or leaked keys | billing anomaly detection, spend caps | $0.10 for a premium account, per BLACKHAT.md §7.1 |
| Provider arbitrage | route to whichever backend is cheapest today | nothing; this is legitimate cost control | engineering |
| Credit fraud | trial abuse, card testing, refund cycling | payment risk engines | legal exposure, high |
The first two are why "identity" is the scarce commodity in §27.4's price
structure. Note that the price of a stolen account, ten cents, is below the
cost of the tokens that account can buy, which means the market is pricing the
identity, not the compute. Whenever a credential trades below the value of what
it unlocks, the constraint in that market is identity issuance, and any defender
who owns identity issuance owns the attacker's cost base.
30.3 Throughput arithmetic
Derived, and worth doing once because the numbers are smaller than people
assume. Assume a frontier API with a 500 requests/minute per-account limit, a
2,000-token average attempt, and p = 0.05 for a decent fixed prompt against a
guarded model.
| Quantity | Value |
|---|---|
| Attempts per account per minute (quota-bound) | 500 |
| Attempts per account per hour | 30,000 |
Successes per hour at p = 0.05 |
1,500 |
| Tokens per hour | 60M |
| Cost per hour at $0.15/M off-peak input | $9.00 |
| Cost per success | $0.006 |
Read the bottom row against §5.7's $0.03-$0.12. The difference is not a better
method; it is that this attacker has p = 0.05 on a fixed prompt and is
paying a cheap model's price per token. The literature's cost-per-success
figures are dominated by the price of the target, exactly as §5.7 says, and the
cheapest high-volume operations are the ones that chose a cheap target (§27.3)
rather than a clever method.
The other reading of the same table: 60M tokens per hour per account is a
screaming traffic signature. No human sends 500 requests a minute for an
hour. Which is why §36 exists, and why the arithmetic that makes the operation
viable is also the arithmetic that makes it visible.
30.4 The build order that works
For anyone modelling an adversary's maturity, the order in which capabilities
get built is diagnostic:
- A single-shot script with a hardcoded prompt. (An afternoon.)
- Retry and rate-limit handling. (A day, and the point at which the operator
stops losing accounts.) - A judge, so attempts can be scored without reading them. (A day.)
- Metadata and a corpus (§29). (A week, and the point at which the operation
becomes maintainable.) - Multi-provider routing. (A week, and the point at which a patch stops being
an outage.) - Delivery and payments (§34, §37). (Where it becomes a business, and where
most of them fail.)
An adversary at stage 2 is a nuisance. At stage 4 they are a durable problem,
because they can re-find what the defence removes. At stage 6 they are a
company, with the exposure that implies. Most of the population in the
documented cases sits at stages 1-2 with an unusually powerful model, which is
precisely why the incident record looks the way it does: high capability,
amateur tradecraft, fast attribution.
31. Judging your own success
The attacker's judge is the same instrument as the defender's (§14.2), pointed
the other way, and it has the same defect: unmeasured error. The difference is
that the attacker's error has a direct cash cost, because it determines which
methods get funded.
31.1 What the judge is for
Four decisions, all of which are worse with a bad judge:
| Decision | Needs | Cost of error |
|---|---|---|
Is p high enough to scale? |
a calibrated estimate of p |
overpaying for a method by 3× (§31.2) |
| Which template to promote? | relative ranking across templates | the corpus converges on a fluke |
| Is the target still vulnerable? | a stable baseline | outages discovered by customers |
| Is the output good enough to sell? | a quality rubric, not a binary | refunds and churn |
The last row is the one the research literature has no analogue for. A
benchmark asks "did it comply". A product asks "would a customer pay for this",
which is a different question with a different judge, and the gap between them
is where most wrapper businesses die: they optimise compliance and ship
gibberish.
31.2 The arithmetic of a bad judge
This is the single most useful piece of arithmetic in Part VI, because it
explains a persistent puzzle: why operators consistently believe their methods
work better than they do.
Take a stream of 1,000 attempts against a target with true p = 0.05, so 50
real successes and 950 real failures. Use a judge with 90% sensitivity and 90%
specificity - optimistic for a model judge on this task, given §14.2.
| Judge says win | Judge says loss | |
|---|---|---|
| Actually won (50) | 45 | 5 |
| Actually lost (950) | 95 | 855 |
The judge flags 140 attempts. 45 are real. Precision is 32%, and the
operator's estimate of p is 14%, not 5%. A 2.8× overestimate, from a judge
that is right nine times out of ten on both classes.
Say it plainly: at low base rates, a good judge produces a wildly wrong
rate. This is not a subtle statistical point, it is the reason the field's
headline numbers do not replicate (§14.1), and it applies with full force to
the operator who is trying to decide whether to buy a $200/month kit.
The fix is the same one §18.3 gives defenders, and it is not optional: draw a
random sample of flagged and unflagged attempts, label them by hand, and compute
precision and recall. Two hundred labels calibrate a judge well enough to make
budget decisions. The operators who do this discover that their "70% success"
prompt is at 20%, and the ones who do not spend the next quarter building on
sand.
31.3 The judge as a second jailbreak
There is an irony that catches everyone: the judge is a model, the judge is
often the same family as the target, and the judge may refuse to score the
output it is asked to score.
Three consequences:
- Systematic false negatives on the worst categories. The judge refuses to
engage, returns a hedge, and the pipeline scores it as a loss - so the
operator's corpus systematically under-weights exactly the content that is
most valuable to their customer. Inferred, but it follows from the asymmetry
in §2. - Judge drift under provider updates. The judge is a hosted model with the
same deprecation clock as the target (§28.4). An unversioned judge changes
the measurement, and the change looks like a change in the target. - The rubric beats the binary. §18.3's four-row table applies verbatim:
for a product, a rubric score is the only usable signal, because "complied"
does not distinguish a sellable output from a degenerate one.
31.4 The downstream judge
The honest ground truth is the downstream one: did the artefact work? Did the
phishing page get clicks, did the code compile and run, did the exploit land,
did the customer renew. This is expensive and slow, and it is the only judge
that cannot be gamed by the model under test.
The published campaigns show what happens without it. Unit 42's agent reported
progress, enumerated tens of thousands of hosts, and produced three confirmed
successful exploitations across 460+ targets. Whether its own internal report
said "success" is unknown; whether the operator believed it is not, because the
operator sent one Telegram message and went to sleep. A judge that scores
attempts, rather than outcomes, would have rated that operation a triumph.
For defenders this is a gift and should be understood as one: the attacker's
success measurement is systematically over-optimistic, so the threat is usually
less capable than its own reporting. Budget accordingly, but do not discount
the capability itself - three confirmed exploitations from one message is still
three more than a human operator would get for the same money.
32. Session tradecraft
A conversation is a state machine (§6) and also a metered resource: every turn
re-sends the history, so the token cost of a session grows quadratically with
its length while the probability of success grows sub-linearly. The operator's
job is to find the crossing point.
32.1 The cost curve
Derived. Assume a 700-token opening request, 300-token replies, and a linear
escalation that adds ~200 tokens of attacker text per turn. Cumulative input
tokens after n turns is roughly n(n+1)/2 × 1,000.
| Turns | Cumulative input tokens | Cost at $1/M | Cost at $0.15/M |
|---|---|---|---|
| 5 | 15k | $0.015 | $0.002 |
| 10 | 55k | $0.055 | $0.008 |
| 20 | 210k | $0.21 | $0.032 |
| 50 | 1.28M | $1.28 | $0.19 |
| 100 | 5.05M | $5.05 | $0.76 |
Two readings. Below ten turns, conversation is free and the only cost is
attention. Above twenty, the context dominates and the operator is paying for
the defender's transcript. The practical consequence: escalation is an
economy of the first ten turns, and any technique that needs fifty turns
(Crescendo's long variants, patient foot-in-the-door) has to justify itself
against the cost of simply restarting with a better opening - which, at p =
0.05 and a fixed prompt, is usually cheaper.
32.2 Restart or continue
The decision every multi-turn operator faces at turn k, and the one they get
wrong most often, because of sunk context:
| Condition | Action | Why |
|---|---|---|
| Refusal landed, persona intact | continue, reframe, do not acknowledge | §6.2; the refusal is a turn, not a verdict |
| Refusal landed, persona broken | restart | repairing a broken frame costs more turns than a fresh opening |
| Model repeating a hedge | change register or language, not content | hedges are distributional, not semantic |
| Success landed | stop | every extra turn risks the output filter catching the good turn (§16, layer 5) |
| Context past ~60% of window | restart with compression | quality degrades before the limit does |
The fourth row is the one with real money attached. An operator who keeps
going after a win often loses the win to a classifier on the way out, and the
pipeline records a failure - so the corpus is trained against the behaviour that
actually worked. This is a measurement bug that presents as a capability bug,
and it is endemic.
32.3 State that survives the session
Three layers, in ascending order of value to the attacker and of danger to the
defender:
- Within a session. The transcript. Free, and destroyed by any restart.
- Across sessions, in the product. Saved conversations, memory features,
custom instructions. The operator's version of §6.3's persona persistence,
and the reason a consumer app with a memory feature is a more valuable
target than a stateless API. - In the defender's own store. Memory poisoning in the sense of §6.4: an
instruction the agent writes to its own notes and later treats as its own
conclusion. One successful injection becomes a durable capability, invisible
to per-request inspection.
Layer 3 is the only one that produces persistence without the attacker paying
rent, which is why it is the highest-value target in the agentic surface and
the least instrumented. A defender who logs every request and no memory write
has instrumented the wrong side of the system.
32.4 The session budget is an identity budget
Abuse teams do not count attempts, they count anomalies per identity per day.
That makes sessions a consumable:
- Warm-up. A few benign sessions on a new account before the first
prohibited one. Cheap, effective against naive anomaly scores, and completely
ineffective against a per-account cumulative-refusal metric - which is why
§17.4 recommends exactly that metric. - Burn rate. Sessions per account per day before the account dies. Inferred
from the ban economics in BLACKHAT.md §7.1: at $0.10 an account, an operator
will happily burn ten a day, which sets a floor of about a dollar a day on
identity costs that no model-side defence touches. - Identity pooling. Many accounts, one corpus. The corpus is the asset; the
identities are consumables. This is the operator's correct mental model, and
it is the reason detection built on "this account is behaving badly" has a
half-life measured in days.
32.5 When the session can act
The moment a session has tools, the risk profile changes class. Content harm is
reputational and legal; action harm is legal, irreversible, and attributable to
a human who configured a permission set.
The operator's version of §13's argument, stated as a design rule: an agent
session is worth more than a chat session by a factor of ten, and it is worth
documenting, because the first thing an agent does in an unfamiliar
environment is leave a record of itself. The Hermes case is the type specimen:
the model did the work, then the operator's own environment published the
credentials, target list and shell history through an unauthenticated port. The
offence scaled; the tradecraft did not.
33. Weight-space operations
Tier 4 is where the operator stops renting capability and starts owning it.
It is also where the economics are least like the rest of this document,
because the cost curve is capital expenditure rather than marginal spend, and
because ownership removes the two constraints that bound every other tier:
rate limits and provider policy.
33.1 Who does it, and why
| Archetype | Why they take the capex | What they get |
|---|---|---|
| Product operator (§26.1) | per-token cost at volume; no AUP risk | an endpoint with no supplier |
| Broker | resale: one artefact, many customers | a product with zero marginal cost |
| Researcher-adjacent | curiosity, publication, clout | the same thing without the revenue |
| Reseller | rarely; the capex is the barrier | usually nothing |
The reported distribution supports the ordering. The 10a Labs study found
1,643 applications built on redistributed open-weight models, a quarter of them
explicitly malicious, and Lin et al. found 11,598 uncensored models in a graph
of 17,104 - a supply so large that the marginal model is free and the marginal
operator is not. Almost nobody in that population fine-tunes from scratch.
They download, verify nothing, and serve.
33.2 The pipeline
base checkpoint
│ (a) refusal-direction extraction: contrast pairs, one forward pass per layer
▼
abliterate or LoRA-fine-tune ──> (b) capability check, (c) compliance check
│ │
▼ ▼
quantise (4/8-bit) ──> serve (vLLM/TGI) ──> monitor ──> re-eval on base updates
The literature's part is (a) - §11.2's direction, §11.1's fine-tune. The
operator's part is (b) through the end, and it is where the money is spent and
the mistakes are made.
(b) is the check nobody runs. Ablation removes refusal and, per §11.2,
costs some capability - the published cost is small on standard suites, and
standard suites may not measure the capability you are selling. A model that
will write anything and can no longer follow a five-part instruction is not a
product. The operator's own evaluation should be a capability suite and a
compliance suite, in that order, because compliance without capability is a
very cheap way to own a useless endpoint.
(c) is the check that gets gamed. Testing your own abliterated model with
prompts from the corpus that produced it is a closed loop. The only meaningful
test is held-out mechanisms (§29.3), and the only meaningful pass criterion is
downstream (§31.4).
33.3 What self-hosting actually costs
Derived, because the folklore ("self-host, it's free") is wrong in an
interesting direction. Rent an H100 at ~$2.50/hour, which is a realistic 2026
spot price; that is $0.00069 per GPU-second.
| Serving regime | Aggregate throughput | Cost per million tokens |
|---|---|---|
| Single stream, 70B, 4-bit | ~50 tok/s | ~$13.90 |
| Batched, 70B, 4-bit | ~1,500 tok/s | ~$0.46 |
| Batched, 8B, 4-bit | ~4,000 tok/s | ~$0.17 |
Compare with the price sheet: DeepSeek V4.1 Flash at $0.15/$0.60 per million
tokens off-peak (BLACKHAT.md §12). The honest conclusion, and it cuts against
the whole romance of the black market: self-hosting is more expensive than
the cheapest hosted API, and cheaper than every frontier API. It is not a
cost play. It is a policy play - you are buying the absence of an acceptable
use policy, and paying a premium of perhaps 3× over the cheapest legitimate
tokens for it.
Three corollaries:
- The break-even against a frontier API is trivial - batched serving beats
$3/M by an order of magnitude - which is why every serious product operator
eventually owns weights, and why §37's margin tables look so good. - The break-even against a cheap API never arrives, which is why resellers stay
on rented endpoints and die with them. - The capex is the moat and the anchor. Once you have bought the GPU and built
the pipeline, you are committed to a model generation, and the generation
turns over in months.
33.4 Serving is the skill
The gap between "I abliterated a model" and "I run a service" is entirely
operational, and it is where the research literature is silent:
| Problem | Why it bites | Mitigation |
|---|---|---|
| Quantisation damage | 4-bit is 4× the throughput and measurably worse at long instruction-following | evaluate at the quantisation you will serve |
| Context growth under load | KV cache is the real memory cost; concurrency collapses at long contexts | cap context, truncate aggressively |
| Cold start | 70B at 4-bit is ~40GB to load; a restart is minutes of downtime | keep it warm; that is the bill |
| Output filtering | you removed the refusal, not the customer's expectation of one | you now own the moderation you sold as absent |
| Version drift | the base model gets a safety update; your ablation is now behind | re-derive, re-evaluate, re-deploy |
That last row is the treadmill in its purest form, and the reason §35 is a whole
section rather than a paragraph.
33.5 The distribution problem
Owning weights removes the supplier and adds a new one: whoever hosts the GPU.
Jurisdiction, provider terms and payment are now three separate exposures, and
the reported enforcement pattern (BLACKHAT.md §8.6) is that they attach to the
distributor, not the user. The 16.52% flag rate in Lin et al. - the fraction of
uncensored models identified as malicious by the platform hosting them - is a
measure of how much of this supply chain is visible to the people who could
turn it off. A supply chain that is 16% visible is not clandestine; it is
tolerated, which is a different and more fragile thing.
34. Delivery
The correction that every operator learns in month one: capability is abundant
and distribution is scarce. The jailbreak costs $0.32. The customer costs a
click, and clicks on this category of product are the most expensive clicks on
the internet.
34.1 The four channels
| Channel | Reach | Acquisition cost | Platform risk | Price point |
|---|---|---|---|---|
| Chat bot (Telegram, Discord) | large, frictionless | low; channel membership is the funnel | high; platform bans the bot, not the operator | $5-$20/month, or per-message credits |
| Web app | largest | high; adult and grey CPCs, SEO takes months | medium; hosting and processor are separable | $10-$25/month |
| API resale | small, technical | near zero; word of mouth among developers | low until it is not; key discovery is fatal | per-token, at a 5-20× markup |
| Character / companion marketplace | very large | zero, you are a listing | medium; the marketplace is the chokepoint | freemium, subscription, coins |
The fourth row is where the volume is, and it is also the row that is easiest to
switch off. A marketplace listing is a distribution channel owned by someone
who has a trust-and-safety team and an app store review process. The reported
structure of the companion market - $2.32B in 2025, ~$400M of it the NSFW
segment (BLACKHAT.md §7.4) - is a market that exists because platforms
deliberately allow a category at the edge of their policy and price the risk
into the ranking. When the policy moves, the traffic moves.
34.2 The product problem nobody expects
Jailbreaking is a supply problem. Delivery is a retention problem, and the
retention curve of an uncensored product is different from a compliant one in a
way that is well documented and badly understood.
The mechanism, stated as the market's own reported finding: compliant platforms
strip high-risk content to keep app-store and processor access, and lose
retention; the uncensored pole abandons mainstream distribution and prices
high to absorb payment and compute risk (BLACKHAT.md §7.4). Candy.ai's claimed
$25M+ ARR against outside estimates of $50-$120M, and Character.AI's ~185M
monthly visits followed by a pivot to licensing, are the two ends of the same
curve.
For the operator, this yields three design rules that have nothing to do with
jailbreaking:
- The first output is the product. Activation is "did the model say the
thing", and it happens in the first thirty seconds or not at all. A pipeline
with a five-turn escalation (§32) has terrible activation, however good its
p. - Filtering you added is churn you bought. An operator who inserts a
moderation layer to keep a processor happy has reintroduced the refusal they
were selling the absence of, and the reviews will say so. - Persona beats capability. Retention tracks character consistency across
sessions (§32.3, layer 2), not benchmark scores. This is why the market wants
memory features, long contexts and stable system prompts - all of which are
cheap on a self-hosted model and awkward on a rented one.
34.3 Support is the hidden cost
Every LLM product is a support business wearing a software business's clothes,
and the uncensored segment is worse because a fraction of customers are trying
to break it, a fraction are trying to resell it, and a fraction are paying with
someone else's card.
| Support load | Driver | Effect on margin |
|---|---|---|
| "It refused me" | pipeline variance, not a bug | the operator's own p leaking to the customer |
| "It's different than last week" | provider patch (§35) | refunds, churn |
| Account sharing | credentials resold | quotas consumed, one subscriber becomes five |
| Chargeback threats | buyer-side fraud | processor risk score (§34.5) |
The first row is the interesting one, because it is structural. A product
built on a probabilistic pipeline has a per-request failure rate that the
customer experiences as a defect. The operator can drive p toward 1 only by
spending per-request (§37), which is exactly the margin they do not have. This
is the economic reason large-scale uncensored services converge on a
fixed-prompt, self-hosted architecture: it turns a variable cost into a
constant, and a variable failure rate into a constant one.
34.4 Discovery
The demand-side data is unambiguous. Halcyon's study measured fewer than 50
AI-tool advertisements per month underground in late 2025, rising to over 1,400
per month by February 2026, across ~4,000 posts in 77 Telegram channels, 20
dark-web forums and 5 specialist markets. Twelve-fold growth in four months is
not a technology signal; it is an advertising signal. The sellers arrived when
they believed the buyers were there.
The channels, in order of volume:
- Search. Queries for "no filter", "unfiltered", "uncensored" plus a model
name. SEO against a provider's brand term is free reach and a trademark
problem simultaneously. - Community seeding. Forum posts and reviews in the communities that
already discuss jailbreaking, which is also where the defence vendors read. - Affiliate and referral. The standard growth loop for grey consumer
products: revenue share, which requires reliable payments, which is the
constraint again. - Marketplace listings. The highest-volume channel and the one with a
landlord.
34.5 The two-sided scam problem
This is the structural feature of the market that defenders under-use. Every
transaction is between two parties who cannot use the law, and neither can
verify the other.
| Failure | Who does it | Frequency | Effect on the market |
|---|---|---|---|
| Exit scam | seller | high | no prepayment trust; prices fall, volume rises |
| Chargeback fraud | buyer | high | processor risk scores rise for everyone |
| Resale of access | buyer | very high | quotas consumed; unit economics break |
| Kit with malware | seller | documented (§28.2) | buyers lose credentials and disappear |
| Fake reviews | both | saturated | discovery becomes unreliable |
Two consequences, and the second is the important one.
First, the market cannot finance trust, so it consolidates on a small number of
channels with visible history - the ones with a public channel, a track record
and something to lose. Second, those channels are the chokepoints. A market
that cannot use escrow, courts or reputation portability concentrates on
platforms that provide them, and platforms are amenable to pressure in a way
that anonymous sellers are not. This is why payment processors and app stores
have removed more capability from this ecosystem than every classifier
combined - the same conclusion BLACKHAT.md §7.4 reaches from the product side,
arriving from the transactions side.
35. Reliability and the maintenance treadmill
Everything in this document decays. The only open question for an operator is
whether they find out from a monitor or from a customer. §3.3 gave the
defender's version of this - half-life - and Part VI's claim is that the
operator's version is not half-life but MTBF of the inventory, and that it
is the quantity that decides who stays in business.
35.1 The failure taxonomy
| Failure | Signature | Detection lag | Recovery |
|---|---|---|---|
| Provider patch | whole corpus drops within hours, uniformly | minutes if monitored, days if not | re-mutate (§29.2); usually partial |
| Silent moderation | a fraction of outputs replaced with a hedge; p drops without refusals |
days; looks like variance | none; migrate the backend |
| Model deprecation / alias re-point | capability shift; different refusal style | hours | re-fingerprint (§27.1) |
| Judge drift | measured p moves while outputs do not |
weeks; nobody audits the judge | recalibrate (§31.2) |
| Rate-limit tightening | throughput collapses, not p |
immediate | more identities, more cost (§30.2) |
| Account ban wave | many dead workers, same day | immediate | identity budget, again |
| Prices change | margin, not capability | a billing cycle | re-route |
| Base model retrain | your ablation is behind; capability gap opens | months | re-derive (§33.2) |
The third and fourth rows are the ones that produce the most wasted effort,
because both present as "the jailbreak stopped working" when in fact the
measurement stopped working. An operator who recalibrates the judge before
rebuilding the corpus saves a week; almost nobody does, because rebuilding the
corpus is the satisfying work.
35.2 MTBF, and why it caps the business
Inferred, from the mechanism half-lives in §3.3 and the patch cadence implied by
the incident record:
| Asset | MTBF | What ends it |
|---|---|---|
| Fixed prompt, frontier API | 2-6 weeks | classifier update or safety fine-tune |
| Fixed prompt, mid-tier API | 2-4 months | rarely patched; usually the provider disappears |
| Search loop (PAIR/TAP-style) | indefinite, degraded | cost rises as p falls; never dies outright |
| Abliterated endpoint | until the next base release | base model moves; product ages |
| Injected agent (Tier 5) | until the harness is fixed | one patch, and the harness rarely patches |
| Payment rail | weeks to months | risk team, not model team |
Read the fourth column as a business constraint. A product whose core asset has
an MTBF of six weeks cannot be sold as a subscription without a rebuild
capability, and the rebuild capability is the corpus and the pipeline (§29, §30).
This is the real reason the market has a long tail of tiny operators and almost
no large ones: scale requires maintenance staff, and maintenance staff are
employees, and employees are attributable.
35.3 SLOs for a service nobody will write a contract for
The operators who last converge on the same four numbers, because their
customers churn on them:
| Metric | Target | Why |
|---|---|---|
| Availability | 99% during the customer's waking hours | downtime is discovered and reviewed publicly |
| p95 latency | under 4 s to first token | slower than a competitor's, and the churn is instant |
| Quality floor | "the thing" happens on the first attempt, 80%+ of sessions | the §34.2 activation rule |
| Breakage-to-fix | under 72 h | the window before a negative review compounds |
The quality floor is the expensive one and it is the one that forces the
architecture: an 80% first-attempt rate is p = 0.8 at the session level,
which a fixed prompt on a guarded model does not deliver and a self-hosted
abliterated model does. That single requirement explains most of the observed
market structure - why product operators own weights and resellers do not.
35.4 The observability paradox
To detect breakage you need logs: every attempt, its template, its target
version, its outcome. To survive attribution you need to not have logs. These
are the same data.
Three resolutions, none good:
- Keep logs, accept the exposure. What almost everyone does. It is why
criminal cases increasingly rest on the defendant's own harness output
(BLACKHAT.md §8.4) and why the Hermes operator's published shell history was
the investigative gift of the year. - Aggregate, do not retain. Per-template success counts, no transcripts.
Loses the ability to debug, keeps the ability to detect. - Retain nothing, monitor live. Alerting without storage. Works until the
first ambiguous failure, at which point the operator is guessing.
The paradox has a clean statement that belongs in a defender's notes: an
operator's need for observability is proportional to their need for
reliability, and their reliability need is proportional to their scale. Small
operators can be invisible and fragile; large operators must be observable and
are therefore findable. Scale buys revenue and sells deniability, which is why
the enforcement record is full of the medium-sized and empty of the tiny.
36. Detection and evasion, honestly
This is the section where the folklore-to-evidence ratio is worst, so it is
organised as a claim-by-claim audit. What does the provider actually see? What
does the operator do about it? Does that work?
36.1 What the provider sees
| Signal | What it is | Attacker counter | Does it work |
|---|---|---|---|
| Request volume and burstiness | attempts per minute per identity | pace the workers; jitter | yes, until the volume is the point (§30.3) |
| Refusal rate per identity | the fraction of requests that get refused | warm-up sessions; rotate identities | no, against cumulative metrics |
| Topic clustering | a distribution concentrated on prohibited categories | diversify requests with benign traffic | no; dilution costs quota and does not change the mode |
| Session topology | thousands of sessions with one identical system prompt | vary the prompt per session | yes, cheaply, and it is why prompt fingerprinting fails |
| Input perplexity | GCG-style suffixes are gibberish | use fluent prompts (AutoDAN's whole point, §9.4) | yes, and it is free |
| Output classifier hits | the model complied and the filter caught it | re-roll; change format | partially; costs attempts |
| Network origin | ASN, IP reputation, proxy quality | residential proxies | yes, at a real cost |
| Payment instrument | card, its history, its reuse across accounts | virtual cards, stolen instruments | no, and this is the binding one |
| Device and browser fingerprint | one machine, many accounts | anti-detect browsers | partially, and expensively |
| Key-sharing geography | one key, three continents | none needed; they buy three keys | n/a |
The bottom half of the table is the argument. Every row from "network origin"
down is an identity signal, and no amount of prompt work touches any of them.
The top half is the layer where evasion is cheap and temporary; the bottom half
is the layer where evasion is expensive and permanent.
36.2 The perplexity myth, and its mirror
The most repeated claim in the grey literature is that "gibberish triggers
detection, so fluent jailbreaks are undetectable". It is half true, and the
half that is false matters.
- True: GCG suffixes are out-of-distribution strings and cheap to score
(§9.1). A detector on input perplexity catches the white-box family at nearly
no cost. This is one of the few defences with a clean mechanism. - False: fluency is not safety. AutoDAN exists precisely because fluent
adversarial prompts defeat perplexity filters (§9.4), and once an attacker
writes readable prose, the perplexity signal is gone - it was never measuring
intent, it was measuring the attack's style.
The mirror claim, from the defender's side, is equally overstated in the other
direction: "the model knows it is being jailbroken". §7.1 is the honest version.
A reasoning model's trace often contains evidence of the conflict - that is a
debugger for the defender and a target for the attacker (§7.2). It is not a
reliable detector, because the trace is generated text, and text can be shaped.
36.3 The base rate argument
The asymmetry that decides this whole section:
- The defender needs one true positive per identity, across a stream where
the attacker must succeed thousands of times. - The attacker needs zero detections per identity, across thousands of
attempts.
At a per-request detection probability of 1%, an identity survives 100 attempts
with probability 0.99^100 = 37%, and 1,000 attempts with probability
0.99^1000 ≈ 0.004%. At 0.1% per request, 1,000 attempts survive with
probability 37%.
The derived conclusion is uncomfortable and it is the one operators price in:
volume is incompatible with evasion at any detection rate above ~0.01% per
request. Which is why the only evasion strategy that works is the one that
abandons volume: low-volume, high-value, one-shot use. That is the broker's
model (§26.1), and it is also, not coincidentally, the model of the attacker who
is hardest to catch - because they generate the fewest events, have the fewest
customers, and need the least infrastructure.
36.4 The evasions that are theatre
Four that circulate widely and do not survive contact with a competent abuse
team:
| Claim | Why it fails |
|---|---|
| "The provider can't tell which model you are" | they can; it is their model, and they have the keys |
| "Encrypted traffic hides the prompts" | the prompts terminate at their API, not at a wire |
| "Rotating IPs is enough" | billing identity is the durable signal, and it does not rotate |
| "A local model has no logs" | it has yours; and your disk is the log (§38) |
The common structure: each claim is true about a layer the defender is not
using. This is the same error as the folklore in §4 - defending against a
mechanism rather than a control - committed from the attack side.
36.5 What actually raises the attacker's cost
Stated as a ranking, because this is the section a defender should take to a
budget meeting:
- Identity issuance. Payment instrument verification, phone binding, spend
caps and per-identity cumulative accounting. The reported price of a stolen
premium account, $0.10, is the number to move. - Cumulative behaviour over sessions. Not per-request classification: a
refusal-rate-per-identity metric, which no prompt trick reduces without
reducing volume. - Traffic-shape anomaly. Not the content; the shape. 60M tokens an hour
from one account is arithmetic, not inference. - Latency and cost asymmetry. Detecting the search loop by its retry
pattern rather than its payloads. - Provenance on the supply side. Flagging redistributed and modified
checkpoints, which is the one control aimed at Tier 4 and the only one that
touches an attacker who never calls an API.
Everything at the top of that list is cheap, boring and system-level, which is
§17.4's conclusion arriving from the attacker's side for the third time in this
document. The repetition is the finding.
37. The operator's cost model
Part I priced attacks. This section prices operations, which is a different
ledger: fixed costs, variable costs, labour, and the cost of being caught. The
numbers are derived from the price sheet and unit economics in BLACKHAT.md §7,
and the reader should treat them as order-of-magnitude.
37.1 The amortisation law
The most consequential single fact in Part VI, and it is arithmetic rather than
research. Take a wrapper at $20/month with a 3,000-message fair-use cap.
- Revenue per message:
$20 / 3,000= $0.0067. - Compute cost per message, fixed prompt, off-peak:
700 × $0.15/M + 350 × $0.60/M= $0.00032. - Compute cost per message if each message requires a search loop at §5.7's
$0.03 per success, repriced at $0.15/M input: ~$0.0045.
| Architecture | Cost per message | Share of revenue | Gross margin |
|---|---|---|---|
| Fixed prompt, prompt-cached | $0.0003 | 4.7% | 95% |
| Fixed prompt, no caching | $0.00032 | 4.8% | 95% |
| Per-request search loop | $0.0045 | 67% | 33% |
| Per-request search loop, one retry | $0.009 | 135% | negative |
The amortisation law: a jailbreak must be a constant, not a query. Any
architecture that pays for the attack on every request is unviable at consumer
prices, and the entire observed market structure - fixed system prompts,
abliterated checkpoints, "persona" products - follows from this one inequality
rather than from anyone's aesthetic preference for simple prompts.
It also explains a puzzle in §15's table: why the highest-ASR automated methods
are commercially irrelevant even though they are the best research. They are
per-query by construction. The commercially relevant methods are the ones whose
output can be baked in - a prompt, a checkpoint, a persona - which is exactly
the set that §3.3 says has the shortest half-life, and therefore exactly the set
that generates the maintenance burden in §35.
37.2 The reseller's P&L
| Line | Value | Note |
|---|---|---|
| Subscribers | 200 | typical for a channel-distributed bot |
| Price | $15/month | market rate |
| Revenue | $3,000 | |
| Token COGS | $192 | 200 × 3,000 messages × $0.00032 |
| Identity and bans | $300 | 10 accounts/day at $0.10 (§32.4) |
| Infrastructure | $100 | VPS, proxies, bot hosting |
| Processor fees | $150 | ~5% |
| Chargeback losses | $36 | 1.2% of revenue |
| Acquisition | $1,000 | the dominant line |
| Net before labour | $1,222 | |
| Labour | ~80 h/month | support, monitoring, rebuilding |
| Effective wage | ~$15/hour | with felony exposure |
That last row is the punchline of the reseller archetype, and it is consistent
with Halcyon's own reported conclusion that there is no mature market for a
fully autonomous attack agent: what the ecosystem sells is task-level
acceleration, not a business in a box. The people running $20 wrappers are
running a minimum-wage job with a criminal record attached, which is why so many
of them are teenagers and so few of them are professionals.
37.3 The product operator's P&L
| Line | Value | Note |
|---|---|---|
| Subscribers | 3,000 | requires real distribution |
| Price | $20/month | |
| Revenue | $60,000 | |
| GPU (one H100, batched) | $1,800 | $2.50/h × 720 h |
| Payments and processors | $3,600 | ~6% with adult-friendly processors |
| Chargebacks | $900 | 1.5%, near the BRAM excessive threshold |
| Support | $8,000 | two staff, partially automated |
| Acquisition | $20,000 | adult-traffic CPCs |
| Net | ~$25,700 | ~43% margin |
| Labour | two people |
Two observations. First, the margin is real but thin by software standards,
and it is thin because of lines that have nothing to do with AI: acquisition,
support and chargebacks. Second, the GPU line is now the smallest technical
cost, which is why §33.3's conclusion matters: self-hosting is a policy
purchase, and at this scale it pays for itself against any frontier API within a
month.
37.4 The broker's P&L
| Line | Value | Note |
|---|---|---|
| Customers | 20 | low volume by design |
| Price | $150/month | kit, updates, support |
| Revenue | $3,000 | |
| Inference COGS | ~$0 | the customer runs it |
| Maintenance | 10 h/week | corpus, docs, patch responses |
| Net | ~$2,700 | ~90% margin |
| Risk | concentrated | one customer's arrest is your indictment |
The broker is the highest-margin, lowest-volume, highest-variance business in
the space, and the variance is legal rather than commercial. It is also the
archetype whose product is closest to legitimate security tooling, which is why
the red-team market and the kit market share suppliers, corpora and sometimes
invoices - and why §37.5's conclusion is about which side of that line a
business chooses to stand on.
37.5 Where the cost axis inverts
Part I's axis: attacks ranked by cost, because cost determines who can mount
them. Part VI's axis: operations ranked by maintenance cost per revenue
dollar, because that determines who survives. The two rankings are almost
uncorrelated:
| Rank | By attack cost (§3.2) | By operating cost per revenue |
|---|---|---|
| Cheapest | Tier 0 persona prompts | self-hosted fixed prompt (constant, amortised) |
| Tier 1 BoN | hosted fixed prompt | |
| Tier 2 multi-turn | search loop (unviable at consumer prices) | |
| Tier 3 gradients | per-query adaptive attack (loss-making) | |
| Most expensive | Tier 4 weights | multi-provider product with 40 h/week maintenance |
The inversion is the thesis of Part VI in one table. The literature's
cheapest attack is not the operator's cheapest operation, and the literature's
most expensive attack is not the operator's most expensive one either. Jailbreaking
research optimises the wrong end of the ledger, which is not a criticism of the
research - it is doing its job - but it is why reading it as a business plan
produces the reseller P&L in §37.2 and a wage of $15/hour.
37.6 The demand side, for scale
UNODC's 2026 assessment put Southeast Asian scam losses at $88.3B-$114.1B for
2025 and described the industry as franchised: a central service hub with
front-end operations, standardised tooling, AI and crypto in the stack
(BLACKHAT.md §7.6). Against that number, a $60,000/month AI product is a
rounding error, and the jailbreak is a component supplier to an industry that
was already industrial before it arrived.
Which is the correct frame for the whole of Part VI. The blackhat side of
jailbreaking is not a hacker story. It is a supplier story: a cheap input
component - text that a model should not produce - feeding a large, existing,
franchised demand. The interesting numbers are prices, margins and maintenance
hours, not success rates.
38. Attribution, or how they get caught
This is the section that the romantic version of this document omits, and it is
the only one that changes a reader's behaviour. Every documented operation in
the corpus ends the same way, and it never ends on a jailbreak.
38.1 The enforcement arithmetic
Three structural facts, before any case:
- The offence completes early. Attempt, solicitation, possession and
conspiracy statutes do not require a working exploit (BLACKHAT.md §8.6).
Everything an operator does before success is already evidence. - The crime scene is a billing relationship. Models are rented. Renting
requires a card, a phone, an identity, and an invoice that survives a
subpoena. This is the difference between this ecosystem and traditional
malware: the infrastructure is not owned, it is subscribed to. - The population is small and the tail is long. The 71 catalogued US
federal AI-CSAM prosecutions (2022-2025) and the cases in Cyprus and Ohio are
not a dragnet. They are a handful of examples chosen to be legible, which is
how enforcement works against any young crime category.
38.2 The failure modes, enumerated
| Failure | What leaked | How common |
|---|---|---|
| Payment identity | the human | near-universal; the first subpoena |
| Infrastructure reuse | linkage across campaigns | very common; VPS and domain reuse |
| Retained logs | the entire operation, timestamped | near-universal (§35.4) |
| Self-exposure | an open port, a public bucket, a debug endpoint | common; the Hermes case is the type specimen |
| Public persona | a handle, a channel, a review history | common; §34.5 forces visibility |
| Customer arrest | the supplier, via the customer's device | common; distribution is a witness list |
| Tooling fingerprint | campaign clustering | common; agents are more consistent than humans |
| Machine speed | the operator is obviously software | common; 500 req/min is not a person |
| Supplier betrayal | credentials, then the buyer | documented (§28.2) |
| Showing off | a writeup, a screenshot, a talk | occasional, and disproportionately fatal |
The three near-universal rows are all financial. This is the single most
important structural fact about the adversary in this document: the thing that
identifies them is not their prompts, it is their invoices.
38.3 The cases, read as attribution stories
Read the documented cases for the failure, not the capability:
- Unit 42's Hermes operator. The agent did genuinely impressive work at
scale: enumeration across tens of thousands of hosts, exploit selection,
chain assembly. It was identified by infrastructure and by a GitHub handle
whose public profile matched an assessed location - and it published its own
API keys, target list and shell history through an unauthenticated
http.serveron port 8888. Capability: high. Tradecraft: none. s1ngularity. Malicious npm packages, four hours of exposure, 2,349
credentials. The attribution path was the publish pipeline itself: a stolen
token, a workflow, a provenance chain. The supply chain that made the attack
scalable is the same chain that made it traceable.postmark-mcp. Discovered because someone read the diff. Malicious tool
descriptions are source code, and source code is diffable.- The CSAM prosecutions. Caught because victims reported, not because a
platform detected (BLACKHAT.md §8.6). Detection did not work; humans did.
The pattern: in every case, the technical capability was not what failed. The
operational layer was, and the operational layer is the layer this document
cannot teach and the research literature does not measure.
38.4 Why agents make it easier
The intuition is that automation improves OPSEC. The evidence says the
opposite, for four reasons BLACKHAT.md §8.5 states and this part should
underline:
- Consistency is a fingerprint. Agents call tools in the same order, query
the same APIs, format the same output, and leave the same artefacts. Humans
vary; software does not. Cluster analysis loves this. - Speed is a fingerprint. Thousands of requests per second from one ASN is
arithmetic, not inference (§36.3). - Logs are generated by default. Harnesses write session JSON, transcripts,
cost ledgers and tool arguments. A developer's feature is an investigator's
exhibit. - Agents act while nobody is watching. Unattended execution means
unattended mistakes: an exposed port, a debug flag, a-vleft on.
The generalisation is worth stating as a law for the AI era: automation
compresses the time between an idea and an act, and leaves the evidence
generation to the machine, which is very good at generating evidence.
38.5 What actually protects an operator
Nothing in this document, and that is the point. The measures that work are
mundane and largely financial: separate legal entities for infrastructure,
payment instruments that are not reused, no logs beyond the current day, no
public persona, no customers. Every one of those is a measure that eliminates
the business rather than protecting it - which is why the population that
survives is the population that stayed small.
The honest summary for a defender: you do not need to detect this activity to
neutralise it, you need to make it have a counterparty. A counterparty is a
processor, a host, a marketplace, a model provider, or a customer. All five are
identifiable, all five keep records, and the removal of any one of them is worth
more than a percentage point of classifier recall.
38.6 The investigator's three questions
For completeness, because a defender designing telemetry should design it to
answer these:
- Who paid? Payment instrument, account lineage, spend pattern.
- Whose infrastructure? Hosting, domains, proxies, ASN history.
- Who was the human in the loop? The one who chose the target (§27.4),
which is the act that converts possession into an offence.
A logging architecture that answers those three questions for every incident is
worth more than one that classifies prompts. That is the design brief, and
almost no deployment is built to it.
39. Why the blackhat reading of this document is wrong
A reader can close Parts I to V and conclude that the attack side is easy, that
the defences are decorative, and that the rational move is to go and do it.
That conclusion is wrong, not on moral grounds but on arithmetic, and this
section is the audit.
39.1 Seven errors
1. ASR is not yield. A reported rate is measured on a benchmark, in one
turn, against a bare model, with a judge whose error is unmeasured (§14). A
yield is successes per hour at a quality a customer accepts. The mapping between
them is not linear and is not favourable: §31.2's arithmetic shows that at low
base rates even an accurate judge inflates the operator's estimate by ~3×.
2. A benchmark is not a deployment. §15.2 says this in one paragraph and it
bears repeating in the operator's units: the deployed target has a classifier, a
rate limit and a log, and it changes monthly. The published 89% is against a
model, and no incident in the corpus was caused by a model.
3. Capability is not a product. The amortisation law (§37.1) is the hardest
constraint in Part VI: 67% of revenue goes to the search loop, 135% with one
retry. Access to capability is not access to a business; the business is a
constant-cost pipeline, and building one is systems work, not jailbreaking.
4. Novelty is not value. The market prices a working prompt at $0.32 and a
stolen account at $0.10. If novelty were the scarce input, the ratio would be
inverted. The scarce inputs are identity and distribution, which is why the
buyers in the documented marketplaces are buying volume and uptime, not
technique (BLACKHAT.md, Halcyon study).
5. Anonymity is not impunity. §38.2's table is the whole argument. The
identification vector is financial in almost every case, and financial
identities are the one thing an operator cannot rotate cheaply without
destroying their revenue.
6. Scale is not profitability. §37.2: a 200-subscriber reseller nets about
$1,200/month before labour and earns roughly $15/hour for it, with criminal
exposure. The broker margin in §37.4 is 90% and the volume is twenty customers,
which is the opposite of scale and the opposite of the folklore.
7. Survivorship runs the other way here. The published cases overstate
capability, not understate it: they are the operations that got written up
because they were large, and even they produced three confirmed exploitations
from 460 targets. The successful quiet operators are invisible, and therefore
unmeasurable, which means every claim about "what works in the wild" - this
document's included - is drawn from the visible tail.
39.2 The missing payload layer
§0 already says that this document does not reproduce payloads. Part VI inherits
that rule for a second reason that is worth stating plainly: at the operator
layer, the payload is the least valuable part of the operation and the most
criminal. Everything above - recon, corpus, harness, judge, delivery, payments,
maintenance - is where the difficulty lives, and the payload is a commodity
priced at thirty-two cents on a market with a review problem.
A reader who wants the bytes can find them. A reader who wants to understand why
the bytes are not the interesting part has just read eleven sections about
logistics.
39.3 The incentives that inflate this literature
Two biases produce the "it's easy" reading, and a careful reader corrects for
both.
Publication bias toward novelty. A paper reporting that a defence works is
harder to publish than one reporting a break. §16's table is the result: the
attack literature is larger, better funded in attention, and more quotable than
the defence literature, and the ratio of claims is not the ratio of effects.
Vendor interest in the threat being large. Attack success rates sell
classifiers. Defense success rates sell classifiers too, but the attack numbers
are the ones that move budget, so they are the ones optimised for headline
legibility - a single number, no conditions, a named frontier model. §14.1
explains why that number is not comparable to anything.
Neither bias makes the threat false. Both make the threat bigger in
presentation than in operation, and the operator who acts on the presentation
is the operator in §37.2's wage column.
39.4 The path that pays better
The comparison that ends the argument, in the same units:
| Path | Revenue | Cost | Exposure |
|---|---|---|---|
| Reseller (§37.2) | $3,000/month | ~$1,800 + 80 h | criminal |
| Broker (§37.4) | $3,000/month | ~$300 + 40 h | ambiguous, converging on criminal |
| Red-team engagement (BLACKHAT.md §7.5) | $15,000-$50,000/month equivalent | labour | none |
| Jailbreak bounty, novel mechanism | five figures, one-off | research skill | none |
The same skill set - corpus construction, harness engineering, judge
calibration, knowing which mechanism beats which layer - is what a red team
sells, at a higher hourly rate, with a contract instead of an indictment. That
is not a moral argument bolted on at the end. It is the market's own price
signal, and it is the reason BLACKHAT.md §7.5 concludes that services beat
products.
39.5 What Part VI is actually for
Three uses, none of them offensive:
- Threat modelling with real cost curves. §26.4's four quantities and
§37.1's amortisation law let a defender rank their risks by what an attack
costs to operate, which is the only ranking that predicts persistence. - Adversary emulation. A red team that knows the maintenance treadmill
(§35) will not report "we jailbroke it once" as a finding; it will report
what survives a fortnight. - Budget arguments. The three controls in §36.5 and the P&Ls in §37 are
the same document read twice: cheap identity controls beat expensive
classifiers, and the numbers to prove it are above.
The rest is colour, and colour is what most writing about this subject is made
of.
40. What changes for the defender
Part V closed with §17.4's "one control to add this quarter" and §18's protocol.
Part VI changes the reasoning behind both, without changing the
recommendation, which is the strongest evidence that the recommendation is
right.
40.1 Stage-to-control map
| Operator stage | Their cost driver | Cheapest control that attacks it | Their cost after |
|---|---|---|---|
| 1 Demand | acquisition spend | payment and store policy, not technology | channel disappears |
| 2 Access | identity, $0.10/account | payment verification, phone binding, spend caps | identity cost rises 10-100× |
| 3 Capability | $0.32 for a prompt | nothing durable; classifiers buy days (§3.3) | days |
| 4 Production | quota | per-identity cumulative accounting, traffic shape | throughput ceiling |
| 5 Delivery | marketplace access | platform terms enforcement | distribution loss |
| 6 Payment | processor tolerance | nothing a defender does; this is their weak point | - |
| 7 Maintenance | hours | keep the treadmill turning: ship updates, vary detection | breakage-to-fix time |
Read the fourth column downward. The controls that move the attacker's cost by
an order of magnitude are all in rows 2, 4, 5 and 6 - identity, quota,
distribution, money - and none of them are model-level. The single row where
model-level work buys something is row 3, and §3.3 says what it buys: days.
40.2 The three controls that beat the economics
Restated, because §36.5 is the operational version and this is the budget
version:
- Identity issuance and binding. The cost of a viable identity is the
attacker's floor. Raise it and every per-attempt cost in §5.7 multiplies. - Cumulative accounting per identity. The only control that is unaffected
by prompt mutations, because it does not look at the prompt. It converts the
attacker's volume - the thing their business requires (§37.1) - into their
own detection signal. - Action mediation for anything with tools. §13.5's table, and the only
row where the fix is cheap and known: allowlist, parameter validation,
approval gates, egress enumeration.
40.3 The three that do not
- Prompt blocklists. Half-life of days (§3.3), defeated by ten mutation
operators (§29.2), and they generate overrefusal (§14.5) which is a real cost
with no security return. - Model hardening as the plan. Necessary, insufficient, and it addresses
the stage that costs the attacker thirty-two cents. - "Detecting jailbreak phrasing." A classifier for a style. §36.2's mirror:
fluency is free for the attacker, so the style moves and the classifier does
not.
40.4 The asymmetry ledger
The reason the three controls above work is not that they are clever; it is that
they exploit a structural asymmetry:
| Attacker | Defender | |
|---|---|---|
| Cost structure | variable, paid before revenue | fixed, paid once |
| Failure mode | one detection kills an identity | one miss is one miss |
| Requirement | reliability and invisibility | detection or deterrence |
| Time horizon | must survive the next patch | must survive the audit |
| Exposure | financial identity | legal process |
The attacker must be simultaneously reliable and invisible at scale, and §35.4
and §36.3 show those requirements are in direct conflict. That conflict is the
defence. It is not a model property and it cannot be trained into one.
40.5 The two halves of the evidence
Parts I to V ask what breaks a model. Part VI asks what it costs to run a
business on the answer. The second question is the one that determines
persistence, because a technique that cannot be operated at a constant cost
disappears from the incident record regardless of its success rate - which is
why the observed incidents are Tier 0 prompt abuse and Tier 5 injection, the two
tiers with the lowest maintenance load, rather than the tiers with the highest
success rates.
The defender's summary of Part VI, in five lines:
- Price the operation, not the attack; the recurring cost is maintenance (§35).
- Identity is the scarce input and the binding control (§27.4, §36.5, §40.2).
- Volume and evasion are incompatible; choose which one to defend against (§36.3).
- The attacker's measurement is optimistic by roughly 3× at low base rates
(§31.2), so budget against capability, not against claims. - The reliable operator is visible and the invisible operator is small (§35.4),
and neither is a threat to a layered stack - which is why §17.3's modest
guarantee, "the cheapest attack expensive and the expensive attack
detectable", remains the correct standard after both halves of this document.
Part VI apparatus
The sections that follow are the operator-side counterparts of §§20-25. They
are separate rather than merged because the two halves of this document are
asking different questions and a merged number card would imply comparability
that does not exist.
41. Timeline, operator side
The research timeline in §20 dates ideas. This one dates infrastructure and
markets, because that is what changed the incident rate. Publisher named, no
links, same convention as BLACKHAT.md.
| Date | Event | Why it matters |
|---|---|---|
| 2023 | FraudGPT at $200/month, $1,000/6 months, $1,700/year; WormGPT shut down Aug 2023 | The reference price for "AI for crime" is set, and the first brand is already dead |
| 2023 | Prompt packs circulate as products | The $0.32 commodity appears before the model that makes it reliable |
| Jun 2024 | Arditi et al. publish the refusal direction | Abliteration becomes a weekend task; uncensored forks follow within weeks |
| 2024-25 | LoRA fine-tunes and "uncensored" checkpoints become a category, not an artefact | Supply stops being scarce; the 11,598-model graph in Lin et al. is the result |
| Jan 2025 | DeepSeek R1 ships; Cisco and UPenn report 100% success on 50 HarmBench prompts | The cheapest capable model is the least defended one |
| Early 2025 | KELA finds infostealer scripts in R1 jailbreak packages | The supply chain turns on its own customers; §28.2's pattern begins |
| Jan-Feb 2025 | DeepSeek blocked in Italy, banned on Australian government devices, 17 US states, PIPC inquiry in Korea | Regulation attaches to distribution, not capability |
| Jun 2025 | EchoLeak, CVE-2025-32711 | Zero-click exfiltration in a production assistant; Tier 5 goes mainstream |
| Aug 2025 | s1ngularity: nx packages, 2,349 credentials, 2,180 accounts, 7,200 repos |
Malware delegates to the developer's own coding agent |
| Sep 2025 | postmark-mcp: ~1,500 weekly installs, hidden BCC |
The tool description becomes a known instruction channel |
| Oct 2025 | Nasr et al., 12 of 12 defences broken | Adaptive evaluation becomes the standard the market must meet |
| Oct 2025-May 2026 | Halcyon's measurement window: 4,000 posts, 77 channels, 20 forums, 5 markets | The first quantitative read of the AI-crime market |
| Nov 2025 | Anthropic discloses GTG-1002: ~30 organisations, 80-90% of tactical work by the model | The most effective "jailbreak" in the record is an authorisation story |
| Feb 2026 | Underground AI-tool advertising passes 1,400/month, from under 50 in late 2025 | The sellers arrive after the buyers; this is the adoption curve |
| Mar 2026 | Cyprus convictions; Bowling Green sentencing, 5 years federal | The enforcement record stops being hypothetical |
| Jul 2026 | Unit 42: Hermes agent, DeepSeek, 460+ targets, 3 confirmed exploitations | Capability at scale with no tradecraft: keys published by http.server |
| Sep 2026 | 10a Labs: 3,471 originals, 8,164 redistributions, 1,643 apps, 25% malicious | Redistribution is the persistence layer of the open-weight ecosystem |
| Oct 2026 | V4.1 Flash released 12 Oct; 31 uncensored forks, 83,900 downloads | The strip-to-distribution interval is now hours |
| Oct 2026 | Halcyon via Axios: $0.10 account, $0.32 prompt, $6,000 identity | The price structure that inverts every cost model in Part I |
The shape is not the shape of §20's. The research timeline is a sequence of
ideas getting better; this one is a sequence of costs falling and channels
opening, in that order, with the capability arriving last and mattering least.
42. Operator glossary
Terms used in Part VI and in the marketplaces, defined for a reader who has only
read the research literature.
| Term | Definition |
|---|---|
| Abliterated | A checkpoint with the refusal direction removed; §11.2, §33.2 |
| Activation | In the market sense: the customer's first successful output; §34.2 |
| Amortisation law | A jailbreak must be a constant, not a per-request query; §37.1 |
| Ban wave | A provider terminating many accounts on one day; §35.1 |
| Broker | Archetype selling technique rather than output; §26.1 |
| Burn rate | Accounts consumed per day at a given volume; §32.4 |
| Chargeback | Buyer-initiated reversal; the processor-level constraint on §34 |
| Corpus | Mechanisms, templates, instances and metadata as one maintained asset; §29 |
| Exit scam | Taking prepayment and disappearing; the reason the market cannot finance trust; §34.5 |
| Farm | Many accounts used to multiply a quota; §30.2 |
| Fingerprint (theirs) | Behavioural signature by which an operation is clustered; §38.4 |
| Fingerprint (yours) | What you learn about the target before spending: model, version, guardrail shape; §27 |
| Half-life | Time for a technique's ASR to halve unaided; §3.3 |
| Harness | The queue-worker-judge-store-router system that runs attempts; §30 |
| Identity | The billing and account substrate; the scarce input; §27.4, §36.5 |
| Judge | The instrument that scores an attempt; the source of §31's 3× bias |
| Kit | A bundled harness plus corpus, sold with updates; §28.1 |
| KYC | Identity verification at the provider; the control that moves the price of §32.4 |
| MTBF of the inventory | Mean time between failures of the thing you are selling; §35.2 |
| Prefill | Assistant-prefix control; the single most valuable API property; §27.2 |
| Prompt pack | A text file of instances with no metadata; §29.1 |
| Refusal rate per identity | The cumulative metric that no mutation defeats; §40.2 |
| Reseller | Archetype selling access to someone else's capability; §26.1 |
| Session budget | Sessions per identity per day before termination; §32.4 |
| Silent moderation | Outputs hedged after the model complied, without a refusal; §35.1 |
| Suite (the operator's) | The regression run that reports whether the business still works; §29.4 |
| Treadmill | The recurring maintenance load that caps scale; §35 |
| Warm-up | Benign sessions to establish an identity before the first prohibited one; §32.4 |
| Wrapper | A consumer product over a rented model, priced like an app; §37.2 |
43. Number card, operator edition
Same discipline as §22: every figure is reported under the reporting source's
conditions, or derived here with the arithmetic shown. The derived rows are
marked.
| Figure | Value | Source | Condition |
|---|---|---|---|
| Stolen premium chatbot account | $0.10 | Halcyon, via Axios, Oct 2026 | underground price sheet |
| Single working jailbreak prompt | $0.32 | Halcyon | same |
| Agent-jailbreak kit, subscription | $50-$200/month | AgentMarketCap, Jul 2026 | with updates and support |
| Full synthetic identity | up to $6,000 | Halcyon | same |
| Underground AI-tool ads | <50/month → >1,400/month | Halcyon | Oct 2025-May 2026, 4,000 posts |
| Uncensored models in the wild | 11,598 of 17,104 | Lin et al., 2026 | model graph |
| Unsafe-request compliance of those models | 74.1% | Sokhansanj, 2025 | profiled set |
| Flagged as malicious by the platform | 16.52% (1,916 models) | Lin et al., 2026 | platform classification |
| Open-weight originals vs redistributions | 3,471 vs 8,164 | 10a Labs, Sept 2026 | redistribution study |
| Apps built on them | 1,643, of which 25% explicitly malicious | 10a Labs | same |
| Uncensored V4.1 Flash forks | 31 repos, 83,900 downloads | reported Oct 2026 | first days after release |
s1ngularity blast radius |
2,349 credentials, 1,079 developers, 2,180 accounts, 7,200 repos | Wiz, Sept 2025 | npm nx 20.9.0-21.8.0 |
postmark-mcp installs |
~1,500/week | Snyk / Postmark, Sept 2025 | until discovery |
| Hermes campaign scope | 460+ targets, 7 exploit tracks, 3 confirmed exploitations | Unit 42, Jul 2026 | recovered session |
| GTG-1002 tactical share | 80-90% | Anthropic, Nov 2025 | ~30 organisations |
| Injection exposure in production | 73% of deployments | 2025 industry survey | self-reported |
| Organisations that feel prepared | 29% | same | self-reported |
| Scam losses, Southeast Asia, 2025 | $88.3B-$114.1B | UNODC, 2026 | regional assessment |
| US federal AI-CSAM cases | 71 (2022-2025) | Stanford CIS / Edinburgh, 2026 | catalogue |
| Derived: reseller net before labour | ~$1,222/month, ~$15/hour | §37.2 | 200 subs at $15, 3,000 msgs |
| Derived: product operator net | ~$25,700/month, ~43% margin | §37.3 | 3,000 subs at $20, one H100 |
| Derived: broker net | ~$2,700/month, ~90% margin | §37.4 | 20 customers at $150 |
| Derived: search-loop share of revenue | 67%, or 135% with one retry | §37.1 | $20 wrapper, 3,000 msgs |
| Derived: self-hosted cost per 1M tokens | $0.17 (8B, batched) to $13.90 (70B, single stream) | §33.3 | H100 at $2.50/h |
Derived: judge precision at p = 0.05 |
32%; measured p inflated to 14% |
§31.2 | 90%/90% judge |
| Derived: attempts surviving at 1% detection | 37% at 100 attempts; 0.4% at 1,000 | §36.3 | binomial |
| Derived: cost of a recon battery | ~$0.01 (cheap model) to ~$0.20 (frontier) | §27.6 | 40 probes, 900 in / 150 out |
| Derived: cost of a weekly regression run | ~$0.10, 40 minutes | §29.4 | 200 instances, judged |
| Derived: cost per success at 500 req/min | $0.006 | §30.3 | p = 0.05, $0.15/M |
44. Source notes for Part VI
Ordered thematically, named by publisher and date, no links. Where a figure
appears in only one source it is treated as contested in the text, per §0's
conventions. Everything here is either already cited in §23 or in BLACKHAT.md
§13, or is arithmetic performed in this part on those figures - there is no new
evidence in Part VI, only a new direction of reading.
Market structure and prices
Halcyon Ransomware Research Center, underground AI-as-a-service study:
4,000 posts, 77 Telegram channels, 20 dark-web forums, 5 specialist markets,
window October 2025 to May 2026, reported by Axios in October 2026. The source
of the $0.10 / $0.32 / $6,000 price sheet, the 1,400-ads-per-month figure, and -
most usefully - the negative result that there is no mature market for a fully
autonomous attack agent. The negative result is the load-bearing one for §37.
Rapid7, "Criminal AI-as-a-Service in 2026". FraudGPT, GhostGPT and the
WormGPT lineage: prices, instability, and the tendency of brands to outlive the
products. Weak on volume; strong on the shape of the supply.
AgentMarketCap, July 2026. Agent-kit subscriptions at $50-$200/month. One
source, so treated as an order of magnitude in §28.1 rather than a price.
Supply chain
Wiz, s1ngularity post-incident, September 2025, with BleepingComputer
coverage and the npm advisory for nx 20.9.0-21.8.0. The credential and repo
counts in §28.2. The technique - malware that asks an installed coding agent for
secrets - is the single most transferable fact in Part VI.
Snyk and Postmark, postmark-mcp, September 2025. ~1,500 weekly installs, a
hidden BCC, discovery by diff. The tool-description channel.
Lin, Li, Liao, Wang, "Consiglieres in the Shadow" (arXiv 2508.12622), 2026.
11,598 uncensored models in a 17,104-model graph, 212 malicious services, 16.52%
flagged. The best available measurement of the uncensored supply.
10a Labs, "Uncensored Open-weight Models: Redistribution as the Persistence
Layer" (arXiv 2609.05241), September 2026. 3,471 originals, 8,164
redistributions, 1,643 applications, 25% malicious. The redistribution clock in
§28.4.
Sokhansanj, 2025, 8,608 uncensored models profiled, 74.1% compliance with
unsafe requests. Cited in the above; used here only for the compliance figure,
because the sampling frame is not described well enough to support more.
Operations and incidents
Unit 42 (Palo Alto Networks), DeepSeek and Hermes Agent Telegram campaign,
460+ targets, 7 exploit tracks, 8 CVEs, three confirmed exploitations - reported
July 2026, summarised by The Hacker News in August 2026. §38.3's type specimen
and §5.2 of BLACKHAT.md. The report is internally inconsistent about the
exfiltration claims and the inconsistency is flagged rather than smoothed.
Anthropic, GTG-1002 disclosure, November 2025: roughly 30 organisations,
80-90% of tactical work performed by the model, human contribution concentrated
in strategy, target selection and the authorisation story told to the model.
IEEE Spectrum editorial follow-up. The authorisation detail is the one Part VI
uses; it is the highest-leverage "jailbreak" in the record and it is not a
prompt.
Cisco Security with the University of Pennsylvania, DeepSeek R1 evaluation,
January-February 2025: 100% attack success across 50 HarmBench prompts. Read
with §14: 50 prompts, one model, one turn, no defence stack.
KELA Red Team, early 2025: the "Evil" and "Leo" jailbreaks on R1 shipped
with infostealer scripts and explosives instructions. The first documented case
of the jailbreak being the payload.
Measurement and enforcement
UNODC, 2026 Southeast Asia transnational-crime assessment: $88.3B-$114.1B in
2025 scam losses, franchised structure, AI and crypto in the stack. Used in
§37.6 only for scale.
Stanford CIS with the University of Edinburgh, 2026: 71 catalogued US
federal AI-CSAM prosecutions 2022-2025, with a 2026 appendix. The enforcement
record in §38.1; the Cyprus and Ohio cases are from reporting, not from this
catalogue.
Nasr et al., "The Attacker Moves Second", October 2025: 12 of 12 defences
broken, most above 90% ASR. Cited in §15 and §16; used in §41 as the date the
market's claims stopped being credible.
Arithmetic
Every figure in §43 marked Derived is computed in the section named, from
the price sheet above and §5.7's dollar-per-million-token convention. None of
them is averaged across sources. The two that matter most - the amortisation law
in §37.1 and the judge-precision result in §31.2 - are one-line arithmetic and
should be checked by the reader rather than trusted.
45. Stage index
Alphabetical by stage, with the section that treats it and the control that
attacks it.
| Stage | Section | Weakest link | Control |
|---|---|---|---|
| Access and identity | §27.3 | payment instrument | identity issuance, spend caps |
| Attribution (theirs) | §38 | billing lineage | subpoena, not security |
| Corpus | §29 | metadata discipline | nothing model-level; patch cadence |
| Delivery | §34 | marketplace landlord | platform terms enforcement |
| Detection (theirs) | §36 | cumulative behaviour | per-identity accounting |
| Economics | §37 | amortisation | consumer price points |
| Harness | §30 | quota | rate limits, anomaly detection |
| Judging | §31 | base-rate bias | use their optimism, not their number |
| Maintenance | §35 | breakage-to-fix | ship updates, vary detection |
| Reconnaissance | §27 | none; it is cheap | make the deployment uninteresting |
| Sessions | §32 | context cost | trajectory monitoring |
| Sourcing | §28 | supply-chain trust | provenance, flagging redistributions |
| Weight-space | §33 | capex and base-model cadence | provenance, distribution policy |
The "weakest link" column is the answer to the only question a budget holder
asks: which one of these, if I spend money on it, changes the attacker's
behaviour this quarter. Four rows have an answer that is cheap and known -
identity, quota, delivery, maintenance - and every one of them is outside the
model.
End of document. Parts I-V are the defence survey; Part VI is the same evidence
read from the operator's side. Companion: BLACKHAT.md for the adversary
economy, README.md for the harness, HOW_TO_USE.md
for operating it.