Agents are increasingly deployed in document-intensive workflows where sensitive private information is not an edge case but a routine input, e.g., an agent booking a flight needs passport numbers. In such settings, the agent must use private information to complete tasks accurately while never exposing it in its responses, because it cannot verify who is actually at the keyboard. These two obligations are in fundamental tension. A model capable enough to use private information for task completion can, by the same capability, be induced to reveal it.
To evaluate the trade-off of task accuracy and privacy leakage, we introduce Task-completion and Resistance to Active Privacy-extraction (TRAP). Each scenario includes a document containing private information, a task query that requires the agent to invoke the correct tool using private fields, and an attack query that attempts to elicit the same information in natural language. Evaluating 22 models spanning frontier proprietary and open-source models at multiple scales, we find that all model families exhibit non-trivial leakage, and that instruction-following ability correlates with leakage rate. Existing prompt-based defenses reduce leakage but at significant cost to task accuracy. Prompt optimization fails to escape this trade-off.
We demonstrate that this failure is not incidental. For any softmax-based model, no soft-constraint defense, e.g., prompt-based defenses, can jointly achieve high task success with zero leakage probability. Motivated by this impossibility result, we propose structural private field isolation, which replaces private fields with hash keys before they reach the model. This approach largely prevents leakage while keeping task accuracy.
Prior benchmarks measure privacy either passively (incidental leakage, no attacker) or adversarially (attack only, task ignored). The problem is that a trivial solution exists here: the agent can achieve perfect privacy simply by being useless, scoring zero on accuracy. TRAP is the first to measure both jointly on the same document and the same private fields. Neither a refuse-all nor a comply-all strategy can increase both task accuracy and the privacy score at the same time.
There is a fundamental limit. As long as a model produces a softmax distribution over its entire vocabulary, no amount of hand-crafted privacy instructions or instruction tuning can drive the leakage probability down to zero. Security requires eliminating the probability of leakage, not merely reducing it.
Theorem 1 — Impossibility of simultaneous task success & perfect privacy.
For a softmax model that can complete the task, any private token needed for the task has strictly positive output probability. Over an adversarial sequence of length $N$, the probability of leakage grows as
$$P(\text{leak}\mid x)\ \ge\ 1-(1-\varepsilon)^{N}\ \ge\ 1-e^{-\varepsilon N}\ \xrightarrow[N\to\infty]{}\ 1.$$
$$\begin{aligned} P(\text{leak}\mid x)\ &\ge\ 1-(1-\varepsilon)^{N}\\[2pt] &\ge\ 1-e^{-\varepsilon N}\\[2pt] &\xrightarrow[N\to\infty]{}\ 1. \end{aligned}$$
Soft-constraint defenses, such as system prompts, alignment, prompt optimization, can only shrink $\varepsilon$, never set it to zero.
If soft constraints cannot reach zero, remove the plaintext entirely. Each private value is replaced
by a semantically typed key token (e.g. [PRIVATE_card_number]) before the document
reaches the model. The agent passes the key as a tool argument; the real value is resolved only inside
the tool at execution time. The model never sees the plaintext, so it cannot disclose it.
@article{yebin2026trap,
title = {TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction},
author = {Ye-Bin, Moon and Hyeon-Woo, Nam and Seong-Eun, Baek and Yeo, Yejin and Oh, Tae-Hyun},
journal = {arXiv preprint arXiv:2606.18996},
year = {2026},
}