PIDS-Bench: Why Prompt Injection Detectors Fail in Real Use

👁 114 reads

A new benchmark published in IEEE Access reveals that prompt injection detectors perform far worse in real-world conditions than their test scores suggest - even when they achieve near-perfect accuracy on standard datasets.

The authors - Yusuf Khalid Shire and Sang-Chul Kim - published PIDS-Bench in IEEE Access (vol. 14, pp. 134184-134205, 2026), a benchmark that evaluates prompt injection detectors under significantly harder conditions than those used in standard testing. The paper runs 21 pages and contains 3 figures and 18 tables.

The Problem with Current Evaluation Methods

Prompt injection detectors are typically evaluated using a single metric - aggregated F1 score on test data drawn from the same distribution as the training data. The authors point out that this approach hides a critical issue: what happens to legitimate queries that the detector incorrectly classifies as attacks? False positives have direct operational costs, yet they are rarely measured. PIDS-Bench is described as “frozen,” meaning its contents do not change - allowing different systems to be compared on identical data.

What PIDS-Bench Measures

The benchmark evaluates detectors along multiple axes simultaneously, using fixed decision thresholds. The dataset includes four categories of inputs: data from the same distribution as training, “hard-benign prompts” (queries that mimic attack structure but have no malicious intent), obfuscated attacks, and data with domain and structural distribution shifts. Seven detectors were evaluated: fine-tuned base models, external prompt injection classifiers, and broader safety classifiers, with a simple rule-based detector used as a baseline.

Key Findings

The core finding is striking: a detector achieving F1 above 0.98 on an internal test set still misclassifies approximately one-third of an external subset of benign queries. This external subset comes from public datasets and is limited to security-related content. The authors tested the full range of decision thresholds across five different training seeds - no internal detector achieved both F1 >= 0.95 and a false positive rate on hard-benign queries of FPR <= 0.10 on this challenging distribution.

Data Provenance Matters

The authors discovered a pattern they call “provenance-sensitive over-defense.” Adding hard negative examples to training (hard-negative augmentation) almost completely eliminates the over-blocking problem on carefully curated test examples, but does not solve it for data from external sources. This pattern appears in both tested fine-tuned model architectures and does not weaken as the augmentation data pool grows - FPR on external data remains well above the 0.10 target. The authors explicitly note they did not test whether augmentation matched to the external data distribution would close this gap. Threshold calibration and curation-style augmentation alone are not sufficient.

What This Means in Practice

In our view, this work highlights a concrete practical problem: if you deploy a prompt injection detector and evaluate it only on your own test data, you may have a false sense of security. An F1 score close to 1.0 does not protect you from the system blocking every third normal user query when it encounters content from a different domain. The benchmark is frozen and publicly available, meaning creators of new detectors can use it as a common reference point - valuable because until now, everyone measured their system differently. At the same time, the authors themselves acknowledge they did not test the most obvious solution: augmentation matched to the external data distribution, so it remains unclear whether the problem is even solvable through that approach.

Source: arXiv cs.AI: PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift (document dated 2026-09-15). This analysis is based on the full text of the paper, not media reports.

// read moreGemini 4 Argon Prompts: 12 Tested ExamplesBenchmarksAI Model API Pricing: Real Costs per 1,000 PromptsBenchmarksAI Detector: How It Works, Why It Fails, and What to DoGuides

Kinetyra