New Method Tests AI Companion Agents More Fairly
Researchers solve a key problem in AI companion testing: simulated users are too cooperative. A disclosure-gate mechanism now measures real engagement.
👁 112 reads
A paper from arXiv (cs.CL, cs.AI, cs.HC) submitted September 1, 2026 by Yao Liu and Yu He addresses a specific methodological problem in evaluating companion agents - AI systems designed to engage users in conversation.
The Problem with Current AI Companion Testing
Evaluating AI systems increasingly relies on another language model acting as a simulated user - this allows tests to scale without involving real people. The authors call this approach “now standard in scalable evaluation.” The problem, however, is well known and - as they write - “repeatedly diagnosed”: the simulated user is overly cooperative. In practice, this means the tested system can achieve high scores not because it actually encourages the conversation partner to open up, but simply because it asks many questions. The artificial user will answer anyway.
What Is a “Disclosure Gate”
The authors propose a mechanism called a disclosure gate. Its principle: information from the simulated user is released gradually, depending on the tested agent’s behavior. The simulator state is described as “a ladder of five ordered gates, merged into three observable depth layers.” The gate behavior is learned from a synthetic branch of the training corpus, while the real branch provides patterns of how people speak and react. After training, the simulator does not need to be told at runtime which gate contains a given conversation element - it decides independently.
How the Authors Verify This
Tests were conducted on the English corpus of the published CompanionBench benchmark, covering 12 tested systems. The authors set two acceptance criteria: the ranking must be order-preserving, and absolute scores must be scale-stable. Of the candidates examined, only one meets both criteria - the simulator the authors are releasing. Its correlation with the benchmark’s original simulator is 0.993.
For comparison: using an off-the-shelf frontier model as a simulator “barely shifts the ranking, but shifts every score upward” - which the authors describe as a shift invisible to someone checking only the ranking. This is an important warning: you might not notice the error if you only look at system order.
The authors also check their solution’s stability after removing gate information from training data. The largest ranking shift among the 12 systems then exceeds the noise band determined by re-running the environment with a new random seed - but individual system scores show no detectable change. This distinction between ranking stability and absolute score stability is one of the article’s key threads.
Relationship to CompanionBench
The authors explicitly note that the environment they specify is the same one already used by the benchmark CompanionBench. That publication describes the mechanism in about four hundred words. This article provides what was missing: specification, ablation, human studies, negative controls, and downstream sensitivity analysis.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →What This Means
In our view, this article solves a real, practical problem: evaluating AI companions is difficult because it’s hard to measure whether a system truly builds conversation partner trust. The disclosure-gate mechanism is a thoughtful answer - instead of measuring agent activity, it measures the simulated user’s willingness to open up. The high 0.993 correlation with the original simulator is a good sign. What’s concerning, however, is what the authors say about frontier models as simulators: they shift scores upward in a way invisible in rankings - a subtle error that could go unnoticed in many evaluation projects. This article’s value also lies in supplementing an existing benchmark with documentation it was lacking.
Source: arXiv cs.AI: Disclosure-Gated User Simulation for Companion-Agent Evaluation (document from 2026-09-02). This overview is based on the full document text, not other media reports.
More on this from QUANTSLANT
state of ai·Mathematical Ontologies Correct LLM Hallucinations - StudyA neuro-symbolic pipeline using OpenMath ontology improves LLM accuracy on math tasks - but only when retrieval quality is high, arxiv study shows.