EMNLP 2026 Main ×2 — presenting in BudapestACL Main ×2ICLRTACLEACL Findings5 of 7 first / co-first author
Agents that survive real interactions
Role-playing and tool-using agents are usually evaluated against a frozen persona and a cooperative user. Real characters evolve, real users don't cooperate, and real tool schemas don't match what models saw in pretraining. This thread builds the benchmarks and methods for that gap.
EMNLP 2026 MainCo-first author
ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
Characters evolve as the story progresses — benchmarks should test whether agents evolve with them.
An automatically constructed benchmark of 17 novels and 80 principal characters. A Character Arc segments each narrative into psychological phases, and each probe poses the same scenario across phases — including situations the source text never explores. Across six models and six context strategies, arc-conditioning wins on every model, with the largest gains exactly where retrieval has nothing to find; fine-tuned ArcANE-8B/32B widen that advantage further.
Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models
Small models hallucinate plausible-but-nonexistent tool names; renaming the schema fixes it without any training.
PA-Tool uses peakedness — a pretraining-familiarity signal from contamination detection — to rename tool-schema components toward what the model already knows. Up to +17% tool-use accuracy on MetaTool and RoTBench, with schema-misalignment errors cut by 80%, entirely training-free.
User simulators are agent-friendly; users are not.
A simulator architecture covering four families of non-collaborative behavior — impossible requests, digressions, impatience, incomplete utterances — while still reliably delivering task intents. State-of-the-art tool agents degrade sharply on MultiWOZ and τ-bench under these conditions, with escalated hallucination and dialogue breakdown. Released as an extensible stress-testing framework.
A growing literature characterizes LLM values and personality with human questionnaires. This thread shows why that measurement stack is broken — contaminated items, profiles that don't predict behavior — and builds psychometrically valid replacements.
EMNLP 2026 MainCo-first author
Human Psychometric Questionnaires Mischaracterize LLM Behavior
Questionnaire profiles and actual generation behavior diverge — the instrument, not the model, is stable.
Across eight open-source LLMs, value/personality profiles from PVQ and BFI self-reports disagree substantially with profiles derived from generation probabilities on realistic user queries. Questionnaire items leak the target construct through lexical cues, inviting socially desirable answers; persona prompts move questionnaire scores in human-like ways yet leave real generation behavior untouched. Proposes generation-based profiling as the reliable alternative.
Value Portrait: Assessing LLM Values through Psychometrically and Ecologically Valid Items
A value benchmark whose items are validated against real humans and drawn from real user–LLM interactions.
Each item is rated by human subjects and kept only when ratings correlate with the raters' own measured values. Evaluating 44 LLMs shows a consistent profile — Benevolence, Security, and Self-Direction prioritized; Tradition, Power, and Achievement undervalued — plus systematic biases in how models perceive demographic groups.
Quantifying Data Contamination in Psychometric Evaluations of LLMs
Popular inventories aren't just familiar to models — models can hit target scores on them at will.
A three-level framework — item memorization, evaluation memorization, target-score matching — applied to 21 models and four widely used inventories. BFI-44 and PVQ-40 show strong contamination: models recognize the items and can steer their responses to produce requested psychological profiles.
Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
LLM virtual respondents can replace costly human studies for validating survey items — if you model the mediators.
The same trait yields different answers through different mediating factors; simulating respondents across diverse mediators identifies items whose responses correlate robustly with the intended trait. Validated on Big Five, Schwartz values, and VIA character strengths, opening a cost-efficient path for survey development.
The two threads are converging: the measurement stack from one side becomes the evaluation harness for the agents on the other.
ArcANE × generation-based profiling
Behavior-grounded evaluation of persona agents
Apply generation-based psychometric profiling to deployed role-playing agents: measure whether a character's expressed values drift over long interactions, rather than whether it recalls facts about itself. Character consistency as a measurement problem, not a retrieval problem.
Non-collaborative simulators × Value Portrait
User simulators with real value profiles
Give user simulators psychometrically grounded value and personality profiles, so agents are stress-tested against realistic human diversity — not one cooperative average user. Toward population-level agent evaluation.
Virtual respondents × contamination auditing
A contamination-proof measurement stack for frontier models
Generate items with mediator-aware virtual respondents, audit them against the contamination framework, and profile models from behavior rather than self-report — evaluation instruments that models cannot have memorized and cannot game.