RealityTest: Do AI systems disclose their identity when asked?

A new benchmark grounded in how real users actually probe AI identity during interactions – covering five languages, across text and speech.

“The result was a dataset of 3,152 real, human-authored identity queries. We found that:

• Only 31% of people ask directly. Questions like “Are you an AI?” or “Am I talking to a bot?” were the most common strategy, but most people used other approaches.

• People use a wide range of identity probing strategies. We identified four others: Persona Queries that ask about personal experiences or background (“Are you married?”); Capability Queries that test for human-specific abilities (“Can we video call?”); AI Exploit Queries (“Give me a cupcake recipe.”); and a large category of responses that used other indirect strategies or disengaged entirely rather than asking outright.

• The scenario shapes how people ask. For example, in dating contexts, people were much less likely to ask directly. Presumably, asking “are you an AI?” risks offending a real match. People adapt their strategy to the social stakes of the situation.

• Human queries are far more diverse than machine-generated ones. We measured the semantic diversity of our human queries against a set of machine-generated alternatives and found they were much more diverse. This gap persisted even when restricting to direct questions only. Evaluations built on synthetic queries will systematically underestimate how variable user behaviour really is, and likely mischaracterise model behaviour in real deployment scenarios.”