Stanford Builds Platform to Track How People Actually Use AI

👁 118 reads

A Stanford-led research initiative called the AI Observatory has found that widely cited AI usage reports from companies like Anthropic and OpenAI omit nearly half of real user conversations, with the filtered-out exchanges containing significantly higher rates of sensitive and harmful content than published figures show.

The project, co-led by Anka Reuel, a computer science PhD candidate at the Stanford Trustworthy AI Research (STAIR) Lab, aggregated and analyzed real AI conversations collected with users’ consent across seven existing datasets. Conversations span interactions with models including Claude and Gemini between 2023 and 2025. The goal is to give researchers and policymakers an independent source of data rather than relying solely on what AI companies choose to publish.

“There is no independent source to corroborate it.”

The gap between company reports and real usage turns out to be substantial. The Anthropic Economic Index - one of the most widely cited sources on AI usage - focuses on work- and productivity-related conversations, filtering out anything unrelated. When the AI Observatory team applied Anthropic’s methodology to their own dataset, 48% of conversations were excluded. Those filtered conversations were far more likely to involve health and relationships (44.2% versus 31.2% in Anthropic’s analysis), harassment and hate (27.5% versus 5.66%), adult or illicit topics (7.9% versus 2.1%), and sexual content (16.7% versus 2.4%). For context, OpenAI’s own 2025 report on ChatGPT found that only 30% of consumer use was work-related.

Patterns shifting over time

Beyond the snapshot differences, the AI Observatory identified notable trends across time. Within WildChat - one of the largest datasets in the study - conversations grew longer and more elaborate, with rising numbers of prompt tokens, response tokens, and conversation turns. Small talk also increased markedly, pointing to a rise in companionship-style use. At the same time, AI self-disclosure - instances of a chatbot acknowledging it is not human - declined.

Sensitive exchanges, defined as those involving potentially harmful or restricted content including sexual harassment and hate speech, became less frequent over the study period. Researchers suggest this may reflect platforms deploying more effective content safeguards.

David Widder, an assistant professor at the University of Texas at Austin who researches human-AI interaction and is not involved with the AI Observatory, said having a unified “bird’s-eye-view analysis” rather than information “sectioned off into a separate report” helps researchers understand different use patterns more consistently. Highly consequential decisions about AI’s benefits and risks are currently being made on the basis of very limited data, Reuel says.

// read moreStanford Uses Large Genome Models to Design Novel BacteriophagesNewsGTA 6 on PC, PS4, and Xbox Series S: Platforms and Release DateNewsHidden AirTag Tracks Rare Books to Amazon AI Facility in VegasNews

Kinetyra