Benchmark finds AI agents remember conversations but struggle to infer user habits

Anyone who uses an AI assistant over many months might reasonably expect it to pick up on their routines. A new benchmark suggests that the memory systems built into today's AI agents are poorly equipped to do so. In a preprint posted to arXiv, which has not yet been peer reviewed, researchers at Shanghai Jiao Tong University report that the best of eight memory methods they tested chose the right answer on habit-related questions just 33.61% of the time, averaged across four domains.

The study concerns agents built on large language models (LLMs), the systems behind modern chatbots, which increasingly carry out tasks for the same person across many separate conversations. Over time, users reveal habits through what they choose and do, rather than by stating them outright. The difficulty, the authors argue, is that any single interaction may offer only partial evidence. It might hint at a habit without making clear whether it really is one, or when it applies. Agents typically rely on memory systems, which store, summarise or retrieve past exchanges so that they can be used later. The authors argue that keeping a record of the past does not guarantee that an agent can work out what that record says about a person.

To measure the difference, the team built HABIT-Bench. It uses synthetic user histories assembled from task-oriented dialogue data, meaning conversations in which someone is trying to get a specific job done. The researchers combined this data with what they call controlled habit graphs, which let them specify the habits underlying each simulated user. Each history spans hundreds of sessions across four domains: food, finance, software and travel. Agents then face four-option multiple-choice questions that test whether they can infer a habit and apply it appropriately to a new request. The questions deliberately include awkward cases. Some involve temporary exceptions that should not be mistaken for a new pattern, some involve habits that change over time, and some involve suggestions the user has not accepted. Independent human reviewers checked a sample of the tasks.

The researchers tested eight memory methods paired with two LLMs, Qwen3-235B-A22B and GLM-5.3-Flash. The top performer reached 33.61% macro accuracy. Macro accuracy is calculated separately for each domain and then averaged, so every domain counts equally. For context, with four options on each question, random guessing would be expected to succeed roughly a quarter of the time. The authors also report that performance remained limited even when the relevant annotated evidence was handed to the model directly, rather than left for the memory system to find. They conclude that there is a persistent gap between having access to past interactions and being able to infer and appropriately use a user's habits.

That last finding suggests that better retrieval alone is unlikely to close the gap. The harder problem appears to be reasoning from scattered, weak clues. Because the user histories are synthetic, it also remains to be seen how closely these results reflect the habits of real people working with real agents.