PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin, Yuhao Chen, Haotian Zhang, Chao Zhang, Derong Xu, Jia Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, Tong Xu

arXiv 2026arXiv

In brief

PERMA is a benchmark for personalized memory agents built on how preferences actually form: gradually, across related events, inside noisy interaction histories. Instead of needle-in-a-haystack recall of static preferences, it arranges temporally ordered interaction events across sessions and domains, adds text variability and linguistic alignment to mimic erratic real-world inputs, and probes persona consistency with both multiple-choice and interactive tasks. Memory systems that link related interactions extract preferences more precisely at lower token cost than semantic retrieval over raw dialogues — but they still struggle with temporal depth and cross-domain interference.

Key takeaways

• Real personalization is evolutionary: preferences emerge gradually and accumulate across interactions, so PERMA evaluates persona consistency over time rather than static preference recall.
• Temporally ordered interaction events span multiple sessions and domains, with preference-related queries inserted along the timeline.
• Text variability and linguistic alignment simulate erratic user inputs and individual idiolects found in real-world data.
• Both multiple-choice and interactive tasks probe an agent's understanding of the persona along the interaction timeline.
• Advanced memory systems that link related interactions extract more precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues.
• Even advanced systems struggle to maintain a coherent persona across temporal depth and cross-domain interference — robust personalized memory management in agents remains open.

Abstract

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

BibTeX

@misc{perma,
  title     = {{PERMA}: Benchmarking Personalized Memory Agents via {Event-Driven} Preference and Realistic Task Environments},
  author    = {Liu, Shuochen and Zhu, Junyi and Shu, Long and Lin, Junda and Chen, Yuhao and Zhang, Haotian and Zhang, Chao and Xu, Derong and Li, Jia and Tang, Bo and Li, Zhiyu and Xiong, Feiyu and Chen, Enhong and Xu, Tong},
  year      = {2026},
  eprint    = {2603.23231},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2603.23231},
}