MEMCUT Benchmark Shows Agent Memory Deletion Leaves Secrets Recoverable

Hong Kong University of Science and Technology (Guangzhou), together with the University of Science and Technology of China and HKUST, introduced MEMCUT, a benchmark designed to evaluate whether large-model agents truly forget sensitive information after a user requests deletion. The benchmark proposes two complementary metrics: native recovery, which measures what the agent can retrieve through its own memory retriever, and residual recovery, which measures whether the underlying evidence still supports reconstructing the secret.

Built through fully manual logical design, the dataset spans 15 multi-session scenarios over one to three months, containing 43,402 dialogue turns and protecting 456 sensitive secrets. Each secret is linked to an average of 8.1 implicit evidence pieces distributed across different sessions, forming evidence chains that can be pieced together to reveal the original secret.

Across evaluations covering five memory architectures and four large models, residual recovery rates reached 85%-99% when only direct disclosure records were deleted. Even with LLM-assisted cleanup of related records, 39%-76% of secrets remained inferable. The study also found that as cleanup intensity increased, the hidden-risk ratio grew in many configurations, meaning lower surface-level answers do not guarantee that underlying data has been erased. The core bottleneck was identified as insufficient retrieval coverage: existing semantic-similarity retrievers fail to reach implicit evidence that is logically related but differently worded.

MEMCUT shifts evaluation from direct memory recall to the derivability of multi-hop implicit evidence chains, offering a standardized diagnostic tool for building trustworthy, privacy-compliant agents.