Memory Is A Bandwidth Problem

A UX Collective piece treats AI chat recall failure as an architectural mistake: messaging-app patterns applied to knowledge work, RAG layers as industry admission that the original architecture was insufficient, sixty years of HCI research describing the missing piece. The framing is implementation, not invention.

I think the framing understates something. There are fundamental information theory constraints that show up as memory problems in any sufficiently large or longitudinal system. Landauer’s principle puts a minimum energy cost on bit operations. The speed of light bounds how fast bits move between storage and processor. Associative memory architectures have capacity limits relative to unit count. The tradeoff curve between latency, capacity, and bandwidth is fixed. Engineering shifts position along it.

Persistent memory and direct file access don’t dissolve the problem. They relocate it. The retrieval function still has to decide what to surface, and that function is itself bandwidth-constrained. Embedding similarity, ranking, summarization, all inherit the same physical constraints as the storage layer they sit on top of.

We have an illusion that information is accessible at uniform latency. It isn’t, and never has been. Search engines, file systems, our own memories, all use ranking, caching, tiering, prefetching to produce the feeling of uniform access. The cold tail gets pushed progressively further from the processor. Items that don’t rank high effectively don’t exist at retrieval time, even when they’re the right fact. We mistake comprehensiveness for accessibility.

Human memory works the same way. Reconstruction at retrieval time costs less than maintaining everything in fast storage. Forgetting does work. Borges’ Funes is the limit case where total recall becomes a form of cognitive disability.

LLMs hit this acutely. Attention is the retrieval mechanism inside the model, and attention scales quadratically with context length. Push memory outside the model and the bottleneck moves to the retrieval function over the external store. The bits still have to move. The constraint is a property of high complexity information systems, not a property of any particular implementation.

So the architectural critique in the article is correct as far as it goes. AI chat did inherit the wrong shape for knowledge work. Persistent, addressable, retrievable artifacts are the right next move. But the move doesn’t escape the underlying constraint. It produces a better tradeoff position, with new manifestations of the same problem at the new scale.

The oldest message in my AI chat history that I’d want back tomorrow is probably findable. The one I’d want back in five years, across millions of messages, almost certainly isn’t, no matter what architecture ships. That’s not a UX failure. That’s physics.