CUSTOMER MEMORY VS. A GENERAL-PURPOSE LLM
When a general-purpose LLM is enough — and when it stops working
Everyone asks it eventually: why not point Claude or ChatGPT at customer calls, interviews, tickets, and reviews? Here’s where each approach genuinely wins—with the numbers.
USE A GENERAL PURPOSE LLM
Small set of documents, a question you'll ask once? Use Claude or ChatGPT. Under roughly 30 documents the whole set fits in one context window, the model reads every word, and you'll get a better answer than any memory system will give you. No setup, no cost, no us.
We'd rather say that plainly than sell past it. Everything below is about what happens when those conditions stop holding — which, for most teams, is sooner than they expect.
First, the objection everyone gets wrong
“It won’t fit in the context window.”
That is not the problem, and repeating it makes the rest of the argument easy to dismiss. A capable model can fan out—one sub-agent per call, ticket, or response—and read everything in parallel, at any volume. Every item genuinely gets read, which even beats retrieval at counting.
What actually breaks, and what it costs
Four things go wrong when raw data is re-read on every question—and all four get worse as the archive grows.
Three things that never work, at any size
Those four problems are problems of scale. These three don’t depend on scale at all—they fail at five calls exactly as they fail at five thousand.
No memory between sessions
“What changed at this account since our last conversation?” needs state. A general-purpose model starts cold every time.
No knowledge of your business
Taxonomy, product aliases, segments, and CRM context must be re-explained in every prompt—unless memory carries it forward.
No consistency between answers
Ask the same question twice and the judgment calls get made again. Tags applied once make trends and counts real.
Where the line falls
It’s not a size limit. It’s volume × how often you ask—and both flip together.
A general-purpose LLM is the right call when
The whole set fits in one context window. The question is one-off. Someone can supply business context each time. Nobody needs the answer to be reproducible six months from now.
The archive keeps growing. Questions come back. The question has a time dimension. The work should run unattended. Answers must carry your taxonomy, segments, and product mapping automatically.
The objections, answered straight
These come up in every evaluation. Here are the real answers.
“Context windows keep getting bigger.”
They do, and it doesn’t help. Cost per question, the concurrency ceiling, the aggregation step, and the lack of state are untouched by a bigger window.
“Isn’t memory less accurate than raw text?”
On a one-off question over a small set—yes. Compression trades some coverage for speed, cost, and repeatability: a decisive trade at 1,000.
“Models keep getting cheaper.”
Cheaper tokens shrink both sides equally. Reading everything on every question stays linear and permanent; memory is paid once at ingestion.
“We don’t want another tool to be locked into.”
The memory is yours—normalized, tagged, and callable by whatever needs it. That is the point of a layer rather than a feature.
THE HONEST SUMMARY