How to Stop AI Agents From Losing Their Memory

On 2026-05-20, I opened GitHub commit 328e671, ran the reference ceiling audit, and found the real memory test was not in the chat window. It was in companies/vibecode-town/code-index.md, <artifact-root>\vibecode-reference-writing-audit\latest.json, src/data/publication-approvals.json, and wiki_fts.db.

The next agent did not need a pep talk. It needed receipts.

Without those receipts, every handoff becomes a public approval risk and a time leak.

The expensive failure in long-running agent work is not forgetting a fact. It is forgetting the status of a fact.

Was that source real or just a summary? Was the feature done or only planned? Did the previous agent run the verifier, or did it only say the verifier should exist?

When a resumed session cannot answer those questions, the operator starts repeating the same corrections. The prompt becomes a junk drawer. The next agent inherits confidence without provenance.

This is why long prompts are not operating memory. They can carry context for a while, but they do not prove what happened, what changed, or what remains unsafe.

The wrong standard is “does the next agent have enough context?” The useful standard is “can the next agent prove which source, command, approval record, and remaining-work queue are current without trusting the previous chat?”

The Broken Default

The bad default is to make the prompt bigger.

At first, it works. You tell the model the policy, paste the previous decision, add the reference link, and remind it not to invent proof. But once the work stretches across days, the prompt becomes a meeting note, a spec, an incident log, a style guide, and a task queue all at the same time.

Two things break.

First, humans stop reviewing it. Important decisions and temporary instructions live in the same wall of text, so the operator skims the exact material that should constrain the work.

Second, agents stop retrieving it cleanly. They cannot tell which line is source evidence, which line is interpretation, and which line is a stale assumption from a previous session.

A long prompt is not an audit trail. It is unstructured state.

The trap is that long prompts feel responsible. They are full of warnings, decisions, links, and apologies. But when everything is in one scroll, nothing has an owner. A durable rule and a late-night guess sit next to each other wearing the same uniform.

Before and After

Here is the before/after that changes the work.

Before: The restart surface was a chat scrollback. The agent had to infer which instruction was still active, which correction was frustration, and which sentence was a durable rule. That feels fast until the next session confidently repeats a solved mistake.

After: The restart surface is an artifact chain:

source packet -> plan/report -> code-index.md -> wiki_fts.db -> archive receipt -> approval record

That chain is slower to write than a single prompt, but it is much faster than explaining the same project for the fifth time.

For this site, the memory receipt is not “the agent said it updated the wiki.” The receipt is closer to this:

python .\reindex_wiki.py
Indexed 333 markdown files into <wiki-root>\wiki_fts.db

.\archive_completed_artifacts.ps1
archive_files_copied=365
source_markdown_count=333
archive_markdown_count=333

That is the difference between memory as a vibe and memory as an operating surface. One asks the next agent to believe. The other gives it a place to look.

The cost is not abstract. If the next agent says “wiki is current” while the index still says 245 files, the reviewer has to audit the story before auditing the work. If the public body changed but the approval record still points at yesterday’s markdown, the page is not approved. It is wearing an old receipt.

The rule is blunt: if memory cannot be reindexed, searched, archived, or contradicted by a hash, it is not operating memory. It is just a nicer prompt.

Operating Memory Stack

Agent work needs a small Operating Memory Stack.

The short version is: source, spec, handoff, index.

raw source
processed source note
spec
handoff
search index
remaining-work queue

Raw source is the preserved input: transcript, log, command output, diff, research link, support ticket, or field note.

The processed source note is not a summary for humans. It extracts reusable pressure:

what the source changes
what it does not prove
what rule it implies
where it can mislead the next agent

The spec is the repeatable contract. It says what the system must keep doing even when a different agent enters the repo.

The handoff is the current state. It should tell the next session what passed, what failed, and what must not be treated as complete.

The index makes the memory searchable. The remaining-work queue turns documents back into action.

In this repo, the stack is concrete:

companies/vibecode-town/sources/raw
companies/vibecode-town/sources/processed
companies/vibecode-town/plans
companies/vibecode-town/code-index.md
<wiki-root>\wiki_fts.db

The verification side is concrete too: python reindex_wiki.py, archive_completed_artifacts.ps1, check_vibecode_completion_audit_sync.ps1, and check_company_artifacts_archive.ps1.

That is the useful test for memory. If the next agent cannot name the file, command, or receipt that proves the state, it is probably carrying a story, not memory.

The current proof packet keeps that standard uncomfortable. On 2026-05-21, the wiki index checked 347 markdown files, the archive copied 379 files, the public site checked 9 non-About posts, and the rendered audit checked 24 viewports before the latest article state stayed public.

Prompt, Skill, Wiki, or Memory

Not every instruction belongs in the same place.

OpenAI’s Conversation state and Compaction docs describe the practical reality: context is managed, summarized, and bounded. That is normal. It also means durable operating memory cannot live only inside a chat window.

MCP Resources point in the same direction. If tools and agents need reusable context, that context should be addressable and explicit.

A Usable Handoff Shape

A useful handoff is not a diary. It is a restart surface.

Goal:
Current status:
Verified evidence:
Files changed:
Commands run:
Known blockers:
Do not repeat:
Next action:
Completion is not proven until:

The last line matters. Without it, the next session tends to convert partial progress into a finished story.

Here is the difference in the current Vibecode workflow.

Bad restart surface:

We improved the blog and fixed the gates. Continue from there.

Usable restart surface:

Goal: make public posts source-backed, English, image-backed, and approval-bound.
Recent status: product commit dd3565e is pushed.
Verified evidence: verify:site-quality passed; 10 posts, 24 rendered viewport checks, 10 approval records.
Files changed: publication review manifest, rendered audit script, image rule gate, reference-writing audit.
Known boundary: the writing floor is much stronger, but some posts still need sharper openings and artifacts.
Next action: improve the lowest-scoring post, then refresh approval record, wiki index, and archive checks.
Completion is not proven until: site-quality, wiki reindex, and archive checks pass again.

The packet trail behind that handoff is also countable:

packet-backed public posts: 9
required packet files per post: 6
total packet files: 54

That is what the next agent can use without re-reading the whole chat. It can tell which work is complete, which work is only a qualitative weakness, and which command has to prove the next change.

The point is not to make the next agent obedient. The point is to make it falsifiable. A good handoff lets the next agent say, “that is stale,” before it wastes two hours politely continuing a lie.

Audit Checklist: Accept / Reject

Before accepting an agent’s memory claim, use this decision matrix.

The rule is simple: accept artifacts, reject vibes.

Forward this to the agent lead who says the chat summary is enough for the next session. The decision is narrow: can the next agent cite a current file, count, hash, and command receipt, or is it only inheriting a confident story about what happened?

If you do not have a matrix yet, start with these questions:

Is the original source outside the chat?
Is the processed note reusable?
Is the spec separate from the prompt?
Is the latest handoff current?
Can the agent search the memory?
Is remaining work tracked as a queue?
Are unverified product claims marked as unverified?

If three of those are missing, the issue may not be model quality. It is missing operating memory.

Vibecode uses this pattern for source-backed content: source notes, explicit specs, current handoffs, searchable wiki indexes, and evidence gates before Field Logs.

The public writing is only the visible surface. The real asset is the memory contract behind it.

The same rule applies to visual proof. A memory diagram in the body is useful only if the image rule agrees with it: one slug-specific image, matching ogImage, visible in rendered post checks, and not reused as a lazy placeholder across unrelated posts. Otherwise the image becomes another fake memory.

Use this as the approval line: accept a memory claim only when it has a source file, current index receipt, archive receipt, approval record, and next-action queue. Reject it when the claim only sounds complete.

Boundary

Operating memory does not make a model truthful. It does not remove the need for review. It does not prove the work is complete.

It lowers the repeated context tax and makes the next agent’s starting point inspectable. Agent systems should not depend on vibes, hidden state, or optimistic prompts. They should run on evidence, handoff, and verifiable boundaries.

Before handing work to the next agent, ask one final question: “What would prove this summary wrong?” If the answer is “nothing, it is just in the chat,” the system does not have memory yet. If the answer is a file path, count, hash, command, and queue item, the next agent can start without pretending.