Context continuity: why capable agents still lose the thread

Context continuity is the ability of an AI system to preserve, carry forward, and selectively reconstruct the state needed to continue work across conversation, context-window, application, model, machine, and collaborator boundaries.

The phrase is useful, but it is not yet a settled academic category with one standard benchmark. We use it to distinguish a system property from three narrower ideas: the tokens currently inside a model's context window, a store that can retrieve similar passages, and a summary of an earlier interaction. Each can contribute to continuity; none guarantees it alone.

Continuity does not mean replaying everything. It means preserving enough evidence that the system can assemble the smallest trustworthy context for the work in front of it.

That distinction matters because a model can be highly capable within one call and still be operationally amnesiac over the life of a project. The intelligence is present; the state management around it is missing.

The obvious response to forgetting is a larger context window. Research shows why that is incomplete. Lost in the Middle found a pronounced positional effect: models often performed best when relevant information was near the beginning or end, and worse when it was buried in the middle—even when the model was explicitly built for long context.[1]

RULER widened the test beyond finding one needle. It added multiple needles, multi-hop tracing, and aggregation. Almost all tested models degraded as input length and task complexity grew; only half maintained the benchmark's satisfactory level at 32K despite every model claiming at least that window size.[2] The nominal window is capacity, not a promise that every token will influence the answer equally.

LongMemEval reaches the same problem across time instead of one prompt. Its 500 questions test five abilities a useful memory system needs: extraction, reasoning across sessions, temporal reasoning, knowledge updates, and abstention. Tested long-context and commercial systems still showed substantial degradation over sustained histories.[3]

Compaction and summaries remain valuable. Anthropic describes compaction as a first lever for long-horizon coherence, while warning that aggressive compression can remove subtle details whose importance only becomes clear later.[6] The design implication is not “never summarize.” It is “never make the summary the only surviving account.”

Earlier agent-memory work points toward a hierarchy rather than one giant prompt. MemGPT treats active context like RAM backed by external storage that can be paged in through tools.[4] Generative Agents keeps a comprehensive memory stream, derives higher-level reflections, and retrieves memories using relevance, recency, and importance.[5] Anthropic's current guidance expresses the same resource constraint directly: useful context is the smallest set of high-signal tokens that improves the desired outcome.[6]

Pantheon is a continuity layer for desktop AI agents, not a model and not another chat client. Every handoff starts in auto mode: MCP reads a session record only through a verified adapter for the calling client; otherwise the agent supplies the real turns still visible in its context. Pantheon stores that episode as a named project conscience in a GitHub or Google Drive vault controlled by the user. MCP gives different agent hosts a common tool surface for handing off, resuming, searching, and opening evidence.[7]

The central design decision is archive first, distil second. The packet that orients the next agent can remain short because the underlying sessions and files survive outside it. If the packet is insufficient, the agent has stable ways to inspect what it summarized.

This is deliberately file-shaped. Plain text gives humans and agents the same inspectable substrate. Git adds version history and transport without turning Pantheon's application database into the owner of the conversation. Conversation content stays in the user's repository; the web application stores account, pairing, sharing, and token metadata.

The design is also model-neutral. A conscience is not serialized into one vendor's proprietary memory API. Any connected MCP client can receive the same orientation and call the same retrieval tools, subject to what that host supports.

1. 1

   Capture the episode

   On handoff, Pantheon tries verified local capture first and automatically requests the fullest transcript visible to the agent only when capture is unavailable, then archives new turns without duplicating the stored prefix.

2. 2

   Build a bounded orientation

   It writes a session packet and rebuilds the space-level handoff from all sessions: totals, recent spoken turns, earlier packet summaries, session references, and artifact names.

3. 3

   Version the state

   The changed files are committed and pushed to the user’s private vault. A later handoff appends another episode or another tail; it does not flatten the project into one mutable recap.

4. 4

   Resume with a map

   The receiving agent gets the packet, durable memory, every session’s identity and size, artifact names, and promoted skill triggers. This creates orientation without spending the window on the entire archive.

5. 5

   Retrieve the evidence

   For every natural-language historical question—after resume or directly with /pantheon-retrieve—the agent ranks bounded connected evidence, verifies exact surrounding turns, and cites the archive. This includes requests for unknown names, files, commands, or quotes; deterministic search is reserved for literal text already supplied or discovered.

The packet is not the memory. It is the index card attached to the memory. The transcript, decisions, files, and version history are what make its claims reopenable.

Pantheon applies ideas from TopoCompress selectively rather than treating the paper as a replacement for its archive, resume packet, or structured memory. TopoCompress frames long-context compression as selecting coherent evidence spans: score them for lexical and semantic relevance, connect sequentially or semantically related spans, propagate relevance through that graph, and remove redundancy under a limited context budget.[8]

Pantheon adapts that pattern as a query-time evidence selector. The source transcript is never compressed away. pantheon_retrieve first ranks exact, lexical, and typo-tolerant matches; expands around neighboring turns; builds a bounded candidate graph from sequential, lexical, and vector similarity; propagates relevance; then applies a redundancy penalty while fitting evidence into an explicit token budget. Every selected span retains its stable session_id#turn_id citation.

A managed embedding interface is available for future semantic encoders, but the current default is model-free. This keeps retrieval private, predictable, and deployable through the CLI and web application without a multi-gigabyte model download. Learned embeddings should be enabled only after they improve Pantheon-specific evidence recall enough to justify their infrastructure, latency, and privacy costs.

Context continuity is infrastructure, not a guarantee of correct reasoning. A receiving model can ignore a clue, search poorly, or apply an old decision to a changed situation. Pantheon improves what can be carried forward and inspected; it does not make the model infallible.

• Capture is limited by the host record

  Pantheon can archive only what a supported local session or the agent's current context contains. Some clients keep full turns; others omit tool results or have already dropped older messages. Pantheon reports that boundary rather than inventing missing evidence.

• Browser-only agents lack full-fidelity capture

  A remote MCP server cannot read a browser client's private conversation log. The agent supplies every real turn still visible to it, so handoff works without user action but may be truncated.

• Semantic retrieval is not yet learned

  Ranked retrieval combines lexical relevance, typo-tolerant local feature vectors, neighboring turns, and graph propagation. It does not yet use a deployed BGE-M3 or comparable learned embedding index, so distant paraphrases can still require a focused follow-up query.

• Distillation remains a judgement

  A packet and a proposed skill are interpretations made by a model. The raw episode remains available beneath the packet, while skills stay inert drafts until a human promotes them.

• A Git vault is controlled, not magical

  Users can rewrite or delete their own repository, and access security still depends on GitHub and machine credentials. User ownership makes those controls explicit; it does not remove the need to secure them.

The next useful research step is evaluation at the handoff boundary: measure whether a different agent can recover the correct decision, evidence, artifact, and unresolved work after multiple sessions—not merely whether a stored sentence is retrievable. LongMemEval's separation of extraction, temporal reasoning, updates, and abstention offers a useful starting vocabulary for that work.[3]

Models will gain larger windows, and compaction will improve. Neither removes the architectural need to decide what persists, where it lives, how it is updated, and how another agent can recover the evidence without ingesting an entire project history.

Pantheon's answer is intentionally concrete: a loss-minimizing archive, a compact orientation layer, granular retrieval, model-neutral MCP tools, and a private Git repository the user owns. In short: preserve the thread, disclose it progressively, and make every summary reopenable.

Research was reviewed from primary papers and official protocol or engineering documentation. Claims about Pantheon describe the current implementation rather than results from an independent benchmark.

1. [1]
   Lost in the Middle: How Language Models Use Long Contexts

   Liu et al. show that retrieval and question-answering performance can change sharply with the position of relevant information, often degrading when it appears in the middle of long inputs.

2. [2]
   RULER: What’s the Real Context Size of Your Long-Context Language Models?

   The COLM 2024 benchmark evaluates 17 models on 13 retrieval, multi-hop, and aggregation tasks; despite claimed windows of at least 32K, only half maintained satisfactory performance at 32K.

3. [3]
   LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

   Wu et al., ICLR 2025, test information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention across sustained histories.

4. [4]
   MemGPT: Towards LLMs as Operating Systems

   Packer et al. frame long-running memory as a hierarchy: a finite active context backed by external storage that can be paged in through tool calls.

5. [5]
   Generative Agents: Interactive Simulacra of Human Behavior

   Park et al., UIST 2023, combine a comprehensive memory stream with retrieval, reflection, and planning to support behavior coherent with earlier experience.

6. [6]
   Effective context engineering for AI agents

   Anthropic’s Applied AI team describes context as a finite resource with diminishing returns and recommends compaction, structured memory, and just-in-time retrieval for long-horizon work.

7. [7]
   Model Context Protocol specification

   MCP defines an open client-host-server protocol through which applications can discover and invoke tools and retrieve external context.

8. [8]
   TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories

   Asante and Li propose training-free extractive compression using dense and lexical query relevance, sequential and semantic graph edges, relevance propagation, and redundancy-aware span selection.