Adding Memory to LLMs and Agents

Read-only Memory

Read-only Memory

Adding memory to LLMs and agents let’s you do a few interesting things:

1. Create agents with infinite memory for past conversations.

2. Read documents in chunks.

3. Have the assistant remember user preferences.

Inspired by the MemGPT paper from over one year ago, I build an LLM system with read-only local memory - allocated within the context.

This is the first in a two part series - in the next part I’ll look at how to add read/write memory to store user preferences and allow for reading documents in chunks (actually, as Cursor does for reading long files).

Enjoyed the video or have Qs? Drop a comment below or reply to this email with a 🤙.

Cheers, Ronan

🛠 Explore Fine-tuning, Inference, Vision, Audio, and Evaluation Tools

💡 Consulting (Technical Assistance OR Market Insights)

🤝 Join the Trelis Team

💸 Grants Program

---

Adding Read-Only Memory to LLMs and LLM Agents

Large language models (LLMs) can be enhanced with memory systems that allow them to access information beyond their context window. This video examines how to implement a read-only memory system that enables an LLM to retrieve and reference past conversations.

Core Memory Architecture

The system consists of two main components:

1. Local memory: The LLM's context window, limited to a fixed token count (e.g., 2,500 tokens)

2. Disk memory: A database storing the complete conversation history

The local memory is further divided into:

1. System message (500 tokens)

2. Read-only retrieval block (500 tokens)

3. Recent chat history (remaining tokens)

Memory Management System

The implementation uses a First-In-First-Out (FIFO) approach:

1. Recent conversations stay in local memory until pushed out by newer ones

2. All conversations are saved to the disk database

3. The LLM can query past conversations through search commands

Search and Retrieval

The system implements a paginated search mechanism:

1. LLM issues search commands using XML-style tags: <fetch_memory>query</fetch_memory>

2. Results are returned in pages of 3 conversation turns

3. LLM can request subsequent pages using <fetch_memory_page>2</fetch_memory_page>

4. Search uses keyword matching (can be upgraded to BM25 or embeddings)

Technical Implementation Details

Context Management:

1. Total context = 2,500 tokens

2. System message = 500 tokens

3. Read-only memory = 500 tokens

4. User/Assistant messages = 250 tokens each

5. Remaining space allocated to recent chat history

Database Structure:

1. Conversations stored in JSON format

2. Each entry includes:

   1. User message

   2. Assistant response

   3. Timestamp

   4. Token count

Search Implementation:

1. Case-insensitive keyword matching

2. Results grouped into conversation turns

3. Pagination with 3 turns per page

4. Search state maintained for multi-page retrieval

Command Processing

The system processes LLM commands through:

1. Regular expression extraction of search queries

2. Token counting for context management

3. Result formatting and injection into read-only memory

4. Confirmation messages back to the LLM

Future Enhancements

The system can be improved through:

1. Postgres implementation with BM25 search

2. Vector embeddings for semantic search

3. Date-based retrieval

4. Rate limiting and throttling

5. Multi-user support

6. Authentication

Technical Requirements

The implementation requires:

1. Python environment

2. Anthropic's Claude API

3. JSON for storage (or Postgres for production)

4. Regular expressions for command parsing

5. Token counting utilities

This memory system enables LLMs to maintain conversational context beyond their standard context window while maintaining a clean, modular architecture that can be extended for more complex use cases.