Luna/Documentation/References/ZeroClaw/Memory.md

Memory

ZeroClaw implements a hybrid memory system that combines traditional keyword search with modern vector similarity. This approach ensures that exact matches (like function names or specific terminology) and semantic matches (concepts and related ideas) are both discoverable.

---

Memory Trait

The foundation of the memory system is the Memory trait. Any backend implementation must satisfy this interface to be used by the system.

pub struct MemoryEntry {
    pub id: String,
    pub key: String,
    pub content: String,
    pub category: MemoryCategory,
    pub timestamp: String,
    pub session_id: Option<String>,
    pub score: Option<f64>,
}

pub enum MemoryCategory {
    Core,
    Daily,
    Conversation,
    Custom(String),
}

#[async_trait]
pub trait Memory: Send + Sync {
    fn name(&self) -> &str;

    async fn store(
        &self,
        key: &str,
        content: &str,
        category: MemoryCategory,
        session_id: Option<&str>,
    ) -> anyhow::Result<()>;

    async fn recall(
        &self,
        query: &str,
        limit: usize,
        session_id: Option<&str>,
    ) -> anyhow::Result<Vec<MemoryEntry>>;

    async fn get(&self, key: &str) -> anyhow::Result<Option<MemoryEntry>>;

    async fn list(
        &self,
        category: Option<&MemoryCategory>,
        session_id: Option<&str>,
    ) -> anyhow::Result<Vec<MemoryEntry>>;

    async fn forget(&self, key: &str) -> anyhow::Result<bool>;

    async fn count(&self) -> anyhow::Result<usize>;

    async fn health_check(&self) -> bool;
}

---

SQLite Backend

The SqliteMemory implementation serves as the primary persistent backend. It uses rusqlite and is tuned for high performance and reliability in a local environment.

Performance Tuning

The backend initializes with specific PRAGMAs to optimize for concurrent access and speed:

conn.execute_batch(
    "PRAGMA journal_mode = WAL;
     PRAGMA synchronous  = NORMAL;
     PRAGMA mmap_size    = 8388608;
     PRAGMA cache_size   = -2000;
     PRAGMA temp_store   = MEMORY;",
)?;

• WAL mode: Enables concurrent reads even during write operations.
• mmap (8MB): Allows the OS to handle hot reads through memory mapping.
• temp_store MEMORY: Ensures temporary tables never touch the disk.

---

Schema Design

The system maintains a relational table for core data and a virtual table for full-text search (FTS5). Triggers keep these in sync automatically.

-- Core memories table
CREATE TABLE IF NOT EXISTS memories (
    id          TEXT PRIMARY KEY,
    key         TEXT NOT NULL UNIQUE,
    content     TEXT NOT NULL,
    category    TEXT NOT NULL DEFAULT 'core',
    embedding   BLOB,
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL,
    session_id  TEXT
);

-- FTS5 virtual table for keyword search
CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5(
    key, content, content=memories, content_rowid=rowid
);

-- Sync triggers
CREATE TRIGGER IF NOT EXISTS memories_ai AFTER INSERT ON memories BEGIN
    INSERT INTO memories_fts(rowid, key, content)
    VALUES (new.rowid, new.key, new.content);
END;

---

Hybrid Search

The recall method executes a hybrid search strategy. It merges results from two distinct search mechanisms:

1. Keyword Search: Uses FTS5 BM25 scoring for exact word matches.
2. Vector Search: Computes cosine similarity between query embeddings and stored memory embeddings.

Score Fusion

Results are combined using a weighted average. The defaults are typically:

• Vector Weight: 0.7
• Keyword Weight: 0.3

If vector results are unavailable (e.g., if embeddings are disabled), the system falls back to keyword-only search. If both high-level search mechanisms return no results, it uses a final LIKE %query% fallback to ensure maximum recall.

---

Embedding Cache

To avoid redundant API calls to embedding providers, ZeroClaw uses an internal LRU (Least Recently Used) cache stored in SQLite.

CREATE TABLE IF NOT EXISTS embedding_cache (
    content_hash TEXT PRIMARY KEY,
    embedding    BLOB NOT NULL,
    created_at   TEXT NOT NULL,
    accessed_at  TEXT NOT NULL
);

Whenever a memory is stored or a query is processed, the system checks this cache first using a deterministic content hash. Eviction occurs once the cache reaches its configured limit (default 10,000 entries).

---

Backend Implementations

ZeroClaw supports several backend types configured via Configuration:

1. SQLite: The standard persistent local backend using FTS5 and BLOB embeddings.
2. PostgreSQL: Used for distributed or cloud-hosted deployments (requires pgvector).
3. Lucid: A bridge backend that synchronizes local SQLite memory with remote services.
4. Markdown: A simple file-based implementation that stores memories as .md files in the workspace.
5. Qdrant: A dedicated vector database backend for high-scale semantic search.
6. None: An explicit no-op backend that disables all memory persistence.

---

Timeout Guards

Database operations are protected by timeout guards. Specifically, opening a SQLite connection is capped at 300 seconds to prevent the system from hanging on locked or slow filesystems.

const SQLITE_OPEN_TIMEOUT_CAP_SECS: u64 = 300;

---

Relevance to Luna

ZeroClaw's memory architecture serves as the blueprint for Memory. While Luna currently has an IMemoryStore interface, the existing file-based implementation lacks sophisticated recall.

Adopting the ZeroClaw pattern provides:

• Structured Categories: Using MemoryCategory (Core, Daily, Conversation) for better context management.
• Hybrid Recall: Moving beyond simple file reading to a ranked keyword+vector search.
• SQLite Reference: Luna can adopt the SqliteMemory implementation directly to replace the current unstructured storage.

Cross-links: Core, Configuration