Agent Memory Engine (3/10) — memcore Completed: 18 Modules, 3,300 Lines

20 Tables, 8 CLI Commands, 295 Curated Rows, and Graceful Degradation

ํ•ต์‹ฌ ์š”์•ฝ

  • memcore is a library that rebuilds a file-based memory pipeline on top of a single SQLite file. 18 modules, approximately 3,300 lines.
  • 8 CLI commands cover the full operational lifecycle, each schedulable with a single cron entry: migrate / lint / warn / decay / wiki-lint / stats / backfill-vectors / ontology-sync.
  • The core design principle is graceful degradation — when a dependency is absent, functionality degrades rather than the system halting.

What You Will Take Away

  • Module partitioning and table design for a SQLite single-file agent memory engine
  • Hybrid prefetch strategy that falls back to FTS5 when vector search is unavailable
  • Pattern of delegating memory lifecycle operations (decay, validation, cleanup) to CLI commands and cron
  • How to decouple memory implementation from the host runtime using the MemoryProvider interface

Scope — Differences from the Initial Build

The initial version of memcore included 9 tables, 51 tests, and a migration path from the file-based bank/ structure. The "completed form" covered in this article adds 20 tables, a full CLI command set, a vector search module, and ontology synchronization. In other words, the target is a version where operational lifecycle and fallback strategies are embedded inside the engine itself, not just the storage layer.

Module Structure (18 Modules)

# Module Responsibility
1 core SQLite connection, WAL mode, schema initialization
2 ingest retain-extract / retain-merge unification
3 prefetch FTS5 hybrid search (topic → FTS5 → LIKE)
4 dialectic U-tag 3-phase (observe → hypothesize → verify)
5 decay opinions confidence decay (0.02/day, remove < 0.30)
6 lint Retain tag format validation
7 entities Entity staleness detection (30 days)
8 topics Topic registry + M2M relationships
9 bank_migrate bank/ → SQLite conversion (READ-ONLY, --incremental)
10 decisions LLM decision queue (TOPIC_CLASSIFY, CONFLICT_RESOLVE, CHANGELOG_SUMMARIZE)
11 archive memory/ → archived/ relocation
12 housekeeping recall TTL, session cleanup, stale orphan removal
13 vectors sqlite-vec + bge-m3 embeddings (optional)
14 wiki Karpathy LLM Wiki pattern (topic page CRUD)
15 wiki_lint 7 wiki checks (contradiction / stale / orphan / gap / size / citation / frontmatter)
16 ontology CLAUDE.md → DB one-way cache (agent / relation / persona layers)
17 promotion Cascade promotion (local → lessons → global)
18 stats Statistics / health check

Responsibilities are partitioned in the following order: storage (core) → write (ingest/promotion) → read (prefetch) → cleanup (decay/housekeeping/archive) → validation (lint/wiki_lint) → auxiliary (vectors/wiki/ontology) → observability (stats). Each module has no knowledge of another module's internals; it shares only the connection object and public schema exposed by core.

20 Tables

Group Table Purpose
Meta meta Schema version, configuration
Knowledge curated, curated_fts Core knowledge + FTS5 index
Topics topics, topic_curated Topic registry + M2M mapping
Entities entities Per-project state
Episodes episodes Daily log
Identity identity MEMORY.md cache
U-tag u_patterns, u_observations, u_hypotheses, u_verifications Dialectic 3-phase
Decisions decisions LLM decision queue
Wiki wiki_pages, wiki_sources, wiki_log Karpathy wiki
Vectors vec_curated sqlite-vec embeddings (optional)
Ontology ont_agents, ont_relations, ont_layers CLAUDE.md cache
Promotions promotions local → lessons → global history

The tables split into three conceptual layers: fact-recording layer (curated, episodes, entities, identity), reasoning-process layer (u_patterns family, decisions, promotions), and auxiliary index layer (curated_fts, vec_curated, topic_curated). The schema ensures the fact-recording layer remains independently queryable even when the auxiliary index layer is empty.

8 CLI Commands

memcore migrate          # bank/ → SQLite conversion
memcore lint             # Retain tag + data integrity check
memcore warn             # Memory warning report
memcore decay            # Run opinions confidence decay
memcore stats            # Statistics / health check
memcore wiki-lint        # 7 wiki checks
memcore backfill-vectors # Bulk sqlite-vec embedding generation
memcore ontology-sync    # CLAUDE.md → DB synchronization

The CLI is a mechanism for pushing maintenance tasks that can be decoupled from the runtime to the outside. Operations like decay, lint, and stats must run periodically but do not need to run inside the conversation loop. Delegating them to cron/launchd reduces the runtime's responsibility to prefetch and ingest only. The system gains a self-maintenance cycle without an orchestration framework.

Current Data State

Item Count
curated rows 295
topics 15
entities 4
topic_curated links 514
Distribution knowledge 150 / pattern 28 / daily 17 / world 14 / identity 13 / experience 10 / opinion 4

The fact that topic_curated links (514) are approximately 1.74× curated rows (295) indicates that each item is linked to an average of 1–2 topics. Multi-topic linking, not single tagging, is the default form. The distribution shows knowledge-type entries accounting for over half. The low opinion count is indirect evidence that decay is operating as intended, removing low-confidence entries.

Core Design Principle — Graceful Degradation

memcore is designed so that code paths remain valid when a dependency is absent.

  • No sqlite-vec: falls back to FTS5 text search. Semantic search degrades to keyword search, but the path remains.
  • No bge-m3: vector build step is skipped. prefetch operates on FTS5 alone.
  • No CLAUDE.md: the ontology cache is empty; no other module is affected.
  • No wiki pages: wiki_lint returns "0 pages checked" and exits normally.

The practical advantage of this approach is tolerance for installation profile variance. The full-spec configuration is Apple Silicon + oMLX + sqlite-vec; the minimum configuration requires only Python + SQLite. Deploying the same codebase across different environments reduces branching install scripts and replaces them with in-runtime feature flags.

Interface with the Host System — MemoryProvider

memcore couples to the host runtime exclusively through the MemoryProvider abstract interface. The implementation is MemcoreProvider(MemoryProvider).

  • prefetch → memcore.prefetch.search
  • on_session_end → memcore.ingest
  • on_memory_write → memcore.promotion (passes Tier gate)
  • system_prompt_block → injects top-N curated entries into context

The host runtime has no knowledge of memcore's existence. It only needs to know the method signatures of the interface. This boundary ensures the host code does not change when the storage layer is replaced or upgraded. The same principle applies to vector backend replacement (sqlite-vec → Qdrant/FAISS) or memory engine experimentation (running two implementations in parallel A/B).

Limitations and Applicability

  • Single-file SQLite: appropriate for agent scenarios with low concurrent write volume. Not suited for multi-writer workloads.
  • Embedding cost: backfill-vectors is optimized for batch processing. Streaming or real-time embedding requires a separate path.
  • One-way ontology cache: reverse synchronization from DB back to CLAUDE.md is out of scope. Markdown is the source of truth.
  • CLI-driven operations: environments without cron/launchd (e.g., pure serverless) require an external scheduler.

The applicable scope can be summarized as: "local execution, single writer, agent environments with high dependency variance that require an observable and stable memory store."

Open Questions

  • How should the consistency model be defined when multiple agents reference the same memcore file concurrently?
  • Should the decay and promotion policy constants (0.02/day, removal threshold < 0.30, etc.) adapt based on corpus size?
  • If the storage layer is replaced with a non-SQLite backend (e.g., key-value store + external FTS) while preserving the MemoryProvider contract, which features degrade?

3,300 lines represents the scale of "one year's accumulated memory judgment logic redistributed onto single-file storage." The substance of the file-based → SQLite transition is not a format change. It is the ability to reduce each operational lifecycle step to a single CLI invocation and delegate it to an external scheduler.

Series overview: Series index

๋Œ“๊ธ€

๋Œ“๊ธ€ ์“ฐ๊ธฐ

์ด ๋ธ”๋กœ๊ทธ์˜ ์ธ๊ธฐ ๊ฒŒ์‹œ๋ฌผ

Agent Memory Engine (2/10) — Building an AI Agent Memory System with SQLite Alone

← 1/10 SQLite Memory Engine f… ๐Ÿ“š Series Index 3/10 memcore Completed → A memory engine that runs without daemons, without dependencies, anywhere ํ•ต์‹ฌ ์š”์•ฝ This post covers how to design an AI agent memory engine (memcore) on a single-file SQLite backend. - Structure: 25 modules, ~6,464 lines, 22 DB tables - Principles: no daemon, single-file portability, no required external dependencies, host-agnostic - Core techniques: 4-layer hybrid search (topic · vector · FTS5 · LIKE), U-tag dialectic-based confidence evolution, ingestion-layer bias rejection rules What This Post Covers One approach to building AI agent memory without a server or cloud vector DB. Specifically: (1) the limitations of text-file-based memory, (2) vector DB alternatives compared and selection criteria, (3) how to layer search tiers, (4) how to evolve confidence scores over time, and (5) how to block biased memory accumulation at the ingestion stage. Design Principles: Why Single-File SQLite Text-file-based ...
Read more »

"ML Foundations (9/9) — PyTorch vs TensorFlow, and the Road to Local LLMs"

์ด๋ฏธ์ง€
← 8/9 Deep Learning Architec… ๐Ÿ“š Series Index (series end) The final part. So far we've focused on models. Now we focus on the tools that actually run them . We'll lay out the philosophical difference between PyTorch and TensorFlow, and trace how Hugging Face transformers , llama.cpp , MLX , and Ollama built the bridge to running large language models on your own machine. By the end you should have the full mental model of "download a pretrained LLM and serve it locally." 0. Learning Objectives Explain the eager-vs-graph difference between PyTorch and TensorFlow. Explain, in graph terms, how autograd automates the backward pass. Use Hugging Face transformers ' AutoModel , AutoTokenizer , and pipeline abstractions. Describe llama.cpp's GGUF quantization, INT4 inference, and CPU-first flow. Describe Apple MLX's use of unified memory and how it differs from PyTorch. Run a local LLM with Ollama and call it through an OpenAI-compatible API. ...
Read more »

"RAG Core Study (14/26) — Evaluation Sets with RAGAS & DeepEval"

์ด๋ฏธ์ง€
Series overview: Series index ๐Ÿ“š Series Index Without an evaluation set, every RAG improvement is a story, not evidence. RAG systems fail in more than one place: retrieval, context selection, answer generation, and grounding. That means "looks better" is not a metric. Part 14 explains how to build a golden dataset , how RAGAS and DeepEval fit into the workflow, and why teams must separate retrieval quality from answer quality if they want tuning decisions to hold up. 0. Prerequisites Part 13 reranking — system quality now depends on multi-stage ranking. Part 1 grounding — a correct-looking answer is not enough. Part 12 Hybrid — multiple retrieval settings need comparison on fixed data. 1. Learning Objectives Build a small but useful golden dataset for RAG. Distinguish retrieval metrics from answer metrics . Use RAGAS and DeepEval in the right roles. Avoid the common traps in judge-model-based evaluation. 2. ํ•ต์‹ฌ ์š”์•ฝ An evaluation set for RAG is ...
Read more »

"ML Foundations (8/9) — Deep Learning Architectures: CNN, RNN, Attention"

์ด๋ฏธ์ง€
← 7/9 Deep Learning Training ๐Ÿ“š Series Index 9/9 PyTorch vs TensorFlow → What did we keep adding on top of an MLP? Vision went CNN, sequences went RNN/LSTM, and eventually both converged on Attention and the Transformer. Each architecture is easier to remember if you read it as what it gave up to gain something else . This part is the one-line summary of the decisive inflection points: LeNet → AlexNet → ResNet → LSTM → Attention → Transformer. 0. Learning Objectives Explain why CNN's convolution, pooling, and weight sharing match images so well. Trace what got unblocked from LeNet → AlexNet → VGG → ResNet as depth grew. State BPTT for RNNs and the vanishing/exploding gradient problem. Write the LSTM gate equations (forget, input, output) with intuition. Write the attention formula in query/key/value form and the scaled dot-product variant. State the precise reasons Transformers replaced RNNs (parallelism + long-range dependencies). 1. ํ•ต์‹ฌ ์š”์•ฝ CNN : weight sharin...
Read more »

"ML Foundations (7/9) — Deep Learning Training: Optimizers, Regularization, Initialization"

← 6/9 Neural Networks ๐Ÿ“š Series Index 8/9 Deep Learning Architec… → Building an MLP, as we did in Part 6, is not the same as getting it to train well . The right optimizer, regularization, initialization, and learning rate — when these line up, deep networks converge. When they don't, the network refuses to learn at all. This part is the catalog of those crafts, with formulas, paper citations, and code in one place. 0. Learning Objectives Compare and write the update rules for SGD, Momentum, Nesterov, and Adam. Explain how Dropout, BatchNorm, and LayerNorm work and where they belong in a model. Derive the variance formulas for Xavier and He initialization and match them to activations. Implement step, cosine, and warmup learning-rate schedules in PyTorch. Explain why gradient clipping is effectively required for RNNs and Transformers. Diagnose the most common training failures (NaN, plateau, overfitting) and apply first-line fixes. 1. ํ•ต์‹ฌ ์š”์•ฝ SGD : \(w \leftarro...
Read more »

OpenClaw to Hermes Migration (2/13) — What to Preserve, Partially Port, or Discard

← 1/13 Current Structure Inve… ๐Ÿ“š Series Index 3/13 What Moves to Hermes → A code review record classifying operational scripts into three tiers: preserve, partial-preserve, and discard. What This Post Covers An asset classification framework (preserve / partial-preserve / discard) for migrating a legacy agent system to a successor A concrete case applying the Strangler Fig + Subset Migration strategy to a memory pipeline Classification results evaluated across LOC, external dependencies, and replaceability axes A method for slimming down a bloated single file (1,310 LOC) using the migration as a natural entry point Problem Definition: Port Everything or Cut Deep? The OpenClaw inventory comprises multiple agents, multiple scheduler/daemon processes, and multiple operational scripts. When moving to the successor system, Hermes, the choice is not binary. A full port carries accumulated technical debt along with it; a minimal port discards hard-won domain logic. Code-revie...
Read more »

AI Agents I Built (5/7) — Building an Automated Blogger API Publishing System

← 4/7 My Life Organizer Agent ๐Ÿ“š Series Index 6/7 Dual → Designing a Markdown-Based Auto-Publish Pipeline with Blogger API v3 + OAuth 2.0 Key Summary Blogger API v3 + OAuth 2.0 enables a pipeline that converts Markdown files into publishable HTML posts blogger_publish.py handles the full flow — frontmatter parsing → body cleanup → HTML conversion → CSS injection → API post — in a single script For bulk publishing, quota management (delay, retry, TOC suppression) is the critical design concern Platform Selection Criteria API access is mandatory for AI agent-driven content publishing. A comparison of platforms: Platform API Cost AdSense Decision WordPress.com REST API Paid plan required Paid only Cost overhead Ghost REST API Self-hosted or paid Manual setup Ops overhead Blogger v3 API Free Native integration Adopted Blogger is free, exposes a REST API, and integrates natively with AdSense. For personal blog automation, this combination i...
Read more »