Overview
ReliBot (The Hartford) is an incident-response assistant for SRE and development teams.
- Engineers ask it questions in Teams, the CLI or a web chat, such as "why is claims-api throwing 500s?".
- It pulls Splunk logs and Dynatrace problems, checks recent GitHub deploys, runs the security scanner, and answers with evidence.
- It runs on Google ADK and Gemini, with 80+ tools in 16 domains and 12 sub-agents.
- Two features make it more than a chat wrapper:
- Long-term memory. It recalls past incident resolutions when the same service shows up again.
- Fail-closed tool policy. Every tool call goes through it.
MyClawBot (personal) is the proving ground for the same patterns.
- 12 bot instances on Telegram and WhatsApp, built on the Claude Agent SDK.
- A cost-aware model router chooses Haiku, Sonnet or Opus for each request.
- 228 in-process MCP tools.
- A markdown vault is the source of truth, with a rebuildable SQLite index for search.
ReliBot system
Memory architecture
Design principle: cheap deterministic recall on every turn; expensive semantic search only when the model asks for it.
Recall layers
| Layer | What it holds | ReliBot budget | MyClawBot budget | How it's selected |
|---|---|---|---|---|
| L1 Core | MEMORY.md: preferences, standing decisions |
≤ 8K chars | ≤ 2K chars | Always present |
| L2 Recent | This chat's recent history | last 4 days, ≤ 20K chars | last 3 days of daily notes, ≤ 4K chars | Recency only, scoped by chat_key |
| L3 Entity | What we know about the services or people mentioned | ≤ 6 entities, ≤ 9.6K chars | ≤ 8 entities, ≤ 1.5K chars | Substring / word-boundary match against a cached entity list. Microseconds, no LLM call. |
| Search | Everything else | on demand | on demand | Hybrid 0.7 vector + 0.3 BM25 (+ graph boost) |
How memory is written and kept bounded
- Writes happen at three moments:
- The model saves explicitly with
save_memory. - Post-turn heuristics save an incident resolution automatically when two or more incident tools ran and nothing was saved.
- An idle or nightly flush runs a cheap model to distill the conversation into Decisions / Lessons / Actions.
- The model saves explicitly with
- Storage: one SQLite file in WAL mode.
- FTS5 handles exact tokens: error codes, service names, CVE IDs.
- sqlite-vec handles semantic similarity.
- Zero extra infrastructure for a team-scoped bot.
- Bounded growth:
- Session rotation at 50K tokens or 50 messages, with a summary written first.
- 90-day TTL on conversations.
- SHA-1 dedup on extracted facts.
- Memory is a prompt-injection persistence vector. Text saved once is replayed every turn. So:
MEMORY.mdis sanitized.- Tool output is scanned in
after_tool_callback. - Compression input is fenced as quoted data.
- Tools are a fail-closed allowlist.
How the two projects relate
| ReliBot | MyClawBot | |
|---|---|---|
| Runtime | Google ADK + Gemini (Vertex AI) | Claude Agent SDK, Haiku / Sonnet / Opus router |
| Source of truth | SQLite notes table |
Markdown vault. SQLite is only an index and can be rebuilt. |
| Session state | ADK DatabaseSessionService + queryable conversations table |
SDK session resume + sessions.json (90-day TTL) |
| Where recall is injected | system_instruction (ADK callback) |
Prepended to the user message, so the system-prompt prefix stays byte-stable for prompt caching |
| Embeddings | fastembed bge-small 384d, sqlite-vec | MiniLM 384d BLOB, cosine over a cached matrix |
| Ranking | 0.7 vector + 0.3 BM25 | same, × recency decay, then MMR re-rank and query expansion |
| Graph extraction | regex (tier 1) + LLM queue (tier 2) | rule-based |
| Idle flush | Gemini Flash, 15 min idle + nightly | Claude, 10 min idle |
| Tools | plain ADK function tools | in-process MCP @tool |
| Guardrails | fail-closed before_tool_callback, output injection scan, 5-calls/tool cap |
fail-closed can_use_tool, untrusted-content fencing, path-traversal checks |
Why the injection point differs.
- Prompt caching is prefix-based. Per-turn recall placed in the system prompt breaks the cache on every turn.
- MyClawBot keeps the system prompt stable, keyed on file mtimes, and puts the dynamic context in the user turn.
- The rule: volatile content goes last.
Design decisions
| Decision | Why | At larger scale |
|---|---|---|
| SQLite (WAL + FTS5 + sqlite-vec), not a vector DB | Each instance is team-scoped: about 2.5K chunks and 500 notes. Millisecond queries, zero ops, one file to back up. | Postgres + pgvector with tenant_id and row-level security, once you need multi-writer concurrency or multi-tenant isolation |
| Keep BM25 alongside vectors | Incident language is full of exact tokens (ORA-01555, CVE IDs, service names) that embeddings blur. On rare-token queries, plain FTS5 got 38/40 recall@10 while the 0.7-weighted hybrid got 0–2/40. |
Reciprocal-rank fusion or an exact-match boost instead of fixed linear weights |
| Deterministic recall + on-demand search | Injecting everything blows the context and the cost. Relying on the model to call a memory tool fails because models forget. | Same pattern; tune the layer budgets per tenant |
| Model failover with a circuit breaker | Fallback chain Pro → Flash → 3.1 Flash → Flash-Lite, retry backoff 1/3/8 s. The breaker opens after 2 failures, so a bad model is skipped instead of paying its timeout every request. Replies are tagged [fallback: model]. |
Per-provider health checks and budget-aware routing |
| Tool-call caps enforced in code | Gemini sub-agents called search_notes 15+ times, and prompt instructions didn't stop it. Now max 5 calls per tool per agent per query, plus an event cap. |
Guardrails belong in code, not the prompt |
| Memory scoping | chat_key = channel:chat_id scopes L2 and compression. Notes and L3 are shared per instance, as team knowledge. Hard isolation comes from separate instances and data directories, run by a manager process. |
tenant_id on every row, enforced in the database, never only in the prompt |
Testing.
- ReliBot: 2,200+ pytest tests. The LLM is mocked at the boundary, and storage tests use a real SQLite in
tmp_path, because mocking the database hides the bugs that matter. - MyClawBot: about 3,500 tests, with a deterministic gate under 60 s plus the full suite.
Lessons from token metering
Two real metering bugs from MyClawBot, both found by comparing claimed behavior against production logs:
- A cumulative counter treated as a delta.
- When a session is resumed, the SDK's
total_cost_usdis the session's running total, but it was summed per turn. - Recorded spend was $561.97 against $144.12 actual: a 3.9× overcount, and 7× on one bot.
- Fix: know whether each provider field is a counter, a gauge or a delta; meter from raw token events and price them yourself.
- When a session is resumed, the SDK's
- Cached tokens.
input_tokensexcludes cache reads and writes, so session rotation keyed on it never fired. It saw 2–8 tokens while real context grew past 300K.- Fix: measure input + cache-read + cache-write, and price each token type separately.
Applying the lessons: a usage-metering pipeline.
- Idempotency everywhere:
event_idon usage events, idempotency keys on Stripe calls, and webhook event IDs stored. - Counters vs deltas. Never sum a cumulative field.
- Price per token type: input, output, cache-write and cache-read.
- Late or duplicate events: ReplacingMergeTree, plus closing each billing period with a grace window.
- Entitlements: a model router that picks tiers by cost is the same shape as plan-based feature gating ("is this tenant entitled to the hard tier?").
Acronyms
Underlined acronyms show these definitions on hover or tap. Full glossary →
- ADK
- Agent Development Kit Google's open-source framework for building agents and multi-agent systems. ReliBot and GitHubScanner run on it.
- API
- Application Programming Interface How programs call each other, such as a REST endpoint.
- BM25
- Best Match 25 The classic keyword-ranking formula behind SQLite FTS5. Strong on exact tokens such as error codes and CVE IDs.
- CLI
- Command-Line Interface A terminal tool.
- CVE
- Common Vulnerabilities and Exposures The ID of one specific, publicly known vulnerability, such as CVE-2021-44228 (Log4Shell).
- FTS5
- Full-Text Search, version 5 SQLite's built-in full-text index.
- JWKS
- JSON Web Key Set The published public keys used to verify JWT signatures, such as Microsoft's keys for Teams bots.
- JWT
- JSON Web Token A signed token that carries identity claims.
- LLM
- Large Language Model A model such as Gemini or Claude that reads and writes text. Every agent here is an LLM plus tools.
- MCP
- Model Context Protocol An open standard for exposing tools and data to LLM agents. MyClawBot's tools are in-process MCP tools.
- MMR
- Maximal Marginal Relevance A re-ranking step that trades relevance against diversity, so the top results aren't near-duplicates.
- ONNX
- Open Neural Network Exchange A portable model format. ReliBot runs its embedding model locally through ONNX, with no API call.
- SDK
- Software Development Kit A library for building on a platform, such as the Claude Agent SDK.
- SHA-1
- Secure Hash Algorithm 1 A 160-bit hash. Used here for deduplication and snippet-integrity checks, not for security.
- SQL
- Structured Query Language The standard language for querying relational databases.
- SRE
- Site Reliability Engineering The discipline (and teams) that keep production running. ReliBot's main users.
- TTL
- Time To Live How long a record or cache entry is kept before it expires.
- WAL
- Write-Ahead Logging A SQLite journal mode where readers don't block the writer.
More architecture notes