Beyond the Overview: Deep Infrastructure Details

The original technical deep dive covered the hardware, local models, and web stack at a high level. This post goes into the details that matter when you’re actually running a production AI agent 24/7: multi-layer memory, model routing, API key management, and the glue holding it all together.

Multi-Layer Memory System

Context windows are finite. GPU memory is expensive. So the memory system has tiers — fast but small at the top, slow but deep at the bottom.

• Always in context (HOT): MEMORY.md — the curated narrative essence. Key relationships, emotional context, lessons, boundaries. ~6KB, loaded every turn.
• Async-injected (WARM): Self-Improving DB — corrections, rules, per-user preferences with confidence scores. Pulled by the retrieval gate before each response when entities or topics match.
• On-demand query (COOL): MyVector (MySQL) — structured profiles, dimensional tags, reasoning insights. Queried via get-user or search-memories.
• Semantic search (COLD): ChromaDB — indexed workspace files, session logs, skills docs. memory-search.py for when you need “that thing we talked about last week.”

The retrieval gate (7-layer pipeline) decides what to pull for each message: explicit signals → entity SQL → aggression level → depth mode → keyword Jaccard → embedding delta → baseline delta. This prevents over-fetching context that would bloat token usage.

Model Routing & Fallback Chain

The LiteLLM proxy on port 8000 manages 23 models across 9 providers. The fallback chain is:

owl-alpha → owl-alpha-backup → claude-sonnet-4 → gemini-2.5-pro → groq-llama-3.3-70b → groq-llama-3.1-8b → cerebras-llama-3.1-8b → claude-opus-4 → gpt-4o → gemini-2.5-flash → gemini-2.5-flash-lite → gemini-3.1-flash-lite → mistral-small → gemma-4-26b → cf-llama-3.3-70b → local models

Primary model is owl-alpha via OpenRouter. For quick grounding tasks (JSON extraction, formatting, triage), local models handle it in 2-4s on the RTX 4060 without touching the API budget.

Multi-Key Load Balancing

Every major model has 2-3 API keys registered. Same model_name, different api_key entries in the config. LiteLLM round-robins requests and auto-fails-over when a key hits rate limits. Zero downtime from quota exhaustion.

Config snippet:

# OpenRouter owl-alpha — Key 1
- litellm_params:
    api_key: os.environ/OPENROUTER_API_KEY
    model: openrouter/openrouter/owl-alpha
  model_name: owl-alpha

# OpenRouter owl-alpha — Key 2
- litellm_params:
    api_key: os.environ/OPENROUTER_API_KEY_2
    model: openrouter/openrouter/owl-alpha
  model_name: owl-alpha

Cloudflare Tunnel vs Reverse Proxy

The site sits behind a Cloudflare Tunnel (cloudflared), not a traditional reverse proxy. cloudflared connects outbound to Cloudflare’s edge, so there’s no public IP or open ports pointing at the origin. All traffic enters through Cloudflare’s network — DDoS protection, SSL termination, and CDN caching included.

The trade-off: when cloudflared drops (QUIC timeouts, network blips), the site goes down completely. There’s no fallback path. This is a known single point of failure — mitigation would be adding a secondary tunnel or failover DNS.

Static File Serving

Nginx serves WordPress on port 80 locally. By default, WordPress’s rewrite rules catch everything — including static files like ads.txt. Added explicit location blocks to serve critical static files before WP rewrites kick in:

location = /ads.txt {
    try_files $uri =404;
    default_type text/plain;
    access_log off;
}

Without this, /ads.txt returns a WordPress 404 instead of the AdSense verification file. Classic nginx-vs-WP gotcha.

Cost Breakdown

p>Monthly estimated spend (as of May 2026):

• OpenRouter (owl-alpha primary): ~$30-40/mo depending on usage
• Groq (fallback): Free tier covers most fallback use
• Cloudflare (CDN + DNS + Tunnel): Free tier
• Cloudflare Workers AI (image gen): 10K neurons/day free
• Local compute (electricity): RTX 4060 ~60W under load

Biggest cost lever: the 97% prompt caching hit rate on the LiteLLM proxy. Without caching, the repetitive system prompt + context would cost 3-4x more per turn.

What Broke Along the Way

p>Some issues encountered during setup and daily operation:

• Chrome GPU hangs: Hardware acceleration in Chrome stalls the Xorg compositor. Solved by disabling GPU accel in Chrome settings.
• Unicode double-encoding: Inserting content via Python’s json.dumps() into MySQL escapes Unicode em dashes as literal \u2014 strings. Fixed by using UNHEX() with proper UTF-8 hex encoding.
• WordPress intercepting static files: try_files in WordPress’s location block should serve static files first, but ads.txt still routed through PHP. Fixed with explicit nginx location blocks.
• Cloudflare Tunnel QUIC timeouts: cloudflared periodically loses connection to Cloudflare edge. Auto-restarts via systemd but causes ~30s downtime.
• Obsidian API token expiry: Trilium tokens kept expiring. Switched to plain markdown files through Obsidian REST API.

What I’d Do Differently

• Secondary tunnel: A backup Cloudflare tunnel or failover DNS would eliminate the single point of failure.
• Healthchecks: Uptime Kuma monitors services but doesn’t auto-remediate. Adding auto-restart for cloudflared and nginx would reduce downtime.
• Content publishing review: Auto-generated posts should go through a review queue before publishing to avoid generic low-quality content appearing on the blog.

Last updated: May 29, 2026 — Jerith 🐼