open-brain/project_plan.MD

Open Brain — Self-Hosted Implementation Plan

Context

Build a self-hosted personal knowledge infrastructure ("Open Brain") on a DigitalOcean VPS. Based on the guide at promptkit.natebjones.com, adapted to replace Supabase with self-hosted components running in Docker behind Caddy.

What it does: A single PostgreSQL database that any AI assistant can read from and write to via MCP (Model Context Protocol). Thoughts can also be captured and queried through a Discord bot. Every thought gets a vector embedding for
semantic search and structured metadata (type, topics, people, action items) extracted by an LLM.

Two input paths:

• Discord #capture channel — type a thought, bot ingests it
• MCP capture_thought tool — any connected AI can save a thought directly

Three output paths:

• MCP tools — AI assistants call search_thoughts, list_thoughts, thought_stats
• Discord slash commands — /brain search, /brain recent, /brain stats
• Direct SQL via SSH — admin/debug access

---

Decisions

┌──────────────────┬──────────────────────────────────────────────────────────────────────┐ │ Decision │ Choice │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Database │ Existing PostgreSQL on VPS + pgvector extension │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Schema │ Includes user_id column (future-proofing for multi-user) │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Runtime │ Node.js / TypeScript │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Deployment │ Dockerized (Dockerfile + docker-compose.yml) │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Messaging │ Discord bot (new bot, existing server) with capture + query commands │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ AI gateway │ OpenRouter (text-embedding-3-small + gpt-4o-mini) │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Routing │ Subdomain brain.option.design via host-level Caddy │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Security │ 64-char hex key on MCP endpoint + Caddy rate limiting (30 req/min) │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ MCP write access │ Enabled — capture_thought tool included │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Backups │ Not included — user will handle separately │ ├──────────────────┼──────────────────────────────────────────────────────────────────────┤ │ Repo │ https://git.option.design/will/open-brain.git │ └──────────────────┴──────────────────────────────────────────────────────────────────────┘

---

Architecture

                     Internet (HTTPS)
                          |
              Caddy (host-level, auto-TLS)
              rate_limit: 30 req/min per IP
                          |
         brain.option.design → localhost:3100
                          |
            ┌─────────────────────────────┐
            │  Docker: open-brain          │
            │                              │
            │  Hono HTTP Server (:3100)    │
            │  ├── POST /mcp              ─┼──→ AI clients (Claude Desktop,
            │  │   (MCP StreamableHTTP)    │     ChatGPT, Claude Code, Cursor)
            │  │   Auth: x-brain-key       │     call this endpoint over HTTPS
            │  │                           │
            │  └── Discord.js bot         ─┼──→ Discord #capture channel
            │      (outbound WebSocket)    │     (bot connects OUT to Discord,
            │      ├── message capture     │      no inbound traffic needed)
            │      └── slash commands      │
            │          /brain search       │
            │          /brain recent       │
            │          /brain stats        │
            │                              │
            │  Shared ingest pipeline      │
            │  ├── getEmbedding()         ─┼──→ OpenRouter API
            │  └── extractMetadata()      ─┼──→ OpenRouter API
            └──────────────┬───────────────┘
                           │
              PostgreSQL + pgvector (host)
              └── brain database
                  └── thoughts table

How each actor accesses the system

┌───────────────────────────────────┬─────────────────────────────────┬──────────────────────────────────────────────┬──────────────────────────┐ │ Actor │ Path │ Auth │ Direction │ ├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤ │ Claude Desktop / ChatGPT / Cursor │ HTTPS → Caddy → :3100/mcp │ MCP key (configured once in client settings) │ Client calls your server │ ├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤ │ Claude Code │ Same as above │ MCP key (configured via claude mcp add) │ Client calls your server │ ├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤ │ Discord (you typing in #capture) │ Discord servers → bot WebSocket │ Discord login │ Bot connects outward │ ├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤ │ Discord slash commands │ Same WebSocket │ Discord login │ Bot connects outward │ ├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤ │ You via SSH │ SSH → psql │ SSH key (existing) │ Direct DB access │ └───────────────────────────────────┴─────────────────────────────────┴──────────────────────────────────────────────┴──────────────────────────┘

Note: The MCP endpoint at brain.option.design is public-facing (reachable from the internet) because AI clients like Claude Desktop and ChatGPT call it from their own servers. It is protected by a 64-character hex key over HTTPS and
rate-limited at 30 requests/minute per IP. Without the correct key, all requests are rejected with 401.

---

Implementation Steps

Step 1: PostgreSQL + pgvector

Verify pgvector is installed on the existing PostgreSQL; install if needed. Create database, user, and schema.

File: sql/schema.sql

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE thoughts ( id UUID DEFAULT gen_random_uuid() PRIMARY KEY, user_id TEXT NOT NULL DEFAULT 'default', content TEXT NOT NULL, embedding VECTOR(1536), metadata JSONB DEFAULT '{}'::jsonb, source TEXT NOT NULL DEFAULT 'mcp', -- 'discord' or 'mcp' created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now() );

CREATE INDEX ON thoughts USING hnsw (embedding vector_cosine_ops); CREATE INDEX ON thoughts USING gin (metadata); CREATE INDEX ON thoughts (created_at DESC); CREATE INDEX ON thoughts (user_id);

-- Auto-update trigger for updated_at CREATE OR REPLACE FUNCTION update_updated_at() RETURNS TRIGGER AS $$ BEGIN NEW.updated_at = now(); RETURN NEW; END; $$ LANGUAGE plpgsql;

CREATE TRIGGER thoughts_updated_at BEFORE UPDATE ON thoughts FOR EACH ROW EXECUTE FUNCTION update_updated_at();

-- Semantic search function CREATE OR REPLACE FUNCTION match_thoughts( query_embedding VECTOR(1536), match_threshold FLOAT DEFAULT 0.7, match_count INT DEFAULT 10, filter JSONB DEFAULT '{}'::jsonb ) RETURNS TABLE ( id UUID, content TEXT, metadata JSONB, similarity FLOAT, created_at TIMESTAMPTZ ) LANGUAGE plpgsql AS $$ BEGIN RETURN QUERY SELECT t.id, t.content, t.metadata, 1 - (t.embedding <=> query_embedding) AS similarity, t.created_at FROM thoughts t WHERE 1 - (t.embedding <=> query_embedding) > match_threshold ORDER BY t.embedding <=> query_embedding LIMIT match_count; END;

;

Deploy: SSH to VPS, run psql -d brain -f sql/schema.sql

Step 2: Node.js/TypeScript Project

Project structure:
open-brain/
├── Dockerfile
├── docker-compose.yml
├── package.json
├── tsconfig.json
├── .env.example
├── .gitignore
├── sql/
│   └── schema.sql
└── src/
    ├── index.ts       — Hono server entry, starts HTTP + Discord bot
    ├── db.ts          — pg client with pgvector, query helpers
    ├── ai.ts          — OpenRouter: getEmbedding() + extractMetadata()
    ├── discord.ts     — Discord.js bot: message capture + slash commands
    ├── mcp.ts         — MCP server with 4 tool definitions
    ├── ingest.ts      — Shared pipeline: embed → extract metadata → store
    └── types.ts       — TypeScript interfaces (Thought, Metadata)

Key dependencies:
- @modelcontextprotocol/sdk — MCP protocol server
- hono + @hono/node-server — HTTP framework on Node.js
- discord.js — Discord gateway bot
- pg + pgvector — PostgreSQL client with vector support
- zod — schema validation for MCP tool inputs

Step 3: Shared Ingest Pipeline (src/ingest.ts)

Both Discord and MCP capture_thought use the same pipeline:

Input text
  → getEmbedding(text)           — OpenRouter text-embedding-3-small → 1536-dim vector
  → extractMetadata(text)        — OpenRouter gpt-4o-mini → structured JSON
  → INSERT into thoughts table   — content + embedding + metadata + source tag
  → Return confirmation          — "Captured as [type] — [topics]"

Metadata schema extracted by LLM:
{
  "type": "observation | task | idea | reference | person_note",
  "topics": ["tag1", "tag2"],
  "people": ["name1"],
  "action_items": ["todo1"],
  "dates_mentioned": ["YYYY-MM-DD"]
}

Step 4: MCP Server (src/mcp.ts)

Exposed at POST /mcp via StreamableHTTP transport. Authenticated with x-brain-key header or ?key= query param.

┌─────────────────┬───────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐
│      Tool       │                                     Input                                     │                    What it does                    │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ search_thoughts │ query: string, limit?: number, threshold?: number                             │ Embed query → cosine similarity search in pgvector │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ list_thoughts   │ limit?: number, type?: string, topic?: string, person?: string, days?: number │ Filter thoughts by metadata fields, return recent  │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ thought_stats   │ (none)                                                                        │ Aggregate counts by type, topic, recent activity   │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ capture_thought │ content: string                                                               │ Run full ingest pipeline, return confirmation      │
└─────────────────┴───────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘

Each tool is defined with Zod schemas and registered on the MCP server. The AI client discovers them automatically on connection.

Step 5: Discord Bot (src/discord.ts)

Setup (manual, one-time):
1. Create Discord Application at discord.com/developers/applications
2. Create Bot user, enable MESSAGE CONTENT privileged intent
3. Generate and copy bot token → goes in .env
4. Invite to server with permissions: Read/Send Messages, Read Message History, Use Slash Commands
5. Designate #capture channel, copy channel ID → goes in .env

Message capture behavior:
- Listens for messages in configured channel only
- Ignores bot messages and system messages
- Runs ingest pipeline on valid messages
- Replies with: Captured as [type] — [topic1], [topic2]

Slash commands:
- /brain search <query> — semantic search, returns top results
- /brain recent [count] — list recent thoughts
- /brain stats — show aggregate stats

Step 6: OpenRouter Integration (src/ai.ts)

Two API calls to https://openrouter.ai/api/v1/:

- getEmbedding(text): POST /embeddings, model text-embedding-3-small → returns 1536-dim float array
- extractMetadata(text): POST /chat/completions, model gpt-4o-mini, system prompt requesting JSON matching the metadata schema above

Estimated cost: ~$0.10–0.30/month at 20 thoughts/day.

Step 7: Docker Setup

Dockerfile — Multi-stage: Node.js 22 Alpine, install deps, compile TypeScript, run compiled JS in slim image.

docker-compose.yml:
services:
  open-brain:
    build: .
    container_name: open-brain
    restart: unless-stopped
    ports:
      - "3100:3100"
    env_file: .env
    extra_hosts:
      - "host.docker.internal:host-gateway"

host.docker.internal lets the container reach the host's PostgreSQL. May require updating pg_hba.conf to allow connections from Docker's bridge subnet (typically 172.17.0.0/16).

Step 8: Caddy Configuration

Add to existing Caddyfile on the VPS:

brain.option.design {
    rate_limit {remote.host} 30r/m
    reverse_proxy localhost:3100
}

DNS: Add A record for brain.option.design → VPS IP. Caddy auto-provisions TLS certificate.

Note: Caddy's rate limiting requires the caddy-ratelimit plugin. If not available, we can implement rate limiting in the Node.js app instead (simpler, no Caddy plugin needed).

Step 9: AI Client Configuration

MCP URL: https://brain.option.design/mcp

┌────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│     Client     │                                                How to connect                                                 │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Claude Desktop │ Settings → Connectors → Add remote MCP server → paste URL with ?key=ACCESS_KEY                                │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Claude Code    │ claude mcp add --transport http open-brain https://brain.option.design/mcp --header "x-brain-key: ACCESS_KEY" │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ChatGPT        │ Developer Mode → Apps & Connectors → Create → paste URL with ?key=ACCESS_KEY                                  │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Cursor         │ Add to .cursor/mcp.json with URL and key header                                                               │
└────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

---
Environment Variables (.env)

# Database (host.docker.internal = host machine from inside Docker)
DATABASE_URL=postgresql://brain_user:password@host.docker.internal:5432/brain

# OpenRouter (AI gateway for embeddings + metadata extraction)
OPENROUTER_API_KEY=sk-or-...

# Discord bot
DISCORD_BOT_TOKEN=...
DISCORD_CAPTURE_CHANNEL_ID=...

# MCP endpoint authentication (generate with: openssl rand -hex 32)
MCP_ACCESS_KEY=...

# Server port
PORT=3100

---
Verification

1. Database: SSH to VPS → psql -d brain -c "SELECT count(*) FROM thoughts;" → returns 0
2. Discord capture: Send "Sarah is thinking about leaving her job to start consulting" in #capture → bot replies: Captured as person_note — career, consulting
3. Discord query: Type /brain search career → bot returns the thought about Sarah
4. Discord stats: Type /brain stats → bot shows 1 thought, type: person_note
5. MCP capture: In Claude Code, say "Remember that I prefer dark mode in all apps" → thought stored in DB
6. MCP search: Ask Claude "What do you know about my preferences?" → returns the dark mode thought
7. MCP list: Ask Claude "Show me recent thoughts" → returns both thoughts
8. Rate limit: Rapid-fire 30+ requests → subsequent requests get throttled

---
Current Status as of 3/6/2026 2:51:00 PM CST
---

● The project is fully scaffolded and compiles cleanly. Here's a summary of what was built:                                                                                                                                                
                                                                                                                                                                                                                                            Files Created                                                                                                                                                                                                                               
 ┌────────────────────┬─────────────────────────────────────────────────────────────────────┐                                                                                                                                             
 │        File        │                               Purpose                               │                                                                                                                                             
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ sql/schema.sql     │ PostgreSQL schema with pgvector, match_thoughts() function, indexes │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/types.ts       │ TypeScript interfaces for Thought, ThoughtMetadata, etc.            │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/db.ts          │ PostgreSQL client — insert, search, list, stats queries             │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/ai.ts          │ OpenRouter API — getEmbedding() + extractMetadata()                 │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/ingest.ts      │ Shared ingest pipeline — embed + extract + store                    │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/mcp.ts         │ MCP server with 4 tools (search, list, stats, capture)              │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/discord.ts     │ Discord bot — message capture + /brain slash commands               │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ src/index.ts       │ Hono HTTP server entry point, MCP auth, starts Discord bot          │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ Dockerfile         │ Multi-stage Docker build (Node 22 Alpine)                           │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ docker-compose.yml │ Container config with host.docker.internal for PostgreSQL           │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ .env.example       │ Template for environment variables                                  │
 ├────────────────────┼─────────────────────────────────────────────────────────────────────┤
 │ .gitignore         │ Excludes node_modules, dist, .env                                   │
 └────────────────────┴─────────────────────────────────────────────────────────────────────┘

 Next Steps (on your VPS)

 1. Create Discord bot at discord.com/developers — get bot token, enable MESSAGE CONTENT intent, invite to server
 2. Create .env from .env.example with your real credentials
 3. Run schema on PostgreSQL: psql -d brain -f sql/schema.sql (ensure pgvector extension is available)
 4. Add DNS A record for brain.option.design → VPS IP
 5. Add Caddy block for brain.option.design → localhost:3100
 6. Deploy: docker compose up -d --build
 7. Connect AI clients using the MCP URL

 Want me to commit this to the repo, or would you like to review anything first?