will/open-brain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e90ea1591b9ea25ca4b43061a22ee42847e404a5
open-brain/project_plan.MD
YOUNG e90ea1591b Initial project scaffold: Open Brain self-hosted knowledge infrastructure 
Node.js/TypeScript server with MCP endpoint for AI assistant integration,
Discord bot for thought capture and slash commands, PostgreSQL + pgvector
for semantic search, and OpenRouter for embeddings/metadata extraction.
Dockerized for deployment behind Caddy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 14:53:41 -06:00

377 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
Open Brain — Self-Hosted Implementation Plan
Context
Build a self-hosted personal knowledge infrastructure ("Open Brain") on a DigitalOcean VPS. Based on the guide at promptkit.natebjones.com, adapted to replace Supabase with self-hosted components running in Docker behind Caddy.
What it does: A single PostgreSQL database that any AI assistant can read from and write to via MCP (Model Context Protocol). Thoughts can also be captured and queried through a Discord bot. Every thought gets a vector embedding for
semantic search and structured metadata (type, topics, people, action items) extracted by an LLM.
Two input paths:
- Discord #capture channel — type a thought, bot ingests it
- MCP capture_thought tool — any connected AI can save a thought directly
Three output paths:
- MCP tools — AI assistants call search_thoughts, list_thoughts, thought_stats
- Discord slash commands — /brain search, /brain recent, /brain stats
- Direct SQL via SSH — admin/debug access
---
Decisions
┌──────────────────┬──────────────────────────────────────────────────────────────────────┐
│ Decision │ Choice │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Database │ Existing PostgreSQL on VPS + pgvector extension │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Schema │ Includes user_id column (future-proofing for multi-user) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Runtime │ Node.js / TypeScript │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Deployment │ Dockerized (Dockerfile + docker-compose.yml) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Messaging │ Discord bot (new bot, existing server) with capture + query commands │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ AI gateway │ OpenRouter (text-embedding-3-small + gpt-4o-mini) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Routing │ Subdomain brain.option.design via host-level Caddy │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Security │ 64-char hex key on MCP endpoint + Caddy rate limiting (30 req/min) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ MCP write access │ Enabled — capture_thought tool included │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Backups │ Not included — user will handle separately │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Repo │ https://git.option.design/will/open-brain.git │
└──────────────────┴──────────────────────────────────────────────────────────────────────┘
---
Architecture
Internet (HTTPS)
|
Caddy (host-level, auto-TLS)
rate_limit: 30 req/min per IP
|
brain.option.design → localhost:3100
|
┌─────────────────────────────┐
│ Docker: open-brain │
│ │
│ Hono HTTP Server (:3100) │
│ ├── POST /mcp ─┼──→ AI clients (Claude Desktop,
│ │ (MCP StreamableHTTP) │ ChatGPT, Claude Code, Cursor)
│ │ Auth: x-brain-key │ call this endpoint over HTTPS
│ │ │
│ └── Discord.js bot ─┼──→ Discord #capture channel
│ (outbound WebSocket) │ (bot connects OUT to Discord,
│ ├── message capture │ no inbound traffic needed)
│ └── slash commands │
│ /brain search │
│ /brain recent │
│ /brain stats │
│ │
│ Shared ingest pipeline │
│ ├── getEmbedding() ─┼──→ OpenRouter API
│ └── extractMetadata() ─┼──→ OpenRouter API
└──────────────┬───────────────┘
│
PostgreSQL + pgvector (host)
└── brain database
└── thoughts table
How each actor accesses the system
┌───────────────────────────────────┬─────────────────────────────────┬──────────────────────────────────────────────┬──────────────────────────┐
│ Actor │ Path │ Auth │ Direction │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Claude Desktop / ChatGPT / Cursor │ HTTPS → Caddy → :3100/mcp │ MCP key (configured once in client settings) │ Client calls your server │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Claude Code │ Same as above │ MCP key (configured via claude mcp add) │ Client calls your server │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Discord (you typing in #capture) │ Discord servers → bot WebSocket │ Discord login │ Bot connects outward │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Discord slash commands │ Same WebSocket │ Discord login │ Bot connects outward │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ You via SSH │ SSH → psql │ SSH key (existing) │ Direct DB access │
└───────────────────────────────────┴─────────────────────────────────┴──────────────────────────────────────────────┴──────────────────────────┘
Note: The MCP endpoint at brain.option.design is public-facing (reachable from the internet) because AI clients like Claude Desktop and ChatGPT call it from their own servers. It is protected by a 64-character hex key over HTTPS and
rate-limited at 30 requests/minute per IP. Without the correct key, all requests are rejected with 401.
---
Implementation Steps
Step 1: PostgreSQL + pgvector
Verify pgvector is installed on the existing PostgreSQL; install if needed. Create database, user, and schema.
File: sql/schema.sql
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE thoughts (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
user_id TEXT NOT NULL DEFAULT 'default',
content TEXT NOT NULL,
embedding VECTOR(1536),
metadata JSONB DEFAULT '{}'::jsonb,
source TEXT NOT NULL DEFAULT 'mcp', -- 'discord' or 'mcp'
created_at TIMESTAMPTZ DEFAULT now(),
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX ON thoughts USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON thoughts USING gin (metadata);
CREATE INDEX ON thoughts (created_at DESC);
CREATE INDEX ON thoughts (user_id);
-- Auto-update trigger for updated_at
CREATE OR REPLACE FUNCTION update_updated_at() RETURNS TRIGGER AS $$
BEGIN NEW.updated_at = now(); RETURN NEW; END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER thoughts_updated_at BEFORE UPDATE ON thoughts
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
-- Semantic search function
CREATE OR REPLACE FUNCTION match_thoughts(
query_embedding VECTOR(1536),
match_threshold FLOAT DEFAULT 0.7,
match_count INT DEFAULT 10,
filter JSONB DEFAULT '{}'::jsonb
) RETURNS TABLE (
id UUID, content TEXT, metadata JSONB,
similarity FLOAT, created_at TIMESTAMPTZ
) LANGUAGE plpgsql AS $$
BEGIN
RETURN QUERY
SELECT t.id, t.content, t.metadata,
1 - (t.embedding <=> query_embedding) AS similarity,
t.created_at
FROM thoughts t
WHERE 1 - (t.embedding <=> query_embedding) > match_threshold
ORDER BY t.embedding <=> query_embedding
LIMIT match_count;
END;
$$;
Deploy: SSH to VPS, run psql -d brain -f sql/schema.sql
Step 2: Node.js/TypeScript Project
Project structure:
open-brain/
├── Dockerfile
├── docker-compose.yml
├── package.json
├── tsconfig.json
├── .env.example
├── .gitignore
├── sql/
│ └── schema.sql
└── src/
├── index.ts — Hono server entry, starts HTTP + Discord bot
├── db.ts — pg client with pgvector, query helpers
├── ai.ts — OpenRouter: getEmbedding() + extractMetadata()
├── discord.ts — Discord.js bot: message capture + slash commands
├── mcp.ts — MCP server with 4 tool definitions
├── ingest.ts — Shared pipeline: embed → extract metadata → store
└── types.ts — TypeScript interfaces (Thought, Metadata)
Key dependencies:
- @modelcontextprotocol/sdk — MCP protocol server
- hono + @hono/node-server — HTTP framework on Node.js
- discord.js — Discord gateway bot
- pg + pgvector — PostgreSQL client with vector support
- zod — schema validation for MCP tool inputs
Step 3: Shared Ingest Pipeline (src/ingest.ts)
Both Discord and MCP capture_thought use the same pipeline:
Input text
→ getEmbedding(text) — OpenRouter text-embedding-3-small → 1536-dim vector
→ extractMetadata(text) — OpenRouter gpt-4o-mini → structured JSON
→ INSERT into thoughts table — content + embedding + metadata + source tag
→ Return confirmation — "Captured as [type] — [topics]"
Metadata schema extracted by LLM:
{
"type": "observation | task | idea | reference | person_note",
"topics": ["tag1", "tag2"],
"people": ["name1"],
"action_items": ["todo1"],
"dates_mentioned": ["YYYY-MM-DD"]
}
Step 4: MCP Server (src/mcp.ts)
Exposed at POST /mcp via StreamableHTTP transport. Authenticated with x-brain-key header or ?key= query param.
┌─────────────────┬───────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐
│ Tool │ Input │ What it does │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ search_thoughts │ query: string, limit?: number, threshold?: number │ Embed query → cosine similarity search in pgvector │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ list_thoughts │ limit?: number, type?: string, topic?: string, person?: string, days?: number │ Filter thoughts by metadata fields, return recent │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ thought_stats │ (none) │ Aggregate counts by type, topic, recent activity │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ capture_thought │ content: string │ Run full ingest pipeline, return confirmation │
└─────────────────┴───────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘
Each tool is defined with Zod schemas and registered on the MCP server. The AI client discovers them automatically on connection.
Step 5: Discord Bot (src/discord.ts)
Setup (manual, one-time):
1. Create Discord Application at discord.com/developers/applications
2. Create Bot user, enable MESSAGE CONTENT privileged intent
3. Generate and copy bot token → goes in .env
4. Invite to server with permissions: Read/Send Messages, Read Message History, Use Slash Commands
5. Designate #capture channel, copy channel ID → goes in .env
Message capture behavior:
- Listens for messages in configured channel only
- Ignores bot messages and system messages
- Runs ingest pipeline on valid messages
- Replies with: Captured as [type] — [topic1], [topic2]
Slash commands:
- /brain search <query> — semantic search, returns top results
- /brain recent [count] — list recent thoughts
- /brain stats — show aggregate stats
Step 6: OpenRouter Integration (src/ai.ts)
Two API calls to https://openrouter.ai/api/v1/:
- getEmbedding(text): POST /embeddings, model text-embedding-3-small → returns 1536-dim float array
- extractMetadata(text): POST /chat/completions, model gpt-4o-mini, system prompt requesting JSON matching the metadata schema above
Estimated cost: ~$0.10–0.30/month at 20 thoughts/day.
Step 7: Docker Setup
Dockerfile — Multi-stage: Node.js 22 Alpine, install deps, compile TypeScript, run compiled JS in slim image.
docker-compose.yml:
services:
open-brain:
build: .
container_name: open-brain
restart: unless-stopped
ports:
- "3100:3100"
env_file: .env
extra_hosts:
- "host.docker.internal:host-gateway"
host.docker.internal lets the container reach the host's PostgreSQL. May require updating pg_hba.conf to allow connections from Docker's bridge subnet (typically 172.17.0.0/16).
Step 8: Caddy Configuration
Add to existing Caddyfile on the VPS:
brain.option.design {
rate_limit {remote.host} 30r/m
reverse_proxy localhost:3100
}
DNS: Add A record for brain.option.design → VPS IP. Caddy auto-provisions TLS certificate.
Note: Caddy's rate limiting requires the caddy-ratelimit plugin. If not available, we can implement rate limiting in the Node.js app instead (simpler, no Caddy plugin needed).
Step 9: AI Client Configuration
MCP URL: https://brain.option.design/mcp
┌────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Client │ How to connect │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Claude Desktop │ Settings → Connectors → Add remote MCP server → paste URL with ?key=ACCESS_KEY │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Claude Code │ claude mcp add --transport http open-brain https://brain.option.design/mcp --header "x-brain-key: ACCESS_KEY" │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ChatGPT │ Developer Mode → Apps & Connectors → Create → paste URL with ?key=ACCESS_KEY │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Cursor │ Add to .cursor/mcp.json with URL and key header │
└────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
---
Environment Variables (.env)
# Database (host.docker.internal = host machine from inside Docker)
DATABASE_URL=postgresql://brain_user:password@host.docker.internal:5432/brain
# OpenRouter (AI gateway for embeddings + metadata extraction)
OPENROUTER_API_KEY=sk-or-...
# Discord bot
DISCORD_BOT_TOKEN=...
DISCORD_CAPTURE_CHANNEL_ID=...
# MCP endpoint authentication (generate with: openssl rand -hex 32)
MCP_ACCESS_KEY=...
# Server port
PORT=3100
---
Verification
1. Database: SSH to VPS → psql -d brain -c "SELECT count(*) FROM thoughts;" → returns 0
2. Discord capture: Send "Sarah is thinking about leaving her job to start consulting" in #capture → bot replies: Captured as person_note — career, consulting
3. Discord query: Type /brain search career → bot returns the thought about Sarah
4. Discord stats: Type /brain stats → bot shows 1 thought, type: person_note
5. MCP capture: In Claude Code, say "Remember that I prefer dark mode in all apps" → thought stored in DB
6. MCP search: Ask Claude "What do you know about my preferences?" → returns the dark mode thought
7. MCP list: Ask Claude "Show me recent thoughts" → returns both thoughts
8. Rate limit: Rapid-fire 30+ requests → subsequent requests get throttled
---
Current Status as of 3/6/2026 2:51:00 PM CST
---
● The project is fully scaffolded and compiles cleanly. Here's a summary of what was built:
Files Created
┌────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ File │ Purpose │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ sql/schema.sql │ PostgreSQL schema with pgvector, match_thoughts() function, indexes │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/types.ts │ TypeScript interfaces for Thought, ThoughtMetadata, etc. │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/db.ts │ PostgreSQL client — insert, search, list, stats queries │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/ai.ts │ OpenRouter API — getEmbedding() + extractMetadata() │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/ingest.ts │ Shared ingest pipeline — embed + extract + store │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/mcp.ts │ MCP server with 4 tools (search, list, stats, capture) │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/discord.ts │ Discord bot — message capture + /brain slash commands │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/index.ts │ Hono HTTP server entry point, MCP auth, starts Discord bot │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Dockerfile │ Multi-stage Docker build (Node 22 Alpine) │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ docker-compose.yml │ Container config with host.docker.internal for PostgreSQL │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ .env.example │ Template for environment variables │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ .gitignore │ Excludes node_modules, dist, .env │
└────────────────────┴─────────────────────────────────────────────────────────────────────┘
Next Steps (on your VPS)
1. Create Discord bot at discord.com/developers — get bot token, enable MESSAGE CONTENT intent, invite to server
2. Create .env from .env.example with your real credentials
3. Run schema on PostgreSQL: psql -d brain -f sql/schema.sql (ensure pgvector extension is available)
4. Add DNS A record for brain.option.design → VPS IP
5. Add Caddy block for brain.option.design → localhost:3100
6. Deploy: docker compose up -d --build
7. Connect AI clients using the MCP URL
Want me to commit this to the repo, or would you like to review anything first?
Reference in New Issue View Git Blame Copy Permalink