will/open-brain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial project scaffold: Open Brain self-hosted knowledge infrastructure

Browse Source 
Node.js/TypeScript server with MCP endpoint for AI assistant integration,
Discord bot for thought capture and slash commands, PostgreSQL + pgvector
for semantic search, and OpenRouter for embeddings/metadata extraction.
Dockerized for deployment behind Caddy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
YOUNG
2026-03-06 14:53:41 -06:00
commit e90ea1591b
16 changed files with 3332 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

377
project_plan.MD Normal file
View File

@@ -0,0 +1,377 @@
Open Brain — Self-Hosted Implementation Plan
Context
Build a self-hosted personal knowledge infrastructure ("Open Brain") on a DigitalOcean VPS. Based on the guide at promptkit.natebjones.com, adapted to replace Supabase with self-hosted components running in Docker behind Caddy.
What it does: A single PostgreSQL database that any AI assistant can read from and write to via MCP (Model Context Protocol). Thoughts can also be captured and queried through a Discord bot. Every thought gets a vector embedding for
semantic search and structured metadata (type, topics, people, action items) extracted by an LLM.
Two input paths:
- Discord #capture channel — type a thought, bot ingests it
- MCP capture_thought tool — any connected AI can save a thought directly
Three output paths:
- MCP tools — AI assistants call search_thoughts, list_thoughts, thought_stats
- Discord slash commands — /brain search, /brain recent, /brain stats
- Direct SQL via SSH — admin/debug access
---
Decisions
┌──────────────────┬──────────────────────────────────────────────────────────────────────┐
│ Decision │ Choice │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Database │ Existing PostgreSQL on VPS + pgvector extension │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Schema │ Includes user_id column (future-proofing for multi-user) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Runtime │ Node.js / TypeScript │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Deployment │ Dockerized (Dockerfile + docker-compose.yml) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Messaging │ Discord bot (new bot, existing server) with capture + query commands │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ AI gateway │ OpenRouter (text-embedding-3-small + gpt-4o-mini) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Routing │ Subdomain brain.option.design via host-level Caddy │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Security │ 64-char hex key on MCP endpoint + Caddy rate limiting (30 req/min) │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ MCP write access │ Enabled — capture_thought tool included │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Backups │ Not included — user will handle separately │
├──────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Repo │ https://git.option.design/will/open-brain.git │
└──────────────────┴──────────────────────────────────────────────────────────────────────┘
---
Architecture
Internet (HTTPS)
|
Caddy (host-level, auto-TLS)
rate_limit: 30 req/min per IP
|
brain.option.design → localhost:3100
|
┌─────────────────────────────┐
│ Docker: open-brain │
│ │
│ Hono HTTP Server (:3100) │
│ ├── POST /mcp ─┼──→ AI clients (Claude Desktop,
│ │ (MCP StreamableHTTP) │ ChatGPT, Claude Code, Cursor)
│ │ Auth: x-brain-key │ call this endpoint over HTTPS
│ │ │
│ └── Discord.js bot ─┼──→ Discord #capture channel
│ (outbound WebSocket) │ (bot connects OUT to Discord,
│ ├── message capture │ no inbound traffic needed)
│ └── slash commands │
│ /brain search │
│ /brain recent │
│ /brain stats │
│ │
│ Shared ingest pipeline │
│ ├── getEmbedding() ─┼──→ OpenRouter API
│ └── extractMetadata() ─┼──→ OpenRouter API
└──────────────┬───────────────┘
│
PostgreSQL + pgvector (host)
└── brain database
└── thoughts table
How each actor accesses the system
┌───────────────────────────────────┬─────────────────────────────────┬──────────────────────────────────────────────┬──────────────────────────┐
│ Actor │ Path │ Auth │ Direction │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Claude Desktop / ChatGPT / Cursor │ HTTPS → Caddy → :3100/mcp │ MCP key (configured once in client settings) │ Client calls your server │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Claude Code │ Same as above │ MCP key (configured via claude mcp add) │ Client calls your server │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Discord (you typing in #capture) │ Discord servers → bot WebSocket │ Discord login │ Bot connects outward │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ Discord slash commands │ Same WebSocket │ Discord login │ Bot connects outward │
├───────────────────────────────────┼─────────────────────────────────┼──────────────────────────────────────────────┼──────────────────────────┤
│ You via SSH │ SSH → psql │ SSH key (existing) │ Direct DB access │
└───────────────────────────────────┴─────────────────────────────────┴──────────────────────────────────────────────┴──────────────────────────┘
Note: The MCP endpoint at brain.option.design is public-facing (reachable from the internet) because AI clients like Claude Desktop and ChatGPT call it from their own servers. It is protected by a 64-character hex key over HTTPS and
rate-limited at 30 requests/minute per IP. Without the correct key, all requests are rejected with 401.
---
Implementation Steps
Step 1: PostgreSQL + pgvector
Verify pgvector is installed on the existing PostgreSQL; install if needed. Create database, user, and schema.
File: sql/schema.sql
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE thoughts (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
user_id TEXT NOT NULL DEFAULT 'default',
content TEXT NOT NULL,
embedding VECTOR(1536),
metadata JSONB DEFAULT '{}'::jsonb,
source TEXT NOT NULL DEFAULT 'mcp', -- 'discord' or 'mcp'
created_at TIMESTAMPTZ DEFAULT now(),
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX ON thoughts USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON thoughts USING gin (metadata);
CREATE INDEX ON thoughts (created_at DESC);
CREATE INDEX ON thoughts (user_id);
-- Auto-update trigger for updated_at
CREATE OR REPLACE FUNCTION update_updated_at() RETURNS TRIGGER AS $$
BEGIN NEW.updated_at = now(); RETURN NEW; END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER thoughts_updated_at BEFORE UPDATE ON thoughts
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
-- Semantic search function
CREATE OR REPLACE FUNCTION match_thoughts(
query_embedding VECTOR(1536),
match_threshold FLOAT DEFAULT 0.7,
match_count INT DEFAULT 10,
filter JSONB DEFAULT '{}'::jsonb
) RETURNS TABLE (
id UUID, content TEXT, metadata JSONB,
similarity FLOAT, created_at TIMESTAMPTZ
) LANGUAGE plpgsql AS $$
BEGIN
RETURN QUERY
SELECT t.id, t.content, t.metadata,
1 - (t.embedding <=> query_embedding) AS similarity,
t.created_at
FROM thoughts t
WHERE 1 - (t.embedding <=> query_embedding) > match_threshold
ORDER BY t.embedding <=> query_embedding
LIMIT match_count;
END;
$$;
Deploy: SSH to VPS, run psql -d brain -f sql/schema.sql
Step 2: Node.js/TypeScript Project
Project structure:
open-brain/
├── Dockerfile
├── docker-compose.yml
├── package.json
├── tsconfig.json
├── .env.example
├── .gitignore
├── sql/
│ └── schema.sql
└── src/
├── index.ts — Hono server entry, starts HTTP + Discord bot
├── db.ts — pg client with pgvector, query helpers
├── ai.ts — OpenRouter: getEmbedding() + extractMetadata()
├── discord.ts — Discord.js bot: message capture + slash commands
├── mcp.ts — MCP server with 4 tool definitions
├── ingest.ts — Shared pipeline: embed → extract metadata → store
└── types.ts — TypeScript interfaces (Thought, Metadata)
Key dependencies:
- @modelcontextprotocol/sdk — MCP protocol server
- hono + @hono/node-server — HTTP framework on Node.js
- discord.js — Discord gateway bot
- pg + pgvector — PostgreSQL client with vector support
- zod — schema validation for MCP tool inputs
Step 3: Shared Ingest Pipeline (src/ingest.ts)
Both Discord and MCP capture_thought use the same pipeline:
Input text
→ getEmbedding(text) — OpenRouter text-embedding-3-small → 1536-dim vector
→ extractMetadata(text) — OpenRouter gpt-4o-mini → structured JSON
→ INSERT into thoughts table — content + embedding + metadata + source tag
→ Return confirmation — "Captured as [type] — [topics]"
Metadata schema extracted by LLM:
{
"type": "observation | task | idea | reference | person_note",
"topics": ["tag1", "tag2"],
"people": ["name1"],
"action_items": ["todo1"],
"dates_mentioned": ["YYYY-MM-DD"]
}
Step 4: MCP Server (src/mcp.ts)
Exposed at POST /mcp via StreamableHTTP transport. Authenticated with x-brain-key header or ?key= query param.
┌─────────────────┬───────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐
│ Tool │ Input │ What it does │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ search_thoughts │ query: string, limit?: number, threshold?: number │ Embed query → cosine similarity search in pgvector │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ list_thoughts │ limit?: number, type?: string, topic?: string, person?: string, days?: number │ Filter thoughts by metadata fields, return recent │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ thought_stats │ (none) │ Aggregate counts by type, topic, recent activity │
├─────────────────┼───────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────┤
│ capture_thought │ content: string │ Run full ingest pipeline, return confirmation │
└─────────────────┴───────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘
Each tool is defined with Zod schemas and registered on the MCP server. The AI client discovers them automatically on connection.
Step 5: Discord Bot (src/discord.ts)
Setup (manual, one-time):
1. Create Discord Application at discord.com/developers/applications
2. Create Bot user, enable MESSAGE CONTENT privileged intent
3. Generate and copy bot token → goes in .env
4. Invite to server with permissions: Read/Send Messages, Read Message History, Use Slash Commands
5. Designate #capture channel, copy channel ID → goes in .env
Message capture behavior:
- Listens for messages in configured channel only
- Ignores bot messages and system messages
- Runs ingest pipeline on valid messages
- Replies with: Captured as [type] — [topic1], [topic2]
Slash commands:
- /brain search <query> — semantic search, returns top results
- /brain recent [count] — list recent thoughts
- /brain stats — show aggregate stats
Step 6: OpenRouter Integration (src/ai.ts)
Two API calls to https://openrouter.ai/api/v1/:
- getEmbedding(text): POST /embeddings, model text-embedding-3-small → returns 1536-dim float array
- extractMetadata(text): POST /chat/completions, model gpt-4o-mini, system prompt requesting JSON matching the metadata schema above
Estimated cost: ~$0.10–0.30/month at 20 thoughts/day.
Step 7: Docker Setup
Dockerfile — Multi-stage: Node.js 22 Alpine, install deps, compile TypeScript, run compiled JS in slim image.
docker-compose.yml:
services:
open-brain:
build: .
container_name: open-brain
restart: unless-stopped
ports:
- "3100:3100"
env_file: .env
extra_hosts:
- "host.docker.internal:host-gateway"
host.docker.internal lets the container reach the host's PostgreSQL. May require updating pg_hba.conf to allow connections from Docker's bridge subnet (typically 172.17.0.0/16).
Step 8: Caddy Configuration
Add to existing Caddyfile on the VPS:
brain.option.design {
rate_limit {remote.host} 30r/m
reverse_proxy localhost:3100
}
DNS: Add A record for brain.option.design → VPS IP. Caddy auto-provisions TLS certificate.
Note: Caddy's rate limiting requires the caddy-ratelimit plugin. If not available, we can implement rate limiting in the Node.js app instead (simpler, no Caddy plugin needed).
Step 9: AI Client Configuration
MCP URL: https://brain.option.design/mcp
┌────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Client │ How to connect │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Claude Desktop │ Settings → Connectors → Add remote MCP server → paste URL with ?key=ACCESS_KEY │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Claude Code │ claude mcp add --transport http open-brain https://brain.option.design/mcp --header "x-brain-key: ACCESS_KEY" │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ChatGPT │ Developer Mode → Apps & Connectors → Create → paste URL with ?key=ACCESS_KEY │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Cursor │ Add to .cursor/mcp.json with URL and key header │
└────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
---
Environment Variables (.env)
# Database (host.docker.internal = host machine from inside Docker)
DATABASE_URL=postgresql://brain_user:password@host.docker.internal:5432/brain
# OpenRouter (AI gateway for embeddings + metadata extraction)
OPENROUTER_API_KEY=sk-or-...
# Discord bot
DISCORD_BOT_TOKEN=...
DISCORD_CAPTURE_CHANNEL_ID=...
# MCP endpoint authentication (generate with: openssl rand -hex 32)
MCP_ACCESS_KEY=...
# Server port
PORT=3100
---
Verification
1. Database: SSH to VPS → psql -d brain -c "SELECT count(*) FROM thoughts;" → returns 0
2. Discord capture: Send "Sarah is thinking about leaving her job to start consulting" in #capture → bot replies: Captured as person_note — career, consulting
3. Discord query: Type /brain search career → bot returns the thought about Sarah
4. Discord stats: Type /brain stats → bot shows 1 thought, type: person_note
5. MCP capture: In Claude Code, say "Remember that I prefer dark mode in all apps" → thought stored in DB
6. MCP search: Ask Claude "What do you know about my preferences?" → returns the dark mode thought
7. MCP list: Ask Claude "Show me recent thoughts" → returns both thoughts
8. Rate limit: Rapid-fire 30+ requests → subsequent requests get throttled
---
Current Status as of 3/6/2026 2:51:00 PM CST
---
● The project is fully scaffolded and compiles cleanly. Here's a summary of what was built:
Files Created
┌────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ File │ Purpose │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ sql/schema.sql │ PostgreSQL schema with pgvector, match_thoughts() function, indexes │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/types.ts │ TypeScript interfaces for Thought, ThoughtMetadata, etc. │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/db.ts │ PostgreSQL client — insert, search, list, stats queries │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/ai.ts │ OpenRouter API — getEmbedding() + extractMetadata() │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/ingest.ts │ Shared ingest pipeline — embed + extract + store │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/mcp.ts │ MCP server with 4 tools (search, list, stats, capture) │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/discord.ts │ Discord bot — message capture + /brain slash commands │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ src/index.ts │ Hono HTTP server entry point, MCP auth, starts Discord bot │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Dockerfile │ Multi-stage Docker build (Node 22 Alpine) │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ docker-compose.yml │ Container config with host.docker.internal for PostgreSQL │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ .env.example │ Template for environment variables │
├────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ .gitignore │ Excludes node_modules, dist, .env │
└────────────────────┴─────────────────────────────────────────────────────────────────────┘
Next Steps (on your VPS)
1. Create Discord bot at discord.com/developers — get bot token, enable MESSAGE CONTENT intent, invite to server
2. Create .env from .env.example with your real credentials
3. Run schema on PostgreSQL: psql -d brain -f sql/schema.sql (ensure pgvector extension is available)
4. Add DNS A record for brain.option.design → VPS IP
5. Add Caddy block for brain.option.design → localhost:3100
6. Deploy: docker compose up -d --build
7. Connect AI clients using the MCP URL
Want me to commit this to the repo, or would you like to review anything first?