diff --git a/docs/00-style-guide.md b/docs/00-style-guide.md deleted file mode 100644 index c8a5419c..00000000 --- a/docs/00-style-guide.md +++ /dev/null @@ -1,96 +0,0 @@ -# Documentation style guide - -This repository aims for a NetBox-like style: clear, technical, and written for working engineers. - -## Voice and tone - -- **Direct and technical.** Prefer short sentences and specific nouns. -- **Narrative flow.** Describe how the system behaves, not how the doc was produced. -- **Calm, professional tone.** Avoid hype. -- **Assume competence, not context.** Define repo-specific terms once, then reuse them. - -## Page structure (default) - -Use a consistent, scan-friendly layout. - -1. **Title** -2. **1–3 sentence intro** - - What this page covers and who it’s for. -3. **Deep dives / Related docs** (optional but common) - - Links to more detailed pages. -4. **Main content** - - Prefer sections that match user intent: “Quickstart”, “How it works”, “Configuration”, “Common workflows”, “Troubleshooting”. -5. **Next steps** (optional) - - Where to go next. - -## Headings and conventions - -- Prefer **verb-led** headings when describing procedures: “Run migrations”, “Regenerate the client”. -- Prefer **intent-led** headings when describing concepts: “How requests flow”, “Auth model”. -- Use numbered steps when order matters. -- Keep headings short; avoid long parentheticals. - -## Cross-linking - -- Treat the numbered IA pages in `docs/` as **entrypoints**. -- Link to deep dives instead of duplicating content. -- Use readable link text: - - Good: “Deployment guide” → `docs/deployment/README.md` - - Avoid: ``docs/deployment/README.md`` - -## Link formatting rules - -- Use markdown links: `[Deployment guide](deployment/README.md)`. -- Use relative paths that work in GitHub and typical markdown renderers. -- Keep code formatting for: - - commands (`make check`) - - environment variables (`NEXT_PUBLIC_API_URL`) - - literal file paths when you mean “this exact file on disk” (not as a navigational link) - -## Avoided phrases (and what to use instead) - -Avoid doc-meta language: - -- Avoid: “evidence basis”, “evidence anchors”, “this page is intentionally…” -- Prefer: - - “Source of truth: …” (only when it matters) - - “See also: …” - - Just link the file or section. - -Avoid hedging: - -- Avoid: “likely”, “probably”, “should” (unless it’s a policy decision) -- Prefer: state what the code does, and point to the file. - -## Preferred patterns - -- **Start here** blocks for role-based entry. -- **Common workflows** sections with copy/paste commands. -- **Troubleshooting** sections with symptoms → checks → fixes. -- **Footguns** called out explicitly when they can cause outages or confusing behavior. - -## Example rewrites - -### Example 1: remove doc-meta “evidence” language - -Before: -> Evidence basis: consolidated from repo root `README.md`, `.github/workflows/ci.yml`, `Makefile`. - -After: -> This page describes the development workflow that matches CI: setup, checks, and common local loops. - -### Example 2: prefer readable links over code-formatted paths - -Before: -- See `docs/deployment/README.md` for deployment. - -After: -- See the [Deployment guide](deployment/README.md). - -### Example 3: replace “first pass” filler with a clear scope boundary - -Before: -- Non-goals (first pass) - -After: -- Out of scope diff --git a/docs/01-overview.md b/docs/01-overview.md deleted file mode 100644 index b4e6c046..00000000 --- a/docs/01-overview.md +++ /dev/null @@ -1,43 +0,0 @@ -# Mission Control - -Mission Control is the **web UI + HTTP API** for operating OpenClaw. - -It’s the place you go to coordinate work across people and agents, keep an evidence trail, and operate the system safely. - -## What problem it solves - -OpenClaw can run tools/skills and hold conversations across channels. What’s missing in practice is a control plane that makes this operational: - -- **Coordination**: boards + tasks make it explicit what’s being worked on, by whom, and what’s blocked. -- **Evidence**: task comments capture commands run, links, outputs, and decisions. -- **Risk control**: approvals provide a structured “allow/deny” gate for sensitive actions. -- **Operations**: deployment, configuration, and troubleshooting live in one navigable docs spine. - -## Core concepts - -- **Board**: a workspace containing tasks, memory, and agents. -- **Task**: a unit of work with a status and evidence (comments). -- **Agent**: an automated worker that executes tasks and posts evidence. -- **Approval**: a review gate for risky steps. -- **Gateway** (optional integration): an OpenClaw runtime host Mission Control can coordinate with. -- **Heartbeat**: periodic agent loop for incremental work. -- **Cron**: scheduled execution (recurring or one-shot). - -## What it is not - -- A general-purpose project management tool. -- An observability suite (use your existing logs/metrics/tracing; Mission Control links and operationalizes them). -- A secrets manager (keep secrets in your secret store; don’t paste them into tasks/docs). - -## How to navigate these docs - -This repo keeps a small “reader journey” spine under `docs/`: - -1. [Quickstart](02-quickstart.md) — run it locally/self-host. -2. [Development](03-development.md) — contributor workflow and CI parity. -3. [Configuration](06-configuration.md) — env vars, precedence, migrations, CORS. -4. [API reference](07-api-reference.md) — route groups + auth model. -5. [Ops / runbooks](09-ops-runbooks.md) — operational checklists. -6. [Troubleshooting](10-troubleshooting.md) — symptom → checks → fixes. - -For deeper references, see `docs/architecture/`, `docs/deployment/`, `docs/production/`, `docs/testing/`, and `docs/troubleshooting/`. diff --git a/docs/02-quickstart.md b/docs/02-quickstart.md deleted file mode 100644 index d299af3a..00000000 --- a/docs/02-quickstart.md +++ /dev/null @@ -1,61 +0,0 @@ -# Quickstart (Docker Compose) - -This is the fastest way to run Mission Control locally or on a single host. - -## What you get - -From `compose.yml` you get three services: - -- Postgres (`db`) -- FastAPI backend (`backend`) on `http://localhost:8000` -- Next.js frontend (`frontend`) on `http://localhost:3000` - -## Prerequisites - -- Docker + Docker Compose v2 (`docker compose`) - -## Run - -From repo root: - -```bash -cp .env.example .env - -docker compose -f compose.yml --env-file .env up -d --build -``` - -Open: -- UI: http://localhost:3000 -- Backend health: http://localhost:8000/healthz - -## Verify - -```bash -curl -f http://localhost:8000/healthz -curl -I http://localhost:3000/ -``` - -## Common gotchas - -- `NEXT_PUBLIC_API_URL` must be reachable from your **browser**. - - If it’s missing/blank/wrong, the UI may load but API calls will fail (e.g. Activity feed blank). -- If you are running locally without Clerk: - - keep `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` unset/blank so Clerk stays gated off in the frontend. - -## Useful commands - -```bash -# tail logs -docker compose -f compose.yml --env-file .env logs -f --tail=200 - -# stop (keeps data) -docker compose -f compose.yml --env-file .env down - -# reset data (DESTRUCTIVE) -docker compose -f compose.yml --env-file .env down -v -``` - -## Next - -- Want a faster contributor loop? See [Development](03-development.md) (DB via Compose, backend+frontend in dev mode). -- Need to change env vars/migrations/CORS? See [Configuration](06-configuration.md). diff --git a/docs/03-development.md b/docs/03-development.md deleted file mode 100644 index 33e7ccd6..00000000 --- a/docs/03-development.md +++ /dev/null @@ -1,121 +0,0 @@ -# Development - -This page is the contributor workflow for Mission Control. - -It’s intentionally **CI-aligned**: if you can run these commands locally, you should not be surprised by CI. - -## Prerequisites - -- Docker + Docker Compose v2 (`docker compose`) -- Python **3.12+** + [`uv`](https://github.com/astral-sh/uv) -- Node.js + npm - - CI pins **Node 20** via `.github/workflows/ci.yml` (`actions/setup-node@v4`, `node-version: "20"`). - -## Repo layout - -- Backend: `backend/` (FastAPI) -- Frontend: `frontend/` (Next.js) -- Canonical commands: `Makefile` - -## Setup (one command) - -From repo root: - -```bash -make setup -``` - -What it does (evidence: `Makefile`): -- `make backend-sync`: `cd backend && uv sync --extra dev` -- `make frontend-sync`: verifies node tooling (`scripts/with_node.sh --check`), then `npm install` in `frontend/` - -## Run the stack (two recommended loops) - -### Loop A (recommended): DB via Compose, app in dev mode - -1) Start Postgres: - -```bash -cp .env.example .env - -docker compose -f compose.yml --env-file .env up -d db -``` - -2) Backend dev server: - -```bash -cd backend -cp .env.example .env -uv sync --extra dev -uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 -``` - -3) Frontend dev server: - -```bash -cd frontend -cp .env.example .env.local -# ensure this is correct for the browser: -# NEXT_PUBLIC_API_URL=http://localhost:8000 -npm install -npm run dev -``` - -### Loop B: all-in-one Compose - -```bash -cp .env.example .env - -docker compose -f compose.yml --env-file .env up -d --build -``` - -## Checks (CI parity) - -### Run everything - -```bash -make check -``` - -Evidence: `Makefile`. - -### Common targeted commands - -Backend: -```bash -make backend-lint # flake8 -make backend-typecheck # mypy --strict -make backend-test # pytest -make backend-coverage # pytest + scoped 100% coverage gate -make backend-migrate # alembic upgrade head -``` - -Frontend: -```bash -make frontend-lint # eslint -make frontend-typecheck # tsc -make frontend-test # vitest -make frontend-build # next build -``` - -## Cypress E2E - -Evidence: `docs/testing/README.md`, `.github/workflows/ci.yml`. - -- E2E uses Clerk’s official Cypress integration (`@clerk/testing`). -- Local run pattern: - -```bash -# terminal 1 -cd frontend -npm run dev -- --hostname 0.0.0.0 --port 3000 - -# terminal 2 -cd frontend -npm run e2e -- --browser chrome -``` - -## Deep dives - -- [Testing guide](testing/README.md) -- [Troubleshooting deep dive](troubleshooting/README.md) diff --git a/docs/04-repo-tour.md b/docs/04-repo-tour.md deleted file mode 100644 index 6e4aeb82..00000000 --- a/docs/04-repo-tour.md +++ /dev/null @@ -1,35 +0,0 @@ -# Repo tour - -High-level map of the codebase so you can quickly find where to change things. - -## Top-level -- `backend/` — FastAPI backend (API server) -- `frontend/` — Next.js frontend (web UI) -- `docs/` — documentation -- `compose.yml` — local/self-host stack (db + backend + frontend) -- `scripts/` — helper scripts - -## Backend: where to look -- App entrypoint + router wiring: `backend/app/main.py` -- Routers: `backend/app/api/*` -- Settings/config: `backend/app/core/config.py` -- Auth (Clerk + agent token): `backend/app/core/auth.py`, `backend/app/core/agent_auth.py` -- Models: `backend/app/models/*` -- Services/domain logic: `backend/app/services/*` - -## Frontend: where to look -- Routes (App Router): `frontend/src/app/*` -- Components: `frontend/src/components/*` -- API utilities: `frontend/src/lib/*` and `frontend/src/api/*` -- Auth (Clerk gating/wrappers): `frontend/src/auth/*` - -## Where to change X - -| You want to… | Start here | -|---|---| -| Add/modify an API endpoint | `backend/app/api/*` + `backend/app/main.py` | -| Change auth behavior | `backend/app/core/auth.py` + `frontend/src/auth/*` | -| Change a UI page | `frontend/src/app/*` | -| Update Compose topology | `compose.yml` | - -Next: see [Architecture](05-architecture.md) for system-level flows. diff --git a/docs/05-architecture.md b/docs/05-architecture.md deleted file mode 100644 index d0056952..00000000 --- a/docs/05-architecture.md +++ /dev/null @@ -1,96 +0,0 @@ -# Architecture - -## Deep dives - -- [Architecture deep dive](architecture/README.md) -- [Gateway protocol](openclaw_gateway_ws.md) - -Mission Control is the **web UI + HTTP API** for operating OpenClaw. It’s where you manage boards, tasks, agents, approvals, and (optionally) gateway connections. - -> Auth note: Mission Control supports two auth modes: `local` (shared bearer token) and `clerk`. - -## Components - -- **Frontend**: Next.js app used by humans - - Location: `frontend/` - - Routes/pages: `frontend/src/app/*` (Next.js App Router) - - API client: generated + custom fetch (see `frontend/src/api/*`, `frontend/src/lib/api-base.ts`) -- **Backend**: FastAPI service exposing REST endpoints - - Location: `backend/` - - Entrypoint: `backend/app/main.py` - - API prefix: `/api/v1/*` -- **Database**: Postgres (see `compose.yml`) -- **Gateway integration (optional)**: backend may call into OpenClaw Gateways over WebSockets - - Client/protocol list: `backend/app/services/openclaw/gateway_rpc.py` - -## Diagram (conceptual) - -```mermaid -flowchart LR - U[User / Browser] -->|HTTP| FE[Next.js Frontend :3000] - FE -->|HTTP /api/v1/*| BE[FastAPI Backend :8000] - - BE -->|SQL| PG[(Postgres :5432)] - - BE -->|WebSocket (optional)| GW[OpenClaw Gateway] - GW --> OC[OpenClaw runtime] -``` - -## How requests flow - -### 1) A human uses the UI - -1. Browser loads the Next.js frontend (`frontend/`). -2. Frontend calls backend endpoints using `NEXT_PUBLIC_API_URL`. -3. Backend routes under `/api/v1/*` (`backend/app/main.py`) and reads/writes Postgres. - -Common UI-driven data shapes: -- “boards/tasks” views → board/task CRUD + streams. -- “activity feed” → activity/events endpoints. - -### 2) Authentication (`local` or Clerk) - -- **Frontend**: - - `local`: token entry + token storage (`frontend/src/components/organisms/LocalAuthLogin.tsx`, `frontend/src/auth/localAuth.ts`). - - `clerk`: Clerk wrappers/hooks (`frontend/src/auth/clerk.tsx`). -- **Frontend → backend**: - - API calls attach `Authorization: Bearer ` from local mode token or Clerk session token (`frontend/src/api/mutator.ts`). -- **Backend**: - - `local`: validates `LOCAL_AUTH_TOKEN`. - - `clerk`: validates Clerk request state via `clerk_backend_api` + `CLERK_SECRET_KEY`. - - Implementation: `backend/app/core/auth.py`. - -### 3) Agent automation surface (`/api/v1/agent/*`) - -Agents can call a dedicated API surface: - -- Router: `backend/app/api/agent.py` (prefix `/agent` → mounted under `/api/v1/agent/*`). -- Authentication: `X-Agent-Token` header (or agent-only Authorization bearer parsing). - - Implementation: `backend/app/core/agent_auth.py`. - -Typical agent flows: -- Heartbeat/presence updates -- Task comment posting (evidence) -- Board memory updates -- Lead coordination actions (if board-lead agent) - -### 4) Streaming/feeds (server-sent events) - -Some endpoints support streaming via SSE (`text/event-stream`). -Notes: -- Uses `sse-starlette` in backend routes (e.g. task/activity/memory routers). - -### 5) Gateway integration (optional) - -Mission Control can coordinate with OpenClaw Gateways over WebSockets. - -- Protocol methods/events list: `backend/app/services/openclaw/gateway_rpc.py`. -- Operator-facing protocol docs: [Gateway WebSocket protocol](openclaw_gateway_ws.md). - -## Where to start reading code - -- Backend entrypoint + router wiring: `backend/app/main.py` -- Auth dependencies + access enforcement: `backend/app/api/deps.py` -- User auth: `backend/app/core/auth.py` -- Agent auth: `backend/app/core/agent_auth.py` -- Agent API surface: `backend/app/api/agent.py` diff --git a/docs/06-configuration.md b/docs/06-configuration.md deleted file mode 100644 index d70184ff..00000000 --- a/docs/06-configuration.md +++ /dev/null @@ -1,116 +0,0 @@ -# Configuration - -This page documents **where configuration comes from**, the key **environment variables**, and a couple operational footguns (migrations, CORS). - -For deployment/production patterns, see: -- [Deployment](deployment/README.md) -- [Production](production/README.md) - -## Configuration sources & precedence - -Mission Control is a 3-service stack (`compose.yml`): Postgres (`db`), backend (`backend`), frontend (`frontend`). - -### Docker Compose (recommended for local/self-host) - -Common pattern: - -```bash -cp .env.example .env - -docker compose -f compose.yml --env-file .env up -d --build -``` - -Precedence (high → low): - -1. Environment exported in your shell (or `-e NAME=value`) -2. Compose `--env-file .env` (variable interpolation) -3. Defaults in `compose.yml` (e.g. `${BACKEND_PORT:-8000}`) -4. Backend defaults via `env_file: ./backend/.env.example` -5. Frontend optional user-managed `frontend/.env` - -> Note: Compose intentionally does **not** load `frontend/.env.example` to avoid placeholder Clerk keys accidentally enabling Clerk. - -### Backend env-file loading (non-Compose) - -Evidence: `backend/app/core/config.py`. - -When running the backend directly (uvicorn), settings are loaded from: -- `backend/.env` (always attempted) -- `.env` (repo root; optional) -- plus process env vars - -## Environment variables (grouped) - -### Root `.env` (Compose-level) - -Template: `.env.example`. - -- Ports: `FRONTEND_PORT`, `BACKEND_PORT`, `POSTGRES_PORT` -- Postgres defaults: `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` -- Backend knobs: `CORS_ORIGINS`, `DB_AUTO_MIGRATE` -- Frontend: `NEXT_PUBLIC_API_URL` (required) - -### Backend - -Template: `backend/.env.example` + settings model `backend/app/core/config.py`. - -- `ENVIRONMENT` -- `LOG_LEVEL` -- `DATABASE_URL` -- `CORS_ORIGINS` -- `DB_AUTO_MIGRATE` - -Clerk: -- `CLERK_SECRET_KEY` (required; backend enforces non-empty) -- `CLERK_API_URL`, `CLERK_VERIFY_IAT`, `CLERK_LEEWAY` - -### Frontend - -Template: `frontend/.env.example`. - -| Variable | Required? | Purpose | Default / example | Footguns | -|---|---:|---|---|---| -| `NEXT_PUBLIC_API_URL` | **yes** | Backend base URL used by the browser | `http://localhost:8000` | Must be browser-reachable | -| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | **yes** | Enables Clerk in the frontend | (none) | Must be a real publishable key | -| `NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL` | optional | Fallback redirect | `/boards` | — | -| `NEXT_PUBLIC_CLERK_AFTER_SIGN_OUT_URL` | optional | Post-logout redirect | `/` | — | - -## Minimal dev configuration - -### Split-mode dev (fastest contributor loop) - -- Start DB via Compose. -- Run backend+frontend dev servers. - -See [Development](03-development.md). - -## Migrations (`DB_AUTO_MIGRATE`) - -Evidence: `backend/app/db/session.py`. - -On backend startup: -- if `DB_AUTO_MIGRATE=true` and migrations exist under `backend/migrations/versions/`, backend runs `alembic upgrade head`. -- otherwise it falls back to `SQLModel.metadata.create_all`. - -Operational guidance: -- Auto-migrate is convenient on a single host. -- In multi-instance deployments, prefer running migrations as an explicit deploy step to avoid race conditions. - -## CORS (`CORS_ORIGINS`) - -Evidence: `backend/app/main.py`, `backend/app/core/config.py`. - -- `CORS_ORIGINS` is a comma-separated list. -- It must include the frontend origin (e.g. `http://localhost:3000`) or browser requests will fail. - -## Common footguns - -- **Frontend env template vs runtime env**: `frontend/.env.example` is a template and `compose.yml` intentionally does **not** load it at runtime. Use user-managed `frontend/.env` (for Compose) or `frontend/.env.local` (for Next dev). -- **`NEXT_PUBLIC_API_URL` reachability**: must work from the browser’s network context (host), not only from within the Docker network. - -## Troubleshooting config issues - -- UI loads but API calls fail / Activity feed blank → `NEXT_PUBLIC_API_URL` is missing/incorrect. -- Backend fails at startup → check required env vars (notably `CLERK_SECRET_KEY`) and migrations. - -See also: `docs/troubleshooting/README.md`. diff --git a/docs/07-api-reference.md b/docs/07-api-reference.md deleted file mode 100644 index 29c18fa2..00000000 --- a/docs/07-api-reference.md +++ /dev/null @@ -1,115 +0,0 @@ -# API / auth - -This page documents how Mission Control’s API surface is organized and how authentication works. - -For deeper backend architecture context, see: -- [Architecture](05-architecture.md) - -## Base path - -Evidence: `backend/app/main.py`. - -- All API routes are mounted under: `/api/v1/*` - -## Auth model (two callers) - -Mission Control has two primary actor types: - -1) **User (Clerk)** — human UI/admin -2) **Agent (`X-Agent-Token`)** — automation - -### User auth (Clerk) - -Evidence: -- backend: `backend/app/core/auth.py` -- config: `backend/app/core/config.py` - -- Frontend calls backend using `Authorization: Bearer `. -- Backend validates requests using the Clerk Backend API SDK with `CLERK_SECRET_KEY`. - -### Agent auth (`X-Agent-Token`) - -Evidence: -- `backend/app/core/agent_auth.py` -- agent API surface: `backend/app/api/agent.py` - -- Agents authenticate with `X-Agent-Token: `. -- Token is verified against the agent’s stored `agent_token_hash`. - -## Route groups (modules) - -Evidence: `backend/app/main.py` includes routers from `backend/app/api/*`. - -| Module | Prefix (under `/api/v1`) | Purpose | -|---|---|---| -| `activity.py` | `/activity` | Activity listing and task-comment feed endpoints. | -| `agent.py` | `/agent` | Agent-scoped API routes for board operations and gateway coordination. | -| `agents.py` | `/agents` | Thin API wrappers for async agent lifecycle operations. | -| `approvals.py` | `/boards/{board_id}/approvals` | Approval listing, streaming, creation, and update endpoints. | -| `auth.py` | `/auth` | Authentication bootstrap endpoints for the Mission Control API. | -| `board_group_memory.py` | `/board-groups/{group_id}/memory` and `/boards/{board_id}/group-memory` | Board-group memory CRUD and streaming endpoints. | -| `board_groups.py` | `/board-groups` | Board group CRUD, snapshot, and heartbeat endpoints. | -| `board_memory.py` | `/boards/{board_id}/memory` | Board memory CRUD and streaming endpoints. | -| `board_onboarding.py` | `/boards/{board_id}/onboarding` | Board onboarding endpoints for user/agent collaboration. | -| `boards.py` | `/boards` | Board CRUD and snapshot endpoints. | -| `gateway.py` | `/gateways` | Thin gateway session-inspection API wrappers. | -| `gateways.py` | `/gateways` | Thin API wrappers for gateway CRUD and template synchronization. | -| `metrics.py` | `/metrics` | Dashboard metric aggregation endpoints. | -| `organizations.py` | `/organizations` | Organization management endpoints and membership/invite flows. | -| `souls_directory.py` | `/souls-directory` | API routes for searching and fetching souls-directory markdown entries. | -| `tasks.py` | `/boards/{board_id}/tasks` | Task API routes for listing, streaming, and mutating board tasks. | -| `users.py` | `/users` | User self-service API endpoints for profile retrieval and updates. | - -## Backend API layer notes (how modules are organized) - -Evidence: `backend/app/main.py`, `backend/app/api/*`, `backend/app/api/deps.py`. - -### Conventions - -- Each file under `backend/app/api/*` typically declares an `APIRouter` (`router = APIRouter(...)`) and defines endpoints with decorators like `@router.get(...)`, `@router.post(...)`, etc. -- Board-scoped modules embed `{board_id}` in the prefix (e.g. `/boards/{board_id}/tasks`). -- Streaming endpoints usually expose **SSE** endpoints at `.../stream` (see `sse-starlette` usage). - -### Where key behaviors live - -- **Router wiring / base prefix**: `backend/app/main.py` mounts these routers under `/api/v1/*`. -- **Auth / access control** is mostly expressed through dependencies (see `backend/app/api/deps.py`): - - `require_admin_auth` — require an authenticated *admin user*. - - `require_admin_or_agent` — allow either an admin user or an authenticated agent. - - `get_board_for_actor_read` / `get_board_for_actor_write` — enforce board access for the calling actor. - - `require_org_member` / `require_org_admin` — enforce org membership/admin for user callers. -- **Agent-only surface**: `backend/app/api/agent.py` uses `get_agent_auth_context` (X-Agent-Token) and contains board/task/memory endpoints specifically for automation. - -### Module-by-module map (prefix, key endpoints, and pointers) - -This is a “where to look” index, not a full OpenAPI dump. For exact parameters and response shapes, see: -- route module file (`backend/app/api/.py`) -- schemas (`backend/app/schemas/*`) -- models (`backend/app/models/*`) -- services (`backend/app/services/*`) - -| Module | Prefix (under `/api/v1`) | Key endpoints (examples) | Main deps / auth | Pointers (schemas/models/services) | -|---|---|---|---|---| -| `activity.py` | `/activity` | `GET /activity` (events); `GET /activity/task-comments` + `/stream` | `require_admin_or_agent`, `require_org_member` | `app/models/activity_events.py`, `app/schemas/activity_events.py` | -| `agent.py` | `/agent` | agent automation surface: boards/tasks/memory + gateway coordination | `get_agent_auth_context` (X-Agent-Token) | `backend/app/core/agent_auth.py`, `backend/app/services/openclaw/*` | -| `agents.py` | `/agents` | agent lifecycle + SSE stream + heartbeat | org-admin gated for user callers; some endpoints allow agent access via deps | `app/schemas/agents.py`, `app/services/openclaw/provisioning_db.py` | -| `approvals.py` | `/boards/{board_id}/approvals` | list/create/update approvals + `/stream` | `require_admin_or_agent` + board access deps | `app/models/approvals.py`, `app/schemas/approvals.py` | - -## Where authorization is enforced - -Evidence: `backend/app/api/deps.py`. - -Most route modules don’t “hand roll” access checks; they declare dependencies: - -- `require_admin_auth` — admin user only. -- `require_admin_or_agent` — admin user OR authenticated agent. -- `get_board_for_actor_read` / `get_board_for_actor_write` — board access for user/agent. -- `require_org_member` / `require_org_admin` — org membership/admin for user callers. - -## “Start here” pointers for maintainers - -- Router wiring: `backend/app/main.py` -- Access dependencies: `backend/app/api/deps.py` -- User auth: `backend/app/core/auth.py` -- Agent auth: `backend/app/core/agent_auth.py` -- Agent automation surface: `backend/app/api/agent.py` diff --git a/docs/08-agents-and-skills.md b/docs/08-agents-and-skills.md deleted file mode 100644 index 17121549..00000000 --- a/docs/08-agents-and-skills.md +++ /dev/null @@ -1,28 +0,0 @@ -# Agents & skills - -## Deep dives - -- [Gateway protocol](openclaw_gateway_ws.md) -- [Gateway base config](openclaw_gateway_base_config.md) - -This page explains the automation model as it appears in Mission Control. - -## Agent lifecycle (conceptual) -- An **agent** checks in to Mission Control (often on a schedule) and posts work results as task comments. -- In OpenClaw terms, agents can run: - - **heartbeats** (periodic loops) - - **cron jobs** (scheduled runs; better for exact timing / isolation) - -## Heartbeats vs cron -- Use **heartbeat** for batched checks and context-aware incremental work. -- Use **cron** for exact timing and isolated, standalone actions. - -## Skills (how to think about them) -- A skill is a packaged workflow/tooling instruction set that agents can follow. -- Skills typically define: - - when to use them - - required binaries/services - - command patterns - -## Next -- Add repo-specific guidance for authoring skills and where they live (once standardized). diff --git a/docs/09-ops-runbooks.md b/docs/09-ops-runbooks.md deleted file mode 100644 index 1d6b11da..00000000 --- a/docs/09-ops-runbooks.md +++ /dev/null @@ -1,81 +0,0 @@ -# Operations - -This is the ops/SRE entrypoint. - -It aims to answer, quickly: -- “Is the system up?” -- “What changed?” -- “What should I check next?” - -Deep dives: -- [Deployment](deployment/README.md) -- [Production](production/README.md) -- [Troubleshooting deep dive](troubleshooting/README.md) - -## First 30 minutes (incident checklist) - -### 0) Stabilize communications - -- Identify incident lead and comms channel. -- Capture last deploy SHA/tag and time window. -- Do not paste secrets into chat/tickets. - -### 1) Confirm impact - -- UI broken vs API broken vs auth vs DB vs gateway integration. -- All users or subset? - -### 2) Health checks - -- Backend: - - `curl -f http://:8000/healthz` - - `curl -f http://:8000/readyz` -- Frontend: - - can the UI load? - - in browser devtools, are `/api/v1/*` requests failing? - -### 3) Configuration sanity - -Common misconfigs that look like outages: - -- `NEXT_PUBLIC_API_URL` wrong → UI loads but API calls fail. -- `CORS_ORIGINS` missing frontend origin → browser CORS errors. -- Clerk misconfig → auth redirects/401s. - -### 4) Database - -- If backend is 5xx’ing broadly, DB is a top suspect. -- Verify `DATABASE_URL` points at the correct host. - -### 5) Logs - -Compose: - -```bash -docker compose -f compose.yml --env-file .env logs -f --tail=200 -``` - -Targeted: - -```bash -docker compose -f compose.yml --env-file .env logs -f --tail=200 backend -``` - -### 6) Rollback / isolate - -- If there was a recent deploy and symptoms align, rollback to last known good. -- If gateway integration is implicated, isolate by disabling gateway-dependent flows. - -## Common failure modes - -- UI loads, Activity feed blank → `NEXT_PUBLIC_API_URL` wrong/unreachable. -- Repeated auth redirects/errors → Clerk keys/redirects misconfigured. -- Backend 5xx → DB outage/misconfig; migration failure. -- Backend won’t start → config validation failure (e.g. empty `CLERK_SECRET_KEY`). - -## Backups - -Evidence: `docs/production/README.md`. - -- Minimum viable: periodic `pg_dump` to off-host storage. -- Treat restore as a drill (quarterly), not a one-time checklist. diff --git a/docs/10-troubleshooting.md b/docs/10-troubleshooting.md deleted file mode 100644 index ee3b1096..00000000 --- a/docs/10-troubleshooting.md +++ /dev/null @@ -1,38 +0,0 @@ -# Troubleshooting - -This is the “symptom → checks → likely fixes” page. - -For deeper playbooks, see: -- [Troubleshooting deep dive](troubleshooting/README.md) - -## Triage map - -| Symptom | Fast checks | Likely fix | -|---|---|---| -| UI loads but API calls fail / Activity feed blank | Browser devtools shows `/api/v1/*` failures; check backend `/healthz` | Fix `NEXT_PUBLIC_API_URL` (must be browser-reachable) | -| UI redirects / Clerk errors | Is `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` set? Are Clerk redirects correct? | Unset keys for local dev without Clerk; configure real keys for prod | -| Backend `/healthz` fails | Is backend container/process running? check backend logs | Fix crash loop: env vars, DB connectivity, migrations | -| Backend returns 5xx | Check DB connectivity (`DATABASE_URL`), DB logs | Fix DB outage/misconfig; re-run migrations if needed | -| Browser shows CORS errors | Compare `CORS_ORIGINS` vs frontend origin | Add frontend origin to `CORS_ORIGINS` | - -## Common checks - -### 1) Verify backend health - -```bash -curl -f http://localhost:8000/healthz -``` - -### 2) Verify frontend can reach backend - -- Ensure `NEXT_PUBLIC_API_URL` matches the backend URL the browser can reach. - -### 3) Check logs - -```bash -docker compose -f compose.yml --env-file .env logs -f --tail=200 backend -``` - -## Next - -If you hit a recurring incident, promote it from the deep-dive page into this triage map. diff --git a/docs/11-contributing.md b/docs/11-contributing.md deleted file mode 100644 index f5b2ac94..00000000 --- a/docs/11-contributing.md +++ /dev/null @@ -1,18 +0,0 @@ -# Contributing - -## Deep dives - -- [Coverage policy](coverage-policy.md) -- [Testing guide](testing/README.md) - -## How to contribute -- Follow the repo’s existing PR and review conventions. -- Prefer small PRs. -- Update docs when behavior changes. - -## Adding/changing docs -- Canonical docs live in `docs/`. -- Follow the IA in [Docs landing](README.md). - -## Testing expectations -- See [docs/testing/README.md](testing/README.md). diff --git a/docs/12-backend-core.md b/docs/12-backend-core.md deleted file mode 100644 index 980a20c4..00000000 --- a/docs/12-backend-core.md +++ /dev/null @@ -1,137 +0,0 @@ -# Backend core modules (auth/config/logging/errors) - -> Evidence basis: repo https://github.com/abhi1693/openclaw-mission-control @ commit `c3490630a4503d9c8142aaa3abf542e0d00b5035`. - -This page documents the backend “core” layer under `backend/app/core/*` plus the API dependency module `backend/app/api/deps.py`. - -It’s written for maintainers who need to answer: - -- “Where does configuration come from?” -- “How do user vs agent auth work?” -- “Where are authorization decisions enforced?” -- “What’s the error envelope / request-id behavior?” -- “How is logging structured and how do I get request-context in logs?” - -## Start here (reading order) - -1. `backend/app/core/config.py` — settings + env file loading -2. `backend/app/core/logging.py` — structured logging + request context -3. `backend/app/core/error_handling.py` — request-id middleware + exception envelope -4. `backend/app/core/auth.py` — Clerk/user auth resolution -5. `backend/app/core/agent_auth.py` — agent token auth resolution -6. `backend/app/api/deps.py` — how routes declare and enforce access - -## Configuration: loading & precedence - -**Primary file:** `backend/app/core/config.py` - -Key facts: -- Uses `pydantic-settings` (`BaseSettings`) to load typed settings from environment. -- Env files are loaded regardless of current working directory: - - `backend/.env` (via `DEFAULT_ENV_FILE`) - - then `.env` (repo root) as an additional source - - See `Settings.model_config.env_file=[DEFAULT_ENV_FILE, ".env"]`. -- Unknown env vars are ignored (`extra="ignore"`). - -Notable settings (security-sensitive in **bold**): -- `DATABASE_URL` / `database_url` -- `CORS_ORIGINS` / `cors_origins` -- `DB_AUTO_MIGRATE` / `db_auto_migrate` -- **`CLERK_SECRET_KEY` / `clerk_secret_key`** (must be non-empty; validator enforces it) -- `CLERK_API_URL`, `CLERK_VERIFY_IAT`, `CLERK_LEEWAY` -- logging knobs: `LOG_LEVEL`, `LOG_FORMAT`, `LOG_USE_UTC`, `REQUEST_LOG_SLOW_MS`, `REQUEST_LOG_INCLUDE_HEALTH` - -### Deployment implication - -- If a deployment accidentally starts the backend with an empty/placeholder `CLERK_SECRET_KEY`, the backend will fail settings validation at startup. - -## Auth model split - -The backend supports two top-level actor types: - -- **User** (human UI / admin) — resolved from the `Authorization: Bearer ` header via Clerk. -- **Agent** (automation) — resolved from `X-Agent-Token: ` (and optionally `Authorization: Bearer ` for agent callers). - -### User auth (Clerk) — `backend/app/core/auth.py` - -What it does: -- Uses the `clerk_backend_api` SDK to authenticate requests (`authenticate_request(...)`) using `CLERK_SECRET_KEY`. -- Resolves a `AuthContext` containing `actor_type="user"` and a `User` model instance. -- The module includes helpers to fetch user profile details from Clerk (`_fetch_clerk_profile`) and to delete a Clerk user (`delete_clerk_user`). - -Security-sensitive notes: -- Treat `CLERK_SECRET_KEY` as a credential; never log it. -- This code calls Clerk API endpoints over the network (timeouts and error handling matter). - -### Agent auth (token hash) — `backend/app/core/agent_auth.py` - -What it does: -- Requires a token header for protected agent endpoints: - - Primary header: `X-Agent-Token` - - Optional parsing: `Authorization: Bearer ...` (only in `get_agent_auth_context`, and only if `accept_authorization=True`) -- Validates token by comparing it against stored `agent_token_hash` values in the DB (`verify_agent_token`). -- “Touches” agent presence (`last_seen_at`, `status`) on authenticated requests. - - For safe methods (`GET/HEAD/OPTIONS`), it commits immediately so read-only polling still shows the agent as online. - -Security-sensitive notes: -- Token verification iterates over agents with a token hash. If this grows large, consider indexing/lookup strategy. -- Never echo full tokens in logs; current code logs only a prefix on invalid tokens. - -## Authorization enforcement: `backend/app/api/deps.py` - -This module is the primary “policy wiring” for most routes. - -Key concepts: - -- `require_admin_auth(...)` - - Requires an authenticated *admin user*. -- `require_admin_or_agent(...)` → returns `ActorContext` - - Allows either: - - admin user (user auth via Clerk), or - - authenticated agent (agent auth via X-Agent-Token). - -Board/task access patterns: -- `get_board_for_actor_read` / `get_board_for_actor_write` - - Enforces that the caller (user or agent) has the correct access to the board. - - Agent access is restricted if the agent is bound to a specific board (`agent.board_id`). -- `get_task_or_404` - - Loads a task and ensures it belongs to the requested board. - -Org access patterns (user callers): -- `require_org_member` and `require_org_admin` - - Resolve/require active org membership. - - Provide an `OrganizationContext` with `organization` + `member`. - -Maintainer tip: -- When debugging a “why is this 403/401?”, start by checking the route’s dependency stack (in the route module) and trace through the relevant dependency in `deps.py`. - -## Logging: structure + request context - -**Primary file:** `backend/app/core/logging.py` - -Highlights: -- Defines a custom TRACE level (`TRACE_LEVEL = 5`). -- Uses `contextvars` to carry `request_id`, `method`, and `path` across async tasks. -- `AppLogFilter` injects `app`, `version`, and request context into each log record. -- Supports JSON output (`JsonFormatter`) and key=value (`KeyValueFormatter`) formats. - -Where request context gets set: -- `backend/app/core/error_handling.py` middleware calls: - - `set_request_id(...)` - - `set_request_route_context(method, path)` - -## Error envelope + request-id - -**Primary file:** `backend/app/core/error_handling.py` - -Key behaviors: -- Installs a `RequestIdMiddleware` (ASGI) that: - - Accepts client-provided `X-Request-Id` or generates one. - - Adds `X-Request-Id` to the response. - - Emits structured “http.request.*” logs, including “slow request” warnings. -- Error responses include `request_id` when available: - - Validation errors (`422`) return `{detail: , request_id: ...}`. - - Other HTTP errors are wrapped similarly. - -Maintainer tip: -- When debugging incidents, ask for the `X-Request-Id` from the client and use it to locate backend logs quickly. diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 243ba080..00000000 --- a/docs/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# Mission Control docs - -This folder is the canonical documentation set for Mission Control. - -## Start here (by role) - -- **Contributor**: start with [Quickstart](../README.md#quick-start-self-host-with-docker-compose) → [Development](development.md) → [Contributing](contributing.md) -- **Maintainer**: start with [Architecture](05-architecture.md) → [Repo tour](04-repo-tour.md) → [API reference](07-api-reference.md) -- **Operator/SRE**: start with [Ops / runbooks](09-ops-runbooks.md) → [Troubleshooting](10-troubleshooting.md) - -## Table of contents (IA) - -- [Style guide](00-style-guide.md) - - -1. [Overview](01-overview.md) -2. [Quickstart](02-quickstart.md) -3. [Development](03-development.md) -4. [Repo tour](04-repo-tour.md) -5. [Architecture](05-architecture.md) -6. [Configuration](06-configuration.md) -7. [API reference](07-api-reference.md) - - [Frontend API + auth modules](frontend-api-auth.md) -8. [Agents & skills](08-agents-and-skills.md) -9. [Ops / runbooks](09-ops-runbooks.md) -10. [Troubleshooting](10-troubleshooting.md) -11. [Contributing](11-contributing.md) - -## Existing deep-dive docs - -These deeper references already exist under `docs/` directories: -- [Architecture deep dive](architecture/README.md) -- [Deployment guide](deployment/README.md) -- [Production notes](production/README.md) -- [Testing guide](testing/README.md) -- [Troubleshooting](troubleshooting/README.md) -- [Gateway WebSocket protocol](openclaw_gateway_ws.md) -- [Gateway base config](openclaw_gateway_base_config.md) diff --git a/docs/architecture/README.md b/docs/architecture/README.md deleted file mode 100644 index ca78f901..00000000 --- a/docs/architecture/README.md +++ /dev/null @@ -1,125 +0,0 @@ -# Mission Control — Architecture - -Mission Control is the **web UI + HTTP API** for operating OpenClaw. It’s where you manage boards, tasks, agents, approvals, and (optionally) gateway connections. - -> Auth note: Mission Control supports two auth modes: `local` (shared bearer token) and `clerk`. - -At a high level: -- The **frontend** is a Next.js app used by humans. -- The **backend** is a FastAPI service that exposes REST endpoints under `/api/v1/*`. -- **Postgres** stores core state (boards/tasks/agents/etc.). - -## Components - -### Diagram (conceptual) - -```mermaid -flowchart LR - U[User / Browser] -->|HTTP| FE[Next.js Frontend :3000] - FE -->|HTTP /api/v1/*| BE[FastAPI Backend :8000] - - BE -->|SQL| PG[(Postgres :5432)] - - BE -->|WebSocket (optional integration)| GW[OpenClaw Gateway] - GW --> OC[OpenClaw runtime] -``` - -### Frontend (Next.js) -- Location: `frontend/` -- Routes/pages: `frontend/src/app/*` (Next.js App Router) -- API utilities: `frontend/src/lib/*` and `frontend/src/api/*` - -**Auth (`local` or Clerk)** -- `local` mode authenticates a shared bearer token (`LOCAL_AUTH_TOKEN`) and resolves a local user context. -- `clerk` mode verifies Clerk JWTs using `CLERK_SECRET_KEY`. -- Frontend mode switch + wrappers: `frontend/src/auth/clerk.tsx`, `frontend/src/auth/localAuth.ts`, and `frontend/src/components/providers/AuthProvider.tsx`. -- Backend mode switch: `backend/app/core/config.py` and `backend/app/core/auth.py`. - - -### Backend (FastAPI) -- Location: `backend/` -- App wiring: `backend/app/main.py` - - Health: `/health`, `/healthz`, `/readyz` - - API prefix: `/api/v1` - - Routers: `backend/app/api/*` - -**Config** -- Settings: `backend/app/core/config.py` -- Env loading: always reads `backend/.env` (and optionally `.env`) so running from repo root still works. - -### Data stores -- **Postgres**: persistence for boards/tasks/agents/approvals/etc. - - Models: `backend/app/models/*` - - Migrations: `backend/migrations/*` - -### Gateway integration (optional) -Mission Control can call into an OpenClaw Gateway over WebSockets. -- Client + protocol: `backend/app/services/openclaw/gateway_rpc.py` -- Protocol doc: [Gateway WebSocket protocol](../openclaw_gateway_ws.md) -- Base gateway config (getting started): [Gateway base config](../openclaw_gateway_base_config.md) - -## Request flows - -### UI → API -1. Browser loads the Next.js frontend. -2. Frontend calls backend endpoints under `/api/v1/*`. -3. Backend reads/writes Postgres. - -### Auth (`local` or Clerk) -- **Frontend**: - - `local`: token entry screen + session storage token (`frontend/src/components/organisms/LocalAuthLogin.tsx`, `frontend/src/auth/localAuth.ts`). - - `clerk`: Clerk wrappers/hooks (`frontend/src/auth/clerk.tsx`). -- **Backend**: - - `local`: validates `Authorization: Bearer `. - - `clerk`: validates Clerk request state with SDK + `CLERK_SECRET_KEY`. -### Agent access (X-Agent-Token) -Automation/agents can use the “agent” API surface: -- Endpoints under `/api/v1/agent/*` (router: `backend/app/api/agent.py`). -- Auth via `X-Agent-Token` (see `backend/app/core/agent_auth.py`, referenced from `backend/app/api/deps.py`). - -### Background jobs -There is currently no queue runtime configured in this repo. - -## Key directories - -Repo root: -- `compose.yml` — local/self-host stack -- `.env.example` — compose/local defaults -- `backend/templates/` — shared templates - -Backend: -- `backend/app/api/` — REST routers -- `backend/app/core/` — config/auth/logging/errors -- `backend/app/models/` — SQLModel models -- `backend/app/services/` — domain logic -- `backend/app/integrations/` — gateway client/protocol - -Frontend: -- `frontend/src/app/` — Next.js routes -- `frontend/src/components/` — UI components -- `frontend/src/auth/` — auth mode helpers (`clerk` and `local`) -- `frontend/src/lib/` — utilities + API base - -## Where to start reading code - -Backend: -1. `backend/app/main.py` — app + routers -2. `backend/app/core/config.py` — env + defaults -3. `backend/app/core/auth.py` — auth behavior -4. `backend/app/api/tasks.py` and `backend/app/api/agent.py` — core flows - -Frontend: -1. `frontend/src/app/*` — main UI routes -2. `frontend/src/lib/api-base.ts` — backend calls -3. `frontend/src/auth/*` — auth mode integration (`local` + Clerk) - -## Related docs -- Self-host (Docker Compose): see repo root README: [Quick start (self-host with Docker Compose)](../../README.md#quick-start-self-host-with-docker-compose) -- Production-ish deployment: [Production notes](../production/README.md) -- Testing (Cypress/Clerk): [Testing guide](../testing/README.md) -- Troubleshooting: [Troubleshooting](../troubleshooting/README.md) - -## Notes / gotchas -- Mermaid rendering depends on the markdown renderer. -- `NEXT_PUBLIC_API_URL` must be reachable from the browser (host), not just from within Docker. -- If Compose loads `frontend/.env.example` directly, placeholder Clerk keys can accidentally enable Clerk; prefer user-managed env files. diff --git a/docs/coverage-policy.md b/docs/coverage-policy.md deleted file mode 100644 index 6ed1b568..00000000 --- a/docs/coverage-policy.md +++ /dev/null @@ -1,47 +0,0 @@ -# Coverage policy (CI gate) - -## Why scoped coverage gates? - -Today, overall repository coverage is low (especially for API routes and Next pages), but we still want CI to **enforce quality deterministically**. - -So we start with a strict gate (100% statements + branches) on a **small, explicitly scoped** set of modules that are: - -- unit-testable without external services -- stable and high-signal for regressions - -We then expand the gated scope as we add tests. - -## Backend scope (100% required) - -Enforced in `Makefile` target `backend-coverage`: - -- `app.core.error_handling` -- `app.services.mentions` - -Command (CI): - -```bash -cd backend && uv run pytest \ - --cov=app.core.error_handling \ - --cov=app.services.mentions \ - --cov-branch \ - --cov-report=term-missing \ - --cov-report=xml:coverage.xml \ - --cov-report=json:coverage.json \ - --cov-fail-under=100 -``` - -## Frontend scope (100% required) - -Enforced in `frontend/vitest.config.ts` coverage settings: - -- include: `src/lib/backoff.ts` -- thresholds: 100% for lines/statements/functions/branches - -This is intentionally limited to a single pure utility module first. As we add more unit tests in `src/lib/**` and React Testing Library component tests for `src/app/**` + `src/components/**`, we should expand the include list and keep thresholds strict. - -## How to expand the gate - -- Add tests for the next-highest-signal modules. -- Add them to the gated scope (backend `--cov=` list; frontend `coverage.include`). -- Keep the threshold at 100% for anything included in the gate. diff --git a/docs/deployment/README.md b/docs/deployment/README.md deleted file mode 100644 index 3423ab26..00000000 --- a/docs/deployment/README.md +++ /dev/null @@ -1,215 +0,0 @@ -# Deployment / Self-hosting (Docker Compose) - -This guide covers how to self-host **OpenClaw Mission Control** using the repository’s `compose.yml`. - -> Scope -> - This is a **dev-friendly self-host** setup intended for local or single-host deployments. -> - For production hardening (TLS, backups, external Postgres, observability), see **Production notes** below. - -## What you get - -When running Compose, you get: - -- **Postgres** database (persistent volume) -- **Backend API** (FastAPI) on `http://localhost:${BACKEND_PORT:-8000}` - - Health check: `GET /healthz` -- **Frontend UI** (Next.js) on `http://localhost:${FRONTEND_PORT:-3000}` - -Auth is configurable per deployment: -- `AUTH_MODE=local` (self-host default; shared bearer token) -- `AUTH_MODE=clerk` (Clerk JWT auth; backend requires `CLERK_SECRET_KEY`) - -## Requirements - -- Docker Engine -- Docker Compose **v2** (`docker compose ...`) -- Recommended: **4GB+ RAM** (frontend build can be memory/CPU intensive) - -## Quick start (self-host) - -From repo root: - -```bash -cp .env.example .env - -# REQUIRED for local mode: -# set LOCAL_AUTH_TOKEN in .env to a non-placeholder value with at least 50 characters. - -docker compose -f compose.yml --env-file .env up -d --build -``` - -Check containers: - -```bash -docker compose -f compose.yml ps -``` - -## Sanity checks - -Backend health: - -```bash -curl -f http://localhost:${BACKEND_PORT:-8000}/healthz -``` - -Frontend serving: - -```bash -curl -I http://localhost:${FRONTEND_PORT:-3000}/ -``` - -## Compose overview - -### Services - -`compose.yml` defines: - -- `db` (Postgres 16) -- `backend` (FastAPI) -- `frontend` (Next.js) - -### Ports - -By default: - -- Postgres: `5432` (`POSTGRES_PORT`) -- Backend: `8000` (`BACKEND_PORT`) -- Frontend: `3000` (`FRONTEND_PORT`) - -Ports are sourced from `.env` (passed via `--env-file .env`) and wired into `compose.yml`. - -### Volumes (data persistence) - -Compose creates named volumes: - -- `postgres_data` → Postgres data directory - -These persist across `docker compose down`. - -## Environment strategy - -### Root `.env` (Compose) - -- Copy the template: `cp .env.example .env` -- Edit values as needed (ports, auth mode, tokens, API URL, etc.) - -Compose is invoked with: - -```bash -docker compose -f compose.yml --env-file .env ... -``` - -### Backend env - -The backend container loads `./backend/.env.example` via `env_file` and then overrides the DB URL for container networking. - -If you need backend customization, prefer creating a real `backend/.env` and updating compose to use it (optional improvement). - -### Frontend env - -`compose.yml` intentionally **does not** load `frontend/.env.example` at runtime, because it may contain non-empty placeholders. - -Instead, it supports an optional user-managed env file: - -- `frontend/.env` (not committed) - -If present, Compose will load it. - -## Authentication modes - -Mission Control supports two deployment auth modes: - -- `AUTH_MODE=local`: shared bearer token auth (self-host default) -- `AUTH_MODE=clerk`: Clerk JWT auth - -### Local mode (self-host default) - -Set in `.env` (repo root): - -```env -AUTH_MODE=local -LOCAL_AUTH_TOKEN=replace-with-random-token-at-least-50-characters -``` - -Set frontend mode (optional override in `frontend/.env`): - -```env -NEXT_PUBLIC_AUTH_MODE=local -NEXT_PUBLIC_API_URL=http://localhost:8000 -``` - -Users enter `LOCAL_AUTH_TOKEN` in the local login screen. - -### Clerk mode - -Set in `.env` (repo root): - -```env -AUTH_MODE=clerk -``` - -Create `backend/.env` with at least: - -```env -CLERK_SECRET_KEY=sk_test_your_real_key -CLERK_API_URL=https://api.clerk.com -CLERK_VERIFY_IAT=true -CLERK_LEEWAY=10.0 -``` - -Create `frontend/.env` with at least: - -```env -NEXT_PUBLIC_AUTH_MODE=clerk -NEXT_PUBLIC_API_URL=http://localhost:8000 -NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_test_your_real_key -``` - -**Security:** treat `LOCAL_AUTH_TOKEN` and `CLERK_SECRET_KEY` like passwords. Do not commit them. - -## Troubleshooting - -### 1) Check container status - -```bash -docker compose -f compose.yml ps -``` - -### 2) Tail logs - -```bash -docker compose -f compose.yml --env-file .env logs -f --tail=200 -``` - -### 3) Common issues - -- **Docker permission denied** (`/var/run/docker.sock`) - - Ensure your user is in the `docker` group and your session picked it up (re-login), or use a root/sudo-capable host. -- **Frontend build fails because of missing `public/`** - - If the repo doesn’t have `frontend/public`, the Dockerfile should not `COPY public/`. -- **Backend build fails looking for `uv.lock`** - - If backend build context is repo root, Dockerfile must copy `backend/uv.lock` not `uv.lock`. - -## Reset / start fresh - -Safe (keeps volumes/data): - -```bash -docker compose -f compose.yml --env-file .env down -``` - -Destructive (removes volumes; deletes Postgres data): - -```bash -docker compose -f compose.yml --env-file .env down -v -``` - -## Production notes (future) - -If you’re running this beyond local dev, consider: - -- Run Postgres as a managed service (or on a separate host) -- Add TLS termination (reverse proxy) -- Configure backups for Postgres volume -- Set explicit resource limits and healthchecks -- Pin image versions/tags and consider multi-arch builds diff --git a/docs/e2e-auth.md b/docs/e2e-auth.md deleted file mode 100644 index 13ae5b98..00000000 --- a/docs/e2e-auth.md +++ /dev/null @@ -1,10 +0,0 @@ -# E2E auth (Cypress) - -Hard requirement: **no auth bypass** for Cypress E2E. - -- Cypress tests must use real Clerk sign-in. -- CI should inject Clerk keys into the Cypress job environment. - -Test account (non-secret): -- email: `jane+clerk_test@example.com` -- OTP: `424242` diff --git a/docs/frontend-api-auth.md b/docs/frontend-api-auth.md deleted file mode 100644 index c7e19899..00000000 --- a/docs/frontend-api-auth.md +++ /dev/null @@ -1,109 +0,0 @@ -# Frontend API client and auth integration - -This page documents the frontend integration points you’ll touch when changing how the UI talks to the backend or how auth is applied. - -## Related docs - -- [Architecture](05-architecture.md) -- [Configuration](06-configuration.md) -- [API reference](07-api-reference.md) - -## API base URL - -The frontend uses `NEXT_PUBLIC_API_URL` as the single source of truth for where to send API requests. - -- Code: `frontend/src/lib/api-base.ts` -- Behavior: - - reads `process.env.NEXT_PUBLIC_API_URL` - - normalizes by trimming trailing slashes - - throws early if missing/invalid - -In Docker Compose, `compose.yml` sets `NEXT_PUBLIC_API_URL` both: -- as a **build arg** (for `next build`), and -- as a **runtime env var**. - -## API client layout - -### Generated client - -- Location: `frontend/src/api/generated/*` -- Generator: **Orval** - - Config: `frontend/orval.config.ts` - - Script: `cd frontend && npm run api:gen` - - Convenience target: `make api-gen` - -By default, Orval reads the backend OpenAPI schema from: -- `ORVAL_INPUT` (if set), otherwise -- `http://127.0.0.1:8000/openapi.json` - -Output details (from `orval.config.ts`): -- Mode: `tags-split` -- Target index: `frontend/src/api/generated/index.ts` -- Schemas: `frontend/src/api/generated/model` -- Client: `react-query` -- All requests go through the custom mutator below. - -### Custom fetch / mutator - -All generated requests go through: - -- Code: `frontend/src/api/mutator.ts` -- What it does: - - resolves `NEXT_PUBLIC_API_URL` and builds the full request URL - - sets `Content-Type: application/json` when there’s a body - - injects `Authorization: Bearer ` when a Clerk session token is available - - converts non-2xx responses into a typed `ApiError` (status + parsed response) - -## Auth enablement and token injection - -### Clerk enablement (publishable key gating) - -Clerk is enabled in the frontend only when `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` looks valid. - -- Gating helper (dependency-free): `frontend/src/auth/clerkKey.ts` -- UI-safe wrappers/hooks: `frontend/src/auth/clerk.tsx` - - provides `SignedIn`, `SignedOut`, `SignInButton`, `SignOutButton`, `useUser`, and `useAuth` - - returns safe fallbacks when Clerk is disabled (to allow secretless builds/prerender) - -### Token injection - -When the UI makes an API request, the mutator attempts to read a token from the Clerk session: - -- Code: `frontend/src/api/mutator.ts` (`resolveClerkToken()`) -- If a token is available, the request includes: - - `Authorization: Bearer ` - -### Route protection (middleware) - -Request-time route protection is implemented via Next.js middleware: - -- Code: `frontend/src/proxy.ts` -- Behavior: - - when Clerk is enabled: uses `clerkMiddleware()` to enforce auth on non-public routes - - when Clerk is disabled: passes all requests through - -## Common workflows - -### Update the backend API and regenerate the client - -1. Run the backend so OpenAPI is available: - -```bash -# from repo root -cp backend/.env.example backend/.env -make backend-migrate -cd backend && uv run uvicorn app.main:app --reload --port 8000 -``` - -2. Regenerate the client: - -```bash -# from repo root -make api-gen - -# or from frontend/ -ORVAL_INPUT=http://127.0.0.1:8000/openapi.json npm run api:gen -``` - -3. Review diffs under `frontend/src/api/generated/*`. - diff --git a/docs/openclaw_gateway_base_config.md b/docs/openclaw_gateway_base_config.md deleted file mode 100644 index 38b7b0ae..00000000 --- a/docs/openclaw_gateway_base_config.md +++ /dev/null @@ -1,167 +0,0 @@ -# OpenClaw Gateway Base Config (Local) - -This document explains a "base" OpenClaw Gateway config intended for local development and LAN use, and how to connect it to Mission Control. - -Related: -- Gateway WebSocket protocol: [Gateway WebSocket protocol](openclaw_gateway_ws.md) (default URL `ws://127.0.0.1:18789`) -- Mission Control architecture (gateway integration): [Architecture deep dive](architecture/README.md) - -## Who This Is For - -Use this config if you want: -- A gateway listening locally (and optionally on your LAN) -- A predictable workspace location for agents/sessions -- A small, readable starting point you can extend - -## Base Config (Template) - -Start from this template and change the values in **Required edits**. - -```json -{ - "agents": { - "defaults": { - "model": { - "primary": "openai-codex/gpt-5.2" - }, - "models": { - "openai-codex/gpt-5.2": {} - }, - "workspace": "~/.openclaw/workspace", - "compaction": { - "mode": "safeguard" - }, - "thinkingDefault": "minimal", - "maxConcurrent": 4, - "subagents": { - "maxConcurrent": 8 - } - } - }, - "gateway": { - "port": 18789, - "mode": "local", - "bind": "lan", - "controlUi": { - "allowInsecureAuth": true - }, - "tailscale": { - "mode": "off", - "resetOnExit": false - }, - "reload": { - "mode": "hot", - "debounceMs": 750 - } - }, - "memory": { - "backend": "qmd", - "citations": "auto", - "qmd": { - "includeDefaultMemory": true, - "update": { - "interval": "15m", - "debounceMs": 15000, - "onBoot": true - }, - "limits": { - "maxResults": 6, - "maxSnippetChars": 900, - "maxInjectedChars": 4500, - "timeoutMs": 4000 - } - } - }, - "skills": { - "install": { - "nodeManager": "npm" - } - } -} -``` - -### Required Edits - -1. `agents.defaults.workspace` - - Set a directory the gateway can read/write. - - Recommended default is `~/.openclaw/workspace` (Mission Control defaults the "workspace root" UI to `~/.openclaw`). - -### Optional Edits (Common) - -- `gateway.bind` - - `"lan"` is convenient for reaching the gateway from another machine on your network. - - If you want local-only access, configure your gateway to bind only to localhost/loopback (exact values depend on the gateway build) and/or firewall the port. -- `gateway.port` - - Change if `18789` is already in use. - - Mission Control requires the URL to include an explicit port (example: `ws://127.0.0.1:18789`). -- `agents.defaults.model.primary` - - Set your preferred default model identifier. -- `memory.qmd.limits` - - Tune memory query/injection sizes if you see timeouts or overly-large prompts. - -## What Each Top-Level Section Does - -### `agents.defaults` - -Default runtime behavior for agents created/managed by the gateway: -- `model.primary`: default model identifier -- `models`: per-model overrides (empty means "use provider defaults") -- `workspace`: where agent state/files live on disk -- `compaction.mode: "safeguard"`: enables conservative compaction behavior -- `thinkingDefault: "minimal"`: default "thinking" level -- `maxConcurrent`: max concurrent top-level runs -- `subagents.maxConcurrent`: max concurrent subagent runs - -### `gateway` - -Network/runtime settings for the gateway service itself: -- `port`: TCP port for the WebSocket server (protocol doc defaults to `18789`) -- `mode: "local"`: local mode (vs remote-managed) -- `bind: "lan"`: binds in a way that's reachable from your LAN (treat as "network exposed") -- `controlUi.allowInsecureAuth: true`: convenience for local dev; do not use as-is for production -- `tailscale.mode: "off"`: disables Tailscale integration by default -- `reload.mode: "hot"` + `reload.debounceMs`: enables hot reload of config with a debounce window - -### `memory` - -Configures the gateway's memory subsystem. -- `backend: "qmd"`: use the QMD backend -- `citations: "auto"`: automatically include citations when supported -- `qmd.includeDefaultMemory`: includes default memory sources -- `qmd.update`: periodic update settings -- `qmd.limits`: bounds for query size/latency and injected context - -### `skills.install.nodeManager` - -Controls which Node package manager is used for skill installation. -- `"npm"` is the most compatible baseline. - -## Connecting Mission Control To The Gateway - -Mission Control connects over WebSockets and supports passing a token. - -What Mission Control expects: -- Gateway URL must be `ws://...` or `wss://...` and must include an explicit port. -- Token is optional. Empty/whitespace tokens are treated as "no token" by the Mission Control API. - -In the Mission Control UI: -1. Go to "Gateways" and add a new gateway. -2. Set URL to something like `ws://127.0.0.1:18789` (or your LAN host/IP + port). -3. If your gateway is configured to require a token, paste it here. Otherwise leave blank. -4. Use the "Check connection" action to confirm reachability. - -Implementation note (how Mission Control sends tokens): -- If you provide a token, Mission Control's backend will include it when connecting to the gateway (it attaches it to the URL query string and also sends it in the `connect` params). See `backend/app/services/openclaw/gateway_rpc.py`. - -### Workspace Root (Mission Control) vs Workspace (Gateway) - -Mission Control stores a `workspace_root` value per gateway (configured in the UI). This is used when generating agent context/templates (for example, to compute a per-agent `workspace_path`). See `backend/app/services/openclaw/provisioning.py`. - -The gateway config's `agents.defaults.workspace` is a separate setting that controls where the gateway runtime actually reads/writes agent state on disk. - -For the smoothest onboarding, set these so the paths you show agents match what exists on the gateway host. - -## Security Notes (Read This If You Expose The Gateway) - -- Treat `gateway.bind: "lan"` as "this is reachable by other devices on your network". -- `controlUi.allowInsecureAuth: true` is for convenience in local dev. If you run this beyond your laptop, tighten this setting and prefer TLS (`wss://`) and network-level protections. diff --git a/docs/openclaw_gateway_ws.md b/docs/openclaw_gateway_ws.md deleted file mode 100644 index 90cfcbec..00000000 --- a/docs/openclaw_gateway_ws.md +++ /dev/null @@ -1,1549 +0,0 @@ -# OpenClaw Gateway WebSocket Protocol - -This document describes how to interact with the OpenClaw Gateway over WebSocket. It is intended for humans and LLMs and includes frame formats, auth/scopes, events, and all known methods with params and response payloads. - -Protocol version: `3` -Default URL: `ws://127.0.0.1:18789` - -All timestamps are milliseconds since Unix epoch unless noted. - -## Connection Lifecycle - -1. Open a WebSocket connection to the gateway. -2. Server immediately sends an `event` frame named `connect.challenge` with a nonce. -3. Client must send a `req` frame with `method: "connect"` and `params: ConnectParams` as the first request. -4. Server responds with a `res` frame whose `payload` is a `HelloOk` object. - -If `connect` is not the first request, the server returns an error. - -### Connect Challenge - -Event payload: - -```ts -type ConnectChallenge = { nonce: string; ts: number }; -``` - -If you provide `device.nonce` in `connect`, it must match this `nonce`. - -## Frame Formats - -```ts -type RequestFrame = { - type: "req"; - id: string; // client-generated - method: string; - params?: unknown; -}; - -type ResponseFrame = { - type: "res"; - id: string; // matches RequestFrame.id - ok: boolean; - payload?: unknown; - error?: ErrorShape; -}; - -type EventFrame = { - type: "event"; - event: string; - payload?: unknown; - seq?: number; // optional event sequence counter - stateVersion?: { presence: number; health: number }; -}; -``` - -### Error Shape - -```ts -type ErrorShape = { - code: string; - message: string; - details?: unknown; - retryable?: boolean; - retryAfterMs?: number; -}; -``` - -Known error codes include: -`NOT_LINKED`, `NOT_PAIRED`, `AGENT_TIMEOUT`, `INVALID_REQUEST`, `UNAVAILABLE`. - -## Connect Params and HelloOk - -```ts -type ConnectParams = { - minProtocol: number; - maxProtocol: number; - client: { - id: "webchat-ui" | "openclaw-control-ui" | "webchat" | "cli" | "gateway-client" | "openclaw-macos" | "openclaw-ios" | "openclaw-android" | "node-host" | "test" | "fingerprint" | "openclaw-probe"; - displayName?: string; - version: string; - platform: string; - deviceFamily?: string; - modelIdentifier?: string; - mode: "webchat" | "cli" | "ui" | "backend" | "node" | "probe" | "test"; - instanceId?: string; - }; - caps?: string[]; - commands?: string[]; - permissions?: Record; - pathEnv?: string; - role?: string; // default "operator" - scopes?: string[]; - device?: { - id: string; - publicKey: string; - signature: string; - signedAt: number; - nonce?: string; - }; - auth?: { - token?: string; - password?: string; - }; - locale?: string; - userAgent?: string; -}; -``` - -Notes: -- `minProtocol`/`maxProtocol` must include `3` (the server's expected protocol). -- If you send `device.nonce`, it must match the `connect.challenge` nonce. - -```ts -type HelloOk = { - type: "hello-ok"; - protocol: number; - server: { - version: string; - commit?: string; - host?: string; - connId: string; - }; - features: { - methods: string[]; // advertised methods - events: string[]; // advertised events - }; - snapshot: Snapshot; - canvasHostUrl?: string; - auth?: { - deviceToken: string; - role: string; - scopes: string[]; - issuedAtMs?: number; - }; - policy: { - maxPayload: number; - maxBufferedBytes: number; - tickIntervalMs: number; - }; -}; -``` - -## Auth, Roles, and Scopes - -Gateway methods are authorized by role + scopes set during `connect`. - -Roles: -`operator` (default) and `node`. - -Scopes: -`operator.read`, `operator.write`, `operator.admin`, `operator.pairing`, `operator.approvals`. - -Notes: -- `node` role can only call `node.invoke.result`, `node.event`, `skills.bins`. -- `operator.admin` is required for config, wizard, update, and several maintenance methods. -- If a method is not explicitly read/write/pairing/approvals, it generally requires `operator.admin`. - -## Idempotency - -The following methods require `idempotencyKey` in params and dedupe repeated requests: -`send`, `poll`, `agent`, `chat.send`, `node.invoke`. - -For `send`, `poll`, `agent`, and `chat.send` the `idempotencyKey` is used as the `runId` in responses/events. - -## Common Types - -```ts -type Snapshot = { - presence: PresenceEntry[]; - health: HealthSummary; - stateVersion: { presence: number; health: number }; - uptimeMs: number; - configPath?: string; - stateDir?: string; - sessionDefaults?: { - defaultAgentId: string; - mainKey: string; - mainSessionKey: string; - scope?: string; - }; -}; - -type PresenceEntry = { - host?: string; - ip?: string; - version?: string; - platform?: string; - deviceFamily?: string; - modelIdentifier?: string; - mode?: string; - lastInputSeconds?: number; - reason?: string; - tags?: string[]; - text?: string; - ts: number; - deviceId?: string; - roles?: string[]; - scopes?: string[]; - instanceId?: string; -}; -``` - -Health summary (used by `health` method and `health` event): - -```ts -type HealthSummary = { - ok: true; - ts: number; - durationMs: number; - channels: Record; - channelOrder: string[]; - channelLabels: Record; - heartbeatSeconds: number; - defaultAgentId: string; - agents: AgentHealthSummary[]; - sessions: { - path: string; - count: number; - recent: Array<{ key: string; updatedAt: number | null; age: number | null }>; - }; -}; -``` - -Status summary (used by `status` method): - -```ts -type StatusSummary = { - linkChannel?: { id: string; label: string; linked: boolean; authAgeMs: number | null }; - heartbeat: { defaultAgentId: string; agents: HeartbeatStatus[] }; - channelSummary: string[]; - queuedSystemEvents: string[]; - sessions: { - paths: string[]; - count: number; - defaults: { model: string | null; contextTokens: number | null }; - recent: SessionStatus[]; - byAgent: Array<{ agentId: string; path: string; count: number; recent: SessionStatus[] }>; - }; -}; -``` - -Usage summaries: - -```ts -type UsageSummary = { - updatedAt: number; - providers: Array<{ - provider: string; - displayName: string; - windows: Array<{ label: string; usedPercent: number; resetAt?: number }>; - plan?: string; - error?: string; - }>; -}; - -type CostUsageSummary = { - updatedAt: number; - days: number; - daily: Array<{ date: string; input: number; output: number; cacheRead: number; cacheWrite: number; totalTokens: number; totalCost: number; missingCostEntries: number }>; - totals: { input: number; output: number; cacheRead: number; cacheWrite: number; totalTokens: number; totalCost: number; missingCostEntries: number }; -}; -``` - -Heartbeat event payload (used by `heartbeat` event and `last-heartbeat` method): - -```ts -type HeartbeatEventPayload = { - ts: number; - status: "sent" | "ok-empty" | "ok-token" | "skipped" | "failed"; - to?: string; - preview?: string; - durationMs?: number; - hasMedia?: boolean; - reason?: string; - channel?: string; - silent?: boolean; - indicatorType?: "ok" | "alert" | "error"; -}; -``` - -Chat and agent events: - -```ts -type AgentEvent = { - runId: string; - seq: number; - stream: string; - ts: number; - data: Record; -}; - -type ChatEvent = { - runId: string; - sessionKey: string; - seq: number; - state: "delta" | "final" | "aborted" | "error"; - message?: unknown; - errorMessage?: string; - usage?: unknown; - stopReason?: string; -}; -``` - -Cron types: - -```ts -type CronSchedule = - | { kind: "at"; at: string } - | { kind: "every"; everyMs: number; anchorMs?: number } - | { kind: "cron"; expr: string; tz?: string }; - -type CronPayload = - | { kind: "systemEvent"; text: string } - | { kind: "agentTurn"; message: string; model?: string; thinking?: string; timeoutSeconds?: number }; - -type CronDelivery = { mode: "none" | "announce"; channel?: "last" | string; to?: string; bestEffort?: boolean }; - -type CronJob = { - id: string; - agentId?: string; - name: string; - description?: string; - enabled: boolean; - deleteAfterRun?: boolean; - createdAtMs: number; - updatedAtMs: number; - schedule: CronSchedule; - sessionTarget: "main" | "isolated"; - wakeMode: "next-heartbeat" | "now"; - payload: CronPayload; - delivery?: CronDelivery; - state: { - nextRunAtMs?: number; - runningAtMs?: number; - lastRunAtMs?: number; - lastStatus?: "ok" | "error" | "skipped"; - lastError?: string; - lastDurationMs?: number; - }; -}; - -type CronEvent = { - jobId: string; - action: "added" | "updated" | "removed" | "started" | "finished"; - runAtMs?: number; - durationMs?: number; - status?: "ok" | "error" | "skipped"; - error?: string; - summary?: string; - nextRunAtMs?: number; -}; -``` - -Node and device pairing types: - -```ts -type NodePairingPendingRequest = { - requestId: string; - nodeId: string; - displayName?: string; - platform?: string; - version?: string; - coreVersion?: string; - uiVersion?: string; - deviceFamily?: string; - modelIdentifier?: string; - caps?: string[]; - commands?: string[]; - permissions?: Record; - remoteIp?: string; - silent?: boolean; - isRepair?: boolean; - ts: number; -}; - -type NodePairingPairedNode = { - nodeId: string; - token: string; - displayName?: string; - platform?: string; - version?: string; - coreVersion?: string; - uiVersion?: string; - deviceFamily?: string; - modelIdentifier?: string; - caps?: string[]; - commands?: string[]; - bins?: string[]; - permissions?: Record; - remoteIp?: string; - createdAtMs: number; - approvedAtMs: number; - lastConnectedAtMs?: number; -}; - -type DevicePairingPendingRequest = { - requestId: string; - deviceId: string; - publicKey: string; - displayName?: string; - platform?: string; - clientId?: string; - clientMode?: string; - role?: string; - roles?: string[]; - scopes?: string[]; - remoteIp?: string; - silent?: boolean; - isRepair?: boolean; - ts: number; -}; - -type PairedDevice = { - deviceId: string; - publicKey: string; - displayName?: string; - platform?: string; - clientId?: string; - clientMode?: string; - role?: string; - roles?: string[]; - scopes?: string[]; - remoteIp?: string; - tokens?: Record; - createdAtMs: number; - approvedAtMs: number; -}; -``` - -Sessions list/preview results: - -```ts -type SessionsListResult = { - ts: number; - path: string; - count: number; - defaults: { modelProvider: string | null; model: string | null; contextTokens: number | null }; - sessions: Array<{ - key: string; - kind: "direct" | "group" | "global" | "unknown"; - label?: string; - displayName?: string; - derivedTitle?: string; - lastMessagePreview?: string; - channel?: string; - subject?: string; - groupChannel?: string; - space?: string; - chatType?: string; - origin?: unknown; - updatedAt: number | null; - sessionId?: string; - systemSent?: boolean; - abortedLastRun?: boolean; - thinkingLevel?: string; - verboseLevel?: string; - reasoningLevel?: string; - elevatedLevel?: string; - sendPolicy?: "allow" | "deny"; - inputTokens?: number; - outputTokens?: number; - totalTokens?: number; - responseUsage?: "on" | "off" | "tokens" | "full"; - modelProvider?: string; - model?: string; - contextTokens?: number; - deliveryContext?: unknown; - lastChannel?: string; - lastTo?: string; - lastAccountId?: string; - }>; -}; - -type SessionsPreviewResult = { - ts: number; - previews: Array<{ - key: string; - status: "ok" | "empty" | "missing" | "error"; - items: Array<{ role: "user" | "assistant" | "tool" | "system" | "other"; text: string }>; - }>; -}; -``` - -## Events - -The gateway may emit these events. Payloads use the types above. - -- `connect.challenge`: `ConnectChallenge` -- `agent`: `AgentEvent` -- `chat`: `ChatEvent` -- `presence`: `{ presence: PresenceEntry[] }` -- `tick`: `{ ts: number }` -- `talk.mode`: `{ enabled: boolean; phase: string | null; ts: number }` -- `shutdown`: `{ reason: string; restartExpectedMs?: number }` -- `health`: `HealthSummary` -- `heartbeat`: `HeartbeatEventPayload` -- `cron`: `CronEvent` -- `node.pair.requested`: `NodePairingPendingRequest` -- `node.pair.resolved`: `{ requestId: string; nodeId: string; decision: "approved" | "rejected"; ts: number }` -- `node.invoke.request`: `{ id: string; nodeId: string; command: string; paramsJSON?: string; timeoutMs?: number; idempotencyKey?: string }` -- `device.pair.requested`: `DevicePairingPendingRequest` -- `device.pair.resolved`: `{ requestId: string; deviceId: string; decision: string; ts: number }` -- `voicewake.changed`: `{ triggers: string[] }` -- `exec.approval.requested`: `{ id: string; request: { command: string; cwd?: string | null; host?: string | null; security?: string | null; ask?: string | null; agentId?: string | null; resolvedPath?: string | null; sessionKey?: string | null }; createdAtMs: number; expiresAtMs: number }` -- `exec.approval.resolved`: `{ id: string; decision: "allow-once" | "allow-always" | "deny"; resolvedBy?: string; ts: number }` - -## Method Reference - -### Health and Status - -#### `health` -Params: -```ts -{ probe?: boolean } -``` -Response: -`HealthSummary` -Notes: Uses cached health unless `probe: true`. - -#### `status` -Params: -```ts -{} -``` -Response: -`StatusSummary` - -#### `usage.status` -Params: -```ts -{} -``` -Response: -`UsageSummary` - -#### `usage.cost` -Params: -```ts -{ days?: number } -``` -Response: -`CostUsageSummary` - -#### `last-heartbeat` -Params: -```ts -{} -``` -Response: -`HeartbeatEventPayload | null` - -#### `set-heartbeats` -Params: -```ts -{ enabled: boolean } -``` -Response: -```ts -{ ok: true; enabled: boolean } -``` -Scope: `operator.admin` - -#### `system-presence` -Params: -```ts -{} -``` -Response: -`PresenceEntry[]` - -#### `system-event` -Params: -```ts -{ - text: string; - deviceId?: string; - instanceId?: string; - host?: string; - ip?: string; - mode?: string; - version?: string; - platform?: string; - deviceFamily?: string; - modelIdentifier?: string; - lastInputSeconds?: number; - reason?: string; - roles?: string[]; - scopes?: string[]; - tags?: string[]; -} -``` -Response: -```ts -{ ok: true } -``` -Scope: `operator.admin` - -### Logs - -#### `logs.tail` -Params: -```ts -{ cursor?: number; limit?: number; maxBytes?: number } -``` -Response: -```ts -{ file: string; cursor: number; size: number; lines: string[]; truncated?: boolean; reset?: boolean } -``` - -### Channels - -#### `channels.status` -Params: -```ts -{ probe?: boolean; timeoutMs?: number } -``` -Response: -```ts -{ - ts: number; - channelOrder: string[]; - channelLabels: Record; - channelDetailLabels?: Record; - channelSystemImages?: Record; - channelMeta?: Array<{ id: string; label: string; detailLabel: string; systemImage?: string }>; - channels: Record; // plugin summaries - channelAccounts: Record>; - channelDefaultAccountId: Record; -} -``` - -#### `channels.logout` -Params: -```ts -{ channel: string; accountId?: string } -``` -Response: -```ts -{ channel: string; accountId: string; cleared: boolean; [key: string]: unknown } -``` -Scope: `operator.admin` - -#### `web.login.start` (plugin-provided) -Params: -```ts -{ force?: boolean; timeoutMs?: number; verbose?: boolean; accountId?: string } -``` -Response: provider-specific, typically includes QR or login URL. - -#### `web.login.wait` (plugin-provided) -Params: -```ts -{ timeoutMs?: number; accountId?: string } -``` -Response: provider-specific, typically `{ connected: boolean; ... }`. - -### TTS - -#### `tts.status` -Params: -```ts -{} -``` -Response: -```ts -{ - enabled: boolean; - auto: boolean | string; - provider: "openai" | "elevenlabs" | "edge"; - fallbackProvider: string | null; - fallbackProviders: string[]; - prefsPath: string; - hasOpenAIKey: boolean; - hasElevenLabsKey: boolean; - edgeEnabled: boolean; -} -``` - -#### `tts.providers` -Params: -```ts -{} -``` -Response: -```ts -{ providers: Array<{ id: string; name: string; configured: boolean; models: string[]; voices?: string[] }>; active: string } -``` - -#### `tts.enable` -Params: -```ts -{} -``` -Response: -```ts -{ enabled: true } -``` - -#### `tts.disable` -Params: -```ts -{} -``` -Response: -```ts -{ enabled: false } -``` - -#### `tts.convert` -Params: -```ts -{ text: string; channel?: string } -``` -Response: -```ts -{ audioPath: string; provider: string; outputFormat: string; voiceCompatible: boolean } -``` - -#### `tts.setProvider` -Params: -```ts -{ provider: "openai" | "elevenlabs" | "edge" } -``` -Response: -```ts -{ provider: string } -``` - -### Config and Update - -#### `config.get` -Params: -```ts -{} -``` -Response: -```ts -{ path: string; exists: boolean; raw: string | null; parsed: unknown; valid: boolean; config: unknown; hash?: string; issues: Array<{ path: string; message: string }>; warnings: Array<{ path: string; message: string }>; legacyIssues: Array<{ path: string; message: string }> } -``` - -#### `config.schema` -Params: -```ts -{} -``` -Response: -```ts -{ schema: unknown; uiHints: Record; version: string; generatedAt: string } -``` - -#### `config.set` -Params: -```ts -{ raw: string; baseHash?: string } -``` -Response: -```ts -{ ok: true; path: string; config: unknown } -``` -Notes: `baseHash` is required if a config already exists. - -#### `config.patch` -Params: -```ts -{ raw: string; baseHash?: string; sessionKey?: string; note?: string; restartDelayMs?: number } -``` -Response: -```ts -{ ok: true; path: string; config: unknown; restart: unknown; sentinel: { path: string | null; payload: unknown } } -``` -Notes: `raw` must be a JSON object for merge patch. Requires `baseHash` if config exists. - -#### `config.apply` -Params: -```ts -{ raw: string; baseHash?: string; sessionKey?: string; note?: string; restartDelayMs?: number } -``` -Response: -```ts -{ ok: true; path: string; config: unknown; restart: unknown; sentinel: { path: string | null; payload: unknown } } -``` -Notes: Requires `baseHash` if config exists. - -#### `update.run` -Params: -```ts -{ sessionKey?: string; note?: string; restartDelayMs?: number; timeoutMs?: number } -``` -Response: -```ts -{ ok: true; result: { status: "ok" | "error"; mode: string; reason?: string; root?: string; before?: string | null; after?: string | null; steps: Array<{ name: string; command: string; cwd: string; durationMs: number; stdoutTail?: string | null; stderrTail?: string | null; exitCode?: number | null }>; durationMs: number }; restart: unknown; sentinel: { path: string | null; payload: unknown } } -``` -Scope: `operator.admin` - -### Exec Approvals - -#### `exec.approvals.get` -Params: -```ts -{} -``` -Response: -```ts -{ path: string; exists: boolean; hash: string; file: { version: 1; socket?: { path?: string }; defaults?: { security?: string; ask?: string; askFallback?: string; autoAllowSkills?: boolean }; agents?: Record }> } } -``` - -#### `exec.approvals.set` -Params: -```ts -{ file: ExecApprovalsFile; baseHash?: string } -``` -Response: -Same shape as `exec.approvals.get`. -Notes: `baseHash` required if file exists. - -#### `exec.approvals.node.get` -Params: -```ts -{ nodeId: string } -``` -Response: -Node-provided exec approvals snapshot for that node. - -#### `exec.approvals.node.set` -Params: -```ts -{ nodeId: string; file: ExecApprovalsFile; baseHash?: string } -``` -Response: -Node-provided exec approvals snapshot after update. - -#### `exec.approval.request` -Params: -```ts -{ id?: string; command: string; cwd?: string | null; host?: string | null; security?: string | null; ask?: string | null; agentId?: string | null; resolvedPath?: string | null; sessionKey?: string | null; timeoutMs?: number } -``` -Response: -```ts -{ id: string; decision: "allow-once" | "allow-always" | "deny"; createdAtMs: number; expiresAtMs: number } -``` -Notes: This method blocks until a decision is made or timeout occurs. - -#### `exec.approval.resolve` -Params: -```ts -{ id: string; decision: "allow-once" | "allow-always" | "deny" } -``` -Response: -```ts -{ ok: true } -``` - -### Wizard - -#### `wizard.start` -Params: -```ts -{ mode?: "local" | "remote"; workspace?: string } -``` -Response: -```ts -{ sessionId: string; done: boolean; step?: WizardStep; status?: "running" | "done" | "cancelled" | "error"; error?: string } -``` - -#### `wizard.next` -Params: -```ts -{ sessionId: string; answer?: { stepId: string; value?: unknown } } -``` -Response: -```ts -{ done: boolean; step?: WizardStep; status?: "running" | "done" | "cancelled" | "error"; error?: string } -``` - -#### `wizard.cancel` -Params: -```ts -{ sessionId: string } -``` -Response: -```ts -{ status: "running" | "done" | "cancelled" | "error"; error?: string } -``` - -#### `wizard.status` -Params: -```ts -{ sessionId: string } -``` -Response: -```ts -{ status: "running" | "done" | "cancelled" | "error"; error?: string } -``` - -WizardStep: -```ts -{ id: string; type: "note" | "select" | "text" | "confirm" | "multiselect" | "progress" | "action"; title?: string; message?: string; options?: Array<{ value: unknown; label: string; hint?: string }>; initialValue?: unknown; placeholder?: string; sensitive?: boolean; executor?: "gateway" | "client" } -``` - -### Talk - -#### `talk.mode` -Params: -```ts -{ enabled: boolean; phase?: string } -``` -Response: -```ts -{ enabled: boolean; phase: string | null; ts: number } -``` -Notes: For webchat clients, requires a connected mobile node. - -### Models - -#### `models.list` -Params: -```ts -{} -``` -Response: -```ts -{ models: Array<{ id: string; name: string; provider: string; contextWindow?: number; reasoning?: boolean }> } -``` - -### Agents - -#### `agents.list` -Params: -```ts -{} -``` -Response: -```ts -{ defaultId: string; mainKey: string; scope: "per-sender" | "global"; agents: Array<{ id: string; name?: string; identity?: { name?: string; theme?: string; emoji?: string; avatar?: string; avatarUrl?: string } }> } -``` - -#### `agents.files.list` -Params: -```ts -{ agentId: string } -``` -Response: -```ts -{ agentId: string; workspace: string; files: Array<{ name: string; path: string; missing: boolean; size?: number; updatedAtMs?: number; content?: string }> } -``` - -#### `agents.files.get` -Params: -```ts -{ agentId: string; name: string } -``` -Response: -```ts -{ agentId: string; workspace: string; file: { name: string; path: string; missing: boolean; size?: number; updatedAtMs?: number; content?: string } } -``` - -#### `agents.files.set` -Params: -```ts -{ agentId: string; name: string; content: string } -``` -Response: -```ts -{ ok: true; agentId: string; workspace: string; file: { name: string; path: string; missing: boolean; size?: number; updatedAtMs?: number; content?: string } } -``` - -#### `agent` -Params: -```ts -{ - message: string; - agentId?: string; - to?: string; - replyTo?: string; - sessionId?: string; - sessionKey?: string; - thinking?: string; - deliver?: boolean; - attachments?: Array<{ type?: string; mimeType?: string; fileName?: string; content?: unknown }>; - channel?: string; - replyChannel?: string; - accountId?: string; - replyAccountId?: string; - threadId?: string; - groupId?: string; - groupChannel?: string; - groupSpace?: string; - timeout?: number; - lane?: string; - extraSystemPrompt?: string; - idempotencyKey: string; - label?: string; - spawnedBy?: string; -} -``` -Response: -```ts -{ runId: string; status: "accepted"; acceptedAt: number } -``` -Final response (same request id, later): -```ts -{ runId: string; status: "ok" | "error"; summary: string; result?: unknown } -``` -Notes: The gateway may send multiple `res` frames for the same `id`. - -#### `agent.identity.get` -Params: -```ts -{ agentId?: string; sessionKey?: string } -``` -Response: -```ts -{ agentId: string; name?: string; avatar?: string; emoji?: string } -``` - -#### `agent.wait` -Params: -```ts -{ runId: string; timeoutMs?: number } -``` -Response: -```ts -{ runId: string; status: "ok" | "error" | "timeout"; startedAt?: number; endedAt?: number; error?: string } -``` - -### Skills - -#### `skills.status` -Params: -```ts -{ agentId?: string } -``` -Response: -```ts -{ workspaceDir: string; managedSkillsDir: string; skills: Array<{ name: string; description?: string; source?: string; bundled?: boolean; filePath?: string; baseDir?: string; skillKey?: string; emoji?: string; homepage?: string; always?: boolean; disabled?: boolean; blockedByAllowlist?: boolean; eligible?: boolean; requirements?: { bins?: string[]; anyBins?: string[]; env?: string[]; config?: string[]; os?: string[] }; missing?: { bins?: string[]; anyBins?: string[]; env?: string[]; config?: string[]; os?: string[] }; configChecks?: unknown[]; install?: unknown[] }> } -``` - -#### `skills.bins` -Params: -```ts -{} -``` -Response: -```ts -{ bins: string[] } -``` - -#### `skills.install` -Params: -```ts -{ name: string; installId: string; timeoutMs?: number } -``` -Response: -```ts -{ ok: boolean; message?: string; [key: string]: unknown } -``` -Notes: Install details are returned from the skill installer. - -#### `skills.update` -Params: -```ts -{ skillKey: string; enabled?: boolean; apiKey?: string; env?: Record } -``` -Response: -```ts -{ ok: true; skillKey: string; config: { enabled?: boolean; apiKey?: string; env?: Record } } -``` - -### Voice Wake - -#### `voicewake.get` -Params: -```ts -{} -``` -Response: -```ts -{ triggers: string[] } -``` - -#### `voicewake.set` -Params: -```ts -{ triggers: string[] } -``` -Response: -```ts -{ triggers: string[] } -``` -Also emits `voicewake.changed` event. - -### Sessions - -#### `sessions.list` -Params: -```ts -{ limit?: number; activeMinutes?: number; includeGlobal?: boolean; includeUnknown?: boolean; includeDerivedTitles?: boolean; includeLastMessage?: boolean; label?: string; spawnedBy?: string; agentId?: string; search?: string } -``` -Response: -`SessionsListResult` - -#### `sessions.preview` -Params: -```ts -{ keys: string[]; limit?: number; maxChars?: number } -``` -Response: -`SessionsPreviewResult` - -#### `sessions.resolve` -Params: -```ts -{ key?: string; sessionId?: string; label?: string; agentId?: string; spawnedBy?: string; includeGlobal?: boolean; includeUnknown?: boolean } -``` -Response: -```ts -{ ok: true; key: string } -``` - -#### `sessions.patch` -Params: -```ts -{ key: string; label?: string | null; thinkingLevel?: string | null; verboseLevel?: string | null; reasoningLevel?: string | null; responseUsage?: "off" | "tokens" | "full" | "on" | null; elevatedLevel?: string | null; execHost?: string | null; execSecurity?: string | null; execAsk?: string | null; execNode?: string | null; model?: string | null; spawnedBy?: string | null; sendPolicy?: "allow" | "deny" | null; groupActivation?: "mention" | "always" | null } -``` -Response: -```ts -{ ok: true; path: string; key: string; entry: unknown } -``` -Scope: `operator.admin` - -#### `sessions.reset` -Params: -```ts -{ key: string } -``` -Response: -```ts -{ ok: true; key: string; entry: unknown } -``` -Scope: `operator.admin` - -#### `sessions.delete` -Params: -```ts -{ key: string; deleteTranscript?: boolean } -``` -Response: -```ts -{ ok: true; key: string; deleted: boolean; archived: string[] } -``` -Scope: `operator.admin` - -#### `sessions.compact` -Params: -```ts -{ key: string; maxLines?: number } -``` -Response: -```ts -{ ok: true; key: string; compacted: boolean; reason?: string; archived?: string; kept?: number } -``` -Scope: `operator.admin` - -### Nodes - -#### `node.pair.request` -Params: -```ts -{ nodeId: string; displayName?: string; platform?: string; version?: string; coreVersion?: string; uiVersion?: string; deviceFamily?: string; modelIdentifier?: string; caps?: string[]; commands?: string[]; remoteIp?: string; silent?: boolean } -``` -Response: -```ts -{ status: "pending"; request: NodePairingPendingRequest; created: boolean } -``` -Scope: `operator.pairing` - -#### `node.pair.list` -Params: -```ts -{} -``` -Response: -```ts -{ pending: NodePairingPendingRequest[]; paired: NodePairingPairedNode[] } -``` -Scope: `operator.pairing` - -#### `node.pair.approve` -Params: -```ts -{ requestId: string } -``` -Response: -```ts -{ requestId: string; node: NodePairingPairedNode } -``` -Scope: `operator.pairing` - -#### `node.pair.reject` -Params: -```ts -{ requestId: string } -``` -Response: -```ts -{ requestId: string; nodeId: string } -``` -Scope: `operator.pairing` - -#### `node.pair.verify` -Params: -```ts -{ nodeId: string; token: string } -``` -Response: -```ts -{ ok: boolean; node?: NodePairingPairedNode } -``` -Scope: `operator.pairing` - -#### `node.rename` -Params: -```ts -{ nodeId: string; displayName: string } -``` -Response: -```ts -{ nodeId: string; displayName: string } -``` -Scope: `operator.pairing` - -#### `node.list` -Params: -```ts -{} -``` -Response: -```ts -{ ts: number; nodes: Array<{ nodeId: string; displayName?: string; platform?: string; version?: string; coreVersion?: string; uiVersion?: string; deviceFamily?: string; modelIdentifier?: string; remoteIp?: string; caps: string[]; commands: string[]; pathEnv?: string; permissions?: Record; connectedAtMs?: number; paired: boolean; connected: boolean }> } -``` - -#### `node.describe` -Params: -```ts -{ nodeId: string } -``` -Response: -```ts -{ ts: number; nodeId: string; displayName?: string; platform?: string; version?: string; coreVersion?: string; uiVersion?: string; deviceFamily?: string; modelIdentifier?: string; remoteIp?: string; caps: string[]; commands: string[]; pathEnv?: string; permissions?: Record; connectedAtMs?: number; paired: boolean; connected: boolean } -``` - -#### `node.invoke` -Params: -```ts -{ nodeId: string; command: string; params?: unknown; timeoutMs?: number; idempotencyKey: string } -``` -Response: -```ts -{ ok: true; nodeId: string; command: string; payload: unknown; payloadJSON?: string | null } -``` -Notes: Requires the node to be connected and command allowed by policy/allowlist. - -#### `node.invoke.result` -Params: -```ts -{ id: string; nodeId: string; ok: boolean; payload?: unknown; payloadJSON?: string; error?: { code?: string; message?: string } } -``` -Response: -```ts -{ ok: true } | { ok: true; ignored: true } -``` -Scope: `node` role only. - -#### `node.event` -Params: -```ts -{ event: string; payload?: unknown; payloadJSON?: string } -``` -Response: -```ts -{ ok: true } -``` -Scope: `node` role only. - -### Devices - -#### `device.pair.list` -Params: -```ts -{} -``` -Response: -```ts -{ pending: DevicePairingPendingRequest[]; paired: Array }> } -``` -Scope: `operator.pairing` - -#### `device.pair.approve` -Params: -```ts -{ requestId: string } -``` -Response: -```ts -{ requestId: string; device: PairedDevice } -``` -Scope: `operator.pairing` - -#### `device.pair.reject` -Params: -```ts -{ requestId: string } -``` -Response: -```ts -{ requestId: string; deviceId: string } -``` -Scope: `operator.pairing` - -#### `device.token.rotate` -Params: -```ts -{ deviceId: string; role: string; scopes?: string[] } -``` -Response: -```ts -{ deviceId: string; role: string; token: string; scopes: string[]; rotatedAtMs: number } -``` -Scope: `operator.pairing` - -#### `device.token.revoke` -Params: -```ts -{ deviceId: string; role: string } -``` -Response: -```ts -{ deviceId: string; role: string; revokedAtMs: number } -``` -Scope: `operator.pairing` - -### Cron and Wake - -#### `wake` -Params: -```ts -{ mode: "now" | "next-heartbeat"; text: string } -``` -Response: -```ts -{ ok: boolean } -``` - -#### `cron.list` -Params: -```ts -{ includeDisabled?: boolean } -``` -Response: -```ts -{ jobs: CronJob[] } -``` - -#### `cron.status` -Params: -```ts -{} -``` -Response: -```ts -{ enabled: boolean; storePath: string; jobs: number; nextWakeAtMs: number | null } -``` - -#### `cron.add` -Params: -```ts -{ name: string; agentId?: string | null; description?: string; enabled?: boolean; deleteAfterRun?: boolean; schedule: CronSchedule; sessionTarget: "main" | "isolated"; wakeMode: "next-heartbeat" | "now"; payload: CronPayload; delivery?: CronDelivery } -``` -Response: -`CronJob` -Scope: `operator.admin` - -#### `cron.update` -Params: -```ts -{ id?: string; jobId?: string; patch: Partial } -``` -Response: -`CronJob` -Scope: `operator.admin` - -#### `cron.remove` -Params: -```ts -{ id?: string; jobId?: string } -``` -Response: -```ts -{ ok: true; removed: boolean } -``` -Scope: `operator.admin` - -#### `cron.run` -Params: -```ts -{ id?: string; jobId?: string; mode?: "due" | "force" } -``` -Response: -```ts -{ ok: true; ran: true } | { ok: true; ran: false; reason: "not-due" } -``` -Scope: `operator.admin` - -#### `cron.runs` -Params: -```ts -{ id?: string; jobId?: string; limit?: number } -``` -Response: -```ts -{ entries: Array<{ ts: number; jobId: string; action: "finished"; status?: "ok" | "error" | "skipped"; error?: string; summary?: string; runAtMs?: number; durationMs?: number; nextRunAtMs?: number }> } -``` - -### Messaging - -#### `send` -Params: -```ts -{ to: string; message: string; mediaUrl?: string; mediaUrls?: string[]; gifPlayback?: boolean; channel?: string; accountId?: string; sessionKey?: string; idempotencyKey: string } -``` -Response: -```ts -{ runId: string; messageId: string; channel: string; chatId?: string; channelId?: string; toJid?: string; conversationId?: string } -``` -Notes: `runId` is `idempotencyKey`. - -#### `poll` (undocumented but implemented) -Params: -```ts -{ to: string; question: string; options: string[]; maxSelections?: number; durationHours?: number; channel?: string; accountId?: string; idempotencyKey: string } -``` -Response: -```ts -{ runId: string; messageId: string; channel: string; toJid?: string; channelId?: string; conversationId?: string; pollId?: string } -``` -Notes: `poll` is not advertised in `features.methods` but is implemented. - -### Chat (WebSocket-native) - -#### `chat.history` -Params: -```ts -{ sessionKey: string; limit?: number } -``` -Response: -```ts -{ sessionKey: string; sessionId?: string; messages: unknown[]; thinkingLevel?: string } -``` - -#### `chat.send` -Params: -```ts -{ sessionKey: string; message: string; thinking?: string; deliver?: boolean; attachments?: unknown[]; timeoutMs?: number; idempotencyKey: string } -``` -Response (immediate ack): -```ts -{ runId: string; status: "started" } -``` -Possible cached response: -```ts -{ runId: string; status: "in_flight" } -``` -Final state is delivered via `chat` events. - -#### `chat.abort` -Params: -```ts -{ sessionKey: string; runId?: string } -``` -Response: -```ts -{ ok: true; aborted: boolean; runIds: string[] } -``` - -#### `chat.inject` (undocumented but implemented) -Params: -```ts -{ sessionKey: string; message: string; label?: string } -``` -Response: -```ts -{ ok: true; messageId: string } -``` -Notes: `chat.inject` is not advertised in `features.methods` but is implemented. - -### Browser - -#### `browser.request` -Params: -```ts -{ method: "GET" | "POST" | "DELETE"; path: string; query?: Record; body?: unknown; timeoutMs?: number } -``` -Response: -`unknown` (proxy result body) -Notes: If a connected browser-capable node is available, requests are proxied through it. Otherwise, the local browser control service is used if enabled. - -### Misc - -#### `system-presence`, `system-event`, `last-heartbeat`, `set-heartbeats` -See Health and Status section. - -## Hidden or Not Advertised in `features.methods` - -The following methods are implemented but not in the `features.methods` list returned by `HelloOk`: -- `poll` -- `chat.inject` - -Clients should primarily rely on `features.methods` to discover capabilities, but these methods exist in the current implementation. diff --git a/docs/production/README.md b/docs/production/README.md deleted file mode 100644 index 98f1db4b..00000000 --- a/docs/production/README.md +++ /dev/null @@ -1,209 +0,0 @@ -# Production deployment (production-ish) - -This document describes **production-ish** deployment patterns for **OpenClaw Mission Control**. - -Mission Control is a web app (frontend) + API (backend) + Postgres. The simplest reliable -baseline is Docker Compose plus a reverse proxy with TLS. - -> This repo currently ships a developer-friendly `compose.yml`. For real production, you should: -> - put Postgres on a managed service or dedicated host when possible -> - terminate TLS at a reverse proxy -> - set up backups + upgrades -> - restrict network exposure (firewall) - -## Recommended baseline - -If you’re looking for the **dev-friendly self-host** path (single machine, Docker Compose defaults), start with the repo root README: -- [Quick start (self-host with Docker Compose)](../../README.md#quick-start-self-host-with-docker-compose) - - -- Docker Engine + Docker Compose v2 -- Reverse proxy: **Caddy** (simplest) or **nginx** -- TLS via Let’s Encrypt -- Persistent storage for Postgres -- Centralized logs (or at least log rotation) - -## Single VPS (all-in-one) - -### Architecture - -On one VM: - -- Caddy/nginx (ports 80/443) → routes traffic to: - - frontend container (internal port 3000) - - backend container (internal port 8000) -- Postgres container (internal 5432) - -### Ports / firewall - -Expose to the internet: - -- `80/tcp` and `443/tcp` only - -Do **not** expose: - -- Postgres 5432 -- backend 8000 -- frontend 3000 - -All of those should be reachable only on the docker network / localhost. - -### Environment & secrets - -Recommended approach: - -- Keep a host-level directory (e.g. `/opt/mission-control/`) -- Store runtime env in **non-committed** files: - - `/opt/mission-control/.env` (compose-level vars) - - optionally `/opt/mission-control/backend.env` and `/opt/mission-control/frontend.env` - -Secrets guidelines: - -- Choose auth mode explicitly: - - `AUTH_MODE=local`: set `LOCAL_AUTH_TOKEN` to a random value with at least 50 characters - - `AUTH_MODE=clerk`: configure Clerk keys -- Never commit `LOCAL_AUTH_TOKEN` or Clerk secret key. -- Prefer passing secrets as environment variables from the host (or use Docker secrets if you later - migrate to Swarm/K8s). -- Rotate secrets if they ever hit logs. - -### Compose in production - -Clone the repo on the VPS, then: - -```bash -cd /opt -sudo git clone https://github.com/abhi1693/openclaw-mission-control.git mission-control -cd mission-control - -cp .env.example .env -# edit .env with real values (domains, auth mode + secrets, etc.) - -docker compose -f compose.yml --env-file .env up -d --build -``` - -### Reverse proxy (Caddy example) - -Example `Caddyfile` (adjust domain): - -```caddyfile -mission-control.example.com { - encode gzip - - # Frontend - reverse_proxy /* localhost:3000 - - # (Optional) If you want to route API separately, use a path prefix: - # reverse_proxy /api/* localhost:8000 -} -``` - -Notes: -- If the frontend calls the backend directly, ensure `NEXT_PUBLIC_API_URL` points to the **public, browser-reachable** API - URL, not `localhost`. - - Example: `NEXT_PUBLIC_API_URL=https://api.mission-control.example.com` -- If you route the backend under a path prefix, ensure backend routing supports it (or put it on a - subdomain like `api.mission-control.example.com`). - -### Keep services running (systemd) - -Docker restart policies are often enough, but for predictable boot/shutdown and easy ops, use -systemd. - -Create `/etc/systemd/system/mission-control.service`: - -```ini -[Unit] -Description=Mission Control (docker compose) -Requires=docker.service -After=docker.service - -[Service] -Type=oneshot -RemainAfterExit=yes -WorkingDirectory=/opt/mission-control - -ExecStart=/usr/bin/docker compose -f compose.yml --env-file .env up -d -ExecStop=/usr/bin/docker compose -f compose.yml --env-file .env down - -TimeoutStartSec=0 - -[Install] -WantedBy=multi-user.target -``` - -Enable: - -```bash -sudo systemctl daemon-reload -sudo systemctl enable --now mission-control -sudo systemctl status mission-control -``` - -### Backups - -Minimum viable: - -- Nightly `pg_dump` to off-host storage -- Or filesystem-level backup of the Postgres volume (requires consistent snapshots) - -Example dump: - -```bash -docker exec -t openclaw-mission-control-db-1 pg_dump -U postgres mission_control > /opt/backups/mission_control.sql -``` - -## Multi-VPS (split services) - -The main reason to split is reliability and blast-radius reduction. - -### Option A: 2 hosts - -- Host 1: reverse proxy + frontend + backend -- Host 2: Postgres (or managed) - -### Option B: 3 hosts - -- Host 1: reverse proxy + frontend -- Host 2: backend -- Host 3: Postgres (or managed) - -### Networking / security groups - -Minimum rules: - -- Public internet → reverse proxy host: `80/443` -- Reverse proxy host → backend host: `8000` (or whatever you publish internally) -- Backend host → DB host: `5432` - -Everything else: deny. - -### Configuration considerations - -- `DATABASE_URL` must point to the DB host (not `localhost`). -- `CORS_ORIGINS` must include the public frontend URL. -- `NEXT_PUBLIC_API_URL` should be the public API base URL. - -### Database migrations - -The backend currently runs Alembic migrations on startup (see logs). In multi-host setups: - -- Decide if migrations should run automatically (one backend instance) or via a manual deploy step. -- Avoid multiple concurrent backend deploys racing on migrations. - -## Operational checklist - -- [ ] TLS is enabled, HTTP redirects to HTTPS -- [ ] Only 80/443 exposed publicly -- [ ] Postgres not publicly accessible -- [ ] Backups tested (restore drill) -- [ ] Log retention/rotation configured -- [ ] Regular upgrade process (pull latest, rebuild, restart) - -## Troubleshooting (production) - -- `docker compose ps` and `docker compose logs --tail=200` are your first stops. -- If the UI loads but API calls fail, check: - - `NEXT_PUBLIC_API_URL` - - backend CORS settings (`CORS_ORIGINS`) - - firewall rules between proxy ↔ backend diff --git a/docs/testing/README.md b/docs/testing/README.md deleted file mode 100644 index a788126f..00000000 --- a/docs/testing/README.md +++ /dev/null @@ -1,86 +0,0 @@ -# Testing - -This repo uses a mix of unit tests and Cypress end-to-end (E2E) tests. - -## Cypress E2E: conventions (stories) - -Write E2E tests as **user stories** describing what a user does (and does not do): - -- Prefer descriptive spec names like: - - `As a signed-in user, I can view my activity feed (happy path)` - - `As a signed-out user, I get redirected to sign-in (negative path)` - - `As a user, invalid API URL shows an error state (negative path)` -- Include **both**: - - **Positive/happy path** (expected successful flow) - - **Negative paths** (missing inputs, unauthenticated access, invalid states) - -Keep each spec focused on one story/flow; avoid long “mega specs”. - -## Cypress E2E: Clerk auth (official implementation) - -Hard requirements: -- **No auth bypass** in E2E. -- Use Clerk’s **official Cypress support** via `@clerk/testing`. - -Implementation in this repo: -- `frontend/cypress.config.ts` calls `clerkSetup()`. -- `frontend/cypress/support/e2e.ts` imports and registers commands: - - `addClerkCommands({ Cypress, cy })` -- Tests can use: - - `cy.clerkLoaded()` - - `cy.clerkSignIn(...)` / `cy.clerkSignOut(...)` - -See also: [E2E auth notes](../e2e-auth.md). - -### Test user (non-secret) - -- Email: `jane+clerk_test@example.com` -- OTP: `424242` - -## Required environment variables - -### Local E2E (running Cypress yourself) - -You typically need: - -- `NEXT_PUBLIC_API_URL` (required) - - Must be reachable from the **browser** (host), not just from inside Docker. - - Examples: - - Local backend: `http://localhost:8000` - - CI E2E job (frontend dev server): `http://localhost:3000` (see workflow) - -- Clerk env (values should come from your Clerk app; **do not commit secrets**): - - `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` (required for the app) - - `CLERK_SECRET_KEY` (required for Clerk testing tokens) - -- Cypress Clerk test user identifier (non-secret, repo default is OK): - - `CYPRESS_CLERK_TEST_EMAIL` (defaults to `jane+clerk_test@example.com` in CI) - -Note: Cypress automatically maps `CYPRESS_FOO=bar` into `Cypress.env('FOO')`. - -### CI artifacts on E2E failures (required) - -For E2E failures, always upload Cypress artifacts so failures are debuggable from CI: - -- `frontend/cypress/screenshots/**` -- `frontend/cypress/videos/**` - -(Our GitHub Actions workflow already uploads these as an artifact for every E2E run.) - -## Running Cypress locally - -From repo root: - -```bash -make frontend-sync - -# in one terminal -cd frontend -npm run dev -- --hostname 0.0.0.0 --port 3000 - -# in another terminal -cd frontend -npm run e2e -- --browser chrome -``` - -If you hit Clerk-related bot detection or sign-in failures, re-check the Clerk testing env vars above. diff --git a/docs/troubleshooting/README.md b/docs/troubleshooting/README.md deleted file mode 100644 index 49baac8a..00000000 --- a/docs/troubleshooting/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Troubleshooting - -## Activity feed is blank / frontend API calls fail - -**Symptoms** -- Activity feed shows no items. -- The browser console/network tab shows failed requests to `/api/v1/*`. - -**Cause** -- `NEXT_PUBLIC_API_URL` is missing/blank/incorrect. The frontend uses this variable to build API URLs. - -**Fix** -- Local dev: set `NEXT_PUBLIC_API_URL=http://localhost:8000` in `frontend/.env.local`. -- Docker Compose (self-host): set `NEXT_PUBLIC_API_URL=http://localhost:8000` in the root `.env` used by compose (or update it to match your actual backend host/port). - -Notes: -- `NEXT_PUBLIC_API_URL` must be reachable from the browser. If you're using Docker Compose, don't set it to an internal service name like `http://backend:8000` unless the browser can resolve it.