Merge pull request #81 from abhi1693/docs/frontend-api-auth-modules

docs: backend API + core module guides
2026-02-11 18:54:49 +05:30
parent c6417bcffb a2a4159a51
commit 0ff645f795
4 changed files with 296 additions and 1 deletions
--- a/docs/07-api-reference.md
+++ b/docs/07-api-reference.md
@@ -24,7 +24,7 @@ It is derived from `backend/app/main.py` (router registration) and `backend/app/
 | `agents.py` | `/agents` | Thin API wrappers for async agent lifecycle operations. |
 | `approvals.py` | `/boards/{board_id}/approvals` | Approval listing, streaming, creation, and update endpoints. |
 | `auth.py` | `/auth` | Authentication bootstrap endpoints for the Mission Control API. |
-| `board_group_memory.py` | `` | Board-group memory CRUD and streaming endpoints. |
+| `board_group_memory.py` | `/board-groups/{group_id}/memory` and `/boards/{board_id}/group-memory` | Board-group memory CRUD and streaming endpoints. |
 | `board_groups.py` | `/board-groups` | Board group CRUD, snapshot, and heartbeat endpoints. |
 | `board_memory.py` | `/boards/{board_id}/memory` | Board memory CRUD and streaming endpoints. |
 | `board_onboarding.py` | `/boards/{board_id}/onboarding` | Board onboarding endpoints for user/agent collaboration. |
@@ -37,6 +37,54 @@ It is derived from `backend/app/main.py` (router registration) and `backend/app/
 | `tasks.py` | `/boards/{board_id}/tasks` | Task API routes for listing, streaming, and mutating board tasks. |
 | `users.py` | `/users` | User self-service API endpoints for profile retrieval and updates. |

+## Backend API layer notes (how modules are organized)
+
+Evidence: `backend/app/main.py`, `backend/app/api/*`, `backend/app/api/deps.py`.
+
+### Conventions
+
+- Each file under `backend/app/api/*` typically declares an `APIRouter` (`router = APIRouter(...)`) and defines endpoints with decorators like `@router.get(...)`, `@router.post(...)`, etc.
+- Board-scoped modules embed `{board_id}` in the prefix (e.g. `/boards/{board_id}/tasks`).
+- Streaming endpoints usually expose **SSE** endpoints at `.../stream` (see `sse-starlette` usage).
+
+### Where key behaviors live
+
+- **Router wiring / base prefix**: `backend/app/main.py` mounts these routers under `/api/v1/*`.
+- **Auth / access control** is mostly expressed through dependencies (see `backend/app/api/deps.py`):
+  - `require_admin_auth` — require an authenticated *admin user*.
+  - `require_admin_or_agent` — allow either an admin user or an authenticated agent.
+  - `get_board_for_actor_read` / `get_board_for_actor_write` — enforce board access for the calling actor.
+  - `require_org_member` / `require_org_admin` — enforce org membership/admin for user callers.
+- **Agent-only surface**: `backend/app/api/agent.py` uses `get_agent_auth_context` (X-Agent-Token) and contains board/task/memory endpoints specifically for automation.
+
+### Module-by-module map (prefix, key endpoints, and pointers)
+
+This is a “where to look” index, not a full OpenAPI dump. For exact parameters and response shapes, see:
+- route module file (`backend/app/api/<module>.py`)
+- schemas (`backend/app/schemas/*`)
+- models (`backend/app/models/*`)
+- services (`backend/app/services/*`)
+
+| Module | Prefix (under `/api/v1`) | Key endpoints (examples) | Main deps / auth | Pointers (schemas/models/services) |
+|---|---|---|---|---|
+| `activity.py` | `/activity` | `GET /activity` (events); `GET /activity/task-comments` + `/stream` | `require_admin_or_agent`, `require_org_member` | `app/models/activity_events.py`, `app/schemas/activity_events.py` |
+| `agent.py` | `/agent` | agent automation surface: boards/tasks/memory + gateway coordination | `get_agent_auth_context` (X-Agent-Token) | `backend/app/core/agent_auth.py`, `backend/app/services/openclaw/*` |
+| `agents.py` | `/agents` | agent lifecycle + SSE stream + heartbeat | org-admin gated for user callers; some endpoints allow agent access via deps | `app/schemas/agents.py`, `app/services/openclaw/provisioning_db.py` |
+| `approvals.py` | `/boards/{board_id}/approvals` | list/create/update approvals + `/stream` | `require_admin_or_agent` + board access deps | `app/models/approvals.py`, `app/schemas/approvals.py` |
+| `auth.py` | `/auth` | `POST /auth/bootstrap` | `get_auth_context` (Clerk/user) | `backend/app/core/auth.py`, `app/schemas/users.py` |
+| `board_group_memory.py` | `/board-groups/{group_id}/memory` and `/boards/{board_id}/group-memory` | list/create + `/stream` (group + board-context views) | `require_admin_or_agent`, `require_org_member`, board access deps | `app/models/board_group_memory.py`, `app/schemas/board_group_memory.py` |
+| `board_groups.py` | `/board-groups` | CRUD + snapshot + heartbeat apply | `require_org_member` / `require_org_admin` | `app/services/board_group_snapshot.py` |
+| `board_memory.py` | `/boards/{board_id}/memory` | list/create + `/stream` | `require_admin_or_agent` + board access deps | `app/models/board_memory.py`, `app/schemas/board_memory.py` |
+| `board_onboarding.py` | `/boards/{board_id}/onboarding` | start/answer/confirm onboarding; agent update callback | mix of admin-user and admin-or-agent deps | `app/models/board_onboarding.py`, `app/services/openclaw/onboarding_service.py` |
+| `boards.py` | `/boards` | list/create/update/delete; `/snapshot`; `/group-snapshot` | user/org access (`require_org_member`); read access via `get_board_for_actor_read` in some paths | `app/services/board_snapshot.py`, `app/schemas/boards.py` |
+| `gateway.py` | `/gateways` | session inspection + supported commands list | org-admin user only | `backend/app/services/openclaw/gateway_rpc.py`, `app/schemas/gateway_api.py` |
+| `gateways.py` | `/gateways` | gateway CRUD + template sync | org-admin user only | `app/models/gateways.py`, `app/services/openclaw/*` |
+| `metrics.py` | `/metrics` | `GET /metrics/dashboard` | org-member user access | `app/schemas/metrics.py` |
+| `organizations.py` | `/organizations` | org create + invites/membership flows | user (Clerk) + org member/admin checks | `app/services/organizations.py` |
+| `souls_directory.py` | `/souls-directory` | search + fetch markdown entries | `require_admin_or_agent` | `app/services/souls_directory.py`, `app/schemas/souls_directory.py` |
+| `tasks.py` | `/boards/{board_id}/tasks` | list/create/update/delete tasks; comments; `/stream` | mix of admin-only + admin-or-agent; board access via deps | `app/models/tasks.py`, `app/schemas/tasks.py`, `app/services/task_dependencies.py` |
+| `users.py` | `/users` | `GET/PATCH/DELETE /users/me` | user (Clerk) | `app/models/users.py`, `app/schemas/users.py` |
+
 ## `/activity` — `activity.py`
 *Activity listing and task-comment feed endpoints.*

--- a/docs/12-backend-core.md
+++ b/docs/12-backend-core.md
@@ -0,0 +1,137 @@
+# Backend core modules (auth/config/logging/errors)
+
+> Evidence basis: repo https://github.com/abhi1693/openclaw-mission-control @ commit `c3490630a4503d9c8142aaa3abf542e0d00b5035`.
+
+This page documents the backend “core” layer under `backend/app/core/*` plus the API dependency module `backend/app/api/deps.py`.
+
+It’s written for maintainers who need to answer:
+
+- “Where does configuration come from?”
+- “How do user vs agent auth work?”
+- “Where are authorization decisions enforced?”
+- “What’s the error envelope / request-id behavior?”
+- “How is logging structured and how do I get request-context in logs?”
+
+## Start here (reading order)
+
+1. `backend/app/core/config.py` — settings + env file loading
+2. `backend/app/core/logging.py` — structured logging + request context
+3. `backend/app/core/error_handling.py` — request-id middleware + exception envelope
+4. `backend/app/core/auth.py` — Clerk/user auth resolution
+5. `backend/app/core/agent_auth.py` — agent token auth resolution
+6. `backend/app/api/deps.py` — how routes declare and enforce access
+
+## Configuration: loading & precedence
+
+**Primary file:** `backend/app/core/config.py`
+
+Key facts:
+- Uses `pydantic-settings` (`BaseSettings`) to load typed settings from environment.
+- Env files are loaded regardless of current working directory:
+  - `backend/.env` (via `DEFAULT_ENV_FILE`)
+  - then `.env` (repo root) as an additional source
+  - See `Settings.model_config.env_file=[DEFAULT_ENV_FILE, ".env"]`.
+- Unknown env vars are ignored (`extra="ignore"`).
+
+Notable settings (security-sensitive in **bold**):
+- `DATABASE_URL` / `database_url`
+- `CORS_ORIGINS` / `cors_origins`
+- `DB_AUTO_MIGRATE` / `db_auto_migrate`
+- **`CLERK_SECRET_KEY` / `clerk_secret_key`** (must be non-empty; validator enforces it)
+- `CLERK_API_URL`, `CLERK_VERIFY_IAT`, `CLERK_LEEWAY`
+- logging knobs: `LOG_LEVEL`, `LOG_FORMAT`, `LOG_USE_UTC`, `REQUEST_LOG_SLOW_MS`, `REQUEST_LOG_INCLUDE_HEALTH`
+
+### Deployment implication
+
+- If a deployment accidentally starts the backend with an empty/placeholder `CLERK_SECRET_KEY`, the backend will fail settings validation at startup.
+
+## Auth model split
+
+The backend supports two top-level actor types:
+
+- **User** (human UI / admin) — resolved from the `Authorization: Bearer <token>` header via Clerk.
+- **Agent** (automation) — resolved from `X-Agent-Token: <token>` (and optionally `Authorization: Bearer <token>` for agent callers).
+
+### User auth (Clerk) — `backend/app/core/auth.py`
+
+What it does:
+- Uses the `clerk_backend_api` SDK to authenticate requests (`authenticate_request(...)`) using `CLERK_SECRET_KEY`.
+- Resolves a `AuthContext` containing `actor_type="user"` and a `User` model instance.
+- The module includes helpers to fetch user profile details from Clerk (`_fetch_clerk_profile`) and to delete a Clerk user (`delete_clerk_user`).
+
+Security-sensitive notes:
+- Treat `CLERK_SECRET_KEY` as a credential; never log it.
+- This code calls Clerk API endpoints over the network (timeouts and error handling matter).
+
+### Agent auth (token hash) — `backend/app/core/agent_auth.py`
+
+What it does:
+- Requires a token header for protected agent endpoints:
+  - Primary header: `X-Agent-Token`
+  - Optional parsing: `Authorization: Bearer ...` (only in `get_agent_auth_context`, and only if `accept_authorization=True`)
+- Validates token by comparing it against stored `agent_token_hash` values in the DB (`verify_agent_token`).
+- “Touches” agent presence (`last_seen_at`, `status`) on authenticated requests.
+  - For safe methods (`GET/HEAD/OPTIONS`), it commits immediately so read-only polling still shows the agent as online.
+
+Security-sensitive notes:
+- Token verification iterates over agents with a token hash. If this grows large, consider indexing/lookup strategy.
+- Never echo full tokens in logs; current code logs only a prefix on invalid tokens.
+
+## Authorization enforcement: `backend/app/api/deps.py`
+
+This module is the primary “policy wiring” for most routes.
+
+Key concepts:
+
+- `require_admin_auth(...)`
+  - Requires an authenticated *admin user*.
+- `require_admin_or_agent(...)` → returns `ActorContext`
+  - Allows either:
+    - admin user (user auth via Clerk), or
+    - authenticated agent (agent auth via X-Agent-Token).
+
+Board/task access patterns:
+- `get_board_for_actor_read` / `get_board_for_actor_write`
+  - Enforces that the caller (user or agent) has the correct access to the board.
+  - Agent access is restricted if the agent is bound to a specific board (`agent.board_id`).
+- `get_task_or_404`
+  - Loads a task and ensures it belongs to the requested board.
+
+Org access patterns (user callers):
+- `require_org_member` and `require_org_admin`
+  - Resolve/require active org membership.
+  - Provide an `OrganizationContext` with `organization` + `member`.
+
+Maintainer tip:
+- When debugging a “why is this 403/401?”, start by checking the route’s dependency stack (in the route module) and trace through the relevant dependency in `deps.py`.
+
+## Logging: structure + request context
+
+**Primary file:** `backend/app/core/logging.py`
+
+Highlights:
+- Defines a custom TRACE level (`TRACE_LEVEL = 5`).
+- Uses `contextvars` to carry `request_id`, `method`, and `path` across async tasks.
+- `AppLogFilter` injects `app`, `version`, and request context into each log record.
+- Supports JSON output (`JsonFormatter`) and key=value (`KeyValueFormatter`) formats.
+
+Where request context gets set:
+- `backend/app/core/error_handling.py` middleware calls:
+  - `set_request_id(...)`
+  - `set_request_route_context(method, path)`
+
+## Error envelope + request-id
+
+**Primary file:** `backend/app/core/error_handling.py`
+
+Key behaviors:
+- Installs a `RequestIdMiddleware` (ASGI) that:
+  - Accepts client-provided `X-Request-Id` or generates one.
+  - Adds `X-Request-Id` to the response.
+  - Emits structured “http.request.*” logs, including “slow request” warnings.
+- Error responses include `request_id` when available:
+  - Validation errors (`422`) return `{detail: <errors>, request_id: ...}`.
+  - Other HTTP errors are wrapped similarly.
+
+Maintainer tip:
+- When debugging incidents, ask for the `X-Request-Id` from the client and use it to locate backend logs quickly.
--- a/docs/README.md
+++ b/docs/README.md
@@ -17,6 +17,7 @@ This folder is the canonical documentation set for Mission Control.
 5. [Architecture](05-architecture.md)
 6. [Configuration](06-configuration.md)
 7. [API reference](07-api-reference.md)
+   - [Frontend API + auth modules](frontend-api-auth.md)
 8. [Agents & skills](08-agents-and-skills.md)
 9. [Ops / runbooks](09-ops-runbooks.md)
 10. [Troubleshooting](10-troubleshooting.md)
--- a/docs/frontend-api-auth.md
+++ b/docs/frontend-api-auth.md
@@ -0,0 +1,109 @@
+# Frontend API client and auth integration
+
+This page documents the frontend integration points you’ll touch when changing how the UI talks to the backend or how auth is applied.
+
+## Related docs
+
+- [Architecture](05-architecture.md)
+- [Configuration](06-configuration.md)
+- [API reference](07-api-reference.md)
+
+## API base URL
+
+The frontend uses `NEXT_PUBLIC_API_URL` as the single source of truth for where to send API requests.
+
+- Code: `frontend/src/lib/api-base.ts`
+- Behavior:
+  - reads `process.env.NEXT_PUBLIC_API_URL`
+  - normalizes by trimming trailing slashes
+  - throws early if missing/invalid
+
+In Docker Compose, `compose.yml` sets `NEXT_PUBLIC_API_URL` both:
+- as a **build arg** (for `next build`), and
+- as a **runtime env var**.
+
+## API client layout
+
+### Generated client
+
+- Location: `frontend/src/api/generated/*`
+- Generator: **Orval**
+  - Config: `frontend/orval.config.ts`
+  - Script: `cd frontend && npm run api:gen`
+  - Convenience target: `make api-gen`
+
+By default, Orval reads the backend OpenAPI schema from:
+- `ORVAL_INPUT` (if set), otherwise
+- `http://127.0.0.1:8000/openapi.json`
+
+Output details (from `orval.config.ts`):
+- Mode: `tags-split`
+- Target index: `frontend/src/api/generated/index.ts`
+- Schemas: `frontend/src/api/generated/model`
+- Client: `react-query`
+- All requests go through the custom mutator below.
+
+### Custom fetch / mutator
+
+All generated requests go through:
+
+- Code: `frontend/src/api/mutator.ts`
+- What it does:
+  - resolves `NEXT_PUBLIC_API_URL` and builds the full request URL
+  - sets `Content-Type: application/json` when there’s a body
+  - injects `Authorization: Bearer <token>` when a Clerk session token is available
+  - converts non-2xx responses into a typed `ApiError` (status + parsed response)
+
+## Auth enablement and token injection
+
+### Clerk enablement (publishable key gating)
+
+Clerk is enabled in the frontend only when `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` looks valid.
+
+- Gating helper (dependency-free): `frontend/src/auth/clerkKey.ts`
+- UI-safe wrappers/hooks: `frontend/src/auth/clerk.tsx`
+  - provides `SignedIn`, `SignedOut`, `SignInButton`, `SignOutButton`, `useUser`, and `useAuth`
+  - returns safe fallbacks when Clerk is disabled (to allow secretless builds/prerender)
+
+### Token injection
+
+When the UI makes an API request, the mutator attempts to read a token from the Clerk session:
+
+- Code: `frontend/src/api/mutator.ts` (`resolveClerkToken()`)
+- If a token is available, the request includes:
+  - `Authorization: Bearer <token>`
+
+### Route protection (middleware)
+
+Request-time route protection is implemented via Next.js middleware:
+
+- Code: `frontend/src/proxy.ts`
+- Behavior:
+  - when Clerk is enabled: uses `clerkMiddleware()` to enforce auth on non-public routes
+  - when Clerk is disabled: passes all requests through
+
+## Common workflows
+
+### Update the backend API and regenerate the client
+
+1. Run the backend so OpenAPI is available:
+
+```bash
+# from repo root
+cp backend/.env.example backend/.env
+make backend-migrate
+cd backend && uv run uvicorn app.main:app --reload --port 8000
+```
+
+2. Regenerate the client:
+
+```bash
+# from repo root
+make api-gen
+
+# or from frontend/
+ORVAL_INPUT=http://127.0.0.1:8000/openapi.json npm run api:gen
+```
+
+3. Review diffs under `frontend/src/api/generated/*`.
+