fb8a932923
Rate-limit the optional agent bearer path after user auth resolution so mixed user/agent routes no longer leave an unthrottled PBKDF2 path. Stop logging token prefixes on agent auth failures and require a locally supplied token for backend/.env.test instead of committing one. Update tests and docs to cover agent bearer fallback, configurable webhook signature headers, and the operator-facing security settings added by the hardening work. Co-Authored-By: Claude <noreply@anthropic.com>
3.7 KiB
Operations
Runbooks and operational notes for running Mission Control.
Health checks
Backend exposes:
/healthz— liveness/readyz— readiness
Example:
curl -f http://localhost:8000/healthz
curl -f http://localhost:8000/readyz
Logs
Docker Compose
# tail everything
docker compose -f compose.yml --env-file .env logs -f --tail=200
# tail just backend
docker compose -f compose.yml --env-file .env logs -f --tail=200 backend
The backend supports slow-request logging via REQUEST_LOG_SLOW_MS.
Backups
The DB runs in Postgres (Compose db service) and persists to the postgres_data named volume.
Minimal backup (logical)
Example with pg_dump (run on the host):
# load variables from .env (trusted file only)
set -a
. ./.env
set +a
: "${POSTGRES_DB:?set POSTGRES_DB in .env}"
: "${POSTGRES_USER:?set POSTGRES_USER in .env}"
: "${POSTGRES_PORT:?set POSTGRES_PORT in .env}"
: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env (strong, unique value; not \"postgres\")}"
PGPASSWORD="$POSTGRES_PASSWORD" pg_dump \
-h 127.0.0.1 -p "$POSTGRES_PORT" -U "$POSTGRES_USER" \
-d "$POSTGRES_DB" \
--format=custom > mission_control.backup
Note
For real production, prefer automated backups + retention + periodic restore drills.
Upgrades / rollbacks
Upgrade (Compose)
docker compose -f compose.yml --env-file .env up -d --build
Rollback
Rollback typically means deploying a previous image/commit.
Warning
If you applied non-backward-compatible DB migrations, rolling back the app may require restoring the database.
Rate limiting
The backend applies per-IP rate limits on sensitive endpoints:
| Endpoint | Limit | Window |
|---|---|---|
| Agent authentication | 20 requests | 60 seconds |
| Webhook ingest | 60 requests | 60 seconds |
Rate-limited requests receive HTTP 429 Too Many Requests.
Set RATE_LIMIT_BACKEND to choose the storage backend:
| Backend | Value | Operational notes |
|---|---|---|
| In-memory (default) | memory |
Per-process limits; each worker tracks independently. No external dependencies. |
| Redis | redis |
Limits are shared across all workers. Set RATE_LIMIT_REDIS_URL or it falls back to RQ_REDIS_URL. Connectivity is validated at startup; transient Redis failures fail open (requests allowed, warning logged). |
When using the in-memory backend in multi-process deployments, also apply rate limiting at the reverse proxy layer (nginx limit_req, Caddy rate limiting, etc.).
Common issues
Frontend loads but API calls fail
- Confirm
NEXT_PUBLIC_API_URLis set and reachable from the browser. - Confirm backend CORS includes the frontend origin (
CORS_ORIGINS).
Auth mismatch
- Backend:
AUTH_MODE(localorclerk) - Frontend:
NEXT_PUBLIC_AUTH_MODEshould match
Webhook signature errors (403)
If a webhook has a secret configured, inbound payloads must include a valid HMAC-SHA256 signature. If the webhook also sets signature_header, that exact header name must be used. Otherwise the backend checks these defaults:
X-Hub-Signature-256: sha256=<hex-digest>(GitHub-style)X-Webhook-Signature: sha256=<hex-digest>
Missing or invalid signatures return 403 Forbidden. If you see unexpected 403s on webhook ingest, verify that the sending service is computing the HMAC correctly using the webhook's secret and sending it in the configured header.
Webhook payload too large (413)
Webhook ingest enforces a 1 MB payload size limit by default. Payloads exceeding this return 413 Content Too Large. If you need to raise the limit, set WEBHOOK_MAX_PAYLOAD_BYTES; otherwise consider sending a URL reference instead of inline content.