chore(backend): upgrade deps and remove redis/rq

This commit is contained in:
Abhimanyu Saharan
2026-02-10 16:05:49 +05:30
parent dcdc0a25b1
commit 5c25c4bb91
17 changed files with 538 additions and 439 deletions

View File

@@ -11,9 +11,6 @@ POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_PORT=5432
# --- redis ---
REDIS_PORT=6379
# --- backend settings (see backend/.env.example for full list) ---
CORS_ORIGINS=http://localhost:3000
DB_AUTO_MIGRATE=true

View File

@@ -16,10 +16,10 @@
From repo root:
- `make setup`: install/sync backend + frontend dependencies.
- `make check`: CI-equivalent suite (lint, typecheck, tests/coverage, frontend build).
- `docker compose -f compose.yml --env-file .env up -d --build`: run full stack (Postgres + Redis included).
- `docker compose -f compose.yml --env-file .env up -d --build`: run full stack.
Fast local dev:
- `docker compose -f compose.yml --env-file .env up -d db redis`
- `docker compose -f compose.yml --env-file .env up -d db`
- Backend: `cd backend && uv sync --extra dev && uv run uvicorn app.main:app --reload --port 8000`
- Frontend: `cd frontend && npm install && npm run dev`
- API client: `make api-gen` (backend must be running on `127.0.0.1:8000`).

View File

@@ -13,7 +13,7 @@ OpenClaw Mission Control is under active development. Expect breaking changes an
- **Frontend:** Next.js app (default http://localhost:3000)
- **Backend:** FastAPI service (default http://localhost:8000)
- **Data:** Postgres + Redis
- **Data:** Postgres
- **Gateway integration:** see [`docs/openclaw_gateway_ws.md`](./docs/openclaw_gateway_ws.md)
> Note on auth (Clerk)
@@ -65,13 +65,13 @@ docker compose -f compose.yml --env-file .env logs -f --tail=200
# Rebuild a single service
docker compose -f compose.yml --env-file .env up -d --build backend
# Reset data (DESTRUCTIVE: deletes Postgres/Redis volumes)
# Reset data (DESTRUCTIVE: deletes Postgres volume)
docker compose -f compose.yml --env-file .env down -v
```
## Quick start (local development)
This is the fastest workflow for contributors: run Postgres/Redis via Docker, and run the backend + frontend in dev mode.
This is the fastest workflow for contributors: run Postgres via Docker, and run the backend + frontend in dev mode.
### Prerequisites
@@ -79,12 +79,12 @@ This is the fastest workflow for contributors: run Postgres/Redis via Docker, an
- Python **3.12+** + [`uv`](https://github.com/astral-sh/uv)
- Node.js (recommend 18+) + npm
### 1) Start Postgres + Redis
### 1) Start Postgres
```bash
cp .env.example .env
docker compose -f compose.yml --env-file .env up -d db redis
docker compose -f compose.yml --env-file .env up -d db
```
### 2) Backend (FastAPI)
@@ -103,7 +103,7 @@ uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
Notes:
- If you run the DB/Redis containers, the backend should use the defaults in `backend/.env` (`localhost:5432` and `localhost:6379`).
- If you run the DB container, the backend should use the default in `backend/.env` (`localhost:5432`).
- Database migrations:
```bash
@@ -137,7 +137,6 @@ When running Cypress (`cd frontend && npm run e2e`), make sure `NEXT_PUBLIC_API_
- **Mission Control backend** exposes a REST API at `/api/v1/*` and also hosts health endpoints (`/healthz`, `/readyz`).
- **Mission Control frontend** calls the backend via `NEXT_PUBLIC_API_URL`.
- **Postgres** stores boards/tasks/agents/etc.
- **Redis** is used for background work (RQ).
- **OpenClaw Gateway** connectivity is over WebSockets; protocol details live in [`docs/openclaw_gateway_ws.md`](./docs/openclaw_gateway_ws.md).
## Common commands
@@ -178,7 +177,7 @@ You likely have `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` set (even to a placeholder).
- Remove the `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` line from `frontend/.env.local`, **or** set it to an empty value.
### Backend cant connect to Postgres/Redis
### Backend cant connect to Postgres
- Confirm containers are up:
@@ -188,7 +187,6 @@ You likely have `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` set (even to a placeholder).
- If youre running backend locally (not in compose), make sure `backend/.env` points to `localhost`:
- `DATABASE_URL=postgresql+psycopg://postgres:postgres@localhost:5432/mission_control`
- `REDIS_URL=redis://localhost:6379/0`
### Port already in use
@@ -197,7 +195,6 @@ Adjust ports in `.env` (copied from `.env.example`):
- `FRONTEND_PORT`
- `BACKEND_PORT`
- `POSTGRES_PORT`
- `REDIS_PORT`
## Star History

View File

@@ -1,7 +1,6 @@
ENVIRONMENT=dev
LOG_LEVEL=INFO
DATABASE_URL=postgresql+psycopg://postgres:postgres@localhost:5432/mission_control
REDIS_URL=redis://localhost:6379/0
CORS_ORIGINS=http://localhost:3000
BASE_URL=

View File

@@ -11,16 +11,15 @@ This directory contains the **Mission Control backend API** (FastAPI + SQLModel)
- Python **3.12+**
- [`uv`](https://github.com/astral-sh/uv) (recommended; used by this repo)
- Postgres (local or Docker)
- Redis (local or Docker)
## Quick start (local backend + Docker Postgres/Redis)
## Quick start (local backend + Docker Postgres)
From the repo root:
```bash
# start dependencies
cp .env.example .env
docker compose -f compose.yml --env-file .env up -d db redis
docker compose -f compose.yml --env-file .env up -d db
# run backend
cd backend
@@ -56,7 +55,6 @@ A starter file exists at `backend/.env.example`.
- Default: `postgresql+psycopg://postgres:postgres@localhost:5432/openclaw_agency`
- Recommended local/dev default (matches `backend/.env.example`):
`postgresql+psycopg://postgres:postgres@localhost:5432/mission_control`
- `REDIS_URL` (default: `redis://localhost:6379/0`)
- `CORS_ORIGINS` (comma-separated)
- Example: `http://localhost:3000`
- `BASE_URL` (optional)
@@ -153,16 +151,6 @@ uv run python scripts/export_openapi.py
- If backend runs **locally** (not in compose), `DATABASE_URL` should usually point at `localhost`.
### Backend cant connect to Redis
- Ensure the Redis container is up:
```bash
docker compose -f compose.yml --env-file .env logs -f --tail=200 redis
```
- Confirm `REDIS_URL=redis://localhost:6379/0` when running backend locally.
### CORS issues from the frontend
- Set `CORS_ORIGINS=http://localhost:3000` (or a comma-separated list) in `backend/.env`.

View File

@@ -25,7 +25,6 @@ class Settings(BaseSettings):
environment: str = "dev"
database_url: str = "postgresql+psycopg://postgres:postgres@localhost:5432/openclaw_agency"
redis_url: str = "redis://localhost:6379/0"
# Clerk auth (auth only; roles stored in DB)
clerk_secret_key: str = Field(min_length=1)

View File

@@ -35,7 +35,7 @@ class GatewayConfig:
def _build_gateway_url(config: GatewayConfig) -> str:
base_url = (config.url or "").strip()
base_url: str = (config.url or "").strip()
if not base_url:
message = "Gateway URL is not configured for this board."
raise OpenClawGatewayError(message)
@@ -44,11 +44,11 @@ def _build_gateway_url(config: GatewayConfig) -> str:
return base_url
parsed = urlparse(base_url)
query = urlencode({"token": token})
return urlunparse(parsed._replace(query=query))
return str(urlunparse(parsed._replace(query=query)))
async def _await_response(
ws: websockets.WebSocketClientProtocol,
ws: websockets.ClientConnection,
request_id: str,
) -> object:
while True:
@@ -70,7 +70,7 @@ async def _await_response(
async def _send_request(
ws: websockets.WebSocketClientProtocol,
ws: websockets.ClientConnection,
method: str,
params: dict[str, Any] | None,
) -> object:
@@ -102,7 +102,7 @@ def _build_connect_params(config: GatewayConfig) -> dict[str, Any]:
async def _ensure_connected(
ws: websockets.WebSocketClientProtocol,
ws: websockets.ClientConnection,
first_message: str | bytes | None,
config: GatewayConfig,
) -> None:

View File

@@ -1,18 +0,0 @@
"""RQ queue and Redis connection helpers for background workers."""
from __future__ import annotations
from redis import Redis
from rq import Queue
from app.core.config import settings
def get_redis() -> Redis:
"""Create a Redis client from configured settings."""
return Redis.from_url(settings.redis_url)
def get_queue(name: str) -> Queue:
"""Return an RQ queue bound to the configured Redis connection."""
return Queue(name, connection=get_redis())

View File

@@ -12,36 +12,34 @@ name = "openclaw-agency-backend"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"fastapi==0.128.0",
"uvicorn[standard]==0.30.6",
"sqlmodel==0.0.22",
"sqlalchemy[asyncio]==2.0.34",
"alembic==1.13.2",
"psycopg[binary]==3.3.2",
"pydantic-settings==2.5.2",
"python-dotenv==1.0.1",
"websockets==12.0",
"rq==1.16.2",
"redis==5.1.1",
"sse-starlette==2.1.3",
"alembic==1.18.3",
"clerk-backend-api==4.2.0",
"fastapi==0.128.6",
"fastapi-pagination==0.15.10",
"jinja2==3.1.6",
"fastapi-pagination==0.15.9",
"clerk-backend-api==1.4.1",
"psycopg[binary]==3.3.2",
"pydantic-settings==2.12.0",
"python-dotenv==1.2.1",
"sqlalchemy[asyncio]==2.0.46",
"sqlmodel==0.0.32",
"sse-starlette==3.2.0",
"uvicorn[standard]==0.40.0",
"websockets==16.0",
]
[project.optional-dependencies]
dev = [
"black==24.10.0",
"flake8==7.1.1",
"httpx==0.27.0",
"isort==5.13.2",
"mypy==1.11.2",
"pytest==8.3.3",
"pytest-asyncio==0.24.0",
"pytest-cov==6.0.0",
"coverage[toml]==7.6.10",
"ruff==0.6.9",
"aiosqlite==0.21.0",
"aiosqlite==0.22.1",
"black==26.1.0",
"coverage[toml]==7.13.4",
"flake8==7.3.0",
"httpx==0.28.1",
"isort==7.0.0",
"mypy==1.19.1",
"pytest==9.0.2",
"pytest-asyncio==1.3.0",
"pytest-cov==7.0.0",
"ruff==0.15.0",
]
[tool.mypy]

View File

@@ -18,7 +18,7 @@ async def run() -> None:
from app.models.boards import Board
from app.models.gateways import Gateway
from app.models.users import User
from app.services.openclaw import GatewayAgentIdentity
from app.services.openclaw.shared import GatewayAgentIdentity
await init_db()
async with async_session_maker() as session:

View File

@@ -52,7 +52,10 @@ def _parse_args() -> argparse.Namespace:
async def _run() -> int:
from app.db.session import async_session_maker
from app.models.gateways import Gateway
from app.services.openclaw import GatewayTemplateSyncOptions, sync_gateway_templates
from app.services.openclaw.provisioning import (
GatewayTemplateSyncOptions,
sync_gateway_templates,
)
args = _parse_args()
gateway_id = UUID(args.gateway_id)

790
backend/uv.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -17,18 +17,6 @@ services:
timeout: 3s
retries: 20
redis:
image: redis:7-alpine
volumes:
- redis_data:/data
ports:
- "${REDIS_PORT:-6379}:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 20
backend:
build:
# Build from repo root so the backend image can include repo-level assets
@@ -40,14 +28,11 @@ services:
environment:
# Override localhost defaults for container networking
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-mission_control}
REDIS_URL: redis://redis:6379/0
CORS_ORIGINS: ${CORS_ORIGINS:-http://localhost:3000}
DB_AUTO_MIGRATE: ${DB_AUTO_MIGRATE:-true}
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
ports:
- "${BACKEND_PORT:-8000}:8000"
@@ -71,4 +56,3 @@ services:
volumes:
postgres_data:
redis_data:

View File

@@ -8,7 +8,6 @@ At a high level:
- The **frontend** is a Next.js app used by humans.
- The **backend** is a FastAPI service that exposes REST endpoints under `/api/v1/*`.
- **Postgres** stores core state (boards/tasks/agents/etc.).
- **Redis** supports async/background primitives (RQ queue scaffolding exists).
## Components
@@ -20,7 +19,6 @@ flowchart LR
FE -->|HTTP /api/v1/*| BE[FastAPI Backend :8000]
BE -->|SQL| PG[(Postgres :5432)]
BE -->|Redis protocol| R[(Redis :6379)]
BE -->|WebSocket (optional integration)| GW[OpenClaw Gateway]
GW --> OC[OpenClaw runtime]
@@ -50,8 +48,6 @@ flowchart LR
- **Postgres**: persistence for boards/tasks/agents/approvals/etc.
- Models: `backend/app/models/*`
- Migrations: `backend/migrations/*`
- **Redis**: used for background primitives.
- RQ helper: `backend/app/workers/queue.py`
### Gateway integration (optional)
Mission Control can call into an OpenClaw Gateway over WebSockets.
@@ -64,7 +60,7 @@ Mission Control can call into an OpenClaw Gateway over WebSockets.
### UI → API
1. Browser loads the Next.js frontend.
2. Frontend calls backend endpoints under `/api/v1/*`.
3. Backend reads/writes Postgres and may use Redis depending on the operation.
3. Backend reads/writes Postgres.
### Auth (Clerk — required for now)
- **Frontend** enables Clerk when a publishable key is present/valid.
@@ -76,11 +72,8 @@ Automation/agents can use the “agent” API surface:
- Endpoints under `/api/v1/agent/*` (router: `backend/app/api/agent.py`).
- Auth via `X-Agent-Token` (see `backend/app/core/agent_auth.py`, referenced from `backend/app/api/deps.py`).
### Background jobs (RQ / Redis)
The codebase includes RQ/Redis dependencies and a queue helper (`backend/app/workers/queue.py`).
If/when background jobs are added, the expected shape is:
- API enqueues work to Redis.
- A separate RQ worker process executes queued jobs.
### Background jobs
There is currently no queue runtime configured in this repo.
## Key directories

View File

@@ -4,14 +4,13 @@ This guide covers how to self-host **OpenClaw Mission Control** using the reposi
> Scope
> - This is a **dev-friendly self-host** setup intended for local or single-host deployments.
> - For production hardening (TLS, backups, external Postgres/Redis, observability), see **Production notes** below.
> - For production hardening (TLS, backups, external Postgres, observability), see **Production notes** below.
## What you get
When running Compose, you get:
- **Postgres** database (persistent volume)
- **Redis** (persistent volume)
- **Backend API** (FastAPI) on `http://localhost:${BACKEND_PORT:-8000}`
- Health check: `GET /healthz`
- **Frontend UI** (Next.js) on `http://localhost:${FRONTEND_PORT:-3000}`
@@ -61,7 +60,6 @@ curl -I http://localhost:${FRONTEND_PORT:-3000}/
`compose.yml` defines:
- `db` (Postgres 16)
- `redis` (Redis 7)
- `backend` (FastAPI)
- `frontend` (Next.js)
@@ -70,7 +68,6 @@ curl -I http://localhost:${FRONTEND_PORT:-3000}/
By default:
- Postgres: `5432` (`POSTGRES_PORT`)
- Redis: `6379` (`REDIS_PORT`)
- Backend: `8000` (`BACKEND_PORT`)
- Frontend: `3000` (`FRONTEND_PORT`)
@@ -81,7 +78,6 @@ Ports are sourced from `.env` (passed via `--env-file .env`) and wired into `com
Compose creates named volumes:
- `postgres_data` → Postgres data directory
- `redis_data` → Redis data directory
These persist across `docker compose down`.
@@ -100,7 +96,7 @@ docker compose -f compose.yml --env-file .env ...
### Backend env
The backend container loads `./backend/.env.example` via `env_file` and then overrides DB/Redis URLs for container networking.
The backend container loads `./backend/.env.example` via `env_file` and then overrides the DB URL for container networking.
If you need backend customization, prefer creating a real `backend/.env` and updating compose to use it (optional improvement).
@@ -174,8 +170,6 @@ docker compose -f compose.yml --env-file .env logs -f --tail=200
- If the repo doesnt have `frontend/public`, the Dockerfile should not `COPY public/`.
- **Backend build fails looking for `uv.lock`**
- If backend build context is repo root, Dockerfile must copy `backend/uv.lock` not `uv.lock`.
- **Redis warning about `vm.overcommit_memory`**
- Usually non-fatal for dev; for stability under load, set `vm.overcommit_memory=1` on the host.
## Reset / start fresh
@@ -185,7 +179,7 @@ Safe (keeps volumes/data):
docker compose -f compose.yml --env-file .env down
```
Destructive (removes volumes; deletes Postgres/Redis data):
Destructive (removes volumes; deletes Postgres data):
```bash
docker compose -f compose.yml --env-file .env down -v
@@ -195,7 +189,7 @@ docker compose -f compose.yml --env-file .env down -v
If youre running this beyond local dev, consider:
- Run Postgres and Redis as managed services (or on separate hosts)
- Run Postgres as a managed service (or on a separate host)
- Add TLS termination (reverse proxy)
- Configure backups for Postgres volume
- Set explicit resource limits and healthchecks

View File

@@ -2,11 +2,11 @@
This document describes **production-ish** deployment patterns for **OpenClaw Mission Control**.
Mission Control is a web app (frontend) + API (backend) + Postgres + Redis. The simplest reliable
Mission Control is a web app (frontend) + API (backend) + Postgres. The simplest reliable
baseline is Docker Compose plus a reverse proxy with TLS.
> This repo currently ships a developer-friendly `compose.yml`. For real production, you should:
> - put Postgres/Redis on managed services or dedicated hosts when possible
> - put Postgres on a managed service or dedicated host when possible
> - terminate TLS at a reverse proxy
> - set up backups + upgrades
> - restrict network exposure (firewall)
@@ -33,7 +33,6 @@ On one VM:
- frontend container (internal port 3000)
- backend container (internal port 8000)
- Postgres container (internal 5432)
- Redis container (internal 6379)
### Ports / firewall
@@ -44,7 +43,6 @@ Expose to the internet:
Do **not** expose:
- Postgres 5432
- Redis 6379
- backend 8000
- frontend 3000
@@ -160,13 +158,13 @@ The main reason to split is reliability and blast-radius reduction.
### Option A: 2 hosts
- Host 1: reverse proxy + frontend + backend
- Host 2: Postgres + Redis (or managed)
- Host 2: Postgres (or managed)
### Option B: 3 hosts
- Host 1: reverse proxy + frontend
- Host 2: backend
- Host 3: Postgres + Redis (or managed)
- Host 3: Postgres (or managed)
### Networking / security groups
@@ -175,14 +173,12 @@ Minimum rules:
- Public internet → reverse proxy host: `80/443`
- Reverse proxy host → backend host: `8000` (or whatever you publish internally)
- Backend host → DB host: `5432`
- Backend host → Redis host: `6379`
Everything else: deny.
### Configuration considerations
- `DATABASE_URL` must point to the DB host (not `localhost`).
- `REDIS_URL` must point to the Redis host.
- `CORS_ORIGINS` must include the public frontend URL.
- `NEXT_PUBLIC_API_URL` should be the public API base URL.
@@ -197,7 +193,7 @@ The backend currently runs Alembic migrations on startup (see logs). In multi-ho
- [ ] TLS is enabled, HTTP redirects to HTTPS
- [ ] Only 80/443 exposed publicly
- [ ] Postgres/Redis not publicly accessible
- [ ] Postgres not publicly accessible
- [ ] Backups tested (restore drill)
- [ ] Log retention/rotation configured
- [ ] Regular upgrade process (pull latest, rebuild, restart)
@@ -209,4 +205,3 @@ The backend currently runs Alembic migrations on startup (see logs). In multi-ho
- `NEXT_PUBLIC_API_URL`
- backend CORS settings (`CORS_ORIGINS`)
- firewall rules between proxy ↔ backend

View File

@@ -128,7 +128,7 @@ npm run api:gen # regenerate typed API client via Orval
There is a `frontend/Dockerfile` used by the root `compose.yml`.
If youre working on self-hosting, prefer running compose from the repo root so the backend/db/redis are aligned with the documented ports/env.
If youre working on self-hosting, prefer running compose from the repo root so the backend/db are aligned with the documented ports/env.
## Troubleshooting