Day 2 Operations

​

Production checklist

​

What to back up

​

The encryption key is the most important thing to back up

• Generate the key once, set it explicitly and keep a copy in your secrets manager (Vault, AWS Secrets Manager, Kubernetes Secret with restricted access, and so on).
• Store it separately from your database backups. Anyone holding both can read your credentials.
• If you do not set it, the container generates one and writes it to /app/backend/data/encryption.key. That path must be on a persistent volume, and the file must be backed up. On Kubernetes, always set the key through a Secret, because the chart does not provide storage for this file.

# Generate a key (Fernet, 44 characters)
openssl rand -base64 32 | tr '+/' '-_'

​

Back up the database

# Logical backup of the bundled PostgreSQL
docker compose exec -T postgres \
  pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB" \
  > bow-$(date +%F).dump

# Restore into an empty database
docker compose exec -T postgres \
  pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists \
  < bow-2026-01-01.dump

• Take at least a daily backup, and always take one immediately before an upgrade. Migrations run automatically on startup and there are no downgrade migrations, so a backup is your rollback path.
• Copy backups off the host or cluster, and keep several generations.
• Test a restore into a scratch environment on a schedule. An untested backup is not a backup.
• Do not use SQLite in production. It is a single file on a single node, and it prevents running more than one replica.

​

Back up files and configuration

• Snapshot or copy the Docker volumes uploads_data, branding_data and app_data (Compose), or the equivalent persistent volumes.
• Keep bow-config.yaml, .env, the Compose file, the Caddyfile and your Helm values in version control. Keep secrets out of Git and in a secrets manager.
• Do not forget the Caddy certificate volume (caddy_data) if you want to avoid re-issuing TLS certificates after a rebuild.

​

High availability

​

Requirements for running two replicas

(BOW_DB_POOL_SIZE + BOW_DB_MAX_OVERFLOW) × workers per replica × replicas

​

Scheduler and background listeners

​

Rolling updates and in-flight work

• Agent runs execute inside the replica that received the request. When a replica is stopped, its in-flight runs end. Give pods a longer terminationGracePeriodSeconds than the 30 second default, and deploy during quiet hours.
• If a browser reconnects to a different replica mid-run, it sees the last saved state of the run rather than the live stream.
• Every container applies database migrations when it starts. For releases that change the schema, avoid starting several new replicas at once. Scale to one replica (or use the Recreate strategy), let it migrate and become healthy, then scale back up.
• On Kubernetes, enable the PodDisruptionBudget (podDisruptionBudget.enabled=true) so node drains never take down both replicas, and spread replicas across nodes or zones.

# values.yaml (excerpt)
# Note: the chart always pulls the image (imagePullPolicy: Always), so a
# pinned tag is what protects you from unexpected changes.
replicaCount: 2
image:
  tag: "0.0.587"      # pin a version
podDisruptionBudget:
  enabled: true
terminationGracePeriodSeconds: 120
config:
  secretRef: bow-secrets   # holds BOW_ENCRYPTION_KEY, BOW_DATABASE_URL, ...
  forwardedAllowIps: "*"   # only if the ingress is the sole peer

​

Upgrades

• Pin versions. Images are published as bagofwords/bagofwords:<version>, <version>-<date> and latest. The Compose file and Helm chart default to latest with always-pull. Override this in production.
• Migrations are automatic. On start, the container runs the database migrations and exits if they fail after three attempts. Because there is no downgrade path, rolling back means restoring the pre-upgrade database backup and redeploying the previous image tag.
• Helm chart 2.0.0 changed the Deployment selector labels. Upgrading from chart 1.x fails with field is immutable: spec.selector. Delete the Deployment (kubectl delete deployment <release> -n <namespace>) and run the upgrade again.
• Air-gapped installs upgrade by loading the new bundle’s images and running docker compose up -d --pull never. See Air-gapped downloads.

​

Monitoring

​

Health checks

​

Logs

​

Traces

otel:
  enabled: true
  service_name: bagofwords-backend
  traces_endpoint: http://otel-collector:4317
  protocol: grpc

​

What to alert on

• /health failing, or containers restarting repeatedly.
• Database: connection count approaching max_connections, disk usage, replication lag, failed backups.
• Disk usage on the uploads and artifact volumes.
• Spikes in failed agent runs. Use Runs and traces to investigate.
• LLM spend and errors. Track them in Usage, cost and quality and set quotas.
• TLS certificate expiry, if you manage certificates yourself.

​

Audit logs

​

Security hardening

• HTTPS everywhere. Keep Caddy or your ingress terminating TLS, and set base_url to the public https:// URL. It is used for SSO redirects, and SAML requires HTTPS.
• Change default credentials. Replace the sample PostgreSQL password (bowpassword in the Compose file) and restrict who can reach the database port. Do not publish PostgreSQL to the internet.
• Keep secrets out of config files. Reference them as ${VAR} in bow-config.yaml and inject them from Docker secrets, a Kubernetes Secret (config.secretRef) or your secrets manager.
• Set FORWARDED_ALLOW_IPS correctly. Behind a proxy or ingress, the default (127.0.0.1) makes every visitor look like the same client, which breaks per-client rate limiting. Set it to your proxy’s address.
• Use SSO. Configure SSO, set auth.mode to sso_only if you want to disable local passwords, and review who has admin roles. Disable open sign-ups with allow_uninvited_signups: false.
• Limit network access. Enable the chart’s NetworkPolicy and allow outbound traffic only to your LLM providers, data sources and identity provider. Use HTTPS_PROXY and NO_PROXY if egress goes through a proxy.
• Least-privilege data access. Give each data source a read-only database user, and use row-level security where users should see different data.
• Telemetry. Anonymous product telemetry is enabled by default. Set telemetry.enabled: false in bow-config.yaml (Helm: config.telemetryEnabled=false) if your policy requires it, and always in air-gapped environments.
• Scan and patch. Pull updated images regularly, scan them in your registry, and patch the host and PostgreSQL on a schedule.

​

Capacity and cost

• Size for concurrency. Each worker runs up to BOW_MAX_CONCURRENT_AGENTS agent runs at once (default 12), and extra runs queue. Watch CPU and memory under real load and adjust workers (UVICORN_WORKERS) and resources. The Helm chart requests 2 CPUs and 900Mi by default and sets no memory limit.
• Control LLM spend. Set usage quotas, and review cost regularly. Keep a second model configured, or use fallback on the enterprise plan, so an outage at one LLM provider does not stop your users.
• Protect your data sources. Add rate limits (enterprise) or use a read replica for analytics so heavy questions cannot slow your production systems.
• Retention. Step payloads are purged automatically (14 days by default). Plan for database growth from reports, traces and audit logs, and monitor disk space.

​

Routine operations calendar

​

Disaster recovery in short

1. Provision a new environment and deploy the same image version you were running.
2. Restore the PostgreSQL backup.
3. Restore the file volumes (uploads, branding_uploads, artifact storage).
4. Supply the original BOW_ENCRYPTION_KEY and your bow-config.yaml.
5. Start the application, point DNS at it, and verify sign-in, a data source and a test question.