Operations runbook

Deploy

Single process (mandatory — see AGENTS.md #1). On the VM, as a dedicated service user:

python3 -m venv venv
venv/bin/pip install --upgrade pip
venv/bin/pip install -r requirements.txt      # ranges (recommended); requirements.lock = exact pins
cp config.example.toml config.toml
# edit config.toml: [okta], [pps] creds, [[auth.users]], and a real secret_key:
python -c "import secrets; print(secrets.token_urlsafe(48))"
chmod 600 config.toml
venv/bin/python wsgi.py

The config file must be writable by the service user (the admin panel rewrites it via temp-file + atomic rename). A read-only mount breaks admin saves. wsgi.py refuses to boot with the placeholder or a <32 char secret_key.

Run under systemd — a ready-to-import unit and step-by-step instructions live in deploy/:

sudo cp deploy/ppsq.service /etc/systemd/system/ppsq.service
sudo systemctl daemon-reload && sudo systemctl enable --now ppsq

Keep it a single instance — do not template it or add web-server workers. Put a TLS-terminating reverse proxy (nginx) in front; set [app] cookie_secure = true and listen = "127.0.0.1".

Logs

Application log (app.log): requests, job lifecycle, PPS API calls, errors. Viewable in the admin panel (Logs → Application) with errors in red, or tail -f app.log.
Operations/audit log (worker.log): one line per message acted on, e.g. job=#12 user=alice@x action=report_release step=move result=ok src='Quarantine' dst='Debugging - Josef' localguid=… guid=… from=… rcpt=… subject=…. This is the "who released/deleted what" audit trail. Grep it: grep result=FAILED worker.log. Report & Release writes one line per step (step=move, then step=release), each with that step's own src/dst; single actions have no step= field.

Both rotate at 5 MB × 5 files. The PPS x-pps-reqid appears in error lines for cross-referencing PPS's own webservices log.

Background jobs

View in the admin panel (Jobs) — all users, with status and progress. Failed jobs have a Retry button (POST /api/admin/jobs/<id>/retry).
Jobs persist in jobs.db and survive a restart; a job interrupted mid-run is requeued automatically on boot and resumes at the pipeline step it had reached.
A Report & Release job shows as waiting while it sits out step_delay_seconds between two steps. It is not stuck: other jobs run during the wait, and it is re-claimed within a second of the delay expiring.

Backup

Back up config.toml and jobs.db. For a consistent jobs.db copy while running (WAL mode):

sqlite3 jobs.db ".backup '/backup/jobs.db'"

config.toml.bak holds the previous config after each admin save (one-step rollback).

Rotating credentials

PPS password / Okta secret: enter the new value in the admin panel (PPS section) — it's write-only and applies live (PPS) or after restart (Okta, edit the file). Use the Test connection button before relying on it.
secret_key: edit config.toml and restart. This invalidates all sessions (everyone re-logs-in).
Client TLS cert: replace the file at client_cert; see certs/README.md.

Common issues

Symptom	Cause / fix
`400 No required SSL certificate was sent`	PPS/proxy wants mutual TLS. Set `[pps] client_cert`.
Every login fails `mismatching_state`	Cookie/SameSite or a proxy dropping cookies; ensure `cookie_secure`/HTTPS and `redirect_uri` match.
`database is locked`	WAL/busy_timeout missing — should not happen; verify no second process opened `jobs.db`.
Admin save fails to persist	`config.toml`/its directory not writable by the service user.
Folder shows few/zero messages	`default_days_back` window too small, or `list_query` not match-all for this PPS.
Messages older than N days missing	Bounded by `default_days_back` — raise it.

operations.md 3.9 KB Pysyvä linkki Historia Raaka