# Operations runbook

## Deploy

Single process (mandatory — see AGENTS.md #1). On the VM, as a dedicated service user:

```bash
python3 -m venv venv
venv/bin/pip install --upgrade pip
venv/bin/pip install -r requirements.txt      # ranges (recommended); requirements.lock = exact pins
cp config.example.toml config.toml
# edit config.toml: [okta], [pps] creds, [[auth.users]], and a real secret_key:
python -c "import secrets; print(secrets.token_urlsafe(48))"
chmod 600 config.toml
venv/bin/python wsgi.py
```

The config file must be **writable by the service user** (the admin panel rewrites it via
temp-file + atomic rename). A read-only mount breaks admin saves. `wsgi.py` refuses to boot
with the placeholder or a `<32` char `secret_key`.

Run under systemd — a ready-to-import unit and step-by-step instructions live in
[`deploy/`](../deploy/README.md):

```bash
sudo cp deploy/ppsq.service /etc/systemd/system/ppsq.service
sudo systemctl daemon-reload && sudo systemctl enable --now ppsq
```

Keep it a **single instance** — do not template it or add web-server workers. Put a
TLS-terminating reverse proxy (nginx) in front; set `[app] cookie_secure = true` and
`listen = "127.0.0.1"`.

## Logs

- **Application log** (`app.log`): requests, job lifecycle, PPS API calls, errors. Viewable
  in the admin panel (Logs → Application) with errors in red, or `tail -f app.log`.
- **Operations/audit log** (`worker.log`): one line per message acted on, e.g.
  `job=#12 user=alice@x action=report_release result=ok src='Quarantine' dst='Debugging - Josef' localguid=… guid=… from=… rcpt=… subject=…`.
  This is the "who released/deleted what" audit trail. Grep it: `grep result=FAILED worker.log`.

Both rotate at 5 MB × 5 files. The PPS `x-pps-reqid` appears in error lines for
cross-referencing PPS's own webservices log.

## Background jobs

- View in the admin panel (Jobs) — all users, with status and progress. Failed jobs have a
  **Retry** button (`POST /api/admin/jobs/<id>/retry`).
- Jobs persist in `jobs.db` and survive a restart; a job interrupted mid-run is requeued
  automatically on boot.

## Backup

Back up `config.toml` and `jobs.db`. For a consistent `jobs.db` copy while running (WAL mode):

```bash
sqlite3 jobs.db ".backup '/backup/jobs.db'"
```

`config.toml.bak` holds the previous config after each admin save (one-step rollback).

## Rotating credentials

- **PPS password / Okta secret:** enter the new value in the admin panel (PPS section) — it's
  write-only and applies live (PPS) or after restart (Okta, edit the file). Use the **Test
  connection** button before relying on it.
- **`secret_key`:** edit `config.toml` and restart. This invalidates all sessions (everyone
  re-logs-in).
- **Client TLS cert:** replace the file at `client_cert`; see `certs/README.md`.

## Common issues

| Symptom | Cause / fix |
|---------|-------------|
| `400 No required SSL certificate was sent` | PPS/proxy wants mutual TLS. Set `[pps] client_cert`. |
| Every login fails `mismatching_state` | Cookie/SameSite or a proxy dropping cookies; ensure `cookie_secure`/HTTPS and `redirect_uri` match. |
| `database is locked` | WAL/busy_timeout missing — should not happen; verify no second process opened `jobs.db`. |
| Admin save fails to persist | `config.toml`/its directory not writable by the service user. |
| Folder shows few/zero messages | `default_days_back` window too small, or `list_query` not match-all for this PPS. |
| Messages older than N days missing | Bounded by `default_days_back` — raise it. |