AGENTS.md — PPS Quarantine Manager

Read this first. It is the map plus the non-obvious invariants that are expensive to rediscover. Human-readable too, but written so an agent can act correctly without re-deriving the traps.

What this is

A Flask web app for operators to manage a Proofpoint Protection Server (PPS) email quarantine: list quarantined mail, read messages, and run bulk actions (Release, Report & Release, Delete, Move) as asynchronous background jobs. Okta OIDC login, an allowed-users list with roles, and an admin panel that edits the config live.

Run it

python3 -m venv venv && venv/bin/pip install -r requirements.txt
cp config.example.toml config.toml   # then edit: Okta, PPS creds, users, secret_key
venv/bin/python wsgi.py              # NOT app.py — see "App factory" below

Dev without Okta: set [auth] mode = "static" and use [auth.static] credentials. Tests / alternate config: PPSQ_CONFIG=/path/to/config.toml. Plain-http localhost dev also needs PPSQ_ALLOW_INSECURE=1 (allows an http:// PPS base_url).

venv/bin/python -m pytest       # 57 tests, no Okta tenant or real config needed

Module map

File	Role
`wsgi.py`	Composition root. The only place that builds the store/queue/prefs, starts the worker thread, configures logging, and serves via waitress.
`app.py`	`create_app(store, queue, prefs)` — Flask routes. No import-time side effects.
`config_store.py`	`ConfigStore`: tomlkit load/validate/atomic-write, immutable snapshots, PPSClient rebuild. The heart of the hot-reload design.
`auth.py`	Okta OIDC + static dev mode; `login_required`/`admin_required`; `is_allowed`/`safe_next` (pure); CSRF.
`pipeline.py`	Data-driven Report & Release step registry (release → move → delete). No Flask.
`worker.py`	`JobQueue`: SQLite-backed background job thread; ops audit log.
`prefs.py`	`PrefStore`: per-user preferences in jobs.db (key-value).
`admin.py`	`/admin` page + `/api/admin/*` (config, pps-test, jobs, logs).
`logs.py`	Safe backwards log tailing + server-side level classification.
`pps_client.py`	Thin PPS REST client (`search`/`get_raw`/`act`). Cheap to rebuild.
`static/`, `templates/`	Vanilla-JS frontend; `app.js` (quarantine), `admin.js` (panel).

Invariants — break these and things fail silently

Single process only. The worker thread and both SQLite connections live in-process. Running multiple web workers (--workers N, gunicorn forks) starts competing queue threads on one DB and an in-process config lock that doesn't coordinate file writes. wsgi.py serves single-process on purpose. Do not "scale" it horizontally.
App factory, no import side effects. import app must never read config or start a thread — otherwise every test touches the real config.toml and spawns a worker. Construction happens only in wsgi.py (prod) and create_app(...) (tests).
localguid is folder-local and dies on move; guid is stable. A message's localguid is only valid in its current folder and becomes invalid the moment the message is moved/deleted. The guid is global and stable. To act on a message after a relocating step, re-find its current localguid by matching guid (pipeline.refind_localguids).
Report & Release: release FIRST, with NO deletedfolder. release must run before move, and must not pass deletedfolder, so the message stays in place and its localguid survives for the move. Passing deletedfolder on release relocates the message and invalidates every localguid — the exact bug the guid-refind fallback exists to paper over. See the comment in pipeline._release.
Snapshot per job / per request. Config is an immutable Snapshot; apply() builds a new one and rebinds one attribute (atomic under the GIL). A worker reads one snapshot at job start and threads it through, so a mid-job admin edit affects the next job, never a job in flight. Never re-read store.snapshot() inside a running job.
WAL + busy_timeout are mandatory. Two connections (queue + prefs) share jobs.db. Without PRAGMA journal_mode=WAL and busy_timeout, a prefs write concurrent with the per-chunk job-progress write throws database is locked. Set in JobQueue._init_db and every prefs connection.
PPS search: 1000-row cap, no offset paging. Each search returns at most 1000 rows, newest first. The client pages the whole folder backwards by date using the oldest date seen as a before cursor. page.oldest MUST be computed over the raw batch, before the acted-on filter (app.py api_messages), or paging terminates early and silently truncates the folder. >1000 messages sharing one exact timestamp is unpageable (accepted edge case). Coverage is also bounded by default_days_back.
list_query = "from=*" is a workaround, not a preference. The PPS API rejects a folder-only search; it requires a from/rcpt/subject filter. from=* means "everything". If a deployment doesn't treat it as match-all, change it (e.g. rcpt=@yourdomain.com).
Secrets are write-only. pps.password, okta.client_secret, auth.static.password, app.secret_key are never returned by any endpoint (ConfigStore.redacted() emits a <name>_set boolean). A blank value on save means "leave unchanged". Do not add an endpoint that echoes these back.
Log source is an enum, never a path. /api/admin/logs?source=ops|app resolves to a path server-side. A ?path= parameter would be arbitrary file read — never add one.

Config zones

config.toml is one file with two zones. The admin panel writes only the admin zone and round-trips the file with tomlkit (comments/formatting preserved).

Server-only (edit the file, restart): [okta], [auth] mode + [auth.static], [app] secret_key/listen/port/db_path/worker_log/app_log/cookie_secure.
Admin-editable (panel, mostly live): [pps], [quarantine] incl. [quarantine.report_release], [auth] users + denied_message, [app] log_level.

ADMIN_EDITABLE in config_store.py is the explicit allowlist. Adding an editable key means adding it there. docs/configuration.md is the full reference.

Conventions

from __future__ import annotations at the top of every module.
Loggers are namespaced pps.<area> (pps.app, pps.worker, pps.pipeline, …).
%-style lazy log formatting.
Private helpers prefixed _; routes named api_* / admin_*.
Bare except Exception only where justified, always with # noqa: BLE001 + a reason.
PPS failures raise PPSError (carries .status/.body/.reqid), funneled through _pps_error_response → 502.
Schema evolution: PRAGMA table_info + ALTER TABLE ADD COLUMN (see worker._init_db).

Extending

New Report & Release action: add a @step("name") function in pipeline.py and put "name" in ALLOWED_STEPS (canonical order). If it relocates messages, teach pipeline_destination. The admin checkboxes render from the schema — no frontend change.
New config key: add to config.example.toml (with a comment), read it from the snapshot, and if admin-editable add it to ADMIN_EDITABLE + validation in config_store.py.
New per-user pref: add a PrefSpec to PREF_SPECS in prefs.py (default path + coercion + validator). No migration needed (key-value store).

AGENTS.md 7.4 KB Geçmiş Ham