proofpoint-quarantine-portal/docs/architecture.md

Architecture

Overview

                    ┌─────────────────────────── single process ───────────────────────────┐
   browser ──HTTP──▶│  Flask (app.py, waitress)                                              │
                    │    │                                                                    │
                    │    ├─ auth.py ......... Okta OIDC / static; login_required/admin_required
                    │    ├─ ConfigStore ..... immutable snapshots of config.toml (tomlkit)   │
                    │    ├─ PrefStore ....... per-user prefs (jobs.db)                        │
                    │    └─ /api/actions ──▶ JobQueue.enqueue() ──▶ SQLite jobs table         │
                    │                              ▲                      │                    │
                    │                    worker thread (daemon) ◀─────────┘                    │
                    │                              │                                           │
                    │                              └─ pipeline.py ──▶ PPSClient ──HTTPS──▶ PPS │
                    └────────────────────────────────────────────────────────────────────────┘

Everything runs in one process (see AGENTS.md invariant #1). The browser fires an action and gets an immediate 202; the daemon worker thread drains the SQLite queue and calls PPS. Job progress is polled via /api/jobs; it is cosmetic — closing the page does not affect processing, and JobQueue._recover() requeues interrupted jobs after a restart.

Request lifecycle

1. wsgi.py builds ConfigStore, JobQueue, PrefStore, calls create_app, starts the worker, serves.
2. Each request: auth.login_required/admin_required gate, then the view takes one store.snapshot() and reads all config from it.
3. State-changing /api/* requests must carry X-CSRF-Token (checked in before_request).

Configuration hot-reload

config.toml is read once into an immutable Snapshot. The admin panel's PUT /api/admin/config validates a patch, atomically rewrites the file with tomlkit (comments preserved), rebuilds the PPSClient only if a connection key changed, and swaps in a new snapshot. Readers are lock-free (attribute rebind is atomic under the GIL). See docs/configuration.md for which keys apply live vs. need a restart.

Background jobs

• Tables: jobs(id, action, folder, user, extra, status, processed, total, error, run_after, state, …) and job_items(job_id, localguid, guid, folder, subject, sender, recipient).
• The worker claims the lowest pending job whose run_after has passed, snapshots config once, chunks the items (chunk_size), runs one PPS action — or, for Report & Release, one pipeline.run_step per configured step — writes one audit line per message per step to worker.log, and marks the job done/failed.
• Waiting doesn't block the queue. Between two Report & Release steps the job is parked (_defer): progress goes into jobs.state, run_after is set to now + delay, and the worker picks up other jobs until the wait is over. On resume it re-reads the messages' current localguids by guid — cached handles are never reused across a step.
• retry(job_id) requeues a failed job from the top (clears state/run_after).

Data stores

Store	Where	Contents
config.toml	file (tomlkit)	all configuration; 0600; .bak on each write
jobs.db	SQLite (WAL)	jobs, job_items, user_prefs
worker.log	rotating file	per-message audit trail
app.log	rotating file	application log (errors visible in admin panel)

Why single-process

The worker thread and both SQLite connections share process memory, and the config lock is an in-process RLock. Multiple processes would run competing queue workers and uncoordinated config writes. Scale vertically (threads), never horizontally. This is a hard constraint.