implementing s3 upload of older backups

backup-server/.htaccess

@@ -1,13 +1,13 @@
 Options -Indexes
 
 <IfModule mod_authz_core.c>
-    <FilesMatch "^(config\.php|.*\.json|.*\.zip)$">
+    <FilesMatch "^(config\.php|lib\.php|s3\.php|.*\.json|.*\.zip|.*\.log)$">
         Require all denied
     </FilesMatch>
 </IfModule>
 
 <IfModule !mod_authz_core.c>
-    <FilesMatch "^(config\.php|.*\.json|.*\.zip)$">
+    <FilesMatch "^(config\.php|lib\.php|s3\.php|.*\.json|.*\.zip|.*\.log)$">
         Order allow,deny
         Deny from all
     </FilesMatch>

backup-server/README.md

@@ -37,6 +37,39 @@ Retention can be changed in `config.php` with `BACKUP_SERVER_RETENTION` and in t
 
 Retention is applied after every successful upload and after retention changes in the management UI.
 
+With S3 enabled (see below) this value controls only the **local** copies. The minimum is `1`, so at least the newest backup always stays on local disk.
+
+## S3 archive (optional)
+
+The server can additionally archive every backup to an S3-compatible object storage (e.g. Hetzner Object Storage, MinIO). Local disk then acts as a small hot cache with the newest backups, while the bucket holds the complete archive.
+
+Enable it in `config.php`:
+
+```php
+define("BACKUP_SERVER_S3_ENABLED", true);
+define("BACKUP_SERVER_S3_ENDPOINT", "https://fsn1.your-objectstorage.com");
+define("BACKUP_SERVER_S3_REGION", "fsn1");
+define("BACKUP_SERVER_S3_BUCKET", "my-backup-bucket");
+define("BACKUP_SERVER_S3_PREFIX", "psa-backups");
+define("BACKUP_SERVER_S3_ACCESS_KEY", "...");
+define("BACKUP_SERVER_S3_SECRET_KEY", "...");
+```
+
+By default the server uses **virtual-hosted-style** addressing (`https://<bucket>.<endpoint>/<key>`), which Hetzner and most S3-compatible providers expect. If your provider requires **path-style** (`https://<endpoint>/<bucket>/<key>`), set `BACKUP_SERVER_S3_PATH_STYLE` to `true`.
+
+Behavior:
+
+- Every received backup is stored locally first and then uploaded to S3 (AWS Signature V4, no SDK required). Objects are stored as `<prefix>/<instance>/<filename>`.
+- A local copy is only deleted after it fell out of the local retention window **and** its S3 copy is confirmed. While S3 is unreachable, local copies accumulate beyond the retention setting instead of being deleted.
+- S3 failures never fail a client upload. They are logged to `backups/s3.log` and shown in the management UI; failed uploads are retried on the next upload for that instance or via the "Retry S3 uploads now" button.
+- S3 has its own count-based retention ("S3 backups retained per instance", default `BACKUP_SERVER_S3_RETENTION` = 365). Backups that age out of S3 are deleted from the bucket.
+- S3-only backups remain listed in the management UI and are downloaded through the server, so the bucket can (and should) stay private.
+- When enabling S3 on an installation with existing backups, use "Retry S3 uploads now" once to backfill the archive; otherwise the first client upload per instance flushes the whole backlog within that request.
+
+Limitations: uploads to S3 hold the whole file in memory, so a single backup must fit into PHP's `memory_limit`. Downloads from S3 are streamed and have no such limit. Concurrent uploads for the same instance may race on `index.json` (pre-existing limitation).
+
+Troubleshooting: S3 errors are written to `backups/s3.log` with the provider's error code and, on a rejected request, a diagnostic showing the HTTP status chain, any redirect target, and the request id. `AccessDenied` or a redirect in the status chain usually means the addressing style is wrong — try flipping `BACKUP_SERVER_S3_PATH_STYLE`. `SignatureDoesNotMatch` usually means a wrong region or secret key. The server never follows S3 redirects, so a `3xx` in the log is reported rather than silently retried against the wrong host.
+
 ## Storage
 
 Backups are stored under:
@@ -57,4 +90,4 @@ Allowed instances and UI retention settings are stored in:
 backups/settings.json
 ```
 
-With the included `.htaccess`, ZIP and JSON files are not directly readable through Apache. Downloads should use the authenticated management UI.
+With the included `.htaccess`, ZIP, JSON, and log files as well as the internal includes (`lib.php`, `s3.php`) are not directly readable through Apache. Downloads should use the authenticated management UI.

backup-server/config.sample.php

@@ -10,3 +10,21 @@ define("BACKUP_SERVER_RETENTION", 30);
 define("BACKUP_SERVER_BACKUP_DIR", __DIR__ . "/backups/");
 define("BACKUP_SERVER_INDEX_FILE", BACKUP_SERVER_BACKUP_DIR . "index.json");
 define("BACKUP_SERVER_SETTINGS_FILE", BACKUP_SERVER_BACKUP_DIR . "settings.json");
+
+// S3-compatible object storage (optional). Leave disabled for local-only behavior.
+// When enabled, every upload is archived to S3 and only the newest local copies
+// are kept on disk. All five connection values must be filled in.
+define("BACKUP_SERVER_S3_ENABLED", false);
+define("BACKUP_SERVER_S3_ENDPOINT", "https://fsn1.your-objectstorage.com"); // e.g. Hetzner
+define("BACKUP_SERVER_S3_REGION", "fsn1");
+define("BACKUP_SERVER_S3_BUCKET", "");
+define("BACKUP_SERVER_S3_PREFIX", "psa-backups"); // key prefix inside the bucket, may be ""
+define("BACKUP_SERVER_S3_ACCESS_KEY", "");
+define("BACKUP_SERVER_S3_SECRET_KEY", "");
+// Addressing style. false = virtual-hosted (https://<bucket>.<endpoint>/<key>),
+// which Hetzner and most providers expect. Set true only if your provider
+// requires path-style (https://<endpoint>/<bucket>/<key>).
+define("BACKUP_SERVER_S3_PATH_STYLE", false);
+define("BACKUP_SERVER_S3_TIMEOUT", 120);   // seconds per HTTP request
+define("BACKUP_SERVER_S3_RETENTION", 365); // default S3 backups kept per instance
+define("BACKUP_SERVER_LOG_FILE", BACKUP_SERVER_BACKUP_DIR . "s3.log");

backup-server/lib.php

@@ -0,0 +1,425 @@
+<?php
+
+declare(strict_types=1);
+
+// Shared helpers for the backup server. Included by upload.php and manage.php.
+
+$backupServerConfigFile = __DIR__ . "/config.php";
+if (is_file($backupServerConfigFile)) {
+    require_once $backupServerConfigFile;
+}
+
+if (!defined("BACKUP_SERVER_RETENTION")) {
+    define("BACKUP_SERVER_RETENTION", 30);
+}
+if (!defined("BACKUP_SERVER_BACKUP_DIR")) {
+    define("BACKUP_SERVER_BACKUP_DIR", __DIR__ . "/backups/");
+}
+if (!defined("BACKUP_SERVER_INDEX_FILE")) {
+    define("BACKUP_SERVER_INDEX_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/index.json");
+}
+if (!defined("BACKUP_SERVER_SETTINGS_FILE")) {
+    define("BACKUP_SERVER_SETTINGS_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/settings.json");
+}
+if (!defined("BACKUP_SERVER_S3_ENABLED")) {
+    define("BACKUP_SERVER_S3_ENABLED", false);
+}
+if (!defined("BACKUP_SERVER_S3_ENDPOINT")) {
+    define("BACKUP_SERVER_S3_ENDPOINT", "");
+}
+if (!defined("BACKUP_SERVER_S3_REGION")) {
+    define("BACKUP_SERVER_S3_REGION", "");
+}
+if (!defined("BACKUP_SERVER_S3_BUCKET")) {
+    define("BACKUP_SERVER_S3_BUCKET", "");
+}
+if (!defined("BACKUP_SERVER_S3_PREFIX")) {
+    define("BACKUP_SERVER_S3_PREFIX", "");
+}
+if (!defined("BACKUP_SERVER_S3_ACCESS_KEY")) {
+    define("BACKUP_SERVER_S3_ACCESS_KEY", "");
+}
+if (!defined("BACKUP_SERVER_S3_SECRET_KEY")) {
+    define("BACKUP_SERVER_S3_SECRET_KEY", "");
+}
+if (!defined("BACKUP_SERVER_S3_PATH_STYLE")) {
+    define("BACKUP_SERVER_S3_PATH_STYLE", false);
+}
+if (!defined("BACKUP_SERVER_S3_TIMEOUT")) {
+    define("BACKUP_SERVER_S3_TIMEOUT", 120);
+}
+if (!defined("BACKUP_SERVER_S3_RETENTION")) {
+    define("BACKUP_SERVER_S3_RETENTION", 365);
+}
+if (!defined("BACKUP_SERVER_LOG_FILE")) {
+    define("BACKUP_SERVER_LOG_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/s3.log");
+}
+
+require_once __DIR__ . "/s3.php";
+
+function backupServerEnsureDirectory(string $dir): void
+{
+    if (!is_dir($dir) && !mkdir($dir, 02775, true) && !is_dir($dir)) {
+        throw new RuntimeException("Directory cannot be created: " . $dir);
+    }
+
+    @chmod($dir, 02775);
+}
+
+function backupServerReadJsonFile(string $file): array
+{
+    if (!is_file($file)) {
+        return [];
+    }
+
+    $decoded = json_decode((string) file_get_contents($file), true);
+    if (!is_array($decoded)) {
+        throw new RuntimeException("JSON file is invalid: " . basename($file));
+    }
+
+    return $decoded;
+}
+
+function backupServerWriteJsonFile(string $file, array $data): void
+{
+    backupServerEnsureDirectory(dirname($file));
+
+    $json = json_encode(
+        $data,
+        JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE,
+    );
+    if ($json === false) {
+        throw new RuntimeException("JSON cannot be encoded.");
+    }
+
+    $tmpFile = $file . ".tmp";
+    if (file_put_contents($tmpFile, $json . PHP_EOL, LOCK_EX) === false) {
+        throw new RuntimeException("JSON cannot be written.");
+    }
+
+    @chmod($tmpFile, 0664);
+    if (!rename($tmpFile, $file)) {
+        @unlink($tmpFile);
+        throw new RuntimeException("JSON cannot be saved.");
+    }
+
+    @chmod($file, 0664);
+}
+
+function backupServerReadIndex(): array
+{
+    $index = backupServerReadJsonFile((string) BACKUP_SERVER_INDEX_FILE);
+    $backups = isset($index["backups"]) && is_array($index["backups"])
+        ? $index["backups"]
+        : [];
+
+    return ["backups" => array_values($backups)];
+}
+
+function backupServerWriteIndex(array $backups): void
+{
+    backupServerWriteJsonFile((string) BACKUP_SERVER_INDEX_FILE, [
+        "backups" => array_values($backups),
+    ]);
+}
+
+function backupServerValidateInstance(string $instance): string
+{
+    $instance = trim($instance);
+    if (
+        $instance === "" ||
+        strlen($instance) > 120 ||
+        preg_match('/^[A-Za-z0-9][A-Za-z0-9._-]*$/', $instance) !== 1
+    ) {
+        throw new RuntimeException("Invalid instance identifier.");
+    }
+
+    return $instance;
+}
+
+function backupServerInstanceDir(string $instance): string
+{
+    return rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . DIRECTORY_SEPARATOR . $instance;
+}
+
+function backupServerBackupPath(string $instance, string $filename): string
+{
+    return backupServerInstanceDir($instance) . DIRECTORY_SEPARATOR . $filename;
+}
+
+function backupServerGetSettings(): array
+{
+    $settings = backupServerReadJsonFile((string) BACKUP_SERVER_SETTINGS_FILE);
+    $retention = isset($settings["retention"])
+        ? max(1, (int) $settings["retention"])
+        : max(1, (int) BACKUP_SERVER_RETENTION);
+    $s3Retention = isset($settings["s3_retention"])
+        ? max(1, (int) $settings["s3_retention"])
+        : max(1, (int) BACKUP_SERVER_S3_RETENTION);
+    $instances =
+        isset($settings["instances"]) && is_array($settings["instances"])
+            ? $settings["instances"]
+            : [];
+    $allowedInstances = [];
+
+    foreach ($instances as $instance) {
+        try {
+            $allowedInstances[] = backupServerValidateInstance((string) $instance);
+        } catch (Throwable $exception) {
+            continue;
+        }
+    }
+    $allowedInstances = array_values(array_unique($allowedInstances));
+    sort($allowedInstances);
+
+    return [
+        "retention" => $retention,
+        "s3_retention" => $s3Retention,
+        "instances" => $allowedInstances,
+    ];
+}
+
+function backupServerWriteSettings(array $settings): void
+{
+    $instances =
+        isset($settings["instances"]) && is_array($settings["instances"])
+            ? $settings["instances"]
+            : backupServerGetSettings()["instances"];
+    $allowedInstances = [];
+
+    foreach ($instances as $instance) {
+        $allowedInstances[] = backupServerValidateInstance((string) $instance);
+    }
+    $allowedInstances = array_values(array_unique($allowedInstances));
+    sort($allowedInstances);
+
+    backupServerWriteJsonFile((string) BACKUP_SERVER_SETTINGS_FILE, [
+        "retention" => max(1, (int) ($settings["retention"] ?? BACKUP_SERVER_RETENTION)),
+        "s3_retention" => max(1, (int) ($settings["s3_retention"] ?? BACKUP_SERVER_S3_RETENTION)),
+        "instances" => $allowedInstances,
+    ]);
+}
+
+function backupServerLog(string $message, array $context = []): void
+{
+    $line = date(DATE_ATOM) . " " . $message;
+    $encoded = @json_encode($context, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
+    if (is_string($encoded) && $encoded !== "[]") {
+        $line .= " " . $encoded;
+    }
+
+    @file_put_contents((string) BACKUP_SERVER_LOG_FILE, $line . PHP_EOL, FILE_APPEND | LOCK_EX);
+}
+
+function backupServerUpdateIndexRecord(string $instance, string $filename, callable $update): void
+{
+    $index = backupServerReadIndex();
+    foreach ($index["backups"] as $position => $backup) {
+        if (
+            is_array($backup) &&
+            ($backup["instance"] ?? "") === $instance &&
+            ($backup["filename"] ?? "") === $filename
+        ) {
+            $index["backups"][$position] = $update($backup);
+        }
+    }
+
+    backupServerWriteIndex($index["backups"]);
+}
+
+function backupServerIndexInstances(): array
+{
+    $instances = [];
+    foreach (backupServerReadIndex()["backups"] as $backup) {
+        if (is_array($backup)) {
+            $instance = (string) ($backup["instance"] ?? "");
+            if ($instance !== "") {
+                $instances[$instance] = true;
+            }
+        }
+    }
+
+    return array_keys($instances);
+}
+
+// Uploads every local backup of the instance that is not yet confirmed in S3,
+// oldest first. Serves both the immediate upload after receiving a backup and
+// the opportunistic retry of earlier failures. Stops at the first failure
+// because the endpoint is then most likely unreachable.
+function backupServerSyncInstanceS3(string $instance): array
+{
+    $result = ["uploaded" => 0, "pending" => 0, "error" => null];
+    if (!backupS3Enabled()) {
+        return $result;
+    }
+
+    $pending = [];
+    foreach (backupServerReadIndex()["backups"] as $backup) {
+        if (!is_array($backup) || ($backup["instance"] ?? "") !== $instance) {
+            continue;
+        }
+        if (!empty($backup["s3_uploaded_at"])) {
+            continue;
+        }
+        $filename = basename((string) ($backup["filename"] ?? ""));
+        if ($filename === "" || !is_file(backupServerBackupPath($instance, $filename))) {
+            continue;
+        }
+        $backup["filename"] = $filename;
+        $pending[] = $backup;
+    }
+
+    usort($pending, function ($left, $right) {
+        return strcmp((string) ($left["uploaded_at"] ?? ""), (string) ($right["uploaded_at"] ?? ""));
+    });
+
+    foreach ($pending as $position => $backup) {
+        $filename = (string) $backup["filename"];
+        $key = (string) ($backup["s3_key"] ?? "");
+        if ($key === "") {
+            $key = backupS3ObjectKey($instance, $filename);
+        }
+
+        try {
+            backupS3PutFile(backupServerBackupPath($instance, $filename), $key);
+        } catch (Throwable $exception) {
+            $result["pending"] = count($pending) - $position;
+            $result["error"] = $exception->getMessage();
+            backupServerUpdateIndexRecord($instance, $filename, function (array $record) use ($key, $exception) {
+                $record["s3_key"] = $key;
+                $record["s3_last_error"] = $exception->getMessage();
+                $record["s3_last_attempt_at"] = date(DATE_ATOM);
+                return $record;
+            });
+            backupServerLog("S3 upload failed", [
+                "instance" => $instance,
+                "filename" => $filename,
+                "key" => $key,
+                "error" => $exception->getMessage(),
+            ]);
+            return $result;
+        }
+
+        backupServerUpdateIndexRecord($instance, $filename, function (array $record) use ($key) {
+            $record["s3_key"] = $key;
+            $record["s3_uploaded_at"] = date(DATE_ATOM);
+            unset($record["s3_last_error"], $record["s3_last_attempt_at"], $record["s3_expired"]);
+            return $record;
+        });
+        $result["uploaded"]++;
+    }
+
+    return $result;
+}
+
+// Applies both retention tiers for one instance. S3 keeps the newest
+// s3_retention archived backups; local keeps the newest retention copies but
+// never deletes a file whose S3 upload is still pending.
+function backupServerApplyRetention(string $instance): void
+{
+    $index = backupServerReadIndex();
+    $settings = backupServerGetSettings();
+    $s3Enabled = backupS3Enabled();
+    $instanceBackups = [];
+    $otherBackups = [];
+
+    foreach ($index["backups"] as $backup) {
+        if (!is_array($backup)) {
+            continue;
+        }
+        if (($backup["instance"] ?? "") === $instance) {
+            $instanceBackups[] = $backup;
+        } else {
+            $otherBackups[] = $backup;
+        }
+    }
+
+    usort($instanceBackups, function ($left, $right) {
+        return strcmp((string) ($right["uploaded_at"] ?? ""), (string) ($left["uploaded_at"] ?? ""));
+    });
+
+    if ($s3Enabled) {
+        $archivedSeen = 0;
+        foreach ($instanceBackups as $position => $backup) {
+            if (empty($backup["s3_uploaded_at"])) {
+                continue;
+            }
+            $archivedSeen++;
+            if ($archivedSeen <= $settings["s3_retention"]) {
+                continue;
+            }
+
+            $filename = basename((string) ($backup["filename"] ?? ""));
+            $key = (string) ($backup["s3_key"] ?? "");
+            if ($key === "" && $filename !== "") {
+                $key = backupS3ObjectKey($instance, $filename);
+            }
+
+            try {
+                if ($key !== "") {
+                    backupS3DeleteObject($key);
+                }
+            } catch (Throwable $exception) {
+                backupServerLog("S3 retention delete failed", [
+                    "instance" => $instance,
+                    "filename" => $filename,
+                    "key" => $key,
+                    "error" => $exception->getMessage(),
+                ]);
+                continue;
+            }
+
+            unset($backup["s3_uploaded_at"], $backup["s3_key"]);
+            $backup["s3_expired"] = true;
+            $instanceBackups[$position] = $backup;
+        }
+    }
+
+    $localSeen = 0;
+    $kept = [];
+    foreach ($instanceBackups as $backup) {
+        $filename = basename((string) ($backup["filename"] ?? ""));
+        $path = $filename !== "" ? backupServerBackupPath($instance, $filename) : "";
+        $localExists = $path !== "" && is_file($path);
+        $inS3 = !empty($backup["s3_uploaded_at"]);
+
+        if (!$localExists) {
+            if ($inS3) {
+                $kept[] = $backup;
+            }
+            // Present in neither store: drop the orphaned record.
+            continue;
+        }
+
+        $localSeen++;
+        if ($localSeen <= $settings["retention"]) {
+            $kept[] = $backup;
+            continue;
+        }
+
+        if ($inS3) {
+            @unlink($path);
+            $backup["local_deleted_at"] = date(DATE_ATOM);
+            $kept[] = $backup;
+            continue;
+        }
+
+        if ($s3Enabled && empty($backup["s3_expired"])) {
+            // The only copy lives locally until the S3 upload succeeds.
+            $kept[] = $backup;
+            continue;
+        }
+
+        // S3 disabled (legacy behavior) or the backup already aged out of S3.
+        @unlink($path);
+    }
+
+    backupServerWriteIndex(array_merge($otherBackups, $kept));
+}
+
+function backupServerApplyRetentionAll(): void
+{
+    foreach (backupServerIndexInstances() as $instance) {
+        backupServerApplyRetention($instance);
+    }
+}

backup-server/manage.php

@@ -2,25 +2,7 @@
 
 declare(strict_types=1);
 
-$baseDir = __DIR__;
-$configFile = $baseDir . "/config.php";
-
-if (is_file($configFile)) {
-    require_once $configFile;
-}
-
-if (!defined("BACKUP_SERVER_RETENTION")) {
-    define("BACKUP_SERVER_RETENTION", 30);
-}
-if (!defined("BACKUP_SERVER_BACKUP_DIR")) {
-    define("BACKUP_SERVER_BACKUP_DIR", $baseDir . "/backups/");
-}
-if (!defined("BACKUP_SERVER_INDEX_FILE")) {
-    define("BACKUP_SERVER_INDEX_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/index.json");
-}
-if (!defined("BACKUP_SERVER_SETTINGS_FILE")) {
-    define("BACKUP_SERVER_SETTINGS_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/settings.json");
-}
+require_once __DIR__ . "/lib.php";
 
 if (session_status() === PHP_SESSION_NONE) {
     ini_set("session.use_strict_mode", "1");
@@ -75,139 +57,6 @@ function backupManageCsrfIsValid(string $token): bool
         hash_equals($_SESSION["backup_server_csrf_token"], $token);
 }
 
-function backupManageEnsureDirectory(string $dir): void
-{
-    if (!is_dir($dir) && !mkdir($dir, 02775, true) && !is_dir($dir)) {
-        throw new RuntimeException("Directory cannot be created: " . $dir);
-    }
-
-    @chmod($dir, 02775);
-}
-
-function backupManageReadJsonFile(string $file): array
-{
-    if (!is_file($file)) {
-        return [];
-    }
-
-    $decoded = json_decode((string) file_get_contents($file), true);
-    if (!is_array($decoded)) {
-        throw new RuntimeException("JSON file is invalid: " . basename($file));
-    }
-
-    return $decoded;
-}
-
-function backupManageWriteJsonFile(string $file, array $data): void
-{
-    backupManageEnsureDirectory(dirname($file));
-
-    $json = json_encode(
-        $data,
-        JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE,
-    );
-    if ($json === false) {
-        throw new RuntimeException("JSON cannot be encoded.");
-    }
-
-    $tmpFile = $file . ".tmp";
-    if (file_put_contents($tmpFile, $json . PHP_EOL, LOCK_EX) === false) {
-        throw new RuntimeException("JSON cannot be written.");
-    }
-
-    @chmod($tmpFile, 0664);
-    if (!rename($tmpFile, $file)) {
-        @unlink($tmpFile);
-        throw new RuntimeException("JSON cannot be saved.");
-    }
-
-    @chmod($file, 0664);
-}
-
-function backupManageReadIndex(): array
-{
-    $index = backupManageReadJsonFile((string) BACKUP_SERVER_INDEX_FILE);
-    $backups = isset($index["backups"]) && is_array($index["backups"])
-        ? $index["backups"]
-        : [];
-
-    return ["backups" => array_values($backups)];
-}
-
-function backupManageWriteIndex(array $backups): void
-{
-    backupManageWriteJsonFile((string) BACKUP_SERVER_INDEX_FILE, [
-        "backups" => array_values($backups),
-    ]);
-}
-
-function backupManageReadSettings(): array
-{
-    $settings = backupManageReadJsonFile((string) BACKUP_SERVER_SETTINGS_FILE);
-    $retention = isset($settings["retention"])
-        ? max(1, (int) $settings["retention"])
-        : max(1, (int) BACKUP_SERVER_RETENTION);
-    $instances =
-        isset($settings["instances"]) && is_array($settings["instances"])
-            ? $settings["instances"]
-            : [];
-    $allowedInstances = [];
-
-    foreach ($instances as $instance) {
-        try {
-            $allowedInstances[] = backupManageValidateInstance((string) $instance);
-        } catch (Throwable $exception) {
-            continue;
-        }
-    }
-    $allowedInstances = array_values(array_unique($allowedInstances));
-    sort($allowedInstances);
-
-    return [
-        "retention" => $retention,
-        "instances" => $allowedInstances,
-    ];
-}
-
-function backupManageWriteSettings(array $settings): void
-{
-    $instances =
-        isset($settings["instances"]) && is_array($settings["instances"])
-            ? $settings["instances"]
-            : backupManageReadSettings()["instances"];
-    $allowedInstances = [];
-
-    foreach ($instances as $instance) {
-        $allowedInstances[] = backupManageValidateInstance((string) $instance);
-    }
-    $allowedInstances = array_values(array_unique($allowedInstances));
-    sort($allowedInstances);
-
-    backupManageWriteJsonFile((string) BACKUP_SERVER_SETTINGS_FILE, [
-        "retention" => max(1, (int) ($settings["retention"] ?? BACKUP_SERVER_RETENTION)),
-        "instances" => $allowedInstances,
-    ]);
-}
-
-function backupManageInstanceDir(string $instance): string
-{
-    return rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . DIRECTORY_SEPARATOR . $instance;
-}
-
-function backupManageValidateInstance(string $instance): string
-{
-    $instance = trim($instance);
-    if (
-        $instance === "" ||
-        strlen($instance) > 120 ||
-        preg_match('/^[A-Za-z0-9][A-Za-z0-9._-]*$/', $instance) !== 1
-    ) {
-        throw new RuntimeException("Invalid instance.");
-    }
-
-    return $instance;
-}
-
 function backupManageValidateFilename(string $filename): string
 {
     $filename = basename(trim($filename));
@@ -234,7 +83,7 @@ function backupManageFormatBytes(int $bytes): string
 
 function backupManageFindBackup(string $instance, string $filename): ?array
 {
-    foreach (backupManageReadIndex()["backups"] as $backup) {
+    foreach (backupServerReadIndex()["backups"] as $backup) {
         if (!is_array($backup)) {
             continue;
         }
@@ -246,11 +95,6 @@ function backupManageFindBackup(string $instance, string $filename): ?array
     return null;
 }
 
-function backupManageBackupPath(string $instance, string $filename): string
-{
-    return backupManageInstanceDir($instance) . DIRECTORY_SEPARATOR . $filename;
-}
-
 function backupManageSendDownload(string $instance, string $filename): void
 {
     $backup = backupManageFindBackup($instance, $filename);
@@ -258,29 +102,46 @@ function backupManageSendDownload(string $instance, string $filename): void
         throw new RuntimeException("Backup not found.");
     }
 
-    $path = backupManageBackupPath($instance, $filename);
-    $size = filesize($path);
-    $handle = fopen($path, "rb");
-    if ($size === false || $handle === false) {
-        throw new RuntimeException("Backup cannot be opened.");
+    $path = backupServerBackupPath($instance, $filename);
+    if (is_file($path)) {
+        $size = filesize($path);
+        $handle = fopen($path, "rb");
+        if ($size === false || $handle === false) {
+            throw new RuntimeException("Backup cannot be opened.");
+        }
+
+        header("Content-Type: application/zip");
+        header("Content-Disposition: attachment; filename=\"" . addcslashes($instance . "-" . $filename, "\"\\") . "\"");
+        header("Content-Length: " . (string) $size);
+        header("Cache-Control: private, no-store");
+        header("X-Content-Type-Options: nosniff");
+
+        fpassthru($handle);
+        fclose($handle);
+        exit;
     }
 
-    header("Content-Type: application/zip");
-    header("Content-Disposition: attachment; filename=\"" . addcslashes($instance . "-" . $filename, "\"\\") . "\"");
-    header("Content-Length: " . (string) $size);
-    header("Cache-Control: private, no-store");
-    header("X-Content-Type-Options: nosniff");
+    if (!empty($backup["s3_uploaded_at"])) {
+        if (!backupS3Enabled()) {
+            throw new RuntimeException("Backup is stored in S3, but S3 is not configured. See config.php.");
+        }
 
-    fpassthru($handle);
-    fclose($handle);
-    exit;
+        $key = (string) ($backup["s3_key"] ?? "");
+        if ($key === "") {
+            $key = backupS3ObjectKey($instance, $filename);
+        }
+
+        backupS3SendObjectToOutput($key, $instance . "-" . $filename, (int) ($backup["size"] ?? 0));
+    }
+
+    throw new RuntimeException("Backup not found in any store.");
 }
 
 function backupManageDeleteBackup(string $instance, string $filename): void
 {
-    $index = backupManageReadIndex();
+    $index = backupServerReadIndex();
     $kept = [];
-    $found = false;
+    $found = null;
 
     foreach ($index["backups"] as $backup) {
         if (
@@ -288,97 +149,56 @@ function backupManageDeleteBackup(string $instance, string $filename): void
             ($backup["instance"] ?? "") === $instance &&
             ($backup["filename"] ?? "") === $filename
         ) {
-            $found = true;
+            $found = $backup;
             continue;
         }
         $kept[] = $backup;
     }
 
-    if (!$found) {
+    if ($found === null) {
         throw new RuntimeException("Backup not found.");
     }
 
-    $path = backupManageBackupPath($instance, $filename);
-    if (is_file($path)) {
-        unlink($path);
-    }
-
-    backupManageWriteIndex($kept);
-}
-
-function backupManageApplyRetentionForInstance(string $instance): void
-{
-    $index = backupManageReadIndex();
-    $retention = backupManageReadSettings()["retention"];
-    $instanceBackups = [];
-    $otherBackups = [];
-
-    foreach ($index["backups"] as $backup) {
-        if (!is_array($backup)) {
-            continue;
-        }
-        if (($backup["instance"] ?? "") === $instance) {
-            $instanceBackups[] = $backup;
-        } else {
-            $otherBackups[] = $backup;
+    // Delete the S3 object first: if that fails, nothing is changed, so no
+    // object is ever stranded in the bucket without an index record.
+    if (!empty($found["s3_uploaded_at"])) {
+        if (!backupS3Enabled()) {
+            throw new RuntimeException("Backup has an S3 copy, but S3 is not configured. See config.php.");
         }
-    }
-
-    usort($instanceBackups, function ($left, $right) {
-        return strcmp((string) ($right["uploaded_at"] ?? ""), (string) ($left["uploaded_at"] ?? ""));
-    });
 
-    $keep = array_slice($instanceBackups, 0, $retention);
-    $remove = array_slice($instanceBackups, $retention);
-
-    foreach ($remove as $backup) {
-        $filename = basename((string) ($backup["filename"] ?? ""));
-        if ($filename !== "") {
-            $path = backupManageBackupPath($instance, $filename);
-            if (is_file($path)) {
-                @unlink($path);
-            }
+        $key = (string) ($found["s3_key"] ?? "");
+        if ($key === "") {
+            $key = backupS3ObjectKey($instance, $filename);
         }
+        backupS3DeleteObject($key);
     }
 
-    backupManageWriteIndex(array_merge($otherBackups, $keep));
-}
-
-function backupManageApplyRetentionAll(): void
-{
-    $instances = [];
-    foreach (backupManageReadIndex()["backups"] as $backup) {
-        if (is_array($backup)) {
-            $instance = (string) ($backup["instance"] ?? "");
-            if ($instance !== "") {
-                $instances[$instance] = true;
-            }
-        }
+    $path = backupServerBackupPath($instance, $filename);
+    if (is_file($path)) {
+        unlink($path);
     }
 
-    foreach (array_keys($instances) as $instance) {
-        backupManageApplyRetentionForInstance($instance);
-    }
+    backupServerWriteIndex($kept);
 }
 
 function backupManageAddInstance(string $instance): void
 {
-    $instance = backupManageValidateInstance($instance);
-    $settings = backupManageReadSettings();
+    $instance = backupServerValidateInstance($instance);
+    $settings = backupServerGetSettings();
     $settings["instances"][] = $instance;
-    backupManageWriteSettings($settings);
+    backupServerWriteSettings($settings);
 }
 
 function backupManageRemoveInstance(string $instance): void
 {
-    $instance = backupManageValidateInstance($instance);
-    $settings = backupManageReadSettings();
+    $instance = backupServerValidateInstance($instance);
+    $settings = backupServerGetSettings();
     $settings["instances"] = array_values(
         array_filter($settings["instances"], function ($existing) use ($instance) {
             return $existing !== $instance;
         }),
     );
-    backupManageWriteSettings($settings);
+    backupServerWriteSettings($settings);
 }
 
 function backupManageGroupBackupsByInstance(array $backups): array
@@ -390,11 +210,15 @@ function backupManageGroupBackupsByInstance(array $backups): array
         }
         $instance = (string) ($backup["instance"] ?? "");
         $filename = basename((string) ($backup["filename"] ?? ""));
-        if ($instance === "" || $filename === "" || !is_file(backupManageBackupPath($instance, $filename))) {
+        if ($instance === "" || $filename === "") {
             continue;
         }
         $backup["filename"] = $filename;
-        $backup["size"] = (int) (filesize(backupManageBackupPath($instance, $filename)) ?: ($backup["size"] ?? 0));
+        $path = backupServerBackupPath($instance, $filename);
+        $backup["local_exists"] = is_file($path);
+        $backup["size"] = $backup["local_exists"]
+            ? (int) (filesize($path) ?: ($backup["size"] ?? 0))
+            : (int) ($backup["size"] ?? 0);
         $grouped[$instance][] = $backup;
     }
 
@@ -409,6 +233,42 @@ function backupManageGroupBackupsByInstance(array $backups): array
     return $grouped;
 }
 
+function backupManageStorageLabel(array $backup): string
+{
+    $local = !empty($backup["local_exists"]);
+    $inS3 = !empty($backup["s3_uploaded_at"]);
+
+    if ($local && $inS3) {
+        return "Local + S3";
+    }
+    if ($local && backupS3Enabled() && empty($backup["s3_expired"])) {
+        return "Local (S3 pending)";
+    }
+    if ($local) {
+        return "Local";
+    }
+    if ($inS3) {
+        return "S3 only";
+    }
+
+    return "Missing";
+}
+
+function backupManageLogTail(int $lines): array
+{
+    $file = (string) BACKUP_SERVER_LOG_FILE;
+    if (!is_file($file)) {
+        return [];
+    }
+
+    $content = @file($file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
+    if (!is_array($content)) {
+        return [];
+    }
+
+    return array_slice($content, -$lines);
+}
+
 if ($_SERVER["REQUEST_METHOD"] === "POST") {
     $action = (string) ($_POST["action"] ?? "");
 
@@ -432,11 +292,14 @@ if ($_SERVER["REQUEST_METHOD"] === "POST") {
     } else {
         try {
             if ($action === "update_retention") {
-                $retention = max(1, (int) ($_POST["retention"] ?? BACKUP_SERVER_RETENTION));
-                $settings = backupManageReadSettings();
-                $settings["retention"] = $retention;
-                backupManageWriteSettings($settings);
-                backupManageApplyRetentionAll();
+                $settings = backupServerGetSettings();
+                $settings["retention"] = max(1, (int) ($_POST["retention"] ?? BACKUP_SERVER_RETENTION));
+                if (isset($_POST["s3_retention"])) {
+                    $settings["s3_retention"] = max(1, (int) $_POST["s3_retention"]);
+                }
+                backupServerWriteSettings($settings);
+                // May issue S3 deletes for backups that now age out of the archive.
+                backupServerApplyRetentionAll();
                 $messages[] = "Retention updated.";
             } elseif ($action === "add_instance") {
                 backupManageAddInstance((string) ($_POST["instance"] ?? ""));
@@ -446,15 +309,31 @@ if ($_SERVER["REQUEST_METHOD"] === "POST") {
                 $messages[] = "Instance removed.";
             } elseif ($action === "download") {
                 backupManageSendDownload(
-                    backupManageValidateInstance((string) ($_POST["instance"] ?? "")),
+                    backupServerValidateInstance((string) ($_POST["instance"] ?? "")),
                     backupManageValidateFilename((string) ($_POST["filename"] ?? "")),
                 );
             } elseif ($action === "delete") {
                 backupManageDeleteBackup(
-                    backupManageValidateInstance((string) ($_POST["instance"] ?? "")),
+                    backupServerValidateInstance((string) ($_POST["instance"] ?? "")),
                     backupManageValidateFilename((string) ($_POST["filename"] ?? "")),
                 );
                 $messages[] = "Backup deleted.";
+            } elseif ($action === "s3_sync") {
+                if (!backupS3Enabled()) {
+                    throw new RuntimeException("S3 is not configured. See config.php.");
+                }
+                $uploaded = 0;
+                $pending = 0;
+                foreach (backupServerIndexInstances() as $syncInstance) {
+                    $result = backupServerSyncInstanceS3($syncInstance);
+                    $uploaded += $result["uploaded"];
+                    $pending += $result["pending"];
+                    if ($result["error"] !== null) {
+                        $errors[] = "S3 upload for " . $syncInstance . " failed: " . $result["error"];
+                    }
+                    backupServerApplyRetention($syncInstance);
+                }
+                $messages[] = "S3 sync finished: " . $uploaded . " uploaded, " . $pending . " still pending.";
             }
         } catch (Throwable $exception) {
             $errors[] = $exception->getMessage();
@@ -463,14 +342,36 @@ if ($_SERVER["REQUEST_METHOD"] === "POST") {
 }
 
 try {
-    $settings = backupManageReadSettings();
-    $groupedBackups = backupManageGroupBackupsByInstance(backupManageReadIndex()["backups"]);
+    $settings = backupServerGetSettings();
+    $groupedBackups = backupManageGroupBackupsByInstance(backupServerReadIndex()["backups"]);
 } catch (Throwable $exception) {
-    $settings = ["retention" => max(1, (int) BACKUP_SERVER_RETENTION)];
+    $settings = [
+        "retention" => max(1, (int) BACKUP_SERVER_RETENTION),
+        "s3_retention" => max(1, (int) BACKUP_SERVER_S3_RETENTION),
+        "instances" => [],
+    ];
     $groupedBackups = [];
     $errors[] = $exception->getMessage();
 }
 
+$s3Enabled = backupS3Enabled();
+$s3PendingCount = 0;
+$s3LastErrors = [];
+if ($s3Enabled) {
+    foreach ($groupedBackups as $instanceBackups) {
+        foreach ($instanceBackups as $backup) {
+            if (!empty($backup["local_exists"]) && empty($backup["s3_uploaded_at"])) {
+                $s3PendingCount++;
+            }
+            if (!empty($backup["s3_last_error"]) && count($s3LastErrors) < 5) {
+                $s3LastErrors[] = ($backup["instance"] ?? "") . "/" . ($backup["filename"] ?? "") .
+                    ": " . $backup["s3_last_error"];
+            }
+        }
+    }
+}
+$s3LogTail = backupManageLogTail(20);
+
 ?>
 <!DOCTYPE html>
 <html lang="de">
@@ -510,12 +411,51 @@ try {
             <input type="hidden" name="action" value="update_retention">
             <input type="hidden" name="csrf_token" value="<?php echo backupManageEscape(backupManageCsrfToken()); ?>">
             <p>
-                <label for="retention">Backups retained per instance</label><br>
+                <label for="retention">Local backups retained per instance</label><br>
                 <input type="number" id="retention" name="retention" min="1" required value="<?php echo (int) $settings["retention"]; ?>">
             </p>
+            <?php if ($s3Enabled): ?>
+                <p>
+                    <label for="s3_retention">S3 backups retained per instance</label><br>
+                    <input type="number" id="s3_retention" name="s3_retention" min="1" required value="<?php echo (int) $settings["s3_retention"]; ?>">
+                </p>
+            <?php endif; ?>
             <button type="submit">Save retention</button>
         </form>
 
+        <h2>S3 archive</h2>
+        <?php if (!$s3Enabled): ?>
+            <p>S3 storage is not configured. Set the <code>BACKUP_SERVER_S3_*</code> constants in <code>config.php</code> to enable it.</p>
+        <?php else: ?>
+            <p>
+                Endpoint: <code><?php echo backupManageEscape(BACKUP_SERVER_S3_ENDPOINT); ?></code>,
+                Bucket: <code><?php echo backupManageEscape(BACKUP_SERVER_S3_BUCKET); ?></code>
+                <?php if (trim((string) BACKUP_SERVER_S3_PREFIX, "/") !== ""): ?>
+                    , Prefix: <code><?php echo backupManageEscape(trim((string) BACKUP_SERVER_S3_PREFIX, "/")); ?></code>
+                <?php endif; ?>
+            </p>
+            <p>Pending uploads: <?php echo (int) $s3PendingCount; ?></p>
+            <form method="POST">
+                <input type="hidden" name="action" value="s3_sync">
+                <input type="hidden" name="csrf_token" value="<?php echo backupManageEscape(backupManageCsrfToken()); ?>">
+                <button type="submit">Retry S3 uploads now</button>
+            </form>
+            <?php if (!empty($s3LastErrors)): ?>
+                <p><strong>Recent upload errors:</strong></p>
+                <ul>
+                    <?php foreach ($s3LastErrors as $s3LastError): ?>
+                        <li><?php echo backupManageEscape($s3LastError); ?></li>
+                    <?php endforeach; ?>
+                </ul>
+            <?php endif; ?>
+            <?php if (!empty($s3LogTail)): ?>
+                <details>
+                    <summary>S3 log (last <?php echo count($s3LogTail); ?> lines)</summary>
+                    <pre><?php echo backupManageEscape(implode("\n", $s3LogTail)); ?></pre>
+                </details>
+            <?php endif; ?>
+        <?php endif; ?>
+
         <h2>Upload endpoint</h2>
         <p>Distributed instances should upload to <code>upload.php</code>.</p>
 
@@ -545,7 +485,7 @@ try {
                         <tr>
                             <td><?php echo backupManageEscape($instance); ?></td>
                             <td>
-                                <form method="POST" style="display:inline" onsubmit="return confirm('Remove this allowed instance? Existing backups remain visible.');">
+                                <form method="POST" style="display:inline" onsubmit="return confirm('Remove this allowed instance? Existing backups remain visible; S3 objects remain until retention or manual delete.');">
                                     <input type="hidden" name="action" value="remove_instance">
                                     <input type="hidden" name="csrf_token" value="<?php echo backupManageEscape(backupManageCsrfToken()); ?>">
                                     <input type="hidden" name="instance" value="<?php echo backupManageEscape($instance); ?>">
@@ -570,6 +510,7 @@ try {
                             <th>Uploaded</th>
                             <th>Filename</th>
                             <th>Size</th>
+                            <th>Storage</th>
                             <th>SHA-256</th>
                             <th>Source IP</th>
                             <th>Actions</th>
@@ -581,6 +522,7 @@ try {
                                 <td><?php echo backupManageEscape($backup["uploaded_at"] ?? ""); ?></td>
                                 <td><?php echo backupManageEscape($backup["filename"] ?? ""); ?></td>
                                 <td><?php echo backupManageEscape(backupManageFormatBytes((int) ($backup["size"] ?? 0))); ?></td>
+                                <td title="<?php echo backupManageEscape($backup["s3_last_error"] ?? ""); ?>"><?php echo backupManageEscape(backupManageStorageLabel($backup)); ?></td>
                                 <td><?php echo backupManageEscape($backup["sha256"] ?? ""); ?></td>
                                 <td><?php echo backupManageEscape($backup["source_ip"] ?? ""); ?></td>
                                 <td>
@@ -591,7 +533,7 @@ try {
                                         <input type="hidden" name="filename" value="<?php echo backupManageEscape($backup["filename"] ?? ""); ?>">
                                         <button type="submit">Download</button>
                                     </form>
-                                    <form method="POST" style="display:inline" onsubmit="return confirm('Delete this backup?');">
+                                    <form method="POST" style="display:inline" onsubmit="return confirm('Delete this backup from all stores?');">
                                         <input type="hidden" name="action" value="delete">
                                         <input type="hidden" name="csrf_token" value="<?php echo backupManageEscape(backupManageCsrfToken()); ?>">
                                         <input type="hidden" name="instance" value="<?php echo backupManageEscape($instance); ?>">

backup-server/s3.php

@@ -0,0 +1,343 @@
+<?php
+
+declare(strict_types=1);
+
+// Dependency-free client for S3-compatible object storage (AWS Signature V4,
+// path-style addressing). Requires the BACKUP_SERVER_S3_* constants defined in
+// lib.php / config.php.
+
+function backupS3Config(): array
+{
+    return [
+        "endpoint" => rtrim(trim((string) BACKUP_SERVER_S3_ENDPOINT), "/"),
+        "region" => trim((string) BACKUP_SERVER_S3_REGION),
+        "bucket" => trim((string) BACKUP_SERVER_S3_BUCKET),
+        "prefix" => trim((string) BACKUP_SERVER_S3_PREFIX, "/"),
+        "access_key" => trim((string) BACKUP_SERVER_S3_ACCESS_KEY),
+        "secret_key" => (string) BACKUP_SERVER_S3_SECRET_KEY,
+        "timeout" => max(1, (int) BACKUP_SERVER_S3_TIMEOUT),
+        "path_style" => (bool) BACKUP_SERVER_S3_PATH_STYLE,
+    ];
+}
+
+function backupS3Enabled(): bool
+{
+    if (BACKUP_SERVER_S3_ENABLED !== true) {
+        return false;
+    }
+
+    $config = backupS3Config();
+
+    return $config["endpoint"] !== "" &&
+        $config["region"] !== "" &&
+        $config["bucket"] !== "" &&
+        $config["access_key"] !== "" &&
+        $config["secret_key"] !== "";
+}
+
+function backupS3ObjectKey(string $instance, string $filename): string
+{
+    $config = backupS3Config();
+    $key = $instance . "/" . $filename;
+
+    return $config["prefix"] !== "" ? $config["prefix"] . "/" . $key : $key;
+}
+
+function backupS3EmptyPayloadHash(): string
+{
+    return hash("sha256", "");
+}
+
+function backupS3HttpStatusFromHeaders(array $headers): int
+{
+    $status = 0;
+    foreach ($headers as $header) {
+        if (preg_match('/^HTTP\/\S+\s+(\d+)/', (string) $header, $matches) === 1) {
+            $status = (int) $matches[1];
+        }
+    }
+
+    return $status;
+}
+
+// $legacyHeaders must be the caller's $http_response_header, because PHP only
+// populates that variable in the scope where the HTTP call was made.
+function backupS3ResponseHeaders($legacyHeaders): array
+{
+    if (function_exists("http_get_last_response_headers")) {
+        $lastHeaders = http_get_last_response_headers();
+        return is_array($lastHeaders) ? $lastHeaders : [];
+    }
+
+    return is_array($legacyHeaders) ? $legacyHeaders : [];
+}
+
+function backupS3SignRequest(string $method, string $key, string $payloadHash, array $extraHeaders = []): array
+{
+    $config = backupS3Config();
+
+    $scheme = parse_url($config["endpoint"], PHP_URL_SCHEME);
+    $endpointHost = parse_url($config["endpoint"], PHP_URL_HOST);
+    if (!is_string($scheme) || $scheme === "" || !is_string($endpointHost) || $endpointHost === "") {
+        throw new RuntimeException("S3 endpoint is invalid.");
+    }
+
+    $encodedKey = str_replace("%2F", "/", rawurlencode($key));
+    if ($config["path_style"]) {
+        // https://<endpoint-host>/<bucket>/<key>
+        $host = $endpointHost;
+        $canonicalUri = "/" . rawurlencode($config["bucket"]) . "/" . $encodedKey;
+    } else {
+        // https://<bucket>.<endpoint-host>/<key> (default for Hetzner)
+        $host = $config["bucket"] . "." . $endpointHost;
+        $canonicalUri = "/" . $encodedKey;
+    }
+
+    $port = parse_url($config["endpoint"], PHP_URL_PORT);
+    if (is_int($port)) {
+        $host .= ":" . $port;
+    }
+
+    $url = $scheme . "://" . $host . $canonicalUri;
+
+    $now = gmdate("Ymd\THis\Z");
+    $date = substr($now, 0, 8);
+
+    $headers = array_merge($extraHeaders, [
+        "host" => $host,
+        "x-amz-content-sha256" => $payloadHash,
+        "x-amz-date" => $now,
+    ]);
+    ksort($headers);
+
+    $canonicalHeaders = "";
+    foreach ($headers as $name => $value) {
+        $canonicalHeaders .= $name . ":" . $value . "\n";
+    }
+    $signedHeaders = implode(";", array_keys($headers));
+
+    $canonicalRequest =
+        $method . "\n" .
+        $canonicalUri .
+        "\n\n" .
+        $canonicalHeaders .
+        "\n" .
+        $signedHeaders .
+        "\n" .
+        $payloadHash;
+    $scope = $date . "/" . $config["region"] . "/s3/aws4_request";
+    $stringToSign =
+        "AWS4-HMAC-SHA256\n" .
+        $now .
+        "\n" .
+        $scope .
+        "\n" .
+        hash("sha256", $canonicalRequest);
+    $kDate = hash_hmac("sha256", $date, "AWS4" . $config["secret_key"], true);
+    $kRegion = hash_hmac("sha256", $config["region"], $kDate, true);
+    $kService = hash_hmac("sha256", "s3", $kRegion, true);
+    $kSigning = hash_hmac("sha256", "aws4_request", $kService, true);
+    $signature = hash_hmac("sha256", $stringToSign, $kSigning);
+    $authorization =
+        "AWS4-HMAC-SHA256 Credential=" .
+        $config["access_key"] .
+        "/" .
+        $scope .
+        ", SignedHeaders=" .
+        $signedHeaders .
+        ", Signature=" .
+        $signature;
+
+    $headerString = "";
+    foreach ($headers as $name => $value) {
+        $headerString .= $name . ": " . $value . "\r\n";
+    }
+    $headerString .= "Authorization: " . $authorization . "\r\n";
+
+    return [
+        "url" => $url,
+        "headers" => $headerString,
+        "timeout" => $config["timeout"],
+    ];
+}
+
+// Builds a human-readable suffix for an error message from an S3 response.
+// S3-compatible endpoints return an XML body like
+// <Error><Code>SignatureDoesNotMatch</Code><Message>...</Message></Error>,
+// which pinpoints why a request was rejected.
+function backupS3ErrorDetail(int $status, $response): string
+{
+    $detail = $status > 0 ? " (HTTP " . $status . ")" : "";
+    $body = is_string($response) ? trim($response) : "";
+    if ($body === "") {
+        return $detail . ".";
+    }
+
+    $parts = [];
+    if (preg_match('#<Code>(.*?)</Code>#s', $body, $matches) === 1) {
+        $parts[] = trim($matches[1]);
+    }
+    if (preg_match('#<Message>(.*?)</Message>#s', $body, $matches) === 1) {
+        $parts[] = trim($matches[1]);
+    }
+    if ($parts === []) {
+        $parts[] = substr(preg_replace('/\s+/', " ", $body) ?? "", 0, 300);
+    }
+
+    return $detail . ": " . implode(" - ", $parts);
+}
+
+// Summarizes the response header chain so a failure can be diagnosed from the
+// log: every HTTP status line (reveals redirects), any Location target, and the
+// server's request id. $headers is the raw wrapper header array.
+function backupS3HeaderDiagnostic(array $headers): string
+{
+    $statuses = [];
+    $location = "";
+    $requestId = "";
+    foreach ($headers as $header) {
+        $header = (string) $header;
+        if (preg_match('/^HTTP\/\S+\s+(\d+)/', $header, $matches) === 1) {
+            $statuses[] = $matches[1];
+        } elseif (preg_match('/^Location:\s*(.+)$/i', $header, $matches) === 1) {
+            $location = trim($matches[1]);
+        } elseif (preg_match('/^x-amz-request-id:\s*(.+)$/i', $header, $matches) === 1) {
+            $requestId = trim($matches[1]);
+        }
+    }
+
+    $parts = [];
+    if ($statuses !== []) {
+        $parts[] = "status chain " . implode("->", $statuses);
+    }
+    if ($location !== "") {
+        $parts[] = "redirected to " . $location;
+    }
+    if ($requestId !== "") {
+        $parts[] = "request-id " . $requestId;
+    }
+
+    return $parts === [] ? "" : " [" . implode("; ", $parts) . "]";
+}
+
+function backupS3PutFile(string $localPath, string $key): void
+{
+    // The whole file is held in memory for signing; a backup larger than
+    // memory_limit fails here, stays local, and is retried later.
+    $payload = @file_get_contents($localPath);
+    if ($payload === false) {
+        throw new RuntimeException("Backup file cannot be read for S3 upload.");
+    }
+
+    $request = backupS3SignRequest("PUT", $key, hash("sha256", $payload), [
+        "content-type" => "application/zip",
+    ]);
+
+    $context = stream_context_create([
+        "http" => [
+            "method" => "PUT",
+            "timeout" => $request["timeout"],
+            "ignore_errors" => true,
+            // Never chase a redirect: PHP would re-send the body with a
+            // signature bound to the original host/path, which the target then
+            // rejects. A 3xx must surface so the endpoint config can be fixed.
+            "follow_location" => 0,
+            "max_redirects" => 1,
+            "protocol_version" => 1.1,
+            "header" => $request["headers"] . "Content-Length: " . strlen($payload) . "\r\n",
+            "content" => $payload,
+        ],
+    ]);
+
+    $response = @file_get_contents($request["url"], false, $context);
+    $headers = backupS3ResponseHeaders($http_response_header ?? null);
+    $status = backupS3HttpStatusFromHeaders($headers);
+
+    if ($response === false || $status < 200 || $status >= 300) {
+        throw new RuntimeException(
+            "S3 upload failed" . backupS3ErrorDetail($status, $response) . backupS3HeaderDiagnostic($headers),
+        );
+    }
+}
+
+function backupS3DeleteObject(string $key): void
+{
+    $request = backupS3SignRequest("DELETE", $key, backupS3EmptyPayloadHash());
+
+    $context = stream_context_create([
+        "http" => [
+            "method" => "DELETE",
+            "timeout" => $request["timeout"],
+            "ignore_errors" => true,
+            "follow_location" => 0,
+            "max_redirects" => 1,
+            "protocol_version" => 1.1,
+            "header" => $request["headers"],
+        ],
+    ]);
+
+    $response = @file_get_contents($request["url"], false, $context);
+    $headers = backupS3ResponseHeaders($http_response_header ?? null);
+    $status = backupS3HttpStatusFromHeaders($headers);
+
+    // DELETE is idempotent: an already missing object (404) counts as deleted.
+    if ($response === false || ($status !== 404 && ($status < 200 || $status >= 300))) {
+        throw new RuntimeException(
+            "S3 delete failed" . backupS3ErrorDetail($status, $response) . backupS3HeaderDiagnostic($headers),
+        );
+    }
+}
+
+function backupS3SendObjectToOutput(string $key, string $downloadName, int $fallbackSize): void
+{
+    $request = backupS3SignRequest("GET", $key, backupS3EmptyPayloadHash());
+
+    $context = stream_context_create([
+        "http" => [
+            "method" => "GET",
+            "timeout" => $request["timeout"],
+            "ignore_errors" => true,
+            "follow_location" => 0,
+            "max_redirects" => 1,
+            "protocol_version" => 1.1,
+            "header" => $request["headers"],
+        ],
+    ]);
+
+    $handle = @fopen($request["url"], "rb", false, $context);
+    if ($handle === false) {
+        throw new RuntimeException("S3 download failed (connection error).");
+    }
+
+    $meta = stream_get_meta_data($handle);
+    $headers = isset($meta["wrapper_data"]) && is_array($meta["wrapper_data"])
+        ? $meta["wrapper_data"]
+        : [];
+    $status = backupS3HttpStatusFromHeaders($headers);
+    if ($status < 200 || $status >= 300) {
+        $body = stream_get_contents($handle, 2048);
+        fclose($handle);
+        throw new RuntimeException(
+            "S3 download failed" . backupS3ErrorDetail($status, $body) . backupS3HeaderDiagnostic($headers),
+        );
+    }
+
+    $size = $fallbackSize;
+    foreach ($headers as $header) {
+        if (preg_match('/^Content-Length:\s*(\d+)/i', (string) $header, $matches) === 1) {
+            $size = (int) $matches[1];
+        }
+    }
+
+    header("Content-Type: application/zip");
+    header("Content-Disposition: attachment; filename=\"" . addcslashes($downloadName, "\"\\") . "\"");
+    if ($size > 0) {
+        header("Content-Length: " . (string) $size);
+    }
+    header("Cache-Control: private, no-store");
+    header("X-Content-Type-Options: nosniff");
+
+    fpassthru($handle);
+    fclose($handle);
+    exit;
+}

backup-server/upload.php

@@ -2,25 +2,7 @@
 
 declare(strict_types=1);
 
-$baseDir = __DIR__;
-$configFile = $baseDir . "/config.php";
-
-if (is_file($configFile)) {
-    require_once $configFile;
-}
-
-if (!defined("BACKUP_SERVER_RETENTION")) {
-    define("BACKUP_SERVER_RETENTION", 30);
-}
-if (!defined("BACKUP_SERVER_BACKUP_DIR")) {
-    define("BACKUP_SERVER_BACKUP_DIR", $baseDir . "/backups/");
-}
-if (!defined("BACKUP_SERVER_INDEX_FILE")) {
-    define("BACKUP_SERVER_INDEX_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/index.json");
-}
-if (!defined("BACKUP_SERVER_SETTINGS_FILE")) {
-    define("BACKUP_SERVER_SETTINGS_FILE", rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . "/settings.json");
-}
+require_once __DIR__ . "/lib.php";
 
 header("Content-Type: application/json; charset=utf-8");
 header("Cache-Control: no-store");
@@ -36,123 +18,9 @@ function backupUploadRespond(int $status, array $payload): void
     exit;
 }
 
-function backupUploadEnsureDirectory(string $dir): void
-{
-    if (!is_dir($dir) && !mkdir($dir, 02775, true) && !is_dir($dir)) {
-        throw new RuntimeException("Directory cannot be created.");
-    }
-
-    @chmod($dir, 02775);
-}
-
-function backupUploadNormalizeInstance(string $instance): string
-{
-    $instance = trim($instance);
-    if (
-        $instance === "" ||
-        strlen($instance) > 120 ||
-        preg_match('/^[A-Za-z0-9][A-Za-z0-9._-]*$/', $instance) !== 1
-    ) {
-        throw new RuntimeException("Invalid instance identifier.");
-    }
-
-    return $instance;
-}
-
-function backupUploadReadJsonFile(string $file): array
-{
-    if (!is_file($file)) {
-        return [];
-    }
-
-    $decoded = json_decode((string) file_get_contents($file), true);
-    return is_array($decoded) ? $decoded : [];
-}
-
-function backupUploadWriteJsonFile(string $file, array $data): void
-{
-    backupUploadEnsureDirectory(dirname($file));
-
-    $json = json_encode(
-        $data,
-        JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE,
-    );
-    if ($json === false) {
-        throw new RuntimeException("JSON cannot be encoded.");
-    }
-
-    $tmpFile = $file . ".tmp";
-    if (file_put_contents($tmpFile, $json . PHP_EOL, LOCK_EX) === false) {
-        throw new RuntimeException("JSON cannot be written.");
-    }
-
-    @chmod($tmpFile, 0664);
-    if (!rename($tmpFile, $file)) {
-        @unlink($tmpFile);
-        throw new RuntimeException("JSON cannot be saved.");
-    }
-
-    @chmod($file, 0664);
-}
-
-function backupUploadReadIndex(): array
-{
-    $index = backupUploadReadJsonFile((string) BACKUP_SERVER_INDEX_FILE);
-    $backups = isset($index["backups"]) && is_array($index["backups"])
-        ? $index["backups"]
-        : [];
-
-    return ["backups" => array_values($backups)];
-}
-
-function backupUploadWriteIndex(array $backups): void
-{
-    backupUploadWriteJsonFile((string) BACKUP_SERVER_INDEX_FILE, [
-        "backups" => array_values($backups),
-    ]);
-}
-
-function backupUploadGetRetention(): int
-{
-    $settings = backupUploadReadJsonFile((string) BACKUP_SERVER_SETTINGS_FILE);
-    if (isset($settings["retention"])) {
-        return max(1, (int) $settings["retention"]);
-    }
-
-    return max(1, (int) BACKUP_SERVER_RETENTION);
-}
-
-function backupUploadGetAllowedInstances(): array
-{
-    $settings = backupUploadReadJsonFile((string) BACKUP_SERVER_SETTINGS_FILE);
-    $instances =
-        isset($settings["instances"]) && is_array($settings["instances"])
-            ? $settings["instances"]
-            : [];
-    $allowed = [];
-
-    foreach ($instances as $instance) {
-        $instance = (string) $instance;
-        if (
-            $instance !== "" &&
-            preg_match('/^[A-Za-z0-9][A-Za-z0-9._-]*$/', $instance) === 1
-        ) {
-            $allowed[$instance] = true;
-        }
-    }
-
-    return $allowed;
-}
-
 function backupUploadInstanceIsAllowed(string $instance): bool
 {
-    $allowed = backupUploadGetAllowedInstances();
-    return isset($allowed[$instance]);
-}
-
-function backupUploadInstanceDir(string $instance): string
-{
-    return rtrim((string) BACKUP_SERVER_BACKUP_DIR, "/\\") . DIRECTORY_SEPARATOR . $instance;
+    return in_array($instance, backupServerGetSettings()["instances"], true);
 }
 
 function backupUploadIsZipFile(string $path): bool
@@ -194,48 +62,12 @@ function backupUploadChooseFilename(string $clientFilename, string $instanceDir)
     return $filename;
 }
 
-function backupUploadApplyRetention(string $instance): void
-{
-    $index = backupUploadReadIndex();
-    $retention = backupUploadGetRetention();
-    $instanceBackups = [];
-    $otherBackups = [];
-
-    foreach ($index["backups"] as $backup) {
-        if (!is_array($backup)) {
-            continue;
-        }
-        if (($backup["instance"] ?? "") === $instance) {
-            $instanceBackups[] = $backup;
-        } else {
-            $otherBackups[] = $backup;
-        }
-    }
-
-    usort($instanceBackups, function ($left, $right) {
-        return strcmp((string) ($right["uploaded_at"] ?? ""), (string) ($left["uploaded_at"] ?? ""));
-    });
-
-    $keep = array_slice($instanceBackups, 0, $retention);
-    $remove = array_slice($instanceBackups, $retention);
-    $instanceDir = backupUploadInstanceDir($instance);
-
-    foreach ($remove as $backup) {
-        $filename = basename((string) ($backup["filename"] ?? ""));
-        if ($filename !== "" && is_file($instanceDir . DIRECTORY_SEPARATOR . $filename)) {
-            @unlink($instanceDir . DIRECTORY_SEPARATOR . $filename);
-        }
-    }
-
-    backupUploadWriteIndex(array_merge($otherBackups, $keep));
-}
-
 if ($_SERVER["REQUEST_METHOD"] !== "POST") {
     backupUploadRespond(405, ["success" => false, "error" => "POST required."]);
 }
 
 try {
-    $instance = backupUploadNormalizeInstance((string) ($_POST["instance"] ?? ""));
+    $instance = backupServerValidateInstance((string) ($_POST["instance"] ?? ""));
     if (!backupUploadInstanceIsAllowed($instance)) {
         throw new RuntimeException("Instance is not allowed.");
     }
@@ -258,8 +90,8 @@ try {
         throw new RuntimeException("Uploaded file must be a ZIP file.");
     }
 
-    $instanceDir = backupUploadInstanceDir($instance);
-    backupUploadEnsureDirectory($instanceDir);
+    $instanceDir = backupServerInstanceDir($instance);
+    backupServerEnsureDirectory($instanceDir);
 
     $requestedFilename = (string) ($_POST["filename"] ?? "");
     $clientFilename = $requestedFilename !== "" ? $requestedFilename : (string) ($file["name"] ?? "");
@@ -285,7 +117,7 @@ try {
         throw new RuntimeException("Backup checksum mismatch.");
     }
 
-    $index = backupUploadReadIndex();
+    $index = backupServerReadIndex();
     $record = [
         "instance" => $instance,
         "filename" => $filename,
@@ -296,8 +128,25 @@ try {
         "source_ip" => $_SERVER["REMOTE_ADDR"] ?? "unknown",
     ];
     $index["backups"][] = $record;
-    backupUploadWriteIndex($index["backups"]);
-    backupUploadApplyRetention($instance);
+    backupServerWriteIndex($index["backups"]);
+
+    // S3 problems must never fail the upload: the local copy exists, and the
+    // sync is retried on the next upload or via the management UI.
+    $s3Enabled = backupS3Enabled();
+    $s3Result = ["uploaded" => 0, "pending" => 0, "error" => null];
+    if ($s3Enabled) {
+        try {
+            $s3Result = backupServerSyncInstanceS3($instance);
+        } catch (Throwable $exception) {
+            $s3Result = ["uploaded" => 0, "pending" => 1, "error" => $exception->getMessage()];
+            backupServerLog("S3 sync crashed", [
+                "instance" => $instance,
+                "error" => $exception->getMessage(),
+            ]);
+        }
+    }
+
+    backupServerApplyRetention($instance);
 
     backupUploadRespond(200, [
         "success" => true,
@@ -305,7 +154,12 @@ try {
         "filename" => $filename,
         "size" => $size,
         "sha256" => $sha256,
-        "retention" => backupUploadGetRetention(),
+        "retention" => backupServerGetSettings()["retention"],
+        "s3" => [
+            "enabled" => $s3Enabled,
+            "uploaded" => $s3Enabled && $s3Result["pending"] === 0,
+            "pending" => $s3Result["pending"],
+        ],
     ]);
 } catch (Throwable $exception) {
     backupUploadRespond(400, [