Skip to content

UKwinika Enhanced Automated Backup Script v3.3.0

Latest
Latest

Choose a tag to compare

View all tags
@UkwiNux UkwiNux released this 17 Jul 01:44
3.3.0
bbaedef
This commit was signed with the committer’s verified signature.
UkwiNux Urayayi Kwinika
GPG key ID: 43A2804BC982489A
Verified
Learn about vigilant mode.

UKwinika Enhanced Automated Backup Script — v3.3.0

Release date: July 2026 Author: Urayayi Kwinika License: MIT

This release closes every gap identified in the review of the previous release: one confirmed logic bug, full ShellCheck cleanliness, and a set of production-hardening features modeled on borgmatic. It also adds a bare-metal disaster-recovery runbook and ready-to-use Prometheus alerting rules.

🐛 Fixed

  • Dead-code bug in validate_usb_target(). The success branch of a case statement previously returned immediately, meaning the mkdir -p "$USB_RSYNC_TARGET" auto-creation logic that followed it could never execute (confirmed via ShellCheck SC2317). The USB target directory is now always created if missing, regardless of which branch the containment check takes.
  • ShellCheck now passes with zero findings on both enhanced_automated_backups.sh and backuprestore/ukwinika_automated_restore.sh — previously SC2015, SC2024, and four SC2317 findings on the main script.
    • The SC2015 A && B || C pattern in audit() was rewritten as an explicit if.
    • The SC2024 finding on the PostgreSQL dump redirect was reviewed, confirmed safe (the script always runs as root, so the redirect's file descriptor is opened before sudo drops privilege), and suppressed with an inline rationale comment rather than silently ignored.

✨ Added

  • MIN_BORG_VERSION enforcement (default 1.2.0) — the script now refuses to run against an older, potentially incompatible borg binary instead of failing with a confusing error partway through a backup.
  • Retry/backoff for transient failures — USB mount (USB_RETRY_ATTEMPTS, USB_RETRY_DELAY_SEC) and cloud upload (CLOUD_RETRY_ATTEMPTS, CLOUD_RETRY_DELAY_SEC) now retry with a configurable delay before giving up, rather than failing the whole backup on the first blip.
  • validate subcommand — borgmatic-style configuration validation with zero side effects: checks required variables, DB_TYPE, the installed borg version, and current repository state without touching the repository, USB, or cloud.
  • backup --dry-run — simulates a full backup cycle (borg create --dry-run) without writing an archive, pruning, syncing to USB, uploading to cloud, or updating metrics/notifications.
  • check-if-due subcommand + scheduled due-date tracking (CHECK_STATE_FILE, CHECK_INTERVAL_DAYS, default 7 days) — a borgmatic-style "checks" feature: a full borg check is expensive on large repositories, so this only actually runs one once the configured interval has elapsed, tracked via a timestamp state file. Wired to a new ukwinika-check.timer / ukwinika-check.service pair that evaluates daily.
  • LVM snapshot hook examples (hooks/lvm_snapshot_pre_backup_hook.sh.example, hooks/lvm_snapshot_post_backup_hook.sh.example) — plug into the existing PRE_HOOK / POST_HOOK mechanism to take a point-in-time LVM snapshot before Borg reads, and clean it up afterwards. Closes the filesystem-consistency gap for hosts not already relying solely on DB-aware dumps.
  • docs/DISASTER-RECOVERY.md — a full bare-metal / total-host-loss recovery runbook, distinct from the existing file-level restore-drill checklist: what must survive off-host before disaster strikes, provisioning a replacement host, pointing at a surviving repository copy, verifying integrity before restoring, and rebuilding scheduling and monitoring afterwards.
  • prometheus/ukwinika-backup-alerts.yml — ready-to-use Prometheus alerting rules for stale backups, missing metrics entirely, failed or stale restore drills, and individual failing verification checks. Closes the "metrics exist but nothing alerts on them" gap.

🔧 Changed

  • CI (.github/workflows/test.yml) extended to test validate, backup --dry-run (asserting no archive is created), and check-if-due (asserting the second call within the same interval is skipped), plus presence/syntax checks for all new v3.3.0 supporting files.
  • Makefile's systemd target now creates /var/lib/ukwinika up front and documents the new ukwinika-check.timer in its post-install guidance.
  • Version banners bumped to v3.3.0 across config/, hooks/, systemd/, logrotate/, and docs/.
  • README.md and UKWINIKA-DOCUMENTATION.md fully rewritten to document every new subcommand, configuration variable, and workflow.

New CLI Subcommands at a Glance

Command Effect
enhanced_automated_backups.sh validate Validate config/secrets/borg version. No side effects.
enhanced_automated_backups.sh backup --dry-run Simulate a full backup with no writes anywhere.
enhanced_automated_backups.sh check-if-due Run borg check only if CHECK_INTERVAL_DAYS has elapsed.

New Configuration Variables

MIN_BORG_VERSION, USB_RETRY_ATTEMPTS, USB_RETRY_DELAY_SEC, CLOUD_RETRY_ATTEMPTS, CLOUD_RETRY_DELAY_SEC, CHECK_STATE_FILE, CHECK_INTERVAL_DAYS — see config/ukwinika-backup.conf.example for defaults and inline documentation.

New Files

docs/DISASTER-RECOVERY.md
prometheus/ukwinika-backup-alerts.yml
systemd/ukwinika-check.service
systemd/ukwinika-check.timer
hooks/lvm_snapshot_pre_backup_hook.sh.example
hooks/lvm_snapshot_post_backup_hook.sh.example

Upgrading from 3.2.2

  1. Replace enhanced_automated_backups.sh and backuprestore/ukwinika_automated_restore.sh with the v3.3.0 versions.
  2. Add the new configuration variables to /etc/ukwinika-backup.conf (see config/ukwinika-backup.conf.example for defaults).
  3. Run sudo make systemd again to install ukwinika-check.service / .timer and create /var/lib/ukwinika.
  4. Enable the new timer: 
    sudo systemctl enable --now ukwinika-check.timer
  5. Validate the upgraded configuration before your next scheduled run: 
    sudo enhanced_automated_backups.sh validate
  6. (Optional) Adopt the LVM snapshot hooks if you need filesystem consistency beyond DB-aware dumps.
  7. (Optional) Install prometheus/ukwinika-backup-alerts.yml into your Prometheus rule directory and reload Prometheus.

No breaking changes to existing configuration variables, archive naming, or CLI behavior for backup, restore, list, check, real-time, or init — all new functionality is additive.

Compatibility

  • BorgBackup: 1.2.0 or newer (enforced at runtime via MIN_BORG_VERSION)
  • Distributions: Debian, Ubuntu, RHEL, Rocky Linux, AlmaLinux, CentOS Stream
  • Shell: Bash, set -euo pipefail, ShellCheck-clean

Full Changelog

See CHANGELOG.md for the complete, itemised version history.

UKwinika Notable Advice: A Backup is Only as Good as its Last Successful Restore. Run monthly restore drills, rehearse disaster recovery at least annually, and let validate and check-if-due catch drift before it becomes an incident.

# UKwinika Enhanced Automated Backup Script — v3.3.0

Release date: July 2026
Author: Urayayi Kwinika
License: MIT

This release closes every gap identified in a systems-administrator and Linux-programmer review of v3.2.2: one confirmed logic bug, full ShellCheck cleanliness, and a set of production-hardening features modeled on borgmatic. It also adds a bare-metal disaster-recovery runbook and ready-to-use Prometheus alerting rules.

🐛 Fixed

  • Dead-code bug in validate_usb_target(). The success branch of a case statement previously returned immediately, meaning the mkdir -p "$USB_RSYNC_TARGET" auto-creation logic that followed it could never execute (confirmed via ShellCheck SC2317). The USB target directory is now always created if missing, regardless of which branch the containment check takes.
  • ShellCheck now passes with zero findings on both enhanced_automated_backups.sh and backuprestore/ukwinika_automated_restore.sh — previously SC2015, SC2024, and four SC2317 findings on the main script.
    • The SC2015 A && B || C pattern in audit() was rewritten as an explicit if.
    • The SC2024 finding on the PostgreSQL dump redirect was reviewed, confirmed safe (the script always runs as root, so the redirect's file descriptor is opened before sudo drops privilege), and suppressed with an inline rationale comment rather than silently ignored.

✨ Added

  • MIN_BORG_VERSION enforcement (default 1.2.0) — the script now refuses to run against an older, potentially incompatible borg binary instead of failing with a confusing error partway through a backup.
  • Retry/backoff for transient failures — USB mount (USB_RETRY_ATTEMPTS, USB_RETRY_DELAY_SEC) and cloud upload (CLOUD_RETRY_ATTEMPTS, CLOUD_RETRY_DELAY_SEC) now retry with a configurable delay before giving up, rather than failing the whole backup on the first blip.
  • validate subcommand — borgmatic-style configuration validation with zero side effects: checks required variables, DB_TYPE, the installed borg version, and current repository state without touching the repository, USB, or cloud.
  • backup --dry-run — simulates a full backup cycle (borg create --dry-run) without writing an archive, pruning, syncing to USB, uploading to cloud, or updating metrics/notifications.
  • check-if-due subcommand + scheduled due-date tracking (CHECK_STATE_FILE, CHECK_INTERVAL_DAYS, default 7 days) — a borgmatic-style "checks" feature: a full borg check is expensive on large repositories, so this only actually runs one once the configured interval has elapsed, tracked via a timestamp state file. Wired to a new ukwinika-check.timer / ukwinika-check.service pair that evaluates daily.
  • LVM snapshot hook examples (hooks/lvm_snapshot_pre_backup_hook.sh.example, hooks/lvm_snapshot_post_backup_hook.sh.example) — plug into the existing PRE_HOOK / POST_HOOK mechanism to take a point-in-time LVM snapshot before Borg reads, and clean it up afterwards. Closes the filesystem-consistency gap for hosts not already relying solely on DB-aware dumps.
  • docs/DISASTER-RECOVERY.md — a full bare-metal / total-host-loss recovery runbook, distinct from the existing file-level restore-drill checklist: what must survive off-host before disaster strikes, provisioning a replacement host, pointing at a surviving repository copy, verifying integrity before restoring, and rebuilding scheduling and monitoring afterwards.
  • prometheus/ukwinika-backup-alerts.yml — ready-to-use Prometheus alerting rules for stale backups, missing metrics entirely, failed or stale restore drills, and individual failing verification checks. Closes the "metrics exist but nothing alerts on them" gap.

🔧 Changed

  • CI (.github/workflows/test.yml) extended to test validate, backup --dry-run (asserting no archive is created), and check-if-due (asserting the second call within the same interval is skipped), plus presence/syntax checks for all new v3.3.0 supporting files.
  • Makefile's systemd target now creates /var/lib/ukwinika up front and documents the new ukwinika-check.timer in its post-install guidance.
  • Version banners bumped to v3.3.0 across config/, hooks/, systemd/, logrotate/, and docs/.
  • README.md and UKWINIKA-DOCUMENTATION.md fully rewritten to document every new subcommand, configuration variable, and workflow.

New CLI Subcommands at a Glance

Command Effect
enhanced_automated_backups.sh validate Validate config/secrets/borg version. No side effects.
enhanced_automated_backups.sh backup --dry-run Simulate a full backup with no writes anywhere.
enhanced_automated_backups.sh check-if-due Run borg check only if CHECK_INTERVAL_DAYS has elapsed.

New Configuration Variables

MIN_BORG_VERSION, USB_RETRY_ATTEMPTS, USB_RETRY_DELAY_SEC, CLOUD_RETRY_ATTEMPTS, CLOUD_RETRY_DELAY_SEC, CHECK_STATE_FILE, CHECK_INTERVAL_DAYS — see config/ukwinika-backup.conf.example for defaults and inline documentation.

New Files

docs/DISASTER-RECOVERY.md
prometheus/ukwinika-backup-alerts.yml
systemd/ukwinika-check.service
systemd/ukwinika-check.timer
hooks/lvm_snapshot_pre_backup_hook.sh.example
hooks/lvm_snapshot_post_backup_hook.sh.example

Upgrading from 3.2.2

  1. Replace enhanced_automated_backups.sh and backuprestore/ukwinika_automated_restore.sh with the v3.3.0 versions.
  2. Add the new configuration variables to /etc/ukwinika-backup.conf (see config/ukwinika-backup.conf.example for defaults).
  3. Run sudo make systemd again to install ukwinika-check.service / .timer and create /var/lib/ukwinika.
  4. Enable the new timer: 
    sudo systemctl enable --now ukwinika-check.timer
  5. Validate the upgraded configuration before your next scheduled run: 
    sudo enhanced_automated_backups.sh validate
  6. (Optional) Adopt the LVM snapshot hooks if you need filesystem consistency beyond DB-aware dumps.
  7. (Optional) Install prometheus/ukwinika-backup-alerts.yml into your Prometheus rule directory and reload Prometheus.

No breaking changes to existing configuration variables, archive naming, or CLI behavior for backup, restore, list, check, real-time, or init — all new functionality is additive.

Compatibility

  • BorgBackup: 1.2.0 or newer (enforced at runtime via MIN_BORG_VERSION)
  • Distributions: Debian, Ubuntu, RHEL, Rocky Linux, AlmaLinux, CentOS Stream
  • Shell: Bash, set -euo pipefail, ShellCheck-clean

Full Changelog

See [CHANGELOG.md](https://github.com/UkwiNux/ukwinika-backups/blob/main/CHANGELOG.md) for the complete, itemised version history.

UKwinika Notable Advice: A Backup is Only as Good as its Last Successful Restore. Run monthly restore drills, rehearse disaster recovery at least annually, and let validate and check-if-due catch drift before it becomes an incident.

Assets 2
Loading