Backups

​

What are backups?

• What to back up - The volume (source data)
• Where to store it - The primary repository (and optional mirrors)
• When to run - Cron-based schedule
• What to keep - Retention policy
• What to include/exclude - File patterns and filters
• Who to notify - Alert destinations for success/failure

​

Why backup schedules matter

​

Consistency

​

Automation

1. Volume health verification
2. Snapshot creation
3. Retention policy application
4. Mirror synchronization
5. Notifications

​

Granular control

• Cron timing
• Inclusion/exclusion patterns
• Retention rules
• Mirror repositories
• Notification settings

​

Visibility

• Start and completion timestamps
• Success/warning/error status
• Detailed progress events (files processed, bytes transferred)
• Error messages for failures

​

Backup lifecycle

Scheduled → Validation → Execution → Retention → Mirroring → Notification
                ↓
              Error → Retry (next schedule)

​

Validation phase

1. Schedule enabled - Disabled schedules are skipped (unless manually triggered)
2. Not already running - Prevents duplicate executions
3. Volume exists - Schedule references a valid volume
4. Repository exists - Schedule references a valid repository
5. Volume mounted - Source must be accessible

​

Execution phase

1. Lock acquisition - Repository receives a shared lock (allows concurrent backups, blocks maintenance)
2. Restic invocation - restic backup runs with configured options
3. Progress streaming - Real-time progress events via server-sent events (SSE)
4. Status tracking - lastBackupStatus updated to in_progress

​

Post-backup phase

1. Retention application - restic forget removes old snapshots based on policy
2. Mirror synchronization - Snapshots copied to mirror repositories
3. Cache invalidation - Repository caches cleared to reflect new snapshots
4. Next run calculation - nextBackupAt computed from cron expression
5. Notifications sent - Success/warning/failure alerts delivered

​

Cron expressions

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
│ │ │ │ │
* * * * *

​

Common patterns

0 * * * *

0 2 * * *

0 */6 * * *

0 18 * * 1-5

0 3 * * 0

0 1 1 * *

*/15 * * * *

​

Next run calculation

• Updates after each backup completes
• Accounts for the server’s timezone
• Determines when the scheduler triggers the next execution

​

Inclusion and exclusion patterns

​

Exclude patterns

{
  "excludePatterns": [
    "/tmp/**",
    "*.log",
    "/cache/*",
    "!important.log"
  ]
}

• Paths starting with / are relative to the volume root
• Paths without / are matched anywhere in the tree
• Glob patterns supported: *, **, ?, [...]
• Prefix with ! to negate (include despite other exclusions)

• Temporary files: *.tmp, /tmp/**
• Caches: /var/cache/**, node_modules/**, .cache/**
• Logs: *.log, /var/log/**
• OS files: .DS_Store, Thumbs.db, desktop.ini

​

Include patterns

{
  "includePatterns": [
    "/data/important/**",
    "/configs/*.yaml"
  ]
}

​

Exclude if present

{
  "excludeIfPresent": [
    ".nobackup",
    "SKIP_BACKUP.txt"
  ]
}

• Let developers mark cache directories with .nobackup
• Exclude build artifacts by placing SKIP_BACKUP.txt in target/ or dist/

​

One file system

{
  "oneFileSystem": true
}

• Mount points encountered during traversal are not crossed
• Prevents backing up accidentally mounted filesystems
• Useful for NFS volumes where subdirectories might be remote mounts

​

How backups work in Zerobyte

​

Scheduler

1. Polls backupSchedulesTable every minute
2. Finds schedules where nextBackupAt <= now and enabled = true
3. Calls executeBackup() for each due schedule
4. Continues running even if individual backups fail

​

Backup execution

// Simplified flow from backups.execution.ts
async function executeBackup(scheduleId: number, manual = false) {
  // 1. Validate
  const validation = await validateBackupExecution(scheduleId, manual);
  if (validation.type !== "success") {
    return handleValidationResult(scheduleId, validation);
  }

  const { schedule, volume, repository, organizationId } = validation.context;

  // 2. Emit started event
  serverEvents.emit("backup:started", ...);

  // 3. Update status to in_progress
  await scheduleQueries.updateStatus(scheduleId, organizationId, {
    lastBackupStatus: "in_progress",
  });

  // 4. Acquire repository lock
  const releaseBackupLock = await repoMutex.acquireShared(repository.id, "backup");

  try {
    // 5. Run restic backup
    const result = await restic.backup(repository.config, volumePath, {
      tags: [schedule.shortId],
      exclude: schedule.excludePatterns,
      include: schedule.includePatterns,
      oneFileSystem: schedule.oneFileSystem,
      compressionMode: repository.compressionMode,
      onProgress: (progress) => {
        serverEvents.emit("backup:progress", ...);
      },
    });

    // 6. Apply retention policy
    if (schedule.retentionPolicy) {
      await runForget(scheduleId);
    }

    // 7. Copy to mirrors
    await copyToMirrors(scheduleId, repository, schedule.retentionPolicy);

    // 8. Update status to success
    await scheduleQueries.updateStatus(scheduleId, organizationId, {
      lastBackupStatus: result.exitCode === 0 ? "success" : "warning",
      lastBackupAt: Date.now(),
      nextBackupAt: calculateNextRun(schedule.cronExpression),
    });

    // 9. Emit completed event
    serverEvents.emit("backup:completed", { status: "success", ... });

  } catch (error) {
    // Handle failure
    await scheduleQueries.updateStatus(scheduleId, organizationId, {
      lastBackupStatus: "error",
      lastBackupError: toMessage(error),
    });
    serverEvents.emit("backup:completed", { status: "error", ... });
  } finally {
    releaseBackupLock();
  }
}

​

Progress tracking

{
  percentDone: 45.2,
  filesProcessed: 1234,
  bytesProcessed: 567890123,
  totalFiles: 2730,
  totalBytes: 1256000000,
  currentFile: "/data/photos/IMG_1234.jpg"
}

​

Restic tags

• Filtering snapshots by backup schedule
• Targeted retention (forget only affects snapshots with the schedule’s tag)
• Restore history (see which schedule created each snapshot)

​

Retention policies

​

Mirror repositories

​

How mirrors work

1. Primary backup completes successfully to the main repository
2. For each enabled mirror:
   • Acquire locks on source and mirror repositories
   • Run restic copy to transfer snapshots matching the schedule’s tag
   • Apply retention policy to the mirror (if configured)
   • Update mirror status and timestamps

​

Mirror compatibility

• Local ↔ Local
• S3-compatible ↔ S3-compatible (S3, R2, MinIO, etc.)
• Any ↔ REST server
• Rclone ↔ Rclone (if backends match)

• S3 ↔ Azure (different authentication mechanisms)
• GCS ↔ SFTP (protocol mismatch)

​

Mirror status tracking

• lastCopyAt - When the last copy operation ran
• lastCopyStatus - “success”, “error”, or “in_progress”
• lastCopyError - Error message if copy failed

​

Backup status states

​

Manual backups

POST /api/backups/:scheduleId/execute

• Run immediately regardless of nextBackupAt
• Ignore the enabled flag (run even if schedule is disabled)
• Update lastBackupAt and nextBackupAt as normal
• Count toward the retention policy

​

Stopping backups

POST /api/backups/:scheduleId/stop

1. Aborts the Restic process via signal
2. Releases the repository lock
3. Sets status to “warning” with error “Backup was stopped by user”
4. May leave partial snapshot data (cleaned up by next prune/doctor)

​

Database schema

{
  id: number,                    // Auto-increment primary key
  shortId: string,               // Human-friendly unique identifier
  name: string,                  // Display name
  volumeId: number,              // Foreign key to volumes_table
  repositoryId: string,          // Foreign key to repositories_table
  enabled: boolean,              // Schedule active
  cronExpression: string,        // Cron schedule
  retentionPolicy: RetentionPolicy | null,
  excludePatterns: string[],
  excludeIfPresent: string[],
  includePatterns: string[],
  oneFileSystem: boolean,
  lastBackupAt: number | null,
  lastBackupStatus: "success" | "error" | "in_progress" | "warning" | null,
  lastBackupError: string | null,
  nextBackupAt: number | null,
  sortOrder: number,             // UI display order
  organizationId: string,
  createdAt: number,
  updatedAt: number
}

​

Best practices

*.tmp
*.log
/tmp/**
/var/cache/**
node_modules/**
.git/objects/**

​

Next steps