How-To Take a Backup of Ephemeral Storage – Crusoe Support

Introduction

Ephemeral storage is temporary storage that is often attached to cloud instances or servers. It is typically used for tasks requiring high-performance I/O and is ideal for storing data that does not need to persist beyond the lifecycle of the instance. However, it is important to note that ephemeral data is erased when an instance is stopped or terminated. To ensure critical data is preserved, periodic backups must be taken.

This guide will walk you through the process of backing up data from ephemeral storage (e.g., /raid0) to shared storage, persistent storage, or a network share. We'll also address strategies to automate backups, manage retention policies, and transfer backups to external locations, such as S3 buckets, for long-term preservation.

By following this guide, you will:

Learn how to create backups using tar.
Automate the backup process with cron.
Implement retention policies to manage disk space.
Understand alternative tools like rsync for efficient backups.

Prerequisites

Before proceeding, ensure the following:

Root or sudo access: Backup tasks generally require root permissions.
A mounted persistent storage volume: This could be an attached disk, network share (e.g., NFS mount), or cloud-based persistent disk.
Ephemeral storage mounted: Ensure your ephemeral storage (e.g., /raid0) is available.
Basic Linux knowledge: Familiarity with commands like tar, cron, and rsync.

Step-by-Step Instructions

Step 1: Verify Available Storage

Verify that both your ephemeral storage (/raid0) and the backup location (e.g., /mnt/shared) are properly mounted.

Check mounted file systems:

df -h

Ensure /raid0 (ephemeral storage) and /mnt/shared (backup disk) are listed with enough free space.

Check device mount points:

lsblk

This command displays devices and their mount points. Confirm that /raid0 is available and ready for backup.

Tip: If /raid0 is missing, mount it manually or check the RAID configuration.

Step 2: Create a Backup Directory (Optional)

Organize backups by creating a directory on the persistent storage:

sudo mkdir -p /mnt/shared/backup

This step is optional but improves organization, especially when managing multiple backups.

Step 3: Perform the Backup Using `tar`

To back up data in /raid0, create a compressed archive using the tar command. Add a timestamp to the filename to ensure uniqueness and avoid overwriting files.

Command to perform the backup:

sudo tar -czf /mnt/shared/backup/raid0_backup_$(date +%Y%m%d_%H%M%S).tar.gz -C / raid0

Explanation:

-czf: Compresses (z) and creates (c) an archive file.
/mnt/shared/backup/raid0_backup_<date_time>.tar.gz: Backup file with the current date and time.
-C /: Ensures tar starts from the root directory.
raid0: Specifies the mount point to back up.
$(date +%Y%m%d_%H%M%S): Adds a unique timestamp in the format YYYYMMDD_HHMMSS.

Example Filename:

raid0_backup_20241207_140000.tar.gz

This ensures no duplicates occur, even if backups run multiple times a day.

Step 4: Automate Backups Using `cron`

To ensure regular backups, set up a cron job for automation.

Edit the crontab:

sudo crontab -e

Add a cron job to back up /raid0 daily at 2:00 AM:

0 2 * * * tar -czf /mnt/shared/backup/raid0_backup_$(date +%Y%m%d_%H%M%S).tar.gz -C / raid0

Save and exit. The backup will run automatically at 2:00 AM daily.

Step 5: Implement Backup Retention Policies

To prevent persistent storage from filling up, delete older backups using a retention policy. Use the find command to automate this.

Example: Delete backups older than 7 days:

find /mnt/shared/backup/ -name "raid0_backup_*.tar.gz" -mtime +7 -exec rm {} \;

-mtime +7: Matches files older than 7 days.
-exec rm {}: Deletes the matched files.

Automate retention policy using cron: Add the following to your crontab to run the cleanup daily:

0 3 * * * find /mnt/shared/backup/ -name "raid0_backup_*.tar.gz" -mtime +7 -exec rm {} \;

This will run at 3:00 AM daily, ensuring backups older than 7 days are removed.

Step 6: Use Alternative Tools for Backups

While tar is simple, other tools may be more efficient depending on your needs:

rsync:
- Efficient for incremental backups.
- Copies only changed or new files.
Example Command:
```
rsync -avz /raid0/ /mnt/shared/backup/
```
Use -a for archive mode and -z for compression.
Backup to S3 or Cloud Storage: Many customers export data to external Cloud providers, such as S3.
- Use aws s3 cp to copy the archive to an S3 bucket.
Example Command:
```
aws s3 cp /mnt/shared/backup/raid0_backup_$(date +%Y%m%d_%H%M%S).tar.gz s3://my-backup-bucket/ephemeral-backups/
```
Replace my-backup-bucket with your S3 bucket name.

Tip: Ensure the AWS CLI is installed and configured with proper credentials.

Common Issues and Resolutions

1. Data Loss on Instance Termination:

Cause: Ephemeral data is erased when the instance stops.
Solution: Automate backups with cron and transfer them to persistent or external storage.

2. Insufficient Space on Backup Disk:

Solution: Check available space with df -h and clean up older backups using a retention policy.

3. High CPU Usage During Backup:

Cause: The tar compression (-z) uses significant CPU resources.
Solution: Schedule backups during off-peak hours to minimize impact.

4. Cron Job Not Executing:

Solution: Verify the cron schedule using grep CRON /var/log/syslog and ensure no syntax errors.

Additional Resources

By following this guide, you can efficiently back up ephemeral storage, automate backups, and manage retention to ensure important data is preserved without consuming excessive disk space.