</>
bashintermediate

Cron Data Sync — Database to S3

Automated script to export database tables to compressed CSV and sync to S3 on a schedule.

bash
#!/usr/bin/env bash
set -euo pipefail

# Config
DB_URL="${DATABASE_URL}"
S3_BUCKET="s3://data-exports/daily"
DATE=$(date +%Y-%m-%d)
EXPORT_DIR="/tmp/exports/${DATE}"
TABLES=("users" "orders" "products" "events")

mkdir -p "$EXPORT_DIR"
log() { echo "[$(date '+%H:%M:%S')] $*"; }

log "Starting export for ${DATE}"

# Export each table
for table in "${TABLES[@]}"; do
  log "Exporting ${table}..."
  psql "$DB_URL" -c \
    "\\COPY (SELECT * FROM ${table} WHERE updated_at >= '${DATE}') TO STDOUT CSV HEADER" \
    | gzip > "${EXPORT_DIR}/${table}.csv.gz"

  rows=$(zcat "${EXPORT_DIR}/${table}.csv.gz" | wc -l)
  log "  ${table}: $((rows - 1)) rows exported"
done

# Sync to S3
log "Syncing to ${S3_BUCKET}/${DATE}/"
aws s3 sync "$EXPORT_DIR" "${S3_BUCKET}/${DATE}/" \
  --storage-class STANDARD_IA \
  --only-show-errors

# Cleanup old local exports (keep 7 days)
find /tmp/exports -maxdepth 1 -type d -mtime +7 -exec rm -rf {} +

# Verify upload
count=$(aws s3 ls "${S3_BUCKET}/${DATE}/" | wc -l)
log "Verified: ${count} files uploaded to S3"
log "Export complete"

# Crontab entry: 0 2 * * * /opt/scripts/data-sync.sh >> /var/log/data-sync.log 2>&1

Sponsored

Cloudflare R2

Use Cases

  • Nightly database exports to cloud storage
  • Automated data lake ingestion from PostgreSQL
  • Scheduled incremental data backups

Tags

#bash#cron#s3#backup#data-engineering

Related Snippets

Similar patterns you can reuse in the same workflow.

bashbeginner

Database Backup and Restore to S3

Automated PostgreSQL backup script with compression, S3 upload, retention policy, and restore commands.

Best for: Automated daily database backups to S3

#bash#backup
bashintermediate

Bash ETL Pipeline Script

Build a complete ETL script in Bash with logging, error handling, notifications, and idempotent runs.

Best for: Automating daily data extract and load jobs

#bash#etl
bashadvanced

Spark Submit — Job Launcher Script

Launch PySpark jobs with spark-submit including cluster configuration, dependencies, and monitoring.

Best for: Launching PySpark batch jobs on YARN clusters

#spark#pyspark
bashbeginner

dbt Run and Test — CI/CD Pipeline Script

Bash script for running dbt build with testing, documentation generation, and failure notifications.

Best for: Automating dbt builds in CI/CD pipelines

#dbt#bash