</>
bashbeginner

dbt Run and Test — CI/CD Pipeline Script

Bash script for running dbt build with testing, documentation generation, and failure notifications.

bash
#!/usr/bin/env bash
set -euo pipefail

# Config
DBT_PROJECT_DIR="${1:-.}"
DBT_TARGET="${DBT_TARGET:-production}"
SLACK_WEBHOOK="${SLACK_WEBHOOK_URL:-}"

log() { echo "[$(date '+%H:%M:%S')] $*"; }

send_slack() {
  [[ -z "$SLACK_WEBHOOK" ]] && return
  curl -sf -X POST "$SLACK_WEBHOOK" \
    -H 'Content-Type: application/json' \
    -d "{\"text\": \"$1\"}"
}

cd "$DBT_PROJECT_DIR"

# 1. Check source freshness
log "Checking source freshness..."
if ! dbt source freshness --target "$DBT_TARGET" 2>&1; then
  send_slack "⚠️ dbt source freshness warning — some sources may be stale"
fi

# 2. Run models
log "Running dbt models..."
if ! dbt run --target "$DBT_TARGET" --full-refresh 2>&1; then
  send_slack "❌ dbt run FAILED on target ${DBT_TARGET}"
  exit 1
fi

# 3. Run tests
log "Running dbt tests..."
TEST_OUTPUT=$(dbt test --target "$DBT_TARGET" 2>&1)
TEST_EXIT=$?
echo "$TEST_OUTPUT"

FAILED=$(echo "$TEST_OUTPUT" | grep -c 'FAIL' || true)
PASSED=$(echo "$TEST_OUTPUT" | grep -c 'PASS' || true)

if [[ $TEST_EXIT -ne 0 ]]; then
  send_slack "❌ dbt tests: ${FAILED} failed, ${PASSED} passed"
  exit 1
fi

# 4. Generate docs
log "Generating documentation..."
dbt docs generate --target "$DBT_TARGET"

log "✅ dbt pipeline complete — ${PASSED} tests passed"
send_slack "✅ dbt pipeline complete — ${PASSED} tests passed, 0 failures"

Sponsored

dbt Cloud

Use Cases

  • Automating dbt builds in CI/CD pipelines
  • dbt testing with Slack failure notifications
  • Scheduled dbt runs with documentation generation

Tags

#dbt#bash#ci-cd#testing#data-engineering

Related Snippets

Similar patterns you can reuse in the same workflow.

sqlbeginner

dbt Source Freshness and Testing

Configure dbt source freshness checks and schema tests to validate upstream data pipelines.

Best for: Ensuring upstream data sources are fresh

#dbt#testing
sqlintermediate

dbt Incremental Model Pattern

Build efficient dbt incremental models that process only new or changed data instead of full refreshes.

Best for: Efficient data warehouse builds processing only deltas

#dbt#incremental
sqlintermediate

SQL Data Quality Checks and Assertions

Reusable SQL queries for data quality: null checks, uniqueness, referential integrity, and freshness.

Best for: Automated data quality gates in ETL pipelines

#sql#data-quality
bashintermediate

Bash ETL Pipeline Script

Build a complete ETL script in Bash with logging, error handling, notifications, and idempotent runs.

Best for: Automating daily data extract and load jobs

#bash#etl