#pipeline
17 snippets tagged with #pipeline
Node.js Stream Pipeline with Transform
Build efficient data processing pipelines using Node.js streams for large file handling.
Best for: Processing large CSV files without loading into memory
Streams: Readable, Writable, Transform
Build custom readable, writable, and transform streams for efficient data processing in Node.js.
Best for: Processing large files without loading into memory
Generator Pipeline for Data Processing
Chain generators to build memory-efficient data processing pipelines for large files and streams.
Best for: Large file ETL
AI Prompt Chaining Pattern
Chain multiple LLM calls sequentially where each step's output feeds into the next for complex tasks.
Best for: Complex multi-step AI workflows
Hugging Face Transformers Pipeline
Run text classification, NER, summarisation, and translation tasks with the HF Pipelines API.
Best for: NLP tasks
LangChain Sequential Multi-Step Chain
Build a multi-step reasoning pipeline where each step's output feeds into the next chain.
Best for: multi-step AI pipelines
Haystack Question Answering Pipeline
Build a document retrieval and Q&A pipeline using Haystack 2.0 with OpenAI backend.
Best for: enterprise RAG
Custom sklearn Pipeline with Transformer
Build a custom scikit-learn Pipeline with a custom BaseEstimator Transformer for data preprocessing.
Best for: custom preprocessing
Flyte ML Pipeline in Python
Define a reproducible machine learning workflow with Flyte's Python SDK for data-to-model pipelines.
Best for: ML orchestration
GitHub Actions CI/CD Pipeline
Complete GitHub Actions workflow with test, build, and deploy stages for a Node.js application.
Best for: Automated testing and deployment on push
Python ETL Pipeline Example
Complete extract-transform-load pipeline with error handling, logging, and incremental processing.
Best for: Automating data ingestion from CSV to warehouse
Retry Logic for Data Pipelines
Configurable retry decorator with exponential backoff and jitter for resilient data pipeline tasks.
Best for: Resilient API calls in data pipelines
Databricks Notebook Data Pipeline
Databricks notebook with Delta Lake reads, transformations, merge operations, and table optimization.
Best for: Medallion architecture data pipelines on Databricks
Bash ETL Pipeline Script
Build a complete ETL script in Bash with logging, error handling, notifications, and idempotent runs.
Best for: Automating daily data extract and load jobs
Bash Pipeline Monitoring and Alerting
Monitor data pipeline health with row counts, runtime tracking, SLA checks, and Slack alerting.
Best for: Monitoring data pipeline health and freshness
Dataclasses as Pipeline Data Models
Use Python dataclasses to define typed, immutable data models passed between pipeline stages.
Best for: typed pipeline stages
attrs Classes as Immutable Pipeline Records
Use attrs to create fast, validated, immutable record types for data pipeline stage outputs.
Best for: typed pipeline records