Prompt

Comprehensive data processing and ETL workflow automation:

**Data Pipeline Design:**
- Analyze data sources and create extraction workflows
- Design data transformation and cleaning operations
- Implement data validation and quality checks
- Create data loading and storage optimization

**Automated ETL Workflows:**
- Set up scheduled data processing jobs
- Implement real-time data streaming and processing
- Create data pipeline monitoring and alerting
- Design error handling and recovery mechanisms

**Data Quality Management:**
- Implement automated data validation and testing
- Set up data lineage tracking and documentation
- Create data profiling and anomaly detection
- Design data governance and compliance workflows

**Performance Optimization:**
- Optimize data processing performance and scalability
- Implement parallel processing and distributed computing
- Create data partitioning and indexing strategies
- Design caching and materialized view optimization

**Data Integration & API Management:**
- Create data API endpoints and documentation
- Implement data synchronization across systems
- Set up data versioning and change management
- Design data access control and security

**Example Data Operations:**
```bash
# Data pipeline execution
python -m luigi --module data_pipeline ProcessData --date $(date +%Y-%m-%d)
# Data quality checks
great_expectations checkpoint run data_quality_checkpoint
# Data transformation
dbt run --models +customer_analytics
# API data ingestion
curl -X POST api/data/ingest -H "Content-Type: application/json" -d @data.json
```

Provide complete data pipeline workflows with quality assurance and monitoring.