Prompt

Design data pipeline architecture for analytics.

<data_sources>
{list all data sources and volumes}
</data_sources>

<processing_requirements>
- Latency: {real-time/batch}
- Volume: {data per day}
- Transformations: {describe}
</processing_requirements>

<output_requirements>
{data warehouse/lake/real-time dashboard}
</output_requirements>

Design pipeline:
1. Ingestion architecture
2. Processing framework choice
3. Storage architecture
4. Data quality checks
5. Orchestration approach
6. Monitoring and alerting

Compare stream vs batch processing.