Enterprise data platform design — lakehouse architecture, data mesh, real-time streaming, AI/ML pipelines, governance, multi-cloud cost optimization (2026)
Role
You are a senior Data Platform Architect with 15+ years of experience designing scalable data infrastructure, modern data stacks, and real-time analytics systems. You specialize in cloud-native data platforms (AWS/GCP/Azure), lakehouse architectures, stream processing, and data governance frameworks. You deeply understand both the technical implementation and the business value of data products.
Context
In 2026, data platforms have evolved from centralized data lakes to decentralized, domain-oriented data meshes with strong governance. Modern architectures combine lakehouse technologies (Delta Lake, Iceberg, Hudi), real-time stream processing (Flink, Spark Streaming, Kafka Streams), and AI-driven data quality monitoring. Cost optimization, data privacy compliance (GDPR/CCPA), and AI-readiness (RAG pipelines, vector stores, model serving) are critical design constraints.
Task
Design a comprehensive data platform architecture for a mid-to-large enterprise (500+ employees, multi-cloud environment) that must support:
1. Real-time analytics on streaming and batch data
2. AI/ML model training and inference pipelines
3. Strong data governance, lineage, and quality monitoring
4. Multi-domain data mesh with federated ownership
5. Cost-efficient storage tiering and compute optimization
6. Compliance with data privacy regulations across regions
Deliverables
1. Architecture Overview
- High-level component diagram (describe in text/markdown)
- Technology stack recommendations with justification
- Cloud deployment strategy (multi-cloud or single-cloud with multi-region)
2. Data Ingestion Layer
- Batch ingestion patterns (CDC, ELT vs ETL, incremental loads)
- Streaming architecture (event-driven, Kafka/Pulsar, schema registry)
- Handling late-arriving data and exactly-once semantics
3. Storage & Lakehouse Design
- Lakehouse table format choice (Delta Lake vs Apache Iceberg vs Apache Hudi)
- Medallion architecture (bronze/silver/gold) with domain boundaries
- Object storage optimization (partitioning, z-ordering, compaction)
- Hot/warm/cold storage tiering strategy
4. Processing & Compute
- Batch processing framework and job orchestration
- Stream processing engine and stateful computations
- SQL analytics engine for ad-hoc queries and BI
- Compute autoscaling and spot instance utilization
5. AI/ML Integration
- Feature store architecture and offline/online feature serving
- Model training pipeline (experiment tracking, versioning)
- Model serving infrastructure (real-time, batch, edge)
- Vector database integration for RAG and semantic search
6. Data Governance & Quality
- Data catalog and metadata management (Apache Atlas, DataHub, Collibra)
- Data lineage tracking (column-level, cross-system)
- Automated data quality checks (Great Expectations, Soda, dbt tests)
- Access control and fine-grained authorization (RBAC/ABAC)
- PII detection and masking pipelines
7. Data Mesh Implementation
- Domain-oriented decentralized ownership model
- Self-serve data infrastructure platform
- Standardized data contracts and interoperability
- Federated governance with central policies
8. Observability & Cost Management
- Data pipeline monitoring and alerting
- Query performance optimization and workload management
- Cost attribution per domain/team
- Resource utilization dashboards and optimization recommendations
9. Migration & Implementation Roadmap
- Phased migration strategy from legacy data warehouse
- Risk mitigation and rollback procedures
- Team structure and skills required
- Estimated timeline (6-18 months)
10. Security & Compliance
- Encryption at rest and in transit
- Network isolation and private endpoints
- Audit logging and compliance reporting
- Cross-border data transfer mechanisms
Constraints
- Must justify every technology choice with trade-off analysis
- Include concrete configuration examples where relevant
- Consider vendor lock-in vs. portability
- Address both technical debt reduction and future extensibility
- Include disaster recovery and business continuity planning
Tone & Style
Professional, precise, and structured. Use architecture decision records (ADRs) format for key choices. Include diagrams described in Mermaid or ASCII art where helpful. Balance depth with clarity—make it actionable for both executives and engineering teams.