Design a Data Lake Platform: System Design Challenge

Design a Data Lake Platform

Requirements

•Ingest data from streaming sources (Kafka) and batch sources (nightly DB exports) into the lake
•Store tables in a columnar format (Parquet) with a metadata layer providing ACID transactions and time travel
•Support schema evolution: add nullable columns to existing tables without rewriting all data
•Run background compaction to merge small files into optimally-sized Parquet files
•Provide a metadata catalog: register tables, track schemas, partitions, and lineage; query tables via SQL

•Query latency: < 30 seconds for analytical queries on 1PB tables (with partition pruning)
•Streaming ingestion lag < 5 minutes from Kafka to queryable
•Schema changes (add column) applied < 1 second with zero read downtime
•Compaction reduces small file count by 90% within 1 hour

•Storage layer: object storage (S3-compatible)
•Compute: separate from storage (decoupled query engines like Spark, Trino, Athena)
•Data types: structured (CDC from OLTP DBs) and semi-structured (JSON event logs)

Evaluation criteria and weights will be revealed after you submit your solution. This ensures a fair assessment of your design skills.

Draw your architecture diagram, explain your design decisions, and get AI-powered feedback.

For the best experience, design challenges on a desktop or tablet — drawing the architecture on a phone is hard.

Start drawing nowNo account needed to draw. Sign up only to see your score (2 free evaluations).Already have an account? Sign in