Design a Distributed Search Engine

Requirements

• •Index documents with an inverted index supporting term-based lookup and BM25 relevance scoring
• •Shard the index across multiple nodes to distribute storage and query load
• •Execute queries via scatter-gather: fan out to all shards, merge partial top-K results
• •Support incremental indexing (new documents) without rebuilding the full index
• •Handle node failures gracefully by maintaining index replicas

Non-Functional Requirements

• •Query latency < 200ms (p99) for queries spanning 10 billion docs
• •Index throughput: 50,000 documents/second
• •Near-real-time indexing: new docs searchable within 60 seconds
• •System tolerates loss of 1 shard without data loss (replication factor 2)

Constraints

• •Corpus: 10 billion documents, average size 5KB
• •Queries are keyword-based full-text search with BM25 ranking
• •Near-real-time indexing: new documents should be searchable within 60 seconds

Evaluation Criteria