Design a Log Collection Agent

Requirements

• •Agent collects logs from applications and buffers them locally
• •Agent flushes batches to the central collection service on a schedule or when buffer is full
• •Central service stores logs in an append-only, queryable store
• •Support filtering logs by service name, severity level, and time range
• •Logs should be retained for 30 days then auto-deleted

Non-Functional Requirements

• •Agent should add < 1ms overhead to the application process
• •Log delivery within 10 seconds of generation (best-effort)
• •System ingests ~100,000 log lines/minute across all agents
• •No log loss on agent crash if buffer is flushed to disk

Constraints

• •Logs are structured (JSON lines format)
• •No real-time streaming query required — batch log search is sufficient
• •Agents run on application servers and forward logs to a central backend

Evaluation Criteria