Small Files Problem

The performance and metadata overhead caused by storing very large numbers of undersized files in analytical data systems.

Why it matters

Small Files Problem helps data teams design systems that are reliable, understandable, governable, and efficient at scale.

Design considerations

• Define ownership, contracts, and expected consumers.
• Make failure, replay, compatibility, and observability behavior explicit.
• Measure quality, freshness, cost, and operational burden rather than optimizing only throughput.