Why Chunking Fails

• Traditional RAG systems divide documents into fixed, arbitrary pieces, separating headings from the paragraphs they introduce and detaching tables from their explanatory text.1:02
• When a query requires synthesizing information across multiple nonadjacent sections, similarity search over flattened chunks fails because it cannot natively understand the original document’s hierarchy.

The Structure-First Alternative

• "Chunkless RAG" keeps the document tree intact, allowing an agent to start with an outline, navigate to specific branches, and follow references like footnotes to maintain context.5:16
• Docling, a reconstruction tool, converts raw PDFs into structured trees with preserved reading order and headings, which allows agents to reason over the document as an organization rather than a raw text file.

Engineering Constraints

• Constructing a clean, usable tree from real-world PDFs remains the primary engineering bottleneck, as PDFs are essentially sets of coordinate instructions rather than semantic documents.4:13
• Structure-aware retrieval is more resource-intensive than standard vector search, involving higher latency and more model calls for back-and-forth navigation.5:55

Hybrid Design

• Most professional systems should utilize a hybrid approach: similarity search for broad discovery across millions of documents, followed by structure-based navigation for deep analysis within a single, selected document.6:24