What Chunkless RAG Is—and Where It Beats Flat Chunking

RAG made models easier to ground in external documents. The first wave of implementations also made a simplifying assumption that is now looking expensive: flatten the document, cut it into chunks, embed the chunks, and hope similarity search can recover the author’s intent.

Chunkless RAG starts from a different premise. For dense, structured documents, the problem is often not retrieval power but retrieval shape. If preprocessing destroys headings, tables, references, and section hierarchy, the model is asked to reason over fragments that no longer preserve how the source was written. ^{1, 2}

Why flat chunking is losing credibility

The core critique is straightforward: conventional RAG relies on chunking, and chunking flattens documents into disjointed fragments. In the process, headings get separated from their paragraphs and tables get detached from the surrounding explanation. That is the kind of context loss that can lead to invented joins between ideas. > "Conventional Retrieval-Augmented Generation relies on chunking, which flattens long documents into disjointed text fragments and destroys the author’s intended structure."

— 1 Minute Signal coverage of IBM Technology ¹

This is not just a theory. Multiple sources converge on the same practical point: fixed or semantic chunking can be fragile, expensive, and inconsistent, with quality depending heavily on document type and implementation details. One evaluation found that chunking is “a much more difficult and fragile problem than is often assumed,” while another concluded that the compute cost of semantic chunking is not justified by consistent performance gains. ^{3, 4}

The deeper issue is that “better chunking” still treats each fragment as an isolated retrieval object. That means the system may retrieve the right sentence while missing the section context that explains what the sentence means. ^{5, 6}

What chunkless RAG actually does

Chunkless RAG keeps the document’s structure intact. Instead of pretending a PDF is a bag of text spans, it preserves the tree: titles, headers, sections, page ranges, and cross-references. An agent can then navigate the source more like a human reader using a table of contents than a similarity engine scanning nearest neighbors. ^{2, 7}

That also makes the comparison to adjacent ideas clearer. Contextual embeddings still produce vectors for chunks; they just try to inject more surrounding context into those vectors. Adaptive chunking still chunks; it merely chooses boundaries more intelligently. Chunkless or vectorless retrieval goes further by making document structure itself the retrieval substrate. ^{6, 8, 9}

IBM Technology coverage frames the implementation around preserving the document tree so the agent can follow the original sections, headings, and references. Docling is one enabling tool cited for turning raw PDFs into structured trees with preserved reading order and headings. ¹

"Chunkless RAG challenges this by maintaining the document’s original tree structure. When tasked with answering a specific question, an AI agent navigates through this tree, just as you might thumb through a book’s table of contents to locate specific information."

— 1 Minute Signal coverage of Frank's World ²

The architecture varies, but the pattern is consistent: retrieve the relevant section or node, then answer from that section in context. In some versions, such as Vectorless RAG or PageIndex-style systems, the retrieval step is explicitly tree-based and reasoning-driven rather than embedding-driven. ^{9, 10}

Why static vector search starts to look like the wrong abstraction

Static vector search is good at similarity. It is weaker at structure. That matters because relevance in enterprise documents often depends on references, hierarchy, and relationships that are not obvious from a single chunk alone. BuildFastWithAI puts it plainly: “similarity does not equal relevance.” ¹⁰

"Vectorless RAG is a retrieval approach that replaces semantic similarity search with LLM-powered reasoning over a structured document index. No embeddings, no vector database, no approximate nearest-neighbor search."

— BuildFastWithAI ¹⁰

M-RAG makes a related move from another angle: it decouples retrieval representation from generation content so retrieval can stay lightweight while generation preserves context-rich information. ¹¹

That is why chunkless systems are attractive for annual reports, contracts, policy docs, and other long structured sources. If the question is local and the corpus is small enough to reason over, navigating a tree can outperform nearest-neighbor search because it preserves logical boundaries rather than approximating them with token counts. ^{7, 10}

Fireship’s coverage of Turso is a useful parallel from the storage layer: the appeal of native vector search is that it removes an extra system and embeds embeddings directly into the core file structure. That kind of integration can simplify a stack, but it also shows the basic trade-off. Native search works well when you want one compact retrieval layer; structure-aware retrieval works better when the document’s shape matters more than the embedding similarity itself. ¹²

The catch: chunkless is not free

This is where the trade-off matters for founders. Chunkless RAG is not a universal replacement for vector search. It is a different engineering budget.

Structure-aware systems tend to be more expensive per query. They often require more model calls, more parsing work, and more elaborate debugging of retrieval traces. One source notes that engineering difficulty shifts away from vector tuning and toward PDF parsing, model-call budgets, and trace debugging. Another warns that converting real-world PDFs into a clean tree remains a primary bottleneck because PDFs are coordinate instructions, not semantic documents. ^{1, 9}

"Most importantly, it shifts the engineering difficulty. You stop fighting vector search tuning and start fighting PDF parsing, model-call budgets, and trace debugging."

— Anubhav ⁹

There is also a scope limit. Tree-based or vectorless retrieval tends to fit long, structured, reasoning-heavy corpora better than huge heterogeneous ones. Traditional vector RAG still wins on latency-sensitive or at-scale retrieval across broad content sets. DevShelfHub’s summary is the most useful operational rule here: vectorless RAG is a sharper tool for a different job, not a blanket replacement. ^{7, 10}

What teams should do next

If you are building for high-precision retrieval on dense, structured documents, stop assuming flat chunking is the baseline. The strongest sources here all point toward the same conclusion: static chunking can fracture meaning, while tree-based or context-aware retrieval can preserve it. ^{1, 2, 11}

But most production systems should not swing all the way to one extreme. The cleaner pattern is hybrid: use similarity search for broad discovery, then use structure-based navigation for deep analysis. That recommendation shows up directly in the source set, and it fits the economics. Vector search is fast and scalable; chunkless reasoning is more precise but more expensive. ^{1, 7}

"If you are building for high-precision retrieval on dense, structured documents like annual reports or long-form policy, stop relying on flat chunking."

— 1 Minute Signal coverage of IBM Technology ¹

For builders, the decision is less “Should we replace vector search?” and more “Where does structure matter enough to justify the extra cost?”

A sensible sequencing looks like this:

1. Keep vector search for broad, high-throughput discovery.
2. Add structure-aware retrieval where the source material is dense, hierarchical, or citation-sensitive.
3. Treat parsing quality as a first-class system dependency, not a preprocessing detail.
4. Measure whether your users need similarity or reasoning, because those are not the same problem. ^{1, 7, 10}

Chunkless RAG is a better fit than flat chunking when the document’s structure carries meaning. For heterogeneous or latency-sensitive corpora, vectors still win. For dense structured documents, chunkless retrieval is often the better fit.