Adaptive Aware Chunking: Improving RAG Retrieval

Imagine asking an AI system about the toxic effects of heavy metals in bats, only for it to return research about rats. Both involve mercury, liver damage and oxidative stress, so the passages look remarkably similar. But one describes the wrong animal, making it unsuitable evidence for the question.

‍

‍

This is a problem with Retrieval Augmented Generation (RAG), a technique that allows AI models to answer questions using information retrieved from external documents. In a recent Passion Academy session, Timothy Abayo explored how the way we split and organise documents can affect what a RAG system retrieve  and how adaptive aware chunking can help it find the right evidence.

‍

‍

Slides

‍

‍

‍

‍

‍

‍

The Problem: Relevant Text, Wrong Answer

‍

‍

RAG works by searching a collection of documents for information relevant to a question, then passing the most useful passages to a language model to help generate an answer.

‍

To make this search efficient, documents are usually divided into smaller sections called chunks. These chunks are converted into numerical representations called embeddings, which allow the system to compare their meaning with a user's question.

‍

The challenge is that two passages can be semantically similar without being relevant in exactly the same way.

‍

Consider a researcher asking: “What are the toxic effects of heavy metals in bats?”

‍

A standard retrieval system might return these two findings:

‍

• Mercury causes liver injury in rats.
• Heavy metals accumulate in bat tissue.

‍

Both passages discuss heavy metals and their effects on animals. If the system primarily ranks results by semantic similarity, the rat passage could appear more relevant, even though it does not answer the question about bats.

‍

This is more than a search problem. In research, law, finance and other fields where precise attribution matters, retrieving information about the wrong entity can lead to misleading conclusions.

‍

The underlying issue is that a system can recognise what a passage is about in general without correctly identifying who or what the information relates to.

‍

‍

Why Standard Chunking Loses Context

‍

The problem often begins before the search even takes place.

‍

A common approach to chunking is to split documents into sections of a fixed size, measured in tokens. Tokens are the units of text that language models process. The system might cut a document every 300 tokens, regardless of whether that is a natural place for the text to end.

‍

Imagine a scientific paper with a section titled Heavy Metal Toxicology in Bats. The first chunk contains the heading, but the following chunks contain the actual findings:

• “Lead, mercury and cadmium accumulate in tissues over time.”
• “Oxidative stress then causes liver dysfunction.”
• “Long term exposure weakens the immune response.”

‍

Once these passages are separated from their heading, the connection to bats may no longer be explicit. The findings remain understandable, but the information identifying the species they describe has been lost.

‍

When the system later searches for information about heavy metals, it may struggle to distinguish these passages from similar findings about rats.

‍

This is where the limitations of fixed size chunking become apparent. It treats text as a sequence of tokens rather than something with structure, relationships and context.

‍

‍

The Solution: Adaptive Aware Chunking

‍

The approach explored in Timothy's presentation combines two ideas: adapting how documents are divided and preserving information about the entities discussed in each chunk.

‍

Rather than applying the same chunk size to every document, the system takes the structure of the content into account. It also attaches contextual information, such as the document, section and species, so that the identity of the subject is retained even when it is not repeated in the passage itself.

‍

The process has four main stages.

‍

1. Split documents according to their structure

‍

Instead of cutting text at arbitrary intervals, the system uses section boundaries and the structure of the document to determine where chunks should begin and end.

‍

Chunk sizes can therefore vary between documents. The aim is to keep related information together rather than forcing every passage into the same length.

‍

This does not mean that every section will automatically form a perfect chunk. It means that document structure becomes part of the decision about where to split the text.

‍

‍

2. Attach contextual information to each chunk

‍

Each chunk receives metadata identifying its source, section and relevant species.

‍

For example:

• ‍Document: Bat Toxicology‍
• Section: Heavy Metals‍
• Species: Bats‍
• Text: Lead and mercury accumulate in tissues over time.

‍

The passage no longer needs to repeat the word “bats” for the system to know which species the finding concerns. That information is stored alongside the text and can be used during retrieval.

‍

This is particularly useful when a document introduces a subject once and then refers to it indirectly in subsequent paragraphs.

‍

‍

3. Identify what the question is asking

‍

The system then considers the question itself, including whether it names a specific entity and what kind of answer it requires.

‍

If someone asks about bats, evidence about bats should be prioritised. If they ask for a comparison between bats and rats, both become relevant. If they ask a general question about heavy metals, the system should be able to draw on evidence about different species.

‍

The important distinction is that the system does not simply filter out everything that fails to match a particular entity. It adapts its approach according to the question.

‍

‍

4. Rerank the retrieved passages

‍

Finally, the system adjusts the ranking of candidate passages using additional information beyond semantic similarity.

‍

Timothy's presentation outlines four signals:

‍

• Semantic similarity: How closely does the passage relate to the meaning of the question?
• Entity match: Does it concern the animal or other entity the user asked about?
• Document context: Does it come from a relevant source?
• Section heading: Does it belong to a section relevant to the question?

‍

These signals are combined into a score that helps determine which passages should rank highest.

‍

The distinction is important because semantic similarity remains useful. The aim is not to replace it, but to supplement it with information that helps the system distinguish between passages that discuss similar topics.

‍

‍

‍

One Approach, Three Types of Questions

‍

The value of question aware retrieval becomes clearer when we consider how the system should respond to different questions.

‍

1. When the question names one entity

‍

“What are the effects of heavy metals in bats?”

‍

The system prioritises findings about bats and demotes passages about rats, even if the language is very similar.

‍

The aim is to reduce contamination, meaning the proportion of retrieved evidence that concerns the wrong entity.

‍

‍

2. When the question compares entities

‍

“What are the effects of mercury in bats compared with rats?”

‍

Here, removing rat related evidence would be counterproductive. The system needs to retrieve findings about both species and ensure that neither side of the comparison is overlooked.

‍

The focus shifts from eliminating irrelevant entities to providing balanced coverage while preserving the correct attribution of each finding.

‍

‍

3. When the question is general

‍

“What are the effects of heavy metals?”

‍

The question does not specify an animal. The system should therefore be able to retrieve relevant evidence from across different species rather than favouring one simply because of an entity match.

‍

However, it should still preserve the context of each finding so that the answer does not blur together results from different animals.

‍

The broader principle is that retrieval needs to be sensitive to what the user is asking, rather than applying the same ranking rules to every question.

‍

‍

‍

The Harder Case: When One Document Covers Multiple Entities

‍

Preserving context becomes more challenging when a single document discusses several subjects.

‍

Consider a zoology paper that begins by describing rats, then explains how they groom each other and build nests. Later, it switches to bats and then hedgehogs.

‍

The text might refer to rats as “they” or “their” rather than repeating the word “rats” in every paragraph. A system that processes each paragraph in isolation could lose track of which animal is being discussed.

‍

The approach presented addresses this by carrying the relevant subject through consecutive paragraphs until a new subject is introduced.

‍

In this example, the first three paragraphs can be grouped into a rat related chunk. The paragraph introducing bats starts a new chunk and the paragraph about hedgehogs starts another. The following paragraph about hibernation inherits the hedgehog context.

‍

This is more than attaching a label to an entire document. The document is about zoology, but individual passages concern different animals.

‍

A document level label alone cannot distinguish between them. The system needs to preserve context at the level where the information is actually used.

‍

‍

How Would We Measure the Difference?

‍

Improving retrieval requires looking beyond whether the returned passages contain similar words.

‍

The presentation illustrated this using a scoring example. With cosine similarity alone, a rat passage ranks first with a score of 0.93, while the bat passage scores 0.91. After adding entity and contextual signals, the bat passage ranks first at 0.96, while the rat passage falls to 0.61.

‍

These scores are illustrative, not measured experimental results. They demonstrate how additional context could change the ranking.

‍

The presentation also proposed contamination as a useful measure of retrieval quality. If a system retrieves five passages for a question about bats and two concern rats, contamination is 40%.

‍

For a question about one animal, the target is zero contamination from other animals. For comparison questions, the evaluation also needs to consider whether both subjects are represented adequately. For general questions, the focus shifts to coverage and correct attribution.

‍

This highlights an important point when evaluating RAG systems: retrieving text that looks relevant is not enough. The evidence also needs to be appropriate for the question.

‍

‍

What This Means for Building Better RAG Systems

‍

Adaptive aware chunking addresses a specific part of RAG: how information is prepared and retrieved before it reaches the language model.

‍

It does not eliminate every source of error. The system still needs to identify entities correctly, recognise when the subject changes and preserve the relationships between paragraphs. If these steps go wrong, chunks may receive incorrect labels or lose important context.

‍

There is also a practical consideration. More sophisticated document processing introduces additional complexity and the benefits will depend on the documents being used and the accuracy required by the application.

‍

For a general purpose assistant, retrieving a passage about a related topic might sometimes be sufficient. For a system working with scientific literature, legal documents or financial records, confusing two similar entities could have much more serious consequences.

‍

The lesson is not that every RAG system needs the same chunking strategy. It is that the way information is structured should reflect the demands of the task.

‍

If a system needs to distinguish between similar entities, document preparation and retrieval need to account for those distinctions from the beginning.

‍

‍

The Takeaway

‍

RAG is often discussed in terms of which language model to use or how to improve the final answer. However, what happens before the model generates a response can be just as important.

‍

If a document is split in a way that strips away the context surrounding a finding, the retrieval system may return evidence that looks right but answers the wrong question.

‍

Adaptive aware chunking offers a way to address this by combining more intelligent document splitting, contextual metadata and question sensitive ranking.

‍

For anyone building RAG applications, it is worth looking beyond the language model itself. Better answers depend not only on how well a model reasons, but also on whether it has been given the right evidence to work with.

‍

‍

References

‍

• de Moura Júnior, P. R., Lelong, J., and Blangero, A. (2026). Adaptive Chunking: Optimizing Chunking Method Selection for RAG. LREC 2026. arXiv:2603.25333
• Bora, C. B., Alatlı, O., Bora, S., and Dikenelli, O. (2026). EAR: Entity Aware Partitioning Approach for Retrieval Augmented Generation Development. arXiv:2609.12268

‍