Proposition Chunking
This code implements the proposition chunking method, based on research from Tony Chen, et. al.. The system break downs the input text into propositions that are atomic, factual, self-contained, and concise in nature, encodes the propositions into a vectorstore, which can be later used for retrieval
The motivation behind the propositions chunking method is to build a system that breaks down a text document into concise, factual propositions for more granular and precise information retrieval. Using propositions allows for finer control and better handling of specific queries, particularly for extracting knowledge from detailed or complex texts. The comparison between using smaller proposition chunks and larger document chunks aims to evaluate the effectiveness of granular information retrieval.
What you'll learn
- 1Document Chunking: Splitting a document into manageable pieces for analysis.
- 2Proposition Generation: Using LLMs to break down document chunks into factual, self-contained propositions.
- 3Proposition Quality Check: Evaluating generated propositions based on accuracy, clarity, completeness, and conciseness.
- 4Embedding and Vector Store: Embedding both propositions and larger chunks of the document into a vector store for efficient retrieval.
- 5Retrieval and Comparison: Testing the retrieval system with different query sizes and comparing results from the proposition-based model with the larger chun…
About this tutorial
This hands-on Jupyter notebook is part of RAG Techniques, a free open-source repository by Nir Diamant covering rag techniques with runnable code examples and detailed explanations.
RAG Made Simple
The book that extends this repo: 22 RAG techniques with the intuition behind each, side-by-side comparisons of when each wins (and quietly fails), and original illustrations.
Get it on Amazon⭐ 4.4 stars · 1,500+ readers · Kindle $9.99 · Paperback $24.99 · Free with Kindle Unlimited
