Skip to content
    AI engineering roles via the DiamantAI Collective.See open roles
    FoundationalRAG Techniques

    Proposition Chunking

    This code implements the proposition chunking method, based on research from Tony Chen, et. al.. The system break downs the input text into propositions that are atomic, factual, self-contained, and concise in nature, encodes the propositions into a vectorstore, which can be later used for retrieval

    The motivation behind the propositions chunking method is to build a system that breaks down a text document into concise, factual propositions for more granular and precise information retrieval. Using propositions allows for finer control and better handling of specific queries, particularly for extracting knowledge from detailed or complex texts. The comparison between using smaller proposition chunks and larger document chunks aims to evaluate the effectiveness of granular information retrieval.

    What you'll learn

    • 1
      Document Chunking: Splitting a document into manageable pieces for analysis.
    • 2
      Proposition Generation: Using LLMs to break down document chunks into factual, self-contained propositions.
    • 3
      Proposition Quality Check: Evaluating generated propositions based on accuracy, clarity, completeness, and conciseness.
    • 4
      Embedding and Vector Store: Embedding both propositions and larger chunks of the document into a vector store for efficient retrieval.
    • 5
      Retrieval and Comparison: Testing the retrieval system with different query sizes and comparing results from the proposition-based model with the larger chun…

    About this tutorial

    This hands-on Jupyter notebook is part of RAG Techniques, a free open-source repository by Nir Diamant covering rag techniques with runnable code examples and detailed explanations.

    Free and open-sourceRunnable Jupyter notebookActive community support
    Go deeper · Amazon Bestseller in Generative AI

    RAG Made Simple

    The book that extends this repo: 22 RAG techniques with the intuition behind each, side-by-side comparisons of when each wins (and quietly fails), and original illustrations.

    Get it on Amazon

    ⭐ 4.4 stars · 1,500+ readers · Kindle $9.99 · Paperback $24.99 · Free with Kindle Unlimited

    More Foundational tutorials

    More from RAG Techniques