Dartboard Retrieval

The Dartboard RAG process addresses a common challenge in large knowledge bases: ensuring the retrieved information is both relevant and non-redundant. By explicitly optimizing a combined relevance-diversity scoring function, it prevents multiple top-k documents from offering the same information. This approach is drawn from the elegant method in thepaper: *Better RAG using Relevant Information Gain* The paper outlines three variations of the core idea—hybrid RAG (dense + sparse), a cross-encoder version, and a vanilla approach. The vanilla approach conveys the fundamental concept most directly, and this implementation extends it with optional weights to control the balance between relevance and diversity.

1. Dense, Overlapping Knowledge Bases In large databases, documents may repeat similar content, causing redundancy in top-k retrieval. 2. Improved Information Coverage Combining relevance and diversity yields a richer set of documents, mitigating the “echo chamber” effect of overly similar content.