Skip to content

awesome_rag.md

Latest commit

 

History

History
438 lines (389 loc) · 31.4 KB

awesome_rag.md

File metadata and controls

438 lines (389 loc) · 31.4 KB
Raw

Survey

RAG

Practice

Routing

In response to diverse queries, the RAG system routes to specific pipelines tailored for different scenario, a feature essential for a versatile RAG architecture designed to handle a wide array of situations.

Adaptive Retrieval / Query Classification

Not all queries require retrieval-augmented due to the inherent capabilities of LLMs. Queries requiring retrieval proceed through the RAG modules; others are handled directly by LLMs.

Refine Query\Query Rewriting\Query Disambiguation\Query Decomposition

legacy

“The internet as a knowledge base” setup

  • Tue, 23 May 2023 Query Rewriting for Retrieval-Augmented Large Language Models
  • Sun, 31 Mar 2024 RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation
    • As demonstrated in previous research (Shi et al., 2023a), irrelevant context not only diminishes generation quality but may also obstruct LLMs’ ability to answer queries they are otherwise capable of addressing.
        1. Adaptive Retrieval ) For straightforward queries, LLMs should respond directly rather than incorporating unnecessary context
        1. Query Decomposition ) for complex queries, simply searching with the original query often fails to retrieve adequate information. It’s crucial for LLMs to first break down such queries into simpler, answerable sub-queries, and then search for information relevant to these sub-components
        1. Query Rewriting\Query Disambiguation ) for ambiguous queries with multiple possible answers, using the original query for information retrieval is insufficient. To provide complete and nuanced responses, LLMs must learn to clarify the query, ideally by identifying the user’s intent, and then craft a more targeted search query
  • Mon, 15 Jul 2024 Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems
    • Question Rewriter+ -> Retrieval Trigger -> Knowledge Retriever -> Knowledge Filter -> LLM Reader -> Memory Knowledge Reservoir
    • (1) single query have an inherent upper limit of retrievable relevant information;
    • (2) employing multiple queries that focus on different semantic aspects can surpass the information plateau, enhancing both the precision and recall of information retrieval;
    • (3) The phenomenon of irrelevant knowledge is pervasive in RAG and becomes more pronounced with larger volumes of retrieved external information; and
    • (4) rewriting ambiguous questions into intent-specific questions improves the precision of responses.

"fixed scene" setup

  • Thu, 18 Jan 2024 ChatQA: Surpassing GPT-4 on Conversational QA and RAG
    • Query Rewriting 并没有增加新的信息，最新的 Retrieval(Embedding) Model 大小来到了7B 甚至更大，基础模型也使用LLM as Retrieval。
    • Conversational Query Rewriting Most of the previous solutions are query rewriting methods.
      • The latest turn of question is rewritten to be a standalone query without additional information from previous dialogue history (Vakulenko et al., 2021a; Ye et al., 2023; Mo et al., 2023),
      • so it can be directly used by retrieval model to retrieve relevant context (Vakulenko et al., 2021b; Mele et al., 2021; Raposo et al., 2022; Mo et al., 2023).
      • 使用大 retrieval model 的效果与使用 GPT-3.5-Turbo Query Rewriting 相当
    • fine-tuning a good single-turn retriever on high-quality conversational query context pairs performs on par with leveraging the state-of-the-art rewriter.
    • However, rewriting method requires extra computational time for autoregressive generation process and probably also API cost for using powerful models like GPT-3.5-Turbo.
  • Mon, 1 Jul 2024 Searching for Best Practices in Retrieval-Augmented Generation
    • However, query rewriting and query decomposition did not enhance retrieval performance as effectively
  • Sat, 20 Jul 2024 Golden-Retriever: High-Fidelity Agentic Retrieval Augmented Generation for Industrial Knowledge Base
    • 处理行话
  • Mon, 9 Sep 2024 MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery
    • RQ-RAG 效果比较差

others

Chunking / Chucking Granularity / Chunking Techniques

Retrieval / Embedding Model

跳转

Vector Databases

Generation-Augmented Retrieval

你没看错，是生成增强检索

Reranking

跳转

Retrieval evaluator

  • Mon, 29 Jan 2024 Corrective Retrieval Augmented Generation
    • a lightweight retrieval evaluator is designed to assess the overall quality of retrieved documents for a query, returning a confidence degree based on which different knowledge retrieval actions can be triggered.
    • Confidence: CORRECT INCORRECT AMBIGUOUS

Summarization

Retrieval results may contain redundant or unnecessary information. Additionally, long prompts can slow down the inference process.

Summarization tasks can be extractive or abstractive.

Document Repacking

The performance of subsequent processes, such as LLM response generation, may be affected by the order documents are provided.

  • Thu, 6 Jul 2023 Lost in the Middle: How Language Models Use Long Contexts
    • 将重要的文档排到两边，大模型会偷懒不看中间的文档
    • 太诡异了，如果(2024年)新训练的，标榜长上下文的模型还需要将检索文档重新排序，那就真的太诡异了。

Generate

Verification

Given the input question, the retrieved knowledge, and the generated answer, a small language model is trained to determine whether the generated answer correctly reflects the retrieved knowledge.

Advanced RAG\Agentic RAG

利用llm的Planning&reasoning, 做RAG复杂流程调度

reasoning

reasoning 的风吹到了 RAG

multi-hop QA

  • Fri, 11 Oct 2024 DeepNote: Note-Centric Deep Retrieval-Augmented Generation
    • DeepNote employs notes as carriers for refining and accumulating knowledge.
    • During in-depth exploration, it uses these notes to determine retrieval timing, formulate retrieval queries, and iteratively assess knowledge growth,
    • ultimately leveraging the best note for answer generation.

Best Practice

Benchmark

  • Mon, 4 Sep 2023 Benchmarking Large Language Models in Retrieval-Augmented Generation
    • Noise Robustness
    • Negative Rejection
    • Information Integration
    • Counterfactual Robustness
  • Fri, 7 Jun 2024 CRAG -- Comprehensive RAG Benchmark
    • kdd-cup-2024
    • Task
      • Task 1: Retrieval Summarization.
      • Task 2: KG and Web Retrieval Augmentation.
      • Task 3: End-to-end RAG.
    • Question type
      • Simple \ Simple w. condition \ Set \ Comparison \ Aggregation \ Multi-hop \ Post-processing heavy \ False Premise
    • Metrics
      • We then use a scoring method Scoreh with score 1, 0.5, 0, and -1 for each perfect, acceptable, missing, and incorrect answer, respectively

Evaluate

Fine-Tuning for RAG

RAG 专用模型

随着大模型能力不断提升， zeroshot、in-Context Learning 和 follow instructions 的能力不断增强，专门为 RAG 任务 Fine-Tuning 到底有多大的收益

Add control token

Knowledge Graph RAG

Meta KDD Cup

Cup 2024

2024 年是 Knowledge Graph (KG)

Cup 2025

2025 年是 Vision Question Answering

Long-Context Evaluate

  • Mar, 8 Feb 2023 Needle in a haystack
    • Doing simple retrieval from LLM models at various context lengths to measure accuracy
  • Mon, 28 Aug 2023 LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  • Wed, 21 Feb 2024 ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
    • LRA (Tay et al., 2020) ∼10K
    • LongBench (Bai et al., 2023) ∼10K QA, summarization, synthetic retrieval, and code
    • L-Eval (An et al., 2023) 4K - 60K QA, summarization, math, retrieval, and multiple-choice (MC)
    • LooGLE (Li et al., 2023) ∼20K summary and QA
    • ∞BENCH (ours) ∼200K
      • Retrieve.PassKey
      • Retrieve.Number
      • Retrieve.KV
      • En.Sum
      • En.QA
      • En.MC
      • En.Dia
      • Zh.QA
      • Code.Debug
      • Code.Run
      • Math.Calc
      • Math.Find
  • Tue, 9 Apr 2024 RULER: What's the Real Context Size of Your Long-Context Language Models?
    • https://github.com/NVIDIA/RULER
    • Retrieval: Needle-in-a-haystack (NIAH)
      • Single NIAH (S-NIAH)
      • Multi-keys NIAH (MK-NIAH)
      • Multi-values NIAH (MV-NIAH)
      • Multi-queries NIAH (MQ-NIAH)
    • Multi-hop Tracing: Variable Tracking (VT)
    • Aggregation: Common Words (CWE) and Frequent Words Extraction (FWE)
    • Question Answering (QA)
    • GLM4 (9B) 效果不错
  • Sat, 29 Jun 2024 Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP
    • As a community, we require a more precise vocabulary to understand what makes long-context tasks similar or different.
    • We propose two orthogonal axes of difficulty
      • (I) Dispersion: How hard is it to find the necessary information in the context?
      • (II) Scope: How much necessary information is there to find?
    • Task Design in Long Context
      • Natural construction.
      • Synthetic construction.
  • Fri, 13 Dec 2024 SCBench: A KV Cache-Centric Analysis of Long-Context Methods
      1. KV cache generation,
      1. KV cache compression,
      1. KV cache retrieval, and
      1. KV cache loading
    • Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n2) pre-filling computation perform robustly.
    • we identify attention distribution shift issues in long-generation scenarios.
    • Task performance shows varying decline trends, as illustrated in Fig. 3b. Sparse KV cache methods excel in tasks requiring global information, whereas O(n) memory is essential for tasks involving exact match retrieval.
    • 所以对kv cache 的压缩不容易

Long-Context

  • Wed, 4 Oct 2023 Retrieval meets Long Context Large Language Models
    • GPT-3.5-turbo-16k 和 Llama2-70B 长文本能力真的不太行， 2024年模型长文本能力进步非常快
    • 测试以QA为主，对标LongBench，也跟不上时代了
  • Tue, 23 Jul 2024 Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
    • LongBench and ∞Bench, mainly focus on tasks that are (a) in English, (b) real, and (c) query-based
    • Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance.
    • However, RAG’s significantly lower cost remains a distinct advantage.
    • Based on this observation, we propose SELF-ROUTE, a simple yet effective method that routes queries to RAG or LC based on model self-reflection.
      • For the queries deemed unanswerable, we proceed to the second step, providing the full context to the long-context LLMs to obtain the final prediction (i.e., LC)
      • 相当于一种 Adaptive Retrieval 方法
  • Tue, 27 Aug 2024 Writing in the Margins: Better Inference Pattern for Long Context Retrieval
    • Needle Retrieval and Single-Hop Question Answering
      • Unsurprisingly, RAG emerges as the most optimal pattern for six out of seven evaluated models when extending the context length to 64k tokens in SQuAD. Indeed, for single-hop reasoning tasks, if the filtering process is successful (here we approximate the retriever by an LLM classifier), the challenge is reduced to a trivial task of retrieving a needle from a context window of 4096 tokens. However, this assumption in the RAG setup is overly optimistic because the LLMs used in our experiment are at least 7B in model parameters, and such large models are not typically used as retrievers.
      • 反过来说，如果成本可控，用7B以上的模型做retrievers几乎完美
  • Mon, 9 Sep 2024 MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery
    • LongBench InfBench UltraDomain
    • Using Long LLMs as Memory Model
    • it employs a light but longrange LLM to form the global memory of database, generates draft answers
    • it leverages an expensive but expressive LLM, which generates the ultimate answer based on the retrieved information.
  • Fri, 20 Sep 2024 Introducing Contextual Retrieval
    • A note on simply using a longer prompt
  • Sun, 6 Oct 2024 Inference Scaling for Long-Context Retrieval Augmented Generation
    • IterDRAG 只用从 32k -> 128k 才能超越 RAG 和 DRAG
    • Agentic RAG 需要非常大的 Inference Scaling 才能体现出优势
  • Tue, 5 Nov 2024 Long Context RAG Performance of Large Language Models
    • 给模型更多512token的片段，而不是更长的连续上下文？？？？

approximate Long-Context

使用短上下文近似长上下文效果， 相当于一种特别的 Agentic RAG

Long-Context inference

LAB

trained the retrieval and generative components jointly