04. Self-RAG

In this tutorial Self-RAG La introduces the Retrieval Augmented Generation (RAG) strategy, LangGraph Describe step by step how to implement using.

Self-RAG is a RAG strategy that includes self-reflection and self-evaluation of searched documents and generated responses, which can contribute to improving the performance of RAG-based systems.

What is Self-RAG?

Self-RAG is a RAG strategy that includes additional steps to check and verify for both searched documents and generated responses. In traditional RAGs, if LLM was the main process to generate answers based on the information retrieved, in Self-RAG Self evaluation Verify the following:

  1. Determining the need to search: Determine whether additional searches are needed for current questions.

  2. Evaluate Search Result Relevance: Make sure that the fragments of the searched documents (chunks) help you solve the question.

  3. Validation of response facts: Evaluate whether the generated answer is sufficiently supported by the document chunk provided.

  4. Response quality assessment: Measures whether the generated answer actually solves the question well.

This process goes beyond simply searching and generating answers, allowing you to monitor and improve the quality and realism of the generated responses yourself.

Self-RAG paper shortcut


Self-RAG main concept theorem

The paper proposes the following decision process through Self-RAG.

Determining whether to use Retriever

  • input: x (question) or (x (question), y (generation))

  • Output: yes, no, continue This step determines whether you want to proceed with an additional search, proceed as it is without a search, or wait more.

Relevance Assessment (Retrieval Grader)

  • input: ( x (question) , d (chunk) ) for each d in D

  • Output: relevant or irrelevant Determine if the searched document chunks are actually useful information to answer questions.

Validation Grader

  • input: x (question) , d (chunk) , y (generation) for each d in D

  • Output: {fully supported, partially supported, no support} Determine whether the generated response reflects facts based on search results, or if Hallucination has occurred.

Answer Quality Assessment (Answer Grader)

  • input: x (question) , y (generation)

  • Output: {5, 4, 3, 2, 1} Evaluate by scoring how much the generated response solves the question.


What to cover in this tutorial

This tutorial covers the process of using LangGraph to implement some ideas of the Self-RAG strategy. You will learn how to build and implement your Self-RAG strategy through the following steps:

  • Retriever : Search for documents

  • Retrieval Grader : Evaluate the relevance of the searched document

  • Generate : Create answers to questions

  • Hallucination Grader : Validation of the generated answer (whether hallucinated)

  • Answer Grader : Evaluate relevance to questions in answers

  • Question Re-writer : Query rewrite

  • Graph creation and execution : Build and run graphs with defined nodes


Reference

Preferences

Copy

Copy

Copy

Copy

Basic PDF-based Retrieval Chain creation

Here, we create a Retrieval Chain based on PDF documents. Retrieval Chain with the simplest structure.

However, LangGraph creates Retriever and Chain separately. Only then can you do detailed processing for each node.

Reference

  • As covered in the previous tutorial, we omit the detailed description.

Documents utilized for practice

Software Policy Institute (SPRi)-December 2023

  • Author: Jaeheung Lee (AI Policy Institute Office Liability Institute), Lee Ji-soo (AI Policy Lab Yi Phyang Institute)

  • Link: https://spri.kr/posts/view/23669

  • File name: SPRI_AI_Brief_2023๋…„12์›”ํ˜ธ_F.pdf

Files downloaded for practice data Please copy to folder

Copy

Document Search Evaluator (Retrieval Grader)

Defined in advance to proceed with the relevance assessment for documents in future retrieve nodes.

Copy

retrieval_grader Run to perform a relevance assessment of the retrieved document.

Copy

Copy

  • question : User-entered questions

  • generation : generated response

  • documents : List of documents searched

Defines the state.

Status definition

Copy

Copy

Copy

Generate a user-entered question regroup.

Question Rewriter

Copy

Copy

Copy

yes In case it is relevant. no In case it means it is not relevant.

Evaluate whether the answer generated is a relevant answer to the question.

Evaluate the relevance of the answer

Copy

Copy

Copy

yes If so, it means there will be no halusation of the answer. Contrary, no If it is, I consider the answer to be halusination.

groundedness_grader Generate and generate answers context Based on the answer, proceed with the evaluation of the answer.

Evaluate whether the answer is halusination

Copy

Copy

It's a common Naive RAG chain we know.

The answer generation chain is a chain that generates answers based on the documents retrieved.

Reply generation chain

Copy

Node definition

  • retrieve : Document Search

  • grade_documents : Document evaluation

  • generate : Create answer

  • transform_query : Rewrite question

Copy

Conditional edge definition

decide_to_generate The function determines whether an answer is generated based on the results of the evaluation of the relevance of the retrieved document.

grade_generation_v_documents_and_question The function determines whether the answer is generated based on the results of the evaluation of the relevance of the generated answer to the document and question.

Copy

Graph generation

Generate graphs through previously written nodes and edges.

Copy

Copy

Visualize the graph.

Graph execution

Run the graph you created.

Copy

Copy

If there is a persistent failure in evaluating the relevance of a user's question, you may fall into a recursive state as follows:

Copy

Copy

A logic correction (flex correction) is needed to escape so as not to fall into this recursive state.

Last updated