04. Self-RAG
In this tutorial Self-RAG La introduces the Retrieval Augmented Generation (RAG) strategy, LangGraph Describe step by step how to implement using.
Self-RAG is a RAG strategy that includes self-reflection and self-evaluation of searched documents and generated responses, which can contribute to improving the performance of RAG-based systems.


What is Self-RAG?
Self-RAG is a RAG strategy that includes additional steps to check and verify for both searched documents and generated responses. In traditional RAGs, if LLM was the main process to generate answers based on the information retrieved, in Self-RAG Self evaluation Verify the following:
Determining the need to search: Determine whether additional searches are needed for current questions.
Evaluate Search Result Relevance: Make sure that the fragments of the searched documents (chunks) help you solve the question.
Validation of response facts: Evaluate whether the generated answer is sufficiently supported by the document chunk provided.
Response quality assessment: Measures whether the generated answer actually solves the question well.
This process goes beyond simply searching and generating answers, allowing you to monitor and improve the quality and realism of the generated responses yourself.
Self-RAG main concept theorem
The paper proposes the following decision process through Self-RAG.
Determining whether to use Retriever
input:
x (question)or(x (question), y (generation))Output:
yes, no, continueThis step determines whether you want to proceed with an additional search, proceed as it is without a search, or wait more.
Relevance Assessment (Retrieval Grader)
input: (
x (question),d (chunk)) for eachdinDOutput:
relevantorirrelevantDetermine if the searched document chunks are actually useful information to answer questions.
Validation Grader
input:
x (question),d (chunk),y (generation)for eachdinDOutput:
{fully supported, partially supported, no support}Determine whether the generated response reflects facts based on search results, or if Hallucination has occurred.
Answer Quality Assessment (Answer Grader)
input:
x (question),y (generation)Output:
{5, 4, 3, 2, 1}Evaluate by scoring how much the generated response solves the question.
What to cover in this tutorial
This tutorial covers the process of using LangGraph to implement some ideas of the Self-RAG strategy. You will learn how to build and implement your Self-RAG strategy through the following steps:
Retriever : Search for documents
Retrieval Grader : Evaluate the relevance of the searched document
Generate : Create answers to questions
Hallucination Grader : Validation of the generated answer (whether hallucinated)
Answer Grader : Evaluate relevance to questions in answers
Question Re-writer : Query rewrite
Graph creation and execution : Build and run graphs with defined nodes
Reference
Preferences
Copy
Copy
Copy
Copy
Basic PDF-based Retrieval Chain creation
Here, we create a Retrieval Chain based on PDF documents. Retrieval Chain with the simplest structure.
However, LangGraph creates Retriever and Chain separately. Only then can you do detailed processing for each node.
Reference
As covered in the previous tutorial, we omit the detailed description.
Documents utilized for practice
Software Policy Institute (SPRi)-December 2023
Author: Jaeheung Lee (AI Policy Institute Office Liability Institute), Lee Ji-soo (AI Policy Lab Yi Phyang Institute)
Link: https://spri.kr/posts/view/23669
File name:
SPRI_AI_Brief_2023๋ 12์ํธ_F.pdf
Files downloaded for practice data Please copy to folder
Copy
Document Search Evaluator (Retrieval Grader)
Defined in advance to proceed with the relevance assessment for documents in future retrieve nodes.
Copy
retrieval_grader Run to perform a relevance assessment of the retrieved document.
Copy
Copy
question: User-entered questionsgeneration: generated responsedocuments: List of documents searched
Defines the state.
Status definition
Copy
Copy
Copy
Generate a user-entered question regroup.
Question Rewriter
Copy
Copy
Copy
yes In case it is relevant. no In case it means it is not relevant.
Evaluate whether the answer generated is a relevant answer to the question.
Evaluate the relevance of the answer
Copy
Copy
Copy
yes If so, it means there will be no halusation of the answer. Contrary, no If it is, I consider the answer to be halusination.
groundedness_grader Generate and generate answers context Based on the answer, proceed with the evaluation of the answer.
Evaluate whether the answer is halusination
Copy
Copy
It's a common Naive RAG chain we know.
The answer generation chain is a chain that generates answers based on the documents retrieved.
Reply generation chain
Copy
Node definition
retrieve: Document Searchgrade_documents: Document evaluationgenerate: Create answertransform_query: Rewrite question
Copy
Conditional edge definition
decide_to_generate The function determines whether an answer is generated based on the results of the evaluation of the relevance of the retrieved document.
grade_generation_v_documents_and_question The function determines whether the answer is generated based on the results of the evaluation of the relevance of the generated answer to the document and question.
Copy
Graph generation
Generate graphs through previously written nodes and edges.
Copy
Copy
Visualize the graph.
Graph execution
Run the graph you created.
Copy
Copy
If there is a persistent failure in evaluating the relevance of a user's question, you may fall into a recursive state as follows:
Copy
Copy
A logic correction (flex correction) is needed to escape so as not to fall into this recursive state.
Last updated