CH10 finder (Retriever)

The Retriever phase is the fifth phase of the Retrieval-Augmented Generation (RAG) system, from the stored vector database Searching for documents related to your question is. This step is for user questions The goal is to quickly find the most appropriate information This is a very important process that is directly related to the overall performance of the RAG system.

Searcher Necessity

1. Provide accurate information : The searcher asks the user's questions Search for the most relevant information Thus, the system Generate accurate and useful answers To do. If this process is not done effectively, it is provided as a result The quality of the answer may be poor There is.

2. Reduce response time : Appropriate in database using efficient search algorithm Search for information quickly By doing, overall Reduce system response time Keep it. It has a direct impact on improving user experience.

3. optimization : Through an effective search process Extract only necessary information Optimize the use of system resources by doing, Reduce unnecessary data processing There is.

Motion method

1. Vectorization of questions : Convert user questions to vector form. This course is conducted using techniques similar to the embedding phase. The converted question vector is used as a reference point for subsequent search operations.

2. Vector similarity comparison : Between stored document vectors and question vectors Calculate similarity To. This is mainly done using mathematical methods such as cosine similarity, Max Marginal Relevance (MMR), etc.

3. Top document selection : Based on calculated similarity score Top N most relevant documents selected To. These documents are used to generate answers to your questions in the next step.

4. Return document information : Deliver information from selected documents to the next level (prompt generation). This information may include the content, location, and metadata of the document.

Importance of searcher

The searcher is on the RAG system A key role in determining the quality of information retrieval To. Without an efficient searcher, it is very difficult to quickly and accurately find relevant information in large databases.

Also, the search term Provide appropriate context for user questions Thus, the language model is more Generate accurate answers Help to do it. So the performance of the searcher Direct impact on the overall efficiency and user satisfaction of the RAG system Goes.

Sparse Retriever & Dense Retriever

Sparse Retriever Wow Dense Retriever Is the two main methods used in information retrieval systems. These are used in the field of natural language processing, especially when searching for related documents in large sets of documents.

Sparse Retriever

Sparse Retriever discrete documents and queries (questions) Converted to keyword vector for processing To. This method mainly uses traditional information retrieval techniques such as tumb frequency-historical frequency (TF-IDF) or BM25.

• TF-IDF (Term Frequency-Inverse Document Frequency) : Calculate the importance of a word by reflecting how often it appears in the document and how many documents it appears in. Here, words that appear frequently and rarely throughout the document set are weighted high.

• BM25 : TF-IDF improved model, improving search accuracy by considering the length of the document. By adjusting the weights between long and short documents, the effect of word frequency is relatively adjusted.

The characteristic of Sparse Retriever is that it only takes into account the existence of each word, so the calculation cost is low and the implementation is simple. However, this method does not take into account the semantic association of words, The quality of search results is highly dependent on the choice of keyword To.

Dense Retriever

Dense Retriever uses the latest deep learning techniques to encode documents and queries into consecutive high-dimensional vectors. Dense Retriever can express the meaningful content of the document more abundantly, Search for semantically relevant documents even if the keywords do not match perfectly You can.

Dense Retriever uses distance from vector space (e.g. cosine similarity) Documents most relevant to queries Find. This method is particularly beneficial for understanding the nuances and contexts of language, Provide more accurate search results for complex queries You can.

difference

1. Expression : Sparse Retriever uses discrete keyword-based expressions, while Dense Retriever uses meaningful expressions in a series of vector spaces.

2. Meaningful processing ability : Dense Retriever has a deeper grasp of context and meaning, so you can search for related documents even if the keywords don't exactly match. Sparse Retriever reflects this semantic nuance less.

3. Coverage : Dense Retriever may be more suitable for complex questions or natural language queries, and Sparse Retriever may be more useful for simple and clear keyword searches.

These two methods have their own pros and cons, and should be properly selected according to the use case and requirements.

code

Dense Retriever

Copy

from langchain_community.vectorstores import FAISS

# Step 4: Create DB and save
# Create a vector store.
vectorstore = FAISS.from_documents(documents=split_documents, embedding=embeddings)

# step 5: Dense Retriever generation
# Sparse Retriever
Retrieves and generates information contained in documents.
faiss_retriever = vectorstore.as_retriever()

Sparse Retriever

Copy

from langchain_community.retrievers import BM25Retriever

# step 5: Sparse Retriever generation
# Retrieves and generates information contained in documents.
bm25_retriever = BM25Retriever.from_documents(split_documents)

Reference

• Vector storage support finder

• LangChain Retrievers