10. Hangeulocyte analyzer (Kiwi, Kkma, Okt) + BM25 finder

Define the function to neatly check the output result.

Copy

def pretty_print(docs):
    for i, doc in enumerate(docs):
        if "score" in doc.metadata:
            print(f"[{i+1}] {doc.page_content} ({doc.metadata['score']:.4f})")
        else:
            print(f"[{i+1}] {doc.page_content}")

BM25Retriever with Kiwi talkizer

Copy

# Install required libraries
# !pip install -qU kiwipiepy konlpy langchain-teddynote

Copy

# For comparison BM25Retriever
from langchain_community.retrievers import BM25Retriever

# BM25Retriever using a custom-implemented Korean morphological analyzer (Kiwi)
from langchain_teddynote.retrievers import KiwiBM25Retriever

sample_texts = [
    "Financial insurance is a financial product designed for long-term asset management and risk management.",
    "Financial savings product insurance is a special financial product that has a long-term savings purpose as well as a livestock product provision function.",
    "Financial savings product insurance is a special financial product that has a long-term savings purpose as well as a livestock product provision function.",
    "Financial group bombing insurance is a product that focuses on risk management rather than savings. It is suitable for customers who are willing to take high risks.",
]

Copy

# create a retriever.
kiwi = KiwiBM25Retriever.from_texts(sample_texts)

Copy

# Perform a similarity search.
pretty_print(kiwi.invoke("fiannce and insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. 

Calculate similarity scores by personal needs and on metadata score Added the ability to add

Copy

# perform a similarity search.
pretty_print(kiwi.search_with_score("financial insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2673) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2616) 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. (0.2463) 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. (0.2248) 

Copy

bm25 = BM25Retriever.from_texts(sample_texts)

# BM25 KiwiBM25 비교
print(f'Kiwi:  \t {kiwi.invoke("finance and insurance")[0].page_content}')
print(f'BM25:  \t {bm25.invoke("finance and insurance")[0].page_content}')

Copy

 Kiwi: Financial insurance is a financial product designed for long-term asset management and risk handling. 
BM25: Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 

k value setting

Copy

# create a retriever.
kiwi = KiwiBM25Retriever.from_texts(sample_texts)
kiwi.k = 2
# perform a similarity search.
pretty_print(kiwi.search_with_score("financial insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2673) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2616) 

BM25Retriever used KonlPy (Kkma, Okt)

Copy

# Using a custom implemented Korean morphological analyzer BM25Retriever
from langchain_teddynote.retrievers import (
    KkmaBM25Retriever,
    OktBM25Retriever,
)

Copy

# create a retriever.
kkma = KkmaBM25Retriever.from_texts(sample_texts)
okt = OktBM25Retriever.from_texts(sample_texts)

Copy

# kkma + BM25
pretty_print(kkma.invoke("finance and insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. 
[3] Mr. Financial Services, don't say anything, and do some savings or something. I don't know what's so urgent. 
[4] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 

Copy

# perform a similarity search.
pretty_print(kkma.search_with_score("finance and insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2770) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2701) 
[3] Mr. Financial Services, don't say anything, and do some savings or something. I don't know what's so urgent. (0.2268) 
[4] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. (0.2260) 

Copy

# okt + BM25
pretty_print(okt.invoke("finance and insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. 

Copy

# perform a similarity search.
pretty_print(okt.search_with_score("finance and insurance"))

Copy

 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2672) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2620) 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. (0.2460) 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. (0.2248)