14. Automating evaluation using online evaluation

Online Evaluators

Sometimes you want to evaluate the products recorded in your project.

Copy

# installation
# !pip install -qU langsmith langchain-teddynote

Copy

# Configuration file for managing API KEY as environment variable
from dotenv import load_dotenv

# Load API KEY information
load_dotenv()

Copy

 True 

Copy

# LangSmith Set up tracking. https://smith.langchain.com
# !pip install -qU langchain-teddynote
from langchain_teddynote import logging

# Enter a project name.
logging.langsmith("CH16-Auto-Evaluation-Test")

Copy

 Start tracking LangSmith. 
[Project name] 
CH16-Auto-Evaluation-Test 

Chain setting for online Evaluation

Copy

from myrag import PDFRAG
from langchain_openai import ChatOpenAI

# Creating a PDFRAG object
rag = PDFRAG(
    "data/SPRI_AI_Brief_December 2023 issue호_F.pdf",
    ChatOpenAI(model="gpt-4o-mini", temperature=0),
)

# Create a retriever
retriever = rag.create_retriever()

# Create a chain
rag_chain = rag.create_chain(retriever)

Copy

from langchain_core.runnables import RunnableParallel, RunnablePassthrough

# RunnableParallel Create an object.
evaluation_runnable = RunnableParallel(
    {
        "context": retriever,
        "answer": rag_chain,
        "question": RunnablePassthrough(),
    }
)

Run the test chain to see if Runs reflects the results.

Copy

# Chain execution
_ = evaluation_runnable.invoke("What is the name of the generative AI developed by Samsung Electronics?")

Online LLM-as-judge creation

Online LLM-as-judge creation

Secrets & API Keys designation (OpenAI API Key)

Provider, Model, Prompt Settings

Provider, Model, Prompt Settings

For facts, specify output.context (change to suit your settings)

answer designates output.answer (changed to suit your settings)

Preview ensures that data is entered in the correct place