# 08. HuggingFace Pipeline

### Hugging Face Local Pipelines <a href="#hugging-face-local-pipelines" id="hugging-face-local-pipelines"></a>

`HuggingFacePipeline` Classes allow Hugging Face models to run locally.

[Hugging Face Model Hub ](https://huggingface.co/models)Hosts over 120,000 models, 20,000 datasets, and 50,000 demo apps on the online platform, all open source and publicly available, making it easy for people to collaborate and build ML together.

These models can be used by LangChain by calling through this local pipeline wrapper, or by calling the inference endpoint hosted through the HuggingFaceHub class. For more information on hosted pipeline [HuggingFaceHub ](https://wikidocs.net/huggingface_hub)See the laptop.

For use [PyTorch ](https://pytorch.org/get-started/locally/)Python with [Package transformers ](https://pypi.org/project/transformers/)It should be installed.

Also, for more memory efficient attention implementation `xformer` You can also install.

Copy

```
%pip install --upgrade --quiet  transformers --quiet
```

Set the path to download the model

Copy

```
# Path to download the hugging face model/tokenizer
# (example)
import os

# ./cache/ Set to download to path
os.environ["TRANSFORMERS_CACHE"] = "./cache/"
os.environ["HF_HOME"] = "./cache/"
```

**Model Loading**

Model `from_model_id` You can load it by specifying model parameters using methods.

* `HuggingFacePipeline` Load the pre-learned model of the Hugging Face using classes.
* `from_model_id` Using methods `beomi/llama-2-ko-7b` Specify the model and set the job to "text-generation".
* `pipeline_kwargs` Use parameters to limit the maximum number of tokens to be generated to 10.
* Loaded models `hf` It is assigned to a variable, which allows you to perform text generation tasks.

Model used: <https://huggingface.co/beomi/llama-2-ko-7b>

Copy

```
from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
```

Copy

```
# HuggingFace Download the model.
hf = HuggingFacePipeline.from_model_id(
    model_id="beomi/llama-2-ko-7b",  # Specifies the ID of the model to use..
    task="text-generation",  # Specifies the task to perform, in this case generating text..
    # Set additional arguments to pass to the pipeline. Here we limit the maximum number of tokens to be generated to 10..
    pipeline_kwargs={"max_new_tokens": 512},
)
```

existing `transformers` You can also load by passing the pipeline directly.

Implement a text generation model using HuggingFacePipeline.

* `AutoTokenizer` Wow `AutoModelForCausalLM` Using `beomi/llama-2-ko-7b` Load the model and the talkizer.
* `pipeline` Create a "text-generation" pipline using functions, set the model and talkizer. The maximum number of tokens generated is limited to 10.
* `HuggingFacePipeline` Using class `hf` Create an object and pass the generated pipline.

Created like this `hf` You can use objects to generate text for a given prompt.

Copy

```
from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "beomi/llama-2-ko-7b"  # Specifies the ID of the model to use.
tokenizer = AutoTokenizer.from_pretrained(
    model_id
)  # Loads a tokenizer for the specified model.
model = AutoModelForCausalLM.from_pretrained(model_id)  # Loads the specified model.
# Create a text generation pipeline and set the maximum number of new tokens to generate to 10.
pipe = pipeline("text-generation", model=model,
                tokenizer=tokenizer, max_new_tokens=512)
# HuggingFacePipeline Create an object and pass it to the pipeline it was created from.
hf = HuggingFacePipeline(pipeline=pipe)
```

**Create Chain**

When the model is loaded into memory, it can be configured with a prompt to form a chain.

* `PromptTemplate` Create prompt templates that use classes to define the format of your questions and answers.
* `prompt` With objects `hf` By connecting the object to the pipe line `chain` Create an object.
* `chain.invoke()` Call methods to generate and output answers to a given question.

Copy

```
from langchain.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser

template = """Answer the following question in Korean.
#Question: 
{question}

#Answer: """  # Template defining the question and answer format
prompt = PromptTemplate.from_template(template)  # Creating a prompt object using a template

# Create a chain by connecting prompts and language models
chain = prompt | hf | StrOutputParser()

question = What is the capital of South Korea?" #Question Definition

print(
    chain.invoke({"question": question})
)  # Generate and output answers to questions by calling the chain
```

**GPU Inference**

When running on GPU `device=n` Models can be placed on specific devices by specifying parameters.

Default `-1` In addition, CPU performs reasoning.

If you have multiple GPUs or if your model is too large for a single GPU `device_map="auto"` You can specify

In this case [Accelerate ](https://huggingface.co/docs/accelerate/index)A library is required and is used to automatically determine how to load model weights.

*caution* : `device` Wow `device_map` Silver should not be specified together, it can cause unexpected behavior.

* `HuggingFacePipeline` Using `gpt2` Load the model, `device` Set the parameter to 0 so it runs on the GPU.
* `pipeline_kwargs` Use parameters to limit the maximum number of tokens to be generated to 10.
* `prompt` Wow `gpu_llm` By connecting the pipe line `gpu_chain` Generate.
* `gpu_chain.invoke()` Call methods to generate and output answers to a given question.

Copy

```
gpu_llm = HuggingFacePipeline.from_model_id(
    model_id="beomi/llama-2-ko-7b",  # Specifies the ID of the model to use.
    task="text-generation",  # Set the task to perform, in this case generating text.
    # Specifies the GPU device number to use. Setting to "auto" uses the accelerate library. device=0,
    # Set additional arguments to be passed to the pipeline. Here we limit the maximum number of tokens to be generated to 10.
    pipeline_kwargs={"max_new_tokens": 64},
)

gpu_chain = prompt | gpu_llm  # Create a gpu_chain by concatenating prompt and gpu_llm.

# Create a chain by connecting prompts and language models.
gpu_chain = prompt | gpu_llm | StrOutputParser()

question = "What is the capital of South Korea?" #Question Definition

# Generate and output answers to questions by calling the chain
print(gpu_chain.invoke({"question": question}))
```

**Batch GPU Inference**

When running on a GPU device, you can run inference on the GPU in batch mode.

* `HuggingFacePipeline` Using `beomi/llama-2-ko-7b` Load the model and set it to run on the GPU.
* `gpu_llm` When creating `batch_size` Set to 2, `temperature` to zero, `max_length` Set to 64.
* `prompt` Wow `gpu_llm` By connecting the pipe line `gpu_chain` Generate and set the exit token to "\n\n".
* `gpu_chain.batch()` Using `questions` Generate answers to the questions in parallel.
* Output the generated answer through a repeat statement.

Copy

```
gpu_llm = HuggingFacePipeline.from_model_id(
    model_id="beomi/llama-2-ko-7b",  # Specifies the ID of the model to use.
    task="text-generation",  # Set the task to perform.
    device=0,  # Specifies the device number. -1 means CPU..
    batch_size=2,  # Adjust the batch size. Set it appropriately depending on your GPU memory and model size.
    model_kwargs={
        "temperature": 0,
        "max_length": 256,
    },  # Set additional arguments to pass to the model.
)

# Create a chain by connecting prompts and language models.
gpu_chain = prompt | gpu_llm.bind(stop=["\n\n"])

questions = []
for i in range(4):
    # Create a list of questions
    questions.append({"question": f"number {i} What is this in Korean?"})

answers = gpu_chain.batch(questions)  # Batch process a list of questions to generate answers.
for answer in answers:
    print(answer)  # Prints the generated answer.
```
