APP

The rapid advancement of Large Language Models (LLMs) has revolutionized the field of natural language processing, enabling applications such as conversational AI, text generation, and language translation. One of the most popular open-source frameworks for LLM inference is llama.cpp, which provides a highly efficient and flexible way to deploy LLMs across various hardware platforms, both locally and in the cloud. In this blog post, we will explore the internlm2_5-7b-chat model in GGUF format, which can be utilized by llama.cpp, and provide a step-by-step guide on how to install, download, and deploy this model for inference and service deployment.

Step 1: create a conda environment and install cmake

conda create --name internlm2 python=3.10 -y

conda activate internlm2

pip install cmake

Step 2: clone the source code and build the project

git clone --depth=1 https://github.com/ggerganov/llama.cpp.git

cd llama.cpp

cmake -B build -DGGML_CUDA=ON

cmake --build build --config Release -j

All the built targets can be found in the build/bin subdirectory. For instructions on other platforms, please refer to the official guide.

Download Models

The internlm2_5-7b-chat model is available in GGUF format in both half precision and various low-bit quantized versions, including q5_0, q5_k_m, q6_k, and q8_0. You can download the appropriate model based on your requirements using the following command:

pip install huggingface-hub

huggingface-cli download internlm/internlm2_5-7b-chat-gguf internlm2_5-7b-chat-fp16.gguf --local-dir. --local-dir-use-symlinks False

Inference

Once you have downloaded the model, you can use llama-cli for conducting inference. Here's an example of how to use llama-cli:

build/bin/llama-cli \

--model internlm2_5-7b-chat-fp16.gguf \

--predict 512 \

--ctx-size 4096 \

--gpu-layers 32 \

--temp 0.8 \

--top-p 0.8 \

--top-k 50 \

--seed 1024 \

--color \

--prompt "<|im_start|>system\nYou are an AI assistant whose name is InternLM (书生·浦语).\n- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory (上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n- InternLM (书生·浦语) can understand and communicate fluently in the language chosen by the user such as English and 中文.<|im_end|>\n" \

--interactive \

--multiline-input \

--conversation \

--verbose \

--logdir workdir/logdir \

--in-prefix "<|im_start|>user\n" \

--in-suffix "<|im_end|>\n<|im_start|>assistant\n"

Serving

llama.cpp provides an OpenAI API compatible server - llama-server. You can deploy internlm2_5-7b-chat-fp16.gguf into a service like this:

./build/bin/llama-server -m./internlm2_5-7b-chat-fp16.gguf -ngl 32

At the client side, you can access the service through OpenAI API:

from openai import OpenAI

client = OpenAI(

api_key='YOUR_API_KEY',

base_url='http://localhost:8080/v1'

)

model_name = client.models.list().data[0].id

response = client.chat.completions.create(

model=model_name,

messages=[

{"role": "system", "content": "You are a helpful assistant."},

{"role": "user", "content": " provide three suggestions about time management"},

],

temperature=0.8,

top_p=0.8

)

print(response)