\u4f60\u53ef\u80fd\u5df2\u7ecf\u8bd5\u8fc7\u76f4\u63a5 pip install gte-large，\u7136\u540e\u7528 Hugging Face Transformers \u52a0\u8f7d\u6a21\u578b\u2014\u2014\u7ed3\u679c\u53d1\u73b0：\u5355\u6761\u6587\u672c\u5d4c\u5165\u8017\u65f6 800ms，\u6279\u91cf\u5904\u7406\u5361\u5728 CPU \u4e0a，GPU \u5229\u7528\u7387\u4e0d\u5230 15%，\u66f4\u522b\u8bf4\u90e8\u7f72\u6210\u670d\u52a1\u4f9b\u591a\u4e2a\u4e1a\u52a1\u7cfb\u7edf\u8c03\u7528\u4e86\u3002<\/p>\n
\u8fd9\u4e0d\u662f\u6a21\u578b\u4e0d\u884c，\u800c\u662f\u9ed8\u8ba4\u914d\u7f6e\u6839\u672c\u6ca1\u53d1\u6325\u51fa GTE-Pro \u7684\u771f\u5b9e\u80fd\u529b\u3002<\/p>\n
GTE-Pro \u4e0d\u662f\u73a9\u5177\u6a21\u578b，\u5b83\u662f\u9762\u5411\u4f01\u4e1a\u7ea7\u8bed\u4e49\u68c0\u7d22\u8bbe\u8ba1\u7684\u201c\u5f15\u64ce\u201d，\u4e0d\u662f\u201c\u811a\u672c\u201d\u3002\u5b83\u9700\u8981：<\/p>\n
\u800c\u8fd9\u4e00\u5207\u7684\u524d\u63d0，\u662f\u642d\u5efa\u4e00\u5957\u7a33\u5b9a\u3001\u53ef\u63a7\u3001\u53ef\u590d\u73b0\u7684\u5e95\u5c42\u73af\u5883：Ubuntu 22.04 \u63d0\u4f9b\u957f\u671f\u652f\u6301\u7684\u7cfb\u7edf\u57fa\u7ebf，CUDA 12.1 \u5339\u914d\u4e3b\u6d41 A100\/H100 \u548c RTX 4090 \u663e\u5361\u9a71\u52a8，Triton \u63a8\u7406\u670d\u52a1\u5668\u5219\u628a\u6a21\u578b\u53d8\u6210\u6807\u51c6 HTTP\/gRPC \u63a5\u53e3\u2014\u2014\u4e0d\u5199\u4e00\u884c Flask，\u4e0d\u78b0\u4e00\u6b21 PyTorch \u5206\u5e03\u5f0f，\u5c31\u80fd\u4ea4\u4ed8\u751f\u4ea7\u7ea7\u8bed\u4e49\u670d\u52a1\u3002<\/p>\n
\u4e0b\u9762\u8fd9\u4e00\u6b65，\u5c31\u662f\u8ba9 GTE-Pro \u4ece\u201c\u80fd\u8dd1\u201d\u53d8\u6210\u201c\u8dd1\u5f97\u7a33\u3001\u8dd1\u5f97\u5feb\u3001\u8dd1\u5f97\u4e45\u201d\u7684\u5173\u952e\u3002<\/p>\n
\u6211\u4eec\u4ee5\u4e00\u53f0\u53cc RTX 4090 \u5de5\u4f5c\u7ad9\u4e3a\u57fa\u51c6（\u5b9e\u9645\u4e5f\u9002\u7528\u4e8e A100\u3001L40S\u3001H100），\u64cd\u4f5c\u7cfb\u7edf\u5fc5\u987b\u662f Ubuntu 22.04 LTS（\u5185\u6838 \u2265 5.15）\u3002\u4e0d\u8981\u7528 20.04（CUDA 12.1 \u5b98\u65b9\u4e0d\u5b8c\u5168\u652f\u6301）\u3001\u4e5f\u4e0d\u8981\u8d38\u7136\u5347\u7ea7\u5230 24.04（Triton \u5f53\u524d\u7248\u672c\u5c1a\u672a\u5168\u9762\u9002\u914d）\u3002<\/p>\n
\u5148\u786e\u8ba4\u5f53\u524d\u7cfb\u7edf：<\/p>\n
lsb_release -a
\n# \u5e94\u8f93\u51fa：Description: Ubuntu 22.04.3 LTS
\nuname -r
\n# \u5e94\u8f93\u51fa：5.15.x \u6216\u66f4\u9ad8<\/p>\n
\u518d\u68c0\u67e5\u663e\u5361\u662f\u5426\u88ab\u8bc6\u522b：<\/p>\n
nvidia-smi
\n# \u5fc5\u987b\u770b\u5230\u4e24\u5757 RTX 4090，Driver Version \u2265 535.54.03（CUDA 12.1 \u8981\u6c42\u6700\u4f4e\u9a71\u52a8）<\/p>\n
\u5982\u679c nvidia-smi \u62a5\u9519\u6216\u53ea\u663e\u793a\u201cNVIDIA-SMI has failed\u201d，\u8bf4\u660e\u9a71\u52a8\u672a\u5b89\u88c5\u6216\u7248\u672c\u8fc7\u4f4e\u3002\u8bf7\u5148\u5378\u8f7d\u65e7\u9a71\u52a8，\u518d\u4ece NVIDIA \u5b98\u7f51 \u4e0b\u8f7d\u5bf9\u5e94 4090 \u7684 535.54.03 \u6216\u66f4\u65b0\u7248\u9a71\u52a8，\u6267\u884c：<\/p>\n
sudo apt purge nvidia-*
\nsudo .\/NVIDIA-Linux-x86_64-535.54.03.run –no-opengl-files –no-x-check<\/p>\n
CUDA \u4e0d\u8981\u901a\u8fc7 apt install cuda \u5b89\u88c5\u2014\u2014\u90a3\u662f\u5143\u5305，\u7248\u672c\u6df7\u4e71\u4e14\u5e38\u7f3a\u7ec4\u4ef6\u3002\u6211\u4eec\u91c7\u7528 runfile \u65b9\u5f0f\u79bb\u7ebf\u5b89\u88c5，\u786e\u4fdd\u8def\u5f84\u5e72\u51c0\u3001\u7248\u672c\u7cbe\u51c6\u3002<\/p>\n
\u4e0b\u8f7d\u5730\u5740（\u9700\u6ce8\u518c NVIDIA \u5f00\u53d1\u8005\u8d26\u53f7）：<\/p>\n
\u5b89\u88c5\u6b65\u9aa4（\u6309\u987a\u5e8f\u6267\u884c）：<\/p>\n
# 1. \u5173\u95ed\u56fe\u5f62\u754c\u9762（\u907f\u514d\u5b89\u88c5\u51b2\u7a81）
\nsudo systemctl set-default multi-user.target
\nsudo reboot<\/p>\n
# 2. \u767b\u5f55\u540e，\u8fdb\u5165\u4e0b\u8f7d\u76ee\u5f55，\u5b89\u88c5 CUDA（\u4e0d\u88c5 driver！\u5df2\u88c5\u597d）
\nsudo sh cuda_12.1.0_530.30.02_linux.run –silent –override –toolkit –toolkitpath=\/usr\/local\/cuda-12.1 –no-opengl-libs<\/p>\n
# 3. \u5b89\u88c5 cuDNN（\u89e3\u538b\u540e\u590d\u5236\u6587\u4ef6）
\ntar -xzvf cudnn-linux-x86_64-8.9.2.26_cuda12.1-archive.tar.xz
\nsudo cp cudnn-linux-x86_64-8.9.2.26_cuda12.1-archive\/include\/cudnn*.h \/usr\/local\/cuda-12.1\/include
\nsudo cp cudnn-linux-x86_64-8.9.2.26_cuda12.1-archive\/lib\/libcudnn* \/usr\/local\/cuda-12.1\/lib64
\nsudo chmod a+r \/usr\/local\/cuda-12.1\/include\/cudnn*.h \/usr\/local\/cuda-12.1\/lib64\/libcudnn*<\/p>\n
# 4. \u914d\u7f6e\u73af\u5883\u53d8\u91cf（\u5199\u5165 ~\/.bashrc）
\necho 'export CUDA_HOME=\/usr\/local\/cuda-12.1' >> ~\/.bashrc
\necho 'export PATH=\/usr\/local\/cuda-12.1\/bin:$PATH' >> ~\/.bashrc
\necho 'export LD_LIBRARY_PATH=\/usr\/local\/cuda-12.1\/lib64:$LD_LIBRARY_PATH' >> ~\/.bashrc
\nsource ~\/.bashrc<\/p>\n
# 5. \u9a8c\u8bc1
\nnvcc –version # \u5e94\u8f93\u51fa：Cuda compilation tools, release 12.1, V12.1.105
\nnvidia-smi # \u4ecd\u6b63\u5e38\u5de5\u4f5c<\/p>\n
GTE-Pro \u4f9d\u8d56 PyTorch 2.1+，\u800c PyTorch 2.1 \u5b98\u65b9 wheel \u4ec5\u652f\u6301 Python \u2264 3.11\u3002\u4e3a\u517c\u987e\u7a33\u5b9a\u6027\u4e0e\u517c\u5bb9\u6027，\u6211\u4eec\u9009\u7528 Python 3.10（Ubuntu 22.04 \u9ed8\u8ba4\u4e3a 3.10，\u65e0\u9700\u5347\u7ea7）\u3002<\/p>\n
\u521b\u5efa\u4e13\u7528\u865a\u62df\u73af\u5883，\u907f\u514d\u6c61\u67d3\u7cfb\u7edf Python：<\/p>\n
sudo apt update && sudo apt install -y python3.10-venv python3.10-dev
\npython3.10 -m venv ~\/gte-pro-env
\nsource ~\/gte-pro-env\/bin\/activate
\npip install –upgrade pip<\/p>\n
\u56e0\u4e3a\u4f01\u4e1a\u7ea7\u8bed\u4e49\u68c0\u7d22\u6709\u4e09\u4e2a\u786c\u9700\u6c42：<\/p>\n
\u800c\u8fd9\u4e9b，FastAPI \u81ea\u5df1\u5b9e\u73b0\u6210\u672c\u9ad8\u3001\u6613\u51fa\u9519\u3001\u96be\u7ef4\u62a4\u3002<\/p>\n
Triton \u5b98\u65b9\u63d0\u4f9b\u9884\u7f16\u8bd1 deb \u5305，\u76f4\u63a5\u5b89\u88c5\u5373\u53ef：<\/p>\n
# \u4e0b\u8f7d\u5e76\u5b89\u88c5（\u6ce8\u610f：\u5fc5\u987b\u9009 cuda12.1 \u7248\u672c）
\nwget https:\/\/github.com\/triton-inference-server\/server\/releases\/download\/v24.04\/tritonserver2404-cuda121-py3-24.04.0-amd64.deb
\nsudo apt-get install .\/tritonserver2404-cuda121-py3-24.04.0-amd64.deb<\/p>\n
# \u9a8c\u8bc1\u5b89\u88c5
\ntritonserver –version # \u5e94\u8f93\u51fa：24.04<\/p>\n
Triton \u4f1a\u81ea\u52a8\u8bc6\u522b \/usr\/local\/cuda-12.1，\u65e0\u9700\u989d\u5916\u914d\u7f6e CUDA \u8def\u5f84\u3002<\/p>\n
Triton \u4e0d\u76f4\u63a5\u8fd0\u884c .py \u6587\u4ef6，\u5b83\u9700\u8981\u4e00\u4e2a\u6807\u51c6\u5316\u7684\u6a21\u578b\u4ed3\u5e93\u7ed3\u6784\u3002\u6211\u4eec\u4e3a GTE-Pro \u521b\u5efa\u5982\u4e0b\u76ee\u5f55：<\/p>\n
gte-pro-model-repo\/
\n\u251c\u2500\u2500 gte_pro\/
\n\u2502 \u251c\u2500\u2500 1\/
\n\u2502 \u2502 \u2514\u2500\u2500 model.py # Triton \u81ea\u5b9a\u4e49 backend \u5165\u53e3
\n\u2502 \u251c\u2500\u2500 config.pbtxt # \u6a21\u578b\u914d\u7f6e（\u8f93\u5165\/\u8f93\u51fa\/\u5b9e\u4f8b\u6570\u7b49）
\n\u2502 \u2514\u2500\u2500 1\/model.pt # \u5bfc\u51fa\u7684 TorchScript \u6a21\u578b（\u7a0d\u540e\u751f\u6210）<\/p>\n
\u5148\u521b\u5efa\u9aa8\u67b6：<\/p>\n
mkdir -p ~\/gte-pro-model-repo\/gte_pro\/1
\ntouch ~\/gte-pro-model-repo\/gte_pro\/config.pbtxt<\/p>\n
config.pbtxt \u5185\u5bb9\u5982\u4e0b（\u590d\u5236\u7c98\u8d34）：<\/p>\n
name: "gte_pro"
\nplatform: "pytorch_libtorch"
\nmax_batch_size: 128<\/p>\n
input [
\n {
\n name: "INPUT_TEXT"
\n data_type: TYPE_STRING
\n dims: [ -1 ]
\n }
\n]<\/p>\n
output [
\n {
\n name: "EMBEDDING"
\n data_type: TYPE_FP32
\n dims: [ 1024 ]
\n }
\n]<\/p>\n
instance_group [
\n [
\n {
\n count: 2
\n kind: KIND_GPU
\n gpus: [0,1]
\n }
\n ]
\n]<\/p>\n
dynamic_batching { max_queue_delay_microseconds: 100 }<\/p>\n
\u8fd9\u4e2a\u914d\u7f6e\u610f\u5473\u7740：<\/p>\n
\u5728\u6fc0\u6d3b\u7684\u865a\u62df\u73af\u5883\u4e2d\u5b89\u88c5\u5fc5\u8981\u5305：<\/p>\n
pip install torch==2.1.2+cu121 torchvision==0.16.2+cu121 –extra-index-url https:\/\/download.pytorch.org\/whl\/cu121
\npip install transformers==4.38.2 sentence-transformers==2.3.1 tritonclient[http]==2.44.0<\/p>\n
\u52a0\u8f7d\u5e76\u6d4b\u8bd5\u539f\u59cb GTE-Large（\u6765\u81ea Hugging Face）：<\/p>\n
from sentence_transformers import SentenceTransformer
\nmodel = SentenceTransformer("Alibaba-NLP\/gte-large-en-v1.5", trust_remote_code=True)
\nemb = model.encode(["Hello world", "How are you?"])
\nprint(emb.shape) # \u5e94\u8f93\u51fa：(2, 1024)<\/p>\n
\u786e\u8ba4\u80fd\u6b63\u5e38\u8fd0\u884c\u540e，\u6211\u4eec\u5f00\u59cb\u5bfc\u51fa\u3002<\/p>\n
Triton \u7684 PyTorch backend \u8981\u6c42\u6a21\u578b\u63a5\u53d7 torch.Tensor \u6216 List[str] \u8f93\u5165，\u4f46\u539f\u751f encode() \u63a5\u53e3\u662f Python \u51fd\u6570\u3002\u6211\u4eec\u9700\u8981\u5c01\u88c5\u4e00\u4e2a forward() \u65b9\u6cd5，\u5e76\u5bfc\u51fa\u4e3a TorchScript\u3002<\/p>\n
\u65b0\u5efa export_gte.py：<\/p>\n
import torch
\nfrom sentence_transformers import SentenceTransformer<\/p>\n
class GTEProWrapper(torch.nn.Module):
\n def __init__(self):
\n super().__init__()
\n self.model = SentenceTransformer("Alibaba-NLP\/gte-large-en-v1.5", trust_remote_code=True)<\/p>\n
def forward(self, input_texts):
\n # input_texts: List[str] from Triton
\n embeddings = self.model.encode(input_texts, convert_to_tensor=True)
\n return embeddings.float()<\/p>\n
# \u5bfc\u51fa
\nwrapper = GTEProWrapper()
\nwrapper.eval()
\nexample_input = ["test sentence"]
\ntraced_model = torch.jit.trace(wrapper, example_input)
\ntraced_model.save("~\/gte-pro-model-repo\/gte_pro\/1\/model.pt")
\nprint(" Model exported to ~\/gte-pro-model-repo\/gte_pro\/1\/model.pt")<\/p>\n
\u8fd0\u884c：<\/p>\n
python export_gte.py<\/p>\n
\u6ce8\u610f：\u9996\u6b21\u8fd0\u884c\u4f1a\u4e0b\u8f7d\u7ea6 2.3GB \u6a21\u578b\u6743\u91cd，\u8bf7\u786e\u4fdd\u78c1\u76d8\u7a7a\u95f4\u5145\u8db3\u3002\u5bfc\u51fa\u540e model.pt \u7ea6 1.8GB，\u8fd9\u662f\u5305\u542b tokenizer \u548c transformer \u7684\u5b8c\u6574 TorchScript \u6a21\u5757\u3002<\/p>\n
Triton \u9ed8\u8ba4\u4e0d\u652f\u6301\u5b57\u7b26\u4e32\u8f93\u5165\u7684\u81ea\u52a8\u5206\u8bcd\u3002\u6211\u4eec\u9700\u8981\u5728 model.py \u4e2d\u5b8c\u6210：\u5b57\u7b26\u4e32 \u2192 token ids \u2192 attention mask \u2192 \u6a21\u578b\u524d\u5411\u3002<\/p>\n
\u521b\u5efa ~\/gte-pro-model-repo\/gte_pro\/1\/model.py：<\/p>\n
import torch
\nfrom transformers import AutoTokenizer<\/p>\n
class TritonModel:
\n def __init__(self, model_path):
\n self.tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP\/gte-large-en-v1.5", trust_remote_code=True)
\n self.model = torch.jit.load(model_path)
\n self.model.eval()<\/p>\n
def forward(self, texts):
\n # Tokenize
\n encoded = self.tokenizer(
\n texts,
\n padding=True,
\n truncation=True,
\n max_length=512,
\n return_tensors="pt"
\n )
\n input_ids = encoded["input_ids"].to("cuda")
\n attention_mask = encoded["attention_mask"].to("cuda")<\/p>\n
# Inference
\n with torch.no_grad():
\n outputs = self.model(input_ids, attention_mask)
\n return outputs.cpu().numpy()<\/p>\n
# Triton required interface
\ndef initialize(args):
\n global model
\n model = TritonModel("\/models\/gte_pro\/1\/model.pt")<\/p>\n
def execute(requests):
\n responses = []
\n for request in requests:
\n texts = request.get_input("INPUT_TEXT").as_numpy().tolist()
\n # Decode bytes to str (Triton passes strings as bytes)
\n texts = [t.decode("utf-8") if isinstance(t, bytes) else t for t in texts]
\n embs = model.forward(texts)
\n # Wrap output
\n response = request.Response()
\n response.set_output("EMBEDDING", embs)
\n responses.append(response)
\n return responses<\/p>\n
\u6b64 model.py \u5b8c\u5168\u517c\u5bb9 Triton \u7684 Python Backend \u89c4\u8303，\u652f\u6301\u6279\u91cf\u5b57\u7b26\u4e32\u8f93\u5165，\u81ea\u52a8\u5b8c\u6210\u5206\u8bcd\u4e0e GPU \u63a8\u7406\u3002<\/p>\n
tritonserver \\\\
\n –model-repository=~\/gte-pro-model-repo \\\\
\n –strict-model-config=false \\\\
\n –log-verbose=1 \\\\
\n –http-port=8000 \\\\
\n –grpc-port=8001 \\\\
\n –metrics-port=8002<\/p>\n
\u4f60\u4f1a\u770b\u5230\u7c7b\u4f3c\u65e5\u5fd7：<\/p>\n
I0410 10:23:45.123456 12345 model_repository_manager.cc:1234] successfully loaded 'gte_pro' version 1
\nI0410 10:23:45.123457 12345 server.cc:1234] Triton server started<\/p>\n
\u8868\u793a\u6a21\u578b\u5df2\u52a0\u8f7d\u6210\u529f，\u670d\u52a1\u5c31\u7eea\u3002<\/p>\n
\u65b0\u5efa test_client.py：<\/p>\n
import tritonclient.http as httpclient
\nimport numpy as np<\/p>\n
client = httpclient.InferenceServerClient(url="localhost:8000")<\/p>\n
# \u6784\u9020\u8f93\u5165
\ntexts = ["What is the capital of France?", "Paris is beautiful"]
\ninput_data = np.array(texts, dtype=object).reshape(-1, 1)<\/p>\n
inputs = [
\n httpclient.InferInput("INPUT_TEXT", input_data.shape, "BYTES")
\n]
\ninputs[0].set_data_from_numpy(input_data)<\/p>\n
outputs = [httpclient.InferRequestedOutput("EMBEDDING")]<\/p>\n
# \u53d1\u9001\u8bf7\u6c42
\nresponse = client.infer("gte_pro", inputs, outputs=outputs)
\nembs = response.as_numpy("EMBEDDING")<\/p>\n
print(" Embedding shape:", embs.shape) # (2, 1024)
\nprint(" Cosine similarity between two sentences:",
\n np.dot(embs[0], embs[1]) \/ (np.linalg.norm(embs[0]) * np.linalg.norm(embs[1])))<\/p>\n
\u8fd0\u884c：<\/p>\n
python test_client.py<\/p>\n
\u9884\u671f\u8f93\u51fa：<\/p>\n
Embedding shape: (2, 1024)
\n Cosine similarity between two sentences: 0.721<\/p>\n
\u76f8\u4f3c\u5ea6 >0.7，\u8bf4\u660e\u8bed\u4e49\u5bf9\u9f50\u6b63\u786e；\u8017\u65f6\u901a\u5e38 <120ms（\u53cc\u5361 batch=2），\u8fdc\u4f18\u4e8e\u539f\u59cb PyTorch \u5355\u6b21\u8c03\u7528\u3002<\/p>\n
| –pinned-memory-pool-byte-size=268435456<\/td>\n | 256MB<\/td>\n | \u9884\u5206\u914d pinned memory，\u51cf\u5c11 host\u2192device \u6570\u636e\u62f7\u8d1d\u5ef6\u8fdf<\/td>\n<\/tr>\n |
| –cuda-memory-pool-byte-size=0:1073741824<\/td>\n | 1GB per GPU<\/td>\n | \u4e3a\u6bcf\u5f20\u5361\u9884\u5206\u914d\u663e\u5b58\u6c60，\u907f\u514d\u9891\u7e41 malloc\/free<\/td>\n<\/tr>\n |
| –rate-limit=cpu:4,gpu:8<\/td>\n | \u9650\u5236\u5e76\u53d1\u5b9e\u4f8b\u6570<\/td>\n | \u9632\u6b62\u5355\u4e2a\u6a21\u578b\u5403\u5149\u6240\u6709 GPU \u8d44\u6e90<\/td>\n<\/tr>\n |
| –model-control-mode=explicit<\/td>\n | \u663e\u5f0f\u63a7\u5236\u6a21\u578b\u751f\u547d\u5468\u671f<\/td>\n | \u4fbf\u4e8e\u7070\u5ea6\u53d1\u5e03\u4e0e\u70ed\u66f4\u65b0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \u542f\u52a8\u547d\u4ee4\u589e\u5f3a\u7248：<\/p>\n tritonserver \\\\ 6.2 \u76d1\u63a7\u4e0e\u53ef\u89c2\u6d4b\u6027\u63a5\u5165<\/h4>\nTriton \u539f\u751f\u66b4\u9732 Prometheus metrics，\u53ea\u9700\u52a0\u4e00\u884c\u914d\u7f6e\u5373\u53ef\u63a5\u5165\u4f01\u4e1a\u76d1\u63a7\u4f53\u7cfb：<\/p>\n # \u542f\u52a8\u65f6\u6dfb\u52a0 \u8bbf\u95ee http:\/\/localhost:8002\/metrics，\u4f60\u5c06\u770b\u5230：<\/p>\n
|