{"id":38402,"date":"2025-05-20T09:18:47","date_gmt":"2025-05-20T01:18:47","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/38402.html"},"modified":"2025-05-20T09:18:47","modified_gmt":"2025-05-20T01:18:47","slug":"2025-5-18","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/38402.html","title":{"rendered":"2025\/5\/18"},"content":{"rendered":"

\u7ee7\u7eed\u7814\u7a76\u4e00\u4e0b\u5927\u4f6c\u7684RAG\u9879\u76ee\u3002\u5f00\u59cb\u6211\u7684\u788e\u788e\u5ff5\u3002<\/p>\n

RAG\u53ef\u4ee5\u5206\u6210\u4e24\u90e8\u5206：\u4e00\u4e2a\u662f\u95ee\u7b54，\u4e00\u4e2a\u662f\u6570\u636e\u5904\u7406\u3002<\/p>\n

\u95ee\u7b54\u662f\u4eba\u63d0\u95ee，\u7136\u540e\u67e5\u6570\u636e\u5e93，\u628a\u67e5\u7684\u4e1c\u897f\u7528\u5927\u6a21\u578b\u7ec4\u7ec7\u6210\u4eba\u8bdd，\u56de\u7b54\u4eba\u7684\u63d0\u95ee\u3002<\/p>\n

\u6570\u636e\u5904\u7406\u662f\u628a\u5f53\u4e0b\u77e5\u8bc6\u5e93\u91cc\u7684\u4e1c\u897f（\u4e0d\u7ba1\u662f\u4ec0\u4e48\u7c7b\u578b\u7684\u6570\u636e），\u5168\u5f04\u6210\u8ba1\u7b97\u673a\u8bdd（\u4ee3\u7801\u80fd\u660e\u767d\u7684\u683c\u5f0f）\u5b58\u5230\u6570\u636e\u5e93，\u7136\u540e\u65b9\u4fbf\u4eba\u63d0\u95ee\u7684\u65f6\u5019（\u4e5f\u5c31\u662f\u95ee\u7b54）\u7ed9\u51fa\u53ef\u4ee5\u56de\u7b54\u7684\u77e5\u8bc6\u3002<\/p>\n

\u5982\u679c\u60f3\u8ba9\u9879\u76ee\u8dd1\u8d77\u6765，\u5fc5\u987b\u628aES\u670d\u52a1\u542f\u52a8\u8d77\u6765，\u8be5\u9879\u76ee\u662f\u7528ES\u5b58\u7684\u6570\u636e\u3002<\/p>\n

\u9879\u76ee\u542f\u52a8\u65f6，\u4f1a\u5148\u8fd0\u884cLoadStartup（\u5728Springboot\u5e94\u7528\u542f\u52a8\u65f6），\u521d\u59cb\u5316\u5411\u91cf\u5b58\u50a8（\u5177\u4f53\u521d\u59cb\u5316\u5411\u91cf\u5b58\u50a8\u7528vectorstorage\u7684initCollection()\u65b9\u6cd5，\u6307\u5b9a\u540d\u79f0\u548c\u7ef4\u5ea6，\u5411\u91cf\u7ef4\u5ea6\u662f1024\u4e3a\u4e86\u9002\u914d\u667a\u8c31AI）\u3002\u603b\u4e4b\u5c31\u662f\u81ea\u52a8\u521d\u59cb\u5316\u4e00\u4e2a\u5411\u91cf\u6570\u636e\u5e93\u7684\u96c6\u5408（Collection），\u7528\u4e8e\u5b58\u50a8\u540e\u7eed\u7684\u5411\u91cf\u6570\u636e（\u5982\u6587\u672c\u5d4c\u5165\u5411\u91cf）\u3002<\/p>\n

\u6211\u4eec\u770b\u5230LoadStartup\u7c7b\u6709\u4e00\u4e2a\u6ce8\u89e3@Component<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

\u6240\u4ee5@Service\u3001@Repository\u7b49\u7b49\u8fd9\u4e9b\u6ce8\u89e3，\u672c\u8d28\u4e0a\u90fd\u662f@Component\u3002\u53ea\u662f\u6839\u636e\u5c42\u6b21\u6709\u4e0d\u540c\u53eb\u6cd5\u3002<\/p>\n

\u8fd9\u4e2acollection\u662f\u68ee\u9a6c\u6837\u5b50？\u56de\u5934\u518d\u5199\u5427\u3002<\/p>\n

\n

\u9996\u5148，\u5c31\u662f\u8f93\u5165\u7684\u95ee\u9898\u3002\u6211\u4eec\u8981\u5b58\u7684\u77e5\u8bc6\u4e0d\u4e00\u5b9a\u662f\u4ec0\u4e48\u7c7b\u578b，\u53ef\u80fd\u4f7ftxt，\u53ef\u80fd\u662fword，\u751a\u81f3\u662fpdf\u3002\u90a3\u6211\u4eec\u5c31\u9700\u8981\u628a\u8f93\u5165\u7684\u4e1c\u897f\u5148\u53d8\u6210\u6587\u672c\u3002<\/p>\n

\u9879\u76ee\u8fd0\u884c\u8d77\u6765\u4e4b\u524d，\u70b9\u51fb\u8fd0\u884c\u4e0b\u8f7d\u597d\u7684es\u7684bin\u6587\u4ef6\u5939\u4e0belasticsearch.bat，\u542f\u52a8\u670d\u52a1\u3002<\/p>\n

\u6b64\u65f6\u53ef\u4ee5\u518d\u7ec8\u7aef\u770b\u5230\u53ef\u4ea4\u4e92\u7684shell\u547d\u4ee4\u884c\u3002\u8fd9\u4e2a\u5e94\u8be5\u662f\u901a\u8fc7spring shell\u5de5\u5177\u5305\u5b9e\u73b0\u7684，\u9879\u76ee\u7684pom.xml\u6587\u4ef6\u91cc\u53ef\u4ee5\u770b\u5230\u5df2\u7ecf\u914d\u7f6e\u4e86shell\u7684\u8d77\u6b65\u4f9d\u8d56\u3002\u600e\u4e48\u7528\u8fd9\u4e2ashell\u5305\u5462？\u53ef\u4ee5\u901a\u8fc7\u81ea\u5df1\u7f16\u5199java\u7c7b，\u81ea\u5df1\u505a\u547d\u4ee4\u3002\u524d\u9762\u8bf4\u8fc7\u628aRAG\u5206\u6210\u4e24\u90e8\u5206：\u95ee\u7b54，\u6570\u636e\u5904\u7406\u3002\u4f7f\u7528add\u547d\u4ee4\u5b8c\u6210\u6570\u636e\u5904\u7406\u90e8\u5206\u7684\u5de5\u4f5c，\u4f7f\u7528chat\u547d\u4ee4\u5b8c\u6210\u95ee\u7b54\u90e8\u5206\u7684\u5de5\u4f5c\u3002\u65b0\u5efacommand\u6587\u4ef6\u5939\u6765\u5b58\u653e\u8fd9\u4e24\u4e2a\u7c7b：add\u547d\u4ee4\u7c7b，chat\u547d\u4ee4\u7c7b\u3002<\/p>\n

\u901a\u8fc7 @ShellMethod \u6ce8\u89e3\u5c06 Java \u65b9\u6cd5\u66b4\u9732\u4e3a Shell \u547d\u4ee4\u3002<\/p>\n

\"\"<\/p>\n

@ShellMethod(value = "add local txt data") \/\/ \u58f0\u660e\u8fd9\u662f\u4e00\u4e2aShell\u547d\u4ee4，\u63cf\u8ff0\u4e3a"add local txt data"
\npublic String add(String doc) { \/\/ \u5b9a\u4e49\u547d\u4ee4\u65b9\u6cd5，\u63a5\u6536\u4e00\u4e2a\u5b57\u7b26\u4e32\u53c2\u6570doc（\u6587\u4ef6\u8def\u5f84\u6216\u6587\u672c\u5185\u5bb9）
\n log.info("start add doc."); \/\/ \u6253\u5370\u65e5\u5fd7：\u5f00\u59cb\u5904\u7406\u6587\u6863<\/p>\n

\/\/ 1. \u6587\u672c\u5206\u5757（Chunking）
\n List<ChunkResult> chunkResults = txtChunk.chunk(doc); \/\/ \u8c03\u7528\u5206\u5757\u5de5\u5177，\u5c06\u6587\u6863\u62c6\u5206\u4e3a\u591a\u4e2a\u6587\u672c\u5757<\/p>\n

\/\/ 2. \u5411\u91cf\u5316（Embedding）
\n List<EmbeddingResult> embeddingResults = zhipuAI.embedding(chunkResults); \/\/ \u4f7f\u7528\u667a\u8c31AI（\u6216\u5176\u4ed6\u6a21\u578b）\u5c06\u6587\u672c\u5757\u8f6c\u4e3a\u5411\u91cf<\/p>\n

\/\/ 3. \u5411\u91cf\u5b58\u50a8
\n String collection = vectorStorage.getCollectionName(); \/\/ \u83b7\u53d6\u5411\u91cf\u6570\u636e\u5e93\u7684\u96c6\u5408\u540d（\u7c7b\u4f3c\u8868\u540d）
\n vectorStorage.store(collection, embeddingResults); \/\/ \u5c06\u5411\u91cf\u5b58\u50a8\u5230\u6570\u636e\u5e93\u4e2d<\/p>\n

log.info("finished"); \/\/ \u6253\u5370\u65e5\u5fd7：\u5904\u7406\u5b8c\u6210
\n return "finished docId:{}" + doc; \/\/ \u8fd4\u56de\u5904\u7406\u7ed3\u679c（\u683c\u5f0f\u6709\u8bef，\u5e94\u4e3aString.format）
\n} <\/p>\n

\u6570\u636e\u5904\u7406\u7684\u4e09\u6b65：\u6587\u672c\u5206\u5757\u3001\u5411\u91cf\u5316\u3001\u5411\u91cf\u5b58\u50a8\u3002\u6700\u540e\u8fd4\u56de\u7ed3\u679c\u3002<\/p>\n

\u8fd9\u51e0\u6b65\u5168\u8c03\u7528\u65b9\u6cd5，\u73b0\u5728\u770b\u662f\u4e00\u4e2a\u9ed1\u76d2，\u77e5\u9053\u8f93\u5165\u8f93\u51fa\u548c\u529f\u80fd\u5c31\u884c，\u540e\u9762\u518d\u5177\u4f53\u770b\u9ed1\u76d2\u91cc\u9762\u7684\u4ee3\u7801\u3002<\/p>\n

doc\u53c2\u6570\u662f\u6587\u4ef6\u5185\u5bb9\u8fd8\u662f\u6587\u4ef6\u8def\u5f84\u641e\u4e0d\u61c2？\u8bd5\u7740\u8f93\u51fa\u4e86doc，\u53d1\u73b0\u662f\u6587\u4ef6\u540d\u3002\u4f46\u662f，\u6839\u636e\u6587\u4ef6\u540d\u5c31\u80fd\u627e\u7740？？<\/p>\n

\"\"<\/p>\n

\u53d1\u73b0\u6709\u4e00\u4e2a\u9ed8\u8ba4\u8def\u5f84\/data，\u7136\u540e\u518d\u9ed8\u8ba4\u8def\u5f84\/data\u4e0b\u627edoc\u6587\u4ef6\u540d\u3002\u627e\u4e00\u4e0b\u54ea\u91cc\u8bbe\u7f6e\u7684\u9ed8\u8ba4\u8def\u5f84\u3002<\/p>\n

\"\"<\/p>\n

\/data\u5728chunk\u8fd9\u91cc\u3002<\/p>\n

\u6240\u4ee5\u8fd9\u4e2a\u610f\u601d\u662f，add \u6587\u4ef6\u540d\u3002add\u8fd9\u4e2a\u65b9\u6cd5\u5c31\u6536\u5230\u4e86\u53c2\u6570doc\u6587\u4ef6\u540d\u3002\u7136\u540e\u8fdb\u884c\u6587\u672c\u5206\u5757（\u6570\u636e\u5904\u7406\u7684\u5177\u4f53\u4ee3\u7801\u653e\u5728\/compoents\u6587\u4ef6\u5939），\u8c03\u7528\u4e86chunk\u65b9\u6cd5，\u7136\u540e\u6839\u636e\u9ed8\u8ba4\u8def\u5f84+\u6587\u4ef6\u540d+.txt，\u5c31\u5f97\u5230\u4e00\u6761\u5b8c\u6574\u7684\u8def\u5f84（\u76f8\u5bf9\u8def\u5f84）\u3002<\/p>\n

\u8bfb\u53d6\u6587\u4ef6\u6d41classpathresource(path)<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

\u6765\u56de\u6d41\u8f6c\u7684\u6570\u636e，\u5c01\u88c5\u5728\u5bf9\u8c61\u4e2d，\u800c\u8fd9\u4e9b\u5bf9\u8c61\u7684\u4ee3\u7801\u90fd\u653e\u5728\/domain\u6587\u4ef6\u5939\u91cc\u3002<\/p>\n

\u660e\u5929\u518d\u5199\u3002<\/p>\n

\n

\u7ee7\u7eed\u8fd9\u4e2achunk\u3002<\/p>\n

\u7136\u540e\u4e3a\u4ec0\u4e48\u8981chunk?\u5c0f\u5757\u6587\u672c\u6bd4\u957f\u6587\u672c\u66f4\u9ad8\u6548，\u8282\u7701\u8ba1\u7b97\u8d44\u6e90\u3002\u6309\u7167256\u4e2a\u5b57\u7b26\u5206\u5272\u5b57\u7b26\u4e32\u3002<\/p>\n

AtomicInteger \u662f Java \u4e2d\u4e00\u4e2a\u7ebf\u7a0b\u5b89\u5168\u7684\u539f\u5b50\u6574\u6570\u7c7b，\u5c5e\u4e8e java.util.concurrent.atomic \u5305\u3002\u5b83\u7684\u6838\u5fc3\u4f5c\u7528\u662f\u63d0\u4f9b\u539f\u5b50\u64cd\u4f5c（\u4e0d\u53ef\u4e2d\u65ad\u7684\u5355\u4e00\u64cd\u4f5c），\u786e\u4fdd\u5728\u591a\u7ebf\u7a0b\u73af\u5883\u4e0b\u5bf9\u6574\u6570\u7684\u64cd\u4f5c（\u5982\u9012\u589e\u3001\u9012\u51cf\u3001\u8d4b\u503c\u7b49）\u4e0d\u4f1a\u51fa\u73b0\u7ade\u6001\u6761\u4ef6（Race Condition）\u3002<\/p>\n

\"\"<\/p>\n

\u5f88\u660e\u663e，\u628a\u6bcf\u4e2achunk\u540e\u7684\u5c0f\u5757\u6587\u672c\u5c01\u88c5\u6210\u4e00\u4e2achunkresult\u5bf9\u8c61，\u7136\u540e\u8fd4\u56de\u8fd9\u4e9b\u5bf9\u8c61\u6784\u6210\u7684\u96c6\u5408\u3002<\/p>\n

\u7136\u540e\u8c03\u7528\u667a\u8c31AI\u7684embedding\u65b9\u6cd5\u3002\u53ef\u4ee5\u770b\u5230\u4f20\u8fdb\u53bb\u7684\u53c2\u6570\u662fchunkresult\u5bf9\u8c61\u7684\u96c6\u5408，\u8fd4\u56de\u7684\u662fembeddingresult\u7684\u96c6\u5408\u3002\u5177\u4f53\u770bembedding\u65b9\u6cd5\u91cc\u7684\u4ee3\u7801：\u89c2\u5bdf\u96c6\u5408\u662f\u5426\u4e3a\u7a7a，\u7a7a\u7684\u8bdd\u8fd4\u56de\u7a7a\u96c6\u5408；\u975e\u7a7a\u8fd4\u56deembedding\u540e\u7684\u96c6\u5408（\u8fd9\u91cc\u5c31\u6709\u4e00\u4e2aembedding\u65b9\u6cd5\u4e86）\u3002\u6211\u4eec\u5177\u4f53\u770b\u8fd9\u4e2aembedding\u65b9\u6cd5，\u4e0a\u4e00\u4e2aembedding\u8fd4\u56de\u7684\u662f\u96c6\u5408，\u8fd9\u4e2a\u91cc\u9762embedding\u65b9\u6cd5\u8fd4\u56de\u7684\u5355\u4e2a\u5411\u91cf\u5316\u540e\u7684\u7ed3\u679c\u3002\u65b9\u6cd5\u7684\u91cd\u8f7d，\u53c2\u6570\u4e0d\u540c\u3002\u8bf4\u4e0d\u660e\u767d，\u5177\u4f53\u770b\u4ee3\u7801\u5c31\u61c2\u4e86\u3002<\/p>\n

\/**
\n * \u6279\u91cf
\n * @param chunkResults \u6279\u91cf\u6587\u672c
\n * @return \u5411\u91cf
\n *\/
\n public List<EmbeddingResult> embedding(List<ChunkResult> chunkResults){
\n log.info("start embedding,size:{}",CollectionUtil.size(chunkResults));
\n if (CollectionUtil.isEmpty(chunkResults)){
\n return new ArrayList<>();
\n }
\n List<EmbeddingResult> embeddingResults=new ArrayList<>();
\n for (ChunkResult chunkResult:chunkResults){
\n embeddingResults.add(this.embedding(chunkResult));
\n }
\n return embeddingResults;
\n }<\/p>\n

public EmbeddingResult embedding(ChunkResult chunkResult){
\n String apiKey= this.getApiKey();
\n \/\/log.info("zp-key:{}",apiKey);
\n OkHttpClient.Builder builder = new OkHttpClient.Builder()
\n .connectTimeout(20000, TimeUnit.MILLISECONDS)
\n .readTimeout(20000, TimeUnit.MILLISECONDS)
\n .writeTimeout(20000, TimeUnit.MILLISECONDS)
\n .addInterceptor(new ZhipuHeaderInterceptor(apiKey));
\n OkHttpClient okHttpClient = builder.build();
\n EmbeddingResult embedRequest=new EmbeddingResult();
\n embedRequest.setPrompt(chunkResult.getContent());
\n embedRequest.setRequestId(Objects.toString(chunkResult.getChunkId()));
\n \/\/ \u667a\u8c31embedding
\n Request request = new Request.Builder()
\n .url("https:\/\/open.bigmodel.cn\/api\/paas\/v3\/model-api\/text_embedding\/invoke")
\n .post(RequestBody.create(MediaType.parse(ContentType.JSON.getValue()), GSON.toJson(embedRequest)))
\n .build();
\n try {
\n Response response= okHttpClient.newCall(request).execute();
\n String result=response.body().string();
\n ZhipuResult zhipuResult= GSON.fromJson(result, ZhipuResult.class);
\n EmbeddingResult ret= zhipuResult.getData();
\n ret.setPrompt(embedRequest.getPrompt());
\n ret.setRequestId(embedRequest.getRequestId());
\n return ret;
\n } catch (IOException e) {
\n throw new RuntimeException(e);
\n }<\/p>\n

} <\/p>\n

\u524d\u9762embedding\u65b9\u6cd5\u53ea\u662f\u5c01\u88c5\u6210\u96c6\u5408，\u5c31\u4e0d\u770b\u4e86\u3002\u540e\u9762\u7684embedding\u624d\u662f\u771f\u6b63\u7684\u5411\u91cf\u5316，\u91cd\u5934\u620f\u3002（\u4f46\u662f\u4eba\u5bb6\u7684API\u54b1\u76f4\u63a5\u7528\u5c31\u884c，embedding\u5177\u4f53\u56de\u5934\u518d\u770b\u5427）<\/p>\n

\u76f4\u63a5\u7528\u7684\u667a\u8c31\u7684API，\u6211\u4eec\u76f4\u63a5\u7ed9\u53c2\u5c31\u597d\u4e86\u3002<\/p>\n

\u5148\u83b7\u53d6\u5bc6\u94a5\u3002\/\/\u54b1\u4eec\u5199\u597d\u7684getapikey\u65b9\u6cd5，\u800c\u5b83\u91cc\u9762\u662f\u8c03\u7528\u4e86LLmProperties\u91cc\u9762\u7684\u4e1c\u897f\u3002（\u56de\u5934\u518d\u770b\u5427，\u53cd\u6b63\u80af\u5b9a\u8fd9\u4e2a\u6ce8\u89e3\u6307\u5b9a\u6709\u70b9\u4e1c\u897f\/\/\u7ed1\u5b9a\u524d\u7f00\u201cllm\u201d\u5f00\u5934\u7684\u914d\u7f6e，\u7136\u540e\u518dyaml\u914d\u7f6e\u6587\u4ef6\u91cc\u5b9a\u4e49\u5c5e\u6027）👇<\/p>\n

\"\"<\/p>\n

\u4f7f\u7528 OkHttpClient.Builder \u6784\u5efa\u4e00\u4e2a HTTP \u5ba2\u6237\u7aef，\u914d\u7f6e\u4e86\u8fde\u63a5\u3001\u8bfb\u53d6\u548c\u5199\u5165\u7684\u8d85\u65f6\u65f6\u95f4（\u5747\u4e3a 20 \u79d2）\u3002<\/p>\n

\u6dfb\u52a0\u4e86\u4e00\u4e2a\u81ea\u5b9a\u4e49\u62e6\u622a\u5668 ZhipuHeaderInterceptor，\u7528\u4e8e\u5728\u8bf7\u6c42\u5934\u4e2d\u6dfb\u52a0 API \u5bc6\u94a5\u7b49\u8ba4\u8bc1\u4fe1\u606f\u3002<\/p>\n

\n