网硕互联帮助中心RAG 文档处理管道：MarkItDown、智能分块与向量化
摘要：本文进入 HelloAgents 的 RAG 系统，包括 RAG 体验、RAGTool 初始化、多格式转换、Markdown 分块、Token 估算和向量索引代码。
一、RAG 要解决什么问题
检索增强生成（Retrieval-Augmented Generation，RAG）的核心是把“从模型参数里回忆知识”改成“从外部知识库检索知识，再交给模型生成”。三个词分别对应三个阶段：
- 检索：从知识库中查询相关内容。
- 增强：把检索结果融入提示词，辅助模型生成。
- 生成：输出兼具准确性与透明度的答案。
RAG 的演进可以看成检索能力和模块化程度不断提升的过程。
#mermaid-svg-wJzoyx5Xf5QpTmXM{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wJzoyx5Xf5QpTmXM .error-icon{fill:#552222;}#mermaid-svg-wJzoyx5Xf5QpTmXM .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wJzoyx5Xf5QpTmXM .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .marker.cross{stroke:#333333;}#mermaid-svg-wJzoyx5Xf5QpTmXM svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wJzoyx5Xf5QpTmXM p{margin:0;}#mermaid-svg-wJzoyx5Xf5QpTmXM .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .cluster-label text{fill:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .cluster-label span{color:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .cluster-label span p{background-color:transparent;}#mermaid-svg-wJzoyx5Xf5QpTmXM .label text,#mermaid-svg-wJzoyx5Xf5QpTmXM span{fill:#333;color:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .node rect,#mermaid-svg-wJzoyx5Xf5QpTmXM .node circle,#mermaid-svg-wJzoyx5Xf5QpTmXM .node ellipse,#mermaid-svg-wJzoyx5Xf5QpTmXM .node polygon,#mermaid-svg-wJzoyx5Xf5QpTmXM .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .rough-node .label text,#mermaid-svg-wJzoyx5Xf5QpTmXM .node .label text,#mermaid-svg-wJzoyx5Xf5QpTmXM .image-shape .label,#mermaid-svg-wJzoyx5Xf5QpTmXM .icon-shape .label{text-anchor:middle;}#mermaid-svg-wJzoyx5Xf5QpTmXM .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .rough-node .label,#mermaid-svg-wJzoyx5Xf5QpTmXM .node .label,#mermaid-svg-wJzoyx5Xf5QpTmXM .image-shape .label,#mermaid-svg-wJzoyx5Xf5QpTmXM .icon-shape .label{text-align:center;}#mermaid-svg-wJzoyx5Xf5QpTmXM .node.clickable{cursor:pointer;}#mermaid-svg-wJzoyx5Xf5QpTmXM .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .arrowheadPath{fill:#333333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wJzoyx5Xf5QpTmXM .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-wJzoyx5Xf5QpTmXM .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wJzoyx5Xf5QpTmXM .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-wJzoyx5Xf5QpTmXM .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .cluster text{fill:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM .cluster span{color:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-wJzoyx5Xf5QpTmXM .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-wJzoyx5Xf5QpTmXM rect.text{fill:none;stroke-width:0;}#mermaid-svg-wJzoyx5Xf5QpTmXM .icon-shape,#mermaid-svg-wJzoyx5Xf5QpTmXM .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wJzoyx5Xf5QpTmXM .icon-shape p,#mermaid-svg-wJzoyx5Xf5QpTmXM .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-wJzoyx5Xf5QpTmXM .icon-shape .label rect,#mermaid-svg-wJzoyx5Xf5QpTmXM .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wJzoyx5Xf5QpTmXM .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-wJzoyx5Xf5QpTmXM .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-wJzoyx5Xf5QpTmXM :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
朴素 RAGTF-IDF / BM25 + 直接拼接
高级 RAG稠密嵌入 + 重排序
模块化 RAG混合检索、MQE、HyDE
一个完整 RAG 应用通常包含离线准备和在线问答两个环节：
#mermaid-svg-o2ydqzEik6CVD8N3{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-o2ydqzEik6CVD8N3 .error-icon{fill:#552222;}#mermaid-svg-o2ydqzEik6CVD8N3 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-o2ydqzEik6CVD8N3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-o2ydqzEik6CVD8N3 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-o2ydqzEik6CVD8N3 .marker.cross{stroke:#333333;}#mermaid-svg-o2ydqzEik6CVD8N3 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-o2ydqzEik6CVD8N3 p{margin:0;}#mermaid-svg-o2ydqzEik6CVD8N3 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 .cluster-label text{fill:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 .cluster-label span{color:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 .cluster-label span p{background-color:transparent;}#mermaid-svg-o2ydqzEik6CVD8N3 .label text,#mermaid-svg-o2ydqzEik6CVD8N3 span{fill:#333;color:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 .node rect,#mermaid-svg-o2ydqzEik6CVD8N3 .node circle,#mermaid-svg-o2ydqzEik6CVD8N3 .node ellipse,#mermaid-svg-o2ydqzEik6CVD8N3 .node polygon,#mermaid-svg-o2ydqzEik6CVD8N3 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-o2ydqzEik6CVD8N3 .rough-node .label text,#mermaid-svg-o2ydqzEik6CVD8N3 .node .label text,#mermaid-svg-o2ydqzEik6CVD8N3 .image-shape .label,#mermaid-svg-o2ydqzEik6CVD8N3 .icon-shape .label{text-anchor:middle;}#mermaid-svg-o2ydqzEik6CVD8N3 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-o2ydqzEik6CVD8N3 .rough-node .label,#mermaid-svg-o2ydqzEik6CVD8N3 .node .label,#mermaid-svg-o2ydqzEik6CVD8N3 .image-shape .label,#mermaid-svg-o2ydqzEik6CVD8N3 .icon-shape .label{text-align:center;}#mermaid-svg-o2ydqzEik6CVD8N3 .node.clickable{cursor:pointer;}#mermaid-svg-o2ydqzEik6CVD8N3 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-o2ydqzEik6CVD8N3 .arrowheadPath{fill:#333333;}#mermaid-svg-o2ydqzEik6CVD8N3 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-o2ydqzEik6CVD8N3 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-o2ydqzEik6CVD8N3 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-o2ydqzEik6CVD8N3 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-o2ydqzEik6CVD8N3 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-o2ydqzEik6CVD8N3 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-o2ydqzEik6CVD8N3 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-o2ydqzEik6CVD8N3 .cluster text{fill:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 .cluster span{color:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-o2ydqzEik6CVD8N3 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-o2ydqzEik6CVD8N3 rect.text{fill:none;stroke-width:0;}#mermaid-svg-o2ydqzEik6CVD8N3 .icon-shape,#mermaid-svg-o2ydqzEik6CVD8N3 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-o2ydqzEik6CVD8N3 .icon-shape p,#mermaid-svg-o2ydqzEik6CVD8N3 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-o2ydqzEik6CVD8N3 .icon-shape .label rect,#mermaid-svg-o2ydqzEik6CVD8N3 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-o2ydqzEik6CVD8N3 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-o2ydqzEik6CVD8N3 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-o2ydqzEik6CVD8N3 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
应用阶段
数据准备阶段
数据提取
文本分割
向量化
写入向量数据库
用户提问
检索相关信息
注入 Prompt
LLM 生成答案
二、快速体验
先用 RAGTool 添加三段文本知识，再做一次语义搜索。
from hello_agents import SimpleAgent, HelloAgentsLLM, ToolRegistry
from hello_agents.tools import RAGTool
# 创建LLM实例
llm = HelloAgentsLLM()
# 创建RAG工具
rag_tool = RAGTool(
knowledge_base_path="./knowledge_base",
collection_name="test_collection",
rag_namespace="test"
)
tool_registry = ToolRegistry()
tool_registry.register_tool(rag_tool)
# 创建具有RAG能力的Agent
agent = SimpleAgent(name="知识助手", llm=llm, tool_registry=tool_registry)
# 体验RAG功能
# 添加第一个知识
result1 = rag_tool.execute("add_text",
text="Python是一种高级编程语言，由Guido van Rossum于1991年首次发布。Python的设计哲学强调代码的可读性和简洁的语法。",
document_id="python_intro")
print(f"知识1: {result1}")
# 添加第二个知识
result2 = rag_tool.execute("add_text",
text="机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式。主要包括监督学习、无监督学习和强化学习三种类型。",
document_id="ml_basics")
print(f"知识2: {result2}")
# 添加第三个知识
result3 = rag_tool.execute("add_text",
text="RAG（检索增强生成）是一种结合信息检索和文本生成的AI技术。它通过检索相关知识来增强大语言模型的生成能力。",
document_id="rag_concept")
print(f"知识3: {result3}")
print("\\n=== 搜索知识 ===")
result = rag_tool.execute("search",
query="Python编程语言的历史",
limit=3,
min_score=0.1
)
print(result)
print("\\n=== 知识库统计 ===")
result = rag_tool.execute("stats")
print(result)
三、五层七步架构
与记忆系统不同，RAG 被设计成一条清晰的 pipeline。整体可以概括为“五层七步”。
#mermaid-svg-SEpiQRo9hHPIL2ok{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-SEpiQRo9hHPIL2ok .error-icon{fill:#552222;}#mermaid-svg-SEpiQRo9hHPIL2ok .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-SEpiQRo9hHPIL2ok .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-SEpiQRo9hHPIL2ok .marker{fill:#333333;stroke:#333333;}#mermaid-svg-SEpiQRo9hHPIL2ok .marker.cross{stroke:#333333;}#mermaid-svg-SEpiQRo9hHPIL2ok svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-SEpiQRo9hHPIL2ok p{margin:0;}#mermaid-svg-SEpiQRo9hHPIL2ok .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok .cluster-label text{fill:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok .cluster-label span{color:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok .cluster-label span p{background-color:transparent;}#mermaid-svg-SEpiQRo9hHPIL2ok .label text,#mermaid-svg-SEpiQRo9hHPIL2ok span{fill:#333;color:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok .node rect,#mermaid-svg-SEpiQRo9hHPIL2ok .node circle,#mermaid-svg-SEpiQRo9hHPIL2ok .node ellipse,#mermaid-svg-SEpiQRo9hHPIL2ok .node polygon,#mermaid-svg-SEpiQRo9hHPIL2ok .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-SEpiQRo9hHPIL2ok .rough-node .label text,#mermaid-svg-SEpiQRo9hHPIL2ok .node .label text,#mermaid-svg-SEpiQRo9hHPIL2ok .image-shape .label,#mermaid-svg-SEpiQRo9hHPIL2ok .icon-shape .label{text-anchor:middle;}#mermaid-svg-SEpiQRo9hHPIL2ok .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-SEpiQRo9hHPIL2ok .rough-node .label,#mermaid-svg-SEpiQRo9hHPIL2ok .node .label,#mermaid-svg-SEpiQRo9hHPIL2ok .image-shape .label,#mermaid-svg-SEpiQRo9hHPIL2ok .icon-shape .label{text-align:center;}#mermaid-svg-SEpiQRo9hHPIL2ok .node.clickable{cursor:pointer;}#mermaid-svg-SEpiQRo9hHPIL2ok .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-SEpiQRo9hHPIL2ok .arrowheadPath{fill:#333333;}#mermaid-svg-SEpiQRo9hHPIL2ok .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-SEpiQRo9hHPIL2ok .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-SEpiQRo9hHPIL2ok .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SEpiQRo9hHPIL2ok .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-SEpiQRo9hHPIL2ok .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SEpiQRo9hHPIL2ok .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-SEpiQRo9hHPIL2ok .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-SEpiQRo9hHPIL2ok .cluster text{fill:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok .cluster span{color:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-SEpiQRo9hHPIL2ok .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-SEpiQRo9hHPIL2ok rect.text{fill:none;stroke-width:0;}#mermaid-svg-SEpiQRo9hHPIL2ok .icon-shape,#mermaid-svg-SEpiQRo9hHPIL2ok .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SEpiQRo9hHPIL2ok .icon-shape p,#mermaid-svg-SEpiQRo9hHPIL2ok .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-SEpiQRo9hHPIL2ok .icon-shape .label rect,#mermaid-svg-SEpiQRo9hHPIL2ok .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SEpiQRo9hHPIL2ok .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-SEpiQRo9hHPIL2ok .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-SEpiQRo9hHPIL2ok :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
用户层：RAGTool 统一接口
应用层：智能问答、搜索、管理
处理层：文档解析、分块、向量化
存储层：向量数据库、文档存储
基础层：嵌入模型、LLM、数据库
RAGTool 作为统一入口，创建命名空间隔离的 RAG 管道：
class RAGTool(Tool):
"""RAG工具
提供完整的 RAG 能力：
– 添加多格式文档（PDF、Office、图片、音频等）
– 智能检索与召回
– LLM 增强问答
– 知识库管理
"""
def __init__(
self,
knowledge_base_path: str = "./knowledge_base",
qdrant_url: str = None,
qdrant_api_key: str = None,
collection_name: str = "rag_knowledge_base",
rag_namespace: str = "default"
):
# 初始化RAG管道
self._pipelines: Dict[str, Dict[str, Any]] = {}
self.llm = HelloAgentsLLM()
# 创建默认管道
default_pipeline = create_rag_pipeline(
qdrant_url=self.qdrant_url,
qdrant_api_key=self.qdrant_api_key,
collection_name=self.collection_name,
rag_namespace=self.rag_namespace
)
self._pipelines[self.rag_namespace] = default_pipeline
整条处理流程可以压缩为：
任意格式文档 → MarkItDown转换 → Markdown文本 → 智能分块 → 向量化 → 存储检索
四、多模态文档载入
RAG 的入口统一使用 MarkItDown，将 PDF、Word、Excel、图片、音频等格式转换为 Markdown。这样后续分块和向量化只需处理一种结构化格式。
def _convert_to_markdown(path: str) –> str:
"""
Universal document reader using MarkItDown with enhanced PDF processing.
核心功能：将任意格式文档转换为Markdown文本
支持格式：
– 文档：PDF、Word、Excel、PowerPoint
– 图像：JPG、PNG、GIF（通过OCR）
– 音频：MP3、WAV、M4A（通过转录）
– 文本：TXT、CSV、JSON、XML、HTML
– 代码：Python、JavaScript、Java等
"""
if not os.path.exists(path):
return ""
# 对PDF文件使用增强处理
ext = (os.path.splitext(path)[1] or '').lower()
if ext == '.pdf':
return _enhanced_pdf_processing(path)
# 其他格式使用MarkItDown统一转换
md_instance = _get_markitdown_instance()
if md_instance is None:
return _fallback_text_reader(path)
try:
result = md_instance.convert(path)
markdown_text = getattr(result, "text_content", None)
if isinstance(markdown_text, str) and markdown_text.strip():
print(f"[RAG] MarkItDown转换成功: {path} -> {len(markdown_text)} chars Markdown")
return markdown_text
return ""
except Exception as e:
print(f"[WARNING] MarkItDown转换失败 {path}: {e}")
return _fallback_text_reader(path)
#mermaid-svg-bFRbz3TPq5oxz8kb{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-bFRbz3TPq5oxz8kb .error-icon{fill:#552222;}#mermaid-svg-bFRbz3TPq5oxz8kb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-bFRbz3TPq5oxz8kb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-bFRbz3TPq5oxz8kb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-bFRbz3TPq5oxz8kb .marker.cross{stroke:#333333;}#mermaid-svg-bFRbz3TPq5oxz8kb svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-bFRbz3TPq5oxz8kb p{margin:0;}#mermaid-svg-bFRbz3TPq5oxz8kb .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb .cluster-label text{fill:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb .cluster-label span{color:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb .cluster-label span p{background-color:transparent;}#mermaid-svg-bFRbz3TPq5oxz8kb .label text,#mermaid-svg-bFRbz3TPq5oxz8kb span{fill:#333;color:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb .node rect,#mermaid-svg-bFRbz3TPq5oxz8kb .node circle,#mermaid-svg-bFRbz3TPq5oxz8kb .node ellipse,#mermaid-svg-bFRbz3TPq5oxz8kb .node polygon,#mermaid-svg-bFRbz3TPq5oxz8kb .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-bFRbz3TPq5oxz8kb .rough-node .label text,#mermaid-svg-bFRbz3TPq5oxz8kb .node .label text,#mermaid-svg-bFRbz3TPq5oxz8kb .image-shape .label,#mermaid-svg-bFRbz3TPq5oxz8kb .icon-shape .label{text-anchor:middle;}#mermaid-svg-bFRbz3TPq5oxz8kb .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-bFRbz3TPq5oxz8kb .rough-node .label,#mermaid-svg-bFRbz3TPq5oxz8kb .node .label,#mermaid-svg-bFRbz3TPq5oxz8kb .image-shape .label,#mermaid-svg-bFRbz3TPq5oxz8kb .icon-shape .label{text-align:center;}#mermaid-svg-bFRbz3TPq5oxz8kb .node.clickable{cursor:pointer;}#mermaid-svg-bFRbz3TPq5oxz8kb .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-bFRbz3TPq5oxz8kb .arrowheadPath{fill:#333333;}#mermaid-svg-bFRbz3TPq5oxz8kb .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-bFRbz3TPq5oxz8kb .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-bFRbz3TPq5oxz8kb .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-bFRbz3TPq5oxz8kb .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-bFRbz3TPq5oxz8kb .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-bFRbz3TPq5oxz8kb .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-bFRbz3TPq5oxz8kb .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-bFRbz3TPq5oxz8kb .cluster text{fill:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb .cluster span{color:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-bFRbz3TPq5oxz8kb .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-bFRbz3TPq5oxz8kb rect.text{fill:none;stroke-width:0;}#mermaid-svg-bFRbz3TPq5oxz8kb .icon-shape,#mermaid-svg-bFRbz3TPq5oxz8kb .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-bFRbz3TPq5oxz8kb .icon-shape p,#mermaid-svg-bFRbz3TPq5oxz8kb .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-bFRbz3TPq5oxz8kb .icon-shape .label rect,#mermaid-svg-bFRbz3TPq5oxz8kb .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-bFRbz3TPq5oxz8kb .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-bFRbz3TPq5oxz8kb .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-bFRbz3TPq5oxz8kb :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
MarkItDown
Word / Excel / PPT
JPG / PNG / GIF
MP3 / WAV / M4A
统一 Markdown 文本
五、Markdown 智能分块
由于所有文档都先转换为 Markdown，分块器可以利用标题层级保持语义完整。
#mermaid-svg-Ligh977Subqo7peM{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Ligh977Subqo7peM .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Ligh977Subqo7peM .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Ligh977Subqo7peM .error-icon{fill:#552222;}#mermaid-svg-Ligh977Subqo7peM .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Ligh977Subqo7peM .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Ligh977Subqo7peM .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Ligh977Subqo7peM .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Ligh977Subqo7peM .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Ligh977Subqo7peM .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Ligh977Subqo7peM .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Ligh977Subqo7peM .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Ligh977Subqo7peM .marker.cross{stroke:#333333;}#mermaid-svg-Ligh977Subqo7peM svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Ligh977Subqo7peM p{margin:0;}#mermaid-svg-Ligh977Subqo7peM .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Ligh977Subqo7peM .cluster-label text{fill:#333;}#mermaid-svg-Ligh977Subqo7peM .cluster-label span{color:#333;}#mermaid-svg-Ligh977Subqo7peM .cluster-label span p{background-color:transparent;}#mermaid-svg-Ligh977Subqo7peM .label text,#mermaid-svg-Ligh977Subqo7peM span{fill:#333;color:#333;}#mermaid-svg-Ligh977Subqo7peM .node rect,#mermaid-svg-Ligh977Subqo7peM .node circle,#mermaid-svg-Ligh977Subqo7peM .node ellipse,#mermaid-svg-Ligh977Subqo7peM .node polygon,#mermaid-svg-Ligh977Subqo7peM .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Ligh977Subqo7peM .rough-node .label text,#mermaid-svg-Ligh977Subqo7peM .node .label text,#mermaid-svg-Ligh977Subqo7peM .image-shape .label,#mermaid-svg-Ligh977Subqo7peM .icon-shape .label{text-anchor:middle;}#mermaid-svg-Ligh977Subqo7peM .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Ligh977Subqo7peM .rough-node .label,#mermaid-svg-Ligh977Subqo7peM .node .label,#mermaid-svg-Ligh977Subqo7peM .image-shape .label,#mermaid-svg-Ligh977Subqo7peM .icon-shape .label{text-align:center;}#mermaid-svg-Ligh977Subqo7peM .node.clickable{cursor:pointer;}#mermaid-svg-Ligh977Subqo7peM .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Ligh977Subqo7peM .arrowheadPath{fill:#333333;}#mermaid-svg-Ligh977Subqo7peM .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Ligh977Subqo7peM .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Ligh977Subqo7peM .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Ligh977Subqo7peM .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Ligh977Subqo7peM .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Ligh977Subqo7peM .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Ligh977Subqo7peM .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Ligh977Subqo7peM .cluster text{fill:#333;}#mermaid-svg-Ligh977Subqo7peM .cluster span{color:#333;}#mermaid-svg-Ligh977Subqo7peM div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Ligh977Subqo7peM .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Ligh977Subqo7peM rect.text{fill:none;stroke-width:0;}#mermaid-svg-Ligh977Subqo7peM .icon-shape,#mermaid-svg-Ligh977Subqo7peM .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Ligh977Subqo7peM .icon-shape p,#mermaid-svg-Ligh977Subqo7peM .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Ligh977Subqo7peM .icon-shape .label rect,#mermaid-svg-Ligh977Subqo7peM .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Ligh977Subqo7peM .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Ligh977Subqo7peM .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Ligh977Subqo7peM :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
标准 Markdown 文本
标题层次解析# / ## / ###
段落语义分割
Token 计算分块
重叠策略优化
向量化准备
第一步按标题和空行把文档拆成段落，并记录每个段落所属的标题路径：
def _split_paragraphs_with_headings(text: str) –> List[Dict]:
"""根据标题层次分割段落，保持语义完整性"""
lines = text.splitlines()
heading_stack: List[str] = []
paragraphs: List[Dict] = []
buf: List[str] = []
char_pos = 0
def flush_buf(end_pos: int):
if not buf:
return
content = "\\n".join(buf).strip()
if not content:
return
paragraphs.append({
"content": content,
"heading_path": " > ".join(heading_stack) if heading_stack else None,
"start": max(0, end_pos – len(content)),
"end": end_pos,
})
for ln in lines:
raw = ln
if raw.strip().startswith("#"):
# 处理标题行
flush_buf(char_pos)
level = len(raw) – len(raw.lstrip('#'))
title = raw.lstrip('#').strip()
if level <= 0:
level = 1
if level <= len(heading_stack):
heading_stack = heading_stack[:level–1]
heading_stack.append(title)
char_pos += len(raw) + 1
continue
# 段落内容累积
if raw.strip() == "":
flush_buf(char_pos)
buf = []
else:
buf.append(raw)
char_pos += len(raw) + 1
flush_buf(char_pos)
if not paragraphs:
paragraphs = [{"content": text, "heading_path": None, "start": 0, "end": len(text)}]
return paragraphs
第二步按照 Token 数量组合段落，并通过重叠块保留上下文连续性：
def _chunk_paragraphs(paragraphs: List[Dict], chunk_tokens: int, overlap_tokens: int) –> List[Dict]:
"""基于Token数量的智能分块"""
chunks: List[Dict] = []
cur: List[Dict] = []
cur_tokens = 0
i = 0
while i < len(paragraphs):
p = paragraphs[i]
p_tokens = _approx_token_len(p["content"]) or 1
if cur_tokens + p_tokens <= chunk_tokens or not cur:
cur.append(p)
cur_tokens += p_tokens
i += 1
else:
# 生成当前分块
content = "\\n\\n".join(x["content"] for x in cur)
start = cur[0]["start"]
end = cur[–1]["end"]
heading_path = next((x["heading_path"] for x in reversed(cur) if x.get("heading_path")), None)
chunks.append({
"content": content,
"start": start,
"end": end,
"heading_path": heading_path,
})
# 构建重叠部分
if overlap_tokens > 0 and cur:
kept: List[Dict] = []
kept_tokens = 0
for x in reversed(cur):
t = _approx_token_len(x["content"]) or 1
if kept_tokens + t > overlap_tokens:
break
kept.append(x)
kept_tokens += t
cur = list(reversed(kept))
cur_tokens = kept_tokens
else:
cur = []
cur_tokens = 0
# 处理最后一个分块
if cur:
content = "\\n\\n".join(x["content"] for x in cur)
start = cur[0]["start"]
end = cur[–1]["end"]
heading_path = next((x["heading_path"] for x in reversed(cur) if x.get("heading_path")), None)
chunks.append({
"content": content,
"start": start,
"end": end,
"heading_path": heading_path,
})
return chunks
为了兼容中英文混合文本，Token 长度采用近似估算：
def _approx_token_len(text: str) –> int:
"""近似估计Token长度，支持中英文混合"""
# CJK字符按1 token计算
cjk = sum(1 for ch in text if _is_cjk(ch))
# 其他字符按空白分词计算
non_cjk_tokens = len([t for t in text.split() if t])
return cjk + non_cjk_tokens
def _is_cjk(ch: str) –> bool:
"""判断是否为CJK字符"""
code = ord(ch)
return (
0x4E00 <= code <= 0x9FFF or # CJK统一汉字
0x3400 <= code <= 0x4DBF or # CJK扩展A
0x20000 <= code <= 0x2A6DF or # CJK扩展B
0x2A700 <= code <= 0x2B73F or # CJK扩展C
0x2B740 <= code <= 0x2B81F or # CJK扩展D
0x2B820 <= code <= 0x2CEAF or # CJK扩展E
0xF900 <= code <= 0xFAFF # CJK兼容汉字
)
六、统一嵌入与向量存储
嵌入模型负责把分块文本转换为向量。HelloAgents 优先使用百炼 API，回退到本地 all-MiniLM-L6-v2，最后用 TF-IDF 兜底。
#mermaid-svg-Eafq5pSnnqM6Foo2{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Eafq5pSnnqM6Foo2 .error-icon{fill:#552222;}#mermaid-svg-Eafq5pSnnqM6Foo2 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Eafq5pSnnqM6Foo2 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .marker.cross{stroke:#333333;}#mermaid-svg-Eafq5pSnnqM6Foo2 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Eafq5pSnnqM6Foo2 p{margin:0;}#mermaid-svg-Eafq5pSnnqM6Foo2 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .cluster-label text{fill:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .cluster-label span{color:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .cluster-label span p{background-color:transparent;}#mermaid-svg-Eafq5pSnnqM6Foo2 .label text,#mermaid-svg-Eafq5pSnnqM6Foo2 span{fill:#333;color:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .node rect,#mermaid-svg-Eafq5pSnnqM6Foo2 .node circle,#mermaid-svg-Eafq5pSnnqM6Foo2 .node ellipse,#mermaid-svg-Eafq5pSnnqM6Foo2 .node polygon,#mermaid-svg-Eafq5pSnnqM6Foo2 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .rough-node .label text,#mermaid-svg-Eafq5pSnnqM6Foo2 .node .label text,#mermaid-svg-Eafq5pSnnqM6Foo2 .image-shape .label,#mermaid-svg-Eafq5pSnnqM6Foo2 .icon-shape .label{text-anchor:middle;}#mermaid-svg-Eafq5pSnnqM6Foo2 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .rough-node .label,#mermaid-svg-Eafq5pSnnqM6Foo2 .node .label,#mermaid-svg-Eafq5pSnnqM6Foo2 .image-shape .label,#mermaid-svg-Eafq5pSnnqM6Foo2 .icon-shape .label{text-align:center;}#mermaid-svg-Eafq5pSnnqM6Foo2 .node.clickable{cursor:pointer;}#mermaid-svg-Eafq5pSnnqM6Foo2 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .arrowheadPath{fill:#333333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Eafq5pSnnqM6Foo2 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Eafq5pSnnqM6Foo2 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Eafq5pSnnqM6Foo2 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Eafq5pSnnqM6Foo2 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .cluster text{fill:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 .cluster span{color:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Eafq5pSnnqM6Foo2 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Eafq5pSnnqM6Foo2 rect.text{fill:none;stroke-width:0;}#mermaid-svg-Eafq5pSnnqM6Foo2 .icon-shape,#mermaid-svg-Eafq5pSnnqM6Foo2 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Eafq5pSnnqM6Foo2 .icon-shape p,#mermaid-svg-Eafq5pSnnqM6Foo2 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Eafq5pSnnqM6Foo2 .icon-shape .label rect,#mermaid-svg-Eafq5pSnnqM6Foo2 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Eafq5pSnnqM6Foo2 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Eafq5pSnnqM6Foo2 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Eafq5pSnnqM6Foo2 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
已配置百炼 API
无云端 API
均不可用
Markdown chunks
统一嵌入器
DashScope
all-MiniLM-L6-v2
TF-IDF
批量向量
Qdrant 向量存储
批量索引函数处理分块、编码、维度检查和异常兜底：
def index_chunks(
store = None,
chunks: List[Dict] = None,
cache_db: Optional[str] = None,
batch_size: int = 64,
rag_namespace: str = "default"
) –> None:
"""
Index markdown chunks with unified embedding and Qdrant storage.
Uses百炼 API with fallback to sentence-transformers.
"""
if not chunks:
print("[RAG] No chunks to index")
return
# 使用统一嵌入模型
embedder = get_text_embedder()
dimension = get_dimension(384)
# 创建默认Qdrant存储
if store is None:
store = _create_default_vector_store(dimension)
print(f"[RAG] Created default Qdrant store with dimension {dimension}")
# 预处理Markdown文本以获得更好的嵌入质量
processed_texts = []
for c in chunks:
raw_content = c["content"]
processed_content = _preprocess_markdown_for_embedding(raw_content)
processed_texts.append(processed_content)
print(f"[RAG] Embedding start: total_texts={len(processed_texts)} batch_size={batch_size}")
# 批量编码
vecs: List[List[float]] = []
for i in range(0, len(processed_texts), batch_size):
part = processed_texts[i:i+batch_size]
try:
# 使用统一嵌入器（内部处理缓存）
part_vecs = embedder.encode(part)
# 标准化为List[List[float]]格式
if not isinstance(part_vecs, list):
if hasattr(part_vecs, "tolist"):
part_vecs = [part_vecs.tolist()]
else:
part_vecs = [list(part_vecs)]
# 处理向量格式和维度
for v in part_vecs:
try:
if hasattr(v, "tolist"):
v = v.tolist()
v_norm = [float(x) for x in v]
# 维度检查和调整
if len(v_norm) != dimension:
print(f"[WARNING] 向量维度异常: 期望{dimension}, 实际{len(v_norm)}")
if len(v_norm) < dimension:
v_norm.extend([0.0] * (dimension – len(v_norm)))
else:
v_norm = v_norm[:dimension]
vecs.append(v_norm)
except Exception as e:
print(f"[WARNING] 向量转换失败: {e}, 使用零向量")
vecs.append([0.0] * dimension)
except Exception as e:
print(f"[WARNING] Batch {i} encoding failed: {e}")
# 实现重试机制
# … 重试逻辑 …
print(f"[RAG] Embedding progress: {min(i+batch_size, len(processed_texts))}/{len(processed_texts)}")
下一篇继续看查询侧的高级检索策略：MQE 与 HyDE 如何改善召回率。
评论前必须登录！
注册