Lewis：前面提到的各种技术，为检索实现奠定了基础。在本节中，我们的核心重点是：如何让检索更准确。但检索不准确的原因有很多。
Anna：Lewis，你一下子就说到重点了。确实，在实际项目中，虽然我们想提升检索准确率，但常常不知道该从哪里下手。
图 6.1：提升检索准确率的关键方向。
Lewis：RAG 流程包含多个阶段，可能出错的点也很多。同样，优化其中任何一个环节，都可能提升整个 RAG 系统的准确性。
在本节中，我们将从优化文本块索引的角度，提供一系列具体策略来提升检索质量。
图 6.2：用于提升 RAG 检索质量的索引优化策略概览。
从小到大：节点句子滑动窗口与父子文本块
一般来说，文本 chunk 越小，检索结果越精确。如果文本 chunk 过大，在对其进行 embedding、检索和内容合成时，可能会因为包含太多无关信息而遮蔽语义细节，从而影响检索效果。因此，有时我们会选择以句子作为文本 chunk 的基本单位。
不过，如果只把单个句子作为检索结果传给大模型，生成的答案可能会因为缺少足够上下文而不完整或不准确。
下面这种“small-to-big”策略可以解决这个问题。该策略的核心在于将检索过程和生成过程解耦，即 Decoupling chunks used for retrieval / chunks used for synthesis。这里的 “Synthesis” 指文本生成，有时也称为 composition，即整合来自多个文本块的信息，形成最终答案。
具体来说，就是先使用更小的文本 chunk 进行检索，以提高检索精度；然后在生成阶段引用更大、更完整的文本 chunk，从而为模型提供更多上下文信息。
LlamaIndex 和 LangChain 都提供了实现 “small-to-big” 检索策略的方法。
LlamaIndex 的节点句子滑动窗口检索：检索阶段只使用单个句子，然后返回包含该句子的文本窗口，为模型提供上下文。
LangChain 的子父递归检索：检索阶段使用较小的子文本块；找到相关子块后，再引用其对应的父文本块进行合成。
简而言之，“small-to-big” 检索策略通过细粒度分块提升检索精度，同时在生成过程中保留足够上下文。
节点句子滑动窗口检索
我们先来看 LlamaIndex 的节点句子滑动窗口检索策略，这一策略在第 2 章中已经介绍过。在下面的例子中，红框内区域表示被检索到的“小文本块”；紫色和红色块的组合则表示传给大模型的“大上下文”。
图 6.3：节点句子滑动窗口检索使用精确句子匹配，同时将周围上下文传给 LLM。
每个 node 会被解析成一个单句，并且该 node 的 metadata 中会包含它前后一定数量的句子，从而形成所谓的 window。检索时，通过 metadata replacement，将这个单句替换为该窗口中的上下文，从而利用扩展上下文进行大模型处理。
接下来，我们将使用 LlamaIndex's SentenceWindowNodeParser 和 MetadataReplacementNodePostProcessor 来实现“从小到大”的策略。这里，SentenceWindowNodeParser 负责以单句粒度将课程解析成 nodes；而 MetadataReplacementNodePostProcessor 则用于将检索到的单句替换为包含周围句子的窗口。
在程序中，会构建两类索引：一种使用“从小到大”策略，另一种是普通索引，用于对比检索效果。
from llama_index.core import VectorStoreIndex, Settings, Document
from llama_index.core.node_parser import SentenceWindowNodeParser, SentenceSplitter
from llama_index.llms.deepseek import DeepSeek
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.postprocessor import MetadataReplacementPostProcessor
### 配置全局设置
Settings.llm = DeepSeek(model="deepseek-chat", temperature=0.1)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-zh")
Settings.text_splitter = SentenceSplitter(separator="\n", chunk_size=50, chunk_overlap=0)
### 准备知识文本并创建 Document 对象
game_knowledge = """
“Myth of Extermination: Monkey King” is an action role-playing game. The game is set in a fictional mythological world.
Players take on the role of the Monkey King, Sun Wukong, embarking on an adventure in a world full of Eastern mythological elements.
The game’s combat system is unique, featuring a distinctive “transformation system.” Wukong can switch between different forms during battles.
Each form has its own unique combat style and skill combinations. The Vajra form emphasizes power strikes, delivering overwhelming destructive force.
The Demon Buddha form focuses on magic attacks, capable of unleashing powerful spell damage.
The game world is filled with iconic mythological characters. In addition to the protagonist Wukong, there are also gods and demons from Buddhism, Taoism, and other factions.
These characters may be Wukong’s allies or formidable opponents to be defeated.
The equipment system offers a variety of weapons. In addition to the famous Ruyi Jingu Bang, Wukong can also use other powerful divine artifacts.
Different weapons have their own unique effects, and players must choose flexibly according to the battle scenario.
The visual presentation of the game is highly characteristic of Eastern aesthetics. The scenes blend with the style of ink painting, perfectly presenting elements such as mountains, rivers, and architecture.
Combat special effects incorporate both traditional Chinese cultural elements and modern game’s visual impact.
In terms of difficulty design, boss battles are highly challenging and require players to precisely master the rhythm of combat and skill usage.
At the same time, the game offers multiple difficulty levels, catering to players of different skill levels."""
创建 Document 对象：
documents = [Document(text=game_knowledge)]
创建带上下文窗口的句子解析器，也就是 window parser，每个目标句子保留左右各 2 句作为上下文：
node_parser = SentenceWindowNodeParser.from_defaults(
window_size=4,
window_metadata_key="window",
original_text_metadata_key="original_text"
)
### 使用窗口解析器处理文档
nodes = node_parser.get_nodes_from_documents(documents)
### 使用基础解析器处理文档，用于对比
base_nodes = Settings.text_splitter.get_nodes_from_documents(documents)
### 构建两种索引用于对比
sentence_index = VectorStoreIndex(nodes)
base_index = VectorStoreIndex(base_nodes)
### 创建带上下文窗口的查询引擎
window_query_engine = sentence_index.as_query_engine(
similarity_top_k=2,
node_postprocessors=[
MetadataReplacementPostProcessor(target_metadata_key="window")
]
)
创建基础查询引擎：
base_query_engine = base_index.as_query_engine(
similarity_top_k=2
)
## 测试问答
test_questions = [
"What transformations does Wukong have in the game?",
# "What is the game’s visual style?",
# "How is the game’s difficulty designed?"
]
print("=== Retrieval Results Using Window Parser ===")
for question in test_questions:
print(f"\nQuestion: {question}")
window_response = window_query_engine.query(question)
print(f"Answer: {window_response}")
## 展示检索到的原始句子和窗口内容
print("\nRetrieval Details:")
for node in window_response.source_nodes:
print(f"Original sentence: {node.node.metadata['original_text']}")
print(f"Context window: {node.node.metadata['window']}")
print("---")
print("\n=== Retrieval Results Using Base Parser (Comparison) ===")
for question in test_questions:
print(f"\nQuestion: {question}")
base_response = base_query_engine.query(question)
print(f"Answer: {base_response}")
输出如下：
=== Retrieval Results Using Window Parser ===
Question: What transformations does Wukong have in the game?
Answer: Wukong switches between the Vajra form and the Demon Buddha form in the game.
Retrieval Details:
Original sentence: Wukong can change into different forms during battle,
Context window: Players will play as Sun Wukong, the Great Sage Equal to Heaven, embarking on an adventure in a world full of Eastern mythological elements. The battle system of the game is unique, adopting a special “ transformation system .” Wukong can change into different forms in combat, each with its own combat style and set of skills. The Vajra form focuses on power strikes, delivering overwhelming destructive force; the Demon Buddha form specializes in magical attacks,
---
Original sentence: These characters may be Wukong’s allies,
Context window: able to unleash powerful magic damage. The game world is full of iconic mythological characters. In addition to the protagonist Sun Wukong , there are gods and demons from Buddhist, Taoist, and other sects. These characters may be Wukong’s allies, or could be formidable opponents that must be defeated. The equipment system includes a variety of weapons. Besides the famous Ruyi Jingu Bang , Wukong can also use other divine weapons and treasures.
---
=== Retrieval Results Using Base Parser (Comparison) ===
Question: What transformations does Wukong have in the game?
Answer: Wukong has various forms in the game.
不难看出，SentenceWindowNodeParser 和 MetadataReplacementNodePostProcessor 的组合在这里明显表现更好。
父子文本块检索
接下来，我们介绍 LangChain 提供的父子文本块检索策略。原始文档经过预处理后形成的子块，也称为 sub-documents，会被存储在 vector store 中用于语义检索。当输入 query 时，系统会从 vector store 中检索相关内容。同时，完整的原始文档，也就是 parent document，也会存储在 document store 中。
图 6.4：检索已索引 chunks，同时保留原始文档用于提供上下文。
基于这种机制，vector store 负责相似度匹配和语义检索，而 document store 保留完整上下文信息。当执行检索操作时，系统首先通过 vector store 定位相关片段，然后再通过 document store 获取这些片段所属的完整内容。这种方式同样实现了“小块检索，大块返回”，保证系统在提升检索准确率的同时，也提供足够上下文支持。
这里提到的 parent document 可以是切分前的完整原始文档，也可以是由更大文本块组成的父 chunk。
下面的代码示例演示如何使用 LangChain 的父子文本块检索策略。
在这个例子中，父文本块被设置为比子块更大的 chunk。我们使用两级 splitter：父文本 splitter 用于创建更大的 chunks，以保持上下文完整性；子文本 splitter 用于创建更小的 chunks，以提升检索精度。
在存储方面，采用双层存储策略。vector store，例如 Chroma，用于存储较小的子块，以便相似度检索，实现精确匹配；而 document store，例如 InMemoryStore，则存储较大的父块，用于提供完整上下文信息。
from langchain_deepseek import ChatDeepSeek
from langchain_huggingface import HuggingFaceEmbeddings
### 初始化语言模型和向量嵌入模型
llm = ChatDeepSeek(model="deepseek-chat", temperature=0.1)
embed_model = HuggingFaceEmbeddings(model_name="BAAI/bge-small-zh")
### 准备游戏知识文本，创建 Document 对象
from langchain.schema import Document
game_knowledge = """
‘Chronicles of Godslayer: Monkey’ is an action role-playing game......
"""
## 创建 Document 对象
documents = [Document(page_content=game_knowledge)]
from langchain_text_splitters import RecursiveCharacterTextSplitter
## 父文本块切分器，较大 chunks
parent_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", "。", "！", "？", "；", ",", " ", ""]
)
## 子文本块切分器，较小 chunks
child_splitter = RecursiveCharacterTextSplitter(
chunk_size=200,
chunk_overlap=50,
separators=["\n\n", "\n", "。", "！", "？", "；", ",", " ", ""]
)
## 创建父文本块和子文本块
parent_docs = parent_splitter.split_documents(documents)
child_docs = child_splitter.split_documents(documents)
## 创建存储和 Retriever，建立两级存储系统
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_community.vectorstores import Chroma
vectorstore = Chroma(
collection_name="game_knowledge",
embedding_function=embed_model
)
store = InMemoryStore()
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=store,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
## 添加文本块
retriever.add_documents(documents)
## 自定义 Prompt 模板
from langchain.prompts import PromptTemplate
from langchain.chains import RetrievalQA
prompt_template = """Answer the question based on the following context information. If you cannot find the answer, please say ‘I cannot find relevant information.’
Context:
{context}
Question: {question}
Answer:"""
PROMPT = PromptTemplate(
template=prompt_template,
input_variables=["context", "question"]
)
创建问答链：
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff", # 问答链类型
retriever=retriever, # Retriever
return_source_documents=True, # 是否返回源文档
chain_type_kwargs={"prompt": PROMPT}
)
## 使用实际问题测试系统
test_questions = [
"What forms does Wukong have in the game?",
"What is the visual style of the game?",
]
for question in test_questions:
print(f"\nQuestion: {question}")
result = qa_chain({"query": question})
print(f"\nAnswer: {result['result']}")
print("\nSource documents used:")
for i, doc in enumerate(result["source_documents"], 1):
print(f"\nRelevant document {i}:")
print(f"Length: {len(doc.page_content)} characters")
print(f"Content snippet: {doc.page_content[:150]}...")
print("---")
Alex：Lewis，这里是使用 LangChain 和 LlamaIndex 提供的工具实现了 bottom-up 策略。我可以自己设计类似的索引结构吗？
Lewis：当然可以。Alex，你这里真正要学的是方法论。在真实项目中，我们通常不会再依赖框架，而是会自己控制每一个环节。
从摘要到细节：使用 IndexNode 和 RecursiveRetriever 构建 summary-to-detail 索引
接下来，我们将探索一种与 bottom-up 方法互补、但看起来方向相反的策略——Summary-to-Detail。该策略的核心是先组织高层信息，使用层级索引和递归检索，使搜索和生成过程能够根据需要逐步下钻，从概览级信息过渡到详细信息。
Summary-to-Detail 策略的设计理念如下。
粗粒度：主索引包含摘要信息，可以快速定位目标场景或主题。
细粒度：当需要更详细的信息时，可以递归搜索与主索引关联的二级索引，甚至更深层索引，以获取详细内容。
图 6.5：由摘要引导的检索，从顶层节点进入子文档 chunks。
与 bottom-up 策略不同，Summary-to-Detail 方法强调 indexes 之间的层级关系，并通过 RecursiveRetriever 工具实现多级索引协作，而不仅仅是在单一索引中控制块大小。
具体来说，这种层级结构允许用户逐步深入不同层级的需求。例如，如果一个知识库存储了世界各国按年份划分的 GDP 信息，那么粗粒度 nodes 负责定位每一年对应的整体说明，以确定具体年份；而具体 GDP 数值则存储在该 node 下方的二级索引中。在这个例子中，如果没有层级安排，直接检索某个国家的 GDP 信息，可能会得到与特定年份无关的结果。
LlamaIndex 中 Summary-to-Detail 策略的实现
在 LlamaIndex 中，“Summary-to-Detail” 策略的具体实现如下。
通过 IndexNode 构建层级索引：主索引包含摘要 nodes 和一些轻量文本内容；子索引则为主索引中的每个 IndexNode 创建独立向量索引，用来存储与该主题相关的深入信息。
递归检索逻辑：RecursiveRetriever 是实现 “Summary-to-Detail” 策略的关键。它允许在一次查询中动态跳转到相关二级索引，递归检索所需信息。
下面的代码示例演示如何在 LlamaIndex 中使用 IndexNode 和 RecursiveRetriever 模块进行递归检索。
from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.schema import IndexNode, Document
from llama_index.llms.deepseek import DeepSeek
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.retrievers import RecursiveRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core import get_response_synthesizer
from typing import List
## 配置全局设置
Settings.llm = DeepSeek(model="deepseek-chat", temperature=0.1)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-zh")
## 创建游戏场景描述
scene_descriptions = [
"""
Huaguo Mountain: This is the birthplace of the Great Sage Equal to Heaven, Sun Wukong. Mist shrouds the mountain year-round, and a waterfall plunges down from a height of a thousand meters,
forming the “Heavenly River Waterfall.” Various magical herbs and elixirs grow in the mountain, and many animals that have attained spirit forms reside here.
""",
"""
Shuilian Cave: Located at the peak of Huaguo Mountain, there is a naturally formed curtain of water in front of the cave, making it a sacred place for cultivation.
""",
"""
Dragon Palace of the East Sea: A magnificent palace at the bottom of the East Sea, decorated with coral and luminous pearls. This is where Sun Wukong obtained the Ruyi Jingu Bang.
"""
]
## 将场景描述转换为 Document 对象
documents = [Document(text=desc) for desc in scene_descriptions]
## 使用 Node Parser 将 Documents 转换为 Nodes
doc_nodes = Settings.node_parser.get_nodes_from_documents(documents)
创建表示层级关系的 IndexNode：创建场景细节，模拟详细文档。
scene_details = [
"""
Detailed Setup of Huaguo Mountain
1. Geographical Location: Within the borders of Aolai Country, Dongsheng Divine Continent
2. Natural Environment: Exotic flowers and plants that bloom year-round, clear mountain springs and waterfalls, dense ancient forests
3. Special Areas: Immortal Fruit Garden, where various spiritual fruits are cultivated; Training Ground, a flat and open area for cultivation; Rest Area, a place for the Monkey Tribe to rest
""",
"""
Detailed Setup of Shuilian Cave
1. Architectural Structure: Exterior—a huge natural rock cave; Entrance—a 30-zhang-high water curtain; Interior—a maze of interconnected caves
2. Functional Areas: Cultivation Hall—equipped with all kinds of training apparatus; Treasure Chamber—stores various magical treasures and elixirs, protected by powerful arrays;
Council Hall—can accommodate hundreds of monkeys, a place for important meetings
""",
"""
Detailed Setup of the Dragon Palace of the East Sea
1. Architectural Features: Materials—coral, pearls, luminous pearls; Scale—covering dozens of miles; Style—an underwater palace complex
2. Important Areas: Dragon King’s Treasury—stores countless treasures such as luminous pearls, and sacred objects like the Ruyi Jingu Bang; Armory—a variety of aquatic magical artifacts and divine weapons; Main Hall—the main hall for receiving guests and holding gatherings of the aquatic tribes
"""
]
## 为每个细节创建对应的 IndexNode，并创建关联 Query Engine
index_nodes = []
index_id_query_engine_mapping = {}
for idx, detail_text in enumerate(scene_details):
# 创建 IndexNode，在放入 f-string 前处理文本
index_id = f"detail{idx}"
first_line = detail_text.split('\n')[1].strip()
index_node = IndexNode(text=f"This node contains {first_line}", index_id=index_id)
index_nodes.append(index_node)
# 创建对应 TextNode，并构建独立索引和查询引擎
detail_node = Document(text=detail_text)
detail_index = VectorStoreIndex.from_documents([detail_node])
detail_query_engine = detail_index.as_query_engine()
## 将 Query Engine 添加到 Mapping 中
index_id_query_engine_mapping[index_id] = detail_query_engine
## 输出当前 Mapping 状态
print(f"\nCurrent index ID: {index_id}")
print(f"Index node text: {index_node.text}")
print(f"Length of detailed scene information: {len(detail_text)} characters")
print(f"Query engine type: {type(detail_query_engine).__name__}")
## 合并 Document Nodes 和 Index Nodes
all_nodes = doc_nodes + index_nodes
## 构建主向量索引
vector_index = VectorStoreIndex(all_nodes)
vector_retriever = vector_index.as_retriever(similarity_top_k=2)
## 创建 RecursiveRetriever 对象
recursive_retriever = RecursiveRetriever(
"vector", # root retriever 的 ID
retriever_dict={"vector": vector_retriever}, # Retriever 映射
query_engine_dict=index_id_query_engine_mapping, # Query engine 映射
verbose=True, # 启用详细输出
)
## 创建 RetrieverQueryEngine，并将 response mode 设置为 compact
response_synthesizer = get_response_synthesizer(response_mode="compact")
## 创建 RetrieverQueryEngine，传入 RecursiveRetriever 和 Response Synthesizer
query_engine = RetrieverQueryEngine.from_args(
recursive_retriever,
response_synthesizer=response_synthesizer,
)
## 定义查询函数
def query_scene(question: str):
print(f"Question: {question}\n")
response = query_engine.query(question)
print(f"Answer: {str(response)}\n")
## 示例查询
if __name__ == "__main__":
questions = [
"What is special about Huaguo Mountain?",
"Describe in detail the internal structure of the Water Curtain Cave.",
"What treasures are stored in the Dragon Palace of the East Sea?",
]
for q in questions:
query_scene(q)
输出结果：
上述程序的主要特点在于其多层级知识组织。它通过 IndexNode 建立层级连接，形成一种树状知识图谱。顶层是场景摘要，也就是花果山、水帘洞和龙宫的基本描述；底层则包含场景细节，例如建筑结构、功能分区和其他详细设定。
通过动态路由机制，主 retriever，也就是 similarity_top_k=2，首先进行粗粒度搜索，自动识别需要触发详细检索的 IndexNode，并根据 index_id 路由到对应的子查询引擎。
这种架构特别适合处理需要结合摘要和细节的复杂问答。当知识体系具有天然层级结构时，它尤其有效，可以实现精确的深层信息定位。系统既能快速回应一般性问题，也能处理需要深入细节的复杂查询，在效率和准确性之间取得平衡。
层级合并：HierarchicalNodeParser 和 RAPTOR
理解了“从小到大”和“粗中有细”策略之后，我们再来看另一种同样灵活且巧妙的方法：先将文本块划分成多个层级，然后对它们建立索引，形成各种向量表示，最后在检索过程中整合它们。
图 6.6：层级 chunks 以不同粒度被检索并合并，从而提供更完整上下文。
使用 HierarchicalNodeParser 创建层级索引
假设玩家问：Vajra Form 和 Demon Buddha Form 有什么区别。传统向量检索可能会准确定位到描述这两种形态的具体句子，例如 “Under the Vajra Form, Wukong’s strength increases dramatically” 和 “The Demon Buddha Form grants Wukong powerful magical abilities.” 但这样的答案显然是不完整的，因为形态之间的区别不仅存在于这两句话中，还包括连招特点、武器联动等信息，而这些内容散落在文本的不同位置。
在 LlamaIndex 中，可以使用 HierarchicalNodeParser 将整段游戏知识文本划分为三个层级：大块，2048 个字符，用于保留完整主题段落；中块，512 个字符，用于容纳若干相关句子；小块，128 个字符，用于精确描述。这就像先把游戏攻略分成章节，再分成段落，最后分成句子。
检索时，系统会先使用向量索引找到最相关的小块，确保检索精准。但系统不会停在这里。它会使用 AutoMergingRetriever 自动找到这些小块，也称为 leaf nodes 的“root node”，也就是包含这些小块的更大文本块。如果发现多个小块来自同一个更大块，系统会自动将它们合并，以提供更完整的上下文信息。
例如，在回答形态差异相关问题时，系统可能发现描述基础属性、连招特点和武器协同的句子虽然散落在不同地方，但都在讨论战斗形态特征。此时，系统会自动合并这些信息，以提供更全面的答案。
下面的代码示例演示如何使用 HierarchicalNodeParser 和 AutoMergingRetriever 构建一个层级游戏知识检索系统。我们会同时创建一个基础 retriever 和一个自动合并 retriever，并对比两者的检索结果。完整代码可参考 github.com/PacktPublis…
## 导入必要库并配置基础设置
from llama_index.core import VectorStoreIndex, StorageContext, Document, Settings
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes, get_root_nodes
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.retrievers import AutoMergingRetriever
from llama_index.llms.deepseek import DeepSeek
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
## 配置全局设置
Settings.llm = DeepSeek(model="deepseek-chat", temperature=0.1)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-zh")
## 准备游戏知识文本
game_knowledge = """The combat system of ‘Legend of the Demon God: Monkey King’ is ingeniously designed. Players can freely switch between multiple combat forms during battle, each with its own unique advantage. In Vajra form... in Demon Buddha form...”””
## 创建 Document 对象
documents = [Document(text=game_knowledge)]
创建层级 node parser 并处理文档：使用 HierarchicalNodeParser 创建文本层级结构。
node_parser = HierarchicalNodeParser.from_defaults(
chunk_sizes=[256, 128, 64] # 从 root node 到 leaf node 的 chunk sizes
)
nodes = node_parser.get_nodes_from_documents(documents)
## 获取 Leaf Nodes，最细粒度文本块，以及 Root Nodes
leaf_nodes = get_leaf_nodes(nodes)
root_nodes = get_root_nodes(nodes)
## 构建存储和索引。创建 Document Store 并添加所有 Nodes
docstore = SimpleDocumentStore()
docstore.add_documents(nodes)
## 创建 Storage Context
storage_context = StorageContext.from_defaults(docstore=docstore)
## 为 Leaf Nodes 创建 Vector Index
base_index = VectorStoreIndex(
leaf_nodes,
storage_context=storage_context
)
## 创建基础 Retriever 和自动合并 Retriever
base_retriever = base_index.as_retriever(similarity_top_k=6)
auto_merging_retriever = AutoMergingRetriever(
base_retriever,
storage_context,
verbose=True # 显示合并过程
)
## 准备测试问题
test_questions = [
"What are the characteristics of the Golden Cudgel in its different forms?",
]
print("=== Results from Auto Merging Retriever ===")
for question in test_questions:
print(f"\nQuestion: {question}")
# 使用 auto merging retriever 进行检索
merge_nodes = auto_merging_retriever.retrieve(question)
print(f"Retrieved {len(merge_nodes)} merged nodes:")
for node in merge_nodes:
print(f"\nSimilarity: {node.score}")
print(f"Content: {node.node.text}")
print("-" * 50)
print("\n=== Results from Base Retriever (Comparison) ===")
for question in test_questions:
print(f"\nQuestion: {question}")
# 使用 base retriever 进行检索
base_nodes = base_retriever.retrieve(question)
print(f"Retrieved {len(base_nodes)} base nodes:")
for node in base_nodes:
print(f"\nSimilarity: {node.score}")
print(f"Content: {node.node.text}")
print("-" * 50)
这种设计非常适合《The Legend of Godslayer: Monkey》这类复杂游戏系统，因为游戏中的许多功能彼此关联，玩家的问题也经常需要综合多个方面的信息，才能给出完整答案。例如，某种形态的特点可能涉及基础属性、连招系统、武器特性等多个方面。通过自动检索合并，系统可以在保持答案准确性的同时，提供更加完整、连贯的信息。
=== Results of the Automatic Merged Retriever ===
Question: What are the characteristics of the Ruyi Jingu Bang in different forms?
> Merging 3 nodes into parent node.
> Parent node id: cbb37aab-f161-4bb4-a1aa-f2adf47687a7.
> Parent node text: Exhibits different effects with different forms.
In the Vajra Form, the Ruyi Jingu Bang becomes heavy and thick, and every hit is full of power. In the Demon Buddha Form, the staff can extend and become slender, combining with spells to form long-range attacks. Besides the Ruyi Jingu Bang, there are various divine artifacts and treasures in the game that can be equipped and switched.
> Merging 1 nodes into parent node.
> Parent node id: 7b1eff02-fdfb-4dfb-81cd-0d75bf7868bc.
> Parent node text: The Vajra Form is adept at breaking enemy defenses with continuous heavy attacks, creating openings.
Retrieved 4 merged nodes:
Similarity: 0.8093140209301627
Content: Exhibits different effects with different forms. In the Vajra Form, the Ruyi Jingu Bang becomes heavy and thick, and every hit is full of power. In the Demon Buddha Form, the staff can extend and become slender, combining with spells to form long-range attacks. Besides the Ruyi Jingu Bang, there are various divine artifacts and treasures in the game that can be equipped and switched.
--------------------------------------------------
Similarity: 0.7931933447011333
Content: The Vajra Form is adept at breaking enemy defenses with continuous heavy attacks, creating openings.
--------------------------------------------------
Similarity: 0.7830123004681626
Content: Each form has its own unique advantages.
In the Vajra Form, Wukong’s strength surges, allowing for high physical damage, suitable for close combat.
--------------------------------------------------
Similarity: 0.7490986271802448
Content: The weapon system is also full of variety. The iconic Ruyi Jingu Bang can change its length and thickness at will,
--------------------------------------------------
=== Results of the Basic Retriever (Comparison) ===
Question: What are the characteristics of the Ruyi Jingu Bang in different forms?
Retrieved 3 basic nodes:
Similarity: 0.8440463514850339
Content: Exhibits different effects with different forms. In the Vajra Form, the Ruyi Jingu Bang becomes heavy and thick, and every hit is full of power. In the Demon Buddha Form,
--------------------------------------------------
Similarity: 0.817276846713401
Content: Besides the Ruyi Jingu Bang, there are various divine artifacts and treasures in the game that can be equipped and switched.
--------------------------------------------------
Similarity: 0.7931933447011333
Content: The Vajra Form is adept at breaking enemy defenses with continuous heavy attacks, creating openings.
对于基础 retriever，如果检索范围设置太小，信息可能不完整；如果检索范围设置太大，又可能引入无关信息。相比之下，AutoMergingRetriever 可以自动识别并合并相关文本片段，智能判断哪些信息应该整合在一起，从而提供更完整上下文。这种方法特别适合回答需要综合多处信息的复杂问题。
使用 RAPTOR 递归生成多级索引
RAPTOR，即 Recursive Abstractive Processing for Tree-Organized Retrieval，是斯坦福大学团队提出的一种复杂多级索引结构。它通过精心设计的策略，充分探索和利用每一层的信息。
图 6.7：RAPTOR 风格的递归聚类和摘要，用于多级检索。
RAPTOR 的检索和生成步骤与普通 RAG 相同。它的核心算法主要在索引创建部分，包括层级文档表示和逐层递归摘要。
Leaf nodes 表示原始文档的 chunks 或整个文档。系统会计算 leaf nodes 的 embeddings，并进行聚类，使用 GMM 和 UMAP。
低层文档，也就是 leaf nodes，被聚类后，每个 cluster 生成一个摘要。
这些摘要随后作为更高一层的输入，继续被聚类和摘要，递归执行，直到形成最终树结构。
最终树结构会被展平成一个 retrieval corpus，使用 vector search 进行高效信息查找。
它就像一棵倒置的树，根在上方，叶子在下方，每一层代表不同抽象级别的信息。层级越高，信息越概括、越抽象。每个 node 可以连接上下层相关 nodes，从而允许从宏观到微观层级浏览信息。
RAPTOR 索引生成策略的伪代码如下。
def recursive_embed_cluster_summarize(text_chunks, level=1, max_levels=3):
"""
Recursively perform embedding, clustering, and summarization on the given text chunks.
Each layer represents an abstraction level.
Parameters:
text_chunks: text chunks at the current layer (e.g., document chunks or higher-level summaries)
level: current layer
max_levels: maximum recursion depth
Returns:
List of summaries generated at this layer (to serve as input for the upper layer)
"""
### 对当前层的 Text Chunks 进行 Embedding
embeddings = embed_texts(text_chunks)
### 如果达到最大层级，或无法继续聚类，则结束递归
if level >= max_levels or len(text_chunks) <= 1:
return text_chunks
### 对 Embedding 结果进行聚类，可选 GMM+UMAP 或 K-Means
clusters = cluster_embeddings(embeddings)
### 为每个 Cluster 生成摘要
summaries = []
for cluster in clusters:
cluster_texts = [text_chunks[i] for i in cluster.indices]
summary = summarize_texts(cluster_texts)
summaries.append(summary)
### 递归调用，进入下一层
return recursive_embed_cluster_summarize(summaries, level + 1, max_levels)
Anna：可以这样理解吗：通过让大模型对低层内容进行总结和抽象，每往上一层，摘要就变得更加概括，在保留关键信息的同时去除次要细节？这类似于写文章时的层级关系：章节提纲 → 段落提纲 → 具体内容。例如，整体产品线介绍 → 产品类别概览 → 多个产品文档。同时，低层内容会根据相似度分组，确保相似项被归到一起；比如，把所有与游戏机制相关的文档聚合在一起。
Lewis：是的。论文作者认为，这种结构的主要优势在于，它通过层级化组织实现更有针对性的信息检索。多级摘要有助于访问不同粒度的信息，global clustering 和 local clustering 的结合提升了文档组织质量，而最终展平后的向量存储则保证了检索效率，支持从不同层级和视角进行信息检索。
由于该论文中的实现相对复杂，这里不提供详细示例。感兴趣的学习者可以参考 LangChain 复现代码以及对这篇论文的文档解释。
前向 / 后向链接：通过前后扩展连接相关 nodes
接下来，我们将探索一种叫做 Forward/Backward Augmentation 的技术，Lewis 称之为 forward/backward linking。这项技术利用文档中 nodes 之间的关系，尤其是相邻 nodes 之间的上下文连接，来增强检索上下文。
图 6.8：基于 query 的 node 扩展，跨 leaf、intermediate 和 root 层级。
在需要连贯上下文的问答任务中，例如分析文章或理解故事线，当 query 涉及“某个事件之后”或“某个事件之前”的信息时，检索系统可以自动推断并返回与被查询 node 相关的前文或后文。这增强了检索结果中的上下文信息，从而提供更加全面、连贯的回答。
对于前向 / 后向扩展技术，LlamaIndex 提供了以下两个实现工具：
PrevNextNodePostprocessor：允许在检索时获取当前 node 前后指定数量的 nodes。通过设置要扩展的 node 数量，即 num_nodes 参数，可以控制增强程度。
AutoPrevNextNodePostprocessor：可以从 query 内容中推断是否需要前向 / 后向扩展检索。例如，如果 query 包含 “after” 或 “before” 这样的关键词，postprocessor 会自动调整模式，并选择 “next”，即后向检索；“previous”，即前向检索；或者 “none”，即不进行前后扩展检索。
Alex：咦，Lewis，这项技术是不是很像节点句子滑动窗口检索？
Lewis：哈哈，方法上确实有相似之处。不过我正想问你，你能说出两者有什么区别吗？
Alex：嗯。前向 / 后向扩展检索是检索后的处理步骤；它不涉及索引构建。所以和节点句子滑动窗口检索不同，它不一定要求在切分向量时把文档拆成句子。而且，AutoPrevNextNodePostprocessor 可以根据 query 内容自动判断是否需要扩展，这增加了灵活性。
Lewis：没错。在 LlamaIndex 中，每个 node 的 metadata 都保存了前后 nodes 的 ID，这意味着任何 node 都可以在检索时按需自动扩展。通过理解 query 的具体内容，系统可以决定是否检索相邻 nodes，在不过度牺牲效率的情况下，提供更丰富、更相关的信息。此外，由于这种扩展是作为检索后处理步骤实现的，在 LlamaIndex 中，带有 Postprocessor 后缀的工具都是检索后处理工具，所以它相对独立，也更容易集成进现有 RAG workflow，而不需要重建索引。
下面的代码示例展示了一个智能游戏剧情检索系统，它可以通过分析上下文关系提供完整故事信息。完整代码可参考 github.com/PacktPublis…
from llama_index.core import VectorStoreIndex, StorageContext, Document, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.postprocessor import PrevNextNodePostprocessor, AutoPrevNextNodePostprocessor
from llama_index.llms.deepseek import DeepSeek
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
### 配置全局设置
Settings.llm = DeepSeek(model="deepseek-chat", temperature=0.1)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-zh")
Settings.node_parser = SentenceSplitter()
### 准备游戏剧情文本
game_story = """When Wukong first awakens, he finds himself trapped in an ancient cave. With blurred memories, he only remembers that he is the Great Sage Sun Wukong, but cannot recall why he is there. Inside the cave is a broken mirror; through it, he sees himself covered in wounds, and his legendary golden cudgel is now just a broken hilt. After leaving the cave, Wukong encounters a mysterious old monk. The old monk tells him that this is the ‘Illusory Realm’, a special space between reality and illusion. 500 years ago, Heaven suffered an unprecedented catastrophe: the gods fell, and the celestial world collapsed. At that time, Wukong was wreaking havoc in Heaven but was swept into the disaster, losing most of his powers and memories, and sealed in this world. The old monk suggests Wukong find the scattered fragments of his memory across the Illusory Realm. His first stop is the Wangchuan Temple in the east, where a Mirror of Memory is enshrined, which may help him recover some memories. However, the temple has been occupied by a group of evil demons, so Wukong must defeat them first. Within Wangchuan Temple, Wukong witnesses scenes from the Heaven Catastrophe through the Mirror of Memory. It turns out that a mysterious ancient force was manipulating everything behind the scenes, using the power of ‘the wishes of all beings’ to distort the rules of the universe. Although Wukong was mighty at the time, he still couldn’t prevent the disaster from happening. After regaining these memories, the old monk tells him that the next stop should be the Fiery Mountain in the west. There, a transformed demon clan holds even more truths. But Fiery Mountain is surrounded by blazing flames year-round, making it inaccessible to ordinary beings. Wukong must first find the legendary Samadhi Fire Armor to enter safely. In his search, Wukong reunites with his old friend, the Demon King. The Demon King tells him that after the collapse of Heaven, chaos erupted in the Six Realms, with various factions rising up—some in the name of rebuilding Heaven, others seeking to create new order. A greater disaster is brewing. After obtaining the Samadhi Fire Armor, Wukong successfully infiltrates Fiery Mountain. In a confrontation with the demon clan leader, he finally recalls more truths. The ancient force’s goal was not mere destruction, but to reshape the rules of the world. They believed there were fundamental flaws in the current order that bring suffering to all beings. Returning to the old monk, Wukong says he wants to unite all forces to fight against the power behind the scenes. But the old monk warns him that things may not be as simple as they appear. Whether the world order should be remade is a question without a standard answer. The old monk advises Wukong to seek more truth before making a decision. Wukong decides to set out for the Southern Sinking Star Sea, where, according to legend, there is an ancient library containing classics about the world’s origin. However, before he departs, the Illusory Realm suddenly shudders violently, as if some great upheaval is about to occur..."""
### 创建 Document 对象
documents = [Document(text=game_story)]
### 构建 Document Storage 和 Index，并使用 Settings 中的 node_parser 解析文档
nodes = Settings.node_parser.get_nodes_from_documents(documents)
### 添加 Nodes
docstore = SimpleDocumentStore()
docstore.add_documents(nodes)
### 创建 Storage Context
storage_context = StorageContext.from_defaults(docstore=docstore)
### 构建 Vector Index
index = VectorStoreIndex(nodes, storage_context=storage_context)
### 创建不同查询引擎。基础查询引擎
base_engine = index.as_query_engine(
similarity_top_k=1,
response_mode="tree_summarize"
)
### 带固定前后文的查询引擎
prev_next_engine = index.as_query_engine(
similarity_top_k=1,
node_postprocessors=[
PrevNextNodePostprocessor(docstore=docstore, num_nodes=2)
],
response_mode="tree_summarize"
)
### 带自动前后文的查询引擎
auto_engine = index.as_query_engine(
similarity_top_k=1,
node_postprocessors=[
AutoPrevNextNodePostprocessor(
docstore=docstore,
num_nodes=2,
verbose=True
)
],
response_mode="tree_summarize"
)
### 测试不同类型问题和不同查询引擎
test_questions = [
"What happened after Wukong regained his memory at Wangchuan Temple?", # 应查找后续上下文
"How did Wukong reach the Fire Mountain?", # 应同时查找前后上下文
"Why did Wukong wake up in the cave?", # 应查找前文上下文
]
虽然示例剧情非常简单，但特意设计了一条围绕揭开真相和寻找记忆的剧情线，以便更好展示系统如何处理文本中的前因后果。当玩家询问某个事件之前 / 之后发生了什么时，需要完整上下文，因为单个片段可能无法提供足够信息。
Question: Why did Wukong wake up in the cave?
> Postprocessor Predicted mode: previous
Answer: Wukong woke up in the cave because during the upheaval in Heaven five hundred years ago, when he caused havoc in the Heavenly Palace, the celestial court suffered a catastrophe, the gods fell, and the Heavenly Realm collapsed. Wukong was also swept into it, losing most of his magical power and memory, and was eventually sealed within this special space—the Illusory Realm.
你可以通过调整 chunk_size 和 num_nodes 等参数，平衡上下文范围，也可以修改故事内容，测试系统在不同场景中的表现。
Anna：这里的前向 / 后向扩展技术更关注文档内容的逻辑顺序，而父子文本块索引技术主要依赖层级结构，也就是 child text blocks 和 parent text blocks，来返回上下文，并不依赖内容的顺序关系。从自动化角度看，前向 / 后向扩展技术中的 AutoPrevNextNodePostprocessor 可以根据 query 推断是否需要扩展，而节点句子滑动窗口检索技术和父子文本块索引技术都基于预设范围，不具备自动判断能力。
混合检索：提升检索准确率并扩大覆盖范围
在前面的章节中，我们反复讨论了 hybrid indexing，也就是 hybrid retrieval 的前提和基础，的概念和实现方式。当数据库中存在多种类型的数据库或索引时，可以使用 hybrid retrieval 来提升检索准确率、扩大覆盖范围，并增强系统能力。
使用 ensemble retriever 结合 BM25 和语义检索
在本节中，我们将使用 LangChain 的 Ensemble Retriever 来组合不同类型 retrievers 的结果。混合检索的实现过程如下：
图 6.9：混合检索结合 BM25 关键词搜索和稠密向量检索。
两个 retrievers 并行运行：左侧是基于关键词匹配的 BM25 稀疏 retriever；右侧是执行语义相似度搜索的稠密 retriever。Ensemble Retriever 会组合两个 retrievers 的结果，使用混合检索策略进行整合，最终将相关文档传递给 generator 输出。
该架构的优势在于，它结合了关键词匹配的精准性和语义检索的灵活性。BM25 擅长捕捉精确文本匹配，非常适合查找包含精确术语或紧密相关词汇的文档；而 dense retriever 可以理解 query 背后的上下文或含义，提供更深层的语义相似性。因此，在许多应用场景中，采用混合检索策略通常比单独使用一种 retriever 获得更好结果。
下面的代码示例使用 BM25 retriever 进行关键词匹配，并使用向量存储进行语义检索。随后，EnsembleRetriever 将这两种检索方法组合起来，BM25 和语义检索的权重可以按需调整，以优化检索表现，从而实现 hybrid retrieval。完整代码可参考 github.com/PacktPublis…
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_deepseek import ChatDeepSeek
from langchain.chains import RetrievalQA
### 系统定义文档，关注具体游戏机制和系统
system_docs = [
"“The Legend of Exorcising the God·Husun” features a unique transformation system as the core combat mechanic",
"In Vajra Form, heavy weapons can be used, increasing attack and defense",
"In Demon Buddha Form, magic attacks are emphasized, unleashing powerful spell damage",
"Different forms can be switched at any time in battle for combos",
"The game has three difficulty levels: Normal, Hard, and Asura"
]
### Lore 文档，关注故事和背景设定
lore_docs = [
"The game is set in a fictional mythical world, blending elements of Eastern mythology",
"Sun Wukong is reawakened after being sealed for 500 years in the game",
"There exist multiple factions of gods and demons in the world, such as Buddhism and Daoism",
"The player controls Sun Wukong to uncover the truth among the various forces",
"Scenes include ink-painting-style mountains and architecture"
]
### 创建两种不同 Retriever：BM25 和 Vector Retriever
from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import FAISS
from langchain.retrievers import EnsembleRetriever
### 创建 BM25 Retriever
bm25_retriever = BM25Retriever.from_texts(
system_docs + lore_docs,
metadatas=[{"source": "system" if i < len(system_docs) else "lore"}
for i in range(len(system_docs) + len(lore_docs))]
)
bm25_retriever.k = 2
### 创建 Vector Retriever
embed_model = HuggingFaceEmbeddings(model_name="BAAI/bge-small-zh")
vectorstore = FAISS.from_texts(
system_docs + lore_docs,
embed_model,
metadatas=[{"source": "system" if i < len(system_docs) else "lore"}
for i in range(len(system_docs) + len(lore_docs))]
)
faiss_retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
### 创建 Hybrid Retriever
ensemble_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, faiss_retriever],
weights=[0.5, 0.5]
)
创建使用 hybrid retriever 的 QA chain，以及使用单一 retriever 的 QA chain，用于对比：
llm = ChatDeepSeek(model="deepseek-chat")
## 创建 Hybrid Retrieval QA Chain
ensemble_qa = RetrievalQA.from_chain_type(
llm=llm,
retriever=ensemble_retriever,
return_source_documents=True
)
## 创建单独 Vector Retrieval QA Chain，用于对比
vector_qa = RetrievalQA.from_chain_type(
llm=llm,
retriever=faiss_retriever,
return_source_documents=True
)
## 测试不同类型查询
test_queries = [
"What is the transformation system like in the game?", # 系统设定查询
"What is the world background of the game?", # 背景设定查询
"What combat forms does Wukong have?" # 混合查询
]
for query in test_queries:
print(f"\nQuery: {query}")
print("\n1. Hybrid Retrieval Results:")
ensemble_docs = ensemble_retriever.invoke(query)
print("Retrieved documents:")
for i, doc in enumerate(ensemble_docs, 1):
print(f"{i}. [{doc.metadata['source']}] {doc.page_content}")
print("\n2. Vector Retrieval Results (Comparison):")
vector_docs = faiss_retriever.invoke(query)
print("Retrieved documents:")
for i, doc in enumerate(vector_docs, 1):
print(f"{i}. [{doc.metadata['source']}] {doc.page_content}")
## 测试 QA 表现
print("\n=== QA Performance Test ===")
test_questions = [
"What are the characteristics of the Vajra form?",
"How are the factions distributed in the game?",
]
for question in test_questions:
print(f"\nQuestion: {question}")
print("\n1. Answer using Hybrid Retrieval:")
ensemble_result = ensemble_qa.invoke({"query": question})
print(f"Answer: {ensemble_result['result']}")
print("\nSource documents used:")
for i, doc in enumerate(ensemble_result['source_documents'], 1):
print(f"{i}. [{doc.metadata['source']}] {doc.page_content}")
print("\n2. Answer using pure vector retrieval (for comparison):")
vector_result = vector_qa.invoke({"query": question})
print(f"Answer: {vector_result['result']}")
print("\nSource documents used:")
for i, doc in enumerate(vector_result['source_documents'], 1):
print(f"{i}. [{doc.metadata['source']}] {doc.page_content}")
输出示例如下：
Query: What is the transformation system like in the game?
Hybrid Retrieval Results:
Documents Retrieved:
[lore] The game scenes include ink-painting style mountains and architecture
[system] “Annihilation God Chronicle: Housun” uses a unique transformation system as the core combat mechanic
[lore] The player, as Sun Wukong, must seek the truth among various factions
[system] In combat, you can switch between different forms at any time to execute combos
Vector Retrieval Results (comparison):
Documents Retrieved:
[system] “Annihilation God Chronicle: Housun” uses a unique transformation system as the core combat mechanic
[system] In combat, you can switch between different forms at any time to execute combos
Question: What are the features of the Vajra Form?
Answer using hybrid retrieval:
Answer: In **Vajra Form**, you can use heavy weapons, increasing attack and defense power.
Source documents used:
[lore] The game scenes include ink-painting style mountains and architecture
[system] In **Vajra Form**, you can use heavy weapons, increasing attack and defense power
[lore] The player, as Sun Wukong, must seek the truth among various factions
[system] **Demon Buddha Form** focuses on magical attacks and can unleash powerful spell damage
Answer using pure vector retrieval (comparison):
Answer: In **Vajra Form**, you can use heavy weapons, increasing attack and defense power.
Source documents used:
[system] In **Vajra Form**, you can use heavy weapons, increasing attack and defense power
[system] **Demon Buddha Form** focuses on magical attacks and can unleash powerful spell damage
通过这种方式，你可以清楚观察每个 retriever 返回的具体文档，并理解最终 ensemble retriever 如何平衡和选择这些结果。这对于调试和优化检索策略尤其有帮助。测试过程中，hybrid retrieval 方法展示了同时兼顾关键词匹配和语义相关性的能力，因此可以提供更全面、更准确的搜索结果。这种方法适用于多种应用场景，例如在数字图书馆中查找资源，以及高效搜索大型文档集合。
当我们将上面的代码与第 4 章中的 hybrid retrieval 实现进行对比时，会发现使用 LangChain 框架大大简化了这一过程。具体来说，BM25Retriever 和 vectorstore.as_retriever 这两个组件，封装了信息 embedding、向量存储和检索背后的复杂逻辑，使开发者无需深入底层细节，也能高效实现信息检索。集成不同 retrievers 的关键功能主要由 EnsembleRetriever 类实现，它负责组合不同 retrievers 的结果，并基于预设权重进行综合评估，以优化检索表现。
接下来，我们看看 EnsembleRetriever 类的核心逻辑。
class EnsembleRetriever(BaseRetriever):
retrievers: List[RetrieverLike]
weights: List[float]
c: int = 60
id_key: Optional[str] = None
def weighted_reciprocal_rank(
self, doc_lists: List[List[Document]]
) -> List[Document]:
rrf_score: Dict[str, float] = defaultdict(float)
for doc_list, weight in zip(doc_lists, self.weights):
for rank, doc in enumerate(doc_list, start=1):
rrf_score[
doc.page_content
if self.id_key is None
else doc.metadata[self.id_key]
] += weight / (rank + self.c)
all_docs = chain.from_iterable(doc_lists)
sorted_docs = sorted(
unique_by_key(
all_docs,
lambda doc: doc.page_content
if self.id_key is None
else doc.metadata[self.id_key],
),
reverse=True,
key=lambda doc: rrf_score[
doc.page_content if self.id_key is None else doc.metadata[self.id_key]
],
)
return sorted_docs
这里的关键点是 Weighted Reciprocal Rank Fusion，也就是 ensemble retrieval 的核心算法。它在 weighted_reciprocal_rank 方法中实现。
分数计算语句如下：
rrf_score[doc_key] += weight / (rank + self.c)
对于每个文档，系统会根据它在各个 retriever 结果中的排名计算分数。排名越靠前，也就是 rank 数字越小，分数越高。weight 用于调整不同 retriever 的重要性。之后，使用 unique_by_key 函数去除重复文档，再根据计算出的 RRF scores 对文档排序。最后返回排序并去重后的文档列表。关于 RRF 算法的更多细节，会在后续课程中介绍。
这种实现会组合多个 retrievers 的结果，同时考虑每个 retriever 的权重，以及文档在每个 retriever 中的排名，能够有效融合不同检索策略的优势，从而提升整体检索质量。使用时，只需要初始化 EnsembleRetriever，并提供 retrievers 列表及其对应权重。
ensemble_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, faiss_retriever],
weights=[0.6, 0.4]
)
这个设置给 BM25 retriever 更高权重，可能更适合处理关键词匹配类查询，同时也利用了 Faiss retriever 的语义理解能力。
使用 MultiVectorRetriever 实现多表示索引
多表示索引是一种通过为文档创建多种表示来提升检索效果的技术。事实上，在前面的章节中，我们已经多次实现过各种类型的多表示索引。例如，在 Parent-child text chunk retrieval 中提到的父子文本块，以及 Creating hierarchical indexes with HierarchicalNodeParser 中介绍的 HierarchicalNodeParser，都可以用于为同一文档生成多种表示。
图 6.10：针对非结构化文档、表格和图像的多索引检索。
下面的代码示例演示如何在 LangChain 中使用 MultiVectorRetriever 快速实现多表示索引。
### 加载文档
from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")
docs = loader.load()
### 创建文档摘要
from langchain_core.prompts import ChatPromptTemplate
from langchain_deepseek import ChatDeepSeek
from langchain_core.output_parsers import StrOutputParser
chain = (
{"doc": lambda x: x.page_content}
| ChatPromptTemplate.from_template("Summarize the following document:\n\n{doc}")
| ChatDeepSeek(model="deepseek-chat")
| StrOutputParser()
)
summaries = chain.batch(docs, {"max_concurrency": 5})
### 设置 Multi-Vector Retriever
from langchain.storage import InMemoryByteStore
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.retrievers.multi_vector import MultiVectorRetriever
embed_model = HuggingFaceEmbeddings(model_name="BAAI/bge-m3")
vectorstore = Chroma(collection_name="summaries", embedding_function=embed_model)
store = InMemoryByteStore()
id_key = "doc_id"
retriever = MultiVectorRetriever(
vectorstore=vectorstore,
byte_store=store,
id_key=id_key,
)
### 将文档和摘要添加到 Retriever
import uuid
from langchain_core.documents import Document
doc_ids = [str(uuid.uuid4()) for _ in docs]
summary_docs = [
Document(page_content=s, metadata={id_key: doc_ids[i]})
for i, s in enumerate(summaries)
]
retriever.vectorstore.add_documents(summary_docs)
retriever.docstore.mset(list(zip(doc_ids, docs)))
### 查询 Retriever
query = "Memory in agents"
retrieved_docs = retriever.get_relevant_documents(query, n_results=1)
在这个多表示索引中，相似度搜索是通过摘要完成的，这更省时间，也能获得更精确结果。找到相关摘要之后，可以使用文档的 doc_id 快速定位原始文档。这种设计在处理长文档时特别有用，因为它允许我们在不丢失原始文档完整信息的情况下，保持检索效率。
混合查询与查询路由
现在回过头来看 query routing 技术，我们会意识到，集成检索和多表示检索都与 logical routing 密切相关。一旦构建了多种类型或多层级索引，关键就变成了：如何判断某个具体 query 最适合使用哪一种表示。
例如，对于 “Help me find battle data about Wukong” 这类查询，优先使用关键词或结构化表示进行检索可能更合适，因为这类查询通常要求精确匹配具体信息或数据点。相比之下，对于 “Why is ‘Chronicle of the Godslayer: Monkey’ popular among players?” 这类查询，更适合使用语义表示进行检索，因为它关注的是理解 query 背后的意图，并寻找更相关的内容。
基于 query 特征选择最合适检索路径的过程，本质上就是 query routing。在复杂场景中，可能需要同时利用多种表示，并合并它们的检索结果。这就形成了一种 hybrid query system，可以根据 query 特征动态决定不同检索路径的权重分配。
总结
本节讨论的技术，大多与结构化文本块生成和检索优化密切相关，目标是提供基础思路，启发更深入的应用和创新。下面概览几项关键技术及其作用。
Small to large：该技术结合了检索精度和生成所需的大上下文，从而提升模型回答问题的准确性和相关性。
Coarse to fine：该技术构建层级索引，并为每个 node 生成文本摘要，实现从粗粒度到细粒度的检索过程。
Hierarchical merging：通过为同一个文本块创建多种类型的索引，该技术增强检索全面性并扩大覆盖范围。
Sequential chaining：该技术基于 query 动态扩展上下文，特别适合处理具有强因果关系和时间关联的文档。
Hybrid retrieval：该技术整合多个数据源或存储表示，是实际应用中常见的检索优化策略。
需要注意的是，虽然上述技术提供了有价值的方法论指导，但实际项目中的关键，是识别影响检索质量和整个 RAG 系统质量的瓶颈。
希望通过理解这些技术，你可以在自己的项目中灵活应用，而不是被现有模式束缚，从而创造出最适合自身需求的索引构建方式和高效检索机制。
