RAG & Embeddings & Vector Store 实战版

113 阅读19分钟

RAG & Embeddings & Vector Store 实战版

💡 内容介绍

  1. 如何用你的垂域数据补充 LLM 的能力
  2. 如何构建你的垂域(向量)知识库
  3. 搭建一套完整 RAG 系统需要哪些模块
  4. 搭建 RAG 系统时更多的有用技巧

一、什么是检索增强的生成模型(RAG)

1.1 大模型目前固有的局限性

  1. LLM的知识不是实时的
  2. LLM 可能不知道你私有的领域/业务知识

1.2 检索增强生成

RAG(Retrieval Augmented Generation)顾名思义,通过检索的方法来增强生成模型的能力。

类比:你可以把这个过程想象成开卷考试。让 LLM 先翻书,再回答问题。

二、RAG系统的基本搭建流程

image.png 搭建过程:

  1. 文档加载,并按一定条件切割成片段
  2. 将切割的文本片段灌入检索引擎
  3. 封装检索接口
  4. 构建调用流程:Query -> 检索 -> Prompt -> LLM -> 回复

2.0 环境准备

  1. 包和环境管理工具:Conda
  2. python版本:3.10.16
  3. web交互式开发环境:Jupyter Lab(可以边写代码,边记markdown)
  4. 开发工具:Visual Studio Code(直接下最新版)

2.1 文档的加载与切割

!pip install 只在 Jupyter Notebook 或 Google Colab 这样的环境中有效。

# 安装或升级 OpenAI 的 Python SDK
# !pip install --upgrade openai

# 安装 pdf 解析库
# !pip install pdfminer.six
from pdfminer.high_level import extract_pages
from pdfminer.layout import LTTextContainer

# 写一个extract_text_from_pdf()
def extract_text_from_pdf(filename, page_numbers=None, min_line_length=1):
    '''从 PDF 文件中(按指定页码)提取文字'''
    paragraphs = []
    buffer = ''
    full_text = ''
    # 提取全部文本
    for i, page_layout in enumerate(extract_pages(filename)):
        # 如果指定了页码范围,跳过范围外的页
        if page_numbers is not None and i not in page_numbers:
            continue
        for element in page_layout:
            if isinstance(element, LTTextContainer):
                full_text += element.get_text() + '\n'
    # 按空行分隔,将文本重新组织成段落
    lines = full_text.split('\n')
    for text in lines:
        if len(text) >= min_line_length:
            buffer += (' '+text) if not text.endswith('-') else text.strip('-')
        elif buffer:
            paragraphs.append(buffer)
            buffer = ''
    if buffer:
        paragraphs.append(buffer)
    return paragraphs

# 调用extract_text_from_pdf()
# llama2.pdf是一个pdf文件,放在当前目录以便读取
paragraphs = extract_text_from_pdf("llama2.pdf", min_line_length=10)

for para in paragraphs[:4]:
    print(para+"\n")

运行上述代码,输出:

Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron∗ Louis Martin† Kevin Stone† Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom∗ GenAI, Meta In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based onour human evaluations for helpfulness and safety, may be a suitable substitute for closed source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

2.2 LLM接口封装

# 安装或升级 `python-dotenv` 库,用于加载环境变量
# !pip install -U python-dotenv

from openai import OpenAI
import os

# 加载环境变量
from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv(), verbose=True)  # 读取本地.env 文件,里面定义了 OPENAI_API_KEY

client = OpenAI()

def get_completion(prompt, model="gpt-4o"):
    '''封装 openai 接口'''
    messages = [{"role": "user", "content": prompt}]
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,  # 模型输出的随机性,0 表示随机性最小
    )
    return response.choices[0].message.content
OPENAI_API_KEY=自行获取
OPENAI_BASE_URL=https://api.openai-hk.com/v1

OPENAI_API_KEY获取方式(推荐网站),GPT4.0 API KEY By OPENAI HK 中转ChatGPT (openai-hk.com)

2.3 Prompt 模板

def build_prompt(prompt_template, **kwargs):
    '''将 Prompt 模板赋值'''
    inputs = {}
    for k, v in kwargs.items():
        if isinstance(v, list) and all(isinstance(elem, str) for elem in v):
            val = '\n\n'.join(v)
        else:
            val = v
        inputs[k] = val
    return prompt_template.format(**inputs)
prompt_template = """
你是一个问答机器人。
你的任务是根据下述给定的已知信息回答用户问题。

已知信息:
{context} # 检索出来的原始文档

用户问:
{query} # 用户的提问

如果已知信息不包含用户问题的答案,或者已知信息不足以回答用户的问题,请直接回复"我无法回答您的问题"。
请不要输出已知信息中不包含的信息或答案。
请用中文回答用户问题。
"""

三、向量检索

3.1 什么是向量

向量是一种有大小和方向的数学对象。它可以表示为从一个点到另一个点的有向线段。例如,二维空间中的向量可以表示为 (x,y)(x,y),表示从原点 (0,0)(0,0) 到点 (x,y)(x,y) 的有向线段。

image.png 以此类推,我可以用一组坐标 (x0,x1,,xN1)(x_0, x_1, \ldots, x_{N-1}) 表示一个 NN 维空间中的向量,NN 叫向量的维度。

3.1.1 文本向量(Text Embeddings)
  1. 将文本转成一组 NN 维浮点数,即文本向量又叫 Embeddings
  2. 向量之间可以计算距离,距离远近对应语义相似度大小

image.png

3.1.2 文本向量是怎么得到的(选)

  1. 构建相关(正例)与不相关(负例)的句子对样本
  2. 训练双塔式模型,让正例间的距离小,负例间的距离大

例如:

image.png

扩展阅读:https://www.sbert.net

3.2 向量间的相似度计算

image.png

import numpy as np
from numpy import dot
from numpy.linalg import norm

def cos_sim(a, b):
    '''余弦距离 -- 越大越相似'''
    return dot(a, b)/(norm(a)*norm(b))


def l2(a, b):
    '''欧氏距离 -- 越小越相似'''
    x = np.asarray(a)-np.asarray(b)
    return norm(x)

# 这里用的text-embedding-ada-002
def get_embeddings(texts, model="text-embedding-ada-002", dimensions=None):
    '''封装 OpenAI 的 Embedding 模型接口'''
    if model == "text-embedding-ada-002":
        dimensions = None
    if dimensions:
        data = client.embeddings.create(
            input=texts, model=model, dimensions=dimensions).data
    else:
        data = client.embeddings.create(input=texts, model=model).data
    return [x.embedding for x in data]
    
test_query = ["测试文本"]
    vec = get_embeddings(test_query)[0]
    print(f"Total dimension: {len(vec)}")
    print(f"First 10 elements: {vec[:10]}")
    

Total dimension: 1536
First 10 elements: [-0.007304091472178698, -0.006229960359632969, -0.010646641254425049, 0.0014391535660251975, -0.010704899206757545, 0.029274623841047287, -0.019807705655694008, 0.005487171467393637, -0.016865678131580353, -0.011979292146861553]
query = "国际争端"

# 且能支持跨语言
# query = "global conflicts"

documents = [
    "联合国就苏丹达尔富尔地区大规模暴力事件发出警告",
    "土耳其、芬兰、瑞典与北约代表将继续就瑞典“入约”问题进行谈判",
    "日本岐阜市陆上自卫队射击场内发生枪击事件 3人受伤",
    "国家游泳中心(水立方):恢复游泳、嬉水乐园等水上项目运营",
    "我国首次在空间站开展舱外辐射生物学暴露实验",
]

query_vec = get_embeddings([query])[0]
doc_vecs = get_embeddings(documents)

print("Query与自己的余弦距离: {:.2f}".format(cos_sim(query_vec, query_vec)))
print("Query与Documents的余弦距离:")
for vec in doc_vecs:
    print(cos_sim(query_vec, vec))

print()

print("Query与自己的欧氏距离: {:.2f}".format(l2(query_vec, query_vec)))
print("Query与Documents的欧氏距离:")
for vec in doc_vecs:
    print(l2(query_vec, vec))
Query与自己的余弦距离: 1.00 
Query与Documents的余弦距离: 0.8224810779975097 0.8299968969406545 0.798096878742543 0.7669367418371253 0.7933908049643592 
Query与自己的欧氏距离: 0.00 
Query与Documents的欧氏距离: 0.5958505343143035 0.5831005284209486 0.6354574886516438 0.6827345862824111 0.6428206511188914

3.3 OpenAI 新发布的两个 Embedding 模型

2024 年 1 月 25 日,OpenAI 新发布了两个 Embedding 模型

  • text-embedding-3-large
  • text-embedding-3-small

其最大特点是,支持自定义的缩短向量维度,从而在几乎不影响最终效果的情况下降低向量检索与相似度计算的复杂度。

通俗的说:越大越准、越小越快。 官方公布的评测结果:

image.png

注:MTEB 是一个大规模多任务的 Embedding 模型公开评测集 model = "text-embedding-3-large" dimensions = 256

model = "text-embedding-3-large"
dimensions = 256

query = "国际争端"

# 且能支持跨语言
# query = "global conflicts"

documents = [
    "联合国就苏丹达尔富尔地区大规模暴力事件发出警告",
    "土耳其、芬兰、瑞典与北约代表将继续就瑞典“入约”问题进行谈判",
    "日本岐阜市陆上自卫队射击场内发生枪击事件 3人受伤",
    "国家游泳中心(水立方):恢复游泳、嬉水乐园等水上项目运营",
    "我国首次在空间站开展舱外辐射生物学暴露实验",
]

query_vec = get_embeddings([query], model=model, dimensions=dimensions)[0]
doc_vecs = get_embeddings(documents, model=model, dimensions=dimensions)

print("向量维度: {}".format(len(query_vec)))

print()

print("Query与Documents的余弦距离:")
for vec in doc_vecs:
    print(cos_sim(query_vec, vec))

print()

print("Query与Documents的欧氏距离:")
for vec in doc_vecs:
    print(l2(query_vec, vec))
向量维度: 256

Query与Documents的余弦距离:
0.277412197853655
0.3359302597197623
0.12432423954047654
0.16717977147818797
0.12355951057682035

Query与Documents的欧氏距离:
1.2021545803883171
1.1524493185495321
1.323386397993211
1.2905969591781052
1.323964126465252
扩展阅读:这种可变长度的 Embedding 技术背后的原理叫做 Matryoshka Representation Learning

3.4 向量数据库

向量数据库,是专门为向量检索设计的中间件!

向量数据库其实最早在传统的人工智能和机器学习场景中就有所应用。在大模型兴起后,由于目前大模型的token数限制,很多开发者倾向于将数据量庞大的知识、新闻、文献、语料等先通过嵌入(embedding)算法转变为向量数据,然后存储在Chroma等向量数据库中。当用户在大模型输入问题后,将问题本身也embedding,转化为向量,在向量数据库中查找与之最匹配的相关知识,组成大模型的上下文,将其输入给大模型,最终返回大模型处理后的文本给用户,这种方式不仅降低大模型的计算量,提高响应速度,也降低成本,并避免了大模型的tokens限制,是一种简单高效的处理手段。此外,向量数据库还在大模型记忆存储等领域发挥其不可替代的作用。

3.4.1 Chroma 向量数据库

官方文档:docs.trychroma.com/docs/overvi…

1738993936021.jpg

# !pip install chromadb

import chromadb
from chromadb.config import Settings

# 为了演示方便,我们只取两页(第一章)
paragraphs = extract_text_from_pdf(
    "llama2.pdf",
    page_numbers=[2, 3],
    min_line_length=10
)

class MyVectorDBConnector:
    def __init__(self, collection_name, embedding_fn):
        # 内存模式
        chroma_client = chromadb.Client(Settings(allow_reset=True))
        # 数据持久化
        # chroma_client = chromadb.PersistentClient(path="./chroma")

        # 注意:为了演示,实际不需要每次 reset(),并且是不可逆的!
        chroma_client.reset()

        # 创建一个 collection
        self.collection = chroma_client.get_or_create_collection(name=collection_name)
        self.embedding_fn = embedding_fn

    def add_documents(self, documents):
        '''向 collection 中添加文档与向量'''
        self.collection.add(
            embeddings=self.embedding_fn(documents),  # 每个文档的向量
            documents=documents,  # 文档的原文
            ids=[f"id{i}" for i in range(len(documents))]  # 每个文档的 id
        )

    def search(self, query, top_n):
        '''检索向量数据库'''
        results = self.collection.query(
            query_embeddings=self.embedding_fn([query]),
            n_results=top_n
        )
        return results
        
# 创建一个向量数据库对象
vector_db = MyVectorDBConnector("demo", get_embeddings)
# 向向量数据库中添加文档
vector_db.add_documents(paragraphs)

user_query = "Llama 2有多少参数"
# user_query = "Does Llama 2 have a conversational variant"
results = vector_db.search(user_query, 2)

for para in results['documents'][0]:
    print(para+"\n")
1. Llama 2, an updated version of Llama 1, trained on a new mix of publicly available data. We also increased the size of the pretraining corpus by 40%, doubled the context length of the model, and adopted grouped-query attention (Ainslie et al., 2023). We are releasing variants of Llama 2 with 7B, 13B, and 70B parameters. We have also trained 34B variants, which we report on in this paper but are not releasing.§

 In this work, we develop and release Llama 2, a family of pretrained and fine-tuned LLMs, Llama 2 and Llama 2-Chat, at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested, Llama 2-Chat models generally perform better than existing open-source models. They also appear to be on par with some of the closed-source models, at least on the human evaluations we performed (see Figures 1 and 3). We have taken measures to increase the safety of these models, using safety-specific data annotation and tuning, as well as conducting red-teaming and employing iterative evaluations. Additionally, this paper contributes a thorough description of our fine-tuning methodology and approach to improving LLM safety. We hope that this openness will enable the community to reproduce fine-tuned LLMs and continue to improve the safety of those models, paving the way for more responsible development of LLMs. We also share novel observations we made during the development of Llama 2 and Llama 2-Chat, such as the emergence of tool usage and temporal organization of knowledge.
澄清几个关键概念:
  • 向量数据库的意义是快速的检索;
  • 向量数据库本身不生成向量,向量是由 Embedding 模型产生的;
  • 向量数据库与传统的关系型数据库是互补的,不是替代关系,在实际应用中根据实际需求经常同时使用。
3.4.2 Chroma 向量数据库服务

Server 端

chroma run --path /db_path

Client 端

import chromadb
chroma_client = chromadb.HttpClient(host='localhost', port=8000)
3.4.3 主流向量数据库功能对比

image.png

扩展阅读:https://guangzhengli.com/blog/zh/vector-database

3.5 基于向量检索的RAG

class RAG_Bot:
    def __init__(self, vector_db, llm_api, n_results=2):
        self.vector_db = vector_db
        self.llm_api = llm_api
        self.n_results = n_results

    def chat(self, user_query):
        # 1. 检索
        search_results = self.vector_db.search(user_query, self.n_results)

        # 2. 构建 Prompt
        prompt = build_prompt(
            prompt_template, context=search_results['documents'][0], query=user_query)

        # 3. 调用 LLM
        response = self.llm_api(prompt)
        return response
        
# 创建一个RAG机器人
bot = RAG_Bot(
    vector_db,
    llm_api=get_completion
)

user_query = "llama 2有多少参数?"

response = bot.chat(user_query)

print(response)
Llama 2 有 7B、13B 和 70B 参数的变体。

其实到这里rag就实战就基本完成了。

四、实战 RAG 系统的进阶知识

4.1 文本分割的粒度

缺陷

  1. 粒度太大可能导致检索不精准,粒度太小可能导致信息不全面
  2. 问题的答案可能跨越两个片段
# 创建一个向量数据库对象
vector_db = MyVectorDBConnector("demo_text_split", get_embeddings)
# 向向量数据库中添加文档
vector_db.add_documents(paragraphs)

# 创建一个RAG机器人
bot = RAG_Bot(
    vector_db,
    llm_api=get_completion
)

# user_query = "llama 2有商用许可协议吗"
user_query="llama 2 chat有多少参数"
search_results = vector_db.search(user_query, 2)

for doc in search_results['documents'][0]:
    print(doc+"\n")

print("====回复====")
bot.chat(user_query)

In this work, we develop and release Llama 2, a family of pretrained and fine-tuned LLMs, Llama 2 and Llama 2-Chat, at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested, Llama 2-Chat models generally perform better than existing open-source models. They also appear to be on par with some of the closed-source models, at least on the human evaluations we performed (see Figures 1 and 3). We have taken measures to increase the safety of these models, using safety-specific data annotation and tuning, as well as conducting red-teaming and employing iterative evaluations. Additionally, this paper contributes a thorough description of our fine-tuning methodology and approach to improving LLM safety. We hope that this openness will enable the community to reproduce fine-tuned LLMs and continue to improve the safety of those models, paving the way for more responsible development of LLMs. We also share novel observations we made during the development of Llama 2 and Llama 2-Chat, such as the emergence of tool usage and temporal organization of knowledge.

 2. Llama 2-Chat, a fine-tuned version of Llama 2 that is optimized for dialogue use cases. We release

====回复====

这个回答无法令人满意

改进: 按一定粒度,部分重叠式的切割文本,使上下文更完整

# !pip install nltk

from nltk.tokenize import sent_tokenize
import json

# chunk_size 一般根据文档内容或大小来设置
# overlap_size 一般设置 chunk_size 大小的10%-20%之间
def split_text(paragraphs, chunk_size=300, overlap_size=100):
    '''按指定 chunk_size 和 overlap_size 交叠割文本'''
    sentences = [s.strip() for p in paragraphs for s in sent_tokenize(p)]
    chunks = []
    i = 0
    while i < len(sentences):
        chunk = sentences[i]
        overlap = ''
        prev_len = 0
        prev = i - 1
        # 向前计算重叠部分
        while prev >= 0 and len(sentences[prev])+len(overlap) <= overlap_size:
            overlap = sentences[prev] + ' ' + overlap
            prev -= 1
        chunk = overlap+chunk
        next = i + 1
        # 向后计算当前chunk
        while next < len(sentences) and len(sentences[next])+len(chunk) <= chunk_size:
            chunk = chunk + ' ' + sentences[next]
            next += 1
        chunks.append(chunk)
        i = next
    return chunks
   
此处 sent_tokenize 为针对英文的实现,针对中文的实现请参考 chinese_utils.py
chunks = split_text(paragraphs, 300, 100)

# 创建一个向量数据库对象
vector_db = MyVectorDBConnector("demo_text_split", get_embeddings)
# 向向量数据库中添加文档
vector_db.add_documents(chunks)
# 创建一个RAG机器人
bot = RAG_Bot(
    vector_db,
    llm_api=get_completion
)

# user_query = "llama 2有商用许可协议吗"
user_query="llama 2 chat有多少参数"

search_results = vector_db.search(user_query, 2)
for doc in search_results['documents'][0]:
    print(doc+"\n")

response = bot.chat(user_query)
print("====回复====")
print(response)

输出,略。此时的回答明显质量更高。

4.2 检索后排序

问题: 有时,最合适的答案不一定排在检索的最前面

user_query = "how safe is llama 2"
search_results = vector_db.search(user_query, 5)

for doc in search_results['documents'][0]:
    print(doc+"\n")

response = bot.chat(user_query)
print("====回复====")
print(response)
We believe that the open release of LLMs, when done safely, will be a net benefit to society. Like all LLMs, Llama 2 is a new technology that carries potential risks with use (Bender et al., 2021b; Weidinger et al., 2021; Solaiman et al., 2023).

We also share novel observations we made during the development of Llama 2 and Llama 2-Chat, such as the emergence of tool usage and temporal organization of knowledge. Figure 3: Safety human evaluation results for Llama 2-Chat compared to other open-source and closed source models.

In this work, we develop and release Llama 2, a family of pretrained and fine-tuned LLMs, Llama 2 and Llama 2-Chat, at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested, Llama 2-Chat models generally perform better than existing open-source models.

Additionally, these safety evaluations are performed using content standards that are likely to be biased towards the Llama 2-Chat models. We are releasing the following models to the general public for research and commercial use‡: 1.

We provide a responsible use guide¶ and code examples‖ to facilitate the safe deployment of Llama 2 and Llama 2-Chat. More details of our responsible release strategy can be found in Section 5.3.

====回复====
我无法回答您的问题。

方案:

  1. 检索时过招回一部分文本
  2. 通过一个排序模型对 query 和 document 重新打分排序

image.png

以下代码运行前要确保能访问 Hugging Face!
# !pip install sentence_transformers

from sentence_transformers import CrossEncoder

model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2', max_length=512) # 英文,模型较小
# model = CrossEncoder('BAAI/bge-reranker-large', max_length=512) # 多语言,国产,模型较大

user_query = "how safe is llama 2"
# user_query = "llama 2安全性如何"
scores = model.predict([(user_query, doc) for doc in search_results['documents'][0]])
# 按得分排序
sorted_list = sorted(zip(scores, search_results['documents'][0]), key=lambda x: x[0], reverse=True)
for score, doc in sorted_list:
    print(f"{score}\t{doc}\n")
6.613734722137451	We believe that the open release of LLMs, when done safely, will be a net benefit to society. Like all LLMs, Llama 2 is a new technology that carries potential risks with use (Bender et al., 2021b; Weidinger et al., 2021; Solaiman et al., 2023).

5.310717582702637	In this work, we develop and release Llama 2, a family of pretrained and fine-tuned LLMs, Llama 2 and Llama 2-Chat, at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested, Llama 2-Chat models generally perform better than existing open-source models.

4.709955215454102	We provide a responsible use guide¶ and code examples‖ to facilitate the safe deployment of Llama 2 and Llama 2-Chat. More details of our responsible release strategy can be found in Section 5.3.

4.5439653396606445	We also share novel observations we made during the development of Llama 2 and Llama 2-Chat, such as the emergence of tool usage and temporal organization of knowledge. Figure 3: Safety human evaluation results for Llama 2-Chat compared to other open-source and closed source models.

4.0338897705078125	Additionally, these safety evaluations are performed using content standards that are likely to be biased towards the Llama 2-Chat models. We are releasing the following models to the general public for research and commercial use‡: 1.
一些 Rerank 的 API 服务

4.3 混合检索(Hybrid Search)(选)

实际生产中,传统的关键字检索(稀疏表示)与向量检索(稠密表示)各有优劣。

举个具体例子,比如文档中包含很长的专有名词,关键字检索往往更精准而向量检索容易引入概念混淆。

# 背景说明:在医学中“小细胞肺癌”和“非小细胞肺癌”是两种不同的癌症

query = "非小细胞肺癌的患者"

documents = [
    "玛丽患有肺癌,癌细胞已转移",
    "刘某肺癌I期",
    "张某经诊断为非小细胞肺癌III期",
    "小细胞肺癌是肺癌的一种"
]

query_vec = get_embeddings([query])[0]
doc_vecs = get_embeddings(documents)

print("Cosine distance:")
for vec in doc_vecs:
    print(cos_sim(query_vec, vec))
Cosine distance: 
0.8915956814209096 0.8902380296876382 0.9043403228477503 0.9136486327152477

所以,有时候我们需要结合不同的检索算法,来达到比单一检索算法更优的效果。这就是混合检索

混合检索的核心是,综合文档 dd 在不同检索算法下的排序名次(rank),为其生成最终排序。

一个最常用的算法叫 Reciprocal Rank Fusion(RRF)

rrf(d)=aA1k+ranka(d)rrf(d)=\sum_{a\in A}\frac{1}{k+rank_a(d)}

其中 AA 表示所有使用的检索算法的集合,ranka(d)rank_a(d) 表示使用算法 aa 检索时,文档 dd 的排序,kk 是个常数。

很多向量数据库都支持混合检索,比如 WeaviatePinecone 等。也可以根据上述原理自己实现。 (具体此处就不展开了,感兴趣的可以自行搜索材料)

五、PDF 文档中的表格怎么处理

image.png

  1. 将每页 PDF 转成图片
  2. 识别文档(图片)中的表格
  3. 基于 GPT-4 Vision API 做表格问答
  4. 用 GPT-4 Vision 生成表格(图像)描述,并向量化用于检索

一些面向 RAG 的文档解析辅助工具

  • PyMuPDF: PDF 文件处理基础库,带有基于规则的表格与图像抽取(不准)
  • RAGFlow: 一款基于深度文档理解构建的开源 RAG 引擎,支持多种文档格式(火爆)
  • Unstructured.io: 一个开源+SaaS形式的文档解析库,支持多种文档格式
  • LlamaParse:付费 API 服务,由 LlamaIndex 官方提供,解析不保证100%准确,实测偶有文字丢失或错位发生
  • Mathpix:付费 API 服务,效果较好,可解析段落结构、表格、公式等,贵!

在工程上,PDF 解析本身是个复杂且琐碎的工作。以上工具都不完美,建议在自己实际场景测试后选择使用。

六、说说 GraphRAG

image.png

  1. 什么是 GraphRAG:核心思想是将知识预先处理成知识图谱
  2. 优点:适合复杂问题,尤其是以查询为中心的总结,例如:“XXX团队去年有哪些贡献”
  3. 缺点:知识图谱的构建、清洗、维护更新等都有可观的成本
  4. 建议
    • GraphRAG 不是万能良药
    • 领会其核心思想
    • 遇到传统 RAG 无论如何优化都不好解决的问题时,酌情使用

总结

RAG 的流程

  • 离线步骤:
    1. 文档加载
    2. 文档切分
    3. 向量化
    4. 灌入向量数据库
  • 在线步骤:
    1. 获得用户问题
    2. 用户问题向量化
    3. 检索向量数据库
    4. 将检索结果和用户问题填入 Prompt 模版
    5. 用最终获得的 Prompt 调用 LLM
    6. 由 LLM 生成回复

我用了一个开源的 RAG,不好使怎么办?

  1. 检查预处理效果:文档加载是否正确,切割的是否合理
  2. 测试检索效果:问题检索回来的文本片段是否包含答案
  3. 测试大模型能力:给定问题和包含答案文本片段的前提下,大模型能不能正确回答问题