langchain之Retrievers

Retrievers

retrievers是一个可以根据非结构化提问，返回相关性文档的接口。所有的vector store都可以转换为retrievers。

// 使用asRetriever，将vector store转换为retriever
// 加载vector store
const vectorStore = await FaissStore.load(dir, embedding);
// 
const vectorStoreRetriever = vectorStore.asRetriever();

const chain = RunnableSequence.from([
    {
      context: vectorStoreRetriever,
      question: new RunnablePassthrough(),
    },
    prompt,
    model,
    new StringOutputParser(),
]);
 const answer = await chain.invoke(
    'Which country does the article mainly talk about'
  );

MultiQueryRetriever

MultiQueryRetriever通过使用模型，针对提问的问题，从多个角度生成多个query，以微调prompt的生成过程，和丰富查询结果，可以优化因为提问的措辞或者embdding过程引起的检索结果不佳的问题。

const retriever = MultiQueryRetriever.fromLLM({
    llm: model,
    retriever: vectorStore.asRetriever(),
    // 生成3次查询
    queryCount: 3,
    // debug log
    verbose: true,
  });
const res = await retriever.invoke(
    'Which country does the article talk about'
);
console.log(res);
// prompt 告诉模型，需要生成三个版本的question
You are an AI language model assistant. Your task is
to generate 3 different versions of the given user
question to retrieve relevant documents from a vector database.
By generating multiple perspectives on the user question,
your goal is to help the user overcome some of the limitations
of distance-based similarity search.
Provide these alternative questions separated by newlines between XML tags. For example:
<questions>
Question 1
Question 2
Question 3
</questions>
Original question: Which country does the article talk about
// 问题生成结果
Generated queries: 
Which country is being referred to in the article?
What are the geographical details of the location mentioned in the article?
Can you identify the nation or territory discussed in this article?

Contextual compression

有时查询结果会包含大量的文档内容，和提问最相关的内容会淹没在大量内容中，Contentual compression用于解决这个问题，解决思路是在返回查询结果前，对结果进行压缩后再返回。

// LLMChainExtractor
const compressor = LLMChainExtractor.fromLLM(model);
const vectorStore = await FaissStore.load(dir, embedding);

const retriever = new ContextualCompressionRetriever({
    baseCompressor: compressor,
    baseRetriever: vectorStore.asRetriever(),
    verbose: true,
});