Streaming rendering the returned data

Background

With the repid development of LLM, we want to provide drivers a function in the app, when they arrive a poi place, they can ask our chat bot the related information of this poi, such as recommended food, costs and so on.

As a pre-research function, we develop a sample chat bot in the backoffice website, front-end uses the api that backend provided which returns data after chatGPT returns all in the first version. The issue that most affect user experience is that response time is too lang. We found that chatGPT supports returning streaming data, and in the second version, backend api will return streaming data, and frontend streaming rendering returns data.

The first try - Axios

There is a config param responseType, which indicates the type of data that the server will respond with, options are: 'arraybuffer', 'document', 'json', 'text', 'stream'.

export function chatBot(data) {
  return request({
    responseType: "stream",
    url: `/stream`,
    baseURL,
    method: "post",
    data,
    ...
  });
}

As the above picture shows, axios returns streaming data with combining them together, that is not what we expected.

There is a warning message in the console.

XMLRequest does not support streaming data in the chrome browser.

The solution - Fetch

About fetch, the Request.body and Response.body properties are available, which are getters exposing the body contents as a readable stream.

export const fetchStreamChat = (data, handleStreamRes) => {
  fetch(`${baseURL}/stream`, {
    headers: {
      "Content-Type": "application/json",
    },
    method: "POST",
    body: JSON.stringify(data),
  })
    .then(response => response.body)
    .then(body => {
      readStream(body, handleStreamRes);
    });
};

export async function readStream(stream, handleStreamRes) {
  const reader = stream.getReader();
  let utf8decoder = new TextDecoder();

  let loop = true;
  do {
    const { done, value } = await reader.read();
    loop = !done;
    let u8arr = new Uint8Array(value);
    let msg = utf8decoder.decode(u8arr);
    handleStreamRes(msg);
    if (done) {
      reader.releaseLock();
      stream.cancel();
    }
  } while (loop);
}

handleStreamReswill handle the rendering process.

usage

fetchStreamChat({ user_input: "Hello" }, msg => {
  this.message += msg;
});