Let’s say you have a chat room where people discuss a product or service. Often, people ask questions that could be answered by reading the manual, but you don’t mind them asking because you get feedback about the product from the questions people ask.
However, some questions get asked a lot, and you are falling behind on answering them. What can you do to provide a timely response without giving up the opportunity to start a dialogue?
One thing you can do is introduce an LLM-based agent to the chat, which selectively responds to questions that are answerable from information inside the documentation.
There may not be an LLM that was trained on your latest documentation, but this can be solved by including the docs in the LLM’s “context.” If your documentation is too large to fit in the context of a given LLM, then it needs to be split up, and only the relevant parts added to the context.
In order to find the parts that are relevant, some kind of search needs to be performed. Once all of this is brought together, we have a system that uses Retrieval Augmented Generation (RAG) to provide an answer.
LocalAGI has a number of features that help with creating such a chatbot. Firstly, we have a set of connectors, such as Matrix or Slack, which allow the agent to listen for incoming chat messages. Then there is a filter feature that allows us to quickly discard irrelevant messages, and finally, there is a knowledge base lookup that is done via LocalRecall.
The whole process that happens when a message is received can be observed through the agent’s status page.
Going beyond what I have been able to demonstrate on video so far, let’s say that the bot’s answers are not satisfactory, and a user expresses their displeasure with it. The bot could be given an action that allows them to create a ticket requesting an improvement to the documentation.
If a trusted person corrects the bot’s answer, this interaction could be saved by the bot into its knowledge base and used to provide better answers in the future. Alternatively, it could open a request to update the documentation from which part of its knowledge base is derived.
When an agent hallucinates and tries to answer questions for which the documentation was not provided to it, this can be mitigated by getting the agent to re-evaluate its inputs and outputs. Indeed, we can ask a second agent with a different LLM to do this as well.
Sometimes the user’s question may not contain the right information to produce a good search result. With larger models capable of reasoning, we can ask the LLM to either reformulate the user’s request or to ask for clarification.
In any case, LocalAGI provides a relatively easy way to create a self-hosted chatbot that can do RAG. If you want to try it out, then head over to https://github.com/mudler/LocalAGI. For basic question answering (or troll detection for that matter) even a very small model capabable of running on cheap consumer GPUs can be used.