Now that we’ve covered file types and formatting, let’s dive into text pre-processing. This is the step where we clean and simplify the content in each document to make it easier for your agent to understand and retrieve the right information.

First, it’s essential to remove any irrelevant data. Think about whether each piece of content in your document is useful for answering potential user questions. For example, if you want to answer questions about a product catalog, legal disclaimers that aren’t directly relevant might cause problems. Removing this can significantly reduce noise, making your dataset cleaner and easier to search. It’s also a good idea to clean up any extra metadata, as well as footers or headers that could create distractions during indexing.

Another important part of this process is simplifying the text itself. Jargon, technical language, or overly complex sentences can sometimes introduce ambiguity. If the document is too complex, it may not only slow down processing but also lead to unclear answers. Consider rephrasing dense sections or removing industry-specific terms unless they’re absolutely critical.

If your document contains long paragraphs or complicated sentences, it might even help to use automated simplification tools. These tools can break down dense language into shorter, clearer statements, making it easier for Botpress to chunk and interpret the content accurately.

In short, the goal here is to make the text as straightforward and relevant as possible. By removing unnecessary data and simplifying the language, you’re creating a streamlined, focused dataset that enhances retrieval performance and accuracy.

Remember, a good rule of thumb is to treat your AI agent like a brand new coworker with no context whatsoever about your product, industry, or business.