How Pearler built a suggestion engine for RFPs

Why build a Suggestion Engine?

Answering RFPs and security questionnaires is a time-consuming and tedious process. It requires companies to dedicate a lot of resources to ensure that all the questions are answered accurately and in accordance with the specific requirements of each customer. It is important to take the time to answer all the questions in detail, as it can make the difference between a successful bid and a failed one. Furthermore, security questionnaires can be especially tricky and difficult to answer, as they typically require detailed technical knowledge and understanding of the company's security systems and processes.

Before we started Pearler, we experienced the pain of answering complex security questionnaires and knew there had to be a better way. What we really wanted was a system that would read the question and give us some relevant answers we'd previously given, ordered by how relevant they were. This sounded like a perfect use case for Natural Language Processing (NLP) and Machine Learning (ML).

This realisation that an AI-supported process would make humans more efficient at responding to RFPs is truly a game changer. By leveraging the power of artificial intelligence to automate the tedious and repetitive aspects of the process. The AI-driven automation can reduce the amount of manual effort needed to process, analyze, and respond to RFPs. AI can also identify and highlight key data points, providing insights and guidance to humans to help them make better decisions about responding to RFPs. This automated approach allows human resources to be allocated more efficiently, ultimately streamlining the RFP process and increasing the accuracy and speed of RFP responses.

What didn't we want to make?

It's easy to talk about what we want, but we thought it was equally important to make sure we knew what we didn't want to develop. We wanted to support the human in the loop, to ensure that they are empowered and enhanced through technology, rather than replaced. The answers given in these questionnaires matter and they have real legal and social consequences. Giving correct answer's is very important in maintaining the trust your customers have in your company.

How does it work?

The back-bone of the technology is the Answer Library. Pearler developed a custom library for storing Question and Answer content and relationships. It maps the connection between concepts, and how your company typically answers those questions.

When working through a Questionnaire, Pearler reads the question being asked and provides instant suggestions of previous answers that are relevant to the question being asked.

Answers are able to evolve and grow in detail, becoming higher quality and a more accurate representation of your company. The more frequently answers are used, the stronger the links between the concepts and their usage becomes.

What's the technology behind the magic?

The technology behind the magic is a combination of proprietary technologies, working together with open source technologies. We make heavy use of Natural Language Processing (NLP) and Machine Learning (ML) to understand the Questions, Answers and how they are connected. We use Search technologies to make retrieval fast and relevant. We apply Machine Learning (ML) to score the relevancy of Questions, allowing us to optimize the order they appear to users so they are most relevant.

Critically, the data that flows into the system and how it is prepared and cleansed ensures that the rest of the system can provide. We undertake extensive cleansing and preparation before the data reaches the rest of our system.

What is data cleansing and preparation?

Data cleansing and data preparation are essential steps in making sure that the answers are accurate and up to date. Data cleansing involves eliminating errors, duplicate records, and inconsistencies, while data preparation involves formatting the data in a way that is easier to analyze and interpret. By taking the time to make sure that the data is properly organized and free of errors, companies can ensure that they are submitting accurate and reliable answers to their customers.

What is Machine Learning?

Machine Learning is a powerful tool that enables computers to learn from vast amounts of data and make predictions, decisions, and recommendations without explicit programming. Deep Neural Networks are one of the most powerful and successful applications of Machine Learning, allowing computers to learn complex patterns and relationships in data and make accurate predictions.

Neural network reinforcement models are a type of artificial intelligence that allow machines to learn from their environment and experiences. They work by taking in input from the environment, processing it to learn, and then producing an output based on the input. The model is continuously updated and improved as it receives more input. It is able to adjust its behavior to maximize the reward it receives, which is often a score or a measure of success. This makes it an effective tool for a wide range of tasks, from playing games to controlling robots.