Semantic Search Engine for Bioinformatics Company

OBJECTIVE:

The customer wanted us to build an intelligent search engine that can help him deal with the internal inventory search. The inventory included a considerable number of blood samples. Each blood sample was described using several tags, grouped into subcategories, which were grouped into larger categories, etc.

Customer’s employees were forced to select many tags by hand to get the information they wanted. It tooks several minutes to perform a single search. And what was more disappointing, if an employee makes a single mistake or provides an inaccurate query, he or she will get an empty result page.

The entire data lookup process was a huge disappointment and headache for the personnel and the customer. There were several challenges to overcome to improve the customer’s workflow.

PROCESS:

From the very beginning, the customer provided a list of keywords that describe blood samples. Very soon our team discovered that this list was incomplete and required additional research and it was not enough to complete the project. Similar issues can’t deny our team from completing the project in time.

This way we decided to split the final solution into two pluggable modules. One for intelligent matching, it determined the level of confidence while tagging a blood sample. Another to extract all possible tags from search queries. De facto the second module transferred unstructured user input into structured data.

The first challenge our engineers overcame was the lack of sample data. We trained the custom model based on hundred thousand life science documents related to blood samples from the open datasources. Data Scientist used Word2vec to analyze the connections between the most common words from the thesaurus to find synonyms and determine how these words are related to each other.

As a result, the model could automatically analyze the description and tag of blood samples with a high confidence level — close to 98%.

The module responsible for entity detection in search queries was partially ready. We had already built a similar module while developing a platform for custom chatbot development. All that was left to do was retrain the model according to the list of entities: sample types, geography, diseases, genders, etc.

To achieve a high level of confidence, we analyzed the massive number of user search queries collected from the open data sources. In the end, we compiled a collection of patterns used to form search queries.

SOLUTION:

The final solution consists of three separate interconnected modules hosted in the cloud. Such an approach helps us to maintain the system remotely to avoid on-site personnel training. Cloud architecture makes the application more flexible, cutting down development and maintenance costs.

We are proud to say, two of the modules are powered by machine learning. Query Analysis module uses natural language processing algorithms to extract entities from search queries, while Search Engine module uses the extracted entities to match these entities with synonyms to perform an accurate and fast search.

Modules are built as independent RESTful microservices, which helps us to scale the final solution to any size in the cloud.

We significantly optimized traditional search algorithms. Instead of searching among the whole dataset we processed about 150.000 samples with about 100 tags and performed the search among these tags. We cached all processed samples with Redis, which helped us to implement in-memory data lookups and avoid the bottlenecks of reading/writing data to hard-drive.

Performed optimizations helped us to provide outstanding search quality and blazing speed.

TECHNOLOGIES:

RESULTS:

We successfully implemented a commercial semantic search engine that can handle massive scientific datasets. We used modern Natural Language Processing technologies to extract entities from search queries and categorize scientific texts by tags. The algorithms we built helped the customer to eliminate ineffective search results and significantly improve employee’s satisfaction with the data lookup process.

Some numbers:

NOW:

We have successfully launched a semantic search engine in the middle of March. Now we are maintaining the application, processing new datasets, increasing search quality and scaling the system in the cloud.