Inventory Search Engine for Auto Parts Retailer

OBJECTIVE:

After applying to the project, we discovered, that there is no single catalog of all auto parts. There were several catalogs from different manufacturers and several catalogs for internal use. The existing solution was performing ineffective request chains from one data source to another. From the very beginning, there was a single catalog, that included an enormous number of autoparts and accessories.

Today other catalogs also store a massive amount of data about auto parts and accessories. The number of data increased dramatically. This way we decided to give up the improvement of the existing solution and design the new system that can handle the amount of data. There were several challenges we faced.

PROCESS:

At Azati we enjoy building modular systems while developing small projects: such systems are more straightforward to build, test, scale and maintain. This project is not an exclusion. Our team decided to create two modules: one for data processing, another for user interface generation.

From the very beginning, as the customer wanted the system to re-generate the result page content every time the user changes the input, we decided to use React as a front-end framework for UI generation. It is a powerful tool that can modify the page source code without reloading the browser tab. The first module is based on React and Python Tornado as a back-end for request handling.

The second module was responsible for entity extraction, parts tagging, and data lookup. There were a lot of different files that contained the information we needed, so our team developed a custom rule-based parser that can identify the document type and provider on the fly.

For every provider, we described a set of rules, that are used by a parser to extract data. The parser took the document as an input and provided the structured data as an output.

There were several data sources, where the information about auto parts was represented as MySQ Land Access databases. For these databases, we built an algorithm that asynchronously sends requests to multiple databases at the same time extracting the data and eliminating any delays.

As the core of a search engine, we used enhanced pairwise comparison algorithm, developed by our PHDs and well-tested on some bioinformatics samples during the recent projects. As a result, we developed a prototype that was presented to the customer. He was quite impressed and asked us to improve the prototype and turn it into a solution.

SOLUTION:

The final solution includes interconnected two modules: one for user interface generation,another for document processing and data lookup. These modules are hosted in a self-made cloud inside the client’s infrastructure. As there is a vast amount of information describing autoparts, it takes about 3GB of RAM to store all the attributes and tags in memory. If a customer loads a bigger dataset, the system will automatically allocate additional memory.

TECHNOLOGIES:

RESULTS:

We successfully built an intelligent search engine that aggregates information from various data sources: from documents to external databases. Our team successfully implemented enhanced search algorithm to provide quick and accurate search results. As all data is stored in memory, we made data lookups blazing fast. The engine also offers the most relevant search results according to advanced scoring.

A FEW NUMBERS:

NOW:

We have successfully launched this project in late November 2017. Now we are maintaining this solution and collecting usage statistics. Statistics help us to understand the users better. We will propose the client to improve the search with machine learning, which will make search even more accurate and fast.