Customer Profile Scraping

The first and the most complicated tech-related issue was a set of limitations and restrictions of the websites we wanted to extract data from. For example, the customer wanted the solution to collect information from Yelp, TripAdvisor, Facebook, Airbnb, and other well-known resources with strict privacy policies.

It is a challenging process: if a website detects abnormal behavior or abusive activities, it automatically bans a user account, and from time to time bans the IP address.

This way, if you want to extract information from, for example, Facebook you’ll need some additional resources – a set of accounts and a list of proxies (private proxies or VPN servers) you can rotate to reduce the probability of being banned.

Hopefully, both proxies and accounts you can buy on the aftermarket, and it is not an actual problem. Our team spent plenty amount of time learning how social networks and popular review websites track user behavior and determine bots and crawlers.

The information collected while solving this challenge helped us to build robust and reliable algorithms that look like a regular user for any tracking systems. Sure, the customer paid us extra for doing a research, but the algorithms we developed help the customer cut down maintenance costs in a long-term run.

Another tech-related challenge was data extraction from the HTML. In recent years front-end frameworks like React, Angular, Ember, and Vue became incredibly popular and replaced old-fashioned AJAX calls and jQuery.

It means, that traditional web-scrapers now are less usable than five years ago, as some critical parts of websites are generated using a JavaScript after the main content of a page was already loaded. So, one of the main requirements for a modern web-scraper is JavaScript rendering. And that was an issue.

As JavaScript rendering is a comparably resource-intensive task, it takes much more resources and time to process a single page. Multithreading and multiprocessing are a solution to this issue. The team decided to develop the core of an application with Golang – programming language known for its simultaneous processing capabilities.

The existing open-source Go library – WebLoop helped us to build efficient JS rendering engine, but made an app resource-demanded. The solution could not be hosted on a cheap $5 VPS and required a small but dedicated server with several cores and threads available.

The third challenge was not related to data extraction – it concerned the collected data. Our team quickly understood that data collection and aggregation does not provide the required insights for the customer by itself.

The engine uses basic information about the customer for doing initial research, but if the name and surname are quite common and widespread – it is problematic to find a required John Smith across thousands of John Smiths from different countries across several continents.

This way, intelligent data matching is the main and the most critical issue our team faced. As we were building a prototype, our team discussed the level of data matching complexity with the customer. It would take thousands of dollars to create truly intelligent algorithms, so the customer agreed to match records partly manually, relying on some algorithmic calculations.

We briefly trained a solution to find similarities in usernames, education information, email addresses, natural language, to make an MVP entirely usable.