Sometimes we don't have a public API to call to get the structured data we want to evaluate our KPIs. So, we can scrape the web to do this, but with a cost, in terms of governance and robustness. In this session we see how to solve those issues by setting up the appropriate process and using the right tools to make the crawling process seamless and easy.

Scraping, Selenium, Background job, Headless browser