What do you do when the web site you want data from suddenly shuts down their APIs? You create your own, of course.

I’ll go through how I did it, using cloud and scraping, and show you both code, config and a running application. Combining scraping tools in Python with serverless and suitable tools from Google Cloud Platform, we can bypass the lack of existing APIs.

We’ll spend quite some time on the tricky details and strange errors that took quite some time to master, as the devil’s in the details.

Eventually, we’ll have our own REST API, powered by GCP, providing us the data we want - the way we want them.

We’ll scrape a list of events, and show how visualizing the data can reveal some interesting twists in the data. We’ll see that visualizing the data is a powerful debugging and testing technique.

This talk is about the totally overengineered developer version of what happens when someone says “you can’t do that”. Yes we can.