This is part 3 of my Python, Selenium, Fargate posts. Part 1 — Run a Python Selenium web scraper on AWS Fargate Part 2 — Adding Browsermob Proxy to sniff traffic and have more confidence in whether the website you’re trying to scrape has loaded Part 3 — exception handling strategies for when something inevitably crashes (this) The scrape job that... » read more
This is part 2 of my Python, Selenium, Fargate posts. Part 1 — Run a Python Selenium web scraper on AWS Fargate Part 2 — Adding Browsermob Proxy to sniff traffic and have more confidence in whether the website you’re trying to scrape has loaded (this) Part 3 — exception handling strategies for when something inevitably crashes While trying to get... » read more
In building Torres App the hardest part was getting data about campsite availability from the three different companies that manage the campsites. Initially, I was running the scraper every few days from my computer but that clearly wasn’t viable so I decided to move it to AWS. Getting everything working on AWS required many hours... » read more