A Follow-Up on AutoWeber: The Mistakes I Made In Design

In my previous post, I talked about a proof of concept on developing a self-adapting web scraper.  As I was adding onto the project, I was having difficulty adding constraints for improving structure accuracy.  After some time, I came to one conclusion:  My Initial Design Was Flawed!

Read more

A Proof of Concept on a Self-Adapting Web Scraper

Last year, I created the IssueHunt-Statistics website project on tracking repository, issues, and funding for open source projects.  Shortly after, however, the website changed and my project breaks down.  I did change the scraping code to bring back functionality, only for it to break down again a little while later.

I now have a problem.  I don't want to always spend time constantly reworking the scraping code to make it functional.  I wonder if I could automate this task?

Read more