Scrapper Bots

We work on a lot of integration projects.

Most of the time these integrations come with official APIs.

For example, Facebook provides an official API for accessing social data about yourself.

But what if you wanted to pull data about yourself from your school's website?

The school may or may not have a way to do that.

If they don't, your only option it to manually retreive that data.

But what if that data is updated every day? And you need it for every student?

At that point, it's no longer feasible for a human to pull that data.

That's where scrapper bots come in.

We write scrapper bots using Puppeteer which basically is just a wrapper around a headless Chrome browser.

A headless Chrome browser is simply a Chrome Web Browser without a screen.

We program this browser to load the school web page and log in using our credentials.

Once logged in, the bot collects the data by parsing the HTML on the page and sends it back to our database is a formatted manner.