Edit

Web scraping 101

All you need to know about extracting structured data from web pages, the protections websites employ to prevent it, and how to bypass them.

Web scraping is the process of extracting structured information from a web page. In essence, web scraping automates the process of manually finding and copy/pasting the information on a website you find useful.

In other words, instead of manually visiting each of the 1000 listings for white T-shirts on an e-commerce site and copy-pasting each listing's price, description and seller information, you can create a bot that does it for you. You can then make your bot return the data in a handy format like JSON, HTML or Excel, so you could process and use it.

What can I use web scraping for?

The primary function of web scraping is the extraction of data.

It is about gathering information, which you can then use to make informed decisions in how to price or market your product, find new customers and make decisions that enable you to grow your business.

To see examples of organizations that have already benefitted from web scraping, check out our success stories.

How does web scraping work?

  1. The scraper requests the contents of a particular page from a website (e.g. this week's Top 10 singles on Spotify). The site returns it in HTML format.
  2. It parses (splits up the data and converts it to the required format) the HTML and extracts the data it's been programmed to extract (e.g. the song title and artist name).
  3. The scraper stores the data in the specified format so you can use it manually or in a program.

Is web scraping the same as robotic process automation (RPA)?

While web scraping is a kind of RPA, it focuses on extracting data. RPA focuses on the other tasks in browsers - everything except for extracting information.

RPA allows you to handle use cases like filling forms or uploading files while you get on with more important tasks. And it's not just simple tasks you can automate. How about processing your invoices or automating your sales processes?

Additional resources

If you would like to learn more, check out this helpful article and this guide, which explains the principles of scraping and the difference between crawlers and scrapers.