search docs
Edit

Introduction

Learn how to develop web scrapers with this comprehensive and practical course. Go from beginner to expert, all in one place.

Welcome to the Web Scraping Academy. A comprehensive and practical web scraping course that will take you from an absolute beginner to an expert web scraper developer. This course is sponsored by Apify, the web scraping and automation platform, but we will use only open-source technologies throughout. This means that the skills you learn will be applicable to any scraping project and you'll be able to run it on any computer. No Apify account needed.

If you would like to learn about the Apify platform and how it can help you build, run and scale your web scraping and automation projects, visit the Apify Academy, where we'll teach you all about our actors, proxies, API, scheduling, webhooks and much more.

Why learn scraper development

With so many point-and-click tools and no-code software that can help you extract data from websites, what is the point of learning scraper development? Contrary to what their marketing departments say, a point-and-click or no-code tool will never be as flexible, as powerful, or as optimized as a custom-built scraper.

Any software can do only what it was programmed to do. If you build your own scraper, it can do anything you want. And you can always quickly change it to do more, less, or the same, but faster or cheaper. The possibilities are endless once you know how scraping really works.

Scraper development is a fun and challenging way to learn web development, web technologies, and understand the internet. You will reverse-engineer websites and understand how they work internally, what technologies they use and how they communicate with their servers. You will also master your chosen programming language and core programming concepts. When you truly understand web scraping, learning other technology like React or Next.js will be a piece of cake.

Course summary

When we set out to create the Web Scraping Academy, we wanted to build a complete guide to modern web scraping. A tutorial that a beginner could use to create their first scraper, as well as a resource that professionals will continuously use to learn about advanced and niche web scraping techniques. All chapters include code examples and code-along exercises that you can use to immediately put your scraping skills into action.

This is what you'll learn in the Web Scraping Academy:

Coming soon:

  • Advanced web scraping
    • Advanced data collection techniques
      • API scraping
      • JavaScript rendering and dynamic pages
      • Data in JavaScript objects
    • Advanced crawling techniques
      • Navigation with POST requests
      • Emulating API pagination
      • GraphQL scraping
    • Anti-scraping protection
      • Proxies and staying anonymous
      • Captchas and human behavior
      • Web application firewalls
  • Expert topics
    • Mobile app scraping
    • Overcoming result limits

We release course content as soon as we write it instead of launching it all at the same time much later. If you want to get notified about new chapters sign up for a free Apify account to get our newsletters.

Requirements

You don't need to be a developer or a software engineer to complete this course, but basic programming knowledge is recommended. Don't be afraid, though. We explain everything in great detail in the Web Scraping for Beginners section and provide external references that can help you level up your web scraping and development skills. If you're new to programming, pay very close attention to the instructions and examples. A seemingly insignificant thing like using [] instead of () can make a lot of difference.

As you progress to the Advanced and Pro sections, the coding will get more challenging, but still manageable to a person with an intermediate level of programming skills.

Next up

The course begins with short theory and continues with practical examples of collecting data from the most popular websites on the internet using your browser console. So let's get to it!

If you already have experience with HTML, CSS, and browser DevTools, feel free to skip to the Basics of crawling chapter, or even the Advanced Web Scraping section of the course.