---
title: Web scraping with AI and Apify
url: https://docs.apify.com/academy/scraping-with-ai.md
parents:
  - [Apify documentation](https://docs.apify.com/llms.txt)
  - [Apify Academy](https://docs.apify.com/academy.md)
children:
  - [Developing a scraper with AI chat](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-chat.md)
  - [Developing a scraper with AI agent](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-agent.md)
  - [Using docs as a spec for AI](https://docs.apify.com/academy/scraping-with-ai/docs-driven-prompting.md)
  - [Using examples as a spec for AI](https://docs.apify.com/academy/scraping-with-ai/tests-driven-prompting.md)
  - [Before publishing to Apify Store](https://docs.apify.com/academy/scraping-with-ai/before-publishing-to-apify-store.md)
next: [Developing a scraper with AI chat](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-chat.md)
---

> ## Documentation index
> Fetch the complete documentation index at: https://docs.apify.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web scraping with AI and Apify

**Learn how to use AI to extract information from websites, starting from the absolute basics.**

***

In this practical course, we'll use AI assistants to create an application for watching prices. It'll be able to scrape product pages of an e-commerce website and record prices. Data gathered from several runs of such program would be useful for seeing trends in price changes or detecting discounts.

The end product will, unlike programs vibe-coded carelessly, reach the level of quality allowing for further extensibility and comfortable maintenance, so that it can be published to [Apify Store](https://apify.com/store).

## What we'll do

* Use ChatGPT (AI chat) to create a program which extracts data from a web page.
* Save extracted data in various formats, for example CSV which MS Excel or Google Sheets can open.
* Use Cursor (AI agent) to improve the program so that it is robust and maintainable.
* Save time and effort with the Apify platform.

## Who is this course for

Anyone with basic knowledge of chatting with an AI assistant and affinity to building digital products who wants to start with web scraping can take this course. The course does not expect you to have any prior knowledge of web technologies or scraping.

## Requirements

* Prior experience chatting with AI assistants, such as ChatGPT from OpenAI, Google Gemini, or Claude by Anthropic.
* A macOS, Linux, or Windows machine. If it's your work computer, make sure you have permissions to install new software.
* Familiarity with running commands in terminal (macOS/Linux) or PowerShell (Windows). Just knowing what those are and how to use them is sufficient.

## Course content

## [📄️ Developing a scraper with AI chat](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-chat.md)

[Use ChatGPT and Apify to build a price-tracking web scraper with no coding knowledge. Learn how AI-generated code runs automatically in the cloud.](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-chat.md)

## [📄️ Developing a scraper with AI agent](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-agent.md)

[Improve your Apify scraper using the Cursor AI agent. Set up local development, clean up price data, and push changes back to the Apify platform.](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-agent.md)

## [📄️ Using docs as a spec for AI](https://docs.apify.com/academy/scraping-with-ai/docs-driven-prompting.md)

[Improve your Apify scraper by documenting its behavior first and letting an AI agent follow the documentation as a practical spec.](https://docs.apify.com/academy/scraping-with-ai/docs-driven-prompting.md)

## [📄️ Using examples as a spec for AI](https://docs.apify.com/academy/scraping-with-ai/tests-driven-prompting.md)

[Improve your Apify scraper by adding automated tests with real-world examples that an AI agent can use as a spec.](https://docs.apify.com/academy/scraping-with-ai/tests-driven-prompting.md)

## [📄️ Before publishing to Apify Store](https://docs.apify.com/academy/scraping-with-ai/before-publishing-to-apify-store.md)

[Use AI to prepare your scraper for other users by improving its first-run experience, Apify Store listing, documentation, and maintenance plan.](https://docs.apify.com/academy/scraping-with-ai/before-publishing-to-apify-store.md)