# Apify Academy | Apify Documentation > One section of the Apify documentation. The main index is at https://docs.apify.com/llms.txt ## Apify Academy - [Apify Academy](https://docs.apify.com/academy.md): Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer. - [Wrap open-source as an Actor](https://docs.apify.com/academy/actorization.md): A guide to converting your applications, scripts, and open-source projects into monetizable, cloud-based tools on the Apify platform. - [Advanced web scraping](https://docs.apify.com/academy/advanced-web-scraping.md): Take your scrapers to a production-ready level by learning various advanced concepts and techniques that will help you build highly scalable and reliable crawlers. - [Crawling sitemaps](https://docs.apify.com/academy/advanced-web-scraping/crawling/crawling-sitemaps.md): Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper. - [Crawling with search](https://docs.apify.com/academy/advanced-web-scraping/crawling/crawling-with-search.md): Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper. - [Sitemaps vs search](https://docs.apify.com/academy/advanced-web-scraping/crawling/sitemaps-vs-search.md): Learn how to extract all of a website's listings even if they limit the number of results pages. - [Tips and tricks for robustness](https://docs.apify.com/academy/advanced-web-scraping/tips-and-tricks-robustness.md): Learn how to make your automated processes more effective. Avoid common pitfalls, future-proof your programs and improve your processes. - [AI agent tutorial](https://docs.apify.com/academy/ai/ai-agents.md): In this section of the Apify Academy, we show you how to build an AI agent with the CrewAI Python framework. You’ll learn how to create an agent for Instagram analysis and integrate it with LLMs and Apify Actors. - [Anti-scraping protections](https://docs.apify.com/academy/anti-scraping.md): Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues. - [Anti-scraping mitigation](https://docs.apify.com/academy/anti-scraping/mitigation.md): After learning about the various different anti-scraping techniques websites use, learn how to mitigate them with a few different techniques. - [Bypassing Cloudflare browser check](https://docs.apify.com/academy/anti-scraping/mitigation/cloudflare-challenge.md.md): Learn how to bypass Cloudflare browser challenge with Crawlee. - [Generating fingerprints](https://docs.apify.com/academy/anti-scraping/mitigation/generating-fingerprints.md): Learn how to use two super handy npm libraries to generate fingerprints and inject them into a Playwright or Puppeteer page. - [Proxies](https://docs.apify.com/academy/anti-scraping/mitigation/proxies.md): Learn all about proxies, how they work, and how they can be leveraged in a scraper to avoid blocking and other anti-scraping tactics. - [Using proxies](https://docs.apify.com/academy/anti-scraping/mitigation/using-proxies.md): Learn how to use and automagically rotate proxies in your scrapers by using Crawlee, and a bit about how to obtain pools of proxies. - [Anti-scraping techniques](https://docs.apify.com/academy/anti-scraping/techniques.md): Understand the various common (and obscure) anti-scraping techniques used by websites to prevent bots from accessing their content. - [Browser challenges](https://docs.apify.com/academy/anti-scraping/techniques/browser-challenges.md): Learn how to navigate browser challenges like Cloudflare's to effectively scrape data from protected websites. - [Captchas](https://docs.apify.com/academy/anti-scraping/techniques/captchas.md): Learn about the reasons a bot might be presented a captcha, the best ways to avoid CAPTCHASs in the first place, and how to programmatically solve them. - [Fingerprinting](https://docs.apify.com/academy/anti-scraping/techniques/fingerprinting.md): Understand browser fingerprinting, an advanced technique used by browsers to track user data and even block bots from accessing them. - [Firewalls](https://docs.apify.com/academy/anti-scraping/techniques/firewalls.md): Understand what a web-application firewall is, how they work, and the various common techniques for avoiding them altogether. - [Geolocation](https://docs.apify.com/academy/anti-scraping/techniques/geolocation.md): Learn about the geolocation techniques to determine where requests are coming from, and a bit about how to avoid being blocked based on geolocation. - [Rate-limiting](https://docs.apify.com/academy/anti-scraping/techniques/rate-limiting.md): Learn about rate-limiting, a common tactic used by websites to avoid a large and non-human rate of requests coming from a single IP address. - [Tutorials on Apify API](https://docs.apify.com/academy/api.md): A collection of various tutorials explaining how to interact with the Apify platform programmatically using its API. - [API scraping](https://docs.apify.com/academy/api-scraping.md): Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements. - [General API scraping](https://docs.apify.com/academy/api-scraping/general-api-scraping.md): Learn the benefits and drawbacks of API scraping, how to locate an API, how to utilize its features, and how to work around common roadblocks. - [Dealing with headers, cookies, and tokens](https://docs.apify.com/academy/api-scraping/general-api-scraping/cookies-headers-tokens.md): Learn about how some APIs require certain cookies, headers, and/or tokens to be present in a request in order for data to be received. - [Handling pagination](https://docs.apify.com/academy/api-scraping/general-api-scraping/handling-pagination.md): Learn about the three most popular API pagination techniques and how to handle each of them when scraping an API with pagination. - [Locating API endpoints](https://docs.apify.com/academy/api-scraping/general-api-scraping/locating-and-learning.md): Learn how to effectively locate a website's API endpoints, and learn how to use them to get the data you want faster and more reliably. - [GraphQL scraping](https://docs.apify.com/academy/api-scraping/graphql-scraping.md): Dig into the topic of scraping APIs which use the latest and greatest API technology - GraphQL. GraphQL APIs are very different from regular REST APIs. - [Custom queries](https://docs.apify.com/academy/api-scraping/graphql-scraping/custom-queries.md): Learn how to write custom GraphQL queries, how to pass input values into GraphQL requests as variables, and how to retrieve and output the data from a scraper. - [Introspection](https://docs.apify.com/academy/api-scraping/graphql-scraping/introspection.md): Understand what introspection is, and how it can help you understand a GraphQL API to take advantage of the features it has to offer before writing any code. - [Modifying variables](https://docs.apify.com/academy/api-scraping/graphql-scraping/modifying-variables.md): Learn how to modify the variables of a JSON format GraphQL query to use the API without needing to write any GraphQL language or create custom queries. - [How to retry failed requests](https://docs.apify.com/academy/api/retry-failed-requests.md): Learn how to resurrect your run but retrying only failed requests - [Run Actor and retrieve data via API](https://docs.apify.com/academy/api/run-actor-and-retrieve-data-via-api.md): Learn how to run an Actor/task via the Apify API, wait for the job to finish, and retrieve its output data. Your key to integrating Actors with your projects. - [Tutorials on Apify Actors](https://docs.apify.com/academy/apify-actors.md): A collection of various Actor tutorials to aid you in your journey to becoming a master Actor developer. - [Adding your RapidAPI project to Apify](https://docs.apify.com/academy/apify-actors/adding-rapidapi-project.md): If you've published an API project on RapidAPI, you can expand your project's visibility by listing it on Apify Store. This gives you access to Apify's developer community and ecosystem. - [Introduction to the Apify platform](https://docs.apify.com/academy/apify-platform.md): Learn all about the Apify platform, all of the tools it offers, and how it can improve your overall development experience. - [Tutorials on ready-made Apify scrapers](https://docs.apify.com/academy/apify-scrapers.md): Discover Apify's ready-made web scraping and automation tools. Compare Web Scraper, Cheerio Scraper and Puppeteer Scraper to decide which is right for you. - [Validate your Actor idea](https://docs.apify.com/academy/build-and-publish/actor-ideas/actor-validation.md): Learn how to validate market demand for your Actor using SEO data, community research, and competitive analysis before you build. - [Find ideas for new Actors](https://docs.apify.com/academy/build-and-publish/actor-ideas/find-actor-ideas.md): Learn what kind of software tools are suitable to be packaged and published as Apify Actors and where you can find ideas and inspiration what to build. - [Why publish Actors on Apify](https://docs.apify.com/academy/build-and-publish/why.md): Discover how publishing Actors transforms your code into a revenue-generating product without traditional SaaS overhead. - [Deploying your code to Apify](https://docs.apify.com/academy/deploying-your-code.md): In this course learn how to take an existing project of yours and deploy it to the Apify platform as an Actor. - [Creating dataset schema](https://docs.apify.com/academy/deploying-your-code/dataset-schema.md): Learn how to generate an appealing Overview table interface to preview your Actor results in real time on the Apify platform. - [Publishing your Actor](https://docs.apify.com/academy/deploying-your-code/deploying.md): Push local code to the platform, or create an Actor and integrate it with a Git repository for automatic rebuilds. - [Creating Actor Dockerfile](https://docs.apify.com/academy/deploying-your-code/docker-file.md): Learn to write a Dockerfile for your project so it can run in a Docker container on the Apify platform. - [How to write Actor input schema](https://docs.apify.com/academy/deploying-your-code/input-schema.md): Learn to define your Actor's input UI on the Apify platform with an INPUT_SCHEMA.json file stored inside the .actor configuration directory. - [Managing Actor inputs and outputs](https://docs.apify.com/academy/deploying-your-code/inputs-outputs.md): Learn to accept input into your Actor, process it, and return output. This concept applies to Actors in any language. - [Expert scraping with Apify](https://docs.apify.com/academy/expert-scraping-with-apify.md): After learning the basics of Actors and Apify, learn to develop pro-level scrapers on the Apify platform with this advanced course. - [Webhooks & advanced Actor overview](https://docs.apify.com/academy/expert-scraping-with-apify/actors-webhooks.md): Learn more advanced details about Actors, how they work, and the default configurations they can take. Also, learn how to integrate your Actor with webhooks. - [Apify API & client](https://docs.apify.com/academy/expert-scraping-with-apify/apify-api-and-client.md): Gain an in-depth understanding of the two main ways of programmatically interacting with the Apify platform - through the API, and through a client. - [Bypassing anti-scraping methods](https://docs.apify.com/academy/expert-scraping-with-apify/bypassing-anti-scraping.md): Learn about bypassing anti-scraping methods using proxies and proxy/session rotation together with Crawlee and the Apify SDK. - [Managing source code](https://docs.apify.com/academy/expert-scraping-with-apify/managing-source-code.md): Learn how to manage your Actor's source code more efficiently by integrating it with a GitHub repository. This is standard on the Apify platform. - [Migrations & maintaining state](https://docs.apify.com/academy/expert-scraping-with-apify/migrations-maintaining-state.md): Learn about what Actor migrations are and how to handle them properly so that the state is not lost and runs can safely be resurrected. - [Saving useful run statistics](https://docs.apify.com/academy/expert-scraping-with-apify/saving-useful-stats.md): Understand how to save statistics about an Actor's run, what types of statistics you can save, and why you might want to save them for a large-scale scraper. - [Tasks & storage](https://docs.apify.com/academy/expert-scraping-with-apify/tasks-and-storage.md): Understand how to save the configurations for Actors with Actor tasks. Also, learn about storage and the different types Apify offers. - [Getting started](https://docs.apify.com/academy/getting-started.md): Get started with the Apify platform by creating an account and learning about Apify Console, which is where all Apify Actors are born! - [Actors](https://docs.apify.com/academy/getting-started/actors.md): What is an Actor? How do we create them? Learn the basics of what Actors are, how they work, and try out an Actor yourself right on the Apify platform! - [The Apify API](https://docs.apify.com/academy/getting-started/apify-api.md): Learn how to use the Apify API to programmatically call your Actors, retrieve data stored on the platform, view Actor logs, and more! - [Apify client](https://docs.apify.com/academy/getting-started/apify-client.md): Interact with the Apify API in your code by using the apify-client package, which is available for both JavaScript and Python. - [Creating Actors](https://docs.apify.com/academy/getting-started/creating-actors.md): Build and run your very first Actor directly in Apify Console from a template. This lesson provides hands-on experience with building and running Actors. - [Inputs & outputs](https://docs.apify.com/academy/getting-started/inputs-outputs.md): Create an Actor from scratch which takes an input, processes that input, and then outputs a result that can be used elsewhere. - [Tutorials on scraping with Node.js](https://docs.apify.com/academy/node-js.md): A collection of various Node.js tutorials on scraping sitemaps, optimizing your scrapers, using popular Node.js web scraping libraries, and more. - [Puppeteer and Playwright course](https://docs.apify.com/academy/puppeteer-playwright.md): Learn in-depth how to use two of the most popular Node.js libraries for controlling a headless browser - Puppeteer and Playwright. - [I - Launching a browser](https://docs.apify.com/academy/puppeteer-playwright/browser.md): Understand what the Browser object is in Puppeteer/Playwright, how to create one, and a bit about how to interact with one. - [VI - Creating multiple browser contexts](https://docs.apify.com/academy/puppeteer-playwright/browser-contexts.md): Learn what a browser context is, how to create one, how to emulate devices, and how to use browser contexts to automate multiple sessions at one time. - [Common use cases](https://docs.apify.com/academy/puppeteer-playwright/common-use-cases.md): Learn about some of the most common use cases of Playwright and Puppeteer, and how to handle these use cases when you run into them. - [Downloading files](https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/downloading-files.md): Learn how to automatically download and save files to the disk using two of the most popular web automation libraries, Puppeteer and Playwright. - [Logging into a website](https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/logging-into-a-website.md): Understand the "login flow" - logging into a website, then maintaining a logged in status within different browser contexts for an efficient automation process. - [Paginating through results](https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/paginating-through-results.md): Learn how to paginate through results on websites that use either page number-based pagination or dynamic lazy-loading pagination. - [Scraping iFrames](https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/scraping-iframes.md): Extracting data from iFrames can be frustrating. In this tutorial, we will learn how to scrape information from iFrames using Puppeteer or Playwright. - [Submitting a form with a file attachment](https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/submitting-a-form-with-a-file-attachment.md): Understand how to download a file, attach it to a form using a headless browser in Playwright or Puppeteer, then submit the form. - [III - Executing scripts](https://docs.apify.com/academy/puppeteer-playwright/executing-scripts.md): Understand the two different contexts which your code can be run in, and how to run custom scripts in the context of the browser. - [Extracting data](https://docs.apify.com/academy/puppeteer-playwright/executing-scripts/collecting-data.md): Learn how to extract data from a page with evaluate functions, then how to parse it by using a second library called Cheerio. - [Injecting code](https://docs.apify.com/academy/puppeteer-playwright/executing-scripts/injecting-code.md): Learn how to inject scripts prior to a page's load (pre-injecting), as well as how to expose functions to be run at a later time on the page. - [II - Opening & controlling a page](https://docs.apify.com/academy/puppeteer-playwright/page.md): Learn how to create and open a Page with a Browser, and how to use it to visit and programmatically interact with a website. - [Interacting with a page](https://docs.apify.com/academy/puppeteer-playwright/page/interacting-with-a-page.md): Learn how to programmatically do actions on a page such as clicking, typing, and pressing keys. Also, discover a common roadblock that comes up when automating. - [Page methods](https://docs.apify.com/academy/puppeteer-playwright/page/page-methods.md): Understand that the Page object has many different methods to offer, and learn how to use two of them to capture a page's title and take a screenshot. - [Waiting for elements and events](https://docs.apify.com/academy/puppeteer-playwright/page/waiting.md): Learn the importance of waiting for content and events before running interaction or extraction code, as well as the best practices for doing so. - [V - Using proxies](https://docs.apify.com/academy/puppeteer-playwright/proxies.md): Understand how to use proxies in your Puppeteer and Playwright requests, as well as a couple of the most common use cases for proxies. - [IV - Reading & intercepting requests](https://docs.apify.com/academy/puppeteer-playwright/reading-intercepting-requests.md): You can use DevTools, but did you know that you can do all the same stuff (plus more) programmatically? Read and intercept requests in Puppeteer/Playwright. - [Tutorials on scraping with Python](https://docs.apify.com/academy/python.md): A collection of various Python tutorials to aid you in your journey to becoming a master web scraping and automation developer. - [Run a web server on the Apify platform](https://docs.apify.com/academy/running-a-web-server.md): A web server running in an Actor can act as a communication channel with the outside world. Learn how to set one up with Node.js. - [Web scraping basics for JavaScript devs](https://docs.apify.com/academy/scraping-basics-javascript.md): Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics. - [Crawling websites with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/crawling.md): Lesson about building a Node.js application for watching prices. Using the Fetch API to follow links to individual product pages. - [Extracting data from a web page with browser DevTools](https://docs.apify.com/academy/scraping-basics-javascript/devtools-extracting-data.md): Lesson about using the browser tools for developers to manually extract product data from an e-commerce website. - [Inspecting web pages with browser DevTools](https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting.md): Lesson about using the browser tools for developers to inspect and manipulate the structure of a website. - [Locating HTML elements on a web page with browser DevTools](https://docs.apify.com/academy/scraping-basics-javascript/devtools-locating-elements.md): Lesson about using the browser tools for developers to manually find products on an e-commerce website. - [Downloading HTML with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/downloading-html.md): Lesson about building a Node.js application for watching prices. Using the Fetch API to download HTML code of a product listing page. - [Extracting data from HTML with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/extracting-data.md): Lesson about building a Node.js application for watching prices. Using string manipulation to extract and clean data scraped from the product listing page. - [Using a scraping framework with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/framework.md): Lesson about building a Node.js application for watching prices. Using the Crawlee framework to simplify creating a scraper. - [Getting links from HTML with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/getting-links.md): Lesson about building a Node.js application for watching prices. Using the Cheerio library to locate links to individual product pages. - [Locating HTML elements with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/locating-elements.md): Lesson about building a Node.js application for watching prices. Using the Cheerio library to locate products on the product listing page. - [Parsing HTML with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/parsing-html.md): Lesson about building a Node.js application for watching prices. Using the Cheerio library to parse HTML code of a product listing page. - [Using a scraping platform with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/platform.md): Lesson about building a Node.js application for watching prices. Using the Apify platform to deploy a scraper. - [Saving data with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/saving-data.md): Lesson about building a Node.js application for watching prices. Using the json2csv library to save data scraped from product listing pages in both JSON and CSV. - [Scraping product variants with Node.js](https://docs.apify.com/academy/scraping-basics-javascript/scraping-variants.md): Lesson about building a Node.js application for watching prices. Using browser DevTools to figure out how to extract product variants and exporting them as separate items. - [Web scraping basics for Python devs](https://docs.apify.com/academy/scraping-basics-python.md): Learn how to use Python to extract information from websites in this practical course, starting from the absolute basics. - [Crawling websites with Python](https://docs.apify.com/academy/scraping-basics-python/crawling.md): Lesson about building a Python application for watching prices. Using the HTTPX library to follow links to individual product pages. - [Extracting data from a web page with browser DevTools](https://docs.apify.com/academy/scraping-basics-python/devtools-extracting-data.md): Lesson about using the browser tools for developers to manually extract product data from an e-commerce website. - [Inspecting web pages with browser DevTools](https://docs.apify.com/academy/scraping-basics-python/devtools-inspecting.md): Lesson about using the browser tools for developers to inspect and manipulate the structure of a website. - [Locating HTML elements on a web page with browser DevTools](https://docs.apify.com/academy/scraping-basics-python/devtools-locating-elements.md): Lesson about using the browser tools for developers to manually find products on an e-commerce website. - [Downloading HTML with Python](https://docs.apify.com/academy/scraping-basics-python/downloading-html.md): Lesson about building a Python application for watching prices. Using the HTTPX library to download HTML code of a product listing page. - [Extracting data from HTML with Python](https://docs.apify.com/academy/scraping-basics-python/extracting-data.md): Lesson about building a Python application for watching prices. Using string manipulation to extract and clean data scraped from the product listing page. - [Using a scraping framework with Python](https://docs.apify.com/academy/scraping-basics-python/framework.md): Lesson about building a Python application for watching prices. Using the Crawlee framework to simplify creating a scraper. - [Getting links from HTML with Python](https://docs.apify.com/academy/scraping-basics-python/getting-links.md): Lesson about building a Python application for watching prices. Using the Beautiful Soup library to locate links to individual product pages. - [Locating HTML elements with Python](https://docs.apify.com/academy/scraping-basics-python/locating-elements.md): Lesson about building a Python application for watching prices. Using the Beautiful Soup library to locate products on the product listing page. - [Parsing HTML with Python](https://docs.apify.com/academy/scraping-basics-python/parsing-html.md): Lesson about building a Python application for watching prices. Using the Beautiful Soup library to parse HTML code of a product listing page. - [Using a scraping platform with Python](https://docs.apify.com/academy/scraping-basics-python/platform.md): Lesson about building a Python application for watching prices. Using the Apify platform to deploy a scraper. - [Saving data with Python](https://docs.apify.com/academy/scraping-basics-python/saving-data.md): Lesson about building a Python application for watching prices. Using standard library to save data scraped from product listing pages in popular formats such as CSV or JSON. - [Scraping product variants with Python](https://docs.apify.com/academy/scraping-basics-python/scraping-variants.md): Lesson about building a Python application for watching prices. Using browser DevTools to figure out how to extract product variants and exporting them as separate items. - [Web scraping with AI and Apify](https://docs.apify.com/academy/scraping-with-ai.md): Learn how to use AI to extract information from websites in this practical course, starting from the absolute basics. - [Before publishing to Apify Store](https://docs.apify.com/academy/scraping-with-ai/before-publishing-to-apify-store.md): Use AI to prepare your scraper for other users by improving its first-run experience, Apify Store listing, documentation, and maintenance plan. - [Developing a scraper with AI agent](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-agent.md): Improve your Apify scraper using the Cursor AI agent. Set up local development, clean up price data, and push changes back to the Apify platform. - [Developing a scraper with AI chat](https://docs.apify.com/academy/scraping-with-ai/developing-scraper-with-ai-chat.md): Use ChatGPT and Apify to build a price-tracking web scraper with no coding knowledge. Learn how AI-generated code runs automatically in the cloud. - [Using docs as a spec for AI](https://docs.apify.com/academy/scraping-with-ai/docs-driven-prompting.md): Improve your Apify scraper by documenting its behavior first and letting an AI agent follow the documentation as a practical spec. - [Using examples as a spec for AI](https://docs.apify.com/academy/scraping-with-ai/tests-driven-prompting.md): Improve your Apify scraper by adding automated tests with real-world examples that an AI agent can use as a spec.