# Development

In this section, you can find documentation on creating your own actors using the Apify platform.

**If you're interested in building paid actors, contact [community@apify.com](mailto:community@apify.com) to receive the guide.**

## Before you begin

Here are some resources that will help you go from beginner to pro. If you have any questions, join other devs on our [Discord server](https://discord.gg/jyEM2PRvMU) or contact us at [community@apify.com](mailto:community@apify.com).

- Get familiar with [the Apify platform](https://www.youtube.com/watch?v=XPF0kbyvoOs) and [actors](/actors).
- [Check out our public wiki](https://gitlab.com/apify-public/wiki/-/wikis/home). It contains key information, best practices, and advice for building actors.
- [How to and how not to write an actor](https://gitlab.com/apify-public/wiki/-/wikis/writing-actors/how-to-write-and-not-write-an-actor).
- [Actor performance and optimization](https://gitlab.com/apify-public/wiki/-/wikis/writing-actors/actor-performance-and-optimization).
- [Help articles](https://help.apify.com/en/collections/1669782-publishing-to-apify-store).

## Essential materials

- [Actor input](https://gitlab.com/apify-public/wiki/-/wikis/public-actors/input).
- README:
  - [Quick guide](https://gitlab.com/apify-public/wiki/-/wikis/public-actors/readme).
  - [How to make your README great](https://help.apify.com/en/articles/2912548-how-to-write-great-readme-for-your-actors).
- [How to structure your actor's files](https://gitlab.com/apify-public/wiki/-/wikis/public-actors/structure).
- [Actor building checklist](https://gitlab.com/apify-public/wiki/-/wikis/public-actors/checklist).
- [Naming your actor](https://docs.apify.com/actors/publishing/naming-your-actor).


Read about the technical part of building Apify actors. Learn to define actor inputs, build new versions, persist actor state, and choose base Docker images.

F/PUB0khfmJ1wL40kSDRcOSQnAipufsu/p+US1qmEmM=

Mold our tools any way you want to scrape websites or automate repetitive tasks. Find the solution to your task here or use the Search box above.

Home

l0eW3kGAfe4HGY2VwV0c8AcdQirppTlpQGgmY7BuId4=

Learn about what Apify is and how it can help you. Get familiar with the platform and take you first steps in using actors.

About

qjdvrs5yEfpRiIkvvyFaJv19MhN5/a4WAynImE/8YU0=

Learn how to run any actor in Apify Store or create your own. A step-by-step guide through your first steps on the Apify platform.

Quick start

47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=

Step-by-step tutorial that will help you get started with all Apify Scrapers. Learn the foundations of scraping the web with Apify and creating your own actors.

Getting started

Getting started with Apify Scrapers

m9vaN2PzzDK1jbyIAvs6coFyK2HxrCiT7B4hWYvoA+c=

Learn how to scrape a website using Apify's Web Scraper. Build an actor's page function, extract information from a web page and download your data.

Web Scraper

Learn how to scrape a website using Apify's Cheerio Scraper. Build an actor's page function, extract information from a web page and download your data.

Cheerio Scraper

Learn how to scrape a website using Apify's Puppeteer Scraper. Build an actor's page function, extract information from a web page and download your data.

Puppeteer Scraper

SqP1VXeJsZiTwNnDoTVpY0UzddApydxk5HuUlayJt8s=

Discover Apify's ready-made web scraping and automation tools. Compare Web Scraper, Cheerio Scraper and Puppeteer Scraper to decide which is right for you.

Apify scrapers

BF9v9Dz5RJOYCDjurlkDNQaMeKFhcrwB2fWWrJJxV9A=

Learn how to perform the most common integration workflow - run the job => wait => collect data. Integrate Apify actors with your applications.

Run actor and retrieve data via API

1jBBNsQgM9ieI0VpROYsCf+dtjZDof2P6IksdknWNyc=

Learn how to connect the Apify platform with your projects. You can use our tools in cloud services like Zapier, Integromat, Keboola, and many more.

Integrations

FEnFQxyT8Hhc+cwcuWFt6ASHyHoA3+K0vydOXe0SVnE=

Wait for dynamically loaded content when web scraping. See code examples and a detailed breakdown for setting timeouts and custom wait functions.

Scraping dynamic content

SqcB3xHAzH14EXVXo1cjZcrs2UsqDJgr+QD2e0h7B+E=

Learn how to complete a website's authentication process using headless Chrome and Puppeteer. Automate the filling in of log in details and passwords.

Log into a website using Puppeteer

f4fIz9/gky0kTFsv4iTOiJ4fcs3F3QzuvGm4VyniqV4=

Learn how to transfer cookies from your web browser to your crawlers. Log into websites when web scraping or automating tasks using your existing logins.

Log in by transferring cookies

9KK5VsOmKcktWcUZBuRmN4lehi+6ydglWsC99P+8XmI=

Learn how to make your scrapers more efficient by storing repeated page data. Avoid re-scraping pages and reduce your data extraction costs.

Cache data to improve performance

aWyBC9OUg3A/3fn1w+YJeeCALuIn9D1FaK0zTVrkMBM=

Learn to crawl and scrape data from URLs specified in a spreadsheet with Apify scrapers. Scrape a pre-determined list of web pages with Apify actors.

Crawl URLs from a Google Sheet

sDDF81HTtA/15P/KwiW4sp7t5yefDfBRANDCm3R0UTE=

The sitemap.xml file is a jackpot for every web scraper. Take advantage of this and learn a much easier way to extract data from websites using the Apify SDK.

Scrape websites using the sitemap

QWdrzJ/Ua3BU0uvaQwCyNJ90cng7ygNL7+iZQckUlQ8=

Learn to deal with random crashes in your web scraping and automation jobs. Find our the essentials of debugging and fixing problems in your actors.

How to analyze pages and fix errors

frFBj8KEX7A/5eQ3fxmU4ojvz7rrl4uEKBWzhzOt5kE=

Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper.

Scrape paginated sites

DLEfaLml6njiNDEGJ2n5rnx2KB8J1OIUy/5jpmugWC4=

Learn how to access Apify's REST API endpoints from your PHP projects using the guzzle package. Follow a tutorial to run an actor and download its data.

Use Apify from PHP

nNXXf8w/qF4fscJbe4Qv6PEa6afhuNHs/FuejMEbSzE=

Learn how to scrape the web and automate processes with Apify. From beginner guides for using actors to advanced topics like migrations and performance.

Tutorials

PjQw2/z1FQfkMwu2mB1bu3P41cwL4V/6CxHUlVySxBY=

An introduction to the methods you can use to extract data from websites. Analyze web pages for hidden elements to find the most effective approach.

Web scraping techniques

Oz1YnOZlMR+6GKXjdV9hWzq+QBX1Gsw5pakjcS+n8Y8=

We explore web scraping prevention and methods used to bypass blocking, such as IP address rotation and proxies, emulate browser signatures or shared IP address sessions.

Anti-scraping techniques

Anti-scraping techniques and how to bypass them

DxtmKciwc4UptmAFkE/MfDyMw2w0XMRJt4ZVNuySYFs=

All you need to know about extracting structured data from web pages, the protections websites employ to prevent it, and how to bypass them.

Web scraping 101

7KCNhjp7wG8RM9ikb2P7IkyL/iYKPTG5QjNxTtD8d+M=

Learn how to make your automated processes more effective. Avoid common RPA pitfalls, future-proof your programs and improve your processes.

Tips and tricks

lw2DWAUz+EmaoSDCrggi646I+UIcur9CI+0HnxioQUk=

Learn the basics of robotic process automation. Make your processes on the web and other software more efficient by automating repetitive tasks.

Robotic process automation

a2pvUK8d2cR+ktuTqQW58PiFyvg0KDv+tRZnVyoqyAk=

A step-by-step monitoring tutorial that shows how you can receive notifications when an actor or task run fails or does not return enough results.

Actor or task run failure

kY0mYpk1aKy+6NqkLHQ0OGoT2QteiuHbDc3A81r8Sw0=

A step-by-step monitoring tutorial that shows you how to ensure your data is correctly formatted and unique. Visualize your data using the monitoring dashboard.

Check data quality

crdE3RoLhLjXsOxV8+iLHbQVewi6DcoR+ckN569O6E8=

A step-by-step monitoring tutorial that shows you how to monitor and ensure consistency in named datasets which aggregate data from multiple actors or tasks.

Monitor shared datasets

hpf5dg5UuqVzVd0DkwF+q/NtmHp5x8XXUWYODyEw+YE=

A step-by-step monitoring tutorial that shows you how to monitor multiple runs, validate your results and visualize them using the monitoring dashboard.

Monitor multiple tasks

FfQ/zL1nxJLKJpdlRT6ZxNje7ITbH/29yQe6/+/DvmQ=

Learn how to check the performance of your actors, tasks and storages. Validate your data and receive notifications when your jobs succeed or fail.

Monitoring

IQQt5R7e8dmpvL1Pi8N8rqmsVyN8JO1kx94aW3LU5e0=

Configure your actor's input parameters using the Apify console, locally or via API. Access parameters in key-value stores from your actor's code.

Input

m1YNSGC4nuKCjp0ebQsuZ+7XeVEtjXqAFlYuZb+7ctw=

Learn about your actors' memory and processing power requirements, their relationship with Docker resources, and minimum requirements for different use cases.

Memory and CPU

iqwgKfzD6nLPZ8vwdQuqWiRpTSsjjsHX/pLMnTyQSmE=

Apify actors' platform resource usage is calculated in compute units. Find out what they are, how they work, and how you are charged for running actors.

Compute units and consumption

c8+WQjzfW7Qb967AxYXlTWkjz58aaepqvU6jfam91Fc=

Start your actors from the Apify console or via API. Learn about actor lifecycles, how to specify settings and version, provide input and resurrect finished runs.

Running

Running actors

YliNIYlrg8Y2noErhv739U0bAn69v2EVDvkNY2y3r1c=

Create custom versions of an actor for specific use cases. Set a task's name and input parameters.

Configure

cdYiUN2oYPVk/TFsJ/ZIuap2rKUxoyEXyGvNNVjayc4=

Learn how to set up reusable versions of your actors for specific use cases.

Create

S87dynVsMwG52LgS79qb/FKw8BQ1iWbFSHFFi1BXlvo=

Start your actor tasks from the Apify console or via API.

xJ91kPshbfTPvcRvKTYRakas0GSf1kdqG4Zdxavd1Ms=

Set up configurations of your Apify actors for simplified usage. Create multiple configurations of the same actor tailored to various use cases.

Tasks

ZoqUUTs9WRB46qxxnt/cd+iGaYZuHKjAbEXYk1klXog=

Learn about the Docker images you can use as your actor's base. Choose the correct base image for your actor's type and requirements.

Base Docker images

NZhw8cvepIJa/eAfNw+E2qEvB6qAwBzSqQMb0yENnow=

Apify's conventions for actor build numbers. Learn to use a specific version of your actor in a run, understand an actor's lifecycle and manage its cache.

Builds

/1Ux7QBBZ1oML9z7Ro9UQk3itIPCpdyxTP4J77zZM1k=

Learn how to set up automated builds, deploys, and testing for your actors using GitHub Actions or Bitbucket Pipelines.

Continuous integration

9R+Bi/sOMTJzL3Dw4hlen2Gm6dDMpU0ZY3Xs74/40gI=

Learn how to provide your actor with context that determines its behavior.

Environment variables

/luGgaRIrzuakI9jSfgbddVy8U8S2qA1vMd1yKMrA28=

Learn how to define and easily validate a schema for your actor's input with code examples. Provide an autogenerated input UI to your actor's users.

Input schema

Iq5pzjN7DPOGP7VNIydYnVp7gF8j4GFMbDUlEtkQfMs=

Learn all about an Apify actor's code. Compare deployment options, create a custom Dockerfile, set environment variables and metamorph your actor.

Source code

+Qv/7LzVccKnZDsumXA9EajvpbGs6UiUhKFM29lRwns=

Maintain a long-running actor's state to protect from unexpected restarts. See a code example on how to protect your run in case of server shutdown.

State persistence

Dmq7VzTbUiD3EPH9cmwtjFd2Vhuap2lIUPh2fAHuyTU=

Learn how to automate ongoing maintenance and make sure your actors perform over time. See code examples for configuring the Actor Testing actor.

Testing and maintenance

2d6L6g+zvWMvlw2M1WBLrxmubGfNrFKNvPkXEsTUSD8=

Development

EBIG+jUjRVraL9jJpSuGUzu1nhvYXCknuRuKJL+UeRc=

Learn how actor rentals work on the Apify platform. Find out about running costs, what you need to rent an actor, and how to pay for your rentals.

Paid actors

zSYrgfpHuN+oMnDHxIIMzyzvI19fEZSBoV3UhLqPooQ=

Apify's standards for actor naming. Learn how to choose the right name for scraping and non-scraping actors and how to optimize your actor for search engines.

Naming your actor

tiV62lfoJrCpA/0LlHdRAvCj1pPLpEdgB+jA1O1NlNI=

Optimize your actors to get more relevant visits from search engines like Google. Set search engine-friendly parameters and share your actor with the world.

SEO and promotion

uTUjdyiGyJh/XELDDNrL4dEtSMLP+EXmaMmmD94aERk=

Learn how to make your actor available to the public or keep it private. Prepare your actor for Apify Store with a description and README file.

Publishing

Publishing actors

1JQlRQKYYnhcBMng9Ho+KRQQvh1k8QDH7dr+jl9+1sk=

Learn more about Apify's security practices and data protection measures.

Security

lZdw/EJsoH5KsKGy5uMBj9R2pOaHHdtu3dk6/uRbzKw=

Learn the Apify platform's resource capability and limitations such as max memory, disk size and number of actors/tasks per user or schedule.

Limits

KwlltdqAZX42/k1cYoyy7sF8qG9FxE3k/8WgkwhWQF8=

Find out what you can do with the Apify platform. See code examples for a variety of use cases and learn about programming language support.

Examples

ZZ+POVgxK6yBUxsOHYmeENyEHVL658l7t0jpnpJUXvw=

Learn how to develop, run and share serverless cloud programs. Create your own web scraping and automation tools and publish them on the Apify platform.

Actors

lsaL63B53GWxVBG3AMet2WgjMwK5SLumtGuCf4g5dfU=

Learn how to automatically start your actor and task runs and the basics of *cron* expressions. Set up and manage your schedules from the Apify console or via API.

Schedules

fN/zEZoKxleHcGNitJr3ge+8su1zpIu5To2Jx/L3wuI=

Store and export web scraping, crawling or data processing job results. Learn how to access and manage datasets in the Apify console or via API.

Dataset

khgn8OMsjL8iWAJu+ZEUUs7g0jtoF3E1W1TWP6FSgrQ=

Store anything from actor or task run results JSON documents or images. Learn how to access and manage key-value stores from the Apify console or via API.

Key-value store

evFTHVUm4sT8ICEUl/cuzUCTAEwEe+hmQoTNmx2TIzM=

Queue URLs for an actor to visit in its run. Learn how to share your queues between actor runs. Access and manage request queues from the Apify console or via API.

Request queue

Cxh1zmjEtRtVRCADrPqrM3Fg7lvr15Qq9DArDutaCzc=

Store anything from images and key-value pairs to structured output data. Learn how to access and manage your stored data from the Apify platform or via API.

Storage

U58GVkHas9fP4m8ibpZoDATIkWtKd3KIjjRPOu3MhB4=

Learn how to connect your application to Apify Proxy. See the required parameters such as the correct username and password.

Connection settings

4dQGQco4TVEt3Gdl+YbnwZq8Z3m5yONlgfbsKU4XOmA=

Learn how to connect to Apify's datacenter proxies from your application with Node.js (axios and got), Python 2 and 3 and PHP using code examples.

LYdWlV91bkFzSb8aKHzzED9eN/JSqPP3q9lQeN/PJ8c=

Learn how to reduce blocking when web scraping using IP address rotation. See proxy parameters and learn to implement Apify Proxy in an application.

Datacenter proxy

jJuYE776lR+DL1eDyxuNIR7bKhpu3irYhv/OoASgAO8=

Helpful tips for using your application with Apify's residential proxies. Control traffic, deal with interrupted connections and manage expenses.

ghrfzV8J3S5vTp5MXB3yowDjZtXHzQGYthaHAlXbVZo=

Achieve a higher level of anonymity using IP addresses from human users. Access a wider pool of proxies and reduce blocking by websites' anti-scraping measures.

Residential proxy

rkuJZb69nLyVr8DX1Ynflhaigqwibk8XaRWxVu03jNY=

Learn how to connect to Google SERP proxies from your applications with Node.js (axios and got), Python 2 and 3 and PHP using code examples.

JoTAWWzQxkXMtsx9AEuplJXqa3fU/O1lVQrW8cIs1lM=

Learn how to collect search results from Google Search-powered tools. Get search results from localized domains in multiple countries, e.g. the US and Germany.

Google SERP proxy

5V9rG/t/aFPi5Ai+uGwTa1WCQnGk0+7thgoGdbePfSE=

Useful tips for debugging applications that use Apify Proxy. Check the status of your proxies and view information about the client IP address.

Troubleshooting

Y2jNGSIjt5RuuhpVtRfBA6nK18HwiW1J9979IYQ0/LA=

Learn how to anonymously access websites when running web scraping or automation jobs. Improve data throughput and efficiency of bots, and enable access to websites from various geographies.