search docs
Edit

Storage

Store anything from images and key-value pairs to structured output data. Learn how to access and manage your stored data from the Apify platform or via API.

The Apify platform includes three types of storage you can use both in your actors and outside the Apify platform via API, the Apify SDK and Apify's JavaScript API client and Python API client.

This page contains a brief introduction of the three types of Apify Storage.

  • Dataset - storage for data objects such as scraping output.
  • Key-value store - storage for arbitrary data records such as files, images, and strings.
  • Request queue - a queue of URLs for your actors to visit.

You will then find basic usage information relating to all three types of storage. For example, how to manage your storages in Apify Console, the basics of setting up the Apify SDK, the JavaScript API client and the Python API client, and general information for using storages with the Apify API.

Dataset

Dataset storage allows you to store a series of data objects such as results from web scraping, crawling or data processing jobs. You can export your datasets in JSON, CSV, XML, RSS, Excel or HTML formats.

Dataset graphic

The easiest way to access your datasets is via Apify Console, which provides a user-friendly interface for viewing or downloading the data and editing your datasets' properties.

To manage your datasets, you can use the Apify SDK, JavaScript API client, Python API client, or the Apify API.

See the dataset documentation for details.

Key-value store

The key-value store is ideal for saving data records such as files, screenshots of web pages, and PDFs or for persisting your actors' state. The records are accessible under a unique name and can be written and read quickly.

Key-value store graphic

The easiest way to access your key-value stores is via Apify Console, which provides a user-friendly interface for viewing or downloading the data and editing your key-value stores' properties.

To manage your key-value stores, you can use the Apify SDK, JavaScript API client, Python API client, or the Apify API.

See the key-value store documentation for details.

Request queue

Request queues allow you to dynamically maintain a queue of URLs of web pages. You can use this when recursively crawling websites: you start from initial URLs and add new links as they are found while skipping duplicates.

Request queue graphic

The easiest way to access your request queues is via Apify Console, which provides a user-friendly interface for viewing your request queues and editing your queues' properties.

To manage your request queues, you can use the Apify SDK, JavaScript API client, Python API client, or the Apify API.

See the request queue documentation for details.

Basic usage

There are five ways to access your storage:

Apify Console

To access your storages from Apify Console, go to the Storage section in the left-side menu. From there, you can click through the tabs to view your key-value stores, datasets, request queues and related API endpoints. To view a storage, click its ID.

Storages in app

Only named storages are displayed by default. Select the Include unnamed store checkbox to display all of your storages.

You can edit your stores' names under the Settings tab of their detail page. There, you can also grant access rights to other Apify users.

You can quickly share your storages' contents and details by sharing the URLs you find under the API tab in a store's detail page.

Storage API

These URLs provide links to API endpoints–the places where your data are stored. Endpoints that allow you to read stored information do not require an authentication token. The calls are authenticated using a hard-to-guess ID, so they can be shared freely. Operations such as update or delete, however, will need the authentication token.

Never share a URL containing your authentication token, as this will compromise your account's security.
If the data you want to share requires a token, first download the data, then share it as a file.

Apify SDK

The Apify SDK is a JavaScript/Node.js library which allows you to build your own web scraping and automation solutions. It requires Node.js 10.17 or later, with the exception of Node.js 11.

See the SDK documentation for setup instructions and to learn how to build your own actors.

JavaScript API client

Apify's JavaScript API client (apify-client) allows you to access your datasets from any Node.js application, whether it is running on the Apify platform or elsewhere.

See the client's documentation for help with setup.

Python API client

Apify's Python API client (apify-client) allows you to access your datasets from any Python application, whether it is running on the Apify platform or elsewhere.

See the client's documentation for help with setup.

Apify API

The Apify API allows you to access your storages programmatically using HTTP requests and easily share your crawling results.

In most cases, when accessing your storages via API, you will need to provide a store ID, which you can do in the following formats:

  • WkzbQMuFYuamGv3YF - the store's alpha-numerical ID if the store is unnamed.
  • username~store-name - your username and the store's name separated by a tilde (~) character (e.g. janedoe~ecommerce-scraping-results) if the store is named.

For read (GET) requests, it is enough to use a store's alpha-numerical ID, since the ID is hard to guess and effectively serves as an authentication key.

With other request types and when using the username~store-name, however, you will need to provide your secret API token in your request's Authorization header or as a query parameter. You can find your token on the Integrations page of your Apify account.

See the API documentation for details and a breakdown of each storage API endpoint.

Rate limiting

All API endpoints limit their rate of requests to protect Apify servers from overloading. The default rate limit is 30 requests per second per storage object, with a few exceptions, which are limited to 200 requests per second per storage object:

If a client sends too many requests, the API endpoints respond with the HTTP status code 429 Too Many Requests and the following body:

{
    "error": {
        "type": "rate-limit-exceeded",
        "message": "You have exceeded the rate limit of ... requests per second"
    }
}

See the API documentation for details and to learn what to do if you exceed the rate limit.

Data retention

Unnamed storages expire after 7 days unless otherwise specified.

Named storages are retained indefinitely.

You can edit your storages' names in Apify Console or using the access methods above.

Named and unnamed storages

All storages are created without a name (with only an ID). This allows them to expire after 7 days and not take up your storage space. If you want to preserve a storage, simply give it a name and it will be retained indefinitely.

Storages' names can be up to 63 characters long.

Named and unnamed storages are the same in all regards except their retention period. The only difference is that named storages make it easier to verify you are using the correct store.

For example, the storage names janedoe~my-storage-1 and janedoe~web-scrape-results are easier to tell apart than the alpha-numerical IDs cAbcYOfuXemTPwnIB and CAbcsuZbp7JHzkw1B.

Sharing

You can invite other Apify users to view or modify your storages using the access rights system. See the full list of permissions.

Sharing storages between runs

Any storage can be accessed from any actor or task run as long as you know its name or ID. You can access and manage storages from other runs using the same methods or endpoints as with storages from your current run.

Datasets and key-value stores can be used concurrently by multiple actors. This means that multiple actors or tasks running at the same time can write data to a single dataset or key-value store. The same applies for reading data–multiple runs can read data from datasets and key-value stores concurrently.

Request queues, on the other hand, only allow multiple runs to add new data. A request queue can only be processed by one actor or task run at any one time.

When multiple runs try to write data to a storage at the same time, it isn't possible to control the order in which the data will be written. It will be written whenever the request is processed.
In key-value stores and request queues, the same applies for deleting records: if a request to delete a record is made shortly before a request to read that same record, the second request will fail.

Deleting storages

Named storages are only removed when you request it. You can delete storages in the following ways.