Skip to main content

Dataset

Store and export web scraping, crawling or data processing job results. Learn how to access and manage datasets in Apify Console or via API.

Dataset storage enables you to sequentially save and retrieve data. Each Actor run is assigned its own dataset, which is created when the first item is stored to it.

Datasets usually contain results from web scraping, crawling or data processing jobs. The data can be visualized as a table where each object is a row and its attributes are the columns. The data can be exported in JSON, CSV, XML, RSS, Excel or HTML formats.

Named datasets are retained indefinitely.
Unnamed datasets expire after 7 days unless otherwise specified.
Learn more

Dataset storage is append-only - data can only be added and cannot be changed or deleted.

Basic usage​

There are five ways to access your datasets:

Apify Console​

In Apify Console, you can view your datasets in the Storage section under the Datasets tab.

Only named datasets are displayed by default. Select the Include unnamed datasets checkbox to display all of your datasets.

Datasets in app

To view or download a dataset in the above-mentioned formats, click on its Dataset ID. Under the Settings tab, you can update the dataset's name (and, in turn, its retention period) and access rights. Click on the API button to view and test the dataset's API endpoints.

Datasets detail view

JavaScript SDK​

If you are building a JavaScript Actor, you will be using the JavaScript SDK. The dataset is represented by a Dataset class. You can use the class to specify whether your data is stored locally or in the Apify cloud and push data to the datasets of your choice using the pushData() method. You could also use other methods such as getData(), map() and reduce(), see the example.

If you have chosen to store your dataset locally, you can find it in the location below.

{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json

DATASET_ID refers to the dataset's name or ID. The default dataset will be stored in the default directory.

To add data to the default dataset, you can use the example below:

// Import the JavaScript SDK into your project
import { Actor } from 'apify';

await Actor.init();
// ...

// Add one item to the default dataset
await Actor.pushData({ foo: 'bar' });

// Add multiple items to the default dataset
await Actor.pushData([{ foo: 'hotel' }, { foo: 'cafe' }]);

// ...
await Actor.exit();

Make sure to use the await keyword when calling pushData(), otherwise the Actor process might finish before the data are stored.

If you want to use something other than the default dataset, e.g. a dataset that you share between Actors or between Actor runs, you can use the Actor.openDataset() method.

import { Actor } from 'apify';

await Actor.init();
// ...

// Save a named dataset to a variable
const dataset = await Actor.openDataset('some-name');

// Add data to the named dataset
await dataset.pushData({ foo: 'bar' });

// ...
await Actor.exit();

When using the getData() method, you can specify the data you retrieve using the fields option. It should be an array of field names (strings) that will be included in the results. To include all the results, exclude the fields parameter.

import { Actor } from 'apify';

await Actor.init();
// ...

const dataset = await Actor.openDataset();

// Only get the 'hotel' and 'cafe' fields
const hotelAndCafeData = await dataset.getData({
    fields: ['hotel', 'cafe'],
});

// ...
await Actor.exit();

See the JavaScript SDK documentation and the Dataset class's API reference for details on managing datasets with the JavaScript SDK.

Python SDK​

If you are building a Python Actor, you will be using the Python SDK. The dataset is represented by a Dataset class. You can use the class to specify whether your data is stored locally or in the Apify cloud and push data to the datasets of your choice using the push_data() method. You could also use other methods such as get_data(), map() and reduce().

If you have chosen to store your dataset locally, you can find it in the location below.

{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json

DATASET_ID refers to the dataset's name or ID. The default dataset will be stored in the default directory.

To add data to the default dataset, you can use the example below:

from apify import Actor

async def main():
    async with Actor:
        # Add one item to the default dataset
        await Actor.push_data({'foo': 'bar'})

        # Add multiple items to the default dataset
        await Actor.push_data([{'foo': 'hotel'}, {'foo': 'cafe'}])

If you want to use something other than the default dataset, e.g. a dataset that you share between Actors or between Actor runs, you can use the Actor.open_dataset() method.

from apify import Actor

async def main():
    async with Actor:
        # Save a named dataset to a variable
        dataset = await Actor.open_dataset(name='some-name')

        # Add data to the named dataset
        await dataset.push_data({'foo': 'bar'})

When using the get_data() method, you can specify the data you retrieve using the fields option. It should be an array of field names (strings) that will be included in the results. To include all the results, exclude the fields parameter.

from apify import Actor

async def main():
    async with Actor:
        dataset = await Actor.open_dataset()

        # Only get the 'hotel' and 'cafe' fields
        hotel_and_cafe_data = await dataset.get_data(fields=['hotel', 'cafe'])

See the Python SDK documentation and the Dataset class's API reference for details on managing datasets with the Python SDK.

JavaScript API client​

Apify's JavaScript API client (apify-client) allows you to access your datasets from any Node.js application, whether it is running on the Apify platform or elsewhere.

After importing and initiating the client, you can save each dataset to a variable for easier access.

const myDatasetClient = apifyClient.dataset('jane-doe/my-dataset');

You can then use that variable to access the dataset's items and manage it.

When using the .listItems() method, if you mention the same field name in the field and omit parameters, the omit parameter will prevail and the field will not be returned.

See the JavaScript API client documentation for help with setup and more details.

Python API client​

Apify's Python API client (apify-client) allows you to access your datasets from any Python application, whether it is running on the Apify platform or elsewhere.

After importing and initiating the client, you can save each dataset to a variable for easier access.

my_dataset_client = apify_client.dataset('jane-doe/my-dataset')

You can then use that variable to access the dataset's items and manage it.

When using the .list_items() method, if you mention the same field name in the field and omit parameters, the omit parameter will prevail and the field will not be returned.

See the Python API client documentation for help with setup and more details.

Apify API​

The Apify API allows you to access your datasets programmatically using HTTP requests and easily share your crawling results.

If you are accessing your datasets using the username~store-name store ID format, you will need to use your secret API token. You can find the token (and your user ID) on the Integrations page of your Apify account.

When providing your API authentication token, we recommend using the request's Authorization header, rather than the URL. (More info).

To get a list of your datasets, send a GET request to the Get list of datasets endpoint.

https://api.apify.com/v2/datasets

To get information about a dataset such as its creation time and item count, send a GET request to the Get dataset endpoint.

https://api.apify.com/v2/datasets/{DATASET_ID}

To view a dataset's data, send a GET request to the Get dataset items Apify API endpoint.

https://api.apify.com/v2/datasets/{DATASET_ID}/items

You can specify which data are exported by adding a comma-separated list of fields to the fields query parameter. Likewise, you can also omit certain fields using the omit parameter.

If you both specify and omit the same field in a request, theomitparameter will prevail and the field will not be returned.

In addition, you can set the format in which you retrieve the data using the ?format= parameter. The available formats are json, jsonl, csv, html, xlsx, xml and rss. The default value is json.

To retrieve the hotel and cafe fields, you would send your GET request to the URL below.

https://api.apify.com/v2/datasets/{DATASET_ID}/items?format=json&fields=hotel%2Ccafe

Instead of commas, you will need to use the %2C code, which represents , in URL encoding.
Learn more about URL encoding here.

To add data to a dataset, send a POST request, with a JSON object containing the data you want to add as the payload to the Put items endpoint.

https://api.apify.com/v2/datasets/{DATASET_ID}/items

Pushing data to dataset via API is limited to 200 requests per second to prevent our servers from being overloaded.

Example payload:

[
    {
        "foo": "bar"
    },
    {
        "foo": "hotel"
    },
    {
        "foo": "cafe"
    }
]

See the API documentation for a detailed breakdown of each API endpoint.

Hidden fields​

Top-level fields starting with the # character are considered hidden. These fields may be easily omitted when downloading the data from a dataset by providing the skipHidden=1 or clean=1 query parameters. This provides a convenient way to store debug information that should not appear in the final dataset.

Below is an example of a dataset record containing hidden fields with an HTTP response and error.

{
    "url": "https://example.com",
    "title": "Example page",
    "data": {
        "foo": "bar"
    },
    "#error": null,
    "#response": {
        "statusCode": 201
    }
}

Data without hidden fields are called "clean" and can be downloaded from Apify Console using the "Clean items" link or via API using the clean=true or clean=1 URL parameters.

XML format extension​

When you export results to XML or RSS formats, object property names become XML tags, while the corresponding values become the tags' children.

For example, the JavaScript object:

{
    name: "Rashida Jones",
    address: [
        {
            type: "home",
            street: "21st",
            city: "Chicago",
        },
        {
            type: "office",
            street: null,
            city: null,
        }
    ]
}

becomes the following XML snippet:

<name>Rashida Jones</name>
<address>
    <type>home</type>
    <street>21st</street>
    <city>Chicago</city>
</address>
<address>
    <type>office</type>
    <street/>
    <city/>
</address>

If the JavaScript object contains a property named @, its sub-properties are exported as attributes of the parent XML element. If the parent XML element does not have any child elements, its value is taken from a JavaScript object property named #.

For example, the following JavaScript object:

{
    "address": [{
        "@": {
            "type": "home",
        },
        "street": "21st",
        "city": "Chicago",
    },
    {
        "@": {
            "type": "office",
        },
        "#": "unknown",
    }]
}

will be transformed to the following XML snippet:

<address type="home">
    <street>21st</street>
    <city>Chicago</city>
</address>
<address type="office">unknown</address>

This feature is also useful when customizing your RSS feeds generated for various websites.

By default, the whole result is wrapped in an <items/> element, while each page object is contained in an <item/> element. You can change this using the xmlRoot and xmlRow URL parameters when GETting your data.

Sharing​

You can invite other Apify users to view or modify your datasets with the access rights system. See the full list of permissions.

Sharing datasets between runs​

You can access a dataset from any Actor or task run as long as you know its name or ID.

To access a dataset from another run using the JavaScript SDK or the Python SDK, open it using the same method as you would with any other dataset.

import { Actor } from 'apify';

await Actor.init();

const otherDataset = await Actor.openDataset('old-dataset');
// ...

await Actor.exit();

In the JavaScript API client, you can access a dataset using its client. Once you've opened the dataset, read its contents and add new data like you would do with a dataset from your current run.

const otherDatasetClient = apifyClient.dataset('jane-doe/old-dataset');

Likewise, in the Python API client, you can access a dataset using its client.

other_dataset_client = apify_client.dataset('jane-doe/old-dataset')

The same applies for the Apify API - you can use the same endpoints as you would normally do.

See the Storage overview for details on sharing storages between runs.

Limits​

  • Tabulated data storage formats (the ones that display the data in columns), such as HTML, CSV, and EXCEL, have a maximum limit of 3000 columns. All data that do not fit into this limit will not be retrieved.

  • When using the pushData() method, the size of the data is limited by the receiving API. Therefore, pushData() will only allow objects whose JSON representation is smaller than 9MB. When an array is passed, none of the included objects may be larger than 9MB, however the array itself may be of any size.

  • Dataset names can be up to 63 characters long.

Rate limiting​

When pushing data to a dataset via API, the request rate is limited to 200 per second per dataset. This helps protect Apify servers from being overloaded.

All other dataset API endpoints are limited to 30 requests per second per dataset.

See the API documentation for details and to learn what to do if you exceed the rate limit.