Skip to main content

Key-value store

Store anything from Actor or task run results, JSON documents, or images. Learn how to access and manage key-value stores from Apify Console or via API.

The key-value store is simple storage that can be used for storing any kind of data. It can be JSON or HTML documents, zip files, images, or simply strings. The data are stored along with their MIME content type.

Each Actor run is assigned its own key-value store when it is created. The store contains the Actor's input, and, if necessary, other data such as its output.

Key-value stores are mutable–you can both add entries and delete them.

Named key-value stores are retained indefinitely.
Unnamed key-value stores expire after 7 days unless otherwise specified.
Learn more

Basic usage​

There are five ways to access your key-value stores:

Apify Console​

In Apify Console, you can view your key-value stores in the Storage section under the Key-value stores tab.

Only named key-value stores are displayed by default. Select the Include unnamed key-value stores checkbox to display all of your stores.

Key-value stores in app

To view a key-value store's content, click on its Store ID. Under the Settings tab, you can update the store's name (and, in turn, its retention period) and access rights. Click on the API button to view and test a store's API endpoints.

Key-value stores detail

JavaScript SDK​

If you are building a JavaScript Actor, you will be using the JavaScript SDK. The key-value store is represented by a KeyValueStore class. You can use the class to specify whether your data is stored locally or in the Apify cloud, and get and set values using the getValue() and setValue() methods respectively, or iterate over your key-value store keys using the forEachKey() method.

Each Actor run is associated with the default key-value store, which is created for the Actor run. When running your Actors and storing data locally, you can pass its input using the INPUT.json file in the default key-value store directory.

You can find INPUT.json and other key-value store files in the location below.

{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}

The default key-value store's ID is default. The {KEY} is the record's key and {EXT} corresponds to the record value's MIME content type.

To manage your key-value stores, you can use the following methods. See the KeyValueStore class's API reference for the full list.

import { Actor } from 'apify';

await Actor.init();
// ...

// Get the default input
const input = await Actor.getInput();

// Open a named key-value store
const exampleStore = await Actor.openKeyValueStore('my-store');

// Read a record in the exampleStore storage
const value = await exampleStore.getValue('some-key');

// Write a record to exampleStore
await exampleStore.setValue('some-key', { foo: 'bar' });

// Delete a record from exampleStore
await exampleStore.setValue('some-key', null);

// ...
await Actor.exit();

Note that JSON is automatically parsed to a JavaScript object, text data returned as a string and other data is returned as binary buffer.

import { Actor } from 'apify';

await Actor.init();
// ...

// Get input of your Actor
const input = await Actor.getInput();
const value = await Actor.getValue('my-key');

// ...
await Actor.setValue(
    'OUTPUT',
    imageBuffer,
    { contentType: 'image/jpeg' },
);

// ...
await Actor.exit();

The Actor.getInput() method is not only a shortcut to Actor.getValue('INPUT'); it is also compatible with Actor.metamorph() [docs]. This is because a metamorphed Actor run's input is stored in the INPUT-METAMORPH-1 key instead of INPUT, which hosts the original input.

See the JavaScript SDK documentation and the KeyValueStore class's API reference for details on managing your key-value stores with the JavaScript SDK.

Python SDK​

If you are building a Python Actor, you will be using the Python SDK. The key-value store is represented by a KeyValueStore class. You can use the KeyValueStore class to specify whether your data is stored locally or in the Apify cloud, and get and set values using the get_value() and set_value() methods respectively.

Each Actor run is associated with the default key-value store, which is created for the Actor run. When running your Actors and storing data locally, you can pass its input using the INPUT.json file in the default key-value store directory.

You can find INPUT.json and other key-value store files in the location below.

{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}

The default key-value store's ID is default. The {KEY} is the record's key and {EXT} corresponds to the record value's MIME content type.

To manage your key-value stores, you can use the following methods. See the KeyValueStore class documentation for the full list.

from apify import Actor
from apify.storages import KeyValueStore

async def main():
    async with Actor:
        # Open a named key-value store
        example_store: KeyValueStore = await Actor.open_key_value_store(name='my-store')

        # Read a record in the example_store storage
        value = await example_store.get_value('some-key')

        # Write a record to example_store
        await example_store.set_value('some-key', {'foo': 'bar'})

        # Delete a record from example_store
        await example_store.set_value('some-key', None)

Note that JSON is automatically parsed to a Python dictionary, text data returned as a string and other data is returned as binary buffer.

from apify import Actor

async def main():
    async with Actor:
        value = await Actor.get_value('my-key')
        # ...
        image_buffer = ...  # Get image data
        await Actor.set_value(key='OUTPUT', value=image_buffer, content_type='image/jpeg')

The Actor.get_input() method is not only a shortcut to Actor.get_value('INPUT'); it is also compatible with Actor.metamorph() [docs]. This is because a metamorphed Actor run's input is stored in the INPUT-METAMORPH-1 key instead of INPUT, which hosts the original input.

See the Python SDK documentation and the KeyValueStore class's API reference for details on managing your key-value stores with the Python SDK.

JavaScript API client​

Apify's JavaScript API client (apify-client) allows you to access your key-value stores from any Node.js application, whether it is running on the Apify platform or elsewhere.

After importing and initiating the client, you can save each key-value store to a variable for easier access.

const myKeyValStoreClient = apifyClient.keyValueStore('jane-doe/my-key-val-store');

You can then use that variable to access the key-value store's items and manage it.

See the JavaScript API client documentation for help with setup and more details.

Python API client​

Apify's Python API client (apify-client) allows you to access your key-value stores from any Python application, whether it is running on the Apify platform or elsewhere.

After importing and initiating the client, you can save each key-value store to a variable for easier access.

my_key_val_store_client = apify_client.key_value_store('jane-doe/my-key-val-store')

You can then use that variable to access the key-value store's items and manage it.

See the Python API client documentation for help with setup and more details.

Apify API​

The Apify API allows you to access your key-value stores programmatically using HTTP requests and easily share your crawling results.

If you are accessing your datasets using the username~store-name store ID format, you will need to use your secret API token. You can find the token (and your user ID) on the Integrations page of your Apify account.

When providing your API authentication token, we recommend using the request's Authorization header, rather than the URL. (More info).

To get a list of your key-value stores, send a GET request to the Get list of key-value stores endpoint.

https://api.apify.com/v2/key-value-stores

To get information about a key-value store such as its creation time and item count, send a GET request to the Get store endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}

To get a record (its value) from a key-value store, send a GET request to the Get record endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}

To add a record with a specific key in a key-value store, send a PUT request to the Put record endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}

Example payload:

{
    "foo": "bar",
    "fos": "baz"
}

To delete a record, send a DELETE request specifying the key from a key-value store to the Delete record endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}

See the API documentation for a detailed breakdown of each API endpoint.

Compression​

In the past, every record uploaded using the Put record endpoint was compressed using Gzip before uploading. This has changed. Now, records are stored in the state you upload them. This means it is up to you if the record is stored compressed or uncompressed.

You can compress a record and use the Content-Encoding request header to let our platform know which compression it uses. We recommend compressing large key-value records to save storage space and network traffic.

If you use the JavaScript SDK or our JavaScript API client, your files are compressed automatically by default. We recommend using the JavaScript API client, which compresses your data before they are sent to our servers and decompresses them when you retrieve them. This makes your storage costs as low as possible.

Sharing​

You can invite other Apify users to view or modify your key-value stores with the access rights system. See the full list of permissions.

Sharing key-value stores between runs​

You can access a key-value store from any Actor or task run as long as you know its name or ID.

To access a key-value store from another run using the JavaScript SDK or the Python SDK, open it using the same method as you would do with any other store.

import { Actor } from 'apify';

await Actor.init();

const otherStore = await Actor.openKeyValueStore('old-store');
// ...

await Actor.exit();

In the JavaScript API client, you can access a store using its client. Once you've opened a store, read and manage its contents like you would do with a key-value store from your current run.

const otherStoreClient = apifyClient.keyValueStore('jane-doe/old-store');

Likewise, in the Python API client, you can access a store using its client.

other_store_client = apify_client.key_value_store('jane-doe/old-store')

The same applies for the Apify API - you can use the same endpoints as you would normally do.

See the Storage overview for details on sharing storages between runs.

Data consistency​

Key-value storage uses the AWS S3 service. According to the S3 documentation, it provides strong read-after-write consistency.

Limits​

  • Key-value store names can be up to 63 characters long.