Key-value store

Copy for LLM

The key-value store is simple storage that can be used for storing any kind of data. It can be JSON or HTML documents, zip files, images, or strings. The data are stored along with their MIME content type.

Each Actor run is assigned its own key-value store when it is created. The store contains the Actor's input, and, if necessary, other data such as its output.

Key-value stores are mutable - you can both add entries and delete them.

Retention period

Named key-value stores are retained indefinitely. Unnamed key-value stores expire after 7 days unless otherwise specified. Learn more

Basic usage

You can access your key-value stores in several ways:

• Apify Console - view and manage your key-value stores in a visual interface.
• Apify API - for accessing your key-value stores programmatically.
• Apify API clients - to access your key-value stores from any Node.js/Python application.
• Apify SDKs - when building your own JavaScript/Python Actor.

Apify Console

In Apify Console, you can view your key-value stores in the Storage section under the Key-value stores tab.

To view a key-value store's content, click on its Store ID. Under the Actions menu, you can rename your store (which affects its retention period) and grant access rights using the Share button. To view and test a store's API endpoints, select API.

At the bottom of the page, you can work with records in your key-value store:

• Upload one or more new records and set their keys.
• View individual records.
• Download individual records, or download all records at once.
• Copy shareable links to records.
• Delete records.

Apify API

The Apify API gives you programmatic access to your key-value stores using HTTP requests.

If you are accessing your key-value stores using the username~store-name store ID format, you will need to use your secret API token. You can find the token (and your user ID) on the API & Integrations tab of Settings page of your Apify account.

Authentication

When providing your API authentication token, we recommend using the request's Authorization header, rather than the URL. For more information, refer to the API integration documentation.

To retrieve a list of your key-value stores, send a GET request to the Get list of key-value stores endpoint.

https://api.apify.com/v2/key-value-stores

To get information about a key-value store such as its creation time and item count, send a GET request to the Get store endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}

To get a record (its value) from a key-value store, send a GET request to the Get record endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}

To add a record with a specific key in a key-value store, send a PUT request to the Store record endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}

Example payload:

{
    "foo": "bar",
    "fos": "baz"
}

To delete a record, send a DELETE request specifying the key from a key-value store to the Delete record endpoint.

https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}

For further details and a breakdown of each storage API endpoint, refer to the API documentation.

Apify API Clients

Apify provides API clients for JavaScript and Python applications.

JavaScript API client

With the Apify JavaScript API client (apify-client), you can access your key-value stores from any Node.js application, whether hosted on the Apify platform or externally.

After importing and initializing the client, you can save each key-value store to a variable for easier access.

const myKeyValStoreClient = apifyClient.keyValueStore(
    'jane-doe/my-key-val-store',
);

You can then use that variable to access the key-value store's items and manage it.

Check out the JavaScript API client documentation for help with setup and more details.

Python API client

With the Apify Python API client (apify-client), you can access your key-value stores from any Python application, whether it's running on the Apify platform or externally.

After importing and initializing the client, you can save each key-value store to a variable for easier access.

my_key_val_store_client = apify_client.key_value_store('jane-doe/my-key-val-store')

You can then use that variable to access the key-value store's items and manage it.

Check out the Python API client documentation for help with setup and more details.

Apify SDKs

Apify provides SDKs for JavaScript and Python Actors.

JavaScript SDK

In JavaScript Actors, manage key-value stores with the JavaScript SDK's KeyValueStore class. It works both locally and on the Apify platform. To read and write records, use the getValue() and setValue() methods; to iterate over keys, use forEachKey().

Every Actor run is linked to a default key-value store, created automatically for that run. When you run your Actor locally, you can supply its input by placing an INPUT.json file in the default key-value store's directory.

You can find INPUT.json and other key-value store files in the location below.

{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}

The default key-value store's ID is default. The {KEY} is the record's key and {EXT} corresponds to the record value's MIME content type.

To manage your key-value stores, you can use the following methods. See the KeyValueStore class's API reference for the full list.

import { Actor } from 'apify';

await Actor.init();
// ...

// Get the default input
const input = await Actor.getInput();

// Open a named key-value store
const exampleStore = await Actor.openKeyValueStore('my-store');

// Read a record in the exampleStore storage
const value = await exampleStore.getValue('some-key');

// Write a record to exampleStore
await exampleStore.setValue('some-key', { foo: 'bar' });

// Delete a record from exampleStore
await exampleStore.setValue('some-key', null);

// ...
await Actor.exit();

Automatic parsing

JSON is automatically parsed to a JavaScript object, text data is returned as a string, and other data is returned as a binary buffer.

import { Actor } from 'apify';

await Actor.init();
// ...

// Get input of your Actor
const input = await Actor.getInput();
const value = await Actor.getValue('my-key');

// ...
await Actor.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });

// ...
await Actor.exit();

The Actor.getInput() method is a shortcut to Actor.getValue('INPUT') that also works with Actor.metamorph(). A metamorphed Actor run's input is stored in the INPUT-METAMORPH-1 key instead of INPUT, which holds the original input.

Check out the JavaScript SDK documentation and the KeyValueStore class's API reference for details on managing your key-value stores with the JavaScript SDK.

Python SDK

In Python Actors, manage key-value stores with the Python SDK's KeyValueStore class. It works both locally and on the Apify platform. To read and write records, use the get_value() and set_value() methods.

Every Actor run is linked to a default key-value store, created automatically for that run. When you run your Actor locally, you can supply its input by placing an INPUT.json file in the default key-value store's directory.

You can find INPUT.json and other key-value store files in the location below.

{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}

The default key-value store's ID is default. The {KEY} is the record's key and {EXT} corresponds to the record value's MIME content type.

To manage your key-value stores, you can use the following methods. See the KeyValueStore class documentation for the full list.

from apify import Actor
from apify.storages import KeyValueStore

async def main():
    async with Actor:
        # Open a named key-value store
        example_store: KeyValueStore = await Actor.open_key_value_store(name='my-store')

        # Read a record in the example_store storage
        value = await example_store.get_value('some-key')

        # Write a record to example_store
        await example_store.set_value('some-key', {'foo': 'bar'})

        # Delete a record from example_store
        await example_store.set_value('some-key', None)

Automatic parsing

JSON is automatically parsed to a Python dictionary, text data is returned as a string, and other data is returned as a binary buffer.

from apify import Actor

async def main():
    async with Actor:
        value = await Actor.get_value('my-key')
        # ...
        image_buffer = ...  # Get image data
        await Actor.set_value(key='OUTPUT', value=image_buffer, content_type='image/jpeg')

The Actor.get_input() method is a shortcut to Actor.get_value('INPUT') that also works with Actor.metamorph(). A metamorphed Actor run's input is stored in the INPUT-METAMORPH-1 key instead of INPUT, which holds the original input.

Check out the Python SDK documentation and the KeyValueStore class's API reference for details on managing your key-value stores with the Python SDK.

Compression

Records are stored exactly as you upload them, compressed or uncompressed.

You can compress a record and use the Content-Encoding request header to let the platform know which compression it uses. We recommend compressing large key-value records to save storage space and network traffic.

The JavaScript SDK and the JavaScript API client compress and decompress records automatically.

Share and reuse

You can grant access rights to your key-value store, share it by link, or generate a time-limited pre-signed URL for specific records. See Share storage.

To read from or write to a key-value store that belongs to a different Actor or task run, see Use storage from another run.

Data consistency

Key-value storage uses the AWS S3 service. According to the S3 documentation, it provides strong read-after-write consistency.

Limits

• The maximum length for a key in a key-value store is 256 characters. Keys may only contain the following characters: a-zA-Z0-9!-_.'().

• The maximum length for a key-value store name is 63 characters.

Rate limiting

Operations on a single record (get, put, delete) and getting the list of keys are limited to 200 requests per second per store.

Downloading all records as a ZIP archive is limited to 100 requests per second per store.

All other key-value store API endpoints use the default limit of 60 requests per second per store.

Check out the API documentation for more information and guidance on actions to take if you exceed these rate limits.