Key-value store
Store anything from Actor or task run results, JSON documents, or images. Learn how to access and manage key-value stores from Apify Console or via API.
The key-value store is simple storage that can be used for storing any kind of data. It can be JSON or HTML documents, zip files, images, or simply strings. The data are stored along with their MIME content type.
Each Actor run is assigned its own key-value store when it is created. The store contains the Actor's input, and, if necessary, other data such as its output.
Key-value stores are mutable–you can both add entries and delete them.
Named key-value stores are retained indefinitely.
Unnamed key-value stores expire after 7 days unless otherwise specified.
Learn more
Basic usage
There are five ways to access your key-value stores:
- Apify Console - provides an easy-to-understand interface [details].
- JavaScript SDK - when building your own JavaScript Actor [details].
- Python SDK - when building your own Python Actor [details].
- JavaScript API client - to access your key-value stores from any Node.js application [details].
- Python API client - to access your key-value stores from any Python application [details].
- Apify API - for accessing your key-value stores programmatically [details].
Apify Console
In Apify Console, you can view your key-value stores in the Storage section under the Key-value stores tab.
Only named key-value stores are displayed by default. Select the Include unnamed key-value stores checkbox to display all of your stores.
To view a key-value store's content, click on its Store ID.
Under the Settings tab, you can update the store's name (and, in turn, its retention period) and access rights.
Click on the API
button to view and test a store's API endpoints.
JavaScript SDK
If you are building a JavaScript Actor, you will be using the JavaScript SDK. The key-value store is represented by a KeyValueStore
class. You can use the class to specify whether your data is stored locally or in the Apify cloud, and get and set values using the getValue()
and setValue()
methods respectively, or iterate over your key-value store keys using the forEachKey()
method.
Each Actor run is associated with the default key-value store, which is created for the Actor run. When running your Actors and storing data locally, you can pass its input using the INPUT.json file in the default key-value store directory.
You can find INPUT.json and other key-value store files in the location below.
{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
The default key-value store's ID is default. The {KEY} is the record's key and {EXT} corresponds to the record value's MIME content type.
To manage your key-value stores, you can use the following methods. See the KeyValueStore
class's API reference for the full list.
import { Actor } from 'apify';
await Actor.init();
// ...
// Get the default input
const input = await Actor.getInput();
// Open a named key-value store
const exampleStore = await Actor.openKeyValueStore('my-store');
// Read a record in the exampleStore storage
const value = await exampleStore.getValue('some-key');
// Write a record to exampleStore
await exampleStore.setValue('some-key', { foo: 'bar' });
// Delete a record from exampleStore
await exampleStore.setValue('some-key', null);
// ...
await Actor.exit();
Note that JSON is automatically parsed to a JavaScript object, text data returned as a string and other data is returned as binary buffer.
import { Actor } from 'apify';
await Actor.init();
// ...
// Get input of your Actor
const input = await Actor.getInput();
const value = await Actor.getValue('my-key');
// ...
await Actor.setValue(
'OUTPUT',
imageBuffer,
{ contentType: 'image/jpeg' },
);
// ...
await Actor.exit();
The Actor.getInput()
method is not only a shortcut to Actor.getValue('INPUT')
; it is also compatible with Actor.metamorph()
[docs]. This is because a metamorphed Actor run's input is stored in the INPUT-METAMORPH-1 key instead of INPUT, which hosts the original input.
See the JavaScript SDK documentation and the KeyValueStore
class's API reference for details on managing your key-value stores with the JavaScript SDK.
Python SDK
If you are building a Python Actor, you will be using the Python SDK. The key-value store is represented by a KeyValueStore
class. You can use the KeyValueStore
class to specify whether your data is stored locally or in the Apify cloud, and get and set values using the get_value()
and set_value()
methods respectively.
Each Actor run is associated with the default key-value store, which is created for the Actor run. When running your Actors and storing data locally, you can pass its input using the INPUT.json file in the default key-value store directory.
You can find INPUT.json and other key-value store files in the location below.
{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
The default key-value store's ID is default. The {KEY} is the record's key and {EXT} corresponds to the record value's MIME content type.
To manage your key-value stores, you can use the following methods. See the KeyValueStore
class documentation for the full list.
from apify import Actor
from apify.storages import KeyValueStore
async def main():
async with Actor:
# Open a named key-value store
example_store: KeyValueStore = await Actor.open_key_value_store(name='my-store')
# Read a record in the example_store storage
value = await example_store.get_value('some-key')
# Write a record to example_store
await example_store.set_value('some-key', {'foo': 'bar'})
# Delete a record from example_store
await example_store.set_value('some-key', None)
Note that JSON is automatically parsed to a Python dictionary, text data returned as a string and other data is returned as binary buffer.
from apify import Actor
async def main():
async with Actor:
value = await Actor.get_value('my-key')
# ...
image_buffer = ... # Get image data
await Actor.set_value(key='OUTPUT', value=image_buffer, content_type='image/jpeg')
The Actor.get_input()
method is not only a shortcut to Actor.get_value('INPUT')
; it is also compatible with Actor.metamorph()
[docs]. This is because a metamorphed Actor run's input is stored in the INPUT-METAMORPH-1 key instead of INPUT, which hosts the original input.
See the Python SDK documentation and the KeyValueStore
class's API reference for details on managing your key-value stores with the Python SDK.
JavaScript API client
Apify's JavaScript API client (apify-client
) allows you to access your key-value stores from any Node.js application, whether it is running on the Apify platform or elsewhere.
After importing and initiating the client, you can save each key-value store to a variable for easier access.
const myKeyValStoreClient = apifyClient.keyValueStore('jane-doe/my-key-val-store');
You can then use that variable to access the key-value store's items and manage it.
See the JavaScript API client documentation for help with setup and more details.
Python API client
Apify's Python API client (apify-client
) allows you to access your key-value stores from any Python application, whether it is running on the Apify platform or elsewhere.
After importing and initiating the client, you can save each key-value store to a variable for easier access.
my_key_val_store_client = apify_client.key_value_store('jane-doe/my-key-val-store')
You can then use that variable to access the key-value store's items and manage it.
See the Python API client documentation for help with setup and more details.
Apify API
The Apify API allows you to access your key-value stores programmatically using HTTP requests and easily share your crawling results.
If you are accessing your datasets using the username~store-name store ID format, you will need to use your secret API token. You can find the token (and your user ID) on the Integrations page of your Apify account.
When providing your API authentication token, we recommend using the request's
Authorization
header, rather than the URL. (More info).
To get a list of your key-value stores, send a GET request to the Get list of key-value stores endpoint.
https://api.apify.com/v2/key-value-stores
To get information about a key-value store such as its creation time and item count, send a GET request to the Get store endpoint.
https://api.apify.com/v2/key-value-stores/{STORE_ID}
To get a record (its value) from a key-value store, send a GET request to the Get record endpoint.
https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}
To add a record with a specific key in a key-value store, send a PUT request to the Put record endpoint.
https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}
Example payload:
{
"foo": "bar",
"fos": "baz"
}
To delete a record, send a DELETE request specifying the key from a key-value store to the Delete record endpoint.
https://api.apify.com/v2/key-value-stores/{STORE_ID}/records/{KEY_ID}
See the API documentation for a detailed breakdown of each API endpoint.
Compression
In the past, every record uploaded using the Put record endpoint was compressed using Gzip before uploading. This has changed. Now, records are stored in the state you upload them. This means it is up to you if the record is stored compressed or uncompressed.
You can compress a record and use the Content-Encoding request header to let our platform know which compression it uses. We recommend compressing large key-value records to save storage space and network traffic.
If you use the JavaScript SDK or our JavaScript API client, your files are compressed automatically by default. We recommend using the JavaScript API client, which compresses your data before they are sent to our servers and decompresses them when you retrieve them. This makes your storage costs as low as possible.
Sharing
You can invite other Apify users to view or modify your key-value stores with the access rights system. See the full list of permissions.
Sharing key-value stores between runs
You can access a key-value store from any Actor or task run as long as you know its name or ID.
To access a key-value store from another run using the JavaScript SDK or the Python SDK, open it using the same method as you would do with any other store.
- JavaScript
- Python
import { Actor } from 'apify';
await Actor.init();
const otherStore = await Actor.openKeyValueStore('old-store');
// ...
await Actor.exit();
from apify import Actor
async def main():
async with Actor:
other_store = await Actor.open_key_value_store(name='old-store')
# ...
In the JavaScript API client, you can access a store using its client. Once you've opened a store, read and manage its contents like you would do with a key-value store from your current run.
const otherStoreClient = apifyClient.keyValueStore('jane-doe/old-store');
Likewise, in the Python API client, you can access a store using its client.
other_store_client = apify_client.key_value_store('jane-doe/old-store')
The same applies for the Apify API - you can use the same endpoints as you would normally do.
See the Storage overview for details on sharing storages between runs.
Data consistency
Key-value storage uses the AWS S3 service. According to the S3 documentation, it provides strong read-after-write consistency.
Limits
- Key-value store names can be up to 63 characters long.