Skip to main content

KeyValueStore

The KeyValueStore class represents a key-value store.

You can imagine it as a simple data storage that is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME content type.

Do not instantiate this class directly, use the Actor.open_key_value_store() function instead.

Each crawler run is associated with a default key-value store, which is created exclusively for the run. By convention, the crawler input and output are stored into the default key-value store under the INPUT and OUTPUT key, respectively. Typically, input and output are JSON files, although it can be any other format. To access the default key-value store directly, you can use the KeyValueStore.get_value and KeyValueStore.set_value convenience functions.

KeyValueStore stores its data either on local disk or in the Apify cloud, depending on whether the APIFY_LOCAL_STORAGE_DIR or APIFY_TOKEN environment variables are set.

If the APIFY_LOCAL_STORAGE_DIR environment variable is set, the data is stored in the local directory in the following files:

{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}

Note that {STORE_ID} is the name or ID of the key-value store. The default key-value store has ID: default, unless you override it by setting the APIFY_DEFAULT_KEY_VALUE_STORE_ID environment variable. The {KEY} is the key of the record and {EXT} corresponds to the MIME content type of the data value.

If the APIFY_TOKEN environment variable is set but APIFY_LOCAL_STORAGE_DIR is not, the data is stored in the Apify Key-value store cloud storage.

Index

Methods

Methods

drop

  • async drop(): None

  • Remove the key-value store either from the Apify cloud storage or from the local directory.

    Returns None

get_public_url

  • async get_public_url(key): str

  • Get a URL for the given key that may be used to publicly access the value in the remote key-value store.

    Parameters

    • key: str

      The key for which the URL should be generated.

    Returns str

get_value

  • async get_value(key, default_value): T

  • Get a value from the key-value store.

    Parameters

    • key: str

      Key of the record to retrieve.

    • optionaldefault_value: T

      Default value returned in case the record does not exist.

    Returns T

iterate_keys

  • async iterate_keys(exclusive_start_key): AsyncIterator[IterateKeysTuple]

  • Iterate over the keys in the key-value store.

    Parameters

    • optionalexclusive_start_key: str

      All keys up to this one (including) are skipped from the result.

      Yields: IterateKeysTuple: A tuple (key, info), where key is the record key, and info is an object that contains a single property size indicating size of the record in bytes.

    Returns AsyncIterator[IterateKeysTuple]

open

  • async open(id, name, force_cloud, config): 'KeyValueStore'

  • Open a key-value store.

    Key-value stores are used to store records or files, along with their MIME content type. The records are stored and retrieved using a unique key. The actual data is stored either on a local filesystem or in the Apify cloud.

    Parameters

    • optionalid: str

      ID of the key-value store to be opened. If neither id nor name are provided, the method returns the default key-value store associated with the actor run. If the key-value store with the given ID does not exist, it raises an error.

    • optionalname: str

      Name of the key-value store to be opened. If neither id nor name are provided, the method returns the default key-value store associated with the actor run. If the key-value store with the given name does not exist, it is created.

    • force_cloud: bool

      If set to True, it will open a key-value store on the Apify Platform even when running the actor locally. Defaults to False.

    • optionalconfig: Configuration

      A Configuration instance, uses global configuration if omitted.

    Returns 'KeyValueStore'

set_value

  • async set_value(key, value, content_type): None

  • Set or delete a value in the key-value store.

    Parameters

    • key: str

      The key under which the value should be saved.

    • optionalvalue: T

      The value to save. If the value is None, the corresponding key-value pair will be deleted.

    • optionalcontent_type: str

      The content type of the saved value.

    Returns None