Skip to main content
Version: 0.2

Dataset

{"content": ["The Dataset class represents a store for structured data where each object stored has the same attributes.\n\nYou can imagine it as a table, where each object is a row and its attributes are columns.\nDataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records.\nTypically it is used to store crawling results.\n\nDo not instantiate this class directly, use the Actor.open_dataset() function instead.\n\nDataset stores its data either on local disk or in the Apify cloud,\ndepending on whether the APIFY_LOCAL_STORAGE_DIR or APIFY_TOKEN environment variables are set.\n\nIf the APIFY_LOCAL_STORAGE_DIR environment variable is set, the data is stored in\nthe local directory in the following files:\n``\n{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json\n``\nNote that {DATASET_ID} is the name or ID of the dataset. The default dataset has ID: default,\nunless you override it by setting the APIFY_DEFAULT_DATASET_ID environment variable.\nEach dataset item is stored as a separate JSON file, where {INDEX} is a zero-based index of the item in the dataset.\n\nIf the APIFY_TOKEN environment variable is set but APIFY_LOCAL_STORAGE_DIR is not, the data is stored in the\nApify Dataset cloud storage."]}

Index

Constructors

Methods

Constructors

__init__

  • __init__(id, name, client, config): None

  • {"content": ["Create a Dataset instance.\n\nDo not use the constructor directly, use the Actor.open_dataset() function instead.\n\nArgs:\n id (str): ID of the dataset.\n name (str, optional): Name of the dataset.\n client (ApifyClientAsync or MemoryStorageClient): The storage client which should be used.\n config (Configuration): The configuration which should be used."]}

    Parameters

    • id: str
    • name: Optional[str]
    • client: Union[ApifyClientAsync, MemoryStorageClient]
    • config: Configuration

    Returns None

Methods

push_data

  • async push_data(data): None

  • {"content": ["Store an object or an array of objects to the dataset.\n\nThe size of the data is limited by the receiving API and therefore push_data() will only\nallow objects whose JSON representation is smaller than 9MB. When an array is passed,\nnone of the included objects may be larger than 9MB, but the array itself may be of any size.\n\nArgs:\n data (JSONSerializable): dict or array of dicts containing data to be stored in the default dataset.\n The JSON representation of each item must be smaller than 9MB."]}

    Parameters

    • data: JSONSerializable

    Returns None

get_data

  • async get_data(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden, flatten, view): ListPage

  • {"content": ["Get items from the dataset.\n\nArgs:\n offset (int, optional): Number of items that should be skipped at the start. The default value is 0\n limit (int, optional): Maximum number of items to return. By default there is no limit.\n desc (bool, optional): By default, results are returned in the same order as they were stored.\n To reverse the order, set this parameter to True.\n clean (bool, optional): If True, returns only non-empty items and skips hidden fields (i.e. fields starting with the # character).\n The clean parameter is just a shortcut for skip_hidden=True and skip_empty=True parameters.\n Note that since some objects might be skipped from the output, that the result might contain less items than the limit value.\n fields (list of str, optional): A list of fields which should be picked from the items,\n only these fields will remain in the resulting record objects.\n Note that the fields in the outputted items are sorted the same way as they are specified in the fields parameter.\n You can use this feature to effectively fix the output format.\n omit (list of str, optional): A list of fields which should be omitted from the items.\n unwind (str, optional): Name of a field which should be unwound.\n If the field is an array then every element of the array will become a separate record and merged with parent object.\n If the unwound field is an object then it is merged with the parent object.\n If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object,\n then the item gets preserved as it is. Note that the unwound items ignore the desc parameter.\n skip_empty (bool, optional): If True, then empty items are skipped from the output.\n Note that if used, the results might contain less items than the limit value.\n skip_hidden (bool, optional): If True, then hidden fields are skipped from the output, i.e. fields starting with the # character.\n flatten (list of str, optional): A list of fields that should be flattened\n view (str, optional): Name of the dataset view to be used\n\nReturns:\n ListPage: A page of the list of dataset items according to the specified filters."]}

    Parameters

    • keyword-onlyoffset: Optional[int] = None
    • keyword-onlylimit: Optional[int] = None
    • keyword-onlyclean: Optional[bool] = None
    • keyword-onlydesc: Optional[bool] = None
    • keyword-onlyfields: Optional[List[str]] = None
    • keyword-onlyomit: Optional[List[str]] = None
    • keyword-onlyunwind: Optional[str] = None
    • keyword-onlyskip_empty: Optional[bool] = None
    • keyword-onlyskip_hidden: Optional[bool] = None
    • keyword-onlyflatten: Optional[List[str]] = None
    • keyword-onlyview: Optional[str] = None

    Returns ListPage

export_to

  • async export_to(key, *, to_key_value_store_id, to_key_value_store_name, content_type): None

  • {"content": ["Save the entirety of the dataset's contents into one file within a key-value store.\n\nArgs:\n key (str): The key to save the data under.\n to_key_value_store_id (str, optional): The id of the key-value store in which the result will be saved.\n to_key_value_store_name (str, optional): The name of the key-value store in which the result will be saved.\n You must specify only one of to_key_value_store_id and to_key_value_store_name arguments.\n If you omit both, it uses the default key-value store.\n content_type (str, optional): Either 'text/csv' or 'application/json'. Defaults to JSON."]}

    Parameters

    • key: str
    • keyword-onlyto_key_value_store_id: Optional[str] = None
    • keyword-onlyto_key_value_store_name: Optional[str] = None
    • keyword-onlycontent_type: Optional[str] = None

    Returns None

export_to_json

  • async export_to_json(key, *, from_dataset_id, from_dataset_name, to_key_value_store_id, to_key_value_store_name): None

  • {"content": ["Save the entirety of the dataset's contents into one JSON file within a key-value store.\n\nArgs:\n key (str): The key to save the data under.\n from_dataset_id (str, optional): The ID of the dataset in case of calling the class method. Uses default dataset if omitted.\n from_dataset_name (str, optional): The name of the dataset in case of calling the class method. Uses default dataset if omitted.\n You must specify only one of from_dataset_id and from_dataset_name arguments.\n If you omit both, it uses the default dataset.\n to_key_value_store_id (str, optional): The id of the key-value store in which the result will be saved.\n to_key_value_store_name (str, optional): The name of the key-value store in which the result will be saved.\n You must specify only one of to_key_value_store_id and to_key_value_store_name arguments.\n If you omit both, it uses the default key-value store."]}

    Parameters

    • key: str
    • keyword-onlyfrom_dataset_id: Optional[str] = None
    • keyword-onlyfrom_dataset_name: Optional[str] = None
    • keyword-onlyto_key_value_store_id: Optional[str] = None
    • keyword-onlyto_key_value_store_name: Optional[str] = None

    Returns None

export_to_csv

  • async export_to_csv(key, *, from_dataset_id, from_dataset_name, to_key_value_store_id, to_key_value_store_name): None

  • {"content": ["Save the entirety of the dataset's contents into one CSV file within a key-value store.\n\nArgs:\n key (str): The key to save the data under.\n from_dataset_id (str, optional): The ID of the dataset in case of calling the class method. Uses default dataset if omitted.\n from_dataset_name (str, optional): The name of the dataset in case of calling the class method. Uses default dataset if omitted.\n You must specify only one of from_dataset_id and from_dataset_name arguments.\n If you omit both, it uses the default dataset.\n to_key_value_store_id (str, optional): The id of the key-value store in which the result will be saved.\n to_key_value_store_name (str, optional): The name of the key-value store in which the result will be saved.\n You must specify only one of to_key_value_store_id and to_key_value_store_name arguments.\n If you omit both, it uses the default key-value store."]}

    Parameters

    • key: str
    • keyword-onlyfrom_dataset_id: Optional[str] = None
    • keyword-onlyfrom_dataset_name: Optional[str] = None
    • keyword-onlyto_key_value_store_id: Optional[str] = None
    • keyword-onlyto_key_value_store_name: Optional[str] = None

    Returns None

get_info

  • async get_info(): Optional[Dict]

  • {"content": ["Get an object containing general information about the dataset.\n\nReturns:\n dict: Object returned by calling the GET dataset API endpoint."]}

    Returns Optional[Dict]

iterate_items

  • iterate_items(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden): AsyncIterator[Dict]

  • {"content": ["Iterate over the items in the dataset.\n\nArgs:\n offset (int, optional): Number of items that should be skipped at the start. The default value is 0\n limit (int, optional): Maximum number of items to return. By default there is no limit.\n desc (bool, optional): By default, results are returned in the same order as they were stored.\n To reverse the order, set this parameter to True.\n clean (bool, optional): If True, returns only non-empty items and skips hidden fields (i.e. fields starting with the # character).\n The clean parameter is just a shortcut for skip_hidden=True and skip_empty=True parameters.\n Note that since some objects might be skipped from the output, that the result might contain less items than the limit value.\n fields (list of str, optional): A list of fields which should be picked from the items,\n only these fields will remain in the resulting record objects.\n Note that the fields in the outputted items are sorted the same way as they are specified in the fields parameter.\n You can use this feature to effectively fix the output format.\n omit (list of str, optional): A list of fields which should be omitted from the items.\n unwind (str, optional): Name of a field which should be unwound.\n If the field is an array then every element of the array will become a separate record and merged with parent object.\n If the unwound field is an object then it is merged with the parent object.\n If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object,\n then the item gets preserved as it is. Note that the unwound items ignore the desc parameter.\n skip_empty (bool, optional): If True, then empty items are skipped from the output.\n Note that if used, the results might contain less items than the limit value.\n skip_hidden (bool, optional): If True, then hidden fields are skipped from the output, i.e. fields starting with the # character.\n\nYields:\n dict: An item from the dataset"]}

    Parameters

    • keyword-onlyoffset: int = 0
    • keyword-onlylimit: Optional[int] = None
    • keyword-onlyclean: Optional[bool] = None
    • keyword-onlydesc: Optional[bool] = None
    • keyword-onlyfields: Optional[List[str]] = None
    • keyword-onlyomit: Optional[List[str]] = None
    • keyword-onlyunwind: Optional[str] = None
    • keyword-onlyskip_empty: Optional[bool] = None
    • keyword-onlyskip_hidden: Optional[bool] = None

    Returns AsyncIterator[Dict]

drop

  • async drop(): None

  • {"content": ["Remove the dataset either from the Apify cloud storage or from the local directory."]}

    Returns None

open

  • async open(*, id, name, force_cloud, config): 'Dataset'

  • {"content": ["Open a dataset.\n\nDatasets are used to store structured data where each object stored has the same attributes,\nsuch as online store products or real estate offers.\nThe actual data is stored either on the local filesystem or in the Apify cloud.\n\nArgs:\n id (str, optional): ID of the dataset to be opened.\n If neither id nor name are provided, the method returns the default dataset associated with the actor run.\n If the dataset with the given ID does not exist, it raises an error.\n name (str, optional): Name of the dataset to be opened.\n If neither id nor name are provided, the method returns the default dataset associated with the actor run.\n If the dataset with the given name does not exist, it is created.\n force_cloud (bool, optional): If set to True, it will open a dataset on the Apify Platform even when running the actor locally.\n Defaults to False.\n config (Configuration, optional): A Configuration instance, uses global configuration if omitted.\n\nReturns:\n Dataset: An instance of the Dataset class for the given ID or name."]}

    Parameters

    • keyword-onlyid: Optional[str] = None
    • keyword-onlyname: Optional[str] = None
    • keyword-onlyforce_cloud: bool = False
    • keyword-onlyconfig: Optional[Configuration] = None

    Returns 'Dataset'

Page Options