Dataset
Index
Methods
drop
Remove the dataset either from the Apify cloud storage or from the local directory.
Returns None
export_to
Save the entirety of the dataset’s contents into one file within a key-value store.
Parameters
key: str
The key to save the data under.
optionalto_key_value_store_id: str
The id of the key-value store in which the result will be saved.
optionalto_key_value_store_name: str
The name of the key-value store in which the result will be saved. You must specify only one of
to_key_value_store_id
andto_key_value_store_name
arguments. If you omit both, it uses the default key-value store.optionalcontent_type: str
Either ‘text/csv’ or ‘application/json’. Defaults to JSON.
Returns None
export_to_csv
Save the entirety of the dataset’s contents into one CSV file within a key-value store.
Parameters
key: str
The key to save the data under.
optionalfrom_dataset_id: str
The ID of the dataset in case of calling the class method. Uses default dataset if omitted.
optionalfrom_dataset_name: str
The name of the dataset in case of calling the class method. Uses default dataset if omitted. You must specify only one of
from_dataset_id
andfrom_dataset_name
arguments. If you omit both, it uses the default dataset.optionalto_key_value_store_id: str
The id of the key-value store in which the result will be saved.
optionalto_key_value_store_name: str
The name of the key-value store in which the result will be saved. You must specify only one of
to_key_value_store_id
andto_key_value_store_name
arguments. If you omit both, it uses the default key-value store.
Returns None
export_to_json
Save the entirety of the dataset’s contents into one JSON file within a key-value store.
Parameters
key: str
The key to save the data under.
optionalfrom_dataset_id: str
The ID of the dataset in case of calling the class method. Uses default dataset if omitted.
optionalfrom_dataset_name: str
The name of the dataset in case of calling the class method. Uses default dataset if omitted. You must specify only one of
from_dataset_id
andfrom_dataset_name
arguments. If you omit both, it uses the default dataset.optionalto_key_value_store_id: str
The id of the key-value store in which the result will be saved.
optionalto_key_value_store_name: str
The name of the key-value store in which the result will be saved. You must specify only one of
to_key_value_store_id
andto_key_value_store_name
arguments. If you omit both, it uses the default key-value store.
Returns None
get_data
Get items from the dataset.
Parameters
optionaloffset: int
Number of items that should be skipped at the start. The default value is 0
optionallimit: int
Maximum number of items to return. By default there is no limit.
optionalclean: bool
If True, returns only non-empty items and skips hidden fields (i.e. fields starting with the # character). The clean parameter is just a shortcut for skip_hidden=True and skip_empty=True parameters. Note that since some objects might be skipped from the output, that the result might contain less items than the limit value.
optionaldesc: bool
By default, results are returned in the same order as they were stored. To reverse the order, set this parameter to True.
optionalfields: List[str]
A list of fields which should be picked from the items, only these fields will remain in the resulting record objects. Note that the fields in the outputted items are sorted the same way as they are specified in the fields parameter. You can use this feature to effectively fix the output format.
optionalomit: List[str]
A list of fields which should be omitted from the items.
optionalunwind: str
Name of a field which should be unwound. If the field is an array then every element of the array will become a separate record and merged with parent object. If the unwound field is an object then it is merged with the parent object. If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object, then the item gets preserved as it is. Note that the unwound items ignore the desc parameter.
optionalskip_empty: bool
If True, then empty items are skipped from the output. Note that if used, the results might contain less items than the limit value.
optionalskip_hidden: bool
If True, then hidden fields are skipped from the output, i.e. fields starting with the # character.
optionalflatten: List[str]
A list of fields that should be flattened
optionalview: str
Name of the dataset view to be used
Returns ListPage
get_info
Get an object containing general information about the dataset.
Returns Dict
iterate_items
Iterate over the items in the dataset.
Parameters
offset: int
Number of items that should be skipped at the start. The default value is 0
optionallimit: int
Maximum number of items to return. By default there is no limit.
optionalclean: bool
If True, returns only non-empty items and skips hidden fields (i.e. fields starting with the # character). The clean parameter is just a shortcut for skip_hidden=True and skip_empty=True parameters. Note that since some objects might be skipped from the output, that the result might contain less items than the limit value.
optionaldesc: bool
By default, results are returned in the same order as they were stored. To reverse the order, set this parameter to True.
optionalfields: List[str]
A list of fields which should be picked from the items, only these fields will remain in the resulting record objects. Note that the fields in the outputted items are sorted the same way as they are specified in the fields parameter. You can use this feature to effectively fix the output format.
optionalomit: List[str]
A list of fields which should be omitted from the items.
optionalunwind: str
Name of a field which should be unwound. If the field is an array then every element of the array will become a separate record and merged with parent object. If the unwound field is an object then it is merged with the parent object. If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object, then the item gets preserved as it is. Note that the unwound items ignore the desc parameter.
optionalskip_empty: bool
If True, then empty items are skipped from the output. Note that if used, the results might contain less items than the limit value.
optionalskip_hidden: bool
If True, then hidden fields are skipped from the output, i.e. fields starting with the # character.
Yields: dict: An item from the dataset
Returns AsyncIterator[Dict]
open
Open a dataset.
Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. The actual data is stored either on the local filesystem or in the Apify cloud.
Parameters
optionalid: str
ID of the dataset to be opened. If neither
id
norname
are provided, the method returns the default dataset associated with the actor run. If the dataset with the given ID does not exist, it raises an error.optionalname: str
Name of the dataset to be opened. If neither
id
norname
are provided, the method returns the default dataset associated with the actor run. If the dataset with the given name does not exist, it is created.force_cloud: bool
If set to True, it will open a dataset on the Apify Platform even when running the actor locally. Defaults to False.
optionalconfig: Configuration
A
Configuration
instance, uses global configuration if omitted.
Returns 'Dataset'
push_data
Store an object or an array of objects to the dataset.
The size of the data is limited by the receiving API and therefore
push_data()
will only allow objects whose JSON representation is smaller than 9MB. When an array is passed, none of the included objects may be larger than 9MB, but the array itself may be of any size.Parameters
data: JSONSerializable
dict or array of dicts containing data to be stored in the default dataset. The JSON representation of each item must be smaller than 9MB.
Returns None
The
Dataset
class represents a store for structured data where each object stored has the same attributes.You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records. Typically it is used to store crawling results.
Do not instantiate this class directly, use the
Actor.open_dataset()
function instead.Dataset
stores its data either on local disk or in the Apify cloud, depending on whether theAPIFY_LOCAL_STORAGE_DIR
orAPIFY_TOKEN
environment variables are set.If the
APIFY_LOCAL_STORAGE_DIR
environment variable is set, the data is stored in the local directory in the following files:Note that
{DATASET_ID}
is the name or ID of the dataset. The default dataset has ID:default
, unless you override it by setting theAPIFY_DEFAULT_DATASET_ID
environment variable. Each dataset item is stored as a separate JSON file, where{INDEX}
is a zero-based index of the item in the dataset.If the
APIFY_TOKEN
environment variable is set butAPIFY_LOCAL_STORAGE_DIR
is not, the data is stored in the Apify Dataset cloud storage.