Dataset
apify.storages.Dataset
Index
Methods
drop
Remove the dataset either from the Apify cloud storage or from the local directory.
Returns None
export_to
Save the entirety of the dataset’s contents into one file within a key-value store.
Parameters
key: str
The key to save the data under.
to_key_value_store_id: str | None = Nonekeyword-only
The id of the key-value store in which the result will be saved.
to_key_value_store_name: str | None = Nonekeyword-only
The name of the key-value store in which the result will be saved. You must specify only one of
to_key_value_store_id
andto_key_value_store_name
arguments. If you omit both, it uses the default key-value store.content_type: str | None = Nonekeyword-only
Either ‘text/csv’ or ‘application/json’. Defaults to JSON.
Returns None
export_to_csv
Save the entirety of the dataset’s contents into one CSV file within a key-value store.
Parameters
key: str
The key to save the data under.
from_dataset_id: str | None = Nonekeyword-only
The ID of the dataset in case of calling the class method. Uses default dataset if omitted.
from_dataset_name: str | None = Nonekeyword-only
The name of the dataset in case of calling the class method. Uses default dataset if omitted. You must specify only one of
from_dataset_id
andfrom_dataset_name
arguments. If you omit both, it uses the default dataset.to_key_value_store_id: str | None = Nonekeyword-only
The id of the key-value store in which the result will be saved.
to_key_value_store_name: str | None = Nonekeyword-only
The name of the key-value store in which the result will be saved. You must specify only one of
to_key_value_store_id
andto_key_value_store_name
arguments. If you omit both, it uses the default key-value store.
Returns None
export_to_json
Save the entirety of the dataset’s contents into one JSON file within a key-value store.
Parameters
key: str
The key to save the data under.
from_dataset_id: str | None = Nonekeyword-only
The ID of the dataset in case of calling the class method. Uses default dataset if omitted.
from_dataset_name: str | None = Nonekeyword-only
The name of the dataset in case of calling the class method. Uses default dataset if omitted. You must specify only one of
from_dataset_id
andfrom_dataset_name
arguments. If you omit both, it uses the default dataset.to_key_value_store_id: str | None = Nonekeyword-only
The id of the key-value store in which the result will be saved.
to_key_value_store_name: str | None = Nonekeyword-only
The name of the key-value store in which the result will be saved. You must specify only one of
to_key_value_store_id
andto_key_value_store_name
arguments. If you omit both, it uses the default key-value store.
Returns None
get_data
Get items from the dataset.
Parameters
offset: int | None = Nonekeyword-only
Number of items that should be skipped at the start. The default value is 0
limit: int | None = Nonekeyword-only
Maximum number of items to return. By default there is no limit.
clean: bool | None = Nonekeyword-only
If True, returns only non-empty items and skips hidden fields (i.e. fields starting with the # character). The clean parameter is just a shortcut for skip_hidden=True and skip_empty=True parameters. Note that since some objects might be skipped from the output, that the result might contain less items than the limit value.
desc: bool | None = Nonekeyword-only
By default, results are returned in the same order as they were stored. To reverse the order, set this parameter to True.
fields: list[str] | None = Nonekeyword-only
A list of fields which should be picked from the items, only these fields will remain in the resulting record objects. Note that the fields in the outputted items are sorted the same way as they are specified in the fields parameter. You can use this feature to effectively fix the output format.
omit: list[str] | None = Nonekeyword-only
A list of fields which should be omitted from the items.
unwind: str | None = Nonekeyword-only
Name of a field which should be unwound. If the field is an array then every element of the array will become a separate record and merged with parent object. If the unwound field is an object then it is merged with the parent object. If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object, then the item gets preserved as it is. Note that the unwound items ignore the desc parameter.
skip_empty: bool | None = Nonekeyword-only
If True, then empty items are skipped from the output. Note that if used, the results might contain less items than the limit value.
skip_hidden: bool | None = Nonekeyword-only
If True, then hidden fields are skipped from the output, i.e. fields starting with the # character.
flatten: list[str] | None = Nonekeyword-only
A list of fields that should be flattened
view: str | None = Nonekeyword-only
Name of the dataset view to be used
Returns ListPage
A page of the list of dataset items according to the specified filters.
get_info
Get an object containing general information about the dataset.
Returns dict | None
Object returned by calling the GET dataset API endpoint.
iterate_items
Iterate over the items in the dataset.
Parameters
offset: int = 0keyword-only
Number of items that should be skipped at the start. The default value is 0
limit: int | None = Nonekeyword-only
Maximum number of items to return. By default there is no limit.
clean: bool | None = Nonekeyword-only
If True, returns only non-empty items and skips hidden fields (i.e. fields starting with the # character). The clean parameter is just a shortcut for skip_hidden=True and skip_empty=True parameters. Note that since some objects might be skipped from the output, that the result might contain less items than the limit value.
desc: bool | None = Nonekeyword-only
By default, results are returned in the same order as they were stored. To reverse the order, set this parameter to True.
fields: list[str] | None = Nonekeyword-only
A list of fields which should be picked from the items, only these fields will remain in the resulting record objects. Note that the fields in the outputted items are sorted the same way as they are specified in the fields parameter. You can use this feature to effectively fix the output format.
omit: list[str] | None = Nonekeyword-only
A list of fields which should be omitted from the items.
unwind: str | None = Nonekeyword-only
Name of a field which should be unwound. If the field is an array then every element of the array will become a separate record and merged with parent object. If the unwound field is an object then it is merged with the parent object. If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object, then the item gets preserved as it is. Note that the unwound items ignore the desc parameter.
skip_empty: bool | None = Nonekeyword-only
If True, then empty items are skipped from the output. Note that if used, the results might contain less items than the limit value.
skip_hidden: bool | None = Nonekeyword-only
If True, then hidden fields are skipped from the output, i.e. fields starting with the # character.
Yields: dict: An item from the dataset
Returns AsyncIterator[dict]
open
Open a dataset.
Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. The actual data is stored either on the local filesystem or in the Apify cloud.
Parameters
id: str | None = Nonekeyword-only
ID of the dataset to be opened. If neither
id
norname
are provided, the method returns the default dataset associated with the actor run. If the dataset with the given ID does not exist, it raises an error.name: str | None = Nonekeyword-only
Name of the dataset to be opened. If neither
id
norname
are provided, the method returns the default dataset associated with the actor run. If the dataset with the given name does not exist, it is created.force_cloud: bool = Falsekeyword-only
If set to True, it will open a dataset on the Apify Platform even when running the actor locally. Defaults to False.
config: Configuration | None = Nonekeyword-only
A
Configuration
instance, uses global configuration if omitted.
Returns Dataset
An instance of the
Dataset
class for the given ID or name.
push_data
Store an object or an array of objects to the dataset.
The size of the data is limited by the receiving API and therefore
push_data()
will only allow objects whose JSON representation is smaller than 9MB. When an array is passed, none of the included objects may be larger than 9MB, but the array itself may be of any size.Parameters
data: JSONSerializable
dict or array of dicts containing data to be stored in the default dataset. The JSON representation of each item must be smaller than 9MB.
Returns None
The
Dataset
class represents a store for structured data where each object stored has the same attributes.You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records. Typically it is used to store crawling results.
Do not instantiate this class directly, use the
Actor.open_dataset()
function instead.Dataset
stores its data either on local disk or in the Apify cloud, depending on whether theAPIFY_LOCAL_STORAGE_DIR
orAPIFY_TOKEN
environment variables are set.If the
APIFY_LOCAL_STORAGE_DIR
environment variable is set, the data is stored in the local directory in the following files:Note that
{DATASET_ID}
is the name or ID of the dataset. The default dataset has ID:default
, unless you override it by setting theAPIFY_DEFAULT_DATASET_ID
environment variable. Each dataset item is stored as a separate JSON file, where{INDEX}
is a zero-based index of the item in the dataset.If the
APIFY_TOKEN
environment variable is set butAPIFY_LOCAL_STORAGE_DIR
is not, the data is stored in the Apify Dataset cloud storage.