ApifyFileSystemStorageBackend

Crawlee's file-system storage backend, aware of the Actor input.

Locally the input arrives in the default key-value store as a bare INPUT or INPUT.json file with no metadata sidecar — written by the Apify CLI, a project template, or by hand. This backend adopts such a file as the record under the input key when the store is opened (both files present fails the open), and keeps the input keys when the default store is purged on start. Everything else is inherited from FileSystemStorageBackend.

Actor installs it as the local backend; pass an instance to Actor.init({ storage }) only if you need to construct it yourself.

Hierarchy

• FileSystemStorageBackend
  • ApifyFileSystemStorageBackend

Index

Constructors

• constructor

Methods

• createDatasetBackend
• createKeyValueStoreBackend
• createRequestQueueBackend
• getStorageBackendCacheKey
• purge
• storageExists
• teardown

Constructors

constructor

• new ApifyFileSystemStorageBackend(options): ApifyFileSystemStorageBackend

• Overrides FileSystemStorageBackend.constructor

  Parameters

  • options: ApifyFileSystemStorageOptions

  Returns ApifyFileSystemStorageBackend

Methods

externalinheritedcreateDatasetBackend

• createDatasetBackend(options): Promise<DatasetBackend<Dictionary>>

• Inherited from FileSystemStorageBackend.createDatasetBackend

  Create (or open) a dataset backend. If id is provided, opens the dataset with that ID. If name is provided, opens an existing dataset with that name or creates a new one. If neither is provided, opens or creates the default dataset.

  ---

  Parameters

  • externaloptionaloptions: StorageIdentifier

  Returns Promise<DatasetBackend<Dictionary>>

externalinheritedcreateKeyValueStoreBackend

• createKeyValueStoreBackend(options): Promise<KeyValueStoreBackend>

• Inherited from FileSystemStorageBackend.createKeyValueStoreBackend

  Create (or open) a key-value store backend. If id is provided, opens the key-value store with that ID. If name is provided, opens an existing store with that name or creates a new one. If neither is provided, opens or creates the default key-value store.

  ---

  Parameters

  • externaloptionaloptions: StorageIdentifier

  Returns Promise<KeyValueStoreBackend>

externalinheritedcreateRequestQueueBackend

• createRequestQueueBackend(options): Promise<RequestQueueBackend>

• Inherited from FileSystemStorageBackend.createRequestQueueBackend

  Create (or open) a request queue backend. If id is provided, opens the request queue with that ID. If name is provided, opens an existing queue with that name or creates a new one. If neither is provided, opens or creates the default request queue.

  ---

  Parameters

  • externaloptionaloptions: StorageIdentifier

  Returns Promise<RequestQueueBackend>

externalinheritedgetStorageBackendCacheKey

• getStorageBackendCacheKey(): string

• Inherited from FileSystemStorageBackend.getStorageBackendCacheKey

  Return a cache key that includes the resolved storage directory, so that two FileSystemStorageBackend instances pointing at different directories get separate cache partitions, by including the storage directory in the cache key.

  ---

  Returns string

externalinheritedpurge

• purge(): Promise<void>

• Inherited from FileSystemStorageBackend.purge

  Cleans up the run-scoped storages before the run starts, sweeping the storage directories so that leftovers from a previous process are caught too.

  ---

  Returns Promise<void>

externalinheritedstorageExists

• storageExists(id, type): Promise<boolean>

• Inherited from FileSystemStorageBackend.storageExists

  Check whether a storage with the given ID exists.

  Used internally to resolve ambiguous idOrName strings passed to Dataset.open(), KeyValueStore.open(), and RequestQueue.open().

  ---

  Parameters

  • externalid: string

  • externaltype: Dataset | KeyValueStore | RequestQueue

  Returns Promise<boolean>

externalinheritedteardown

• teardown(): Promise<void>

• Inherited from FileSystemStorageBackend.teardown

  This method should be called at the end of the process, to ensure all data is saved.

  It persists the state of every opened request queue so that requests fetched but not yet handled are not stuck (until their lock expires) for the next consumer of the same on-disk queue.

  ---

  Returns Promise<void>