Pagination

Copy for LLM

Most methods named list() or listSomething() in the Apify client resolve to a PaginatedList. All such pages share a consistent interface for working with paginated data and expose the following fields:

• items - The main results you're looking for.
• total - The total number of items available.
• offset - The starting point of the current page.
• count - The number of items in the current page.
• limit - The maximum number of items per page.
• desc - Whether the items are sorted in descending order.

Some methods paginate differently. KeyValueStoreClient.listKeys() pages by key: to fetch the next page, pass its nextExclusiveStartKey value back as the exclusiveStartKey option. RequestQueueClient.listRequests() pages by cursor: pass its nextCursor value back as the cursor option. listHead() returns only the first requests of the queue. Regardless, the primary results are always stored under the items field, and the limit option controls the number of results returned.

The following example shows how to fetch one page of dataset items:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

const datasetClient = client.dataset('dataset-id');

// Fetch a single page of up to 1000 items in one API call.
const page = await datasetClient.listItems({ limit: 1000, offset: 0 });

// Inspect the pagination metadata returned by the API.
console.log(page.total);

for (const item of page.items) {
    console.log(item);
}

A single call returns at most one page, and the API can cap how many items that page holds. To go through a whole collection, iterate it instead.

Async iteration

The promise that a paginated method returns is also an async iterable. Iterating it with for await fetches as many pages as needed to yield every item matching the options, so you don't need to manage offsets, limits, or cursors yourself. The limit option caps the total, and chunkSize caps the number of items per API call. paginateRequests() yields whole pages of requests.

The following example iterates over every Actor owned by the current user with a collection client's list() method:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

// Iterate over all Actors you own, fetching as many pages as needed.
for await (const actor of client.actors().list({ my: true })) {
    console.log(actor.id);
}

The next example iterates dataset items past a given offset:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

const datasetClient = client.dataset('dataset-id');

// Iterate through 1500 items past the first 100, fetching up to 100 items per API call.
for await (const item of datasetClient.listItems({ limit: 1500, offset: 100, chunkSize: 100 })) {
    console.log(item);
}

The API applies offset and limit of listItems() to the dataset's rows before clean, skipEmpty, skipHidden and unwind reshape them. With any of those options set, a page can hold fewer or more items than the rows it covered, so iteration can yield fewer or more than limit items, each of them once.

To read the items of a run that's still going, use RunClient.iterateDatasetItems(). Its offset and limit also count dataset rows. For details, see Retrieve Actor data.