Streaming resources

Copy for LLM

Certain resources, such as dataset items, key-value store records and logs, support streaming directly from the Apify API. You can then process large resources incrementally without downloading them entirely into memory, which suits large or continuously updated data.

Supported streaming methods:

• LogClient.stream() - Stream a log in real time. Resolves to a Node.js Readable, or undefined when the log doesn't exist. A log client chained off a run or build, such as client.run('run-id').log(), throws a NotFoundError instead, since the run itself may be what is missing.
• DatasetClient.downloadItems() with the stream: true option - Stream dataset items serialized to a format such as CSV or XLSX. Resolves to a Node.js Readable.
• KeyValueStoreClient.getRecord() with the stream: true option - Stream a key-value store record as raw data. Resolves to a record whose value is a Node.js Readable, or undefined when the record doesn't exist.
• RunClient.getStreamedLog() - Redirect the log of a run to a logger while the run is in progress. For details, see Redirecting Actor run logs.

A stream holds its connection open until you consume it to the end or destroy it. Iterating it with for await or passing it to pipeline() from node:stream/promises does either one for you, including when the processing throws.

The following example shows how to stream the log of an Actor run incrementally:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

// Resolves to a Node.js `Readable` that follows the log while the run is in progress.
const logStream = await client.run('MY-RUN-ID').log().stream();

if (logStream) {
    for await (const chunk of logStream) {
        process.stdout.write(chunk);
    }
}

The next example streams a key-value store record straight to a file:

import { createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

// With `stream: true`, the record value is a Node.js `Readable`.
const record = await client.keyValueStore('MY-KVS-ID').getRecord('video.mp4', { stream: true });

if (record) {
    await pipeline(record.value, createWriteStream('video.mp4'));
}

To export a large dataset without holding it in memory, stream its items to a file:

import { createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

// Resolves to a Node.js `Readable` with the items serialized to CSV.
const itemsStream = await client.dataset('MY-DATASET-ID').downloadItems('csv', { stream: true });

await pipeline(itemsStream, createWriteStream('items.csv'));

Streaming needs Node.js, so these methods aren't available in the browser bundle or in other non-Node.js runtimes. For details, see Bundled environments.