Build a custom HTTP client

Copy for LLM

This guide implements a custom HttpClient over the global fetch, which Node.js 22 and browsers provide. It shows the three hooks a transport fills in and how a foreign response API is adapted to the HttpResponse shape the pipeline expects.

For an overview of the architecture and the built-in axios client, see HTTP clients.

Implementation

The client has three parts:

1. sendRequest() sends one prepared request and adapts the Response that fetch resolves to. The pipeline hands it the URL with the query string encoded, the headers merged and the body serialized, so the method only moves bytes. The timeout of the attempt becomes an AbortSignal.timeout(), unless the request runs without one, which the pipeline signals with an undefined timeoutMillis. AbortSignal.any() joins it with the signal of a caller who wants to abort the call. A Readable body is streamed, which fetch requires to be flagged with duplex: 'half'. When the caller asked for a streamed response, the body goes back unread as a Readable, otherwise as a Buffer.
2. isRetryableTransportError() maps the transport's transient failures for the shared retry loop. fetch reports every network failure as a TypeError with the underlying error in cause, so the classification reads the error code from there. A timeout from AbortSignal.timeout() is a DOMException named TimeoutError, which the inherited isTimeoutError() already recognizes, so the override only has to make it retryable.
3. ApifyClient.withCustomHttpClient() connects the client to the resource clients and applies the API token.

import { Readable } from 'node:stream';
import type { ReadableStream as NodeReadableStream } from 'node:stream/web';

import { ApifyClient, HttpClient } from 'apify-client';
import type { HttpRequest, HttpResponse } from 'apify-client';

const RETRYABLE_CODES = new Set(['ECONNREFUSED', 'ECONNRESET', 'EPIPE', 'ETIMEDOUT', 'EAI_AGAIN', 'UND_ERR_SOCKET']);

class FetchHttpClient extends HttpClient {
    override async sendRequest(request: HttpRequest): Promise<HttpResponse> {
        const { method, url, headers, body, timeoutMillis, stream, signal } = request;

        const signals = signal ? [signal] : [];
        if (timeoutMillis !== undefined) signals.push(AbortSignal.timeout(timeoutMillis));

        const response = await fetch(url, {
            method,
            headers,
            // `fetch` in Node.js streams a `Readable` body, which the DOM `BodyInit` type doesn't list.
            body: body as BodyInit | undefined,
            signal: AbortSignal.any(signals),
            ...(body instanceof Readable ? { duplex: 'half' } : {}),
        });

        return {
            status: response.status,
            headers: Object.fromEntries(response.headers),
            body:
                stream && response.body
                    ? Readable.fromWeb(response.body as NodeReadableStream)
                    : Buffer.from(await response.arrayBuffer()),
        };
    }

    override isRetryableTransportError(error: unknown): boolean {
        if (this.isTimeoutError(error)) return true;
        return error instanceof TypeError && RETRYABLE_CODES.has((error.cause as { code?: string })?.code ?? '');
    }
}

const client = ApifyClient.withCustomHttpClient({
    token: 'MY-APIFY-TOKEN',
    httpClient: new FetchHttpClient({ maxRetries: 4, timeoutLongSecs: 60 }),
});

const user = await client.user('me').get();
console.log(user?.username);

The constructor options of HttpClient configure the inherited pipeline: retries, the timeout tiers and their cap, default headers and statistics. A transport with a connection pool of its own also overrides close() to release it.

warning

This example is a compact integration, not a replacement for all built-in client behavior. A production custom client should account for transport-specific details such as proxy configuration, TLS settings, redirects and response resource cleanup. Timeout semantics differ per transport too: AbortSignal.timeout() bounds the whole request, headers and body included, while the timeout of the built-in axios client fires after that long without socket activity, so a response whose body keeps trickling in can outlast it.