Skip to main content

Using HTTPX as the HTTP client

Copy for LLM

This guide shows how to replace the default ImpitHttpClient and ImpitHttpClientAsync with one based on HTTPX. The same approach works for any HTTP library — see Custom HTTP clients for the underlying architecture.

Why HTTPX?

You might want to use HTTPX instead of the default Impit-based client for reasons like:

  • You already use HTTPX in your project and want a single HTTP stack.
  • You need HTTPX-specific features.
  • You want fine-grained control over connection pooling or proxy routing.

Implementation

The implementation involves two steps:

  1. Extend HttpClient (sync) or HttpClientAsync (async) and implement the call method that delegates to HTTPX.
  2. Pass it to ApifyClient.with_custom_http_client to create a client that uses your implementation.

The call method receives parameters like method, url, headers, params, data, json, stream, and timeout. Map them to the corresponding HTTPX arguments — most map directly, except data which becomes HTTPX's content parameter and timeout which needs conversion from timedelta to seconds.

A convenient property of HTTPX is that its httpx.Response object already satisfies the HttpResponse protocol, so you can return it directly without wrapping.

from __future__ import annotations

import asyncio
from typing import TYPE_CHECKING, Any

import httpx

from apify_client import ApifyClientAsync, HttpClientAsync, HttpResponse

if TYPE_CHECKING:
    from datetime import timedelta

TOKEN = 'MY-APIFY-TOKEN'


class HttpxClientAsync(HttpClientAsync):
    """Custom async HTTP client using HTTPX library."""

    def __init__(self) -> None:
        super().__init__()
        self._client = httpx.AsyncClient()

    async def call(
        self,
        *,
        method: str,
        url: str,
        headers: dict[str, str] | None = None,
        params: dict[str, Any] | None = None,
        data: str | bytes | bytearray | None = None,
        json: Any = None,
        stream: bool | None = None,
        timeout: timedelta | None = None,
    ) -> HttpResponse:
        timeout_secs = timeout.total_seconds() if timeout else 0

        # httpx.Response satisfies the HttpResponse protocol,
        # so it can be returned directly.
        return await self._client.request(
            method=method,
            url=url,
            headers=headers,
            params=params,
            content=data,
            json=json,
            timeout=timeout_secs,
        )


async def main() -> None:
    client = ApifyClientAsync.with_custom_http_client(
        token=TOKEN,
        http_client=HttpxClientAsync(),
    )

    actor = await client.actor('apify/hello-world').get()
    print(actor)


if __name__ == '__main__':
    asyncio.run(main())
warning

When using a custom HTTP client, you are responsible for handling retries, timeouts, and error handling yourself. The built-in retry logic with exponential backoff is part of the default ImpitHttpClient and is not applied to custom implementations.

Going further

The example above is minimal on purpose. In a production setup, you might want to extend it with:

  • Retry logic - Use HTTPX's event hooks or utilize library like tenacity to retry failed requests.
  • Custom headers - You can add headers in the call method before delegating to HTTPX.
  • Connection lifecycle - Close the underlying httpx.Client when done by adding a close() method to your custom client.
  • Proxy support - You can pass proxy=... when creating the httpx.Client.
  • Metrics collection - Track request latency, error rates, or other metrics by adding instrumentation in the call method.
  • Logging - Log requests and responses for debugging or auditing purposes.