Skip to main content
Version: Next

HTTP compression

The Apify client compresses request bodies before sending them to the API. It reduces the amount of data transferred over the network, resulting in faster requests and lower bandwidth usage, especially for large payloads such as Actor inputs, dataset uploads, or key-value store records.

How it works

The client compresses request bodies using the compressor configured via the compression parameter (default 'gzip'). The server supports gzip, brotli, and zstd and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, the request carries no Content-Encoding of its own, and the body isn't streamed from a file or an iterator. For details, see Minimum body size, Already-compressed payloads, Pre-compressed bodies, and Streaming uploads.

Minimum body size

The client sends bodies smaller than 1024 bytes without compression and without the Content-Encoding header. A body of this size fits in one network packet, so compression doesn't remove a network round trip and only costs CPU time. For very small bodies, the compression format adds bytes and can make the body larger.

Already-compressed payloads

Some payloads carry their own compression, so compressing them again costs CPU and memory while making the request slightly larger. The client skips compression when the request's Content-Type is one of these:

  • any image/*, audio/*, or video/* type
  • archives such as application/zip, application/gzip, or application/x-7z-compressed
  • office documents and packages built on ZIP, such as .docx, .xlsx, .epub, or .apk
  • web fonts (font/woff, font/woff2)

Two kinds of media type are compressed anyway: raw formats such as image/bmp, image/tiff, and audio/wav, and subtypes with a structured syntax suffix such as image/svg+xml. Set an accurate content_type when uploading media to a key-value store:

import asyncio
from pathlib import Path

from apify_client import ApifyClientAsync

TOKEN = 'MY-APIFY-TOKEN'


async def main() -> None:
apify_client = ApifyClientAsync(TOKEN)
kvs_client = apify_client.key_value_store('MY-KVS-ID')

screenshot = await asyncio.to_thread(Path('screenshot.png').read_bytes)

# The explicit content type lets the client skip compressing the PNG.
await kvs_client.set_record('screenshot', screenshot, content_type='image/png')


if __name__ == '__main__':
asyncio.run(main())

Without an explicit content type, a bytes value is sent as application/octet-stream, which the client can't tell apart from uncompressed binary data and therefore still compresses. A streamed value, such as an open file or an iterator of chunks, is never compressed. For details, see Streaming uploads.

Pre-compressed bodies

A payload can reach the client already encoded, for example a gzipped file read from disk. Set the Content-Encoding header to name the encoding the payload carries. The client then sends the body as it is and forwards the header, so nothing gets compressed twice. set_record exposes the header as its content_encoding argument:

import asyncio
import gzip
from pathlib import Path

from apify_client import ApifyClientAsync

TOKEN = 'MY-APIFY-TOKEN'


async def main() -> None:
apify_client = ApifyClientAsync(TOKEN)
kvs_client = apify_client.key_value_store('MY-KVS-ID')

report = await asyncio.to_thread(Path('report.csv').read_bytes)
compressed_report = await asyncio.to_thread(gzip.compress, report)

# The explicit content encoding stops the client from compressing the bytes again.
await kvs_client.set_record(
'report',
compressed_report,
content_type='text/csv',
content_encoding='gzip',
)


if __name__ == '__main__':
asyncio.run(main())

The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as deflate. It's forwarded for a streamed value too, so an open gzipped file can be passed in place of its bytes. The API accepts gzip, br, zstd, deflate, and identity. Passing identity turns compression off for a single request without changing how the client is configured.

A value that can't be compressed at all - a string, an object serialized to JSON, or a file-like value opened in text mode - is rejected with a TypeError when content_encoding names a compression. Beyond that the client can't verify that the bytes match the header, so set Content-Encoding only when the payload really is encoded that way. Key-value store records are stored exactly as you upload them, which makes the header part of the stored record rather than a transport detail.

Configuration

To choose the compression algorithm, pass compression to the client constructor:

from apify_client import ApifyClient

# Default: gzip, no extra dependency required
client = ApifyClient(token='MY-APIFY-TOKEN')

# Opt in to brotli (requires apify-client[brotli])
client = ApifyClient(token='MY-APIFY-TOKEN', compression='brotli')

# Opt in to zstd (requires apify-client[zstd] before Python 3.14)
client = ApifyClient(token='MY-APIFY-TOKEN', compression='zstd')

Enabling brotli

Brotli is available as an optional extra. Install it alongside the client:

pip install "apify-client[brotli]"
# or
uv add "apify-client[brotli]"

Then pass compression='brotli' to the client constructor. If you request brotli without installing the extra, the client raises a clear ImportError. There is no silent fallback.

Enabling zstd

Python 3.14 and newer ship zstd in the standard library as compression.zstd, so compression='zstd' works there without an extra. On older Python versions, install the optional extra, which brings in the backports.zstd package:

pip install "apify-client[zstd]"
# or
uv add "apify-client[zstd]"

As with brotli, requesting zstd without the extra on Python older than 3.14, or on a Python 3.14 build without zstd support, raises a clear ImportError.

Custom quality and advanced control

For fine-grained control over compression quality, inject an HttpCompressor instance directly instead of a string literal:

from apify_client import ApifyClient
from apify_client.http_compressors import (
BrotliHttpCompressor,
GzipHttpCompressor,
ZstdHttpCompressor,
)

# Brotli at maximum quality
client = ApifyClient(token='MY-APIFY-TOKEN', compression=BrotliHttpCompressor(quality=11))

# Gzip at maximum quality
client = ApifyClient(token='MY-APIFY-TOKEN', compression=GzipHttpCompressor(quality=9))

# Zstd at the highest accepted level
client = ApifyClient(token='MY-APIFY-TOKEN', compression=ZstdHttpCompressor(quality=19))

You can also implement a fully custom compressor by subclassing HttpCompressor. The client calls it only for bodies that reach the minimum body size and aren't already compressed or pre-compressed by the caller:

from apify_client import ApifyClient
from apify_client.http_compressors import HttpCompressor


class IdentityCompressor(HttpCompressor):
content_encoding = 'identity'
"""Value sent in the `Content-Encoding` header."""

def compress(self, data: bytes) -> bytes:
"""Compress a request body.

Args:
data: The raw bytes to compress.

Returns:
The compressed bytes.
"""
return data


client = ApifyClient(token='MY-APIFY-TOKEN', compression=IdentityCompressor())

Comparison

BrotliGzipZstd
Compression ratioTypically better than gzipGoodClose to brotli at high levels
CPU costModerate, depends on qualityLowLow at the default level, higher at higher levels
AvailabilityRequires the brotli extraBuilt-in, no extra neededBuilt-in from Python 3.14, else the zstd extra
Content-Encoding headerbrgzipzstd
Quality range0–111–91–19
Default quality663
Enable via configcompression='brotli'compression='gzip' (default)compression='zstd'
Best forLarge payloads where bandwidth mattersMinimal-dependency environmentsFast compression at a good ratio
tip

For most workloads, the bandwidth savings from brotli outweigh the CPU costs. Install the brotli extra and pass compression='brotli' unless you can't install additional packages in your environment. On Python 3.14 and newer, when the Python build includes zstd, compression='zstd' is an alternative that needs no extra package.