Usage
Learn how to configure and use Apify Proxy. See the required parameters such as the correct username and password.
Connection settings
To connect to the Apify Proxy, you use the HTTP proxy protocol. This means that you need to configure your HTTP client to use the proxy server at proxy.apify.com:8000 and provide it with your Apify Proxy password and the other parameters described below.
The full connection string has the following format:
http://<username>:<password>@proxy.apify.com:8000
| Parameter | Value / explanation |
|---|---|
| Proxy type | HTTP |
| Hostname | proxy.apify.com, alternatively you can use static IP addresses 18.208.102.16, 35.171.134.41. |
| Port | 8000 |
| Username | Specifies the proxy parameters such as groups, session and location. See username parameters below for details. Note: this is not your Apify username. |
| Password | Proxy password. Your password is displayed on the Proxy page in Apify Console. In Apify actors, it is passed as the APIFY_PROXY_PASSWORD environment variable. See the environment variables docs for more details. |
WARNING: All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks – the password is sent unencrypted due to the HTTP protocol's limitations.
Username parameters
The username field enables you to pass parameters like groups, session ID and country for your proxy connection.
For example, if you're using datacenter proxies and want to use the new_job_123 session using the SHADER group, the username will be:
groups-SHADER,session-new_job_123
The table below describes the available parameters.
| Parameter | Type | Description |
groups | Required | Set proxied requests to use servers from the selected groups: - groups-[group name] or auto when using datacenter proxies.- groups-RESIDENTIAL when using residential proxies.- groups-GOOGLE_SERP when using Google SERP proxies. |
|---|---|---|
session | Optional | If specified to The session string can only contain numbers (0-9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~). The maximum length is 50 characters. Session management may work differently for residential and SERP proxies. Check relevant documentations for more details. |
country | Optional | If specified, all proxied requests will use proxy servers from a selected country. Note that if there are no proxy servers from the specified country, the connection will fail. For example groups-SHADER,country-US uses proxies from the SHADER group located in the USA. By default, the proxy uses all available proxy servers from all countries. |
If you want to specify one parameter and not the others, just provide that parameter and omit the others. To use the default behavior (not specifying either groups, session, or country), set the username to auto. auto serves as a placeholder because the username can't be empty.
Code examples
We have code examples for connecting to our proxy using the Apify SDK (JavaScript and Python) and Crawlee and other libraries, as well as examples in PHP.
For code examples related to proxy management in Apify SDK and Crawlee, see:
IP address rotation
Web scrapers can rotate the IP addresses they use to access websites. They assign each request a different IP address, which makes it appear like they are all coming from different users. This greatly enhances performance and data throughout.
Depending on whether you use a browser or HTTP requests for your scraping jobs, IP address rotation works differently.
- Browser – a different IP address is used for each browser.
- HTTP request – a different IP address is used for each request.
Use sessions to controll how you rotate and persist IP addresses. See our guide Anti-scraping techniques to learn more about IP address rotation and our findings on how blocking works.
Sessions
Sessions allow you to use the same IP address for multiple connections. In cases where you need to keep the same session (e.g. when you need to log in to a website), it is best to keep the same proxy and so the IP address. On the other hand by switching the IP address, you can avoid being blocked by the website.
To set a new session, pass the session parameter in your username field when connecting to a proxy. This will serve as the session's ID and an IP address will be assigned to it. To use that IP address in other requests, pass that same session ID in the username field.
We recommend you to use SessionPool abstraction when managing sessions. The created session will then store information such as cookies and can be used to generate browser fingerprints. You can also assign custom user data such as authorization tokens and specific headers.
Sessions are available for datacenter and residential proxies. For datacenter proxies, a session persists for 26 hours (more info). For residential proxies, it persists for 1 minute (more info) but you can prolong the lifetime by regularly using the sessinon. Google SERP proxies do not support sessions.
Proxy groups
You can see which proxy groups you have access to on the Proxy page in the Apify Console. To use a specific proxy group (or multiple groups), specify it in the username parameter.
Troubleshooting
To view your connection status to Apify Proxy, open the URL below in the browser using the proxy. http://proxy.apify.com/. If the proxy connection is working, the page should look something like this:
To test that your requests are proxied and IP addresses are being rotated correctly, open the following API endpoint via the proxy. It shows information about the client IP address.
https://api.apify.com/v2/browser-info/
A different approach to 502 Bad Gateway
There are times when the 502 status code is not comprehensive enough. Therefore, we have modified our server with 590-599 codes instead to provide more insight:
590 Non Successful: upstream responded with non-200 status code.591 RESERVED: this status code is reserved for further use.592 Status Code Out Of Range: upstream responded with status code different than 100-999.593 Not Found: DNS lookup failed -EAI_NODATAorEAI_NONAME.594 Connection Refused: upstream refused connection.595 Connection Reset: connection reset due to loss of connection or timeout.596 Broken Pipe: trying to write on a closed socket.597 Auth Failed: incorrect upstream credentials.598 RESERVED: this status code is reserved for further use.599 Upstream Error: generic upstream error.
The typical issues behind these codes are:
590and592indicate an issue on the upstream side.593indicates an incorrectproxy-chainconfiguration.594,595and596may occur due to connection loss.597indicates incorrect upstream credentials.599is a generic error, where the above is not applicable.