Connect to datacenter proxies
Learn how to connect to Apify's datacenter proxies from your application with Node.js (axios and got), Python 2 and 3 and PHP using code examples.
This page contains code examples for connecting to datacenter proxies using Apify Proxy.
See the connection settings page for connection parameters.
Using the Apify SDK
If you are developing your own Apify actor using the Apify SDK, you can use Apify Proxy in:
- PuppeteerCrawler using the createProxyConfiguration() function.
- CheerioCrawler using the createProxyConfiguration() function.
- requestAsBrowser() function by specifying proxy configuration in the options.
- launchPuppeteer() by specifying the configuration in the function's options.
The Apify SDK's ProxyConfiguration enables you to choose which proxies you use for all connections. You can inspect the current proxy's URL and other attributes using the ProxyInfo property in your crawler's page function.
Rotate IP addresses
IP addresses for each request are selected at random from all available proxy servers.
const Apify = require("apify");
Apify.main(async () => {
const requestList = await Apify.openRequestList(
"my-list", ["http://proxy.apify.com/?format=json"]
);
const proxyConfiguration = await Apify.createProxyConfiguration();
const crawler = new Apify.PuppeteerCrawler({
requestList,
proxyConfiguration,
handlePageFunction: async ({ page, request, proxyInfo }) => {
console.log(await page.content())
},
});
await crawler.run();
});
Single IP address for multiple requests
Use a single IP address until it fails (gets retired).
The maxPoolSize: 1
configuration in PuppeteerCrawler means that a single IP will be used by all browsers until it fails. Then, all running browsers are retired, a new IP is selected and new browsers opened. The browsers all use the new IP.
const Apify = require("apify");
Apify.main(async () => {
const requestList = await Apify.openRequestList(
"my-list", [
"http://proxy.apify.com/?format=json",
"http://proxy.apify.com",
]
);
const proxyConfiguration = await Apify.createProxyConfiguration();
const crawler = new Apify.PuppeteerCrawler({
requestList,
proxyConfiguration,
useSessionPool: true,
sessionPoolOptions: {
sessionOptions: { maxPoolSize: 1 },
},
handlePageFunction: async ({ page, request, proxyInfo }) => {
console.log(await page.content());
},
});
await crawler.run();
});
How to use proxy groups
For simplicity, the examples above use the auto
proxy group, which selects IP addresses from all available groups.
To use IP addresses from specific proxy groups, add a groups
property
to createProxyConfiguration()
and specify the group names. For example:
const proxyConfiguration = await Apify.createProxyConfiguration({
groups: ['GROUP_NAME_1', 'GROUP_NAME_2'],
});
Using standard libraries and languages
You can find your proxy password on the Proxy page of the Apify app.
The
username
field is not your Apify username.
Instead, you specify proxy settings (e.g.groups-SHADER+BUYPROXIES94952
,session-123
).
Useauto
for default settings.
For examples using PHP, you need to have the cURL extension enabled in your PHP installation. See installation instructions for more information.
Examples in Python 2 use the six library. Run pip install six
to enable it.
Use IP rotation
For each request, a random IP address is chosen from all available proxy groups. You can use random IP addresses from proxy groups by specifying the group(s) in the username
parameter.
A random IP address will be used for each request.
const HttpsProxyAgent = require("https-proxy-agent");
const axios = require("axios");
const httpsAgent = new HttpsProxyAgent({
host: "proxy.apify.com",
port: "8000",
// Replace <YOUR_PROXY_PASSWORD> below with your password
// found at https://my.apify.com/proxy
auth: "auto:<YOUR_PROXY_PASSWORD>"
});
const axiosWithProxy = axios.create({ httpsAgent });
async function useProxy() {
const response = await axiosWithProxy.get("http://proxy.apify.com/?format=json");
console.log(response.data);
};
useProxy();
Multiple requests with the same IP address
The IP address in the example is chosen at random from all available proxy groups.
To use this option, set a session name in the username
parameter.
const HttpsProxyAgent = require("https-proxy-agent");
const axios = require("axios");
const httpsAgent = new HttpsProxyAgent({
host: "proxy.apify.com",
port: "8000",
// Replace <YOUR_PROXY_PASSWORD> below with your password
// found at https://my.apify.com/proxy
auth: "session-my_session:<YOUR_PROXY_PASSWORD>"
});
const axiosWithProxy = axios.create({ httpsAgent });
async function useProxy() {
const response = await axiosWithProxy.get("https://api.apify.com/v2/browser-info");
console.log(response.data);
};
useProxy();
// Should return the same clientIp as
useProxy();
groups-SHADER
Use a randomly allocated IP address for multiple requests:
session-new_job_123
Use the same IP address from the SHADER
and BUYPROXIES94952
groups for multiple requests:
groups-SHADER+BUYPROXIES94952,session-new_job_123
Set a session and select an IP from the BUYPROXIES94952
group geolocated in the USA:
groups-BUYPROXIES94952,session-new_job_123,country-US