Skip to main content

Examples

Connect to Google SERP proxies

Learn how to connect to Google SERP proxies from your applications with Node.js (axios and got-scraping), Python 2 and 3 and PHP using code examples.

This page contains code examples for connecting to Google SERP proxies using Apify Proxy.

See the connection settings page for connection parameters.

Using the Apify SDK​

If you are developing your own Apify actor using the Apify SDK and Crawlee, the most efficient way to use Google SERP proxy is CheerioCrawler. This is because Google SERP proxy only returns a page's HTML. Alternatively, you can use the got-scraping NPM package by specifying proxy URL in the options.

Apify Proxy also works with PuppeteerCrawler, launchPuppeteer(), PlaywrightCrawler, launchPlaywright() and JSDOMCrawler. However, CheerioCrawler is simply the most efficient solution for this use case.

Get a list of search results​

Get a list of search results for the keyword wikipedia from the USA (google.com).

import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';

await Actor.init();

const proxyConfiguration = await Actor.createProxyConfiguration({
    groups: ['GOOGLE_SERP'],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ body }) {
        // ...
        console.log(body)
    },
});

await crawler.run(['http://www.google.com/search?q=wikipedia']);

await Actor.exit();

Get a list of shopping results​

Get a list of shopping results for the query Apple iPhone XS 64GB from Great Britain (google.co.uk).

import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';

await Actor.init();

const proxyConfiguration = await Actor.createProxyConfiguration({
    groups: ['GOOGLE_SERP'],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ body }) {
        // ...
        console.log(body)
    },
});

const query = encodeURI('Apple iPhone XS 64GB');
await crawler.run([`http://www.google.co.uk/search?q=${query}&tbm=shop`]);

await Actor.exit();

Using standard libraries and languages​

You can find your proxy password on the Proxy page of the Apify Console.

The username field is not your Apify username.
Instead, you specify proxy settings (e.g. groups-GOOGLE_SERP).
Use groups-GOOGLE_SERP to use proxies from all available countries.

For examples using PHP, you need to have the cURL extension enabled in your PHP installation. See installation instructions for more information.

Examples in Python 2 use the six library. Run pip install six to enable it.

HTML from search results​

Get the HTML of search results for the keyword wikipedia from the USA (google.com).

Select this option by setting the username parameter to groups-GOOGLE_SERP. Add the item you want to search to the query parameter.

import axios from 'axios';

const proxy = {
    protocol: 'http',
    host: 'proxy.apify.com',
    port: 8000,
    // Replace <YOUR_PROXY_PASSWORD> below with your password
    // found at https://console.apify.com/proxy
    auth: { username: 'groups-GOOGLE_SERP', password: <YOUR_PROXY_PASSWORD> },
};

const url = 'http://www.google.com/search';
const params = { q: 'wikipedia' };

const { data } = await axios.get(url, { proxy, params });

console.log(data);

HTML from localized shopping results​

Get HTML of shopping results for the query Apple iPhone XS 64GB from Great Britain (google.co.uk).

Select this option by setting the username parameter to groups-GOOGLE_SERP. In the query parameter, add the item you want to search and specify the shop page as a URL parameter.

Set the domain (your country of choice) in the URL (in the response variable).

import axios from 'axios';

const proxy = {
    protocol: 'http',
    host: 'proxy.apify.com',
    port: 8000,
    // Replace <YOUR_PROXY_PASSWORD> below with your password
    // found at https://console.apify.com/proxy
    auth: { username: 'groups-GOOGLE_SERP', password: <YOUR_PROXY_PASSWORD> },
};

const url = 'http://www.google.co.uk/search';
const params = { q: 'Apple iPhone XS 64GB', tbm: 'shop' }

const { data } = await axios.get(url, { proxy, params });

console.log(data);