Crawl all links on a website

This example uses the enqueueLinks() method to add new links to the RequestQueue as the crawler navigates from page to page. If only the required parameters are defined, all links will be crawled.

Cheerio Crawler

Using CheerioCrawler:

import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';

await Actor.init();

const crawler = new CheerioCrawler({
    async requestHandler({ request, enqueueLinks }) {
        console.log(request.url);
        // Add all links from page to RequestQueue
        await enqueueLinks();
    },
    maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
});

// Run the crawler
await crawler.run(['https://apify.com/']);

await Actor.exit();

Puppeteer Crawler

Using PuppeteerCrawler:

tip

To run this example on the Apify Platform, select the apify/actor-node-puppeteer-chrome image for your Dockerfile.

import { Actor } from 'apify';
import { PuppeteerCrawler } from 'crawlee';

await Actor.init();

const crawler = new PuppeteerCrawler({
    async requestHandler({ request, enqueueLinks }) {
        console.log(request.url);
        // Add all links from page to RequestQueue
        await enqueueLinks();
    },
    maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
});

// Run the crawler
await crawler.run(['https://apify.com/']);

await Actor.exit();

Playwright Crawler

Using PlaywrightCrawler:

tip

To run this example on the Apify Platform, select the apify/actor-node-playwright-chrome image for your Dockerfile.

import { Actor } from 'apify';
import { PlaywrightCrawler } from 'crawlee';

await Actor.init();

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, enqueueLinks }) {
        console.log(request.url);
        // Add all links from page to RequestQueue
        await enqueueLinks();
    },
    maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
});

// Run the crawler
await crawler.run(['https://apify.com/']);

await Actor.exit();