search docs
Edit on GitHub

Request queue

Documentation of Request queues, which allow you to queue URLs for your crawler to visit.

Request queues enable you to enqueue and retrieve requests such as URLs with an HTTP method and other parameters. They are useful not only in web crawling but anywhere you need to process a high number of URLs and enqueue new links.

Request queue storage supports both breadth-first and depth-first crawling orders, as well as custom data attributes. It allows you to query whether specific URLs were already found, push new URLs to the queue and fetch the next URLs to process.

Named request queues are retained indefinitely.
Unnamed request queues expire after 7 days unless otherwise specified.
Learn about named and unnamed queues.

Basic usage

There are four ways to access your request queues:

Apify app

In the Apify app, you can view your request queues in the Storage section under the Request queues tab.

Only named request queues are displayed by default. Select the Include unnamed request queues checkbox to display all of your queues.

Request queues in app

To view a request queue, click on its Queue ID. In the detail page, under the Settings tab, you can update the queue's name (and, in turn, its retention period) and access rights. The API tab allows you to view and test a queue's API endpoints.

Request queues detail

Apify SDK

If you are building an Apify actor, you will be using the Apify software development kit (SDK). In the Apify SDK, the request queue is represented by the RequestQueue class.

You can use the RequestQueue class to specify whether your data is stored locally or in the Apify cloud, enqueue new URLs (see example), and manage your existing queues.

Each actor run is associated with the default request queue, which is created for the actor run when the first request is added to it. Typically, it is used to store URLs to crawl in the specific actor run, however its usage is optional.

You can also create named queues which can be shared between actors or between actor runs.

If you are storing your data locally, you can find your request queue at the following location.

{APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/{STATE}/{NUMBER}.json

The default request queue's ID is default. Each request in the queue is stored as a separate JSON file, where {STATE} is either handled or pending, and {NUMBER} is an integer indicating the request's position in the queue.

To open a request queue, use the Apify.openRequestQueue() method.

// Import the Apify SDK into your project
const Apify = require("apify");

// The optional Apify.main() function performs the
// actor's job and terminates the process when it is finished
Apify.main(async () => {

    // Open the default request queue associated with
    // the actor run
    const queue = await Apify.openRequestQueue();

    // Open the "my-queue" request queue
    const queueWithName = await Apify.openRequestQueue("my-queue");
});

Once a queue is open, you can manage it using the following methods. For a full list of methods, see the RequestQueue class's API reference.

// Enqueue requests
await queue.addRequest({ url: "http://example.com/aaa" });
await queue.addRequest(
    { url: "http://example.com/foo/bar" },
    { forefront: true }
);

// Get the next request from queue
const request1 = await queue.fetchNextRequest();
const request2 = await queue.fetchNextRequest();

// Get a specific request
const specificRequest = await queue.getRequest("shi6Nh3bfs3");

// Reclaim a failed request back to the queue
// and crawl it again
await queue.reclaimRequest(request2);

// Remove a queue
await queue.drop();

For more information on managing your request queues with the Apify SDK, see the SDK documentation and the RequestQueue class's API reference.

JavaScript API client

Apify's JavaScript API client (apify-client) allows you to access your request queues from any Node.js application, whether it is running on the Apify platform or elsewhere.

For help with setting up the client, see the JavaScript API client section on the overview page.

After importing the apify-client package into your application and creating an instance of it, save it to a variable for easier access.

// Save your request queues to a variable for easier access
const requestQueues = apifyClient.requestQueues;

You can then get or create new queues, retrieve existing requests or enqueue new URLs. For a full list of options, see the JavaScript API client's documentation.

// Get the "my-queue" request queue and set it as the default
// to be used in the following commands
const queue = await requestQueues.getOrCreateQueue({
    queueName: "my-queue"
});
apifyClient.setOptions({ queueId: queue.id });

// Add a request to the default queue
await requestQueues.addRequest({ 
    url: "http://example.com",
    uniqueKey: "http://example.com"
});

await requestQueues.addRequest({
    url: 'http://example.com',
    uniqueKey: 'http://example.com'
});
await requestQueues.addRequest({
    url: 'http://example.com/a/b',
    uniqueKey: 'http://example.com/a/b'
});

// Mark request as handled.
request1.handledAt = new Date();
await requestQueues.updateRequest(request1);

For more information on managing your request queues using the JavaScript API client, see its documentation.

Apify API

The Apify API allows you to access your request queues programmatically using HTTP requests.

If you are accessing your request queues using the username~store-name store ID format, you will need to append your secret API token as a query parameter (see below). You can find the token (and your user ID) on the Integrations page of your Apify account.

To get a list of your request queues, send a GET request to the Get list of request queues endpoint, providing your secret API token as a query parameter.

https://api.apify.com/v2/request-queues?token={YOUR_API_TOKEN}

To get information about a request queue such as its creation time and item count, send a GET request to the Get request queue endpoint.

https://api.apify.com/v2/request-queues/{QUEUE_ID}?token={YOUR_API_TOKEN}

To get a request from a queue, send a GET request to the Get request endpoint.

https://api.apify.com/v2/request-queues/{QUEUE_ID}/requests/{REQUEST_ID}?token={YOUR_API_TOKEN}

To add a request to a queue, send a POST request with the request to be added as a JSON object in the request's payload to the Add request endpoint.

https://api.apify.com/v2/request-queues/{QUEUE_ID}/requests?token={YOUR_API_TOKEN}

Example payload:

{
    "uniqueKey": "http://example.com",
    "url": "http://example.com",
    "method": "GET"
}

To update a request in a queue, send a PUT request with the request to update as a JSON object in the request's payload to the Update request endpoint. In the payload, specify the request's ID and add the information you want to update.

https://api.apify.com/v2/request-queues/{QUEUE_ID}/requests/{REQUEST_ID}?token={YOUR_API_TOKEN}

Example payload:

{
    "id": "dnjkDMKLmdlkmlkmld",
    "uniqueKey": "http://example.com",
    "url": "http://example.com",
    "method": "GET"
}

When adding or updating requests, you can optionally provide a clientKey parameter to your request. It must be a string between 1 and 32 characters in length. This identifier is used to determine whether the queue was accessed by multiple clients. If clientKey is not provided, the system considers this API call to come from a new client. For details, see the hadMultipleClients field returned by the Get head operation.
Example: client-abc

For a detailed breakdown of each API endpoint, see the API documentation.

Sharing request queues between runs

You can access a request queue from any actor or task run as long as you know its name or ID.

To access a request queue from another run using the Apify SDK, open it using the Apify.openRequestQueue([queueIdOrName]) method like you would any other queue.

const otherQueue = await Apify.openRequestQueue("old-queue");

To access a request queue using the JavaScript API client, use the getOrCreateQueue() method.

const otherQueue = await requestQueues.getOrCreateQueue({
    queueName: "my-queue"
});

Once you've opened the request queue, you can use it in your crawl or add new requests like you would with a queue from your current run.

The same applies for the Apify API - you can use the same endpoints as you would normally.

For more information on sharing storages between runs, see the Storage overview page.

Limits

While multiple actor or task runs can add new requests to a queue concurrently, only one run can process a queue at any one time.