Skip to main content
Version: 2.3

RequestOptions

Specifies required and optional fields for constructing a Request.

Properties​

url​

Type: string

URL of the web page to crawl. It must be a non-empty string.

uniqueKey​

Type: string

A unique key identifying the request. Two requests with the same uniqueKey are considered as pointing to the same URL.

If uniqueKey is not provided, then it is automatically generated by normalizing the URL. For example, the URL of HTTP://www.EXAMPLE.com/something/ will produce the uniqueKey of http://www.example.com/something.

The keepUrlFragment option determines whether URL hash fragment is included in the uniqueKey or not.

The useExtendedUniqueKey options determines whether method and payload are included in the uniqueKey, producing a uniqueKey in the following format: METHOD(payloadHash):normalizedUrl. This is useful when requests point to the same URL, but with different methods and payloads. For example: form submits.

Pass an arbitrary non-empty text value to the uniqueKey property to override the default behavior and specify which URLs shall be considered equal.

method​

Type: string = "'GET'"

payload​

Type: string | Buffer

HTTP request payload, e.g. for POST requests.

headers​

Type: Object<string, string>

HTTP headers in the following format:

{
    Accept: 'text/html',
    'Content-Type': 'application/json'
}

userData​

Type: Object<string, *>

Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries, failures etc.

keepUrlFragment​

Type: boolean = false

If false then the hash part of a URL is removed when computing the uniqueKey property. For example, this causes the http://www.example.com#foo and http://www.example.com#bar URLs to have the same uniqueKey of http://www.example.com and thus the URLs are considered equal. Note that this option only has an effect if uniqueKey is not set.

useExtendedUniqueKey​

Type: boolean = false

If true then the uniqueKey is computed not only from the URL, but also from the method and payload properties. This is useful when making requests to the same URL that are differentiated by method or payload, such as form submit navigations in browsers.