Specifies required and optional fields for constructing a
URL of the web page to crawl. It must be a non-empty string.
A unique key identifying the request. Two requests with the same
uniqueKey are considered as pointing to the same URL.
uniqueKey is not provided, then it is automatically generated by normalizing the URL. For example, the URL of
will produce the
keepUrlFragment option determines whether URL hash fragment is included in the
uniqueKey or not.
useExtendedUniqueKey options determines whether method and payload are included in the
uniqueKey, producing a
uniqueKey in the following
METHOD(payloadHash):normalizedUrl. This is useful when requests point to the same URL, but with different methods and payloads. For example:
Pass an arbitrary non-empty text value to the
uniqueKey property to override the default behavior and specify which URLs shall be considered equal.
HTTP request payload, e.g. for POST requests.
HTTP headers in the following format:
Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries, failures etc.
false then the hash part of a URL is removed when computing the
uniqueKey property. For example, this causes the
http://www.example.com#bar URLs to have the same
http://www.example.com and thus the URLs are considered equal. Note that this
option only has an effect if
uniqueKey is not set.
true then the
uniqueKey is computed not only from the URL, but also from the method and payload properties. This is useful when making requests
to the same URL that are differentiated by method or payload, such as form submit navigations in browsers.