A step-by-step monitoring tutorial that shows you how to ensure your data is correctly formatted and unique. Visualize your data using the monitoring dashboard.
You want regularly scrape product data using a single scraper.
- Data to always be in the correct format.
- Alerts if items are duplicated.
- Notification when your scheduled run times out or fails.
- Data visualization on a simple dashboard.
If you haven't already, add the monitoring suite to your account.
If you have already added the task, under its Settings tab, give it a name. For example,
monitoring-iphone-offers, since we're monitoring the
We recommend prefixing your monitoring task names with
monitoring-so you could identify them easier.
Under your task's Input tab, set the Mode dropdown to Create configuration.
Next, open the What you want to monitor section. Give the monitoring suite a name in the Monitoring suite name field, e.g.
In the Type of target: dropdown, select Task, since you will be monitoring an Amazon Crawler task.
Target name patterns should be the name of your task,
Select the Notify me whenever actor/task does not succeed option to receive a report when a run finishes unsuccessfully.
Each of your monitoring suites must have a unique name.
This is what the configuration should look like:
Let's say you need each item to always have properties such as
currency and a list of
Open the Validating by a schema section and select the Enable schema validation option.
In the Validation options field, create an object containing a
schemakey. As its value, set an object specifying the format of each of the properties you want to validate.
The monitoring suite uses the ow library for type validation. Make sure to import the library using
/* global ow */.
Validation is done after the each run finishes.
In the Check for duplicates section, select the Enable duplicate items check option.
Set the Unique keys field to
asinto make sure all the ASIN properties are unique.
In the Statistics dashboard section, check the Enable dashboard option to activate data visualization.
Finally, click the Save & Run button. It will create a monitoring configuration and turn the monitoring ON.
Following each of your
amazon-iphone-offers tasks runs, the suite will process your results and report if any of the checks fail. You receive an email with a link to your monitoring project dashboard.
This is what your dashboard can look like after some time: