A step-by-step monitoring tutorial that shows you how to monitor multiple runs, validate your results and visualize them using the monitoring dashboard.
This example walks you through setting up monitoring for a multiple tasks, validating data and setting up a data monitoring dashboard. Though the tutorial focuses on tasks, you can also use it to monitor actors.
You want to monitor multiple actors or tasks at once.
- To validate the default dataset (check structure, item count).
- Notification of run failure.
- Statistics presented on a dashboard.
In this scenario we'll imagine you want to scrape COVID-19 data for several countries: Brazil, Germany, the USA and Singapore. You have created a task from each of the actors tracking those countries.
If you haven't already, add the monitoring suite to your account.
If you have already added the task, under its Settings tab, give it a name. For example,
We recommend prefixing your monitoring task names with
monitoring-so you could identify them easier.
Next, we will configure the monitoring suite.
Under your task's Input tab, set the Mode dropdown to Create configuration.
Next, open the What you want to monitor section. Give the monitoring suite a name in the Monitoring suite name field, e.g.
In the Type of target: dropdown, select Task, since you will be monitoring actor tasks.
Target name patterns should be
^covid-. The task names follow a simple naming convention (all start with covid-), so this name pattern will target all of the above tasks. To select only particular tasks, add separate Target name patterns for each:
Select the Notify me whenever actor/task does not succeed option to receive a report when a run finishes unsuccessfully.
Each of your monitoring suites must have a unique name.
Now let's ensure that your data have the correct format.
We have used four actors from Apify Store. Each actor will return a differently structured dataset, so we need to validate them using a different schema. To do this, you can use the tasks' IDs or filter the them by their names.
Open the Validating by a schema section and select the Enable schema validation option.
In the Validation options field, create a separate object for each task containing
filter, specify the task it is for (e.g.
schema, set an object specifying the format of each of the properties you want to validate.
It is best to set Validation frequency to
Per run, so the data is validated right after it is collected.
The monitoring suite uses the ow library for type validation. Make sure to import the library using
/* global ow */.
To add the monitoring dashboard, open the Statistics dashboard section and enable this feature.