Learn how to check the performance of your actors, tasks and storages. Validate your data and receive notifications when your jobs succeed or fail.
Monitoring allows you to track and observe how software works. It enables you to measure and compare your programs' performance over time, and to be notified when something goes wrong.
You can use data you gain from monitoring to optimize your software and maximize its potential.
In essence, monitoring gives you more time. It allows you to automate processes that are complicated or tedious if you have to do them manually.
If you are scraping your competitor's website every day, you have already automated a part of the process. With monitoring, you can save even more time and ensure all your results are usable even before you need them.
And if the scraping job fails, you will receive a notification, so there will be no unpleasant surprises.
The monitoring suite is made up of a system of actors: checkers and reporters. Based on your configuration, the suite will create schedules and webhooks under your account and use those to run these actors.
As the names suggest, the checkers make sure your data is up to standard. Reporters create visualizations of how your actors perform over time and send notifications about failed runs.
You can use the monitoring suite free of charge, but under the hood, it is a system of actors that consume compute units (CUs). We have tried our best to make it as efficient as possible.
- Medium projects (single monitoring configuration with 5-20 daily running monitored tasks or actors) should consume around 10 CUs per month.
- Bigger projects (more monitoring configurations with more than 20 daily running tasks or actors) can exceed 50 CUs+ per month.
We currently have video tutorials for setting up monitoring and adding data validation.
We also have written tutorials for the following use cases.
To turn off a monitoring task, simply run it with the Mode (in the task's Input section) set to Delete configuration. It will clean up all its resources and turn off all monitoring activity. You will no longer receive notifications.
To make any changes to an existing configuration, choose the Update configuration mode in the task's Input, make your changes and run the monitoring suite task again.
The suite will send notifications to the email address associated with your account by default.
You can disable email notifications or change the email address in a task Input's Email notifications section.
Below is an example of an email notification for a task run that did not return enough results.
To see what went wrong, click on the time next to Actor run to open your task's Runs dashboard. When you click on Details, you'll find a detailed monitoring report in JSON format.