In Statful, alerts are condition-based notifications triggered by the analysis of data ingested by Statful.
An alert may have one or more conditions that will be verified, and if all or any of these match an established threshold, a notification is triggered. To set up an alert, define:
- Source: metric and tags
- Transformation: mean/sum/count/max/min
- Condition: below/above or any/no data received
- Threshold: value
- Duration: time window
An alert condition can be read as: “The mean value of the metric
system.gauge.memory with tag/values
type:used, app:DB and
environment:production has gone
below 100 for
- Conditions are evaluated as a group of one-dimensional logical operations.
- Conditions are based on a simplified query - group by or baseline queries are not supported.
The process for the analysis of triggering conditions to activate or deactivate an alert (ON or OFF) is based on the concept of a rolling temporal window of date coming into Statful. We provide a detailed explanation of how this is achieved both for Transformation and Any/No Data conditions below.
- Received data points are stored in buckets on a per alert basis.
- Data points are pulled from buckets and processed on a rolling window of real-time (based on the alert configuration trigger duration) in which every X seconds all data points received during the period are processed and a comparison with the specified threshold is performed.
- On each processing iteration, data points that fall outside of the defined window of time are removed from the bucket and not considered for processing purposes.
- A processing iteration is only performed if there are at least two data points that represent an interval higher than the defined duration in order to guarantee that a minimum data sample is received before processing.
Any/No Data conditions:
- When receiving a data point, the current timestamp is stored in a bucket on a per alert basis.
- It follows the same rolling window of time approach as the transformation conditions in which the timestamp of the last data point received is checked and compared with the current timestamp - validating if any or no data was received as per defined condition.
- A processing iteration begins the moment the alert is configured.
Alert events, generated by user actions or automatically by data processing, are persisted and associated with the respective alert in order to provide a historical view of the alert.
The possible alert states are:
- OFF - In this status the alert is CLOSED or OFF, being ready for activation when trigger conditions are verified.
- ON - In this status the alert is OPEN or ON, being activated when trigger conditions were verified.
- ACK - In this status the alert has been acknowledged by a user and can be considered as silent for any future notifications.
Depending on the current state of the alert, an alert event may or may not generate an alert notification. Situations that trigger a user notification are:
- OFF to ON -The alert was OPEN after receiving an ON event.
- ON to OFF - The alert was CLOSED after receiving an OFF event.
- ON to ACK -The alert was ACKNOWLEDGE after receiving an ACK event.
- ACK to OFF - The alert was CLOSED after receiving an OFF event.
- ACK to ON - The previous acknowledgment of the alert was canceled after receiving a UNACK event that resets the status to ON.
Notification profiles are used to send notifications via e-mail, Slack or Pargerduty. An alert can have multiple notification profiles associated. A notification profile can be used on different alerts.