Alerts
A Sentinet alert is an event triggered on certain conditions, for example when an SLA is violated or a registered X.509 certificate is about to expire or an OAuth Client Credential is about to expire.
SLA Violation Alerts
An SLA violation can be configured with one or more alerts. Select Service Agreement you are interested to create SLA violations for, and click on the ALERTS tab.
To add a new alert, click the + Add button, to modify an existing alert, click the [...] button.
Each alert is configured with the following properties:
| Property | Description |
|---|---|
| Name | Alert-friendly name |
| Minimum Alert Generation Period | Each alert event of that type is generated not more often as specified by the value of this property. For example, some alerts may be generated not more than every 10 minutes, while others may be generated not more than once a day. |
| Enabled | Enables or disables alert generation. |
| Alert Actions | An alert can be configured with one or more alert actions. For example, an alert event can trigger sending emails and logging error messages in the Windows Event viewer. Custom alert actions can be added to add custom alert processing. |
| Minimum Execution Period | The period of time in which the defined actions are performed, in the examples shown in the figures on this page, the violation alert (above) is generated every minute, but the corresponding “Send Email” event (next figure below) only occurs every 24 hours. |
Alert Actions
Each alert can be configured with one or more independent alert actions, that will execute once an event is triggered. Sentinet provides two built-in alert actions, Send Email(s) and Write to Windows Event Log. Custom alert actions can be registered with Sentinet and configured with alerts.
Every alert action has two common properties -- Minimum Execution Period and Enabled flag. An alert action can be executed upon alert triggering, but no more than every Minimum Execution Period value. For example, some alert actions can be instructed to execute at minimum once a day (for example: Send Email), while others may execute every minute (for example: Write to Windows Event Log). The Minimum Alert Generation Period property of the alert is different from the Minimum Execution Period of the alert action (see example above). Minimum Alert Generation Period may instruct to generate SLA violation events every minute, while Minimum Execution Period may cause Send Email alert action to act upon accumulated events every 1 hour (in this example, it might be impractical to receive emails every minute). The Enabled flag allows stopping and resuming of alert action execution (for example, the Send Email action can be temporarily or permanently disabled from sending emails).
Click one of the alert actions management buttons to add a new or modify an existing alert action.
Select one of the built-in alert actions or a custom alert action from the drop-down list. Before a custom alert action can be selected, it must be registered with Sentinet (see Custom Alert Actions chapter).
Send Email(s) Alert Action
The Send Email(s) alert action automatically delivers emails to all Sentinet users that have specific access to the Repository folder in which the SLA is created. This alert action can be additionally configured with specific email addresses.
The Send one email per address field specifies whether a single email will be sent to all recipients or each email is sent separately.
The Send one email per alert field specifies whether one email may contain multiple alert events or one email will be sent per event occurrence.
The Subject and the Body fields specify text for the email subject and the email body. The Body field may include built-in Alert Data Tokens (in curly brackets) that will be replaced with the actual alert data.
Write to Windows Event Log Alert Action
The Write to Windows Event Log alert action logs custom Windows Events. Third party management tools can be further configured to monitor these specific Windows Events and custom scripts execution.
The Write to Windows Event Log alert is configured with standard Windows Event properties described in https://technet.microsoft.com/en-us/library/cc765981.
The Message Template field may include built-in Alert Data Tokens (in curly brackets) that will be replaced with the actual alert data.
Alert Data Tokens
Sentinet offers a number of built-in Alert Data Tokens which are automatically replaced in the Body field of the Send Email(s) alerts action, and in the Message Template of the Write to Windows Event Log alert action.
Common alert data tokens available for all Sentinet Alerts are:
{Id} - alert database identifier
{TypeId} - alert type database identifier
{TypeName} - alert type friendly name
{Subtype} - alert subtype
{SourceTypeId} - alert source type database identifier
{SourceTypeName} - alert source type friendly name
{Description} - alert description
{DateTime} - alert generation time in the reporting time zone
{DateTimeUtc} - alert generation time in UTC
{Data} - alert custom (source-specific) properties serialized to XML
Service Agreement specific alert data tokens are:
{Data.ServiceAgreementId} - database identifier of the violated service
agreement
{Data.ServiceAgreementName} - friendly name of the violated service
agreement
{Data.ViolationStartDateTimeLocal} - the most recent violation's date
and time reported in the service agreement reporting time zone
{Data.ServiceAgreementTimeZone} - service agreement time zone identifier
{Data.ViolationMetricId} - identifier of the metric that triggered the violation
{Data.ViolationMetricType} - name of the metric that triggered the violation
{Data.ViolationMetricValue} - metric value read when violation was detected
Custom Alert Actions
Before an alert can be configured with a custom alert action, they must be registered with the Sentinet alerting system (see Sentinet Extensibility chapter for generic description of the Sentinet extensibility features). Navigate to Repository root tree item, first select CONFIGURATION, then EXTENSIBILITY tab and click the + Add above the Custom Alert Actions table.
Provide a friendly name for the custom alert action, the assembly that contains the code for the custom alert action implementation and type (.NET class) that implements Sentinet's Alerting API interface. Optional default configuration for the Custom Alert Action (see the Sentinet CustomAlertHandler sample for more details on how to develop custom alert actions).
Once registered, this alert action can be selected from the list of the registered custom alert actions.
The custom alert action component may contain its own (optional) configuration, which can be in free-form text format or in XML as designed by the custom alert action developer.
Custom alert processing is a very powerful Sentinet extensibility feature; it allows developers and administrators to extend the Sentinet Alerting system with active and proactive systems recovery, reliability and high-availability capabilities. For example, an SLA is configured with certain metrics to trigger custom alert processing and when these metrics are violated, can automatically start additional cloud instances to support increased system load.
Expiration Alerts
To configure expiration alerts supported by Sentinet, select Repository tree root element and navigate to the CONFIGURATION->ALERTS tab.
Sentinet can be configured with expiration alerts for 3 groups of monitored entities:
Configured X.509 Certificates
Monitored X.509 Certificates
OAuth Client Credentials
These three groups are described in details below in this chapter.
Configured X.509 Certificates Alerts
X.509 certificates have expiration causing their invalidation, which may immediately break communication between consumer and service applications. Sentinet provides support for monitoring and alerting on certificates that are about to expire to avoid an operational outage. Certificates that are explicitly configured anywhere in the Sentinet, are automatically monitored. By default, Sentinet configures one alert with two alert actions. The alert mechanics are the same as described in the Alerts chapter in this guide. The Certificates Expiration Alert has two alert-specific properties, Check active certificates only and Alert when certificate expires in less than days:
Check active certificates only flag. By default, this flag is set to Yes, which means certificates used with inactive services will not be monitored and alerted on.
Alert when certificate expires in less than. This field, measured in days (by default is 30), instructs Sentinet to start generating certificate expiration alerts for expired certificates, and for those that will expire in less than the specified number of days.
Sentinet administrators can add new certificate expiration alerts and alert actions, as well as modify or delete an existing default alert and its alert actions.
Monitored X.509 Certificates Alerts
Sentinet automatically monitors explicitly configured X.509 certificates, such as Node Identities, Shared Identities, certificates in Access Rules, Outbound Client Identities and Trusted Name Issuers.
Sentinet allows to add any other certificate(s) for monitoring purposes that are not explicitly configured with Sentinet. For example, Nodes' SSL certificates are not necessarily monitored because they may be configured outside of the Sentinet context (for example on SSL-terminated external load-balancers), while they are important part of the Sentinet infrastructure. You can add any number of "external" certificates for additional monitoring and expiration alerting. Click +Add button to add new certificate and to upload its public part from a file (Name field is automatically populated, but can be changed; Description field is optional).
Note
In addition to sending alerts for certificates which have expired or about to expire, Sentinet will also highlight expiration in its UI in bold font as shown on Figure above.
OAuth Client Credentials Expiration
When OAuth Client Credentials are required for virtual services' outbound endpoint(s), the credentials are usually issued by a third-part OAuth Provider infrastructure, and they may be issued with some expiration. To help monitor and be alerted on these expirations, Sentinet users may add expiration date and time to these credentials when they are configured with Sentinet. From that point all OAuth Client Credentials configured with Sentinet will be monitored and alerted on their expiration. The expiration alert configuration has two alert-specific properties, Check active credentials only and Alert when credentials expire in less than days:
Check active credentials only flag. By default, this flag is set to Yes, which means OAuth Client Credentials used with inactive services will not be monitored and alerted on.
Alert when credentials expire in less than. This field, measured in days (by default is 30), instructs Sentinet to start generating OAuth Client Credentials expiration alerts for expired credentials, and for those that will expire in less than the specified number of days.
Alerting System
The Sentinet Alerting System is executed by the Sentinet Agent Windows service configured to execute each scheduled task on its own schedule and based on its own pattern. The relationship between the tasks relevant to the Sentinet Alerting System, alerts and alert action properties is shown in the diagram below.
Agent Task Frequency is configured in the agent's Nevatech.Vsb.Agent.exe.config file, while all other time intervals are configured using the Sentinet Administration Console described earlier in this chapter. Sentinet provides great flexibility in configuring alerts and alert actions and requires a certain level of responsibility of Sentinet users to assign relative time intervals that make sense.