7. Alerting with Alertmanager on IT-CNE Cloud-Native Core

7.1 Tasks: Enable and configure Alertmanager

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.1.1: Install Alertmanager and Thanosruler

Update your monitoring application (charts/user-monitoring/values.yaml) and update the alertmanager.enabled and ruler.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
  enabled: true
# thanos-query
query:
  enabled: true
# grafana
grafana:
  enabled: true
  datasources:
  - name: prometheus
    access: proxy
    editable: false
    type: prometheus
    url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
  enabled: true
# pushgateway
pushgateway:
  enabled: true
# alertmanager
alertmanager:
  enabled: true
# thanos-ruler
ruler:
  enabled: true

Commit and push the changes.

Verify the installation and sync process in the ArgoCD UI. Or execute the following command:

7.2 Tasks: Alertmanager

Mon, 01 Jan 0001 00:00:00 +0000

Task 7.2.1: Configure Slack as alert receiver

As part of our lab setup we already configured an alert receiver for Alertmanager alerts.

We deployed the Mail catcher, which is a very simple component. We can send Emails to the server, and those are then displayed in the Web UI. The Emails are not send anywhere, and this setup should only be used for demo purposes.

When we enabled the alertmanager configuration in the charts/user-monitoring/values.yaml we also deployed the following AlertmanagerConfig Custom Resource

7.3 Tasks: Alertrules and alerts

Mon, 01 Jan 0001 00:00:00 +0000

Note

For doing the alerting lab it’s useful to have a “real” application so that alerts can be provoked. You will use the demo app installed in your monitoring-demo namespace for this purpose.

The example app exposes metrics which are already scraped from lab one.

{job="example-web-python"}

The Prometheus Operator allows you to configure Alerting Rules (PrometheusRules). This enables Kubernetes users to configure and maintain alerting rules for their projects. Furthermore it is possible to treat Alerting Rules like any other Kubernetes resource and lets you manage them in Helm or Kustomize. A PrometheusRule has the following form: