1.1 Tasks: Setup

Task 1.1.1: Getting Started

In Bearbeitung.

Task 1.1.2: Install Prometheus

Mit ArgoCD

In Bearbeitung.

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <user>-app-of-apps
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: 'https://gitea.training.cluster.acend.ch/<user>/prometheus-training-lab-setup'
    path: apps/
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - Replace=true

kubectl argocd create -f user-app-of-apps.yaml

Deploy Prometheus

Wir verwenden diese Helm-Chart: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md

helm install -f kube-prom-stack-values.yaml -n monitoring-system prometheus-stack oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack

# kube-prom-stack-values.yaml
prometheusOperator:
  enabled: true
prometheus:
  enabled: true
  ingress:
    enabled: true
    # ingressClassName: "traefik"
    hosts:
      - prometheus.app.lernvirt.ch

thanosRuler:
  enabled: true
  ingress:
    enabled: true
    # ingressClassName: "traefik"
    hosts:
      - thanosruler.app.lernvirt.ch
grafana:
  enabled: true
  ingress:
    enabled: true
    # ingressClassName: "traefik"
    hosts:
      - grafana.app.lernvirt.ch

alertmanager:
  enabled: true
  ingress:
    enabled: true
    # ingressClassName: "traefik"
    hosts:
      - alertmanager.app.lernvirt.ch

Configure Prometheus

As mentioned in the introduction, configuring a Prometheus on Kubernetes can be done using the Prometheus Operator. We basically need to specify a Prometheus custom resource and the Operator will do its work. In the prometheus custom resource’s spec block you can find various configuration options:

spec:
  enableAdminAPI: true
  evaluationInterval: 30s
  externalLabels:
    monitoring: <user>
  podMonitorNamespaceSelector:
    matchLabels:
      user: <user>
  podMonitorSelector: {}
  portName: web
  probeNamespaceSelector:
    matchLabels:
      user: <user>
  probeSelector: {}
  resources:
    requests:
      memory: 400Mi
  scrapeInterval: 60s
  serviceAccountName: prometheus-<user>
  serviceMonitorNamespaceSelector:
    matchLabels:
      user: <user>
  serviceMonitorSelector: {}

So far we do not have configured anything special yet. We instruct prometheus to automatically discover Probes, PodMonitors and ServiceMonitors in all namespaces with the label user matching your username.

Note

We will learn more about other configuration options (evaluation_interval) later in this training.

Check Prometheus

Is your prometheus running? Use your browser to navigate to https://prometheus.app.lernvirt.ch . You should now see the Prometheus web UI.

Task 1.1.3: Deploy example application

In the next section we are going to deploy our own application and try to monitor their metrics. As stated earlier the folder user-demo is automatically watched and synchronized by ArgoCD from your git repository. Deploy the Acend example Python application, which provides application metrics at /metrics by creating the following file (user-demo/deployment.yaml) in your repo:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: example-web-python
  name: example-web-python
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-web-python
  template:
    metadata:
      labels:
        app: example-web-python
    spec:
      containers:
      - image: quay.io/acend/example-web-python
        name: example-web-python
        resources:
          requests:
            memory: "32Mi"
            cpu: "10m"
          limits:
            memory: "128Mi"
            cpu: "100m"

We can simply deploy the application by pushing the resource into our git repository in the user-demo folder:

git add .
git commit -m "Deploy Demo App"
git push

Use the following command to verify whether pod example-web-python is Ready and Running in your <user> namespace. (use CTRL C to exit the command) Or you can also check in the ArgoCD UI.

kubectl -n $USER get pod -w

To access the deployed resources we also need to create a Service for the new application. Create a file (user-demo/service.yaml) with the following content:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: example-web-python
    prometheus-monitoring: 'true'
  name: example-web-python
spec:
  ports:
    - name: http
      port: 5000
      protocol: TCP
      targetPort: 5000
  selector:
    app: example-web-python
  type: ClusterIP

We can add and push the resource to our git repository to have it synchronized to our namespace:

git add .
git commit -m "Deploy Demo Service"
git push

This created a so-called Kubernetes Service which allows communication via an internal network abstraction.

kubectl -n $USER get services

Task 1.1.4: Create a ServiceMonitor

So far we did not have any interaction with Prometheus at all. Kubernetes-based Prometheus installation use PodMonitors and ServiceMonitors as a service-discovery mechanism. PodMonitors and ServiceMonitors select via labels pods / services for prometheus to scrape at a given endpoint. Check whether the application metrics are actually exposed by opening a shell within the container and curling the metrics endpoint.

# get the pod name
kubectl -n $USER get pod

# exec curl within the pod
kubectl -n $USER exec -it <pod-name> -- curl http://localhost:5000/metrics

Should result in something like:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 541.0
python_gc_objects_collected_total{generation="1"} 344.0
python_gc_objects_collected_total{generation="2"} 15.0
...

Since our newly deployed application now exposes metrics, the next thing we need to do, is to tell our Prometheus server to scrape metrics from the Kubernetes deployment. In a highly dynamic environment like Kubernetes this is done with so called Service Discovery.

Let us create a ServiceMonitor for the example application, which will configure Prometheus to scrape metrics from the example-web-python application every 30 seconds.

For this to work, you need to ensure:

The example-web-python Service is labeled correctly and matches the labels you’ve defined in your ServiceMonitor.
The port name in your ServiceMonitor configuration matches the port name in the Service definition.
- hint: check with kubectl -n $USER get service example-web-python -o yaml
Verify the target in the Prometheus user interface.

Create the following ServiceMonitor user-demo/servicemonitor.yaml in your git repository:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: example-web-python
  name: example-web-python-monitor
spec:
  endpoints:
    - interval: 60s
      port: http
      scheme: http
      path: /metrics
  selector:
    matchLabels:
      prometheus-monitoring: 'true'

Then commit and push the resource again for ArgoCD to synchronize.

git add .
git commit -m "Add ServiceMonitor"
git push

In the Prometheus UI we can check our targets to be scaped. So far no target should appear from your demo application and ServiceMonitor we just deployed. This is happening because we tell Prometheus to look for ServiceMonitors in our <user> namespace, but it is not yet allowed by the Kubernetes API to see resources in this namespace. Therefore we need to add RBAC rules to let Prometheus see and scrape these resources.

We do this again by creating a Role and RoleBinding for the Prometheus’ ServiceAccount. Create the following two files in your git repository:

user-demo/role.yaml Replace the <user> placeholder with your user:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-<user>
rules:
- apiGroups: [""]
  resources:
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]

user-demo/rolebinding.yaml Replace the <user> placeholder with your user:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-<user>
subjects:
- kind: ServiceAccount
  name: prometheus-<user>
  namespace: <user>-monitoring

Then again add and push these files to your git repository and let ArgoCD synchronize them for you.

git add .
git commit -m "Add role and rolebinding"
git push

Note

This part usually is done on a cluster level, but is needed for our current Lab setup.

Verify that the target gets scraped in the Prometheus user interface. Target name: serviceMonitor/<user>/example-web-python-monitor/0 (1/1 up) (it may take up to a minute for Prometheus to load the new configuration and scrape the metrics).

Task 1.1.5: Your first PromQL Query

Switch back to the Graph Tab in the Prometheus UI and enter the following query into the input field:

python_info

hit enter and explore the result should look similar to

python_info{endpoint="http", implementation="CPython", instance="10.244.18.95:5000", job="example-web-python", major="3", minor="11", namespace="<user>", patchlevel="5", pod="example-web-python-7c8b9984d4-z9b92", service="example-web-python", version="3.11.5"}  1