6.1 Tasks: Instrumenting

Task 6.1.1: Get the source code

git clone https://github.com/acend/prometheus-training-go-instrumentation.git

Change into the freshly cloned git repository

cd prometheus-training-go-instrumentation

TODO: create Dockerfile This application creates a simple webserver which listens at http://localhost:8083 and returns the string Prometheus Training after a random interval between 0 and 1000 milliseconds. You can test the app by building it as a docker image using docker build -t prometheus-training-go-instrumentation:local . and then running the built image using docker run --rm -p 8083:8083 prometheus-training-go-instrumentation:local. You can then (in a separate terminal) issue a request using the command curl http://localhost:8083.

Task 6.1.2: Instrument the Go example app

The Prometheus Go client library provides an easy way to add prometheus metrics to our own applications. In this task we will extend a simple Go application to expose these metrics.

Task description:

Look at the described steps and try to understand them. As you are missing the needed tooling on the webshell, you will not be able to execute them yourself.
You will be provided docker commands so you can build and test the added functionality

In order to collect the metrics and use the promhttp.Handler function we first need to download the prometheus/promhttp library:

go get github.com/prometheus/client_golang/prometheus/promhttp

Next we replace the line // insert prometheus library here in the main.go file with the following line:

    "github.com/prometheus/client_golang/prometheus/promhttp"

Finally we need to add a handler for the /metrics endpoint. We replace the line // insert additional handlers here in the main.go file with the following line:

    http.Handle("/metrics", promhttp.Handler())

The main.go of our example app now looks like this:

package main

import (
    "fmt"
    "net/http"
    "math/rand"
    "time"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    http.HandleFunc("/", ServeHandler)
    http.Handle("/metrics", promhttp.Handler())
    err := http.ListenAndServe(":8083", nil)
    if err != nil {
        fmt.Println(err)
    }
}

func ServeHandler(w http.ResponseWriter, r *http.Request) {
    rand.Seed(time.Now().UnixNano())
    n := rand.Intn(1000)
    time.Sleep(time.Duration(n)*time.Millisecond)
    fmt.Fprintf(w, "Prometheus Training")
}

You can now rebuild and rerun the application. For this, switch into the solution folder and run docker build -t prometheus-training-go-instrumentation:local . and docker run --rm -p 8083:8083 prometheus-training-go-instrumentation:local.

Using a second terminal window we can now query the /metrics endpoint of our app and should receive a number of Prometheus metrics

curl localhost:8083/metrics

Expected result:

curl localhost:8083/metrics
# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
# TYPE go_gc_cycles_automatic_gc_cycles_total counter
go_gc_cycles_automatic_gc_cycles_total 0
# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
# TYPE go_gc_cycles_forced_gc_cycles_total counter
go_gc_cycles_forced_gc_cycles_total 0
# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.
# TYPE go_gc_cycles_total_gc_cycles_total counter
go_gc_cycles_total_gc_cycles_total 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
...

Task 6.1.3: Scrape config

TODO: prep image and create deployment + SM here

Task description:

Configure Prometheus to scrape the metrics endpoint of the Go Application

Hints

Add a new Job to the scrape_configs in your /etc/prometheus/prometheus.yml:

scrape_configs:
  ...
  - job_name: "go_app"
    metrics_path: "/metrics"
    static_configs:
      - targets:
          - "localhost:8083"
  ...

Note that you can always validate the configuration with:

promtool check config /etc/prometheus/prometheus.yml

Do not forget to reload or restart the Prometheus server.

Task 6.1.4: Metric names

Study the following metrics and decide if the metric name is ok

http_requests{handler="/", status="200"}

http_request_200_count{handler="/"}

go_memstats_heap_inuse_megabytes{instance="localhost:9090",job="prometheus"}

prometheus_build_info{branch="HEAD",goversion="go1.19.1",instance="localhost:9090",job="prometheus",revision="6d7f26c46ff70286944991f95d791dff03174eea",version="2.39.0"}

prometheus_config_last_reload_success_timestamp{instance="localhost:9090",job="prometheus"}

prometheus_tsdb_lowest_timestamp_minutes{instance="localhost:9090",job="prometheus"}

Hints

The _total suffix should be appended, so http_requests_total{handler="/", status="200"} is better.
There are two issues in http_request_200_count{handler="/"}: The _count suffix is foreseen for histograms, counters can be suffixed with _total. Second, status information should not be part of the metric name, a label {status="200"} is the better option.
The base unit is bytes not megabytes, so go_memstats_heap_inuse_bytes is correct.
Everything is ok with prometheus_build_info and its labels. It’s a good practice to export such base information with a gauge.
In prometheus_config_last_reload_success_timestamp, the base unit is missing, correct is prometheus_config_last_reload_success_timestamp_seconds.
The base unit is seconds for timestamps, so prometheus_tsdb_lowest_timestamp_seconds is correct.

Task 6.1.5: Metric names (optional)

What kind of risk do you have, when you see such a metric

http_requests_total{path="/etc/passwd", status="404"} 1

Hints

There is no potential security vulnerability from exposing the /etc/passwd path, which seems to be handled appropriately in this case: no password is revealed.

From a Prometheus point of view, however, there is the risk of a DDoS attack: An attacker could easily make requests to paths which obviously don’t exist. As every request and therefore path is registered with a label, many new time series are created which could lead to a cardinality explosion and finally to out-of-memory errors.

It’s hard to recover from that!

For this case, it’s better just to count the 404 requests and to lookup the paths in the log files.

http_requests_total{status="404"} 15

Task 6.1.6: Custom metric (optional)

TODO: prepare custom metric lab (if wanted)

Task description:

Create a custom histogram metric with the name training_app_request_duration_seconds which measures the time (per request) used by the ServeHandler handler
Make a couple of requests to the example application
Calculate the 85th percentile of the request duration of the example app

Hints

Download the following two prometheus libraries and add them to the imports in the main.go file:

github.com/prometheus/client_golang/prometheus/promauto
github.com/prometheus/client_golang/prometheus

go get github.com/prometheus/client_golang/prometheus/promauto \
  github.com/prometheus/client_golang/prometheus

Add the imports to main.go (only the relevant section is shown in the example below)

...
import (
    ...
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus"
...

Add a new variable for the histogram metric to the main.go file (only the relevant section is shown in the example below):

...
var (
    responseLatency = promauto.NewHistogram(prometheus.HistogramOpts{
        Name: "training_app_request_duration_seconds",
        Help: "example app response latencies",
    })

)

func main() {
...

Extend the existing ServeHandler function to measure the time between the start end the end of the function and update the histogram accordingly:

main.go (only the relevant section is shown in the example below)

...
func ServeHandler(w http.ResponseWriter, r *http.Request) {
    requestReceived := time.Now()
    rand.Seed(time.Now().UnixNano())
    n := rand.Intn(1000)
    time.Sleep(time.Duration(n)*time.Millisecond)
    fmt.Fprintf(w, "Prometheus Training")
    responseLatency.Observe(time.Since(requestReceived).Seconds())
}

By now your main.go file should look like this:

package main

import (
    "fmt"
    "net/http"
    "math/rand"
    "time"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus"
)

var (
    responseLatency = promauto.NewHistogram(prometheus.HistogramOpts{
        Name: "training_app_request_duration_seconds",
        Help: "example app response latencies",
    })

)

func main() {
    http.HandleFunc("/", ServeHandler)
    http.Handle("/metrics", promhttp.Handler())
    err := http.ListenAndServe(":8083", nil)
    if err != nil {
        fmt.Println(err)
    }
}

func ServeHandler(w http.ResponseWriter, r *http.Request) {
    requestReceived := time.Now()
    rand.Seed(time.Now().UnixNano())
    n := rand.Intn(1000)
    time.Sleep(time.Duration(n)*time.Millisecond)
    fmt.Fprintf(w, "Prometheus Training")
    responseLatency.Observe(time.Since(requestReceived).Seconds())
}

Next we have to build and run the app again:

go run main.go

Now let’s create a couple of requests to our new endpoint. Make sure to run those commands from a second terminal window, while the Go application is still running.

for i in {1..100}; do (curl http://localhost:8083/ -o /dev/null -s &) done

Verify the Prometheus metrics endpoint and look for a metric with the name training_app_request_duration_seconds_bucket

curl http://localhost:8083/metrics

Expected result:

...
# HELP training_app_request_duration_seconds example app response latencies
# TYPE training_app_request_duration_seconds histogram
training_app_request_duration_seconds_bucket{le="0.005"} 0
training_app_request_duration_seconds_bucket{le="0.01"} 1
training_app_request_duration_seconds_bucket{le="0.025"} 1
training_app_request_duration_seconds_bucket{le="0.05"} 4
training_app_request_duration_seconds_bucket{le="0.1"} 10
training_app_request_duration_seconds_bucket{le="0.25"} 24
training_app_request_duration_seconds_bucket{le="0.5"} 55
training_app_request_duration_seconds_bucket{le="1"} 100
training_app_request_duration_seconds_bucket{le="2.5"} 100
training_app_request_duration_seconds_bucket{le="5"} 100
training_app_request_duration_seconds_bucket{le="10"} 100
training_app_request_duration_seconds_bucket{le="+Inf"} 100
training_app_request_duration_seconds_sum 48.73099980000001
training_app_request_duration_seconds_count 100
...

Let us now calculate the 85th percentile by navigating to the Prometheus Web Console by entering the following query in the Prometheus expression browser:

histogram_quantile(0.85, training_app_request_duration_seconds_bucket)

The result describes the number of seconds it took to process 85% of all requests (15% of the requests were slower than the calculated result).