Monitor Application With Prometheus

This page explains how to collect and view Prometheus metrics exposed by a sample application on APPUiO Cloud and how to write alerts using them.

To get notified about firing alerts you need to setup an alert receiver.

See Configure Alert Receivers for more information.

Deploy a Sample Application

  1. Create a new namespace for the sample application

    ZONE="cloudscale-lpg-2"(1)
    APP_NAMESPACE="monitoring-example" (2)
    kubectl create namespace ${APP_NAMESPACE}
    1 The APPUiO Cloud Zone to deploy to
    2 The namespace for the application. Please choose a unique name.
  2. Check that the namespace was created successfully.

    kubectl get namespace ${APP_NAMESPACE}
  3. Install the sample application

    kubectl -n "${APP_NAMESPACE}" apply -f - <<YAML
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: prometheus-example-app
      name: prometheus-example-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prometheus-example-app
      template:
        metadata:
          labels:
            app: prometheus-example-app
        spec:
          containers:
          - image: ghcr.io/rhobs/prometheus-example-app:0.4.1
            imagePullPolicy: IfNotPresent
            name: prometheus-example-app
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
              runAsNonRoot: true
              seccompProfile:
                type: "RuntimeDefault"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: prometheus-example-app
      name: prometheus-example-app
    spec:
      ports:
      - port: 8080
        protocol: TCP
        targetPort: 8080
        name: web
      selector:
        app: prometheus-example-app
      type: ClusterIP
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-production
      name: prometheus-example-app
    spec:
      rules:
      - host: ${APP_NAMESPACE}.apps.$ZONE.appuio.cloud
        http:
          paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: prometheus-example-app
                port:
                  number: 8080
      tls:
      - hosts:
        - ${APP_NAMESPACE}.apps.$ZONE.appuio.cloud
        secretName: prometheus-example-app-tls
    YAML
  4. Check that the application is running

    kubectl -n "${APP_NAMESPACE}" get pod
  5. Check that the route was created

    kubectl -n "${APP_NAMESPACE}" get route -w
  6. After a few seconds, you should be able to access the sample deployment

    curl https://$APP_NAMESPACE.apps.$ZONE.appuio.cloud

    You should also be able to manually access the exported metrics

    curl https://$APP_NAMESPACE.apps.$ZONE.appuio.cloud/metrics

Collect Metrics

  1. Configure a service monitor

    kubectl -n "${APP_NAMESPACE}" apply -f - <<YAML
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      labels:
        k8s-app: prometheus-example-monitor
      name: prometheus-example-monitor
    spec:
      endpoints:
      - interval: 30s
        port: web (1)
        scheme: http
      selector:
        matchLabels:
          app: prometheus-example-app (2)
    YAML
    1 The port on which the application provides the metrics endpoint, defined in the Service resource
    2 A selector that points to the service of the application
    Consult the ServiceMonitor API for more configuration options.
  2. View the collected metrics

    Select Custom Query and enter rate(http_requests_total[1m]). You should see a graph showing the request rate to the example application.

    Collected metrics on https://console.<CLUSTER>.cloud/dev-monitoring/ns/<APP_NAMESPACE>/metrics

Create an Alert

  1. Create the alert rule

    kubectl -n "${APP_NAMESPACE}" apply -f - <<YAML
    apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      name: prometheus-example-alert
    spec:
      groups:
      - name: prometheus-example-alert
        rules:
        - alert: TooManyrequests
          expr: rate(http_requests_total[1m]) > 1 (1)
          labels:
            severity: info
          annotations:
            summary: "There are a lot of requests to the application"
    YAML
    1 The alert rule will fire if the application receives more than 60 requests in one minute.
    Look at the upstream documentation on writing your own alerting rules.
  2. Verify that the alert was picked up by Prometheus

    List of alerts https://console.<CLUSTER>.cloud/dev-monitoring/ns/<APP_NAMESPACE>/alerts

    The alert should be displayed in the list of alerts.

    It may take a few minutes for the alert to show up. If it doesn’t show up, that most likely means that the PrometheusRule CRD isn’t well-formed and might have failed to apply, or that the PromQL expression isn’t valid.

    Check if the PrometheusRule CRD was created successfully.

    kubectl -n "${APP_NAMESPACE}" get prometheusrules

    If this seems correct, check that the PromQL expression is valid by entering it in the UI.

  3. Check that the alert will fire by generating requests

    for i in {1..70}
    do
      curl https://$APP_NAMESPACE.apps.$ZONE.appuio.cloud
    done

    The alert should now be firing.

    Firing alert https://console.<CLUSTER>.cloud/dev-monitoring/ns/<APP_NAMESPACE>/alerts