Prometheus Query Workshop: Setup Requirements

This page is for the instructor or cluster administrator who prepares an OpenShift cluster for the Prometheus Query Workshop. It lists everything the exercises rely on, derived from the commands and manifests in the modules. It has two audiences: the cluster administrator, who prepares the cluster, the operators and the attendee accounts once before the workshop (sections Cluster to Network and Image Access), and the attendee, who only needs a workstation or terminal with the tools listed under Attendee Workstation. Modules 1 to 4 only need read access to the platform metrics; Modules 5 and 6 need additional cluster-wide preparation.

Cluster

  • OpenShift version: the content is written for and tested with OpenShift 4.16 (Prometheus 2.x); later 4.x releases work. From OpenShift 4.19, the Developer perspective of the web console is disabled by default. Modules 5 and 6 use it (Observe → Metrics and Alerts for a project, silencing alerts). Either enable it in the console customization settings (the Perspectives settings of the cluster Console configuration; see the OpenShift documentation of your release), or let attendees use the API alternatives that Module 5 describes.

  • Monitoring stack: the default platform monitoring stack in openshift-monitoring (Prometheus, Thanos Querier, Alertmanager). The exercises use:

    • the web console Observe → Metrics page (Modules 1 to 6)

    • the tenancy ports of the thanos-querier service: 9092 for queries (Modules 1, 4, 5, 6) and 9093 for alerts and rules (Module 5)

    • the tenancy port 9092 of the alertmanager-main service for silences (Module 5, optional step). Alerts of user projects must go to the platform Alertmanager, which is the default.

    • the prometheus-k8s route in openshift-monitoring, only in the optional cluster-administrator examples (Modules 1 and 4)

  • User workload monitoring: must be enabled for Modules 5 and 6 (see Enabling User Workload Monitoring).

  • Topology and size: there is no minimum number of worker nodes. Several queries in Modules 2 and 3 group by node (for example bottomk(3, …​) by (node) and "Are the nodes balanced?"); they are more instructive on a cluster with several nodes, but they also work on single-node OpenShift, where they return one series. Each attendee runs at most these pods in their project: prometheus-example-app (1 replica), api-client (ubi-minimal, idle) and one Grafana instance. Enabling user workload monitoring adds a prometheus-user-workload and a thanos-ruler-user-workload StatefulSet (two replicas each by default, one on single-node OpenShift) in openshift-user-workload-monitoring; plan cluster capacity for those plus one Grafana pod per attendee.

  • Ingress: attendees create routes (oc expose service in Module 5, the grafana-route in Module 6) and must be able to reach *.apps.<cluster-domain> from their terminal and browser.

Operators and Cluster Components

Component Install source Namespace Configuration needed Modules

Platform monitoring (Prometheus, Thanos Querier, Alertmanager)

Built into OpenShift

openshift-monitoring

None. Attendees need the cluster-monitoring-view cluster role (see Attendee Accounts, Projects and Roles).

1–6

User workload monitoring

Built into OpenShift, disabled by default

openshift-user-workload-monitoring

Set enableUserWorkload: true in the cluster-monitoring-config ConfigMap (see Enabling User Workload Monitoring).

5, 6

Grafana Operator (community, version 5)

OperatorHub, catalog community-operators, package grafana-operator, channel v5

openshift-operators (installation mode All namespaces on the cluster)

Install once for the whole cluster (see Installing the Grafana Operator). The community operator is not supported by Red Hat.

6

Enabling User Workload Monitoring

User workload monitoring is disabled by default. Enable it by setting enableUserWorkload: true in the cluster-monitoring-config ConfigMap in the openshift-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

If the ConfigMap already exists, add the enableUserWorkload: true line to its existing config.yaml key instead of replacing the ConfigMap:

oc -n openshift-monitoring edit configmap cluster-monitoring-config

After a few minutes, the prometheus-user-workload and thanos-ruler-user-workload pods run in the openshift-user-workload-monitoring namespace:

oc -n openshift-user-workload-monitoring get pods

User workload monitoring skips the namespaces of the platform (openshift-, kube-) and namespaces labeled openshift.io/user-monitoring=false. Make sure that the attendee projects do not carry that label.

Installing the Grafana Operator

Install the community Grafana Operator once for the whole cluster:

  1. In the Administrator perspective of the web console, navigate to Operators → OperatorHub.

  2. Search for Grafana Operator, select the community operator, and click Install.

  3. Select the v5 update channel and the All namespaces on the cluster installation mode, and click Install.

Alternatively, create a Subscription in the openshift-operators namespace:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: grafana-operator
  namespace: openshift-operators
spec:
  channel: v5
  name: grafana-operator
  source: community-operators
  sourceNamespace: openshift-marketplace

When the operator is installed, project administrators and editors can create its resources (Grafana, GrafanaDatasource, GrafanaDashboard in the grafana.integreatly.org/v1beta1 API group) in their projects. No further role bindings are needed.

Attendee Accounts, Projects and Roles

This repository contains no provisioning automation; the instructor creates the users and projects with the identity provider and tooling of the lab environment. Each attendee needs:

  • A user account that can log in with oc login using a user name and password or a token. The exercises run oc whoami -t to get an OAuth token, which does not work with a client-certificate kubeconfig (such as kubeadmin’s `kubeconfig file).

  • One workshop project of their own, in which they have the admin role (or at least edit). The content does not prescribe a project name; every exercise uses NS=$(oc project -q), so attendees only need to switch to their project. A simple convention is one project per user, named after the user, for example <user>-prometheus. Either let attendees run oc new-project (self-provisioning, enabled by default), or create the projects in advance:

    oc new-project <user>-prometheus
oc adm policy add-role-to-user admin <user> -n <user>-prometheus

  • The cluster-monitoring-view cluster role (Modules 1 to 4). It lets Observe → Metrics in the Administrator perspective show the metrics of every namespace, which the exercises query (for example openshift-monitoring and node metrics). The role allows reading metrics, alerts and rules, and listing the names of all namespaces (so oc get projects shows every project); it grants no access to other projects' resources. The tenancy API examples in Module 1 work with or without it.

    oc adm policy add-cluster-role-to-user cluster-monitoring-view <user>

  • The monitoring-edit role in their project (Modules 5 and 6). As project administrators, attendees can deploy applications, but creating ServiceMonitor and PodMonitor objects, creating PrometheusRule objects, reading the alerts and rules of the project through the tenancy port 9093 and creating silences require one of the monitoring roles:

    oc adm policy add-role-to-user monitoring-edit <user> \
  -n <user>-prometheus

    monitoring-rules-view and monitoring-rules-edit are narrower roles, but Module 5 creates a ServiceMonitor, which needs monitoring-edit.

  • No extra role for Module 6. Attendees create a service account grafana-thanos and bind the view cluster role to it in their own project, which the project admin role allows.

To prepare many users at once, loop over the user names, for example:

for u in user1 user2 user3; do
  oc new-project "$u-prometheus"
  oc adm policy add-role-to-user admin "$u" -n "$u-prometheus"
  oc adm policy add-role-to-user monitoring-edit "$u" \
    -n "$u-prometheus"
  oc adm policy add-cluster-role-to-user cluster-monitoring-view "$u"
done

Instructor-only (not needed by attendees, and attendees cannot do these):

  • Enabling user workload monitoring and installing the Grafana Operator.

  • The port-forward examples in Modules 1 and 2: regular users cannot port-forward into openshift-monitoring. The examples run:

    oc port-forward -n openshift-monitoring pod/prometheus-k8s-0 9090:9090

  • Querying all namespaces through the prometheus-k8s route works for users with cluster-monitoring-view; it is shown as an optional alternative in Modules 1 and 4.

  • oc get nodes (Module 2, Exercise "find all pods running on a specific node") needs permission to list nodes, which cluster-monitoring-view does not grant. Attendees without it can read the node names from the node label of kube_node_info instead.

Attendee Workstation

Attendees work in the OpenShift web console and in a shell that is logged in to the cluster. When the workshop is served together with the OpenShift 102 lab, the introduction page includes the shared CLI tools container of that lab, which provides a terminal with these tools.

Tool Version Used in

oc

Matching the cluster version; tested with the oc client of OpenShift 4.16

All modules: oc login, oc whoami -t, oc run, oc exec, oc apply, oc expose, oc patch, oc extract, oc auth can-i

curl

Any recent version

Module 5: generating traffic to the route of the sample application from the workstation. Optional in Modules 1 and 4 for the prometheus-k8s route. (The tenancy API calls use the curl inside the api-client pod, not the workstation’s.)

Web browser

A browser supported by the OpenShift web console

All modules: Observe → Metrics; Module 6: Grafana UI

bash, date, base64

GNU coreutils date syntax (date -d '+30 minutes'); the content gives the macOS alternative

Module 5 (silence through the Alertmanager API, optional), Module 6 (base64 -d to decode the service account token)

promtool

From a Prometheus release; optional

Module 5: checking rule files before applying them (optional)

No other CLIs (helm, podman, jq, istioctl) are used.

Network and Image Access

Container images that the exercises pull into attendee projects:

Image Used in

quay.io/brancz/prometheus-example-app:v0.5.0

Module 5 (Deployment of the sample application, kept running for Module 6)

registry.access.redhat.com/ubi9/ubi-minimal:latest

Modules 1, 4, 5, 6 (api-client pod that runs curl against the Thanos Querier and Alertmanager tenancy ports)

docker.io/grafana/grafana (tag chosen by the installed Grafana Operator version)

Module 6 (Grafana instance created by the operator)

Grafana Operator bundle and operator images (from the community-operators catalog)

Module 6 (installed by the cluster administrator)

External URLs used during the workshop:

  • https://prometheus.io/docs/introduction/overview/ (reading link on the introduction page; not needed to run the exercises).

  • The exercises themselves fetch nothing from the internet; all API calls go to cluster-internal services (thanos-querier.openshift-monitoring.svc, alertmanager-main.openshift-monitoring.svc) or to routes under *.apps.<cluster-domain>.

For disconnected or restricted-egress clusters:

  • Mirror quay.io/brancz/prometheus-example-app:v0.5.0 and registry.access.redhat.com/ubi9/ubi-minimal:latest, and either configure an ImageDigestMirrorSet/ImageTagMirrorSet for them or tell attendees to replace the image references in the manifests.

  • Mirror the community-operators catalog (at least the grafana-operator package, channel v5) with oc-mirror, together with the operator image and the docker.io/grafana/grafana image that the installed operator version deploys. The Grafana image tag depends on the operator version; find it by creating a test Grafana instance on a connected cluster and running this command in its project:

    oc get pods \
  -l app=grafana \
  -o jsonpath='{..image}'

  • Docker Hub rate limits can affect the docker.io/grafana/grafana pull when many attendees start Grafana at the same time; consider mirroring it even on connected clusters.

Verification Checklist

Instructor (as a cluster administrator):

  1. The cluster version is 4.16 or later:

    oc get clusterversion

  2. User workload monitoring is running; expect prometheus-user-workload- and thanos-ruler-user-workload- pods in Running state:

    oc -n openshift-user-workload-monitoring get pods

  3. The Grafana Operator is installed; expect a grafana-operator.v5.* CSV in phase Succeeded:

    oc get csv -n openshift-operators | grep grafana-operator

  4. Each attendee has the roles; expect yes for every command:

    oc auth can-i get prometheuses/k8s \
  --subresource=api \
  -n openshift-monitoring \
  --as <user>

oc auth can-i create servicemonitors.monitoring.coreos.com \
  -n <user>-prometheus \
  --as <user>

oc auth can-i create prometheusrules.monitoring.coreos.com \
  -n <user>-prometheus \
  --as <user>

oc auth can-i create grafanas.grafana.integreatly.org \
  -n <user>-prometheus \
  --as <user>

    The first command checks the cluster-monitoring-view role, which grants get on the api subresource of the prometheuses resource named k8s (a check without the name answers no). To see the binding directly, run:

    oc get clusterrolebinding -o wide | grep cluster-monitoring-view

    The output must list the user.

  5. The images can be pulled: run the api-client pod from Module 1 in a test project and check that it reaches Running.

Attendee (logged in with oc login, in their workshop project):

  1. These commands print a token and the workshop project:

    oc whoami -t
oc project -q

  2. In the web console, Administrator perspective, Observe → Metrics, this query returns a number (not an empty result or a permission error):

    count(kube_pod_info{namespace="openshift-monitoring"})

  3. These commands print yes:

    oc auth can-i create servicemonitors.monitoring.coreos.com
oc auth can-i create prometheusrules.monitoring.coreos.com
oc auth can-i create grafanas.grafana.integreatly.org