Skip to content

Use Blackbox-Exporter

Add a new target

ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: blackbox-exporter
    caascad.com/prometheus-monitor: caascad
  name: blackbox-exporter-OVERRIDEME # Exemple blackbox-exporter-Grafana
spec:
  endpoints:
  - interval: 60s
    params:
      module:
      - OVERRIDEME # see the available modules below : Example http_2xx
      target:
      - OVERRIDEME # target to check: Example https://grafana.ZONE_NAME.caascad.com
    path: /probe
    port: http
    relabelings:
    - sourceLabels:
      - __param_target
      - __param_module
      separator: /
      targetLabel: target
    - sourceLabels:
      - __param_target
      - __param_module
      separator: /
      targetLabel: instance
    - sourceLabels:
      - __param_module
      targetLabel: module
    - targetLabel: "cc_prom_source"
      replacement: "cloud-caascad"
    metricRelabelings:
     - regex: 'pod'
       action: labeldrop
    scheme: http
    scrapeTimeout: 60s
  jobLabel: blackbox-exporter
  namespaceSelector:
    matchNames:
    - caascad-monitoring
  selector:
    matchLabels:
      app.kubernetes.io/instance: blackbox-exporter
      app.kubernetes.io/name: prometheus-blackbox-exporter

Caution

The .spec.endpoints[].params.target[] is an array. However you can set only one URL there. Any other URL will be ignored.

Available modules

A module is a list of checks that will be performed on a target.

Modules already activated in Blackbox-Exporter:

  • icmp_ckeck

    Allows to test the ping on a service.

  • smtp_check

    Allows to test an smtp server by making a list of commands and checking the response. 

    - expect: "^220 ([^ ]+) ESMTP (.+)$"
- send: "EHLO ProberCheck\r"
- expect: "^250-STARTTLS"
- send: "STARTTLS\r"
- expect: "^220"
- starttls: true
- send: "EHLO ProberCheck\r"
- expect: "^250-AUTH"
- send: "QUIT\r"

  • http_2xx

    Send an http get request on the service and verify that the return code is 200.

  • http_2xx_ssl

    Send an http get request on the service and verify that the return code is 200 and ssl is present.

  • dns_udp

    Use udp protocol and checks the A field.

    The resolved_name is karma.<cluster_name>.caascad.com.

  • dns_tcp

    Use tcp protocol.

    The resolved_name is karma.<cluster_name>.caascad.com.

  • dns_soa

    Checks the SOA field.

    The resolved_name is karma.<cluster_name>.caascad.com.

Note

If you need to use other modules, contact Caascad support.

The full configuration is available here: 

kubectl get cm -n caascad-monitoring -o yaml caascad-blackbox-exporter

See the different possibilities with Blackbox-Exporter: Configuration

Deploy the ServiceMonitor

Just deploy your ServiceMonitor with kubectl (as usual).

Prometheus-operator runs an automatic discovery of ServiceMonitors and PodMonitors and will detect it soon.

Then it will get the informations it needs from the ServiceMonitor or PodMonitor.

It will get more information from the Service or Pod you selected with your monitor definition.

And it will regenerate the configuration of Prometheus. All of this is automatic.

Troubleshooting

Verify Prometheus Target

You can use a port-forward on Prometheus: 

kubectl -n caascad-monitoring port-forward svc/caascad-prometheus 9090:9090 &

Go to Status/Targets in order to verify if the endpoint target is up (search servicemonitor name), and see the error message if the target is down.

Ensure you check on the Grafana

Go to https://grafana.ZONE_NAME.caascad.com/

Select the Blackbox-Exporter dashboard and verify that your target is present and up.

Target is down

Go to Explore tab in Grafana, choose Loki datasource.

Retrieve the blackbox-exporter logs by filtering on the module and the target.

Example with module=https://karma.ocb-demo.caascad.com and target=http_2xx : {app_kubernetes_io_instance="blackbox-exporter"}|="https://karma.ocb-demo.caascad.com"|="http_2xx"

Back to top