Skip to content

Configuration⚓︎

How the BB is configured - with examples etc.

How to put health checks into different namespaces⚓︎

In the reference deployment, all health checks are in resource-health namespace, together with all the other parts of Resource Health BB, such as Check API, Telemetry API, OpenSearch, OpenTelemetry (OTel) Collector, etc.
It could be useful to put health checks into separate namespaces, for example, to have a separate namespace for each user (or user group). You might also wish to run the health checks in a separate (virtual) cluster. The steps below will focus on the use case of putting health checks into per-user namespaces, with some notes on how to run them in a different (virtual) cluster. It should not be difficult to adapt this to other setups.

How to put the checks into per-user namespaces:

  1. Create a namespace for each user

  2. Implement get_k8s_namespace hook to get the namespace based on userid (see Hooks Tutorial)

    Info

    In case you want the checks to run on another cluster, you should also implement get_k8s_config to return an appropriate K8s config.

  3. Now we need to ensure that the health checks can still communicate with the other parts of Resource Health. That means sending the telemetry to OTel Collector, and querying the previous telemetry from OpenSearch. We need to use Fully Qualified Domain Name (FQDN) for both OTel Collector and OpenSearch endpoint. Specifically, use FQDN for DEFAULT_COLLECTOR_URL_NO_PROTOCOL and DEFAULT_PROXY_REMOTE_DOMAIN here. Alternatively, use FQDN for collector_url_no_protocol and proxy_remote_domain parameters to simple_runner_template function for each check template here. So collector_url_no_protocol should be resource-health-opentelemetry-collector.<collector-namespace>.<internal-domain>:4317, for example resource-health-opentelemetry-collector.resource-health.svc.cluster.local:4317 for the reference deployment. proxy_remote_domain should be https://opensearch-cluster-master-headless.<opensearch-namespace>.<internal-domain>:9200.

    Info

    In case you want the checks to run on another cluster, you should ensure that sending telemetry to OTel Collector and querying telemetry from OpenSearch is still possible. For OTel Collector, we would recommend to deploy one in the same cluster as the health checks, based on the OTel Collector in the reference deployment here. The OTel Collector could just forward everything to the main one (if such exists). For OpenSearch, you need to ensure that query endpoint is accessible, for example by exposing it to the outside.

Auth⚓︎

OpenID Connect⚓︎

Resource Health BB is designed to work with any OpenID Provider. To integrate with you OpenID Provider you just need to point OpenSearch, Health Check API, and Telemetry API to it.
Specifically, you need to:

  • Configure OpenSearch to use your OpenID Provider by following this. See an example here.
  • Set OPEN_ID_CONNECT_URL and OPEN_ID_CONNECT_AUDIENCE environment variables here for Health Check Api, and here for Telemetry API.

Alternative Auth Schemes⚓︎

To use some other auth scheme (such as basic HTTP auth), you need to:

  • Configure OpenSearch authentication, see here.
  • Implement Health Check API hooks get_fastapi_security and on_auth based on examples here.
  • Implement Telemetry API hooks get_fastapi_security, on_auth, and get_opensearch_config based on the example here.
  • For health checks to be able to access telemetry using alternative authentication methods:
    • Create a new Docker image which launches an appropriately configured mitmproxy based on this.
    • Update the health check template for accessing telemetry to use the new image based on this and this.

Resource Health APIs⚓︎

Resource Health APIs authentication and authorization is configured through hooks, see Hooks Tutorial and Hooks Documentation.

OpenSearch⚓︎

OpenSearch configuration for the reference deployment is defined here

Index⚓︎

Currently all the health check telemetry is written to ss4o_traces-default-namespace index. The index is created by OTel Collector OpenSearch exporter. In particular, this means that the index will not exist until the first health check telemetry gets to OpenSearch.

Info

ss4o_traces-default-namespace index name is generated by the formula by OTel Collector OpensSearch exporter, see here. So to change the index name you need to:

  • Update the OpenSearch Exporter configuration here.
  • Update document-level security filter to the new namespace.
  • Point Telemetry API to the new index here.
  • Point health checks which use previous telemetry to the new index here.

You should also consider doing index management optimized for data streams, such as rotating backing indices, deleting old data, etc., see Data Streams and Index State Management for more.

Document-level security⚓︎

Each trace is tagged with user.id of the user who created the check. To give a user access to telemetry annotated with their user.id, give the user the role own_trace_data_access, which is defined here. This uses document-level security feature of OpenSearch.

Info

The current user.id annotation is meant to signal who is allowed to view the health check telemetry. For a different or more complex security model, you would need to tag the OTel traces with identifier(s) to be used in authorization decisions, and to create a document-level-security role similar to this, but with a different filtering condition.