Skip to content

Operations BB Architecture⚓︎

Overview⚓︎

The operator of a platform needs to ensure that the platform is running smoothly, with healthy services that are meeting their performance expectations, and with sufficient information available to diagnose issues when they arise. This requires a robust monitoring and logging infrastructure that can provide real-time insights at both Infrastructure and Application level: virtual machines, Kubernetes clusters, and individual services.

Auto-scaling of infrastructure and services is also essential to ensure that the platform can handle varying loads without manual intervention. Provision of suitable performance metrics is crucial for this, allowing the platform to scale up or down based on demand.

The Operations Building Block (BB) provides the necessary tools and practices to achieve this, covering:

  • Health Monitoring
  • Metrics Collection
  • Logging
  • Alerting
  • User interfaces for monitoring and troubleshooting

Modular Approach⚓︎

EOEPCA comprises a set of building blocks that can be used to implement platform capabilities. The platform operator can select the building blocks that are relevant to their platform, and integrate them into their operations. The platform operator would not typically have the depth on knowledge to know how to best monitor each building block.

Each building block should ensure that is is Observable in accordance with the Operations BB. The approach should be modular, such that each building block provides its own observability artefacts, which can then be integrated into the platform’s overall monitoring and logging framework.

Thus, each building block can be expected to provide:

  • Endpoints for health checks
  • Metrics endpoint and associated exporter
  • Log outputs in a suitable format for centralised consumption
  • Pre-configured dashboards for visualisation, that can be imported into the Operations tooling

Where possible, the Operations framework should support the use of a declarative approach for each building block to contribute its observability aspects.

Preferred Software Stack⚓︎

Based upon the use cases and requirements of platforms using EOEPCA, a preference has been expressed for the following software stack for the Operations BB:

  • Prometheus: For metrics collection and alerting. It provides a powerful query language and integrates well with Kubernetes.
  • Grafana: For visualisation of metrics and logs. It supports a wide range of data sources and provides pre-configured dashboards.
  • Loki: For log aggregation. It is designed to work seamlessly with Prometheus and Grafana, allowing for a unified view of metrics and logs.
  • Alertmanager: For managing alerts generated by Prometheus. It can route alerts to various receivers such as email, Slack, or PagerDuty.

The Prometheus Operator defines a set of Kubernetes Custom Resource Definitions (CRDs) that allow for the declarative configuration of Prometheus and its components, including Alertmanager and Grafana. This provides a convenient mechamism through which each building block is able to provide its observability artefacts.

High-level Requirements⚓︎

The high-level requirements to be covered by the Operations BB include:

  1. a deployment of some mechanism for storing and querying of Metrics (preferably using Prometheus)
  2. a deployment of some mechanism for storing and querying of Logs (preferably using Loki)
  3. recommendations, documentation, and some reference implementations helping other building blocks generate telemetry
  4. deployment of a service or a reference implementation to help other building blocks collect, transport, and store telemetry (metrics, logs, and maybe traces) in the systems described in (1.) and (2.)
  5. a deployment of some service for displaying graphs and dashboards (preferably through Grafana)
  6. a documented mechanism for other building blocks to “publish” graphs/dashboards on the system described in (5.)
  7. a documented and predictable mechanism for using the telemetry generated using (4.) in the graphs/dashboards given by (5.) and (6.)
  8. a deployed mechanism, documentation, and/or reference implementation for how to rescale Kubernetes resources in response to telemetry generated using (3.) and (4.)
  9. a documented mechanism for gathering metrics/logs from Kubernetes metrics endpoints, as well as gathering metrics from the Kuberentes cluster (e.g. via the included K8S Prometheus)
  10. design principles:
    1. deployments should be compatible with the type of declarative environment used in the rest of EOEPCA+, especially relevant to (1.), (2.), (5.), and (6.)
    2. access to declared dashboards and stored telemetry (whether accessed through dashboards or other supported means) should respect IAM policies specified by building blocks and administrators, including allowing access to arbitrary end-users when appropriate
    3. the functionality for collecting and utilising telemetry should be compatible with telemetry as generated by the Kubernetes API/control plane itself

Evolution of existing Resource Health BB⚓︎

The existing Resource Health BB provides the capability to define proactive checks to assert the health of resources within the platform. For example, periodically exercising the APIs of the EOEPCA building blocks to ensure they are responding as expected. The outcome of such checks can be delivered through the Operations BB as contribution to the overall BB health monitoring.

Moreover, consideration should be made to evolve the existing Resource Health BB to become the new Operations BB, as it already provides many of the required capabilities. This would allow for a more streamlined approach to platform operations, with a single building block handling both resource health and operational monitoring.

In the case that the Resource Health BB capability is integrated into that of the Operations BB, then the following aspects of the existing Resource Health should be consolidated into the combined solution:

  • Adoption of the same Dashboad solution (e.g. Grafana) - instead of the current OpenSearch-Dashboards/Kibana approach
  • Trace-type telemetry also be taken into account (beyond metrics and logs), ideally in such a way that platforms can use the same infrastructure for traces generated from health checks.
    Currently, OpenSearch is used for storage/access and oTel-collector for transmission
  • Consider building the transmission and instrumentation around the opentelemetry.io project.
    This is especially relevant to (3.) and (4.) and is compatible with the use of Loki and Prometheus for storage and querying. Use of OpenTelemetry instrumentation is especially recommended as a way of gathering and exposing telemetry in other building blocks and the the use of oTel-collector (or other OpenTelemetry-compatible solutions such as https://vector.dev/) for sampling, aggregating, and transporting telemetry in accordance with OpenTelemetry protocols.
  • Try to ensure the availability of gathered telemetry for use in health checks, ideally such that there is a mechanism for cross correlating metrics, logs, and spans.
    This would be at least partially resolved by relying on the OpenTelemetry project as per above.