Spring Boot - Observability

Overview

Spring Boot Observability

Github: https://github.com/gitorko/project71

Application monitoring can be classified into

  1. Observability - Creates metrics, traces, logs that get stored in time-series db and charts are created. eg: Prometheus, Grafana, OpenTelemetry, Jaeger, Zipkin
  2. APM (Application Performance Management) - Runs an agents in the jvm that instruments the bytecode and sends metrics to a remote server, focuses on performance and user experience in the application layer. eg: New Relic, Datadog APM, AppDynamics, Dynatrace, Elastic APM
  3. Monitoring - Checks endpoint uri to monitor health (cpu, memory) infrastructure-centric for alerting. eg: Nagios, Prometheus

Observability

Observability is the ability to observe the internal state of a running system from the outside. Observability has 3 pillars

  1. Metrics: Quantitative data about system performance (e.g., CPU usage, request count, error rates) eg: spring-boot-starter-actuator.
  2. Logs: Event-based data for tracking specific actions and events with correlation/span id (e.g., application logs) eg: micrometer-tracing-bridge-brave.
  3. Traces: Distributed tracing for tracking the path of a request across services (e.g., tracing API calls) eg: zipkin-reporter-brave.

Various tools that help in observability

  1. Prometheus - An open-source systems monitoring and alerting tool. Prometheus scrapes/collects metrics from an endpoint at regular intervals. Stores the data in a time series database.
  2. Grafana - A visualization tool, can pull data from multiple sources (Prometheus) and shows them in graphs.
  3. Zipkin - A distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures.

Micrometer is a vendor-neutral instrumentation library that allows you to collect metrics and traces for observability.

  1. Metrics Collection - Supports Prometheus, Graphite, Datadog, New Relic, etc.
  2. Tracing Support - Used with Brave (Zipkin), OpenTelemetry, Wavefront, etc.
  3. Logging Context Propagation - Adds tracing IDs in logs for better debugging
  4. Spring Integration - Works out of the box with Spring Boot’s Actuator

Spring Observability internally uses Micrometer, so in Spring Boot 3+, you should use Spring Observability APIs for new projects

Logging

Tracing adds spans/traces to all logs.

Metrics

A Meter consists of a name and tags, There are 4 main types of meters.

  1. Timers - Time taken to run something.
  2. Counter - Number of time something was run.
  3. Gauge - Report data when observed. Gauges can be useful when monitoring stats of cache, collections
  4. Distribution summary - Distribution of events.
  5. Binders - Built-in binders to monitor the JVM, caches, ExecutorService, and logging services

Distributed Tracing

Spring Boot samples only 10% of requests to prevent overwhelming the trace backend. Change probability to 1.0 so that every request is sent to the trace backend.

1management:
2  tracing:
3    sampling:
4      probability: 1.0

Code

  1package com.demo.project71.service;
  2
  3import java.util.Random;
  4import java.util.concurrent.TimeUnit;
  5
  6import com.demo.project71.config.RegistryConfig;
  7import io.micrometer.core.annotation.Timed;
  8import io.micrometer.observation.Observation;
  9import io.micrometer.observation.ObservationRegistry;
 10import io.micrometer.observation.annotation.Observed;
 11import lombok.RequiredArgsConstructor;
 12import lombok.SneakyThrows;
 13import lombok.extern.slf4j.Slf4j;
 14import org.springframework.context.annotation.Configuration;
 15import org.springframework.scheduling.annotation.Async;
 16import org.springframework.scheduling.annotation.EnableAsync;
 17import org.springframework.stereotype.Service;
 18
 19@Service
 20@RequiredArgsConstructor
 21@Slf4j
 22@Configuration
 23@EnableAsync
 24public class GreetService {
 25
 26    final ObservationRegistry observationRegistry;
 27
 28    public String sayHello1() {
 29        return Observation.createNotStarted("sayHello1", observationRegistry).contextualName("greet.hello-1").observe(() -> {
 30            log.info("Hello World 1!");
 31            return "Hello World 1!";
 32        });
 33    }
 34
 35    @Observed(contextualName = "greet.hello-2")
 36    public String sayHello2() {
 37        log.info("Hello World 2!");
 38        return "Hello World 2!";
 39    }
 40
 41    public String sayHello3() {
 42        return Observation.createNotStarted("greet.hello-3", observationRegistry).observe(this::sayHello3_NoObs);
 43    }
 44
 45    public String sayHello3_NoObs() {
 46        log.info("Hello World 3!");
 47        return "Hello World 3!";
 48    }
 49
 50    @Timed("greet.sayHello4")
 51    @SneakyThrows
 52    public String sayHello4() {
 53        RegistryConfig.helloApiCounter.increment();
 54        log.info("Hello World 4!");
 55        int sleepTime = new Random().nextInt(5);
 56        log.info("Sleeping for seconds: {}", sleepTime);
 57        TimeUnit.SECONDS.sleep(sleepTime);
 58        return "Hello World 4!";
 59    }
 60
 61    @SneakyThrows
 62    public String sayHello5() throws InterruptedException {
 63        log.info("sayHello5 start - Original span");
 64        inner1();
 65        inner2();
 66        log.info("sayHello5 end - Original span");
 67        return "Hello World 5!";
 68    }
 69
 70    public void inner1() {
 71        Observation.createNotStarted("inner1", observationRegistry).observe(() -> {
 72            log.info("Inner1");
 73        });
 74    }
 75
 76    public void inner2() {
 77        Observation.start("inner2", observationRegistry).observe(() -> {
 78            log.info("Inner2");
 79        });
 80    }
 81
 82    /**
 83     * You can add additional values
 84     * Low cardinality tags will be added to metrics and traces, while high cardinality tags will only be added to traces.
 85     */
 86    public String sayHello6() {
 87        return Observation.createNotStarted("sayHello1", observationRegistry)
 88                .lowCardinalityKeyValue("locale", "en-US")
 89                .highCardinalityKeyValue("userId", "42")
 90                .contextualName("greet.hello-6")
 91                .observe(() -> {
 92                    log.info("Hello World 6!");
 93                    return "Hello World 6!";
 94                });
 95    }
 96
 97    @Async
 98    @SneakyThrows
 99    public void asyncHello() {
100        log.info("Start Async Method");
101        TimeUnit.SECONDS.sleep(1);
102        log.info("End Async Method");
103    }
104}
 1package com.demo.project71.config;
 2
 3import io.micrometer.core.aop.TimedAspect;
 4import io.micrometer.core.instrument.Counter;
 5import io.micrometer.core.instrument.MeterRegistry;
 6import io.micrometer.core.instrument.Metrics;
 7import jakarta.annotation.PostConstruct;
 8import org.springframework.beans.factory.annotation.Value;
 9import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
10import org.springframework.context.annotation.Bean;
11import org.springframework.context.annotation.Configuration;
12import org.springframework.context.annotation.EnableAspectJAutoProxy;
13
14@Configuration
15@EnableAspectJAutoProxy
16public class RegistryConfig {
17
18    public static Counter helloApiCounter;
19
20    /**
21     * Applies common tags on all Meters
22     */
23    @Bean
24    MeterRegistryCustomizer<MeterRegistry> configurer(@Value("${spring.application.name}") String applicationName) {
25        return registry -> registry.config().commonTags("application", applicationName);
26    }
27
28    /**
29     * Enables @Timed annotation
30     */
31    @Bean
32    public TimedAspect timedAspect(MeterRegistry registry) {
33        return new TimedAspect(registry);
34    }
35
36    /**
37     * Creates a Meter
38     */
39    @PostConstruct
40    public void postInit() {
41        helloApiCounter = Metrics.counter("hello.api.count", "type", "order");
42    }
43}
 1package com.demo.project71.config;
 2
 3import java.util.concurrent.Executor;
 4import java.util.concurrent.Executors;
 5
 6import io.micrometer.context.ContextExecutorService;
 7import io.micrometer.context.ContextSnapshotFactory;
 8import lombok.RequiredArgsConstructor;
 9import org.springframework.context.annotation.Configuration;
10import org.springframework.scheduling.annotation.AsyncConfigurer;
11
12@Configuration
13@RequiredArgsConstructor
14class ThreadConfig implements AsyncConfigurer {
15
16    @Override
17    public Executor getAsyncExecutor() {
18        return ContextExecutorService.wrap(Executors.newFixedThreadPool(5), ContextSnapshotFactory.builder().build()::captureAll);
19    }
20
21}

Postman

Import the postman collection to postman

Postman Collection

Setup

 1# Project 71
 2
 3Spring Observability
 4
 5[https://gitorko.github.io/spring-boot-observability/](https://gitorko.github.io/spring-boot-observability/)
 6
 7### Version
 8
 9Check version
10
11```bash
12$java --version
13openjdk 21.0
14```
15
16### Postgres DB
17
18```
19docker run -p 5432:5432 --name pg-container -e POSTGRES_PASSWORD=password -d postgres:9.6.10
20docker ps
21docker exec -it pg-container psql -U postgres -W postgres
22CREATE USER test WITH PASSWORD 'test@123';
23CREATE DATABASE "test-db" WITH OWNER "test" ENCODING UTF8 TEMPLATE template0;
24grant all PRIVILEGES ON DATABASE "test-db" to test;
25
26docker stop pg-container
27docker start pg-container
28```
29
30### Zipkin
31
32To run zipkin server use the docker command
33
34```bash
35docker run -d -p 9411:9411 --name my-zipkin openzipkin/zipkin
36
37docker stop my-zipkin
38docker start my-zipkin
39```
40
41Login to zipkin UI, wait for few seconds for server to be up.
42
43[http://localhost:9411/zipkin/](http://localhost:9411/zipkin/)
44
45## Prometheus
46
47Update the target ip-address in the prometheus.yml file, don't use localhost when using docker container
48
49To start the prometheus docker instance build the docker image & run the image.
50
51```bash
52cd project71
53docker build -f docker/Dockerfile --force-rm -t my-prometheus .
54docker run -d -p 9090:9090 --name my-prometheus my-prometheus
55
56docker stop my-prometheus
57docker start my-prometheus
58```
59
60[http://localhost:9090](http://localhost:9090)
61
62## Grafana
63
64To start the grafana docker instance run the command.
65
66```bash
67docker run --name my-grafana -d -p 3000:3000 grafana/grafana
68
69docker stop my-grafana
70docker start my-grafana
71```
72
73[http://localhost:3000](http://localhost:3000)
74
75```
76user: admin
77password: admin
78```
79
80### Dev
81
82To run the code.
83
84```bash
85./gradlew clean build
86./gradlew bootRun
87```

Open zipkin dashboard

http://localhost:9411/zipkin/

Open prometheus dashboard

http://localhost:9090

Open grafana dashboard

http://localhost:3000

1user: admin
2password: admin

Add the prometheus data source, make sure it's the ip address of your system, don't add localhost

http://IP-ADDRESS:9090

There are existing grafana dashboards that can be imported. Import a dashboard, Download the json file or copy the ID of the dashboard for micrometer dashboard.

https://grafana.com/dashboards/4701

Create a custom dashboard, Add a new panel, add 'hello_api_count_total' metric in the query, save the dashboard.

References

https://micrometer.io/docs

https://prometheus.io/

https://grafana.com/

https://grafana.com/grafana/dashboards/4701

https://grafana.com/grafana/dashboards/

comments powered by Disqus