Spring Boot - Observability

Overview

Spring Boot Observability

Github: https://github.com/gitorko/project71

Monitoring & Observability

  • Monitoring* - ensures the system is healthy. With spring-boot-starter-actuator you can monitor CPU usage, memory usage, request rates, and error rates.
  • Observability - helps you understand issues and derive insights. Micrometer Observability API

Observability is the ability to observe the internal state of a running system from the outside. Observability has 3 pillars

  1. Logging - Logging Correlation IDs - Correlation IDs provide a helpful way to link lines in your log files to spans/traces.
  2. Metrics - Custom metrics to monitor time taken, count invocations etc.
  3. Distributed Tracing - Micrometer Tracing library is a facade for popular tracer libraries. eg: OpenTelemetry, OpenZipkin Brave

Various tools that help in observability

  1. Prometheus — An open-source systems monitoring and alerting tool. Prometheus scrapes/collects metrics from an endpoint at regular intervals. Stores the data in a time series database.
  2. Grafana — A visualization tool, can pull data from multiple sources (Prometheus) and shows them in graphs.
  3. Zipkin — a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data.

Logging

Micrometer tracing adds spans/traces to all logs.

Metrics

A Meter consists of a name and tags, There are 4 main types of meters.

  1. Timers - Time taken to run something.
  2. Counter - Number of time something was run.
  3. Gauge - Report data when observed. Gauges can be useful when monitoring stats of cache, collections
  4. Distribution summary - Distribution of events.
  5. Binders - Built-in binders to monitor the JVM, caches, ExecutorService, and logging services

Distributed Tracing

Spring Boot samples only 10% of requests to prevent overwhelming the trace backend. Change probability to 1.0 so that every request is sent to the trace backend.

1management:
2  tracing:
3    sampling:
4      probability: 1.0

Code

  1package com.demo.project71.service;
  2
  3import java.util.Random;
  4import java.util.concurrent.TimeUnit;
  5
  6import com.demo.project71.config.RegistryConfig;
  7import io.micrometer.core.annotation.Timed;
  8import io.micrometer.observation.Observation;
  9import io.micrometer.observation.ObservationRegistry;
 10import io.micrometer.observation.annotation.Observed;
 11import lombok.RequiredArgsConstructor;
 12import lombok.SneakyThrows;
 13import lombok.extern.slf4j.Slf4j;
 14import org.springframework.context.annotation.Configuration;
 15import org.springframework.scheduling.annotation.Async;
 16import org.springframework.scheduling.annotation.EnableAsync;
 17import org.springframework.stereotype.Service;
 18
 19@Service
 20@RequiredArgsConstructor
 21@Slf4j
 22@Configuration
 23@EnableAsync
 24public class GreetService {
 25
 26    final ObservationRegistry observationRegistry;
 27
 28    public String sayHello1() {
 29        return Observation.createNotStarted("sayHello1", observationRegistry).contextualName("greet.hello-1").observe(() -> {
 30            log.info("Hello World 1!");
 31            return "Hello World 1!";
 32        });
 33    }
 34
 35    @Observed(contextualName = "greet.hello-2")
 36    public String sayHello2() {
 37        log.info("Hello World 2!");
 38        return "Hello World 2!";
 39    }
 40
 41    public String sayHello3() {
 42        return Observation.createNotStarted("greet.hello-3", observationRegistry).observe(this::sayHello3_NoObs);
 43    }
 44
 45    public String sayHello3_NoObs() {
 46        log.info("Hello World 3!");
 47        return "Hello World 3!";
 48    }
 49
 50    @Timed("greet.sayHello4")
 51    @SneakyThrows
 52    public String sayHello4() {
 53        RegistryConfig.helloApiCounter.increment();
 54        log.info("Hello World 4!");
 55        int sleepTime = new Random().nextInt(5);
 56        log.info("Sleeping for seconds: {}", sleepTime);
 57        TimeUnit.SECONDS.sleep(sleepTime);
 58        return "Hello World 4!";
 59    }
 60
 61    @SneakyThrows
 62    public String sayHello5() throws InterruptedException {
 63        log.info("sayHello5 start - Original span");
 64        inner1();
 65        inner2();
 66        log.info("sayHello5 end - Original span");
 67        return "Hello World 5!";
 68    }
 69
 70    public void inner1() {
 71        Observation.createNotStarted("inner1", observationRegistry).observe(() -> {
 72            log.info("Inner1");
 73        });
 74    }
 75
 76    public void inner2() {
 77        Observation.start("inner2", observationRegistry).observe(() -> {
 78            log.info("Inner2");
 79        });
 80    }
 81
 82    /**
 83     * You can add additional values
 84     * Low cardinality tags will be added to metrics and traces, while high cardinality tags will only be added to traces.
 85     */
 86    public String sayHello6() {
 87        return Observation.createNotStarted("sayHello1", observationRegistry)
 88                .lowCardinalityKeyValue("locale", "en-US")
 89                .highCardinalityKeyValue("userId", "42")
 90                .contextualName("greet.hello-6")
 91                .observe(() -> {
 92                    log.info("Hello World 6!");
 93                    return "Hello World 6!";
 94                });
 95    }
 96
 97    @Async
 98    @SneakyThrows
 99    public void asyncHello() {
100        log.info("Start Async Method");
101        TimeUnit.SECONDS.sleep(1);
102        log.info("End Async Method");
103    }
104}
 1package com.demo.project71.config;
 2
 3import io.micrometer.core.aop.TimedAspect;
 4import io.micrometer.core.instrument.Counter;
 5import io.micrometer.core.instrument.MeterRegistry;
 6import io.micrometer.core.instrument.Metrics;
 7import jakarta.annotation.PostConstruct;
 8import org.springframework.beans.factory.annotation.Value;
 9import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
10import org.springframework.context.annotation.Bean;
11import org.springframework.context.annotation.Configuration;
12import org.springframework.context.annotation.EnableAspectJAutoProxy;
13
14@Configuration
15@EnableAspectJAutoProxy
16public class RegistryConfig {
17
18    public static Counter helloApiCounter;
19
20    /**
21     * Applies common tags on all Meters
22     */
23    @Bean
24    MeterRegistryCustomizer<MeterRegistry> configurer(@Value("${spring.application.name}") String applicationName) {
25        return registry -> registry.config().commonTags("application", applicationName);
26    }
27
28    /**
29     * Enables @Timed annotation
30     */
31    @Bean
32    public TimedAspect timedAspect(MeterRegistry registry) {
33        return new TimedAspect(registry);
34    }
35
36    /**
37     * Creates a Meter
38     */
39    @PostConstruct
40    public void postInit() {
41        helloApiCounter = Metrics.counter("hello.api.count", "type", "order");
42    }
43}
 1package com.demo.project71.config;
 2
 3import java.util.concurrent.Executor;
 4import java.util.concurrent.Executors;
 5
 6import io.micrometer.context.ContextExecutorService;
 7import io.micrometer.context.ContextSnapshotFactory;
 8import lombok.RequiredArgsConstructor;
 9import org.springframework.context.annotation.Configuration;
10import org.springframework.scheduling.annotation.AsyncConfigurer;
11
12@Configuration
13@RequiredArgsConstructor
14class ThreadConfig implements AsyncConfigurer {
15
16    @Override
17    public Executor getAsyncExecutor() {
18        return ContextExecutorService.wrap(Executors.newFixedThreadPool(5), ContextSnapshotFactory.builder().build()::captureAll);
19    }
20
21}

Postman

Import the postman collection to postman

Postman Collection

Setup

 1# Project 71
 2
 3Spring Observability
 4
 5[https://gitorko.github.io/spring-boot-observability/](https://gitorko.github.io/spring-boot-observability/)
 6
 7### Version
 8
 9Check version
10
11```bash
12$java --version
13openjdk 21.0
14```
15
16### Postgres DB
17
18```
19docker run -p 5432:5432 --name pg-container -e POSTGRES_PASSWORD=password -d postgres:9.6.10
20docker ps
21docker exec -it pg-container psql -U postgres -W postgres
22CREATE USER test WITH PASSWORD 'test@123';
23CREATE DATABASE "test-db" WITH OWNER "test" ENCODING UTF8 TEMPLATE template0;
24grant all PRIVILEGES ON DATABASE "test-db" to test;
25
26docker stop pg-container
27docker start pg-container
28```
29
30### Zipkin
31
32To run zipkin server use the docker command
33
34```bash
35docker run -d -p 9411:9411 --name my-zipkin openzipkin/zipkin
36
37docker stop my-zipkin
38docker start my-zipkin
39```
40
41Login to zipkin UI, wait for few seconds for server to be up.
42
43[http://localhost:9411/zipkin/](http://localhost:9411/zipkin/)
44
45## Prometheus
46
47Update the target ip-address in the prometheus.yml file, don't use localhost when using docker container
48
49To start the prometheus docker instance build the docker image & run the image.
50
51```bash
52cd project71
53docker build -f docker/Dockerfile --force-rm -t my-prometheus .
54docker run -d -p 9090:9090 --name my-prometheus my-prometheus
55
56docker stop my-prometheus
57docker start my-prometheus
58```
59
60[http://localhost:9090](http://localhost:9090)
61
62## Grafana
63
64To start the grafana docker instance run the command.
65
66```bash
67docker run --name my-grafana -d -p 3000:3000 grafana/grafana
68
69docker stop my-grafana
70docker start my-grafana
71```
72
73[http://localhost:3000](http://localhost:3000)
74
75```
76user: admin
77password: admin
78```
79
80### Dev
81
82To run the code.
83
84```bash
85./gradlew clean build
86./gradlew bootRun
87```

Open zipkin dashboard

http://localhost:9411/zipkin/

Open prometheus dashboard

http://localhost:9090

Open grafana dashboard

http://localhost:3000

1user: admin
2password: admin

Add the prometheus data source, make sure it's the ip address of your system, don't add localhost

http://IP-ADDRESS:9090

There are existing grafana dashboards that can be imported. Import a dashboard, Download the json file or copy the ID of the dashboard for micrometer dashboard.

https://grafana.com/dashboards/4701

Create a custom dashboard, Add a new panel, add 'hello_api_count_total' metric in the query, save the dashboard.

References

https://micrometer.io/docs

https://prometheus.io/

https://grafana.com/

https://grafana.com/grafana/dashboards/4701

https://grafana.com/grafana/dashboards/

comments powered by Disqus