Spring Boot - Observability
Overview
Spring Boot Observability
Github: https://github.com/gitorko/project71
Monitoring & Observability
- Monitoring* - ensures the system is healthy. With
spring-boot-starter-actuator
you can monitor CPU usage, memory usage, request rates, and error rates. - Observability - helps you understand issues and derive insights. Micrometer Observability API
Observability is the ability to observe the internal state of a running system from the outside. Observability has 3 pillars
- Logging - Logging Correlation IDs - Correlation IDs provide a helpful way to link lines in your log files to spans/traces.
- Metrics - Custom metrics to monitor time taken, count invocations etc.
- Distributed Tracing - Micrometer Tracing library is a facade for popular tracer libraries. eg: OpenTelemetry, OpenZipkin Brave
Various tools that help in observability
- Prometheus — An open-source systems monitoring and alerting tool. Prometheus scrapes/collects metrics from an endpoint at regular intervals. Stores the data in a time series database.
- Grafana — A visualization tool, can pull data from multiple sources (Prometheus) and shows them in graphs.
- Zipkin — a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data.
Logging
Micrometer tracing adds spans/traces to all logs.
Metrics
A Meter
consists of a name and tags, There are 4 main types of meters.
- Timers - Time taken to run something.
- Counter - Number of time something was run.
- Gauge - Report data when observed. Gauges can be useful when monitoring stats of cache, collections
- Distribution summary - Distribution of events.
- Binders - Built-in binders to monitor the JVM, caches, ExecutorService, and logging services
Distributed Tracing
Spring Boot samples only 10% of requests to prevent overwhelming the trace backend. Change probability to 1.0 so that every request is sent to the trace backend.
1management:
2 tracing:
3 sampling:
4 probability: 1.0
Code
1package com.demo.project71.service;
2
3import java.util.Random;
4import java.util.concurrent.TimeUnit;
5
6import com.demo.project71.config.RegistryConfig;
7import io.micrometer.core.annotation.Timed;
8import io.micrometer.observation.Observation;
9import io.micrometer.observation.ObservationRegistry;
10import io.micrometer.observation.annotation.Observed;
11import lombok.RequiredArgsConstructor;
12import lombok.SneakyThrows;
13import lombok.extern.slf4j.Slf4j;
14import org.springframework.context.annotation.Configuration;
15import org.springframework.scheduling.annotation.Async;
16import org.springframework.scheduling.annotation.EnableAsync;
17import org.springframework.stereotype.Service;
18
19@Service
20@RequiredArgsConstructor
21@Slf4j
22@Configuration
23@EnableAsync
24public class GreetService {
25
26 final ObservationRegistry observationRegistry;
27
28 public String sayHello1() {
29 return Observation.createNotStarted("sayHello1", observationRegistry).contextualName("greet.hello-1").observe(() -> {
30 log.info("Hello World 1!");
31 return "Hello World 1!";
32 });
33 }
34
35 @Observed(contextualName = "greet.hello-2")
36 public String sayHello2() {
37 log.info("Hello World 2!");
38 return "Hello World 2!";
39 }
40
41 public String sayHello3() {
42 return Observation.createNotStarted("greet.hello-3", observationRegistry).observe(this::sayHello3_NoObs);
43 }
44
45 public String sayHello3_NoObs() {
46 log.info("Hello World 3!");
47 return "Hello World 3!";
48 }
49
50 @Timed("greet.sayHello4")
51 @SneakyThrows
52 public String sayHello4() {
53 RegistryConfig.helloApiCounter.increment();
54 log.info("Hello World 4!");
55 int sleepTime = new Random().nextInt(5);
56 log.info("Sleeping for seconds: {}", sleepTime);
57 TimeUnit.SECONDS.sleep(sleepTime);
58 return "Hello World 4!";
59 }
60
61 @SneakyThrows
62 public String sayHello5() throws InterruptedException {
63 log.info("sayHello5 start - Original span");
64 inner1();
65 inner2();
66 log.info("sayHello5 end - Original span");
67 return "Hello World 5!";
68 }
69
70 public void inner1() {
71 Observation.createNotStarted("inner1", observationRegistry).observe(() -> {
72 log.info("Inner1");
73 });
74 }
75
76 public void inner2() {
77 Observation.start("inner2", observationRegistry).observe(() -> {
78 log.info("Inner2");
79 });
80 }
81
82 /**
83 * You can add additional values
84 * Low cardinality tags will be added to metrics and traces, while high cardinality tags will only be added to traces.
85 */
86 public String sayHello6() {
87 return Observation.createNotStarted("sayHello1", observationRegistry)
88 .lowCardinalityKeyValue("locale", "en-US")
89 .highCardinalityKeyValue("userId", "42")
90 .contextualName("greet.hello-6")
91 .observe(() -> {
92 log.info("Hello World 6!");
93 return "Hello World 6!";
94 });
95 }
96
97 @Async
98 @SneakyThrows
99 public void asyncHello() {
100 log.info("Start Async Method");
101 TimeUnit.SECONDS.sleep(1);
102 log.info("End Async Method");
103 }
104}
1package com.demo.project71.config;
2
3import io.micrometer.core.aop.TimedAspect;
4import io.micrometer.core.instrument.Counter;
5import io.micrometer.core.instrument.MeterRegistry;
6import io.micrometer.core.instrument.Metrics;
7import jakarta.annotation.PostConstruct;
8import org.springframework.beans.factory.annotation.Value;
9import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
10import org.springframework.context.annotation.Bean;
11import org.springframework.context.annotation.Configuration;
12import org.springframework.context.annotation.EnableAspectJAutoProxy;
13
14@Configuration
15@EnableAspectJAutoProxy
16public class RegistryConfig {
17
18 public static Counter helloApiCounter;
19
20 /**
21 * Applies common tags on all Meters
22 */
23 @Bean
24 MeterRegistryCustomizer<MeterRegistry> configurer(@Value("${spring.application.name}") String applicationName) {
25 return registry -> registry.config().commonTags("application", applicationName);
26 }
27
28 /**
29 * Enables @Timed annotation
30 */
31 @Bean
32 public TimedAspect timedAspect(MeterRegistry registry) {
33 return new TimedAspect(registry);
34 }
35
36 /**
37 * Creates a Meter
38 */
39 @PostConstruct
40 public void postInit() {
41 helloApiCounter = Metrics.counter("hello.api.count", "type", "order");
42 }
43}
1package com.demo.project71.config;
2
3import java.util.concurrent.Executor;
4import java.util.concurrent.Executors;
5
6import io.micrometer.context.ContextExecutorService;
7import io.micrometer.context.ContextSnapshotFactory;
8import lombok.RequiredArgsConstructor;
9import org.springframework.context.annotation.Configuration;
10import org.springframework.scheduling.annotation.AsyncConfigurer;
11
12@Configuration
13@RequiredArgsConstructor
14class ThreadConfig implements AsyncConfigurer {
15
16 @Override
17 public Executor getAsyncExecutor() {
18 return ContextExecutorService.wrap(Executors.newFixedThreadPool(5), ContextSnapshotFactory.builder().build()::captureAll);
19 }
20
21}
Postman
Import the postman collection to postman
Setup
1# Project 71
2
3Spring Observability
4
5[https://gitorko.github.io/spring-boot-observability/](https://gitorko.github.io/spring-boot-observability/)
6
7### Version
8
9Check version
10
11```bash
12$java --version
13openjdk 21.0
14```
15
16### Postgres DB
17
18```
19docker run -p 5432:5432 --name pg-container -e POSTGRES_PASSWORD=password -d postgres:9.6.10
20docker ps
21docker exec -it pg-container psql -U postgres -W postgres
22CREATE USER test WITH PASSWORD 'test@123';
23CREATE DATABASE "test-db" WITH OWNER "test" ENCODING UTF8 TEMPLATE template0;
24grant all PRIVILEGES ON DATABASE "test-db" to test;
25
26docker stop pg-container
27docker start pg-container
28```
29
30### Zipkin
31
32To run zipkin server use the docker command
33
34```bash
35docker run -d -p 9411:9411 --name my-zipkin openzipkin/zipkin
36
37docker stop my-zipkin
38docker start my-zipkin
39```
40
41Login to zipkin UI, wait for few seconds for server to be up.
42
43[http://localhost:9411/zipkin/](http://localhost:9411/zipkin/)
44
45## Prometheus
46
47Update the target ip-address in the prometheus.yml file, don't use localhost when using docker container
48
49To start the prometheus docker instance build the docker image & run the image.
50
51```bash
52cd project71
53docker build -f docker/Dockerfile --force-rm -t my-prometheus .
54docker run -d -p 9090:9090 --name my-prometheus my-prometheus
55
56docker stop my-prometheus
57docker start my-prometheus
58```
59
60[http://localhost:9090](http://localhost:9090)
61
62## Grafana
63
64To start the grafana docker instance run the command.
65
66```bash
67docker run --name my-grafana -d -p 3000:3000 grafana/grafana
68
69docker stop my-grafana
70docker start my-grafana
71```
72
73[http://localhost:3000](http://localhost:3000)
74
75```
76user: admin
77password: admin
78```
79
80### Dev
81
82To run the code.
83
84```bash
85./gradlew clean build
86./gradlew bootRun
87```
Open zipkin dashboard
Open prometheus dashboard
Open grafana dashboard
1user: admin
2password: admin
Add the prometheus data source, make sure it's the ip address of your system, don't add localhost
http://IP-ADDRESS:9090
There are existing grafana dashboards that can be imported. Import a dashboard, Download the json file or copy the ID of the dashboard for micrometer dashboard.
https://grafana.com/dashboards/4701
Create a custom dashboard, Add a new panel, add 'hello_api_count_total' metric in the query, save the dashboard.