software development spring boot performance monitoring monitoring

System performance: What you need to know about monoliths and bare metal servers setup optimization - part 1

30-Jan-2025 - Muthomi Kathurima

In one server, with 32 Gigs and 12 cores, we were able to optimise a large monolith to perform over 3 million reads a day, and over 1 million writes efficiently and achieve a 300ms p90 efficiency while at it.

This article is going to detail our approach towards achieving this remarkable numbers, and what we learnt in the process. In the end, this approach is not the gospel truth, but should give you an idea on how to approach it yourself.

In this part, we will talk about the setup, and how to set up monitoring to identify bottlenecks specifically for Spring Boot Backend.

Our Stack

Spring Boot Java Monolith

Postgresql 14 Database

Nginx Web Server

 

Performance Monitoring;

Based on the above, the 3 easiest to monitor and do the greatest impact on was the application, postgresql instance and the server.

Spring Boot;

Spring boot actuator allows us to monitor our app, gather health metrics and the state of the database. It also allows us to be able to have access to operational information about the application. It is also very simple to setup, and only requires installing the dependancy and exposing the required endpoints. `/health` and `/info` are exposed by default.

To learn how to setup Spring boot actuator and configure it securely, read the documentation; https://www.baeldung.com/spring-boot-actuators, and see the endpoints available and how to configure the security around that here: https://docs.spring.io/spring-boot/reference/actuator/endpoints.html

  1. Configure an uptime monitoring service.

To leverage information setup above, there is a need to setup an uptime monitoring tool that notifies when their is service unavailability. Based on needs and criticality you can leverage multiple services with different time checks.

Based on our needs, we engaged the free service onlineornot.com. Other services include uptime.com, newrelic monitoring, Sentry.

This should allow you to know when the service is up, when its down, the times its down, and potentially show you traffic trends causing the outage for action.

   2. Comprehensive Logging

Well written code should have proper logging to show events, change of data, business operations, as well as handled exceptions.

For Spring Boot, Slf4j was our choice of logger to use. There are other alternatives but the latter was easy for us to get started and running. To achieve this, we ensured, going forward, each new pull request was properly logged and did awareness on logging techniques and what a good log represents.

Some of the materials we used for the training, is this article on logging by DataSet. https://www.dataset.com/blog/the-10-commandments-of-logging/

   3. Error Monitoring Tool

To be able to leverage the extensive logging setup done, we require an efficient error monitoring tool, that would allow developers to see the errors being encountered and be able to trace and resolve the issues.

This was on top of configuring journalctl to allow a developer to ssh to the server and view the logs using journalctl -f -u backend.service

Our tool of choice here was Sentry due to more features it offered out of the box.

   4. Performance Monitoring

With all the configurations, there is a need to see the performance of various APIs, database operations and the cost of each of this processes. This also could easily be enabled on Sentry allowing us to see the queries taking the longest time, the apis commonly used and the time taken to process the requests.

On the next article, we will discuss how to utilize the setup and some of the optimizations done as a result.