One of the useful enhancements coming up in Spring Boot 3.4 is the out-of-the-box support for structured logging with support for Elastic Common Schema, Logstash and our own formats. While we could already do this by adding dependent libraries such as SLF4J and Log4J beforehand, it’s nice to see this now being available out of the box. There’s certainly something to be said for having fewer dependencies that you have to manage, and which may potentially have vulnerabilities¹.
To take advantage of this new feature we merely need to add
logging.structured.format.console=ecs
In case we desire to use ECS, which will enable structured logging for us and send it to our console.
Why does structured logging matter?
Our applications are growing increasingly complex, and alongside that the business criticality of these. The days when our clients could easily pull out their binders and notepads are (thankfully) long gone. This means that we need to know what’s happening within our applications both pro- and retroactively is thus critical. Additionally, scaling up and out can be a challenging task when we’re groping blindly, thus we need proper insights into potential bottlenecks.
Also, it’s sadly a fact that much too often developers are kept far away from more detailed observability insights, which in my honest opinion should be brought into play much earlier in the process. Ideally one should even have full tooling on their own development machine which helps them out with observability insights, so the impact of their changes can be, well observed. It’s why using tooling such as Digma, which easily integrates with your IDE and captures, processes and points out certain key issues captured by OpenTelemetry can be vital.
Logging itself is one of the three pillars of observability alongside metrics and traces. It’s vital that we capture this data, so we can see what’s going on in our application and swiftly react to incidents. Especially given we generally use a more distributed system architecture, which also requires us to enable distributed tracing so that our trace- and span-id are also automatically logged, which facilitates log correlation over multiple services. Logs are an indispensable tool for this, given they’re generally made in a human-readable format, thus making them easily and swiftly parsable by us without the need for extra tools.
By making use use a structured logging format such as JSON we can feed them to a log management system.
For reference, such a log entry could look like:
{ "@timestamp": "2024-09-19T13:07:01.407Z", "log.level": "INFO", "process.pid":67457, "process.thread.name":"main", "service.name":"digma-structured-logging-demo", "log.logger":"com.digma.demo.structured_logging.StructuredLoggingDemoApplication", "message":"Started StructuredLoggingDemoApplication in 0.287 seconds (process running for 0.445)", "ecs.version":"8.11" }
Connecting the dots
When using OpenTelemetry (OTel) we can link our logs to specific parts of an application’s functionality.
This is achieved by making each log entry uniquely identifiable by adding a tag which consists of a trace ID and a span ID. The trace ID uniquely identifies your request through your whole system, while the span ID identifies a specific operation within the trace.
We can achieve this traceability by getting the IDs from our current OTel span and adding them to each log message we create. This way we can easily identify a specific flow through the whole system, which facilitates debugging performance issues and troubleshooting.
More information on Structured Logging in Spring Boot 3.4
It is quite likely that you’ll want to include more information, and luckily this can be achieved easily by:
- Adding more information to the Mapped Diagnostic Context:
MDC.put("organization", "Digma"); LOGGER.info("Thank you for reading this blog!"); MDC.remove("organization");
- Making use of the fluent logging application
LOGGER.atInfo().setMessage("Thank you for reading this blog!").addKeyValue("organization", "Digma").log();
- Implementing our own StructuredLogFormatter and referencing it in our application properties.
class DigmaDemoLoggingFormatter implements StructuredLogFormatter<ILoggingEvent> { private final JsonWriter<ILoggingEvent> writer = JsonWriter.<ILoggingEvent>of((members) -> { members.add("time", (event) -> event.getInstant()); members.add("level", (event) -> event.getLevel()); members.add("thread", (event) -> event.getThreadName()); members.add("message", (event) -> event.getFormattedMessage()); }).withNewLineAtEnd(); @Override public String format(ILoggingEvent event) { return this.writer.writeToString(event); } }
Tips: Get the most out of your Structured logging in Spring Boot 3.4
There are certain best practices you ideally keep in mind to get the most mileage out of your logging:
- Decide upon a consistent logging format within the very least your application, and ideally your organization.
- Include all the relevant context and metadata for all involved stakeholders, so properly align. You really don’t want to let another team in the chain down when they need a hand pinpointing something. Cross-team communication is key, development should not be a silo!
- Use a standardized date format, we’ve all encountered our challenges with dates: what’s the format (dd/mm, mm/dd), in which timezone, and so on.
- Make sure no sensitive data is logged, we don’t want a GDPR lawsuit hiding in the corner!
- Be concise, vital information should be logged, but do not go over the top. You do not want to impact the performance by causing too much I/O, or making key information hard to spot.
If you’ve applied the above, you should invest in proper monitoring and parsing of these logs. That way we can set up monitoring, automated alerts and even possible automated actions.
A multitude of use cases
While print statements are certainly useful for tracking bugs, there is so much more that can be achieved with these logs.
- Following a consumer path through our system based upon the traceId to get insights into how, and by who/what our services are being consumed
- Tracing where an oddity happened during our flow in a distributed system
- Gather user insights: which elements are frequently interacted with, are there certain things that we should cache for example? Or on the other hand, is the cache called so infrequently that we’re just wasting resources?
- Audit possible vulnerabilities, we know security audits can be a challenge, so make sure you have the required information at hand
Thoughts
Logging is just the start of gathering insights into your applications, and we need to let go of the bad habit of only looking at them reactively. We need to parse as much data as possible to proactively improve the quality of our applications. Ideally, bring observability forward as part of your development process, this is not merely an operations matter! We all need to do our part to enhance our application. Given this, I always advocate applying tooling like Digma to help parse OTel information to garnish insights.
In case you want to dive deeper, the Spring team has already updated the Structured logging documentation.
References
¹ Log4Shell (CVE-2021-44228) – https://nvd.nist.gov/vuln/detail/CVE-2021-44228
Getting in Touch
I’m always happy to receive feedback/questions or just an interesting conversation.
In case you want to reach out I can be found at:
- https://www.linkedin.com/in/simonverhoeven/
- https://github.com/simonVerhoeven/
- https://twitter.com/Simon_Verhoeven