The tech industry sometimes becomes fascinated by a fresh buzzword every so often, and at the moment it is observability that is having a dominant influence.<\/p>\n
If you spend a sufficient amount of time going through the way vendors market their products, you might come to the conclusion that observability is some kind of magical, costly platform which you buy and that then automatically prevents your Kubernetes clusters from ever failing. However, when you speak to the engineers who actually answer the pagers and are responsible for keeping the systems running, you’ll discover that observability is not a tool\u2014it is a basic feature of your architecture.<\/p>\n
It boils down to one simple question: when failures inevitably occur in a production environment, how hard is it to figure out why?<\/p>\n
Monitoring tells you when a system is failing, while observability lets you ask why without deploying new code to get the answer.<\/p>\n
In order to do this we make use of the “three pillars” of observability: Metrics, Logs, and Traces. Although people tend to use these terms as if they were the same in meetings, they carry out entirely different roles when there is an outage. The following explains what they are, the limitations they have, and how you are to use them together so that you won’t be tearing your hair out at 2:00 AM.<\/p>\n
Envision the metrics as the dashboard of your car; you check it to see how fast you’re travelling, how much fuel you have remaining, or whether the engine temperature is increasing.<\/p>\n
Metrics are just numerical values that are recorded over periods of time, for example CPU utilisation, memory usage, network throughput, or the number of HTTP 500 errors over the last five minutes; when the RED method (Rate, Errors, Duration) is applied, they become the basic vital signs of your application.<\/p>\n
Why we love them:<\/strong><\/p>\n Metrics function as an alarm system since they act as the threshold triggers that cause your PagerDuty alerts to be sent. Yet they have one major drawback: they let you know that something has gone wrong, but they do not provide the high-cardinality context necessary to explain why. It’s helpful to be aware that your error rate has shot up to 10 per cent, but this doesn’t tell you which specific customer caused the error or what payload they sent.<\/p>\n The metrics dashboard is displaying a red alert; the HTTP error rate has just increased on the checkout service. So what do you do? You go and check the logs.<\/p>\n Logs are made up of separate, individual units of information and serve as the permanent, time-stamped records of specific events which occur in your application code. While the dashboard displays the metrics, the logs offer the technician a comprehensive diagnostic reading.<\/p>\n A traditional log line looks something like this:<\/p>\n At 10:14:22 on 9th September 2026, there was an error when user 8942 tried to complete the transaction; the database connection was rejected on port 5432.<\/p><\/blockquote>\n The reality of logging:<\/strong><\/p>\n Logs are invaluable for debugging a local issue in a particular container; but what should you do when a user clicks a button on the frontend and that one request has to move asynchronously through six different microservices before it finally fails?<\/p>\n It is the missing link that traditional monitoring systems completely fail to offer, and it is the only means of surviving a distributed microservices architecture.<\/p>\n Imagine that a user is intending to buy a product; their request is received by the Frontend, which then contacts the Auth Service, which in turn makes a call to the Billing API, which finally communicates with the Inventory Database.<\/p>\n The user is presented with a general timeout error on their screen. When you look at the frontend logs all you see is: Error 500: Upstream timeout<\/span>. That is completely useless. At this point you have to manually check the logs of the Auth service and then those of the Billing API, having to guess where the failure actually came from.<\/p>\n Here’s where distributed tracing proves to be invaluable.<\/strong><\/p>\n A unique, global ID (a Trace ID) is assigned to each request as it enters your infrastructure, and this ID is passed on through the HTTP headers to all the downstream services that it reaches. Each unit of work carried out by a service is recorded as a “Span”.<\/p>\n Rather than guessing which service failed, you open your tracing tool, look at a visual waterfall chart, and instantly see:<\/p>\n The exact original error message isn’t always available, but the traces do take you directly to the particular service and function where the delay occurred, avoiding hours of guessing blindly.<\/p>\n What makes modern observability work is that you shouldn’t depend on just one of these pillars by itself; instead, you have to use all three as part of a highly specific and repeatable workflow. When the pager sounds in the middle of the night, the investigation generally follows this exact procedure:<\/p>\n After rotating the API token, you restart the pods and the metrics on your dashboard turn green once again immediately.<\/p>\n Observability<\/strong> is no magic trick and can’t be obtained simply by buying it off the shelf to fix bad architecture; it is just the practice of making available the right telemetry data\u2014using metrics to generate alerts, traces to isolate problems, and logs to diagnose issues\u2014so that you can answer the correct questions at the right time.<\/p>\n","protected":false},"excerpt":{"rendered":" Logs, Metrics, and Traces: Making Sense of the Observability Buzzword The tech industry sometimes becomes fascinated by a fresh buzzword every so often, and at the moment it is observability that is having a dominant influence. If you spend a sufficient amount of time going through the way vendors market their products, you might come […]<\/p>\n","protected":false},"author":1606,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":3,"footnotes":""},"categories":[2348],"tags":[1892,7501],"class_list":["post-82954","post","type-post","status-publish","format-standard","hentry","category-devops-technology","tag-devops","tag-observability"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n
2. Logs: “Precisely what happened?”<\/h2>\n
\n
3. Traces: “When exactly did the chain break?”<\/h2>\n
\n
Tying It All Together (A Real-World Triage Flow)<\/h2>\n
\n