Monitoring and Observability
Market competitiveness relies on two foundational aspects: understanding market demands and being able to react to those demands with little delay. The majority of this book deals with the second topic. We covered the need for autonomous delivery practices for teams and applicable strategies for testing changes, validating preview builds, and delivering updates quickly yet responsibly.
The topic of understanding - or downright predicting - current market demands combines empirical market research, heuristic feedback loops, and data-driven product decisions. Starting from a solid base of a secure, stable, and responsive application, we may begin the prophetic process of building towards organizational growth.
Through monitoring and observability, we gather information about our software in production. Implementing strategic monitoring and observability allows us to react to errors, analyze system performance, enhance our customer's journeys, and evaluate feature acceptance. Armed with this information, we can make meaningful decisions for our future development.
Monitoring
Users who have lost trust in our offering are costly to reengage with our product. Our brand value decreases exponentially over the time our product is unresponsive. Hence, an unusable or unresponsive system reduces immediate and future revenue. To minimize downtime, we monitor our system health by collecting data from various sources. Crucial detriments to core systems require our immediate attention.
Monitoring endpoints of online services have converged into so-called zPages, popular ones being Healthz, RPCz, Statsz, and Tracez. Querying live data enables us to collect information about the health of our software. The prevalence of zPages across the industry assists us in effectively monitoring third-party products within our systems.
Using these endpoints, we gather runtime aspects such as response time, CPU usage, and memory allocation. When these metrics reach a predetermined adverse threshold, our system sends out alerts to our on-call personnel. In these kinds of emergencies, the sole priority is to get the production system back up and running. We have time to discover and fix the error in relative peace after our organization stops hemorrhaging revenue.
False positives or false negatives depreciate trust in our monitoring system posthaste as alerts in production may happen at any time. If we're woken up at 3 a.m. for a severe alert that turns out to be a false positive, we'll be pretty upset. A high volume of low-impact alerts leads to fatigue and desensitization. Desensitized on-call engineers are more likely to mis-prioritize high-impact alerts.
We continuously adjust our alert system to balance immediacy and volume.
Observability
Quote
"What gets measured gets managed."
- Peter Drucker
Our monitoring tools actively inform us about occurring problems in our software. Our observability tools enable us to investigate our software's behavior during those problems. The foundation of observability is telemetry data and the relationship between data sources. The complexities of observability have become a priority conversation within distributed architectures. The complexities of information aggregation across distributed sources have spawned a multitude of tools.
Opposed to monitoring third-party systems, observing these proves to be difficult. When observing systems, we request data and knowledge of internal processes, which, if not provided by the original authors, are hard to come by. Our industry has consolidated around a data standard called OpenTelemetry. The standard allows us to read and evaluate telemetry data across our proprietary tech and cross-vendor services. The three pillars of telemetry data consist of logs, metrics, and traces.
Logs
Logs are a collection of information serialized as human-readable text providing insight into the happenings of our software. Our developers print output when events happen or certain code procedures are called. Logs are typically either written to a file or collected from stdout and stderr. Across distributed products, we face challenges when aggregating logging information. We need tools to centralize logs across all machines or instances. Logs serialized solely to our customer's machine prove useless to us when investigating errors.
We design logging schemas (or data models) to correlate metadata and context to our message output. An open-source schema called OpenTelemetry has found favor across a broad spectrum of technical fields. For interoperability, OpenTelemetry logs in a standardized parsable format, typically json. A common meta field is the severity level of log entries, which indicates the gravity of the stored information. The OpenTelemetry data model defines the following events, ordered in ascending severity.
| Level | Meaning |
|---|---|
| TRACE | A fine-grained debugging event. Typically disabled in default configurations. |
| DEBUG | A debugging event. |
| INFO | An informational event. Indicates that an event happened. |
| WARN | A warning event. Not an error but more important than an informational event. |
| ERROR | An error event. Something went wrong. |
| FATAL | A fatal error such as an application or system crash. |
Besides giving context, log levels allow us to disable messages below a certain threshold. Filtering logging output reduces the verbosity of our application and thus increases our runtime performance. Logging - while helpful for observability and debugging - brings a significant performance overhead with it.
When building and deploying for Release, we disable DEBUG logs by default. The necessity of printing application state and data for helpful debugging sessions is at odds with the discipline of security and confidentiality for production logs. Regardless of log level, we never print sensitive information.
To further decrease the performance needs of logging, we introduce log sampling. Our logging implementation collects repeated information over our sampling time, leading to lower storage requirements and less performance overhead. The amount of storage needed for application logs escalates rapidly. The effort of sampling logs and eliminating duplicate entries becomes worthwhile in a short time. We log retention length for different entries to store the smallest number needed. Legal compliance forces us to retain certain logs for either a minimum or maximum amount of time.
-
Good read? Unlock the rest of the chapter!
Engineering Collaboration is currently available as an Advanced Reading Copy for select readers.