Maximizing Your Logs with CloudWatch Logs Insights – News Block

When you run your application, it generates logs that record pertinent information about activity or event logs. These logs, which can contain essential information about your app, your users, and everything in between, can be incredibly beneficial sources of information. Effective logs allow you to stop making assumptions about what’s going on in your code and instead focus on making informed decisions. However, the data is practically useless if it is not stored, searched and analyzed correctly.

In this post, I’ll provide a brief overview of good logging practices and examples of how you can use CloudWatch Logs Insights, a program designed to help you interactively find and analyze your log data, to better consume the logs you generate for your application.

Logs can be much more efficient than other means of getting data from your application. For example, with effective logs, you can generate a report, send an email, or debug via the console in one step. Log lines are always tracking, so assuming you set them up correctly, you can use them to get the data you need very quickly.

Logging is ideal for high-traffic endpoints because logging does not require a write to the database at any time. It’s also an easy way to provide information about models and concerns, all in one line.

You typically log in at two main points during your app’s lifecycle:

1. Local Lumber for Development: These logs are produced during development for simple verifications such as print debugging. As a result, longer registration functions are not normally required.

2. Logger implemented for debugging– These logs give you the ability to sculpt the data in real time to make intelligent decisions about what’s actually going on in your application.

When you log into a deployed environment, you want to make sure that you’re using a universal logging feature to standardize your output. If you’re doing a test deployment for a short-term problem and it’s not going to be a permanent fixture in your code, you can place the logs however you see fit. However, for anything you want to monitor long-term by leaving a logger in your codebase, you need to make sure you have the ability to turn it on and off (for example, logger(…) if App.log_x_thing_setting).

If you want to maximize the benefits your logline provides, consider following some of these best practices:

• Strive to convey one logical evaluation per inserted record. Don’t clutter up your records by including lots of data that isn’t relevant to the same topic.
• Be thorough. When you are investigating a problem, make sure that all possible relevant logic gates are covered.
• Try to keep the loggers under logic gates if there are multiple streams.
• You prefer too many individual loggers to fewer, densely packed log lines.
• Use prefixes to make it easier to find specific records or groups of records.
• When comparing the same keys across multiple log lines, make sure the naming and placement are consistent within the line to increase expressiveness.

When I build a logline, I usually include three main sections:

1. Prefix: A unique descriptor for all loglines on the topic (for example, USER_PROFILE:…)
2. Description: A unique descriptor for a single logical expression (for example, POST_EMAIL: would direct us to any information that has been posted to the server’s emails).
3. Key/value strategy: A key or value strategy for everything else in the logline. I try to follow a key value strategy where I include the key, followed by a colon delimiter, followed by the value. I also often use a trailing delimiter to make it easy to tell when I’m using multiple record lines together (eg EMAIL_POST: #{params{’email’} – EXISTING_EMAIL: #{current_user.email}}).

When building loglines, it’s also important to avoid pitfalls, such as:

• Multiple Identifiers: It’s important to separate multiple identifiers, especially when you include sensitive data such as a user’s ID, email, and username. In the event records are ever leaked, you don’t want a user’s sensitive data stored together. When storing IDs, try to replace them with a hash_identifier whenever possible. For example, you can create a unique identifier using multiple data points (eg “#{self.id}#{self.salt}”) so you don’t directly record IDs.
• Identical loglines in multiple locations: Don’t just copy and paste the same log line with the same variable output to four different points within the stream. Always make sure there is some way to tell each logline apart so they are easy to tell apart.
• Registration of line numbers to determine the location: As the code is modified, the line numbers are constantly changing. So if you identify loglines based on line numbers, the references will no longer be relevant. If in doubt, use the naming convention discussed above.
• Full Secrets Log: If you absolutely must commit secret values ​​in production, be sure to obfuscate as much of the key as possible. For example, if it is a long sequence of numbers, reveal only the last identifying digits.
• User credentials record: Most platforms have automatic filters that you can use, like the Rails Params method, to filter credentials, passwords, access tokens, etc. by default. However, if you’re not careful, you could still register them with another registrar. You need to be aware when you’re logging credentials to ensure sensitive data doesn’t bypass the filters that naturally protect it.

The following code is a snippet from a project I’ve been working on recently. We decided to quickly create some logs to understand what happens to which users, in which places, at what times, and in what states. The initial list of records looked like this:

Although our initial log lines were very complete, there were some issues with the formatting:

• Inconsistent spacing: If you review the image above, you will notice that there are missing spaces. These spacing issues make it difficult to quickly review and use the records.
• Inconsistent variable placement: For example, all of the records above are prefixed with “ATS”, but then “Subdomain” is either in different places or is missing altogether.
• Registered inconsistent variable: For example, the third record in the image above includes the gdpr_consent variable that is not included in any of the other records. When we start consuming these logs, this kind of inconsistency will make it much harder to compare them.

Record set extracted (fixed)

To fix the issues mentioned above, I extracted all the logs from the file, put them together on one line, and refactored them to be high-quality log lines:

As you can see, these lines now have constant spacing, prefixes, and registered variables. When dealing with a massive lineup of records, creating this kind of consistency makes it much easier to quickly scan the list and see what stands out.

You don’t necessarily need to generate loglines to consume them. Most frameworks generate their own logs that you can use to your advantage.

For example, CloudWatch Logs Insights gives you valuable information about your application:

Some of the CloudWatch Logs Insights components include:

• Range: In the top right corner, you can set the range of records you’d like to search for. You can use absolute or relative ranges, and you can choose a fixed range or search based on certain dates.
• Log line dropdown: Below the header is a dropdown containing each log line in your project. You can filter through the log lines to choose only the ones you’d like to analyze.
• Query parameters: Below the dropdown list is the query field. Here you can define the parameters of your query. Using @logstream in your query parameters will give you a link to view the log stream in context with the full log set.
• Query results: Once you complete your query, you can view your results in the space below.

When using logline information, it’s best to narrow down the query set as much as possible with your initial filter clauses to prevent your query from returning an overwhelming number of results.

Often times, the success of an app is determined by the developer’s ability to identify user trends and issues and fine-tune the app to maximize the user experience. Whether you’re releasing a new app, fixing a bug, making small changes, or completing a full overhaul, log lines can help you make smart, informed decisions as you go.