AWS CloudWatch: Set Retention Periods for Log Groups

Amazon CloudWatch is a monitoring service that provides users with insights and other data related to their infrastructure and applications. With these insights, users can then optimize the performance of Amazon Web Services (AWS) applications, manage resources, and monitor the health of their system.

Aggregated logs, metrics, and events are the basis for these CloudWatch insights. Logs are created by everything from your operating systems to the applications running on your AWS instance. As your workload grows, so will your number of log files. When you have more log files stored in your AWS instance than you need, then you have a chance to optimize your cloud costs.

In this guide, we’ll briefly talk about CloudWatch logs and log groups, and then show how you can ensure that log groups have retention periods set to limit AWS costs.

Blink Automation: Ensure AWS CloudWatch Log Groups Have Set Retention Periods

Blink + AWS

Try This Automation

CloudWatch Logs and Log Groups

CloudWatch logs let you use one service to centralize logs from all systems and AWS services. That makes it easier to:

• View logs
• Find specific patterns and error codes
• Filter specific fields
• Archive logs for future use

For example, you can use CloudWatch logs to keep up with errors found in application logs of your Amazon EC2 instances. Specify a threshold for an acceptable error rate and receive a notification once the number of errors passes that boundary. Because CloudWatch logs use existing log data, you can configure this without needing to make any code changes.

CloudWatch Log Groups

All logs are presented as a consistent event flow ordered by the time they occur. You can query and sort CloudWatch logs, group them based on certain fields, use query language to set up customized computations, and present log data within a dashboard.

Log groups are collections of log streams, which are individual sequences of log events from a shared source. Each log group shares access, monitoring, and retention control settings. Users can define log groups and control how to assign streams to a log group.

Finding Log Groups Without a Set Retention Period

You can get the details of an Amazon CloudWatch Log Group to verify it has a retention period set. To do this, run the following command with AWS CLI:

aws logs describe-log-groups     
    --log-group-name-prefix my-log-group

CloudWatch log groups do not have a retention period set by default, so log events are retained forever. When the retention period is set, the log group details contain the attribute "retentionInDays". You can see an example of this output below :

{
    "logGroups": [        
        {            
            "storedBytes": 0,            
            "metricFilterCount": 0,            
            "creationTime": 1433189500783,            
            "logGroupName": "my-logs",            
            "retentionInDays":  90,            
            "arn": "arn:aws:logs:us-west-2:0123456789012:log-group:my-logs:*"        
        }    
    ]
}

Using jq with AWS CLI you can get the list, for your current region, of CloudWatch log groups without retention period with the following command:

aws logs describe-log-groups --output json | jq '.logGroups | 
map(select(has(retentionInDays) == false))'

Once you have found the groups that are missing retention periods, next you need to add a retention policy.

Setting Retention Periods for CloudWatch Log Groups

You can set a retention policy that specifies the number of days to retain log events within a log group by using the following syntax in the AWS CLI:

aws logs put-retention-policy     
    --log-group-name my-curr-aws-logs     
    --retention-in-days 10 

Once you have run this command, you can again run the “describe-log-groups” command to verify that this change has taken place. This small change can instantly start saving your organization money by capping log storage.

Automating this AWS CloudWatch Check with Blink

If you are running AWS CloudWatch, checks like this are important to keep your logs from becoming too costly. While you can run optimizations ad hoc, there’s a better way to get a handle on your AWS costs.

With Blink, you can schedule automations like this one to run regularly and locate unneeded resources and comply with best practices.

When this cost optimization automation runs, it executes the following steps:

1. Checks for Log Groups with retention period longer than 365 days or none set.
2. Sends all non-compliant Log Groups in a report via email.

You can use and customize any of the 5K automations in the Blink library, or build automations from scratch to fit your unique needs.

Get started with Blink today to see how easy automation can be.