Step 2: Create the security group and egress rule
Only one egress rule is attached to the Security Group, which enables traffic from the VPC to access the Internet (using the Internet gateway).
Amazon CloudWatch Logs Agent is a software component installed on servers that allows Cloud Engineering teams to monitor and collect log files from the servers and applications in real time. It sends log data to Amazon CloudWatch Logs, where they can be analyzed, searched, and visualized, thus making it easier to troubleshoot issues and monitor system performance. The agent uses a configuration file to specify which log files to monitor, log group and log streams to send logs, metrics collection details, append and aggregation dimensions, and specific metrics like CPU, memory, disk usage, and log file paths.
In this note, I demonstrate how to install and configure the AWS CloudWatch Logs Agent on an Amazon EC2 instance for Linux using the user data script and Terraform. The logic inside the user data script is to:
1. Download and install the Amazon CloudWatch Logs agent installer.
2. Download the config.json from the SSM Parameter store and store it locally
3. Configure and start the CloudWatch Logs agent service.
Note: If you want to examine how to install and configure the Amazon CloudWatch Logs agent on an Amazon EC2 instance for Windows using the user data script, head over to -install-and-configure-cloudwatch-logs-agent-on-amazon-ec2-instance-for-windows-using-user-data.
I separated this use case into eight high-level steps for ease of understanding. 1. Create the network components 2. Create the security group and egress rule 3. Create an Amazon KMS key and policy 4. Create an Amazon CloudWatch Logs Group 5. Create an AWS Systems Manager Parameter Store parameter 6. Create an IAM role to attach the Amazon EC2 instance 7. Update the user data script to install and configure the CloudWatch Logs agent 8. Create the Amazon EC2 instance
If you want to follow along, please refer to the Terraform code in my GitHub repository: kunduso/ec2-userdata-terraform. Please note that thebranch name = add-cloudwatch-agent-linux-ec2.
Step 1: Create the network stack
The network components include the Amazon VPC, a public subnet, a route table, an Internet gateway, and a route in the route table to the Internet using the Internet gateway.
Step 2: Create the security group and egress rule
Only one egress rule is attached to the Security Group, which enables traffic from the VPC to access the Internet (using the Internet gateway).
Step 3: Create an Amazon KMS key and policy
The AWS KMS Key encrypts the data stored in the AWS Systems Manager Parameter Store parameter and the AWS CloudWatch Logs. I provisioned a tighter KMS key policy specifically for this use case using the aws_kms_key_policy. Please note that adding the AWS KMS key, although optional, helps create a secure solution.
Step 4: Create an Amazon CloudWatch Logs Group
The logs group stores logs sent by the CloudWatch Logs agent running on the Amazon EC2 instance and is encrypted using the custom KMS key.
Step 5: Create an AWS Systems Manager Parameter Store parameter
The CloudWatch Logs agent accepts a JSON file with keys and values to configure the service. The JSON file is stored inside the AWS Systems Manager Parameter and downloaded into the Amazon EC2 instance by the logic in the user data script. The data in the parameter store is encrypted using the custom KMS key. This file is stored as a .tftpl file in the code repository since it has a variable reference to the AWS CloudWatch Log group name.
Step 6: Create an IAM role to attach the Amazon EC2 instance
I explained the logic of attaching an IAM role to an Amazon EC2 instance in my note attach-iam-role-to-amazon-ec2-instance. The IAM role attached to the Amazon EC2 instance requires an assume_role_policy. It also requires a list of managed and custom policies. These are:
(a) A custom policy that allows access to the SSM Parameter Store parameter and decrypts it using the custom KMS key.
(b) A managed policy arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy that allows the Amazon EC2 instance to send custom logs from the instance to Amazon CloudWatch.
(c) Optionally, you may attach the managed policy arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore to enable a Cloud Engineer to log into the Amazon EC2 instance using Session Manager from the AWS Cloud console for verification or debugging purposes. For details, please refer to this note -access-amazon-ec2-instance-using-session-manager.
Step 6: Update the user data script to install and configure the CloudWatch Logs agent
The user data script is located as a template file in the user_data folder. The script is configured to download and install the Amazon CloudWatch Logs agent. Then, it connects to the SSM Parameter Store parameter, downloads the parameter value, decrypts it using the custom KMS key, and stores it as a JSON file on the Amazon EC2 instance. Then, it configures the Amazon CloudWatch Logs agent by passing the JSON file and starts the service.
Step 7: Create the Amazon EC2 instance
The last step is to create the Amazon EC2 instance with the properties specified in the previous steps, like the security group, the subnet ID, the instance profile, and the user data script. As shown in the image below, the user data script accepts an SSM Parameter Store parameter name. This parameter is referenced in the user data script to download the JSON configuration file for the Amazon CloudWatch Logs agent. I also enabled the metadata_options {} to follow best practices, enhancing security and control over instance metadata access.
After Terraform provisioned all the resources, I waited a few minutes before accessing the Amazon CloudWatch log group from the AWS Console. That was sufficient time for the Amazon EC2 instance user data script to download the installers, install and configure the CloudWatch Logs agent service, and send the logs to CloudWatch Logs.
Best Practices I implemented a few best practices, such as: 1. Attaching an IAM role with Amazon EC2 instance. 2. Using least privilege IAM permissions. 3. No ingress rule for any IP address on any ports. 4. Automate the installation process via the idempotent user data PowerShell script. 5. Use the IMDv2 metadata options in Amazon EC2 to enhance security and control over instance metadata access. 6. Attach a secure KMS key policy instead of using the default key policy 7. Encrypt the SSM Parameter Store parameter and CloudWatch Logs using KMS Key.
However, there is one more best practice to consider when creating an Amazon EC2 instance: Remove public IP address: The Amazon EC2 instance has a public IP address since it requires Internet access to download the CloudWatch Logs agent installer. However, suppose the installer was stored in an Amazon S3 bucket (such as the Amazon CloudWatch agent installer). We could use a VPC Endpoint to access the installer in that case. That way, the EC2 instance won’t require a public IP address. It is a security best practice to avoid having Amazon EC2 instances with public IP addresses because they can be reached from the Internet, making them vulnerable to attacks. Please note that I could have used a NAT gateway in this use case to prevent the Amazon EC2 instance from having a public IP address to access the Internet, but that would have deviated from the primary use case.
And that concludes this note. If you have any questions or suggestions, please do not hesitate to contact me via the comments below.