# Upload device-side logs to Amazon CloudWatch
You can upload historical, device-side logs into Amazon CloudWatch to monitor and analyze a device's activity in the field. Device-side logs can include system, application, and device logs files. This process uses a CloudWatch Logs rules action parameter to publish device-side logs into a customer-defined [log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html).
## How it works
The process begins when an AWS IoT device sends MQTT messages containing formatted log files to an AWS IoT topic. An AWS IoT rule monitors the message topic and sends the log files to a CloudWatch Logs group that you define. You can then review and analyze the information.
**Topics**
+ [MQTT topics](#upload-mqtt-topics-overview)
+ [Rule action](#upload-rule-action-overview)
### MQTT topics
Choose an MQTT topic name space that you will use to publish the logs. We recommend using this format for the common topic space, `$aws/rules/things/thing_name/logs`, and this format for error topics, `$aws/rules/things/thing_name/logs/errors`. The naming structure for logs and error topics is recommended, but not required. For more information, see [Designing MQTT Topics for AWS IoT Core](https://docs.aws.amazon.com/whitepapers/latest/designing-mqtt-topics-aws-iot-core/designing-mqtt-topics-aws-iot-core.html).
By using the recommended common topic space, you utilize AWS IoT Basic Ingest reserved topics. AWS IoT Basic Ingest securely sends device data to the AWS services that are supported by AWS IoT rule actions. It removes the publish/subscribe message broker from the ingestion path, making it more cost effective. For more information, see [Reducing messaging costs with Basic Ingest](https://docs.aws.amazon.com/iot/latest/developerguide/iot-basic-ingest.html).
If you use batchMode to upload log files, your messages must follow a specific format that includes a UNIX timestamp and message. For more information, see the [MQTT message format requirements for batchMode](https://docs.aws.amazon.com/iot/latest/developerguide/cloudwatch-logs-rule-action.html#cloudwatch-logs-rule-action-message-format) topic within [CloudWatch Logs rule action](https://docs.aws.amazon.com/iot/latest/developerguide/cloudwatch-logs-rule-action.html).
### Rule action
When AWS IoT receives the MQTT messages from the client devices, an AWS IoT rule monitors the customer-defined topic and publishes the contents into a CloudWatch log group that you define. This process uses a CloudWatch Logs rule action to monitor MQTT for batches of log files. For more information, see the [CloudWatch Logs](https://docs.aws.amazon.com/iot/latest/developerguide/cloudwatch-logs-rule-action.html) AWS IoT rule action.
#### Batch mode
`batchMode` is a Boolean parameter within the AWS IoT CloudWatch Logs rule action. This parameter is optional and is off (`false`) by default. To upload device-side log files in batches, you must turn this parameter on (`true`) when you create the AWS IoT rule. For more information, see [CloudWatch Logs](https://docs.aws.amazon.com/iot/latest/developerguide/cloudwatch-logs-rule-action.html) in the [AWS IoT rule actions](https://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.html) section.
# Uploading device-side logs by using AWS IoT rules
You can use the AWS IoT rules engine to upload log records from existing device-side log files (system, application, and device-client logs) to Amazon CloudWatch. When device-side logs are published to an MQTT topic, the CloudWatch Logs rules action transfers the messages to CloudWatch Logs. This process outlines how to upload device logs in batches using the rules action `batchMode` parameter turned on (set to `true`).
To begin uploading device-side logs to CloudWatch, complete the following prerequisites.
## Prerequisites
Before you begin, do the following:
+ Create at least one target IoT device that's registered with AWS IoT Core as an AWS IoT thing. For more information, see [Create a thing object](https://docs.aws.amazon.com/iot/latest/developerguide/create-iot-resources.html#create-aws-thing).
+ Determine the MQTT topic space for ingestion and errors. For more information about MQTT topics and recommended naming conventions, see the [MQTT topics](upload-device-logs-to-cloudwatch.md#upload-mqtt-topics-overview) [MQTT topics](https://docs.aws.amazon.com/iot/latest/developerguide/upload-device-logs-to-cloudwatch.html#upload-mqtt-topics-overview) section in [Upload device-side logs to Amazon CloudWatch](https://docs.aws.amazon.com/iot/latest/developerguide/upload-device-logs-to-cloudwatch.html).
For more information about these prerequisites, see [Upload device-side logs to CloudWatch](https://docs.aws.amazon.com/iot/latest/developerguide/upload-device-logs-to-cloudwatch).
## Creating a CloudWatch log group
To create a CloudWatch log group, complete the following steps. Choose the appropriate tab depending on whether you prefer to perform the steps through the AWS Management Console or the AWS Command Line Interface (AWS CLI).
------
#### [ AWS Management Console ]
**To create a CloudWatch log group by using the AWS Management Console**
1. Open the AWS Management Console and navigate to [CloudWatch](https://console.aws.amazon.com//cloudwatch).
1. On the navigation bar, choose **Logs**, and then **Log groups**.
1. Choose **Create log group**.
1. Update the **Log group name** and, optionally, update the **Retention setting** fields.
1. Choose **Create**.
------
#### [ AWS CLI ]
**To create a CloudWatch log group by using the AWS CLI**
1. To create the log group, run the following command. For more information, see ` [create-log-group](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/create-log-group.html) ` in the AWS CLI v2 Command Reference.
Replace the log group name in the example (`uploadLogsGroup`) with your preferred name.
```
aws logs create-log-group --log-group-name uploadLogsGroup
```
1. To confirm that the log group was created correctly, run the following command.
```
aws logs describe-log-groups --log-group-name-prefix uploadLogsGroup
```
Sample output:
```
{
"logGroups": [
{
"logGroupName": "uploadLogsGroup",
"creationTime": 1674521804657,
"metricFilterCount": 0,
"arn": "arn:aws:logs:us-east-1:111122223333:log-group:uploadLogsGroup:*",
"storedBytes": 0
}
]
}
```
------
## Creating a topic rule
To create an AWS IoT rule, complete the following steps. Choose the appropriate tab depending on whether you prefer to perform the steps through the AWS Management Console or the AWS Command Line Interface (AWS CLI).
------
#### [ AWS Management Console ]
**To create a topic rule by using the AWS Management Console**
1. Open the Rule hub.
1. Open the AWS Management Console and navigate to [AWS IoT](https://console.aws.amazon.com/iot) .
1. On the navigation bar, choose **Message routing** and then **Rules**.
1. Choose **Create rule**.
1. Enter the rule properties.
1. Enter an alphanumeric **Rule name**.
1. (Optional) Enter a **Rule description** and **Tags**.
1. Choose **Next**.
1. Enter a SQL statement.
1. Enter a SQL statement using the MQTT topic that you defined for ingestion.
For example, `SELECT * FROM '$aws/rules/things/thing_name/logs' `
1. Choose **Next**.
1. Enter rule actions.
1. On the **Action 1** menu, choose **CloudWatch logs**.
1. Choose the **Log group name** and then choose the log group that you created.
1. Select **Use batch mode**.
1. Specify the IAM role for the rule.
If you have an IAM role for the rule, do the following.
1. On the **IAM role** menu, choose your IAM role.
If you don't have an IAM role for the rule, do the following.
1. Choose **Create new role**.
1. For **Role name**, enter a unique name and choose **Create**.
1. Confirm that the IAM role name is correct in the **IAM role** field.
1. Choose **Next**.
1. Review the template configuration.
1. Review the settings for the Job template to verify they're correct.
1. When you're done, choose **Create**.
------
#### [ AWS CLI ]
**To create an IAM role and a topic rule by using the AWS CLI**
1. Create an IAM role that grants rights to the AWS IoT rule.
1. Create an IAM policy.
To create an IAM policy, run the following command. Make sure you update the `policy-name` parameter value. For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-policy.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-policy.html) in the AWS CLI v2 Command Reference.
**Note**
If you're using a Microsoft Windows operating system, you might need to replace the end of line marker (\$1) with a tick (`) or another character.
```
aws iam create-policy \
--policy-name uploadLogsPolicy \
--policy-document \
'{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"iot:CreateTopicRule",
"iot:Publish",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:GetLogEvents"
],
"Resource": "*"
}
}'
```
1. Copy the policy ARN from your output into a text editor.
Sample output:
```
{
"Policy": {
"PolicyName": "uploadLogsPolicy",
"PermissionsBoundaryUsageCount": 0,
"CreateDate": "2023-01-23T18:30:10Z",
"AttachmentCount": 0,
"IsAttachable": true,
"PolicyId": "AAABBBCCCDDDEEEFFFGGG",
"DefaultVersionId": "v1",
"Path": "/",
"Arn": "arn:aws:iam::111122223333:policy/uploadLogsPolicy",
"UpdateDate": "2023-01-23T18:30:10Z"
}
}
```
1. Create an IAM role and trust policy.
To create an IAM policy, run the following command. Make sure you update the `role-name` parameter value. For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-role.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/create-role.html) in the AWS CLI v2 Command Reference.
```
aws iam create-role \
--role-name uploadLogsRole \
--assume-role-policy-document \
'{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "iot.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
```
1. Attach the IAM policy to the rule.
To create an IAM policy, run the following command. Make sure you update the `role-name` and `policy-arn` parameter values. For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/attach-role-policy.html) in the AWS CLI v2 Command Reference.
```
aws iam attach-role-policy \
--role-name uploadLogsRole \
--policy-arn arn:aws:iam::111122223333:policy/uploadLogsPolicy
```
1. Review the role.
To confirm that the IAM role was created correctly, run the following command. Make sure you update the `role-name` parameter value. For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/get-role.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iam/get-role.html) in the AWS CLI v2 Command Reference.
```
aws iam get-role --role-name uploadLogsRole
```
Sample output:
```
{
"Role": {
"Path": "/",
"RoleName": "uploadLogsRole",
"RoleId": "AAABBBCCCDDDEEEFFFGGG",
"Arn": "arn:aws:iam::111122223333:role/uploadLogsRole",
"CreateDate": "2023-01-23T19:17:15+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Principal": {
"Service": "iot.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Description": "",
"MaxSessionDuration": 3600,
"RoleLastUsed": {}
}
}
```
1. Create an AWS IoT topic rule in the AWS CLI.
1. To create an AWS IoT topic rule, run the following command. Make sure you update the `--rule-name`,`sql` statement, `description`,`roleARN `, and `logGroupName` parameter values. For more information, see [create-topic-rule](https://awscli.amazonaws.com/v2/documentation/api/2.0.34/reference/iot/create-topic-rule.html) in the AWS CLI v2 Command Reference.
```
aws iot create-topic-rule \
--rule-name uploadLogsRule \
--topic-rule-payload \
'{
"sql":"SELECT * FROM 'rules/things/thing_name/logs'",
"description":"Upload logs test rule",
"ruleDisabled":false,
"awsIotSqlVersion":"2016-03-23",
"actions":[
{"cloudwatchLogs":
{"roleArn":"arn:aws:iam::111122223333:role/uploadLogsRole",
"logGroupName":"uploadLogsGroup",
"batchMode":true}
}
]
}'
```
1. To confirm that the rule was created correctly, run the following command. Make sure you update the `role-name` parameter value. For more information, see [get-topic-rule](https://awscli.amazonaws.com/v2/documentation/api/2.0.34/reference/iot/get-topic-rule.html) in the AWS CLI v2 Command Reference.
```
aws iot get-topic-rule --rule-name uploadLogsRule
```
Sample output:
```
{
"ruleArn": "arn:aws:iot:us-east-1:111122223333:rule/uploadLogsRule",
"rule": {
"ruleName": "uploadLogsRule",
"sql": "SELECT * FROM rules/things/thing_name/logs",
"description": "Upload logs test rule",
"createdAt": "2023-01-24T16:28:15+00:00",
"actions": [
{
"cloudwatchLogs": {
"roleArn": "arn:aws:iam::111122223333:role/uploadLogsRole",
"logGroupName": "uploadLogsGroup",
"batchMode": true
}
}
],
"ruleDisabled": false,
"awsIotSqlVersion": "2016-03-23"
}
}
```
------
## Sending device-side logs to AWS IoT
**To send device-side logs to AWS IoT**
1. To send historical logs to AWS IoT, communicate with your devices to ensure the following.
+ The log information is sent to the correct topic namespace as specified within the *Prerequisites* section of this procedure.
For example, `$aws/rules/things/thing_name/logs`
+ The MQTT message payload is formatted correctly. For more information about MQTT topic and recommended naming convention, see the [MQTT topics](upload-device-logs-to-cloudwatch.md#upload-mqtt-topics-overview) section within [Upload device-side logs to Amazon CloudWatch](upload-device-logs-to-cloudwatch.md) .
1. Confirm that the MQTT messages are received within the AWS IoT MQTT client.
1. Open the AWS Management Console and navigate to [AWS IoT](https://console.aws.amazon.com/iot/home).
1. To view the **MQTT test client**, on the navigation bar, choose **Test**,**MQTT test client**.
1. For **Subscribe to a topic**,**Topic filter**, enter the *topic namespace*.
1. Choose **Subscribe**.
MQTT messages appear in the **Subscriptions** and **Topic** table, as seen in the following. These messages can take up to five minutes to appear.
![\[MQTT messages appearing in the Subscriptions and Topic table.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/uploading-logs-rules-start-messages-sample-mqtt.png)
## Viewing the log data
**To review your log records in CloudWatch Logs**
1. Open the AWS Management Console, and navigate to [CloudWatch](https://console.aws.amazon.com/cloudwatch).
1. On the navigation bar, choose **Logs**,**Logs Insights**.
1. On the **Select log group(s)** menu, choose the log group you specified in the AWS IoT rule.
1. On the **Logs insights** page, choose **Run query**.