# Device Advisor test cases
Device Advisor provides prebuilt tests in six categories.
+ [TLS](device-advisor-tests-tls.md)
+ [MQTT](device-advisor-tests-mqtt.md)
+ [Shadow](device-advisor-tests-shadow.md)
+ [Job execution](device-advisor-tests-job-execution.md)
+ [Permissions and policies](device-advisor-tests-permissions-policies.md)
+ [Long duration tests](device-advisor-tests-long-duration.md)
## Device Advisor test cases to qualify for the AWS Device Qualification Program.
Your device must pass the following tests to qualify according to the [AWS Device Qualification Program](https://aws.amazon.com/partners/programs/dqp/).
**Note**
This is a revised list of the qualification tests.
+ [TLS Connect](device-advisor-tests-tls.md#TLS_Connect) ("TLS Connect")
+ [TLS Incorrect Subject Name Server Cert](device-advisor-tests-tls.md#TLS_Incorrect_Subject_Name) ("Incorrect Subject Common Name (CN) / Subject Alternative Name (SAN)")
+ [TLS Unsecure Server Cert](device-advisor-tests-tls.md#TLS_Unsecure_Server_Cert) ("Not Signed By Recognized CA")
+ [TLS Device Support for AWS IoT Cipher Suites](device-advisor-tests-tls.md#TLS_DeviceSupport_For_IOT) ("TLS Device Support for AWS IoT recommended Cipher Suites")
+ [TLS Receive Maximum Size Fragments](device-advisor-tests-tls.md#TLS_MaximumSize)("TLS Receive Maximum Size Fragments")
+ [TLS Expired Server Cert](device-advisor-tests-tls.md#TLS_Expired_Server_Cert)("Expired server certificate")
+ [TLS Large Size Server Cert](device-advisor-tests-tls.md#TLS_LargeServerCert)("TLS large Size Server Certificate")
+ [MQTT Connect](device-advisor-tests-mqtt.md#MQTT_Connect) ("Device send CONNECT to AWS IoT Core (Happy case)")
+ [MQTT Subscribe](device-advisor-tests-mqtt.md#MQTT_Subscribe) ("Can Subscribe (Happy Case)")
+ [MQTT Publish](device-advisor-tests-mqtt.md#MQTT_Publish) ("QoS0 (Happy Case)")
+ [MQTT Connect Jitter Retries](device-advisor-tests-mqtt.md#MQTT_ConnectJitterBackoff)("Device connect retries with jitter backoff - No CONNACK response")
# TLS
Use these tests to determine if the transport layer security protocol (TLS) between your devices and AWS IoT is secure.
**Note**
Device Advisor now supports TLS 1.3.
## Happy Path
**TLS Connect**
Validates if the device under test can complete the TLS handshake to AWS IoT. This test doesn't validate the MQTT implementation of the client device.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. For best results, we recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_tls_connect_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", //in seconds
},
"test":{
"id":"TLS_Connect",
"version":"0.0.0"
}
}
]
```
**Example Test case outputs:**
+ **Pass** — The device under test completed TLS handshake with AWS IoT.
+ **Pass with warnings** — The device under test completed TLS handshake with AWS IoT, but there were TLS warning messages from the device or AWS IoT.
+ **Fail** — The device under test failed to complete TLS handshake with AWS IoT due to handshake error.
**TLS Receive Maximum Size Fragments**
This test case validates that your device can receive and process TLS maximum size fragments. Your test device must subscribe to a pre-configured topic with QoS 1 to receive a large payload. You can customize the payload with the configuration `${payload}`.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. For best results, we recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"TLS Receive Maximum Size Fragments",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", //in seconds
"PAYLOAD_FORMAT":"{"message":"${payload}"}", // A string with a placeholder ${payload}, or leave it empty to receive a plain string.
"TRIGGER_TOPIC": "test_1" // A topic to which a device will subscribe, and to which a test case will publish a large payload.
},
"test":{
"id":"TLS_Receive_Maximum_Size_Fragments",
"version":"0.0.0"
}
}
]
```
## Cipher Suites
**TLS Device Support for AWS IoT recommended Cipher Suites**
Validates that the cipher suites in the TLS Client Hello message from the device under test contains the recommended [AWS IoT cipher suites](transport-security.md). It provides more insights into cipher suites supported by the device.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_tls_support_aws_iot_cipher_suites_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
},
"test":{
"id":"TLS_Support_AWS_IoT_Cipher_Suites",
"version":"0.0.0"
}
}
]
```
**Example Test case outputs:**
+ **Pass** — The device under test cipher suites contain at least one of the recommended AWS IoT cipher suite and don't contain any unsupported cipher suites.
+ **Pass with warnings** — The device cipher suites contain at least one AWS IoT cipher suite but:
1. It doesn't contain any of the recommended cipher suites
1. It contains cipher suites that aren't supported by AWS IoT.
We suggest that you verify that any unsupported cipher suites are safe.
+ **Fail** — The device under test cipher suites doesn't contain any of the AWS IoT supported cipher suites.
## Larger Size Server Certificate
**TLS large Size Server Certificate**
Validates at your device can complete the TLS handshake with AWS IoT when it receives and processes a larger size server certificate. The size of the server certificate (in bytes) used by this test is larger than what is currently used in the **TLS Connect** test case and IoT Core by 20 During this test case, AWS IoT tests your device’s buffer space for TLS If the buffer space is large enough, the TLS handshake ompletes without errors. This test esn't validate the MQTT implementation of the device. The test case ds after the TLS handshake process completes.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. For best results, we recommend a timeout value of 2 minutes. If this test case fails but the **TLS Connect** test case passes, we recommend you increase your device’s buffer space limit for TLS Increasing the buffer space limit sures that your device can process a larger size server certificate in case the size increases.
```
"tests":[
{
"name":"my_tls_large_size_server_cert_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
},
"test":{
"id":"TLS_Large_Size_Server_Cert",
"version":"0.0.0"
}
}
]
```
**Example Test case outputs:**
+ **Pass** — The device under test completed the TLS handshake with AWS IoT.
+ **Pass with warnings** — The device under test completed the TLS handshake with AWS IoT, but there are TLS warning messages either from the device or AWS IoT.
+ **Fail** — The device under test failed to complete the TLS handshake with AWS IoT because of an error during the handshake process.
## TLS Unsecure Server Cert
**Not Signed By Recognized CA**
Validates that the device under test closes the connection if it's presented with a server certificate without a valid signature from the ATS CA. A device should only connect to an endpoint that presents a valid certificate.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_tls_unsecure_server_cert_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", //in seconds
},
"test":{
"id":"TLS_Unsecure_Server_Cert",
"version":"0.0.0"
}
}
]
```
**Example Test case outputs:**
+ **Pass** — The device under test closed the connection.
+ **Fail** — The device under test completed TLS handshake with AWS IoT.
**TLS Incorrect Subject Name Server Cert / Incorrect Subject Common Name (CN) / Subject Alternative Name (SAN)**
Validates that the device under test closes the connection if it's presented with a server certificate for a domain name that is different than the one requested.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_tls_incorrect_subject_name_cert_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
},
"test":{
"id":"TLS_Incorrect_Subject_Name_Server_Cert",
"version":"0.0.0"
}
}
]
```
**Example Test case outputs:**
+ **Pass** — The device under test closed the connection.
+ **Fail** — The device under test completed the TLS handshake with AWS IoT.
## TLS Expired Server Certificate
**Expired server certificate**
Validates that the device under test closes the connection if it's presented with an expired server certificate.
**Example API test case definition:**
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_tls_expired_cert_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", //in seconds
},
"test":{
"id":"TLS_Expired_Server_Cert",
"version":"0.0.0"
}
}
]
```
**Example Test case outputs:**
+ **Pass** — The device under test refuses to complete the TLS handshake with AWS IoT. The device sends a TLS alert message before it closes the connection.
+ **Pass with warnings** — The device under test refuses to complete the TLS handshake with AWS IoT. However, it doesn’t send a TLS alert message before it closes the connection.
+ **Fail** — The device under test completes the TLS handshake with AWS IoT.
# MQTT
## CONNECT, DISCONNECT, and RECONNECT
**"Device send CONNECT to AWS IoT Core (Happy case)"**
Validates that the device under test sends a CONNECT request.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_mqtt_connect_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
},
"test":{
"id":"MQTT_Connect",
"version":"0.0.0"
}
}
]
```
**"Device can return PUBACK to an arbitrary topic for QoS1"**
This test case will check if the device (client) can return a PUBACK message if it received a publish message from the broker after subscribing to a topic with QoS1.
The payload content and the payload size are configurable for this test case. If the payload size is configured, Device Advisor will overwrite the value for the payload content, and send a predefined payload to the device with the desired size. The payload size is a value between 0 to 128 and cannot exceed 128 KB. AWS IoT Core rejects publish and connect requests larger than 128 KB, as seen in the [AWS IoT Core message broker and protocol limits and quotas](https://docs.aws.amazon.com/general/latest/gr/iot-core.html#message-broker-limits) page.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. `PAYLOAD_SIZE` can be configured to a value between 0 and 128 kilobytes. Defining a payload size overrides the payload content as Device Advisor will be sending a pre-defined payload with the given size back to the device.
```
"tests":[
{
"name":"my_mqtt_client_puback_qos1",
"configuration": {
// optional:"TRIGGER_TOPIC": "myTopic",
"EXECUTION_TIMEOUT":"300", // in seconds
"PAYLOAD_FOR_PUBLISH_VALIDATION":"custom payload",
"PAYLOAD_SIZE":"100" // in kilobytes
},
"test": {
"id": "MQTT_Client_Puback_QoS1",
"version": "0.0.0"
}
}
]
```
**"Device connect retries with jitter backoff - No CONNACK response"**
Validates that the device under test uses the proper jitter backoff when reconnecting with the broker for at least five times. The broker logs the timestamp of the device under test's CONNECT request, performs packet validation, pauses without sending a CONNACK to the device under test, and waits for the device under test to resend the request. The sixth connection attempt is allowed to pass through and CONNACK is allowed to flow back to the device under test.
The preceding process is performed again. In total, this test case requires the device to connect at least 12 times in total. The collected timestamps are used to validate that jitter backoff is used by the device under test. If the device under test has a strictly exponential backoff delay, this test case will pass with warnings.
We recommend implementation of the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test to pass this test case.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes.
```
"tests":[
{
"name":"my_mqtt_jitter_backoff_retries_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
},
"test":{
"id":"MQTT_Connect_Jitter_Backoff_Retries",
"version":"0.0.0"
}
}
]
```
**"Device connect retries with exponential backoff - No CONNACK response"**
Validates that the device under test uses the proper exponential backoff when reconnecting with the broker for at least five times. The broker logs the timestamp of the device under test's CONNECT request, performs packet validation, pauses without sending a CONNACK to the client device, and waits for the device under test to resend the request. The collected timestamps are used to validate that an exponential backoff is used by the device under test.
We recommend implementation of the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test to pass this test case.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes.
```
"tests":[
{
"name":"my_mqtt_exponential_backoff_retries_test",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"600", // in seconds
},
"test":{
"id":"MQTT_Connect_Exponential_Backoff_Retries",
"version":"0.0.0"
}
}
]
```
**"Device re-connect with jitter backoff - After server disconnect"**
Validates if a device under test uses necessary jitter and backoff while reconnecting after it's been disconnected from the server. Device Advisor disconnects the device from the server for at least five times and observes the device's behavior for MQTT reconnection. Device Advisor logs the timestamp of the CONNECT request for the device under test, performs packet validation, pauses without sending a CONNACK to the client device, and waits for the device under test to resend the request. The collected timestamps are used to validate that the device under test uses jitter and backoff while reconnecting. If the device under test has a strictly exponential backoff or doesn't implement a proper jitter backoff mechanism, this test case will pass with warnings. If the device under test has implemented either a linear backoff or a constant backoff mechanism, the test will fail.
To pass this test case, we recommend implementing the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes.
The number of reconnection attempts to validate for backoff can be changed by specifying the `RECONNECTION_ATTEMPTS`. The number must be between 5 and 10. The default value is 5.
```
"tests":[
{
"name":"my_mqtt_reconnect_backoff_retries_on_server_disconnect",
"configuration":{
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"RECONNECTION_ATTEMPTS": 5
},
"test":{
"id":"MQTT_Reconnect_Backoff_Retries_On_Server_Disconnect",
"version":"0.0.0"
}
}
]
```
**"Device re-connect with jitter backoff - On unstable connection"**
Validates if a device under test uses necessary jitter and backoff while reconnecting on an unstable connection. Device Advisor disconnects the device from the server after five successful connections, and observes the device's behavior for MQTT reconnection. Device Advisor logs the timestamp of the CONNECT request for the device under test, performs packet validation, sends back CONNACK, disconnects, log the timestamp of the disconnection, and waits for the device under test to resend the request. The collected timestamps are used to validate that the device under test uses jitter and backoff while reconnecting after successful but unstable connections. If the device under test has a strictly exponential backoff or doesn't implement a proper jitter backoff mechanism, this test case will pass with warnings. If the device under test has implemented either a linear backoff or a constant backoff mechanism, the test will fail.
To pass this test case, we recommend implementing the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes.
The number of reconnection attempts to validate for backoff can be changed by specifying the `RECONNECTION_ATTEMPTS`. The number must be between 5 and 10. The default value is 5.
```
"tests":[
{
"name":"my_mqtt_reconnect_backoff_retries_on_unstable_connection",
"configuration":{
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"RECONNECTION_ATTEMPTS": 5
},
"test":{
"id":"MQTT_Reconnect_Backoff_Retries_On_Unstable_Connection",
"version":"0.0.0"
}
}
]
```
## Publish
**"QoS0 (Happy Case)"**
Validates that the device under test publishes a message with QoS0 or QoS1. You can also validate the topic of the message and payload by specifying the topic value and payload in the test settings.
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_mqtt_publish_test",
"configuration":{
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"TOPIC_FOR_PUBLISH_VALIDATION": "my_TOPIC_FOR_PUBLISH_VALIDATION",
"PAYLOAD_FOR_PUBLISH_VALIDATION": "my_PAYLOAD_FOR_PUBLISH_VALIDATION",
},
"test":{
"id":"MQTT_Publish",
"version":"0.0.0"
}
}
]
```
**"QoS1 publish retry - No PUBACK"**
Validates that the device under test republishes a message sent with QoS1, if the broker doesn't send PUBACK. You can also validate the topic of the message by specifying this topic in the test settings. The client device must not disconnect before republishing the message. This test also validates that the republished message has the same packet identifier as the original. During the test execution, if the device loses connection and reconnects, the test case will reset without failing and the device has to perform the test case steps again.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. It is recommended for at least 4 minutes.
```
"tests":[
{
"name":"my_mqtt_publish_retry_test",
"configuration":{
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"TOPIC_FOR_PUBLISH_VALIDATION": "my_TOPIC_FOR_PUBLISH_VALIDATION",
"PAYLOAD_FOR_PUBLISH_VALIDATION": "my_PAYLOAD_FOR_PUBLISH_VALIDATION",
},
"test":{
"id":"MQTT_Publish_Retry_No_Puback",
"version":"0.0.0"
}
}
]
```
**"Publish Retained messages"**
Validates that the device under test publishes a message with `retainFlag` set to true. You can validate the topic and payload of the message by setting the topic value and payload in the test settings. If the `retainFlag` sent within the PUBLISH packet is not set to true, the test case will fail.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. To run this test case, add the `iot:RetainPublish` action in your [ device role](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html#da-iam-role).
```
"tests":[
{
"name":"my_mqtt_publish_retained_messages_test",
"configuration":{
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"TOPIC_FOR_PUBLISH_RETAINED_VALIDATION": "my_TOPIC_FOR_PUBLISH_RETAINED_VALIDATION",
"PAYLOAD_FOR_PUBLISH_RETAINED_VALIDATION": "my_PAYLOAD_FOR_PUBLISH_RETAINED_VALIDATION",
},
"test":{
"id":"MQTT_Publish_Retained_Messages",
"version":"0.0.0"
}
}
]
```
**"Publish with User Property"**
Validates that the device under test publishes a message with the correct user property. You can validate the user property by setting the name-value pair in the test settings. If the user property is not provided or doesn't match, the test case fails.
*API test case definition:*
This is a MQTT5 only test case.
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_mqtt_user_property_test",
"test":{
"USER_PROPERTIES": [
{"name": "name1", "value":"value1"},
{"name": "name2", "value":"value2"}
],
"EXECUTION_TIMEOUT":"300", // in seconds
},
"test":{
"id":"MQTT_Publish_User_Property",
"version":"0.0.0"
}
}
]
```
## Subscribe
**"Can Subscribe (Happy Case)"**
Validates that the device under test subscribes to MQTT topics. You can also validate the topic that the device under test subscribes to by specifying this topic in the test settings.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_mqtt_subscribe_test",
"configuration":{
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"TOPIC_LIST_FOR_SUBSCRIPTION_VALIDATION":["my_TOPIC_FOR_PUBLISH_VALIDATION_a","my_TOPIC_FOR_PUBLISH_VALIDATION_b"]
},
"test":{
"id":"MQTT_Subscribe",
"version":"0.0.0"
}
}
]
```
**"Subscribe Retry - No SUBACK"**
Validates that the device under test retries a failed subscription to MQTT topics. The server then waits and doesn't send a SUBACK. If the client device doesn't retry the subscription, the test fails. The client device must retry the failed subscription with the same packet Id. You can also validate the topic that the device under test subscribes to by specifying this topic in the test settings. During the test execution, if the device loses connection and reconnects, the test case will reset without failing and the device has to perform the test case steps again.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes.
```
"tests":[
{
"name":"my_mqtt_subscribe_retry_test",
"configuration":{
"EXECUTION_TIMEOUT":"300", // in seconds
// optional:
"TOPIC_LIST_FOR_SUBSCRIPTION_VALIDATION":["myTOPIC_FOR_PUBLISH_VALIDATION_a","my_TOPIC_FOR_PUBLISH_VALIDATION_b"]
},
"test":{
"id":"MQTT_Subscribe_Retry_No_Suback",
"version":"0.0.0"
}
}
]
```
## Keep-Alive
**"Mqtt No Ack PingResp"**
This test case validates if the device under test disconnects when it doesn't receive a ping response. As part of this test case, Device Advisor blocks responses sent from AWS IoT Core for publish, subscribe, and ping requests. It also validates if the device under test disconnects the MQTT connection.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout greater than 1.5 times the `keepAliveTime` value.
The maximum `keepAliveTime` must be no greater than 230 seconds for this test.
```
"tests":[
{
"name":"Mqtt No Ack PingResp",
"configuration":
//optional:
"EXECUTION_TIMEOUT":"306", // in seconds
},
"test":{
"id":"MQTT_No_Ack_PingResp",
"version":"0.0.0"
}
}
]
```
## Persistent Session
**"Persistent Session (Happy Case)"**
This test case validates the device behavior when disconnected from a persistent session. The test case checks if the device can reconnect, resume the subscriptions to its trigger topics without explicitly re-subscribing, receive the stored messages in the topics, and work as expected during a persistent session. When this test case passes, it indicates that the client device is able to maintain a persistent session with the AWS IoT Core broker in an expected manner. For more information on AWS IoT Persistent Sessions, see [ Using MQTT persistent sessions ](https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html#mqtt-persistent-sessions).
In this test case, the client device is expected to CONNECT with the AWS IoT Core with a clean session flag set to false, and then subscribe to a trigger topic. After a successful subscription, the device will be disconnected by AWS IoT Core Device Advisor. While the device is in a disconnected state, a QoS 1 message payload will be stored in that topic. Device Advisor will then allow the client device to re-connect with the test endpoint. At this point, since there is a persistent session, the client device is expected to resume its topic subscriptions without sending any additional SUBSCRIBE packets and receive the QoS 1 message from the broker. After re-connecting, if the client device re-subscribes to its trigger topic again by sending an additional SUBSCRIBE packet and/or if the client fails to receive the stored message from the trigger topic, the test case will fail.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of at least 4 minutes. In the first connection, client device needs to explicitly subscribe to a `TRIGGER_TOPIC` which was not subscribed before. To pass the test case, client device must successfully subscribe to `TRIGGER_TOPIC` with a QoS 1. After re-connecting, the client device is expected to understand that there is an active persistent session; so it should accept the stored message sent by the trigger topic and return PUBACK for that specific message.
```
"tests":[
{
"name":"my_mqtt_persistent_session_happy_case",
"configuration":{
//required:
"TRIGGER_TOPIC": "myTrigger/topic",
// optional:
// if Payload not provided, a string will be stored in the trigger topic to be sent back to the client device
"PAYLOAD": "The message which should be received from AWS IoT Broker after re-connecting to a persistent session from the specified trigger topic.",
"EXECUTION_TIMEOUT":"300" // in seconds
},
"test":{
"id":"MQTT_Persistent_Session_Happy_Case",
"version":"0.0.0"
}
}
]
```
**"Persistent Session - Session Expiry"**
This test case helps to validate device behavior when a disconnected device reconnects to an expired persistent session. After the session expires, we expect the device to resubscribe to the topics previously subscribed to by explicitly sending a new SUBSCRIBE packet.
During the first connection, we expect the test device to CONNECT with the AWS IoT broker, as its `CleanSession` flag is set to false to initiate a persistent session. The device should then subscribe to a trigger topic. Then the device is disconnected by AWS IoT Core Device Advisor, after a successful subscription and initiation of a persistent session. After the disconnection, AWS IoT Core Device Advisor allows the test device to re-connect back with the test endpoint. At this point, when the test device sends another CONNECT packet, AWS IoT Core Device Advisor sends back a CONNACK packet that indicates that the persistent session is expired. The test device needs to interpret this packet properly, and it is expected to re-subscribe to the same trigger topic again as the persistent session is terminated. If the test device does not re-subscribe to its topic trigger again, the test case fails. For the test to pass, the device needs to understand that the persistent session is over, and send back a new SUBSCRIBE packet for the same trigger topic in the second connection.
If this test case passes for a test device, it indicates that the device is able to handle re-connection on expiry of persistent session in an expected way.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of at least 4 minutes. The test device needs to explicitly subscribe to a `TRIGGER_TOPIC`, to which it was not subscribed before. To pass the test case, the test device must send a CONNECT packet with `CleanSession` flag set to false, and successfully subscribe to a trigger topic with a QoS 1. After a successful connection, AWS IoT Core Device Advisor disconnects the device. After the disconnection, AWS IoT Core Device Advisor allows the device to re-connect back, and the device is expected to re-subscribe to the same `TRIGGER_TOPIC` since AWS IoT Core Device Advisor would have terminated the persistent session.
```
"tests":[
{
"name":"my_expired_persistent_session_test",
"configuration":{
//required:
"TRIGGER_TOPIC": "myTrigger/topic",
// optional:
"EXECUTION_TIMEOUT":"300" // in seconds
},
"test":{
"id":"MQTT_Expired_Persistent_Session",
"version":"0.0.0"
}
}
]
```
# Shadow
Use these tests to verify your devices under test use AWS IoT Device Shadow service correctly. See [AWS IoT Device Shadow service](iot-device-shadows.md) for more information. If these test cases are configured in your test suite, then providing a thing is required when starting the suite run.
**MQTT over WebSocket** is not supported at this time.
## Publish
***"Device publishes state after it connects (Happy case)"***
Validates if a device can publish its state after it connects to AWS IoT Core
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_shadow_publish_reported_state",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"SHADOW_NAME": "SHADOW_NAME",
"REPORTED_STATE": {
"STATE_ATTRIBUTE": "STATE_VALUE"
}
},
"test":{
"id":"Shadow_Publish_Reported_State",
"version":"0.0.0"
}
}
]
```
The `REPORTED_STATE` can be provided for additional validation on your device's exact shadow state, after it connects. By default, this test case validates your device publishing state.
If `SHADOW_NAME` is not provided, the test case looks for messages published to topic prefixes of the Unnamed (classic) shadow type by default. Provide a shadow name if your device uses the named shadow type. See [Using shadows in devices](https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-comms-device.html) for more information.
## Update
***"Device updates reported state to desired state (Happy case)"***
Validates if your device reads all update messages received and synchronizes the device's state to match the desired state properties. Your device should publish its latest reported state after synchronizing. If your device already has an existing shadow before running the test, make sure the desired state configured for the test case and the existing reported state do not already match. You can identify Shadow update messages sent by Device Advisor by looking at the **ClientToken** field in the Shadow document as it will be `DeviceAdvisorShadowTestCaseSetup`.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes.
```
"tests":[
{
"name":"my_shadow_update_reported_state",
"configuration": {
"DESIRED_STATE": {
"STATE_ATTRIBUTE": "STATE_VALUE"
},
// optional:
"EXECUTION_TIMEOUT":"300", // in seconds
"SHADOW_NAME": "SHADOW_NAME"
},
"test":{
"id":"Shadow_Update_Reported_State",
"version":"0.0.0"
}
}
]
```
The `DESIRED_STATE` should have at least one attribute and associated value.
If `SHADOW_NAME` is not provided, then the test case looks for messages published to topic prefixes of the Unnamed (classic) shadow type by default. Provide a shadow name if your device uses the named shadow type. See [Using shadows in devices](https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-comms-device.html) for more information.
# Job Execution
**"Device can complete a job execution"**
This test case helps you validate if your device is able to receive updates using AWS IoT Jobs, and publish the status of successful updates. For more information on AWS IoT Jobs, see [ Jobs](https://docs.aws.amazon.com//iot/latest/developerguide/iot-jobs.html).
To successfully run this test case, there are two reserved AWS topics that you need to grant your [Device Role](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html#da-iam-role) . To subscribe to job activity related messages, use the **notify** and **notify-next** topics. Your device role must grant PUBLISH action for the following topics:
+ \$1aws/things/**thingName**/jobs/**jobId**/get
+ \$1aws/things/**thingName**/jobs/**jobId**/update
It is recommended to grant SUBSCRIBE and RECEIVE actions for the following topics:
+ \$1aws/things/**thingName**/jobs/get/accepted
+ \$1aws/things/**thingName**/jobs/**jobId**/get/rejected
+ \$1aws/things/**thingName**/jobs/**jobId**/update/accepted
+ \$1aws/things/**thingName**/jobs/**jobId**/update/rejected
It is recommended to grant SUBSCRIBE action for the following topic:
+ \$1aws/things/**thingName**/jobs/notify-next
For more information about these reserved topics, see reserved topics for [AWS IoT Jobs](https://docs.aws.amazon.com/iot/latest/developerguide/reserved-topics.html#reserved-topics-job).
**MQTT over WebSocket** is not supported at this time.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 3 minutes. Depending on the AWS IoT Job document or source provided, adjust the timeout value (for example, if a job will take a long time to run, define a longer timeout value for the test case). To run the test, either a valid AWS IoT Job document or an already existing job ID is required. An AWS IoT Job document can be provided as a JSON document or an S3 link. If a job document is provided, providing a job ID is optional. If a job ID is provided, Device Advisor will use that ID while creating the AWS IoT Job on your behalf. If the job document is not provided, you can provide an existing ID that is in the same region as you are running the test case. In this case, Device Advisor will use that AWS IoT Job while running the test case.
```
"tests": [
{
"name":"my_job_execution",
"configuration": {
// optional:
// Test case will create a job task by using either JOB_DOCUMENT or JOB_DOCUMENT_SOURCE.
// If you manage the job task on your own, leave it empty and provide the JOB_JOBID (self-managed job task).
// JOB_DOCUMENT is a JSON formatted string
"JOB_DOCUMENT": "{
\"operation\":\"reboot\",
\"files\" : {
\"fileName\" : \"install.py\",
\"url\" : \"${aws:iot:s3-presigned-url:https://s3.amazonaws.com/bucket-name/key}\"
}
}",
// JOB_DOCUMENT_SOURCE is an S3 link to the job document. It will be used only if JOB_DOCUMENT is not provided.
"JOB_DOCUMENT_SOURCE": "https://s3.amazonaws.com/bucket-name/key",
// JOB_JOBID is mandatory, only if neither document nor document source is provided. (Test case needs to know the self-managed job task id).
"JOB_JOBID": "String",
// JOB_PRESIGN_ROLE_ARN is used for the presign Url, which will replace the placeholder in the JOB_DOCUMENT field
"JOB_PRESIGN_ROLE_ARN": "String",
// Presigned Url expiration time. It must be between 60 and 3600 seconds, with the default value being 3600.
"JOB_PRESIGN_EXPIRES_IN_SEC": "Long"
"EXECUTION_TIMEOUT": "300", // in seconds
},
"test": {
"id": "Job_Execution",
"version": "0.0.0"
}
}
]
```
For more information on creating and using job documents see [job document](https://docs.aws.amazon.com//iot/latest/developerguide/iot-jobs.html).
# Permissions and policies
You can use the following tests to determine if the policies attached to your devices’ certificates follow standard best practices.
**MQTT over WebSocket** is not supported at this time.
**"Device certificate attached policies don’t contain wildcards"**
Validates if the permission policies associated with a device follow best practices and do not grant the device more permissions than needed.
*API test case definition:*
`EXECUTION_TIMEOUT` has a default value of 1 minute. We recommend setting a timeout of at least 30 seconds.
```
"tests":[
{
"name":"my_security_device_policies",
"configuration": {
// optional:
"EXECUTION_TIMEOUT":"60" // in seconds
},
"test": {
"id": "Security_Device_Policies",
"version": "0.0.0"
}
}
]
```
# Long duration tests
Long duration tests is a new test suite that monitors a device's behavior when it operates over longer periods of time. Compared to running individual tests that focus on specific behaviors of a device, the long duration test examines the device's behavior in a variety of real-world scenarios over the device's lifespan. Device Advisor orchestrates the tests in the most efficient possible order. The test generates results and logs, including a summary log with useful metrics about the device's performance while under test.
## MQTT long duration test case
In the MQTT long duration test case, the device's behavior is initially observed in happy case scenarios such as MQTT Connect, Subscribe, Publish, and Reconnect. Then, the device is observed in multiple, complex failure scenarios such as MQTT Reconnect Backoff, Long Server Disconnect, and Intermittent Connectivity.
## MQTT long duration test case execution flow
There are three phases in the execution of a MQTT long duration test case:
![\[The "MQTT Long Duration test execution" that shows Basic test execution, Advanced tests execution, and Additional execution time.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/mqtt-execution-flow.png)
### Basic tests execution
In this phase, the test case runs simple tests in parallel. The test validates if the device has the operations selected in the configuration.
The set of basic tests can include the following, based on the operations selected:
#### CONNECT
This scenario validates if the device is able to make a successful connection with the broker.
![\[The basic connection flow that includes a device sending a CONNECT message and Broker responds with a CONNACK message with a successful return code.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/basic-connect.png)
#### PUBLISH
This scenario validates if the device successfully publishes against the broker.
##### QoS 0
This test case validates if the device successfully sends a `PUBLISH` message to the broker during a publish with QoS 0. The test does not wait on the `PUBACK` message to be received by the device.
![\[The PUBLISH QoS 0 flow that includes a device sending a PUBLISH message with QoS 0 level.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/Qos0.png)
##### QoS 1
In this test case, the device is expected to send two `PUBLISH` messages to the broker with QoS 1. After the first `PUBLISH` message, the broker waits for up to 15 seconds before it responds. The device must retry the original `PUBLISH` message with the same packet identifier within the 15 second window. If it does, the broker responds with a `PUBACK` message and the test validates. If the device doesn't retry the `PUBLISH`, the original `PUBACK` is sent to the device and the test is marked as **Pass with warnings**, along with a system message. During the test execution, if the device loses connection and reconnects, the test scenario will reset without failing and the device has to perform the test scenario steps again.
![\[The PUBLISH QoS 1 flow that includes a device sending a PUBLISH message with QoS 1 level and multiple interactions with the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/Qos1.png)
#### SUBSCRIBE
This scenario validates if the device successfully subscribes against the broker.
##### QoS 0
This test case validates if the device successfully sends a `SUBSCRIBE` message to the broker during a subscribe with QoS 0. The test doesn't wait for the device to receive a SUBACK message.
![\[The SUBSCRIBE QoS 0 flow that includes a device sending a SUBSCRIBE message with QoS 0 level and a broker responding with a SUBACK message and Success Maximum QoS 0 code.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/subscribe-Qos0.png)
##### QoS 1
In this test case, the device is expected to send two `SUBSCRIBE` messages to the broker with QoS 1. After the first `SUBSCRIBE` message, the broker waits for up to 15 seconds before it responds. The device must retry the original `SUBSCRIBE` message with the same packet identifier within the 15 second window. If it does, the broker responds with a `SUBACK` message and the test validates. If the device doesn't retry the `SUBSCRIBE`, the original `SUBACK` is sent to the device and the test is marked as **Pass with warnings**, along with a system message. During the test execution, if the device loses connection and reconnects, the test scenario will reset without failing and the device has to perform the test scenario steps again.
![\[The SUBSCRIBE QoS 1 flow that includes a device sending a SUBSCRIBE message with QoS 1 level and multiple interactions with the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/subscribe-Qos1.png)
#### RECONNECT
This scenario validates if the device successfully reconnects with the broker after the device is disconnected from a successful connection. Device Advisor won't disconnect the device if it connected more than once previously during the test suite. Instead, it will mark the test as **Pass**.
![\[The RECONNECT flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/reconnect.png)
### Advanced tests execution
In this phase, the test case runs more complex tests in serial to validate if the device follows best practices. These advanced tests are available for selection and can be opted out if not required. Each advanced test has its own timeout value based on what the scenario demands.
#### RETURN PUBACK ON QoS 1 SUBSCRIPTION
**Note**
Only select this scenario if your device is capable of performing QoS 1 subscriptions.
This scenario validates if, after the device subscribes to a topic and receives a `PUBLISH` message from the broker, it returns a `PUBACK` message.
![\[The RETURN PUBACK ON QoS 1 SUBSCTIPTION flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/return-puback.png)
#### RECEIVE LARGE PAYLOAD
**Note**
Select this scenario only if your device is capable of performing QoS 1 subscriptions.
This scenario validates if the device responds with a `PUBACK` message after receiving a `PUBLISH` message from the broker for a QoS 1 topic with a large payload. The format of the expected payload can be configured using the `LONG_PAYLOAD_FORMAT` option.
![\[The RECEIVE LARGE PAYLOAD flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/large-payload.png)
#### PERSISTENT SESSION
**Note**
Select this scenario only if your device is capable of performing QoS 1 subscriptions and can maintain a persistent session.
This scenario validates the device behavior in maintaining persistent sessions. The test validates when the following conditions are met:
+ The device connects to the broker with an active QoS 1 subscription and persistent sessions enabled.
+ The device successfully disconnects from the broker during the session.
+ The device reconnects to the broker and resumes subscriptions to its trigger topics without explicitly resubscribing to those topics.
+ The device successfully receives messages stored by the broker for its subscribed topics and runs as expected.
For more information on AWS IoT Persistent Sessions, see [Using MQTT persistent sessions](https://docs.aws.amazon.com//iot/latest/developerguide/mqtt.html#mqtt-persistent-sessions).
![\[The PERSISTENT SESSION flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/persistent-session.png)
#### KEEP ALIVE
This scenario validates if the device successfully disconnects after it doesn't receive a ping response from the broker. The connection must have a valid keep-alive timer configured. As part of this test, the broker blocks all responses sent for `PUBLISH`, `SUBSCRIBE`, and `PINGREQ` messages. It also validates if the device under test disconnects the MQTT connection.
![\[The KEEP ALIVE flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/keep-alive.png)
#### INTERMITTENT CONNECTIVITY
This scenario validates if the device can connect back to the broker after the broker disconnects the device at random intervals for a random period of time.
![\[The INTERMITTENT CONNECTIVITY flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/intermittent.png)
#### RECONNECT BACKOFF
This scenario validates if the device has a backoff mechanism implemented when the broker disconnects from it multiple times. Device Advisor reports the backoff type as exponential, jitter, linear or constant. The number of backoff attempts is configurable using the `BACKOFF_CONNECTION_ATTEMPTS` option. The default value is 5. The value is configurable between 5 and 10.
To pass this test, we recommend implementing the [ Exponential Backoff And Jitter](http://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) mechanism on the device under test.
![\[The RECONNECT BACKOFF flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/reconnect-backoff.png)
#### LONG SERVER DISCONNECT
This scenario validates if the device can successfully reconnect after the broker disconnects the device for a long period of time (up to 120 minutes). The time for server disconnection can be configured using the `LONG_SERVER_DISCONNECT_TIME` option. The default value is 120 minutes. This value is configurable from 30 to 120 minutes.
![\[The LONG SERVER DISCONNECT flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/longserver-disconnect.png)
### Additional execution time
The additional execution time is the time the test waits after completing all the above tests and before ending the test case. Customers use this additional time period to monitor and log all communications between the device and the broker. The additional execution time can be configured using the `ADDITIONAL_EXECUTION_TIME` option. By default, this option is set to 0 minutes and can be 0 to 120 minutes.
## MQTT long duration test configuration options
All configuration options provided for the MQTT long duration test are optional. The following options are available:
**OPERATIONS**
The list of operations that the device performs, such as `CONNECT`, `PUBLISH` and `SUBSCRIBE`. The test case runs scenarios based on the specified operations. Operations that aren't specified are assumed valid.
```
{
"OPERATIONS": ["PUBLISH", "SUBSCRIBE"]
//by default the test assumes device can CONNECT
}
```
**SCENARIOS**
Based on the operations selected, the test case runs scenarios to validate the device's behavior. There are two types of scenarios:
+ **Basic Scenarios** are simple tests that validate if the device can perform the operations selected above as part of the configuration. These are pre-selected based on the operations specified in the configuration. No more input is required in the configuration.
+ **Advanced Scenarios** are more complex scenarios that are performed against the device to validate if the device follows best practices when met with real world conditions. These are optional and can be passed as an array of scenarios to the configuration input of the test suite.
```
{
"SCENARIOS": [ // list of advanced scenarios
"PUBACK_QOS_1",
"RECEIVE_LARGE_PAYLOAD",
"PERSISTENT_SESSION",
"KEEP_ALIVE",
"INTERMITTENT_CONNECTIVITY",
"RECONNECT_BACK_OFF",
"LONG_SERVER_DISCONNECT"
]
}
```
**BASIC\$1TESTS\$1EXECUTION\$1TIME\$1OUT:**
The maximum time the test case will wait for all the basic tests to complete. The default value is 60 minutes. This value is configurable from 30 to 120 minutes.
**LONG\$1SERVER\$1DISCONNECT\$1TIME:**
The time taken for the test case to disconnect and reconnect the device during the Long Server Disconnect test. The default value is 60 minutes. This value is configurable from 30 to 120 minutes.
**ADDITIONAL\$1EXECUTION\$1TIME:**
Configuring this option provides a time window after all the tests are completed, to monitor events between the device and broker. The default value is 0 minutes. This value is configurable from 0 to 120 minutes.
**BACKOFF\$1CONNECTION\$1ATTEMPTS:**
This option configures the number of times the device is disconnected by the test case. This is used by the Reconnect Backoff test. The default value is 5 attempts. This value is configurable from 5 to 10.
**LONG\$1PAYLOAD\$1FORMAT:**
The format of the message payload that the device expects when the test case publishes to a QoS 1 topic subscribed by the device.
**API test case definition:**
```
{
"tests":[
{
"name":"my_mqtt_long_duration_test",
"configuration": {
// optional
"OPERATIONS": ["PUBLISH", "SUBSCRIBE"],
"SCENARIOS": [
"LONG_SERVER_DISCONNECT",
"RECONNECT_BACK_OFF",
"KEEP_ALIVE",
"RECEIVE_LARGE_PAYLOAD",
"INTERMITTENT_CONNECTIVITY",
"PERSISTENT_SESSION",
],
"BASIC_TESTS_EXECUTION_TIMEOUT": 60, // in minutes (60 minutes by default)
"LONG_SERVER_DISCONNECT_TIME": 60, // in minutes (120 minutes by default)
"ADDITIONAL_EXECUTION_TIME": 60, // in minutes (0 minutes by default)
"BACKOFF_CONNECTION_ATTEMPTS": "5",
"LONG_PAYLOAD_FORMAT":"{"message":"${payload}"}"
},
"test":{
"id":"MQTT_Long_Duration",
"version":"0.0.0"
}
}
]
}
```
## MQTT long duration test case summary log
The MQTT long duration test case runs for longer duration than regular test cases. A separate summary log is provided, which lists important events such as device connections, publish, and subscribe during the run. Details include what was tested, what was not tested and what failed. At the end of the log, the test includes a summary of all the events that happened during the test case run. This includes:
+ *Keep Alive timer configured on the device.*
+ *Persistent session flag configured on the device.*
+ *The number of device connections during the test run.*
+ *The device reconnection backoff type, if validated for the reconnect backoff test.*
+ *The topics the device published to, during the test case run.*
+ *The topics the device subscribed to, during the test case run.*