# Troubleshoot errors Each test suite execution has a unique execution ID that is used to create a folder named `results/execution-id` in the `results` directory. Individual test group logs are under the `results/execution-id/logs` directory. Use the IDT for FreeRTOS console output to find the execution id, test case id, and test group id of the test case that failed and then open the log file for that test case named `results/execution-id/logs/test_group_id__test_case_id.log`. The information in this file includes: + Full build and flash command output. + Test execution output. + More verbose IDT for FreeRTOS console output. We recommend the following workflow for troubleshooting: 1. If you see the error "*user/role* is not authorized to access this resource", make sure that you configure permissions as specified in [Create and configure an AWS account](dev-tester-prereqs.md#config-aws-account). 1. Read the console output to find information, such as execution UUID and currently executing tasks. 1. Look in the `FRQ_Report.xml` file for error statements from each test. This directory contains execution logs of each test group. 1. Look in the log files under `/results/execution-id/logs`. 1. Investigate one of the following problem areas: + Device configuration, such as JSON configuration files in the `/configs/` folder. + Device interface. Check the logs to determine which interface is failing. + Device tooling. Make sure that the toolchains for building and flashing the device are installed and configured correctly. + For FRQ 1.x.x make sure that a clean, cloned version of the FreeRTOS source code is available. FreeRTOS releases are tagged according to the FreeRTOS version. To clone a specific version of the code, use the following commands: ``` git clone --branch version-number https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout --init --recursive ``` ## Troubleshoot device configuration When you use IDT for FreeRTOS, you must get the correct configuration files in place before you execute the binary. If you're getting parsing and configuration errors, your first step should be to locate and use a configuration template appropriate for your environment. These templates are located in the `IDT_ROOT/configs` directory. If you are still having issues, see the following debugging process. ### Where do I look? Start by reading the console output to find information, such as the execution UUID, which is referenced as `execution-id` in this documentation. Next, look in the `FRQ_Report.xml` file in the `/results/execution-id` directory. This file contains all of the test cases that were run and error snippets for each failure. To get all of the execution logs, look for the file `/results/execution-id/logs/test_group_id__test_case_id.log` for each test case. ### IDT error codes The following table explains the error codes generated by IDT for FreeRTOS: | Error Code | Error Code Name | Possible Root Cause | Troubleshooting | | --- | --- | --- | --- | | 201 | InvalidInputError | Fields in `device.json`, `config.json`, or `userdata.json` are either missing or in an incorrect format. | Make sure required fields are not missing and are in required format in listed files. For more information, see [First test of your microcontroller board](qual-steps.md). | | 202 | ValidationError | Fields in `device.json`, `config.json`, or `userdata.json` contain invalid values. | Check the error message on the right hand side of the error code in the report: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dt-afr-troubleshooting.html) | | 203 | CopySourceCodeError | Unable to copy FreeRTOS source code to specified directory. | Verify the following items: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dt-afr-troubleshooting.html) | | 204 | BuildSourceError | Unable to compile the FreeRTOS source code. | Verify the following items: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dt-afr-troubleshooting.html) | | 205 | FlashOrRunTestError | IDT FreeRTOS is unable to flash or run FreeRTOS on your DUT. | Verify the information under `flashTool` in your `userdata.json` file is correct. For more information, see [Configure build, flash, and test settings](cfg-dt-ud.md). | | 206 | StartEchoServerError | IDT FreeRTOS is unable to start echo server for the WiFi or secure sockets tests. | Verify the ports configured under `echoServerConfiguration` in your `userdata.json` file are not in use or blocked by firewall or network settings. | ### Debug config file parsing errors Occasionally, a typo in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quote from your JSON file. IDT for FreeRTOS performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you are still having issues locating the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint. ### Debug test results parse errors While running a test group from [ FreeRTOS-Libraries-Integration-Tests](https://github.com/FreeRTOS/FreeRTOS-Libraries-Integration-Tests), such as **FullTransportInterfaceTLS, FullPKCS11\$1Core, FullPKCS11\$1Onboard\$1ECC, FullPKCS11\$1Onboard\$1RSA, FullPKCS11\$1PreProvisioned\$1ECC, FullPKCS11\$1PreProvisioned\$1RSA**, or **OTACore**, IDT for FreeRTOS parses the test results from the test device with the serial connection. Sometimes, extra serial outputs on the device may interfere with the parsing of the test results. In the above mentioned case, strange test case failure reasons like strings originating from unrelated device outputs are output. The IDT for FreeRTOS test case log file (which includes all the serial output IDT for FreeRTOS has received during the test) may show the following: ``` TEST(Full_PKCS11_Capabilities, PKCS11_Capabilities) PASS ``` In the above example, the unrelated device output prevents IDT for FreeRTOS from detecting the test result which is **PASS**. Check the following to ensure optimal testing. + Make sure the logging macros used on the device are thread safe. See [Implementing the library logging macros](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-library-logging-macros.html) for more information. + Make sure there are minimal outputs to the serial connection during the tests. Other device outputs can be a problem even if your logging macros are properly thread safe, because the test results will output in separate calls during testing. An IDT for FreeRTOS test case log would ideally show an uninterrupted test results output like below: ``` ---------STARTING TESTS--------- TEST(Full_OTA_PAL, otaPal_CloseFile_ValidSignature) PASS TEST(Full_OTA_PAL, otaPal_CloseFile_InvalidSignatureBlockWritten) PASS ----------------------- 2 Tests 0 Failures 0 Ignored ``` ### Debug integrity check failures If using FRQ 1.x.x version of FreeRTOS the following integrity checks apply. When you run the FreeRTOSIntegrity test group and you encounter failures, first make sure that you haven't modified any of the `freertos` directory files. If you haven’t, and are still seeing issues, make sure you are using the correct branch. If you run IDT's `list-supported-products` command, you can find which tagged branch of the `freertos` repo you should be using. If you cloned the correct tagged branch of the `freertos` repo and still have issues, make sure you have also run the `submodule update` command. The clone workflow for the `freertos` repo is as follows. ``` git clone --branch version-number https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout —init —recursive ``` The list of files the integrity checker looks for are in the `checksums.json` file in your `freertos` directory. To qualify a FreeRTOS port without any modifications to files and the folder structure, make sure that none of the files listed in the '`exhaustive`' and '`minimal`' sections of the `checksums.json` file have been modified. To run with an SDK configured, verify that none of the files under the '`minimal`' section have been modified. If you run IDT with an SDK and have modified some files in your `freertos` directory, then make sure you correctly configure your SDK in your `userdata` file. Otherwise, the Integrity checker will verify all files in the `freertos` directory. ### Debug FullWiFi test group failures If you are using FRQ 1.x.x and encounter failures in the FullWiFi test group, and the "`AFQP_WiFiConnectMultipleAP`" test fails, this could be because both access points aren't in the same subnet as the host computer running IDT. Make sure that both access points are in the same subnet as the host computer running IDT. ### Debug "required parameter missing" errors Because new features are being added to IDT for FreeRTOS, changes to the configuration files might be introduced. Using an old configuration file might break your configuration. If this happens, the `test_group_id__test_case_id.log` file under the `results/execution-id/logs` directory explicitly lists all missing parameters. IDT for FreeRTOS validates your JSON configuration file schemas to ensure that the latest supported version has been used. ### Debug "test could not start" errors You might see errors that point to failures during test start. Because there are several possible causes, check the following areas for correctness: + Make sure that the pool name you've included in your execution command actually exists. This is referenced directly from your `device.json` file. + Make sure that the device or devices in your pool have correct configuration parameters. ### Debug "unable to find start of test results" errors You might see errors when IDT attempts to parse the results output by the device under test. There are several possible causes, so check the following areas for correctness: + Make sure that the device under test has a stable connection to your host machine. You can check the log file for a test that shows these errors to see what IDT is receiving. + If using FRQ 1.x.x, and the device under test is connected via a slow network or other interface, or you do not see the "---------STARTING TESTS---------" flag in a FreeRTOS test group log along with other FreeRTOS test group outputs, you can try increasing the value of `testStartDelayms` in your userdata configuration. For more information, see [Configure build, flash, and test settings](cfg-dt-ud.md). ### Debug a "Test failure:expected \$1\$1 results but saw \$1\$1\$1" errors You might see errors that point to a test failure during testing. The test expects a certain number of results, and does not see it during testing. Some FreeRTOS tests run before IDT sees the output from the device. If you see this error, you can try increasing the value of `testStartDelayms` in your *userdata* configuration. For more information, see [Configure build, flash, and test settings](lts-cfg-dt-ud.md). ### Debug a "\$1\$1\$1\$1\$1\$1\$1\$1 was unselected due to ConditionalTests constraints" errors This means that you are running a test on a device pool that is incompatible with the test. This may happen with the OTA E2E tests. For example, while running the `OTADataplaneMQTT` test group and in your `device.json` config file, you have chosen OTA as **No** or `OTADataPlaneProtocol` as **HTTP**. The test group chosen to run must match your `device.json` capability selections. ### Debug an IDT timeout during device output monitoring IDT can timeout due to a number of reasons. If a timeout happens during the device output monitoring phase of a test, and you can see the results inside of the IDT test case log, it means that the results were incorrectly parsed by IDT. One reason could be the interleaved log messages in the middle of the test results. If this is the case, please refer to the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-porting-ota.html) for further details on how the UNITY logs should be setup. Another reason for a timeout during device output monitoring could be a device rebooting after a single TLS test case failure. The device then runs the flashed image and causes an infinite loop which is seen in the logs. If this happens, make sure your device does not reboot after a test failure. ### Debug a "not authorized to access resource" error You might see the error "*user/role* is not authorized to access this resource" in the terminal output or in the `test_manager.log` file under `/results/execution-id/logs`. To resolve this issue, attach the `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy to your test user. For more information, see [Create and configure an AWS account](dev-tester-prereqs.md#config-aws-account). ### Debug network test errors For network-based tests, IDT starts an echo server that binds to a non-reserved port on the host machine. If you are running into errors due to timeouts or unavailable connections in the WiFi or secure sockets tests, make sure that your network is configured to allow traffic to configured ports in the 1024 - 49151 range. The secure sockets test uses ports 33333 and 33334 by default. The WiFi tests uses port 33335 by default. If these three ports are in use or blocked by firewall or network, you can choose to use different ports in userdata.json for testing. For more information, see [Configure build, flash, and test settings](cfg-dt-ud.md). You can use the following commands to check whether a specific port is in use: + Windows: `netsh advfirewall firewall show rule name=all | grep port` + Linux: `sudo netstat -pan | grep port` + macOS: `netstat -nat | grep port` ### OTA update failures due to same version payload If OTA test cases are failing due to the same version being on the device after an OTA was performed, it may be due to your build system (e.g. cmake) not noticing IDT's changes to the FreeRTOS source code and not building an updated binary. This causes OTA to be performed with the same binary that is currently on the device, and the test to fail. To troubleshoot OTA update failures, start by making sure that you are using the latest supported version of your build system. ### OTA test failure on `PresignedUrlExpired` test case One prerequisite of this test is that the OTA update time should be more than 60 seconds, otherwise the test would fail. If this occurs, the following error message is found in the log: "Test takes less than 60 seconds (url expired time) to finish. Please reach out to us." ### Debug device interface and port errors This section contains information about the device interfaces IDT uses to connect to your devices. #### Supported platforms IDT supports Linux, macOS, and Windows. All three platforms have different naming schemes for serial devices that are attached to them: + Linux: `/dev/tty*` + macOS: `/dev/tty.*` or `/dev/cu.*` + Windows: COM\$1 To check your device port: + For Linux/macOS, open a terminal and run `ls /dev/tty*`. + For macOS, open a terminal and run `ls /dev/tty.*` or `ls /dev/cu.*`. + For Windows, open Device Manager and expand the serial devices group. To verify which device is connected to a port: + For Linux, make sure that the `udev` package is installed, and then run `udevadm info –name=PORT`. This utility prints the device driver information that helps you verify you are using the correct port. + For macOS, open Launchpad and search for **System Information**. + For Windows, open Device Manager and expand the serial devices group. #### Device interfaces Each embedded device is different, which means that they can have one or more serial ports. It is common for devices to have two ports when connected to a machine: + A data port for flashing the device. + A read port to read output. You must set the correct read port in your `device.json` file. Otherwise, reading output from the device might fail. In the case of multiple ports, make sure to use the read port of the device in your `device.json` file. For example, if you plug in an Espressif WRover device and the two ports assigned to it are `/dev/ttyUSB0` and `/dev/ttyUSB1`, use `/dev/ttyUSB1` in your `device.json` file. For Windows, follow the same logic. #### Reading device data IDT for FreeRTOS uses individual device build and flash tooling to specify port configuration. If you are testing your device and don't get output, try the following default settings: + Baud rate: 115200 + Data bits: 8 + Parity: None + Stop bits: 1 + Flow control: None These settings are handled by IDT for FreeRTOS. You do not have to set them. However, you can use the same method to manually read device output. On Linux or macOS, you can do this with the `screen` command. On Windows, you can use a program such as TeraTerm. `Screen: screen /dev/cu.usbserial 115200` `TeraTerm: Use the above-provided settings to set the fields explicitly in the GUI.` ### Development toolchain problems This section discusses problems that can occur with your toolchain. #### Code Composer Studio on Ubuntu Newer versions of Ubuntu (17.10 and 18.04) have a version of the `glibc` package that is not compatible with Code Composer Studio 7.*x* versions. We recommended that you install Code Composer Studio version 8.2 or later. Symptoms of incompatibility might include: + FreeRTOS failing to build or flash to your device. + The Code Composer Studio installer might freeze. + No log output is displayed in the console during the build or flash process. + Build command attempts to launch in GUI mode even when invoked as headless. ### Logging IDT for FreeRTOS logs are placed in a single location. From the root IDT directory, these files are available under `results/execution-id/`: + `FRQ_Report.xml` + `awsiotdevicetester_report.xml` + `logs/test_group_id__test_case_id.log` `FRQ_Report.xml` and `logs/test_group_id__test_case_id.log` are the most important logs to examine. `FRQ_Report.xml` contains information about which test cases failed with a specific error message. You can then use `logs/test_group_id__test_case_id.log` to dig further into the problem to get better context. #### Console errors When AWS IoT Device Tester is run, failures are reported to the console with brief messages. Look in `results/execution-id/logs/test_group_id__test_case_id.log` to learn more about the error. #### Log errors Each test suite execution has a unique execution ID that is used to create a folder named `results/execution-id`. Individual test case logs are under the `results/execution-id/logs` directory. Use the output of the IDT for FreeRTOS console to find the execution id, test case id, and test group id of the test case that failed. Then use this information to find and open the log file for that test case named `results/execution-id/logs/test_group_id__test_case_id.log` The information in this file includes the full build and flash command output, test execution output, and more verbose AWS IoT Device Tester console output. #### S3 bucket issues If you press CTRL\$1C while running IDT, IDT will start a clean up process. Part of that clean up is to remove Amazon S3 resources that have been created as a part of the IDT tests. If the clean up can't finish, you might run into an issue where you have too many Amazon S3 buckets that have been created. This means the next time that you run IDT the tests will start to fail. If you press CTRL\$1C to stop IDT, you must let it finish the clean up process to avoid this issue. You can also delete the Amazon S3 buckets from your account that were created manually. ## Troubleshoot timeout errors If you see timeout errors while running a test suite, increase the timeout by specifying a timeout multiplier factor. This factor is applied to the default timeout value. Any value configured for this flag must be greater than or equal to 1.0. To use the timeout multiplier, use the flag `--timeout-multiplier` when running the test suite. **Example** ``` ./devicetester_linux run-suite --suite-id FRQ_1.0.1 --pool-id DevicePool1 --timeout-multiplier 2.5 ``` ``` ./devicetester_linux run-suite --suite-id FRQ_1 --pool-id DevicePool1 --timeout-multiplier 2.5 ``` ## Cellular feature and AWS charges When the `Cellular` feature is set to `Yes` in your `device.JSON` file, FullSecureSockets will use t.micro EC2 instances for running tests and this may incur additional costs to your AWS account. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/). ## Qualification report generation policy Qualification reports are only generated by AWS IoT Device Tester (IDT) versions that support FreeRTOS versions released within the last two years. If you have questions about the support policy, please contact [AWS Support](https://aws.amazon.com/contact-us/).