# Monitoring Amazon Q Business user conversations with Amazon CloudWatch Logs Amazon Q Business CloudWatch Logs You can use [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.html) to deliver user conversations and response feedback in Amazon Q Business for you to analyze. These logs can be delivered to multiple destinations, such as Amazon CloudWatch, Amazon S3, or Amazon Data Firehose (standard rates apply). We recommend that you set up conversation and feedback logging within five minutes of creating your Amazon Q Business application environment. The following are examples of tasks you can complete with logs from conversations and response feedback in Amazon Q Business: + Identify common user queries and pain points by reviewing the chat message content. + Identify number of system-generated messages that have hallucination. + Monitor the quality of responses by looking at metrics like `isMessageWithNoAnswer`. + Understand user sentiment and satisfaction by analyzing the feedback data, including comments and usefulness ratings. + Generate custom dashboards and reports to track key metrics and trends over time. Amazon Q Business supports the `EVENT_LOGS` log type that tracks the specifics of conversations in an application. You can use `EVENT_LOGS` to monitor Amazon Q Business in all AWS regions where Amazon Q Business is offered. For more information about the AWS Regions and endpoints currently supported by Amazon Q Business, see [Supported Regions](quotas-regions.md#regions). Logs from conversations might include sensitive or personally identifiable data passed in the chats. You can filter out this information from your logs with the Amazon Q Business console. Or you can mask this data on your logs using CloudWatch Logs masking policies. For more information, see [Help protect sensitive log data with masking](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html). **Topics** + [ # Amazon Q Business chat message and feedback log examples ](cw-log-examples.md) + [ # Permissions for monitoring Amazon Q Business with Amazon CloudWatch Logs ](cw-logs-permissions.md) + [ # Enabling Amazon Q Business user conversation logging ](cw-logs-enable-logging.md) + [ # Amazon Q Business conversation log query examples ](cw-logs-common-queries.md) # Amazon Q Business chat message and feedback log examples Log examples The following are examples of Amazon Q Business chat message and feedback logs in CloudWatch Logs. **Chat message** ``` { "application_id": "", "event_timestamp": "2024-09-05T19:43:58Z", "log_type": "Message", "account_id": "", "conversation_id": "", "system_message_id": "", "user_message_id": "", "user_message": string, "system_message": string, "output_metrics_is_message_blocked": boolean, "output_metrics_is_message_with_no_answer": boolean, "user_email": "" } ``` **Amazon Q Business generated chat message (hallucination detected)** ``` { "hallucinated_message": "string", "application_id": "", "event_timestamp": "2024-09-05T19:43:58Z", "log_type": "Message", "account_id": "", "conversation_id": "", "system_message_id": "", "user_message_id": "", "user_message": string, "system_message": string, "output_metrics_is_message_blocked": boolean, "output_metrics_is_message_with_no_answer": boolean, "user_email": "" } ``` **Amazon Q Business generated chat message (no hallucination detected)** ``` { "hallucinated_message": "NO HALLUCINATION DETECTED", "application_id": "", "event_timestamp": "2024-09-05T19:43:58Z", "log_type": "Message", "account_id": "", "conversation_id": "", "system_message_id": "", "user_message_id": "", "user_message": string, "system_message": string, "output_metrics_is_message_blocked": boolean, "output_metrics_is_message_with_no_answer": boolean, "user_email": "" } ``` For system generated messages that have hallucination, you'll see one of the following log descriptions: + Hallucination mitigation disabled – `hallucinated_message: DISABLED` + Hallucination mitigation enabled but not triggered – `hallucinated_message: NOT TRIGGERED` + Hallucination mitigation enabled and triggered, but no hallucinations detected – `hallucinated_message: NO HALLUCINATION DETECTED` + Hallucination mitigation feature enabled and triggered, and hallucinations detected – `hallucinated_message: string` **Feedback** ``` { "application_id": "", "event_timestamp": "2024-09-05T13:13:27Z", "log_type": "Feedback", "account_id": "", "conversation_id": "", "system_message_id": "", "user_message_id": "", "user_message": string, "system_message": string, "usefulness_reason": "NOT_FACTUALLY_CORRECT" | "HARMFUL_OR_UNSAFE" | "INCORRECT_OR_MISSING_SOURCES" | "NOT_HELPFUL" | "FACTUALLY_CORRECT" | "COMPLETE" | "RELEVANT_SOURCES" | "HELPFUL" | "NOT_BASED_ON_DOCUMENTS" | "NOT_COMPLETE" | "NOT_CONCISE" | "OTHER", "usefulness": "NOT_USEFUL" | "USEFUL", "comment": string, "user_email": "" } ``` # Permissions for monitoring Amazon Q Business with Amazon CloudWatch Logs Permissions To set up Amazon CloudWatch Logs for Amazon Q Business, use the following IAM policy to grant the necessary permissions. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "CloudWatchLogsDeliveryPermissions", "Effect": "Allow", "Action": "logs:CreateDelivery", "Resource": [ "arn:aws:logs:us-east-1:111122223333:delivery-source:*", "arn:aws:logs:us-east-1:111122223333:delivery:*", "arn:aws:logs:us-east-1:111122223333:delivery-destination:*" ] }, { "Sid": "QBusinessLogDeliveryPermissions", "Effect": "Allow", "Action": "qbusiness:AllowVendedLogDeliveryForResource", "Resource": [ "arn:aws:qbusiness:us-east-1:111122223333:application/application-id" ] } ] } ``` ------ For example IAM policies with all the required permissions for your specific logging destination, see [Enable logging from AWS services](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.html) in the *Amazon CloudWatch Logs User Guide*. # Enabling Amazon Q Business user conversation logging Enabling logging You can enable Amazon Q Business user conversation logging within the Amazon Q Business console or using the Amazon CloudWatch Logs API operations. When you enable logging, you specify a delivery destination for the logs. If you choose Amazon S3, the prefix of logs delivered to the Amazon S3 bucket is `AWSLogs/account-id/AmazonQBusinessLogs/your-region/application-id/year/month/day/hour/.` The files are compressed and named with `Feedback-20240905T19Z_501fec0f.log.gz ` or `VendedAnalyticsChat-20240905T19Z_d26ccf9e.log.gz` formats. **Important** Logs might include sensitive or personally identifiable data passed in the chats. You can filter out this information from your logs with the Amazon Q Business console. Or you can mask this data on your logs using CloudWatch Logs masking policies. For more information, see [Help protect sensitive log data with masking](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html). **Topics** + [ ## Enabling user conversation logging with the Amazon Q Business console ](#cws-logs-enable-logging-console) + [ ## Enabling user conversation logging with the Amazon CloudWatch Logs API operations ](#cws-logs-enable-logging-api) ## Enabling user conversation logging with the Amazon Q Business console To enable user conversation logging with the Amazon Q Business console, use the admin controls for your environment to configure log delivery, optionally filter out sensitive information, and then enable logging to start streaming conversation and feedback data. **To enable logging** 1. Open the Amazon Q Business console at [Amazon Q Business](https://console.aws.amazon.com/amazonq/business/) and sign in to your account. 1. In **Applications**, choose the name of your application environment. 1. In the navigation pane, choose **Enhancements** and choose **Admin Controls and Guardrails**. 1. In **Log delivery**, choose **Add** and choose one of the following options. + **Amazon CloudWatch Logs** – Enter the **Destination log group ** where the logs will be stored. To filter out sensitive or personally identifiable information, choose **Additional settings - optional** and specify the fields to be logged, the output format, and field delimiter. For more information about log groups, see [Working with log groups and log streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html) in the *Amazon CloudWatch Logs* user guide. + **Amazon S3** – To add delivery to Amazon S3, choose the **Log type** and specify a **Destination S3 bucket**. To filter out sensitive or personally identifiable information, in **Additional settings - optional** specify the fields to be logged, whether to use hive compatible S3 paths, the output format, and the field delimiter. + **Amazon Data Firehose** – To add delivery to Amazon Data Firehose, choose the **Log type** and specify a **Destination delivery stream**. To filter out sensitive or personally identifiable information, in **Additional settings - optional** specify the fields to be logged, the output format, and the field delimiter. For information about creating a delivery stream, see [Create a Firehose delivery stream](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CreateFirehoseStream.html). **Note** If you want the users' email recorded in your logs, it must be added explicitly as a field in **Additional settings**. 1. Choose **Enable logging** start streaming conversation and feedback data to your logging destination. ## Enabling user conversation logging with the Amazon CloudWatch Logs API operations To enable user conversation logging with the Amazon CloudWatch Logs API operations, you call the PutDeliverySource, PutDeliveryDesintation, and CreateDelivery API operations. For information about quotas for these API operations, see [Service quotas](https://docs.aws.amazon.com//general/latest/gr/cwl_region.html#limits_cloudwatch_events). **Note** To enable conversation logging, you need the Amazon Resource Name (ARN) of your environment. To get this ARN, you can use the Amazon Q Business console or the [GetApplication](https://docs.aws.amazon.com/amazonq/latest/api-reference/API_GetApplication.html) API operation. An ARN follows this format: `arn:aws:qbusiness:region:account-id:application/application-id`. **To enable user conversation logging** 1. Create a delivery source with the [PutDeliverySource](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutDeliverySource.html) Amazon CloudWatch Logs API operation. Give the delivery source a name and for `resourceArn`, specify the ARN of your application. For `logType`, specify `EVENT_LOGS`. ``` { "logType": "EVENT_LOGS", "name": "my-q-business-application-delivery-source", "resourceArn": "arn:aws:qbusiness:your-region:your-account-id:application/application-id" } ``` 1. Configure the log delivery destination with the [PutDeliveryDestination](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutDeliveryDestination.html) Amazon CloudWatch Logs API operation. You can choose either Amazon CloudWatch Logs, Amazon S3, or Amazon Data Firehose as the destination for storing logs. You must specify the Amazon Resource Name of one of the destination options for where your logs will be stored. The `outputFormat` of the logs can be one of the following: json, plain, w3c, raw, or parquet. The following shows how to specify an Amazon S3 bucket as a log delivery destination with an `outputFormat` of `json`. ``` { "deliveryDestinationConfiguration": { "destinationResourceArn": "arn:aws:s3:::bucket-name" }, "name": "s3-delivery-destination", "outputFormat": "json", "tags": { "key": "value" } } ``` 1. Enable monitoring with the [CreateDelivery](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateDelivery.html) Amazon CloudWatch Logs API operation. This API operation links the delivery source to the destination you created in the previous steps. ``` { "deliveryDestinationArn": "string", "deliverySourceName": "string", "tags": { "string": "string" } } ``` **Note** If you want the users' email recorded in your logs, it must be added explicitly as a field along with the [other fields](https://docs.aws.amazon.com/amazonq/latest/qbusiness-ug/cw-log-examples.html) that you want in the [https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateDelivery.html#CWL-CreateDelivery-request-recordFields](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateDelivery.html#CWL-CreateDelivery-request-recordFields) parameter as part of calling the `CreateDelivery` operation . # Amazon Q Business conversation log query examples Log query examples You can use [CloudWatch Logs insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) to interact with conversation and feedback logs from Amazon Q Business. The following are examples of common queries. + Query for all the feedback type logs. ``` filter log_type = "Feedback" ``` + Query for all the chat type logs. ``` filter log_type = "VendedAnalyticsChat" ``` + Query for chat logs related to particular conversation. ``` filter conversation_id = ``` + Query for chat logs where customer message with certain pattern. ``` filter customer_message like /pattern/ ``` + Query for chat logs where system response message with certain pattern. ``` filter system_message like /pattern/ ``` + Query for chat logs where system not able to provide answer. ``` filter system_message like /Sorry, I could not find relevant information to complete your request./ ``` + Query for chat logs where system not able to provide answer. ``` filter output_metrics_is_message_with_no_answer = 1 ``` + Query for chat logs where customer message was blocked. ``` filter output_metrics_is_message_blocked = 1 ``` + Query for all the feedback logs where system answer was marked useful. ``` filter usefulness = "USEFUL" ``` + Query for all the feedback logs where system answer was marked not useful. ``` filter usefulness = "NOT_USEFUL" ``` + Query for all the feedback logs where system answer was marked not useful with reason “Other”. ``` filter usefulness = "NOT_USEFUL" and usefulness_reason = "OTHER" ``` + Query all feedback logs where system message was hallucinated ``` filter hallucinated_message != "NOT TRIGGERED" and hallucinated_message != NO HALLUCINATION DETECTED ```