# Monitor collectors with vended logs
<a name="AMP-collector-vended-logs"></a>

Amazon Managed Service for Prometheus collectors provide vended logs to help you monitor and troubleshoot the metrics collection process. These logs are automatically sent to Amazon CloudWatch Logs and provide visibility into service discovery, metric collection, and data export operations. The collector vends logs for three main components of the metrics collection pipeline:

**Topics**
+ [Service discovery logs](#amp-collector-service-discovery-vended-logs)
+ [Collector logs](#amp-collector-vended-logs)
+ [Exporter logs](#amp-exporter-vended-logs)
+ [Understanding and using collector vended logs](#amp-collector-log-details)

## Service discovery logs
<a name="amp-collector-service-discovery-vended-logs"></a>

Service discovery logs provide information about the target discovery process, including:
+ Authentication or permission issues when accessing Kubernetes API resources.
+ Configuration errors in service discovery settings.

The following examples demonstrate common authentication and permission errors you might encounter during service discovery:

**Nonexistent Amazon EKS cluster**  
When the specified Amazon EKS cluster does not exist, you receive the following error:  

```
{
  "component": "SERVICE_DISCOVERY",
  "timestamp": "2025-04-30T17:25:41.946Z",
  "message": {
    "log": "Failed to watch Service - Verify your scraper source exists."
  },
  "scrapeConfigId": "s-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
}
```

**Invalid permissions for services**  
When the collector lacks proper Role-Based Access Control (RBAC) permissions to watch Services, you receive this error:  

```
{
  "component": "SERVICE_DISCOVERY",
  "timestamp": "2025-04-30T17:25:41.946Z",
  "message": {
    "log": "Failed to watch Service - Verify your scraper source permissions are valid."
  },
  "scrapeConfigId": "s-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
}
```

**Invalid permissions for endpoints**  
When the collector lacks proper Role-Based Access Control (RBAC) permissions to watch Endpoints, you receive this error:  

```
{
  "component": "SERVICE_DISCOVERY",
  "timestamp": "2025-04-30T17:25:41.946Z",
  "message": {
    "log": "Failed to watch Endpoints - Verify your scraper source permissions are valid."
  },
  "scrapeConfigId": "s-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
}
```

## Collector logs
<a name="amp-collector-vended-logs"></a>

Collector logs provide information about the metric scraping process, including:
+ Scrape failures due to endpoints not being available.
+ Connection issues when attempting to scrape targets.
+ Timeouts during scrape operations.
+ HTTP status errors returned by scrape targets.

The following examples demonstrate common collector errors you might encounter during the metric scraping process:

**Missing metrics endpoint**  
When the `/metrics` endpoint is not available on the target instance, you receive this error:  

```
{
    "component": "COLLECTOR",
    "message": {
        "log": "Failed to scrape Prometheus endpoint - verify /metrics endpoint is available",
        "job": "pod_exporter",
        "targetLabels": "{__name__=\"up\", instance=\10.24.34.0\", job=\"pod_exporter\"}"
    },
    "timestamp": "1752787969551",
    "scraperId": "s-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
}
```

**Connection refused**  
When the collector cannot establish a connection to the target endpoint, you receive this error:  

```
{
  "scrapeConfigId": "s-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
  "timestamp": "2025-04-30T17:25:41.946Z",
  "message": {
    "message": "Scrape failed",
    "scrape_pool": "pod_exporter",
    "target": "http://10.24.34.0:80/metrics",
    "error": "Get \"http://10.24.34.0:80/metrics\": dial tcp 10.24.34.0:80: connect: connection refused"
  },
  "component": "COLLECTOR"
}
```

## Exporter logs
<a name="amp-exporter-vended-logs"></a>

Exporter logs provide information about the process of sending collected metrics to your Amazon Managed Service for Prometheus workspace, including:
+ Number of metrics and data points processed.
+ Export failures due to workspace issues.
+ Permission errors when attempting to write metrics.
+ Dependency failures in the export pipeline.

The following example demonstrates a common exporter error you might encounter during the metric export process:

**Workspace not found**  
When the target workspace for metric export cannot be found, you receive this error:  

```
{
    "component": "EXPORTER",
    "message": {
        "log": "Failed to export to the target workspace - Verify your scraper destination.",
        "samplesDropped": 5
    },
    "timestamp": "1752787969664",
    "scraperId": "s-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"
}
```

## Understanding and using collector vended logs
<a name="amp-collector-log-details"></a>

### Log structure
<a name="amp-log-structure"></a>

All collector vended logs follow a consistent structure with these fields:

**scrapeConfigId**  
The unique identifier of the scrape configuration that generated the log.

**timestamp**  
The time when the log entry was generated.

**message**  
The log message content, which may include additional structured fields.

**component**  
The component that generated the log (SERVICE\$1DISCOVERY, COLLECTOR, or EXPORTER)

### Using vended logs for troubleshooting
<a name="amp-troubleshooting"></a>

The collector vended logs help you troubleshoot common issues with metrics collection:

1. Service discovery issues
   + Check **SERVICE\$1DISCOVERY** logs for authentication or permission errors.
   + Verify that the collector has the necessary permissions to access Kubernetes resources.

1. Metric scraping issues
   + Check **COLLECTOR** logs for scrape failures.
   + Verify that target endpoints are accessible and returning metrics.
   + Ensure that firewall rules allow the collector to connect to target endpoints.

1. Metric export issues
   + Check **EXPORTER** logs for export failures.
   + Verify that the workspace exists and is correctly configured.
   + Ensure that the collector has the necessary permissions to write to the workspace.

### Accessing collector vended logs
<a name="amp-accessing-logs"></a>

Collector vended logs are automatically sent to Amazon CloudWatch Logs. To access these logs:

1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/).

1. In the navigation pane, choose **Log groups**.

1. Find and select the log group for your collector: `/aws/prometheus/workspace_id/collector/collector_id`.

1. Browse or search the log events to find relevant information.

You can also use CloudWatch Logs Insights to query and analyze your collector logs. For example, to find all service discovery errors:

```
fields @timestamp, message.message
| filter component = "SERVICE_DISCOVERY" and message.message like /Failed/
| sort @timestamp desc
```

### Best practices for monitoring collectors
<a name="amp-monitoring-best-practices"></a>

To effectively monitor your Amazon Managed Service for Prometheus collectors:

1. Set up CloudWatch alarms for critical collector issues, such as persistent scrape failures or export errors. For more information, see [Alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) in the *Amazon CloudWatch User Guide*.

1. Create CloudWatch dashboards to visualize collector performance metrics alongside vended log data. For more information, see [Dashboards](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html) in the *Amazon CloudWatch User Guide*.

1. Regularly review service discovery logs to ensure targets are being discovered correctly.

1. Monitor the number of dropped targets to identify potential configuration issues.

1. Track export failures to ensure metrics are being successfully sent to your workspace.