# Using a Prometheus instance as a collector
You can use a Prometheus instance, running in *agent* mode (known as a *Prometheus agent*), to scrape metrics and send them to your Amazon Managed Service for Prometheus workspace.
The following topics describe different ways to set up a Prometheus instance running in agent mode as a collector for your metrics.
**Warning**
When you create a Prometheus agent, you are responsible for its configuration and maintenance. Avoid exposing Prometheus scrape endpoints to the public internet by [enabling security features](https://prometheus.io/docs/prometheus/latest/configuration/https/).
If you set up multiple Prometheus instances that monitor the same set of metrics and sent them to a single Amazon Managed Service for Prometheus workspace for high availability, you need to set up deduplication. If you don't follow the steps to set up deduplication, you will be charged for all data samples sent to Amazon Managed Service for Prometheus, including duplicate samples. For instructions about setting up deduplication, see [Deduplicating high availability metrics sent to Amazon Managed Service for Prometheus](AMP-ingest-dedupe.md).
**Topics**
+ [Set up ingestion from a new Prometheus server using Helm](AMP-onboard-ingest-metrics-new-Prometheus.md)
+ [Set up ingestion from an existing Prometheus server in Kubernetes on EC2](AMP-onboard-ingest-metrics-existing-Prometheus.md)
+ [Set up ingestion from an existing Prometheus server in Kubernetes on Fargate](AMP-onboard-ingest-metrics-existing-Prometheus-fargate.md)
# Set up ingestion from a new Prometheus server using Helm
The instructions in this section get you up and running with Amazon Managed Service for Prometheus quickly. You set up a new Prometheus server in an Amazon EKS cluster, and the new server uses a default configuration to send metrics to Amazon Managed Service for Prometheus. This method has the following prerequisites:
+ You must have an Amazon EKS cluster from which the new Prometheus server will collect metrics.
+ Your Amazon EKS cluster must have an [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) installed (required by Helm).
+ You must use Helm CLI 3.0 or later.
+ You must use a Linux or macOS computer to perform the steps in the following sections.
## Step 1: Add new Helm chart repositories
To add new Helm chart repositories, enter the following commands. For more information about these commands, see [Helm Repo](https://helm.sh/docs/helm/helm_repo/).
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update
```
## Step 2: Create a Prometheus namespace
Enter the following command to create a Prometheus namespace for the Prometheus server and other monitoring components. Replace *prometheus-namespace* with the name that you want for this namespace.
```
kubectl create namespace prometheus-namespace
```
## Step 3: Set up IAM roles for service accounts
For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running.
With IAM roles for service accounts, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
If you have not already set up these roles, follow the instructions at [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest) to set up the roles. The instructions in that section require the use of `eksctl`. For more information, see [Getting started with Amazon Elastic Kubernetes Service – `eksctl`](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html).
**Note**
When you are not on EKS or AWS and using just access key and secret key to access Amazon Managed Service for Prometheus, you cannot use the `EKS-IAM-ROLE` based SigV4.
## Step 4: Set up the new server and start ingesting metrics
To install the new Prometheus server that sends metrics to your Amazon Managed Service for Prometheus workspace, follow these steps.
**To install a new Prometheus server to send metrics to your Amazon Managed Service for Prometheus workspace**
1. Use a text editor to create a file named `my_prometheus_values_yaml` with the following content.
+ Replace *IAM\$1PROXY\$1PROMETHEUS\$1ROLE\$1ARN* with the ARN of the **amp-iamproxy-ingest-role** that you created in [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest).
+ Replace *WORKSPACE\$1ID* with the ID of your Amazon Managed Service for Prometheus workspace.
+ Replace *REGION* with the Region of your Amazon Managed Service for Prometheus workspace.
```
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
remoteWrite:
- url: https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
```
1. Enter the following command to create the Prometheus server.
+ Replace *prometheus-chart-name* with your Prometheus release name.
+ Replace *prometheus-namespace* with the name of your Prometheus namespace.
```
helm install prometheus-chart-name prometheus-community/prometheus -n prometheus-namespace \
-f my_prometheus_values_yaml
```
**Note**
You can customize the `helm install` command in many ways. For more information, see [Helm install](https://helm.sh/docs/helm/helm_install/) in the *Helm documentation*.
# Set up ingestion from an existing Prometheus server in Kubernetes on EC2
Amazon Managed Service for Prometheus supports ingesting metrics from Prometheus servers in clusters running Amazon EKS and in self-managed Kubernetes clusters running on Amazon EC2. The detailed instructions in this section are for a Prometheus server in an Amazon EKS cluster. The steps for a self-managed Kubernetes cluster on Amazon EC2 are the same, except that you will need to set up the OIDC provider and IAM roles for service accounts yourself in the Kubernetes cluster.
The instructions in this section use Helm as the Kubernetes package manager.
**Topics**
+ [Step 1: Set up IAM roles for service accounts](#AMP-onboard-existing-Prometheus-IRSA)
+ [Step 2: Upgrade your existing Prometheus server using Helm](#AMP-onboard-ingest-metrics-existing-remotewrite)
## Step 1: Set up IAM roles for service accounts
For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running. These roles are also called *service roles*.
With service roles, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
If you have not already set up these roles, follow the instructions at [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest) to set up the roles.
## Step 2: Upgrade your existing Prometheus server using Helm
The instructions in this section include setting up remote write and sigv4 to authenticate and authorize the Prometheus server to remote write to your Amazon Managed Service for Prometheus workspace.
### Using Prometheus version 2.26.0 or later
Follow these steps if you are using a Helm chart with Prometheus Server image of version 2.26.0 or later.
**To set up remote write from a Prometheus server using Helm chart**
1. Create a new remote write section in your Helm configuration file:
+ Replace `${IAM_PROXY_PROMETHEUS_ROLE_ARN}` with the ARN of the **amp-iamproxy-ingest-role** that you created in [Step 1: Set up IAM roles for service accounts](#AMP-onboard-existing-Prometheus-IRSA). The role ARN should have the format of `arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role`.
+ Replace `${WORKSPACE_ID}` with your Amazon Managed Service for Prometheus workspace ID.
+ Replace `${REGION}` with the Region of the Amazon Managed Service for Prometheus workspace (such as `us-west-2`).
```
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
remoteWrite:
- url: https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
```
1. Update your existing Prometheus Server configuration using Helm:
+ Replace `prometheus-chart-name` with your Prometheus release name.
+ Replace `prometheus-namespace` with the Kubernetes namespace where your Prometheus Server is installed.
+ Replace `my_prometheus_values_yaml` with the path to your Helm configuration file.
+ Replace `current_helm_chart_version` with the current version of your Prometheus Server Helm chart. You can find the current chart version by using the [helm list](https://helm.sh/docs/helm/helm_list/) command.
```
helm upgrade prometheus-chart-name prometheus-community/prometheus \
-n prometheus-namespace \
-f my_prometheus_values_yaml \
--version current_helm_chart_version
```
### Using earlier versions of Prometheus
Follow these steps if you are using a version of Prometheus earlier than 2.26.0. These steps use a sidecar approach, because earlier versions of Prometheus don't natively support AWS Signature Version 4 signing process (AWS SigV4).
These instructions assume that you are using Helm to deploy Prometheus.
**To set up remote write from a Prometheus server**
1. On your Prometheus server, create a new remote write configuration. First, create a new update file. We will call the file `amp_ingest_override_values.yaml`.
Add the following values to the YAML file.
```
serviceAccounts:
server:
name: "amp-iamproxy-ingest-service-account"
annotations:
eks.amazonaws.com/role-arn: "${SERVICE_ACCOUNT_IAM_INGEST_ROLE_ARN}"
server:
sidecarContainers:
- name: aws-sigv4-proxy-sidecar
image: public.ecr.aws/aws-observability/aws-sigv4-proxy:1.0
args:
- --name
- aps
- --region
- ${REGION}
- --host
- aps-workspaces.${REGION}.amazonaws.com
- --port
- :8005
ports:
- name: aws-sigv4-proxy
containerPort: 8005
statefulSet:
enabled: "true"
remoteWrite:
- url: http://localhost:8005/workspaces/${WORKSPACE_ID}/api/v1/remote_write
```
Replace `${REGION}` with the Region of the Amazon Managed Service for Prometheus workspace.
Replace `${SERVICE_ACCOUNT_IAM_INGEST_ROLE_ARN}` with the ARN of the **amp-iamproxy-ingest-role** that you created in [Step 1: Set up IAM roles for service accounts](#AMP-onboard-existing-Prometheus-IRSA). The role ARN should have the format of `arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role`.
Replace `${WORKSPACE_ID}` with your workspace ID.
1. Upgrade your Prometheus Helm chart. First, find your Helm chart name by entering the following command. In the output from this command, look for a chart with a name that includes `prometheus`.
```
helm ls --all-namespaces
```
Then enter the following command.
```
helm upgrade --install prometheus-helm-chart-name prometheus-community/prometheus -n prometheus-namespace -f ./amp_ingest_override_values.yaml
```
Replace *prometheus-helm-chart-name* with the name of the Prometheus helm chart returned in the previous command. Replace *prometheus-namespace* with the name of your namespace.
#### Downloading Helm charts
If you don't already have Helm charts downloaded locally, you can use the following command to download them.
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm pull prometheus-community/prometheus --untar
```
# Set up ingestion from an existing Prometheus server in Kubernetes on Fargate
Amazon Managed Service for Prometheus supports ingesting metrics from Prometheus servers in self-managed Kubernetes clusters running on Fargate. To ingest metrics from Prometheus servers in Amazon EKS clusters running on Fargate, override the default configs in a config file named amp\$1ingest\$1override\$1values.yaml as follows:
```
prometheus-node-exporter:
enabled: false
alertmanager:
enabled: false
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
persistentVolume:
enabled: false
remoteWrite:
- url: https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
```
Install Prometheus using the overrides with the following command:
```
helm install prometheus-for-amp prometheus-community/prometheus \
-n prometheus \
-f amp_ingest_override_values.yaml
```
Note that in the Helm chart configuration we disabled the node exporter and the alert manager as well as running the Prometheus server deployment.
You can verify the install with the following example test query.
```
$ awscurl --region region --service aps "https://aps-workspaces.region_id.amazonaws.com/workspaces/workspace_id/api/v1/query?query=prometheus_api_remote_read_queries"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_api_remote_read_queries","instance":"localhost:9090","job":"prometheus"},"value":[1648461236.419,"0"]}]}}21
```