# Amazon Bedrock Marketplace
Use Amazon Bedrock Marketplace to discover, test, and use over 100 popular, emerging, and specialized foundation models (FMs). These models are in addition to the selection of industry-leading models in Amazon Bedrock.
You can discover the models in a single catalog. After you discover the model, you can subscribe to it and deploy it to an endpoint managed by SageMaker AI.
You can access the models that you’ve deployed through Amazon Bedrock’s APIs. Accessing the models through Amazon Bedrock’s APIs allows you to use them natively with Amazon Bedrock's tools such as Agents, Knowledge Bases, and Guardrails.
For a list of Amazon Bedrock Marketplace models, see [Model compatibility](bedrock-marketplace-model-reference.md).
You can access the Amazon Bedrock Marketplace models from the:
+ [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) operation
+ [Converse](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) operation
+ Amazon Bedrock console
For information about setting up Amazon Bedrock Marketplace, see [Set up Amazon Bedrock Marketplace](setup-amazon-bedrock-marketplace.md). For information about licensing terms for individual models, see [Serverless Third-Party Models on Amazon Bedrock](https://aws.amazon.com/legal/bedrock/third-party-models/).
**Important**
After you set up Amazon Bedrock Marketplace, you can either use the code we provide for your entire workflow or get started using the AWS Management Console.
For code that runs an end-to-end workflow, see [End-to-end workflow](bedrock-marketplace-end-to-end-workflow.md). To get started using the AWS Management Console, see [Discover a model](bedrock-marketplace-discover-a-model.md).
**Topics**
+ [Set up Amazon Bedrock Marketplace](setup-amazon-bedrock-marketplace.md)
+ [Controlling Access to Amazon Bedrock Marketplace Models](control-amazon-bedrock-marketplace.md)
+ [End-to-end workflow](bedrock-marketplace-end-to-end-workflow.md)
+ [Discover a model](bedrock-marketplace-discover-a-model.md)
+ [Subscribe to a model](bedrock-marketplace-subscribe-to-a-model.md)
+ [Deploy a model](bedrock-marketplace-deploy-a-model.md)
+ [Bring your own endpoint](bedrock-marketplace-bring-your-own-endpoint.md)
+ [Call the endpoint](bedrock-marketplace-call-the-endpoint.md)
+ [Manage your endpoints](bedrock-marketplace-manage-your-endpoints.md)
+ [Model compatibility](bedrock-marketplace-model-reference.md)
# Set up Amazon Bedrock Marketplace
You can use the [Amazon Bedrock Full Access policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonBedrockFullAccess.html) to provide permissions to SageMaker AI. We recommend using the managed policy, but if you can't use the managed policy, make sure that your IAM role has the following permissions.
The following are the permissions provided by the Amazon Bedrock Full Access policy.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "BedrockAll",
"Effect": "Allow",
"Action": [
"bedrock:*"
],
"Resource": "*"
},
{
"Sid": "DescribeKey",
"Effect": "Allow",
"Action": [
"kms:DescribeKey"
],
"Resource": "arn:*:kms:*:::*"
},
{
"Sid": "APIsWithAllResourceAccess",
"Effect": "Allow",
"Action": [
"iam:ListRoles",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups"
],
"Resource": "*"
},
{
"Sid": "MarketplaceModelEndpointMutatingAPIs",
"Effect": "Allow",
"Action": [
"sagemaker:CreateEndpoint",
"sagemaker:CreateEndpointConfig",
"sagemaker:CreateModel",
"sagemaker:CreateInferenceComponent",
"sagemaker:DeleteInferenceComponent",
"sagemaker:DeleteEndpoint",
"sagemaker:UpdateEndpoint"
],
"Resource": [
"arn:aws:sagemaker:*:*:endpoint/*",
"arn:aws:sagemaker:*:*:endpoint-config/*",
"arn:aws:sagemaker:*:*:model/*"
],
"Condition": {
"StringEquals": {
"aws:CalledViaLast": "bedrock.amazonaws.com"
}
}
},
{
"Sid": "BedrockEndpointTaggingOperations",
"Effect": "Allow",
"Action": [
"sagemaker:AddTags",
"sagemaker:DeleteTags"
],
"Resource": [
"arn:aws:sagemaker:*:*:endpoint/*",
"arn:aws:sagemaker:*:*:endpoint-config/*",
"arn:aws:sagemaker:*:*:model/*"
]
},
{
"Sid": "MarketplaceModelEndpointNonMutatingAPIs",
"Effect": "Allow",
"Action": [
"sagemaker:DescribeEndpoint",
"sagemaker:DescribeEndpointConfig",
"sagemaker:DescribeModel",
"sagemaker:DescribeInferenceComponent",
"sagemaker:ListEndpoints",
"sagemaker:ListTags"
],
"Resource": [
"arn:aws:sagemaker:*:*:endpoint/*",
"arn:aws:sagemaker:*:*:endpoint-config/*",
"arn:aws:sagemaker:*:*:model/*"
],
"Condition": {
"StringEquals": {
"aws:CalledViaLast": "bedrock.amazonaws.com"
}
}
},
{
"Sid": "BedrockEndpointInvokingOperations",
"Effect": "Allow",
"Action": [
"sagemaker:InvokeEndpoint",
"sagemaker:InvokeEndpointWithResponseStream"
],
"Resource": [
"arn:aws:sagemaker:*:*:endpoint/*"
],
"Condition": {
"StringEquals": {
"aws:CalledViaLast": "bedrock.amazonaws.com"
}
}
},
{
"Sid": "DiscoveringMarketplaceModel",
"Effect": "Allow",
"Action": [
"sagemaker:DescribeHubContent"
],
"Resource": [
"arn:aws:sagemaker:*:aws:hub-content/SageMakerPublicHub/Model/*",
"arn:aws:sagemaker:*:aws:hub/SageMakerPublicHub"
]
},
{
"Sid": "AllowMarketplaceModelsListing",
"Effect": "Allow",
"Action": [
"sagemaker:ListHubContents"
],
"Resource": "arn:aws:sagemaker:*:aws:hub/SageMakerPublicHub"
},
{
"Sid": "RetrieveSubscribedMarketplaceLicenses",
"Effect": "Allow",
"Action": [
"license-manager:ListReceivedLicenses"
],
"Resource": [
"*"
]
},
{
"Sid": "PassRoleToSageMaker",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::*:role/*Sagemaker*ForBedrock*"
],
"Condition": {
"StringEquals": {
"iam:PassedToService": [
"sagemaker.amazonaws.com",
"bedrock.amazonaws.com"
]
}
}
},
{
"Sid": "PassRoleToBedrock",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/*AmazonBedrock*",
"Condition": {
"StringEquals": {
"iam:PassedToService": [
"bedrock.amazonaws.com"
]
}
}
}
]
}
```
------
**Important**
The Amazon Bedrock Full Access policy only provides permissions to the Amazon Bedrock API. To use Amazon Bedrock in the AWS Management Console, your IAM role must also have the following permissions:
```
{
"Sid": "AllowConsoleS3AccessForBedrockMarketplace",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetBucketCORS",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:GetBucketLocation"
],
"Resource": "*"
}
```
If you’re writing your own policy, you must include the policy statement that allows the Amazon Bedrock Marketplace action for the resource. For example, the following policy allows Amazon Bedrock to use the `InvokeModel` operation for a model that you’ve deployed to an endpoint.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "BedrockAll",
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel"
],
"Resource": [
"arn:aws:bedrock:us-east-1:111122223333:marketplace/model-endpoint/all-access"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": ["sagemaker:InvokeEndpoint"],
"Resource": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/project": "example-project-id",
"aws:CalledViaLast": "bedrock.amazonaws.com"
}
}
}
]
}
```
------
For more information about setting up Amazon Bedrock, see [Quickstart](getting-started.md).
You might want to use an AWS Key Management Service key to encrypt the endpoint where you've deployed the model. You must modify the preceding policy to have permissions to use the AWS KMS key.
The AWS KMS key must also have permissions to encrypt the endpoint. You must modify the AWS KMS resource policy to encrypt the endpoint. For more information about modifying the policy, see [Using IAM policies with AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/developerguide/iam-policies).
Your AWS KMS key must also have `CreateGrant` permissions. The following is an example of the permissions that must be in the key policy.
```
{
"Sid": "Allow access for AmazonSageMaker-ExecutionRole",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/SagemakerExecutionRole"
},
"Action": "kms:CreateGrant",
"Resource": "*"
}
```
For more information about providing create grant permissions, see [Granting CreateGrant permission](https://docs.aws.amazon.com/kms/latest/developerguide/create-grant-overview.html#grant-creategrant).
# Controlling Access to Amazon Bedrock Marketplace Models
You can use the [Amazon Bedrock Full Access policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonBedrockFullAccess.html) to provide permissions to SageMaker AI. To prevent users from accessing specific Bedrock Marketplace models while maintaining access to all other models, use a deny policy. The following policy demonstrates how to deny access to a specific model.
Denying Access to Specific Models:
**Important**
This policy explicitly denies access to the specified model while allowing access to all other Bedrock Marketplace models (assuming other necessary permissions are in place.
**Allowing Access to Only Specific Models**
To restrict users to accessing only specific Bedrock Marketplace models, use an allow policy with explicit model specifications. The following policy demonstrates how to allow access to only specific models:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "MarketplaceModelAllow",
"Effect": "Allow",
"Action": [
"sagemaker:CreateEndpoint",
"sagemaker:CreateEndpointConfig",
"sagemaker:CreateModel",
"sagemaker:DeleteEndpoint",
"sagemaker:UpdateEndpoint"
],
"Resource": [
"arn:aws:sagemaker:*:*:endpoint/*",
"arn:aws:sagemaker:*:*:endpoint-config/*",
"arn:aws:sagemaker:*:*:model/*"
],
"Condition": {
"StringEquals": {
"aws:CalledViaLast": "bedrock.amazonaws.com",
"aws:ResourceTag/sagemaker-sdk:bedrock": "compatible"
},
"StringLike": {
"aws:ResourceTag/sagemaker-studio:hub-content-arn": "arn:aws:sagemaker:*:aws:hub-content/SageMakerPublicHub/Model//*"
}
}
},
{
"Sid": "BedrockEndpointTaggingOperations",
"Effect": "Allow",
"Action": [
"sagemaker:AddTags",
"sagemaker:DeleteTags"
],
"Resource": [
"arn:aws:sagemaker:*:*:endpoint/*",
"arn:aws:sagemaker:*:*:endpoint-config/*",
"arn:aws:sagemaker:*:*:model/*"
],
"Condition": {
"StringLike": {
"aws:ResourceTag/sagemaker-studio:hub-content-arn": "arn:aws:sagemaker:*:aws:hub-content/SageMakerPublicHub/Model//*"
}
}
}
]
}
```
------
This policy only allows access to the specified model and denies access to all other models. If basing your policy off of `AmazonBedrockFullAccess`, this should replace the `MarketplaceModelEndpointMutatingAPIs` and `BedrockEndpointTaggingOperations` statements.
# End-to-end workflow
After you've set up Amazon Bedrock Marketplace, you can use the following example code in your end-to-end workflow. If you need more context, you can read the sections that follow the code.
```
from botocore.exceptions import ClientError
import pprint
from datetime import datetime
import json
import time
import sys
import boto3
import argparse
SM_HUB_NAME = 'SageMakerPublicHub'
DELIMITER = "\n\n\n\n================================================================================================"
class Bedrock:
def __init__(self, region_name) -> None:
self.region_name = region_name
self.boto3_session = boto3.session.Session()
self.sagemaker_client = self.boto3_session.client(
service_name='sagemaker',
region_name=self.region_name,
)
self.bedrock_client = self.boto3_session.client(
service_name='bedrock',
region_name=self.region_name
)
self.endpoint_paginator = self.bedrock_client.get_paginator('list_marketplace_model_endpoints')
self.bedrock_runtime_client = self.boto3_session.client(
service_name='bedrock-runtime',
region_name=self.region_name)
def list_models(self):
SM_RESPONSE_FIELD_NAME = 'HubContentSummaries'
SM_HUB_CONTENT_TYPE = 'Model'
response = self.sagemaker_client.list_hub_contents(
MaxResults=100,
HubName=SM_HUB_NAME,
HubContentType=SM_HUB_CONTENT_TYPE
)
all_models = Bedrock.extract_bedrock_models(response[SM_RESPONSE_FIELD_NAME])
while ("NextToken" in response) and response["NextToken"]:
response = self.sagemaker_client.list_hub_contents(
MaxResults=100,
HubName=SM_HUB_NAME,
HubContentType=SM_HUB_CONTENT_TYPE,
NextToken=response['NextToken']
)
extracted_models = Bedrock.extract_bedrock_models(response[SM_RESPONSE_FIELD_NAME])
if not extracted_models:
# Bedrock enabled models always appear first, therefore can return when results are empty.
return all_models
all_models.extend(extracted_models)
time.sleep(1)
return all_models
def describe_model(self, hub_name: str, hub_content_name: str):
return self.sagemaker_client.describe_hub_content(
HubName=hub_name,
HubContentType='Model',
HubContentName=hub_content_name
)
def list_endpoints(self):
for response in self.endpoint_paginator.paginate():
for endpoint in response['marketplaceModelEndpoints']:
yield endpoint
def list_endpoints_for_model(self, hub_content_arn: str):
for response in self.endpoint_paginator.paginate(
modelSourceEquals=hub_content_arn):
for endpoint in response['marketplaceModelEndpoints']:
yield endpoint
# acceptEula needed only for gated models
def create_endpoint(self, model, endpoint_config, endpoint_name: str, tags = []):
model_arn = model['HubContentArn']
if self._requires_eula(model=model):
return self.bedrock_client.create_marketplace_model_endpoint(
modelSourceIdentifier=model_arn,
endpointConfig=endpoint_config,
endpointName=endpoint_name,
acceptEula=True,
tags=tags
)
else:
return self.bedrock_client.create_marketplace_model_endpoint(
modelSourceIdentifier=model_arn,
endpointConfig=endpoint_config,
endpointName=endpoint_name,
tags=tags
)
def delete_endpoint(self, endpoint_arn: str):
return self.bedrock_client.delete_marketplace_model_endpoint(endpointArn=endpoint_arn)
def describe_endpoint(self, endpoint_arn: str):
return self.bedrock_client.get_marketplace_model_endpoint(endpointArn=endpoint_arn)['marketplaceModelEndpoint']
def update_endpoint(self, endpoint_arn: str, endpoint_config):
return self.bedrock_client.update_marketplace_model_endpoint(endpointArn=endpoint_arn,
endpointConfig=endpoint_config)
def register_endpoint(self, endpoint_arn: str, model_arn: str):
return self.bedrock_client.register_marketplace_model_endpoint(endpointIdentifier=endpoint_arn,
modelSourceIdentifier=model_arn)['marketplaceModelEndpoint']['endpointArn']
def deregister_endpoint(self, endpoint_arn: str):
return self.bedrock_client.deregister_marketplace_model_endpoint(endpointArn=endpoint_arn)
def invoke(self, endpoint_arn: str, body):
response = self.bedrock_runtime_client.invoke_model(modelId=endpoint_arn, body=body,
contentType='application/json')
return json.loads(response["body"].read())
def invoke_with_stream(self, endpoint_arn: str, body):
return self.bedrock_runtime_client.invoke_model_with_response_stream(modelId=endpoint_arn, body=body)
def converse(self, endpoint_arn: str, conversation):
return self.bedrock_runtime_client.converse(modelId=endpoint_arn, messages=conversation)
def converse_with_stream(self, endpoint_arn: str, conversation):
return self.bedrock_runtime_client.converse_stream(modelId=endpoint_arn, messages=conversation,
inferenceConfig={"maxTokens": 4096, "temperature": 0.5,
"topP": 0.9})
def wait_for_endpoint(self, endpoint_arn: str):
endpoint = self.describe_endpoint(endpoint_arn=endpoint_arn)
while endpoint['endpointStatus'] in ['Creating', 'Updating']:
print(
f"Endpoint {endpoint_arn} status is still {endpoint['endpointStatus']}. Waiting 10 seconds before continuing...")
time.sleep(10)
endpoint = self.describe_endpoint(endpoint_arn=endpoint_arn)
print(f"Endpoint status: {endpoint['status']}")
def _requires_eula(self, model):
if 'HubContentDocument' in model:
hcd = json.loads(model['HubContentDocument'])
if ('HostingEulaUri' in hcd) and hcd['HostingEulaUri']:
return True
return False
@staticmethod
def extract_bedrock_models(hub_content_summaries):
models = []
for content in hub_content_summaries:
if ('HubContentSearchKeywords' in content) and (
'@capability:bedrock_console' in content['HubContentSearchKeywords']):
print(f"ModelName: {content['HubContentDisplayName']}, modelSourceIdentifier: {content['HubContentArn']}")
models.append(content)
return models
def run_script(sagemaker_execution_role: str, region: str):
# Script params
model_arn = 'arn:aws:sagemaker:AWS Region:aws:hub-content/SageMakerPublicHub/Model/example-model-name/hub-content-arn'
model_name = 'example-model-name'
sample_endpoint_name = f'test-ep-{datetime.now().strftime("%Y-%m-%d%H%M%S")}'
sagemaker_execution_role = sagemaker_execution_role
conversation = [
{
"role": "user",
"content": [
{
"text": "whats the best park in the US?"
}
]
}
]
bedrock = Bedrock(region_name=region)
###
### Model discovery
###
# List all models - no new Bedrock Marketplace API here. Uses existing SageMaker APIs
print(DELIMITER)
print("All models:")
all_models = bedrock.list_models()
# Describe a model - no new Bedrock Marketplace API here. Uses existing SageMaker APIs
# Examples:
# bedrock.describe_model("SageMakerPublicHub", "huggingface-llm-amazon-mistrallite")
# bedrock.describe_model("SageMakerPublicHub", "huggingface-llm-gemma-2b-instruct")
print(DELIMITER)
print(f'Describing model: {model_name}')
model = bedrock.describe_model(SM_HUB_NAME, model_name)
pprint.pprint(model)
## If customer wants to use a proprietary model, they need to subscribe to it first
## If customer wants to use a gated model, they need to accept EULA. Note: EULA Acceptance is on-creation, and needs
## to be provided on every call. Cannot un-accept a EULA
## If customer wants to use an open weight model, they can proceed to deploy
###
### Model deployment to create endpoints
###
# # Create endpoint - uses Bedrock Marketplace API
endpoint_arn = bedrock.create_endpoint(
endpoint_name=sample_endpoint_name,
endpoint_config={
"sageMaker": {
"initialInstanceCount": 1,
"instanceType": "ml.g5.2xlarge",
"executionRole": sagemaker_execution_role
# Other fields:
# kmsEncryptionKey: KmsKeyId
# vpc: VpcConfig
}
},
# Optional:
# tags: TagList
model=model
)['marketplaceModelEndpoint']['endpointArn']
# # Describe endpoint - uses Bedrock Marketplace API
endpoint = bedrock.describe_endpoint(endpoint_arn=endpoint_arn)
print(DELIMITER)
print('Created endpoint:')
pprint.pprint(endpoint)
# Wait while endpoint is being created
print(DELIMITER)
bedrock.wait_for_endpoint(endpoint_arn=endpoint_arn)
###
### Currently, customers cannot use self-hosted endpoints with Bedrock Runtime APIs and tools. They can only pass a model ID to the APIs.
### Bedrock Marketplace will enable customers to use self-hosted endpoints through existing Bedrock Runtime APIs and tools
### See below examples of calling invoke_model, invoke_model_with_response_stream, converse and converse_stream
### Customers will be able to use the endpoints with Bedrock dev tools also (Guardrails, Model eval, Agents, Knowledge bases, Prompt flows, Prompt management) - examples not shown below
###
# Prepare sample data for invoke calls by getting default payload in model metadata
model_data = json.loads(bedrock.describe_model('SageMakerPublicHub', model_name)['HubContentDocument'])
payload = list(model_data["DefaultPayloads"].keys())[0]
invoke_body = model_data["DefaultPayloads"][payload]["Body"]
invoke_content_field_name = 'generated_text'
# Invoke model (text) - without stream - uses existing Bedrock Runtime API
print(DELIMITER)
print(f'Invoking model with body: {invoke_body}')
invoke_generated_response = bedrock.invoke(endpoint_arn=endpoint_arn, body=json.dumps(invoke_body))
print(f'Generated text:')
print(invoke_generated_response[invoke_content_field_name])
sys.stdout.flush()
# Converse with model (chat) - without stream - uses existing Bedrock Runtime API
print(DELIMITER)
print(f'Converse model with conversation: {conversation}')
print(bedrock.converse(endpoint_arn=endpoint_arn, conversation=conversation)['output'])
###
## Other endpoint management operations
###
# List all endpoints - uses Bedrock Marketplace API
print(DELIMITER)
print('Listing all endpoints')
for endpoint in bedrock.list_endpoints():
pprint.pprint(endpoint)
# List endpoints for a model
# Example: bedrock.list_endpoints_for_model(hub_content_arn='arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/Model/huggingface-textgeneration1-mpt-7b-storywriter-bf16/3.2.0')
print(DELIMITER)
print(f"Listing all endpoints for model: {model_arn}")
for endpoint in bedrock.list_endpoints_for_model(hub_content_arn=model_arn):
pprint.pprint(endpoint)
# # Update endpoint - uses new API provided by Bedrock Marketplace
updated_endpoint_arn = bedrock.update_endpoint(
endpoint_arn=endpoint_arn,
endpoint_config={
"sageMaker": {
"initialInstanceCount": 2, # update to increase instance count
"instanceType": "ml.g5.2xlarge",
"executionRole": sagemaker_execution_role
# Other fields:
# kmsEncryptionKey: KmsKeyId
# vpc: VpcConfig
}
# Optional:
# tags: TagList
}
)['marketplaceModelEndpoint']['endpointArn']
# Wait while endpoint is being updated
print(DELIMITER)
bedrock.wait_for_endpoint(endpoint_arn=updated_endpoint_arn)
# Confirm endpoint update
updated_endpoint = bedrock.describe_endpoint(endpoint_arn=updated_endpoint_arn)
print(f'Updated endpoint: {updated_endpoint}')
assert updated_endpoint['endpointConfig']['sageMaker']['initialInstanceCount'] == 2
print(DELIMITER)
print(f'Confirmed that updated endpoint\'s initialInstanceCount config changed from 1 to 2')
# Wait while endpoint is being updated
print(DELIMITER)
bedrock.wait_for_endpoint(endpoint_arn=updated_endpoint_arn)
# Deregister endpoint - uses Bedrock Marketplace API
print(DELIMITER)
print(f'De-registering endpoint: {updated_endpoint_arn}')
bedrock.deregister_endpoint(endpoint_arn=updated_endpoint_arn)
try:
pprint.pprint(bedrock.describe_endpoint(endpoint_arn=updated_endpoint_arn))
except ClientError as err:
assert err.response['Error']['Code'] == 'ResourceNotFoundException'
print(f"Confirmed that endpoint {updated_endpoint_arn} was de-registered")
# Re-register endpoint - uses Bedrock Marketplace API
print(DELIMITER)
print(f'Registered endpoint: {bedrock.register_endpoint(endpoint_arn=updated_endpoint_arn, model_arn=model_arn)}')
pprint.pprint(bedrock.describe_endpoint(endpoint_arn=updated_endpoint_arn))
# Delete endpoint - uses Bedrock Marketplace API
print(DELIMITER)
print(f'Deleting endpoint: {updated_endpoint_arn}')
bedrock.delete_endpoint(endpoint_arn=updated_endpoint_arn)
try:
pprint.pprint(bedrock.describe_endpoint(endpoint_arn=updated_endpoint_arn))
except ClientError as err:
assert err.response['Error']['Code'] == 'ResourceNotFoundException'
print(f"Confirmed that endpoint {updated_endpoint_arn} was deleted")
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--sagemaker-execution-role', required=True)
parser.add_argument('--region', required=True)
args = parser.parse_args()
run_script(args.sagemaker_execution_role, args.region)
```
# Discover a model
You can discover both Amazon Bedrock Marketplace models and Amazon Bedrock serverless models from the model catalog.
Use the following procedure to discover an Amazon Bedrock Marketplace model.
**To discover an Amazon Bedrock Marketplace model**
1. Sign in to the AWS Management Console using an IAM role with Amazon Bedrock permissions.
1. In the search bar, specify "Amazon Bedrock" and choose the **Amazon Bedrock** from the dropdown list.
1. From the navigation pane, choose **Model Catalog**.
1. Filter for **Model Collection** = **Bedrock Marketplace** to load Amazon Bedrock Marketplace models.
1. Alternatively, you can search for the model name directly in the search bar.
**Note**
You must subscribe to proprietary models in Amazon Bedrock Marketplace to get access to them.
# Subscribe to a model
To use a model from Amazon Bedrock Marketplace, you subscribe to the model. If you're using a publicly available model, such as a HuggingFace model, you don't need a subscription. Models can be publicly available or proprietary. When you subscribe to a model, you review and accept the prices and EULAs of the model provider.
**Note**
Subscription is not required before first invocation due to auto-enablement, Amazon Bedrock automatically initiates the subscription process in the background. API calls may succeed temporarily while the subscription is being finalized. However, if subscription fails, subsequent requests will fail, see [Access Amazon Bedrock foundation models](model-access.md).
**To subscribe to a model**
1. Sign in to the AWS Management Console using an IAM role with Amazon Bedrock permissions.
1. In the search bar, specify "Amazon Bedrock" and choose the **Amazon Bedrock** from the dropdown list.
1. From the navigation pane, choose Model Catalog.
1. Specify **Model Collection** = **Bedrock Marketplace** to load Amazon Bedrock Marketplace models.
1. Alternatively, you can search for the model name directly in the search bar.
1. Choose the model card to open the **Model Detail** page.
1. Choose **View Subscription Options** to open the subscription modal.
1. Review the offers and the cost.
1. Review the legal terms and conditions.
1. Choose **Subscribe**.
The subscription process is usually complete within 10-15 minutes but can vary based on the requirements of the provider.
The costs are for the software costs only. You will be billed a separate SageMaker AI infrastructure cost for the instance type and number of instances that you select.
# Deploy a model
After you've subscribed to a model, you deploy it to a SageMaker AI endpoint. You make inference calls to the endpoint. The model is hosted by SageMaker AI. During the deployment process, you provide the following information:
+ The name of the SageMaker AI endpoint
+ The number of instances taking inference calls to the endpoint
+ The instance type of the endpoint
You can also configure optional advanced options such as tags.
A SageMaker AI service role is automatically created for SageMaker AI to assume and perform actions on your behalf. For more information about Amazon SageMaker AI permissions, see [How to use SageMaker AI execution roles](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html ).
Alternatively, you can choose an existing role or create a new one. To learn how to deploy an Amazon Bedrock Marketplace model, select the tab corresponding to your method of choice and follow the steps.
**To deploy a model**
1. Sign in to the AWS Management Console using an IAM role with Amazon Bedrock permissions.
1. In the search bar, specify "Amazon Bedrock" and choose the **Amazon Bedrock** from the dropdown list.
1. From the navigation pane, choose **Model Catalog**.
1. Choose the model card for the model that you're deploying.
1. Choose **Deploy**.
1. For **Endpoint Name**, specify the name of the endpoint.
1. Choose the number of instances and select the instance type.
1. Under **Advanced Settings**, you can optionally:
1. Set up your VPC
1. Configure the service access role
1. Customize your encryption settings
1. Add tags
1. Choose **Deploy** to deploy your Amazon Bedrock Marketplace model to an SageMaker AI endpoint. This process usually takes 10-15 minutes.
Use the `CreateMarketplaceModelEndpoint` operation to create an endpoint. Some models have an end-user license agreement (EULA). To accept the EULA, you set the `AcceptEula` to `True`.
The following example uses an example AWS Command Line Interface command to create an endpoint:
```
aws bedrock create-marketplace-model-endpoint --model-source-identifier HubContentArn --endpoint-config "{\"sageMaker\":{\"initialInstanceCount\":1,\"instanceType\":\"ml.g5.xlarge\",\"executionRole\":\"arn:aws:iam::111122223333:role/example-IAM-role\"}}" --endpoint-name "example-endpoint-name"
```
**Note**
The responses for the `CreateMarketplaceModelEndpoint`, `UpdateMarketplaceModelEndpoint` and `RegisterMarketplaceModelEndpoint` operations are the same.
```
{"marketplaceModelEndpoint":{"createdAt":"2024-11-12T02:31:58.201474085Z","endpoint":{"sageMaker":{"executionRole":"arn:aws:iam::111122223333:role/service-role/amazon-sagemaker-execution-role","initialInstanceCount":1,"instanceType":"ml.g5.2xlarge","kmsEncryptionKey":null,"vpc":null}},"endpointArn":"arn:aws:sagemaker:region:555555555555:endpoint/example-endpoint-name","endpointStatus":"Creating","endpointStatusMessage":"","modelSourceIdentifier":"arn:aws:sagemaker:region:aws:hub-content/SageMakerPublicHub/Model/example-model-name/1.2.2","status":"ACTIVE","statusMessage":"","updatedAt":"2024-11-12T02:31:58.201474085Z"}}
```
You can modify the endpoint in both Amazon Bedrock Marketplace and Amazon SageMaker AI. We recommend only modifying the endpoint within Amazon Bedrock. If you modify the endpoint within SageMaker AI, you might not be able to use the endpoint within Amazon Bedrock. The following are the modifications that can cause the endpoint to fail within Amazon Bedrock:
+ Setting [EnableNetworkIsolation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html#sagemaker-CreateModel-request-EnableNetworkIsolation) to `False`
+ Modifying the model definition within the [PrimaryContainer](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html#sagemaker-CreateModel-request-PrimaryContainer) object
For the endpoint to be operational, it must be registered and in service. You can use the following AWS Command Line Interface command to check the status of the endpoint.
```
aws bedrock get-marketplace-model-endpoint --endpoint-arn arn:aws:sagemaker:region:111122223333:endpoint/example-endpoint-name
```
The endpoint must have the following status for you to use it with Amazon Bedrock:
```
endpointStatus: InService
status: REGISTERED
```
If you've made a modification that has caused the endpoint to fail, you can deregister and reregister the endpoint. Use the `DeregisterEndpoint` operation to deregister the endpoint. Use the `RegisterEndpoint` operation to reregister it.
```
aws bedrock deregister-marketplace-model-endpoint --endpoint-arn arn:aws:sagemaker:region:111122223333:endpoint/example-endpoint-name
```
```
aws bedrock register-marketplace-model-endpoint --endpoint-identifier arn:aws:sagemaker:region:111122223333:endpoint/example-endpoint-name --model-source-identifier
HubContentArn
```
# Bring your own endpoint
You can register an endpoint hosting an Amazon Bedrock AWS Marketplace model you've created in SageMaker AI. During the registration process, the models are checked for compatibility with Amazon Bedrock Marketplace requirements. You must have network isolation enabled on your endpoints. Additionally, you can't change the model artifacts from the base model that the provider supplies.
For more information about registering the endpoint, see [Use your SageMaker AI JumpStart Models in Amazon Bedrock](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-use-studio-updated-register-bedrock)
# Call the endpoint
You can start using your model after you've deployed it to an endpoint. You use either the `Converse` or `InvokeModel` operations to call the models.
For a list of models which are supported by `Converse` operation, see [Model compatibility](bedrock-marketplace-model-reference.md). For information about Agents, see [Automate tasks in your application using AI agents](agents.md). For information about knowledge bases, see [Retrieve data and generate AI responses with Amazon Bedrock Knowledge Bases](knowledge-base.md).
**To use a model**
1. Sign in to the AWS Management Console using an IAM role with Amazon Bedrock permissions.
1. In the search bar, specify "Amazon Bedrock" and choose **Amazon Bedrock** from the dropdown list.
1. Choose the tool you're using. **Playground** is the fastest way to access the model that you've deployed.
1. Choose **Select Model**.
1. Choose your model's endpoint.
You can also use the following example AWS Command Line Interface commands to call the endpoint.
```
aws bedrock-runtime converse --model-id arn:aws:sagemaker:region:111122223333:endpoint/example-endpoint-name --messages '[{"role": "user", "content": [{"text": "Describe the purpose of a \"hello world\" program in one line."}]}]'
```
```
aws bedrock-runtime invoke-model --model-id arn:aws:sagemaker:region:111122223333:endpoint/example-endpoint-name --body '{"inputs": "Hello? How are you?", "parameters": {"details": true}}' --cli-binary-format raw-in-base64-out test.txt
```
# Manage your endpoints
You view and manage your Amazon Bedrock Marketplace model endpoints in the following ways:
+ Editing the number of instances or instance types
+ Changing the tags
+ Deleting the endpoint
You can also register and de-register endpoints of Amazon Bedrock Marketplace models that you created from SageMaker AI.
**To manage your endpoints**
1. Sign in to the AWS Management Console using an IAM role with Amazon Bedrock permissions.
1. In the search bar, specify "Amazon Bedrock" and choose **Amazon Bedrock** from the dropdown list.
1. Choose **Marketplace deployments** under **Foundation models**
1. Choose **Register**, **Edit**, or **Delete** from the **Action** bar.
1. To view additional details, choose the endpoint.
You can use the following AWS Command Line Interface command to delete the endpoint.
```
aws bedrock delete-marketplace-model-endpoint --endpoint-arn "arn:aws:sagemaker:region:111122223333:endpoint/example-endpoint"
```
The preceding command doesn't return a response.
You can use the following AWS Command Line Interface command to update the endpoint.
```
aws bedrock update-marketplace-model-endpoint --endpoint-config "{\"sageMaker\":{\"initialInstanceCount\":2,\"instanceType\":\"ml.g5.xlarge\",\"executionRole\":\"arn:aws:iam::111122223333:role/service-role/example-sagemaker-service-role\"}}" --endpoint-arn "arn:aws:sagemaker:region:account-number:endpoint/example-endpoint-name"
```
**Note**
The responses for the `CreateMarketplaceModelEndpoint`, `RegisterMarketplaceModelEndpoint`, and `UpdateMarketplaceModelEndpoint` operations are the same.
# Model compatibility
All models can use the `InvokeModel` operation. Some models can use the `Converse` operation.
For Amazon Bedrock marketplace models, use the [ ApplyGuardrail](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html) API to use Amazon Bedrock Guardrails.
The following table shows the available models and whether they can use the `Converse` operation:
| Name | Converse API Supported | Streaming Supported |
| --- | --- | --- |
| Arcee Lite | Yes | Yes |
| Arcee Nova | Yes | Yes |
| Arcee SuperNova | Yes | Yes |
| Arcee Virtuoso Small | Yes | Yes |
| Aya 101 | No | Yes |
| Bart Large CNN samsum | No | No |
| Bloom 1b1 | No | Yes |
| Bloom 1b7 | No | Yes |
| Bloom 3B | No | Yes |
| Bloom 560m | No | Yes |
| Bloom 7B1 | No | Yes |
| Bloomz 1b1 | No | Yes |
| Bloomz 1b7 | No | Yes |
| BloomZ 3B FP16 | No | Yes |
| BloomZ 7B1 FP16 | No | Yes |
| Bria 2.2HD Commercial | No | No |
| Bria 2.3 Commercial | No | No |
| Bria 2.3 Fast Commercial | No | No |
| CyberAgentLM3-22B-Chat (CALM3-22B-Chat) | Yes | Yes |
| DBRX Base | No | Yes |
| DBRX Instruct | Yes | Yes |
| DeepSeek-R1-Distill-Llama-70B | No | Yes |
| DeepSeek-R1-Distill-Llama-8B | No | Yes |
| DeepSeek-R1-Distill-Qwen-32B | No | Yes |
| DeepSeek-R1-Distill-Qwen-14B | No | Yes |
| DeepSeek-R1-Distill-Qwen-7B | No | Yes |
| DeepSeek-R1-Distill-Qwen-1.5B | No | Yes |
| DeepSeek-R1 | No | Yes |
| Distilbart CNN 12-6 | No | No |
| Distilbart CNN 6-6 | No | No |
| Distilbart xsum 12-3 | No | No |
| DistilGPT 2 | No | Yes |
| Dolly V2 12b BF16 | No | Yes |
| Dolly V2 3b BF16 | No | Yes |
| Dolly V2 7b BF16 | No | Yes |
| Dolphin 2.2.1 Mistral 7B | Yes | Yes |
| Dolphin 2.5 Mixtral 8 7B | Yes | Yes |
| EleutherAI GPT Neo 1.3B | No | Yes |
| EleutherAI GPT Neo 2.7B | No | Yes |
| ESM3-open | No | No |
| EXAONE\$1v3.0 7.8B Instruct | Yes | No |
| Falcon 40B BF16 | No | Yes |
| Falcon Lite | No | Yes |
| Falcon Lite 2 | No | Yes |
| Falcon RW 1B | No | Yes |
| Falcon3 1B Instruct | Yes | Yes |
| Falcon3 3B Base | No | Yes |
| Falcon3 3B Instruct | Yes | Yes |
| Falcon3 7B Base | No | Yes |
| Falcon3 7B Instruct | Yes | Yes |
| Falcon3 10B Base | No | Yes |
| Falcon3 10B Instruct | Yes | Yes |
| Flan-T5 Base | No | Yes |
| Flan-T5 Base Model Fine-tuned on the Samsum Dataset | No | Yes |
| Flan-T5 Large | No | Yes |
| Flan-T5 Small | No | Yes |
| Gemma 2 27B | No | Yes |
| Gemma 2 27B Instruct | Yes | Yes |
| Gemma 2 2B | No | Yes |
| Gemma 2 2B Instruct | Yes | Yes |
| Gemma 2 9B | No | Yes |
| Gemma 2 9B Instruct | Yes | Yes |
| Gemma 2B | No | Yes |
| Gemma 2B Instruct | Yes | Yes |
| Gemma 7B | No | Yes |
| Gemma 7B Instruct | Yes | Yes |
| GPT 2 | No | Yes |
| GPT NeoX 20B FP16 | No | Yes |
| GPT-2 XL | No | Yes |
| GPT-J 6B | No | Yes |
| GPT-Neo 1.3B | No | Yes |
| GPT-Neo 125M | No | Yes |
| GPT-NEO 2.7B | No | Yes |
| Granite 3.0 2B Instruct | Yes | Yes |
| Granite 3.0 8B Instruct | Yes | Yes |
| Gretel Navigator Tabular | Yes | Yes |
| IBM Granite 20B Code Instruct - 8K | Yes | Yes |
| IBM Granite 34B Code Instruct - 8K | Yes | Yes |
| IBM Granite 3B Code Instruct - 128K | Yes | Yes |
| IBM Granite 8B Code Instruct - 128K | Yes | Yes |
| KARAKURI LM 8x7b instruct | No | Yes |
| Liquid | Yes | Yes |
| Llama 3.1 SuperNova Lite | Yes | Yes |
| Llama Spark | Yes | Yes |
| Llama-3-Varco-Offsetbias-8B | No | Yes |
| Llama3 8B SEA-Lion v2.1 Instruct | Yes | Yes |
| MARS6 | No | No |
| Medical LLM - Medium | No | No |
| Medical LLM - Small | No | No |
| Medical Text Translation (EN-ES) | No | No |
| Mistral 7B OpenOrca AWQ | Yes | Yes |
| Mistral 7B OpenOrca GPTQ | Yes | Yes |
| Mistral 7B SFT Alpha | Yes | Yes |
| Mistral 7B SFT Beta | Yes | Yes |
| Mistral Lite | Yes | Yes |
| Mistral Nemo Base 2407 | No | Yes |
| Mistral Nemo Instruct 2407 | Yes | Yes |
| Mistral Trix V1 | No | Yes |
| MPT 7B BF16 | No | Yes |
| MPT 7B Instruct BF16 | No | Yes |
| MPT 7B StoryWriter-65k\$1 BF16 | No | Yes |
| Multilingual GPT | No | Yes |
| NVIDIA Nemotron-4 15B NIM Microservice | Yes | No |
| Open Hermes 2 Mistral 7B | Yes | Yes |
| Phi-2 | No | Yes |
| Phi-3-Mini-128K-Instruct | Yes | Yes |
| Phi-3-Mini-4K-Instruct | Yes | Yes |
| Phi-3.5-mini-instruct | Yes | Yes |
| Pixtral 12B 2409 | Yes | Yes |
| PLaMo API | No | Yes |
| Snowflake Arctic Instruct Vllm | Yes | Yes |
| Solar Mini Chat | Yes | Yes |
| Solar Mini Chat - Quant | Yes | Yes |
| Solar Mini Chat ja | Yes | Yes |
| Solar Mini Chat ja - Quant | Yes | Yes |
| Solar Pro | Yes | Yes |
| Solar Pro (Quantized) | Yes | Yes |
| Stable Diffusion 3.5 Large | No | No |
| Stockmark-LLM-13b | No | No |
| Text Summarization | No | No |
| VARCO LLM KO/EN-13B-IST | Yes | No |
| Whisper Large V3 Turbo | No | Yes |
| Widn Llama3-Tower Vesuvius | Yes | Yes |
| Widn Tower Anthill | Yes | Yes |
| Widn Tower Sugarloaf | Yes | Yes |
| Writer Palmyra Small | No | Yes |
| Writer Palmyra-Fin-70B-32K | Yes | Yes |
| Writer Palmyra-Med-70B-32K | Yes | Yes |
| YARN Mistral 7B 128k | No | Yes |
| Yi-1.5-34B | No | Yes |
| Yi-1.5-34B-Chat | Yes | Yes |
| Yi-1.5-6B | No | Yes |
| Yi-1.5-6B-Chat | Yes | Yes |
| Yi-1.5-9B | No | Yes |
| Yi-1.5-9B-Chat | Yes | Yes |
| Zephyr 7B Alpha | No | Yes |
| Zephyr 7B Beta | No | Yes |
| Zephyr 7B Gemma | No | Yes |