View a markdown version of this page

Automate Lambda test event generation using Amazon Bedrock AgentCore - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Automate Lambda test event generation using Amazon Bedrock AgentCore

Ishita Gupta and Kriti Gupta, Amazon Web Services

Summary

This pattern delivers an AI-powered approach for automatically generating comprehensive test cases for AWS Lambda functions using Amazon Bedrock's generative AI capabilities. This pattern implements a three-agent architecture (Analyzer, Generator, and Validator) deployed on Amazon Bedrock AgentCore that analyzes actual Lambda function code to extract input/output patterns and generates positive, negative, and edge-case test scenarios with realistic data.

The architecture establishes an intelligent workflow where the Analyzer Agent fetches and analyzes Lambda code using Amazon Bedrock, the Generator Agent creates test cases based on learned patterns, and the Validator Agent ensures quality through deduplication and ranking. An Amazon DynamoDB-based memory store enables continuous learning from user feedback, improving test generation accuracy over time. Amazon Cognito provides secure user authentication with JSON Web Token (JWT)-based API authorization. Amazon Bedrock Guardrails provide prompt attack filtering, sensitive information redaction, and content safety enforcement on all Amazon Bedrock API calls. Key Capabilities:

  • Multi-Language Support: Python, Java, C#, JavaScript/TypeScript, and Ruby Lambda functions

  • Intelligent Code Chunking: Automatically splits large codebases into manageable chunks for analysis

  • Parallel Processing: Concurrent Amazon Bedrock API calls for faster test generation (5 concurrent workers for code analysis, parallel per-chunk test generation)

  • Target-Specific Analysis: Focus on specific functions, classes, or files within Lambda code

  • Pattern Learning: Amazon DynamoDB-based memory store with target-specific and global pattern storage

  • Rejection Avoidance: Learns from rejected test cases to avoid common mistakes

  • Continuation Mechanism: Handles incomplete Amazon Bedrock responses with automatic continuation.

  • Ignore Patterns: Exclude test files, dependencies, and non-code files from analysis.

  • Serverless Deployment: Runs on Amazon Bedrock AgentCore with Cognito-based authentication

  • Security Guardrails: Amazon Bedrock Guardrails for prompt attack filtering, Personally identifiable information (PII) redaction, content safety, and denied topic blocking on all AI calls

    This pattern is ideal for development teams and organizations that want to accelerate Lambda testing, improve test coverage, and maintain high code quality through AI-assisted test generation. This pattern uses AWS managed services to simplify test creation, enhance quality through learning, and scale to meet evolving testing needs.

Prerequisites and limitations

Prerequisites

To successfully implement this pattern, make sure that the following are in place:

  • An active AWS account - An AWS account with permissions to access Lambda functions, invoke Amazon Bedrock models, create DynamoDB tables, manage Cognito user pools, and deploy Amazon Bedrock AgentCore agents.

  • Amazon Bedrock model access - Anthropic Claude Sonnet 4 (us.anthropic.claude-sonnet-4-6) enabled in your AWS Region. For setup instructions, see Model access in the Amazon Bedrock documentation.

  • A development environment that consists of:

    - Python 3.11 or higher

    - AWS CLI installed and configured

    - Git for cloning the repository

  • AWS Lambda functions - At least one deployed Lambda function with accessible source code for analysis. The function must include source files (.py, .js, .java, .cs, .rb) in the deployment package, not just compiled bytecode.

  • Supported Lambda runtimes:

    - Python 3.x (all versions)

    - Node.js (JavaScript/TypeScript)

    - Java 8, 11, 17, 21 (requires .java source files in deployment package)

    - .NET Core/.NET 6+ (C#) (requires .cs source files in deployment package)

    - Ruby 2.7, 3.2

Limitations

  • This requires Amazon Bedrock model access in us-east-1 region for Claude Sonnet 4.6. AWS Lambda functions can be in any region.

  • Pattern data in Amazon DynamoDB expires after 90 days (configurable via Time to Live (TTL)) to maintain relevance and control costs.

  • Java and C# Lambda functions must include source files (.java, .cs) in deployment packages. Compiled-only packages (.class, .dll) cannot be analyzed.

  • The system implements in-memory per-user rate limiting (5 requests per 60 seconds). For distributed deployments with multiple AgentCore instances, rate limiting would need to be moved to DynamoDB or Redis for consistency.

  • Some AWS services aren't available in all AWS Regions. For Region availability, see AWS services by Region. For specific endpoints, see the Service endpoints and quotas page, and choose the link for the service.

Product versions

Architecture

Target architecture

The following diagram shows the architecture and workflow for this pattern:

In this workflow:

  1. User provides input: Developer interacts with Streamlit UI (app.py) running locally, provides Lambda function name, optional custom instructions for test generation, target filter to focus on specific functions/classes/files, and ignore patterns to exclude test files or dependencies.

  2. Cognito Authentication: Application authenticates request with Cognito and returns JWT (ID/Access token) to Streamlit.

  3. AgentCore API Invocation: Streamlit invokes AgentCore API with Bearer Token + payload (function name, filters, instructions, ignore patterns).

  4. JWT Validation: AgentCore API validates JWT using Cognito token signature.

  5. Request Routing: AgentCore API routes request to Lambda Test Generator workflow (AgentCore Runtime).

  6. Code Analysis Request: Analyzer Agent initiates a request to fetch the target Lambda function's code and metadata.

  7. AWS Authentication for Lambda Access: Boto3 authenticates with AWS using the AgentCore execution role, requesting read access permissions.

  8. AWS Lambda Code Retrieval and Processing: IAM role authorizes access and fetches Lambda function code as a ZIP file, extracts source files, filters out dependencies (node_modules, venv, etc.) and non-code files, applies user-defined ignore patterns, and chunks code into manageable pieces.

  9. Amazon Bedrock Authentication: Boto3 Amazon Bedrock client authenticates with AWS IAM for Amazon Bedrock access, requesting invoke model permissions to use Anthropic Claude Sonnet 4.6.

  10. AI-Powered Code Analysis: IAM role authorizes and sends code chunks to Amazon Bedrock (us-east-1) using Anthropic Claude Sonnet 4 (us.anthropic.claude-sonnet-4-6) with Amazon Bedrock Guardrail applied for prompt attack filtering and content safety. Amazon Bedrock performs Regex + LLM enhanced analysis to extract actual input patterns (e.g., event['body'], headers['Authorization']), output patterns (e.g., statusCode, response body structure), dependencies, error handling patterns, and edge cases from the code.

  11. Analysis results are packaged: Analyzer Agent packages the analysis results into an AnalysisResult object containing code chunks, input patterns, output patterns, dependencies, error patterns, and metadata, then passes it to the Generator Agent.

  12. Test generation process starts: Generator Agent receives analysis results and initiates a request to generate test cases, first querying DynamoDB memory store for historical patterns to learn from.

  13. Amazon DynamoDB Authentication for Memory Access: Boto3 DynamoDB client authenticates with AWS IAM for read access permissions to retrieve stored patterns.

  14. Historical patterns are retrieved from Database: IAM role authorizes and queries DynamoDB memory store table (lambda-testcase-memory) to fetch previously accepted patterns and rejected patterns (failed tests with rejection reasons) for the specific Lambda function.

  15. Amazon Bedrock Authentication for Test Generation: Boto3 Amazon Bedrock client authenticates again with AWS IAM for Amazon Bedrock access to generate test cases.

  16. AI Test Case Generation: IAM role authorizes and sends analysis results combined with memory patterns to Amazon Bedrock. Amazon Bedrock generates test case schemas with realistic input events based on actual code patterns, creating positive tests (35%), negative tests (35%), and edge cases (30%). Generation happens per-chunk in parallel for efficiency, applies learned patterns from memory, avoids rejected patterns, and chunks generation if needed for large codebases. All Bedrock converse() calls include guardrailConfig for prompt attack filtering, PII redaction, and denied topic enforcement.

  17. Test validation begins: Generator Agent passes the generated test case candidates to the Validator Agent for quality control, deduplication, and final selection.

  18. Validation process is configured: Validator Agent receives test candidates and initiates validation process, requesting access to DynamoDB memory store for scoring and validation.

  19. DynamoDB Authentication for Validation: Boto3 DynamoDB client authenticates with AWS IAM for read access to query memory patterns for validation scoring.

  20. Test quality is assessed: IAM role authorizes and uses DynamoDB memory store to score test cases based on previous pattern success rates, function coverage and code complexity. Validator performs structural validation, deduplication using pattern hashing, quality scoring with confidence boosts for handler functions and error handling, diversity selection to cover different chunks and test types, and selects the top N highest-quality, most diverse test cases.

  21. Final test cases returned to AgentCore: Validator Agent returns the final validated test cases with metadata (confidence scores, descriptions, input events, categories) to the main orchestrator, which formats and returns them back to the Amazon Bedrock AgentCore API

  22. Results delivered to UI: AgentCore API returns the generated tests back to the Streamlit UI for display with analysis summary, generation metadata, and test case details.

  23. User reviews and provides feedback: Developer reviews the displayed test cases in Streamlit UI, evaluates each test for quality and relevance, accepts good test cases or rejects poor ones with specific rejection reasons (missing_auth_headers, wrong_status_code, unrealistic_data, missing_required_fields, incorrect_event_source, etc.) and optional custom notes explaining the rejection, then submits feedback.

  24. Feedback is submitted to the system: Streamlit UI sends the collected feedback (accepted/rejected status, rejection reasons, custom notes) to the AgentCore API (save_feedback).

  25. AgentCore routes feedback: AgentCore API calls Validator agent which contains logic to store feedback in DynamoDB.

  26. Feedback storage process begins: Validator Agent initiates the process to store the feedback.

  27. DynamoDB write access is authenticated: Boto3 DynamoDB client authenticates with AWS IAM (AgentCore Execution role) for write access permissions to store feedback patterns.

  28. Learning patterns are stored: IAM role authorizes and stores user feedback in DynamoDB memory store table. Each pattern is stored with a composite partition key (function_name#target_function or function_name#GLOBAL), composite sort key (FEEDBACK#accepted/rejected#PATTERN#hash), pattern hash for deduplication, test type, input pattern structure, feedback status, rejection reason (if rejected), custom notes, usage count, success rate, timestamp, and TTL of 90 days for automatic cleanup. This stored data enables the system to learn from user feedback and improve future test generation.

Automation and scale

This pattern scales automatically using AWS managed services. Amazon Bedrock handles AI inference on-demand with a 200K token context window and 64K token output per call, and Amazon DynamoDB uses on-demand billing that automatically adjusts to traffic patterns. The system uses parallel processing for speed, the Analyzer Agent makes 5 concurrent Bedrock calls to analyze code chunks (ThreadPoolExecutor with max_workers=5), while the Generator Agent processes chunks in parallel with 5 concurrent workers. A continuation mechanism handles incomplete responses by automatically retrying requests. Rate limiting (5 requests per 60 seconds per user) prevents cost abuse from excessive Bedrock API calls.

Cost optimization includes TTL-based cleanup that removes patterns older than 90 days, composite key queries with begins_with() for instant lookups without table scans, and BatchWriteItem operations that reduce DynamoDB write operations by approximately 90%. Performance depends on function size, small functions (under 10 files) generate 10 test cases in 30–60 seconds, medium functions (10–50 files) take 1–3 minutes, and large functions (50+ files) take 3–5 minutes, though using target filters to focus on specific code sections reduces time by 50–70%.

Tools

AWS services

  • Amazon Bedrock – Provides generative AI capabilities through Anthropic Claude Sonnet 4 for code analysis, test generation, validation, and rejection summarization. Always uses us-east-1 region for model access.

  • Amazon Bedrock AgentCore – Provides serverless agent runtime for deploying and hosting the test generation backend with automatic scaling, OAuth-based authorization, and CloudWatch observability.

  • Amazon Bedrock Guardrails – Provides ML-based security filtering on all Bedrock API calls including prompt attack detection (HIGH strength), content filtering, PII anonymization (email, phone, name), blocking of AWS access keys/private keys/JWT tokens, and denied topic enforcement (exploit code generation, raw source code output). Deployed via CloudFormation.

  • AWS CloudFormation – Automates complete infrastructure provisioning including DynamoDB table, Cognito User Pool, Amazon Bedrock Guardrail with versioning, and IAM execution role for AgentCore.

  • Amazon Cognito – Provides user authentication with email-based sign-up, JWT token issuance, and secure API authorization for the AgentCore backend.

  • Amazon DynamoDB – Stores accepted and rejected test patterns with usage statistics for continuous learning and improvement. Uses composite keys (function_target, pattern_sk) for zero-scan queries and target-specific pattern storage.

  • AWS Lambda – Source of function code for analysis, the GetFunction API retrieves code and configuration. The tool supports Python, Node.js, Java, .NET, and Ruby runtimes.

Other tools

  • Python 3.11+ – Runtime environment for the application and agent orchestration.

  • Streamlit – Web-based user interface for authentication, test generation, feedback collection, and system status monitoring.

  • Boto3 – AWS SDK for Python to interact with Lambda, Amazon Bedrock, DynamoDB, and Cognito services.

Code repository

The code for this pattern is available in Github - Lambda Test Event Generator.

Best practices

This pattern implements the following best practices:

  • Uses least-privilege IAM policies for AWS Lambda (read-only), Amazon Bedrock (invoke), Amazon DynamoDB (query/write), and Amazon Cognito (authentication) access.

  • Implement target-specific pattern learning with global fallback for improved test accuracy.

  • Enable DynamoDB TTL for automatic cleanup of old patterns (90 days) to control storage costs.

  • Use zero-scan queries with composite keys (function_target, pattern_sk) for fast pattern retrieval.

  • Apply ignore patterns to exclude test files, dependencies, and non-code files from analysis.

  • Use multi-language code chunking with Abstract Syntax Tree (AST) parsing (Python) and regex patterns (Java, C#, JS, Ruby).

  • Store patterns with actual values (not just structure) for true deduplication.

  • Deploy backend on Amazon Bedrock AgentCore for serverless scaling and managed infrastructure.

  • Authenticate users via Amazon Cognito with JWT token validation on every API request.

  • Apply Amazon Bedrock Guardrails on all converse() calls for prompt attack filtering, PII redaction, sensitive data blocking (AWS keys, private keys, JWTs), and denied topic enforcement.

  • Sanitize analysis results before returning to users - all raw source code chunks are removed from responses, ensuring Lambda source code never leaves the AgentCore runtime boundary.

  • Validate all API inputs with regex patterns and length limits (function name max 170 chars, custom instructions max 2000 chars, max 50 ignore patterns) to prevent injection and abuse.

  • Implement per-user rate limiting (5 requests per 60 seconds) to prevent cost abuse from excessive Bedrock API calls.

  • Sanitize error messages before returning to users - internal file paths, AWS SDK details, and infrastructure information are never exposed in error responses.

Consider the following additional best practices:

  • Provide specific feedback reasons when rejecting test cases to improve learning accuracy.

  • Generate tests iteratively (2-3 times) for the same function to allow the system to learn and improve.

  • Provide custom instructions when you need specific test scenarios or data formats.

  • Enable IAM Access Analyzer to monitor resource permissions and identify unintended access.

  • Start with small Lambda functions to understand the system before analyzing large codebases.

  • Review IAM policies regularly and remove unused permissions.

  • Use strong password policies and enable Cognito AdvancedSecurityMode for threat detection.

Epics

TaskDescriptionSkills required

Clone the repository.

Clone the GitHub repository to your local system and navigate to the project directory:

git clone https://github.com/aws-samples/sample-lambda-test-event-generator.git

cd sample-lambda-test-event-generator

This repository contains the Python application, CloudFormation template, and configuration files.

App developer

Configure AWS Credentials.

Configure your AWS credentials to enable the AWS CLI to interact with your AWS account and to enable the application to access Lambda functions you want to test.

You can do this using the AWS CLI configuration command:

aws configure

When prompted, provide the following information:

  • AWS Access Key ID: Your AWS access key

  • AWS Secret Access Key: Your AWS secret access key

  • Default region name: The AWS region where you want to deploy the resources (e.g., us-east-1)

  • Default output format: The preferred output format (e.g., json)

App developer
TaskDescriptionSkills required

Deploy infrastructure using CloudFormation.

  1. Deploy the complete backend infrastructure (DynamoDB, Cognito, IAM roles) using the provided CloudFormation template. Run the following command in your CLI:

    aws cloudformation create-stack \
      --stack-name lambda-test-generator-infra \
      --template-body file://cloudformation/complete-infrastructure.yaml \
      --capabilities CAPABILITY_NAMED_IAM \
      --region us-east-1

  2. Wait for the stack deployment to complete:

    aws cloudformation wait stack-create-complete \
      --stack-name lambda-test-generator-infra \
      --region us-east-1

  3. Retrieve all outputs:

    aws cloudformation describe-stacks \
      --stack-name lambda-test-generator-infra \
      --query 'Stacks[0].Outputs' \
      --output table

What gets created:

  • DynamoDB table with TTL enabled, point-in-time recovery, and server-side encryption.

  • Cognito User Pool with email authentication, AdvancedSecurityMode ENFORCED, optional MFA (TOTP), and admin-only user creation.

  • Amazon Bedrock Guardrail with prompt attack filtering, PII redaction, content filtering, and denied topic blocking.

  • IAM execution role for AgentCore with least-privilege policies including bedrock:ApplyGuardrail permission.

Note

The stack creates all required resources. Ensure the CloudFormation template completes successfully before proceeding to the next step.

App developer

Export configuration variables.

Export all values from the CloudFormation stack outputs as environment variables:

export AGENTCORE_ROLE_ARN=$(aws cloudformation describe-stacks \
      --stack-name lambda-test-generator-infra \
      --query 'Stacks[0].Outputs[?OutputKey==`AgentCoreExecutionRoleArn`].OutputValue' \
      --output text)

export DISCOVERY_URL=$(aws cloudformation describe-stacks \
      --stack-name lambda-test-generator-infra \
      --query 'Stacks[0].Outputs[?OutputKey==`DiscoveryUrl`].OutputValue' \
      --output text)

export CLIENT_ID=$(aws cloudformation describe-stacks \
      --stack-name lambda-test-generator-infra \
      --query 'Stacks[0].Outputs[?OutputKey==`ClientId`].OutputValue' \
      --output text)

export DYNAMODB_TABLE=$(aws cloudformation describe-stacks \
      --stack-name lambda-test-generator-infra \
      --query 'Stacks[0].Outputs[?OutputKey==`DynamoDBTableName`].OutputValue' \
      --output text)

export COGNITO_POOL_ID=$(aws cloudformation describe-stacks \
      --stack-name lambda-test-generator-infra \
      --query 'Stacks[0].Outputs[?OutputKey==`UserPoolId`].OutputValue' \
      --output text)

export BEDROCK_GUARDRAIL_ID=$(aws cloudformation describe-stacks \
  --stack-name lambda-test-generator-infra \
  --query 'Stacks[0].Outputs[?OutputKey==`BedrockGuardrailId`].OutputValue' \
  --output text)

export BEDROCK_GUARDRAIL_VERSION=$(aws cloudformation describe-stacks \
  --stack-name lambda-test-generator-infra \
  --query 'Stacks[0].Outputs[?OutputKey==`BedrockGuardrailVersion`].OutputValue' \
  --output text)

These variables are used in subsequent steps for AgentCore configuration and .env file creation.

AWS administrator
TaskDescriptionSkills required

Create virtual environment.

  1. Create and activate a Python virtual environment to isolate project dependencies::

    python3 -m venv venv

source venv/bin/activate

  2. Install the required Python packages from the requirements file:

    pip install -r requirements.txt

    This installs all necessary libraries including boto3 for AWS SDK, python-dotenv for environment configuration, bedrock-agentcore-runtime for AgentCore integration, and Streamlit for the user interface.

    Note

    The virtual environment must be activated (using source venv/bin/activate) each time you open a new terminal session to run the application.

App developer

Configure and deploy AgentCore.

  • Configure the AgentCore agent:

    agentcore configure \
  --entrypoint main.py \
  --name lambda_test_generator \
  --requirements-file requirements.txt \
  --region us-east-1 \
  --execution-role $AGENTCORE_ROLE_ARN

  • When prompted:

    1. Deployment type: 1 (Direct Code Deploy - no Docker needed)

    2. Python version: 2 (PYTHON_3_11 or select your particular Python Version)

    3. S3 bucket: Press Enter (auto-create)

    4. OAuth authorizer: yes

    5. Discovery URL: Paste $DISCOVERY_URL value

    6. Client IDs: Paste $CLIENT_ID value

    7. Audience: Press Enter (leave empty)

    8. Scopes: Press Enter (leave empty)

    9. Custom claims: Press Enter (leave empty)

    10. Request headers: yes

    11. Headers: Authorization

    12. Memory: s (skip - using DynamoDB)

  • Deploy the agent:

    agentcore deploy \
  --env DYNAMODB_TABLE_NAME=$DYNAMODB_TABLE \
  --env AWS_REGION=us-east-1 \
  --env BEDROCK_GUARDRAIL_ID=$BEDROCK_GUARDRAIL_ID \
  --env BEDROCK_GUARDRAIL_VERSION=$BEDROCK_GUARDRAIL_VERSION

  • Get the AgentCore runtime ID and endpoint:

    RUNTIME_ID=$(agentcore status | grep "Agent ARN:" | sed 's/.*runtime\///' | sed 's/[│ ].*//')
AGENTCORE_ENDPOINT="https://bedrock-agentcore-runtime.us-east-1.amazonaws.com/agents/${RUNTIME_ID}/endpoints/DEFAULT"
App developer

Create .env file for local development.

Create a .env file in the project root directory with all configuration values:

cat > .env << EOF
AWS_REGION=us-east-1
DYNAMODB_TABLE_NAME=$DYNAMODB_TABLE
COGNITO_POOL_ID=$COGNITO_POOL_ID
COGNITO_CLIENT_ID=$CLIENT_ID
COGNITO_REGION=us-east-1
AGENTCORE_ENDPOINT=$AGENTCORE_ENDPOINT
BEDROCK_GUARDRAIL_ID=$BEDROCK_GUARDRAIL_ID
BEDROCK_GUARDRAIL_VERSION=$BEDROCK_GUARDRAIL_VERSION
EOF

Test connectivity to AWS services:

aws dynamodb describe-table --table-name $DYNAMODB_TABLE
aws bedrock list-foundation-models --region us-east-1

Both commands should return successful responses. If you encounter permission errors, verify that the IAM policies are properly configured.

App developer
TaskDescriptionSkills required

Create a new Cognito User.

The Cognito User Pool is configured with admin-only user creation for security, so users cannot self-register. Create a user via the AWS CLI using the Cognito Pool ID and Client ID exported from the CloudFormation stack outputs. Create a new user (replace user@example.com with your email):

aws cognito-idp admin-create-user \
  --user-pool-id $COGNITO_POOL_ID \
  --username user@example.com \
  --user-attributes Name=email,Value=user@example.com Name=email_verified,Value=true \
  --temporary-password '[PASSWORD]!' \
  --region us-east-1

Set a permanent password (minimum 8 characters, must include uppercase, lowercase, and a number):

aws cognito-idp admin-set-user-password \
  --user-pool-id $COGNITO_POOL_ID \
  --username user@example.com \
  --password '[PASSWORD]!' \
  --permanent \
  --region us-east-1

Use these credentials to sign in through the Streamlit UI in the next step.

App developer
TaskDescriptionSkills required

Launch Streamlit UI.

Start the application:

streamlit run app.py

The UI will open in your default browser at http://localhost:8501

Sign in with the credentials created in the previous step

App developer

Configure generation options

In the UI, configure the following options:

  1. Lambda function name or ARN: Enter the target Lambda function identifier

  2. Target Filter (Optional): Specify a function, class, or file to focus on (e.g., validate_user, UserService, auth.py)

  3. Ignore Patterns (Optional): Add file or folder patterns to exclude, one per line:

    • tests/ - Ignore tests directory

    • *.test.js - Ignore test files

    • mock_data/ - Ignore mock data

  4. Custom Instructions (Optional): Add specific testing requirements

App developer

Generate test cases.

Click the "Generate Test Cases" button. The system will:

  1. Authenticate with AgentCore using Cognito JWT token.

  2. Fetch Lambda code from AWS.

  3. Apply ignore patterns to exclude unwanted files.

  4. Chunk code using language-specific strategies.

  5. Apply target filter if specified.

  6. Analyze code with Amazon Bedrock.

  7. Query DynamoDB for learned patterns.

  8. Generate test cases.

  9. Validate and rank test cases.

App developer

Review generated tests.

Review each generated test case, which includes:

  • Type: Positive (valid inputs), Negative (invalid inputs), or Edge (boundary conditions)

  • Description: What the test validates

  • Test data: The actual test event payload

DevOps engineer, App developer

Provide feedback.

For each test case, provide feedback:

To accept a test case:

  • Click the "Accept" button

To reject a test case:

  1. Click the "Reject" button

  2. Select a rejection reason from the dropdown or add a custom reason

  3. Click the "Submit Rejection" button

Test engineer

Save feedback to memory.

After reviewing all test cases:

  1. Click the "Save All Feedback to Memory" button

  2. The system validates that all rejected cases have submitted reasons

  3. Feedback is sent to AgentCore API which routes to the Validator Agent.

  4. Feedback is batch-stored to DynamoDB with pattern hash for deduplication.

  5. View the feedback summary showing accepted/rejected counts

Test engineer

Iterate for improvement.

Generate tests 2-3 more times for the same function to improve quality:

  • The system retrieves learned patterns from DynamoDB

  • Quality improves with each iteration based on your feedback

  • Use target filter for focused testing on specific components

  • Use ignore patterns to exclude irrelevant code and improve generation quality

Note

The learning system becomes more effective with repeated use. Each feedback cycle helps the AI understand your testing preferences and generate more relevant test cases for your Lambda functions.

App developer
TaskDescriptionSkills required

Invoke AgentCore API directly.

For automation and integration, invoke the AgentCore backend directly using API calls with Cognito authentication.

Get a Cognito token:

TOKEN=$(agentcore identity get-cognito-inbound-token)

Invoke test generation:

agentcore invoke --bearer-token "$TOKEN" '{"action": "generate_test_cases", "function_name": "your-lambda-function", "num_test_cases": 10,"custom_instructions": "Focus on authentication scenarios", "target_filter": "validate_user","ignore_patterns": ["tests/", "*.test.js"] }'

Or using curl:

curl -X POST "$AGENTCORE_ENDPOINT" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "action": "generate_test_cases", "function_name": "your-lambda-function", "num_test_cases": 10}'
App developer
TaskDescriptionSkills required

Monitor DynamoDB patterns and track costs.

  1. Query the DynamoDB table to review accepted patterns and understand what the system has learned:

    aws dynamodb query \
  --table-name $DYNAMODB_TABLE \
  --key-condition-expression "function_target = :ft AND begins_with(pattern_sk, :prefix)" \
  --expression-attribute-values '{":ft":{"S":"my-function#GLOBAL"},":prefix":{"S":"FEEDBACK#accepted"}}'

    Replace my-function with your actual Lambda function name to view patterns specific to that function.

  2. Track resource consumption to optimize costs:

    • AWS Cost Explorer: Monitor overall spending on Bedrock, DynamoDB, and Lambda services

    • Amazon Bedrock: Review token usage and API call metrics in the Bedrock console

    • DynamoDB: Check request metrics, consumed capacity, and storage usage.

    • AgentCore: Monitor agent runtime metrics and invocation counts in CloudWatch

  3. Use CloudWatch Logs to diagnose issues and monitor system behavior:

    • AgentCore logs: 

      aws logs tail /aws/bedrock-agentcore/runtimes/lambda_test_generator --follow

    • Application errors: Review application-level errors and exceptions

    • Bedrock API responses: Analyze AI model responses and token consumption

    • DynamoDB operations: Monitor read/write patterns and throttling events

Note

Regular monitoring helps identify cost optimization opportunities and ensures the system continues to learn effectively. The TTL configuration on your DynamoDB table automatically cleans up old patterns, helping manage storage costs over time.

AWS administrator
TaskDescriptionSkills required

Delete deployed resources.

To remove all deployed resources:

Delete the AgentCore agent:

agentcore destroy

Delete the CloudFormation stack (deletes DynamoDB table, Cognito User Pool, and IAM role):

aws cloudformation delete-stack 
\ --stack-name lambda-test-generator-infra 
\ --region us-east-1
AWS administrator

Troubleshooting

IssueSolution

"Access Denied" error when fetching Lambda code

  • Verify IAM permissions include lambda:GetFunction and lambda:GetFunctionConfiguration for the target Lambda function

  • Check your AWS CLI credentials

    aws sts get-caller-identity

"DynamoDB write errors"

  • Verify the table name in your .env file matches the deployed table name

  • Check CloudFormation outputs:

    aws cloudformation describe-stacks --stack-name lambda-test-generator-infra --query 'Stacks[0].Outputs[?OutputKey=DynamoDBTableName].OutputValue' --output text

  • Or verify directly:

    aws dynamodb describe-table --table-name $DYNAMODB_TABLE

No test cases generated

Possible causes and solutions:

  • Lambda function code is not accessible: Check IAM permissions for Lambda access

  • Lambda deployment package contains only compiled code: .class, .dll files without source files cannot be analyzed

  • All files filtered out by ignore patterns: Review and adjust your ignore patterns

  • Custom instructions are too restrictive: Simplify or remove custom instructions

  • Verify the Lambda function exists:

    aws lambda get-function --function-name <name>

Java/C# Lambda shows "No source code found"

Java and C# Lambdas require source files in the deployment package:

Java (Maven):

  1. Add source inclusion configuration to pom.xml (see docs/JAVA_SETUP.md)

  2. Rebuild and redeploy your Lambda function

C# (.NET):

  1. Include .cs files in build configuration (see docs/CSHARP_SETUP.md)

  2. Rebuild and redeploy your Lambda function

DynamoDB write errors

  • Check DynamoDB permissions and table status:

  • Verify IAM policy includes dynamodb:PutItem and dynamodb:BatchWriteItem

  • Check table status: 

    aws dynamodb describe-table --table-name $DYNAMODB_TABLE

    The table name includes the stack name suffix. Use the value from your CloudFormation outputs or .env file.

  • Enable CloudWatch Logs for DynamoDB to see detailed error messages

Slow test generation

Optimize generation speed:

  1. Use target filter to focus on specific functions (50-70% faster)

  2. Add ignore patterns to exclude test files, dependencies, and irrelevant code

  3. Reduce code size by filtering out unnecessary files before analysis

AgentCore: "Agent not found"

Verify agent is deployed:

agentcore status

agentcore configure list

AgentCore: "Permission denied" at runtime

Verify execution role has correct policies:

aws iam list-role-policies --role-name agentcore-exec-lambda-test-generator-infra

Cognito: "Invalid username or password"

Verify your credentials are correct. You can create a new user using the Streamlit UI's "Create Account" flow, or via the AWS CLI:

aws cognito-idp admin-create-user --user-pool-id <POOL_ID> --username <username> --temporary-password '<password>' --region us-east-1
aws cognito-idp admin-set-user-password --user-pool-id <POOL_ID> --username <username> --password '<password>' --permanent --region us-east-1

"Bedrock Guardrail not configured" warning in logs

Verify BEDROCK_GUARDRAIL_ID and BEDROCK_GUARDRAIL_VERSION environment variables are set in the AgentCore deployment.

Check CloudFormation outputs: 

aws cloudformation describe-stacks --stack-name lambda-test-generator-infra --query 'Stacks[0].Outputs[?OutputKey==`BedrockGuardrailId`].OutputValue' --output text

"Rate limit exceeded" error

The system limits to 5 requests per 60 seconds per user. Wait the indicated time before retrying. For production use with higher throughput needs, adjust RATE_LIMIT_MAX_REQUESTS and RATE_LIMIT_WINDOW constants in main.py.

Related resources

AWS documentation