Automate Amazon Bedrock bulk predictions: Constructing a scalable and effective process | AWS Machine Learning Blog

Introduction to Amazon Bedrock

Amazon Bedrock is a fully managed service that provides a selection of high-performing foundation models (FMs) from top AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon. This service offers a comprehensive set of tools necessary to develop generative AI applications with a focus on security, privacy, and responsible AI.

Batch Inference in Amazon Bedrock

Batch inference in Amazon Bedrock is designed to efficiently handle large data processing tasks utilizing foundation models (FMs) when real-time results are not immediate. This feature is advantageous for workloads that are not sensitive to latency, such as obtaining embeddings, entity extraction, FM-as-judge evaluations, text categorization, and summarization for business reporting tasks. It is also cost-effective, with a 50% discount on batch inference workloads compared to On-Demand pricing.

Scalable Solution using AWS Lambda and Amazon DynamoDB

To address the limitation of 10 batch inference jobs per model per Region in Amazon Bedrock, a scalable solution using AWS Lambda and Amazon DynamoDB has been developed. This solution automates the monitoring of available job slots and submission of new jobs as slots become available, enhancing the efficiency of batch inference workflows on Amazon Bedrock.

Benefits of Batch Inference in Amazon Bedrock

Batch inference in Amazon Bedrock allows organizations to process large volumes of data asynchronously, making it suitable for scenarios where real-time results are not critical. This capability is particularly useful for tasks like asynchronous embedding generation, large-scale text classification, and bulk content analysis. The cost-effectiveness of batch inference, offered at a 50% discount, enables organizations to process large datasets more economically while handling substantial data volumes.

Automated Pipeline for Batch Inference

The automated pipeline introduced in this post leverages AWS Lambda, Amazon DynamoDB, and Amazon EventBridge to automate batch inference jobs in Amazon Bedrock. This solution enhances the ability to manage large-scale batch processing workflows efficiently and offers a scalable, cost-effective approach to processing large amounts of data using batch inference.

Conclusion

The automated pipeline for batch inference in Amazon Bedrock provides organizations with a scalable, efficient, and cost-effective solution to handle large-scale AI inference tasks. By implementing this solution and optimizing configurations based on workload patterns and requirements, organizations can enhance their capability to process large amounts of data using batch inference effectively.