Attain doubled throughput with 50% cost savings for generative AI inference on Amazon SageMaker using the latest optimization toolkit – Part 2 | AWS Machine Learning Blog

Introduction to Inference Optimization Toolkit

As generative artificial intelligence (AI) inference gains importance in business operations, organizations are looking for ways to optimize generative AI models to enhance productivity while balancing cost-effectiveness. However, the diverse requirements of different use cases present a challenge in selecting the right optimization techniques. In this article, we will explore the new inference optimization toolkit in Amazon SageMaker, designed to improve throughput and reduce costs for generative AI models.

Features of Inference Optimization Toolkit

The inference optimization toolkit in Amazon SageMaker leverages cutting-edge techniques such as compilation, quantization, and speculative decoding to streamline the process of optimizing generative AI models. These techniques help reduce the time needed to optimize models and achieve optimal price-performance ratios for various use cases.

Optimization Techniques

The toolkit uses the Neuron Compiler for compilation, Activation-aware Weight Quantization (AWQ) for quantization, and speculative decoding to enhance inference speed for longer text generation tasks. These techniques are applied to popular generative AI models like Llama 3, Mistral, and Mixtral models to demonstrate significant improvements in throughput and cost reduction.

Deploying Optimized Models

With the inference optimization toolkit, deploying pre-optimized models is simplified. Users can choose from pre-configured optimization settings based on their latency and throughput requirements, ensuring best-in-class cost-performance at scale. Deployment can be done with just a few clicks in SageMaker JumpStart or via the SageMaker Python SDK for custom optimizations.

Custom Optimization Options

For organizations that require tailored optimization configurations, the toolkit offers flexibility to create custom optimizations. Users can choose specific instance types, deployment options, and optimization techniques to meet their unique needs. SageMaker JumpStart provides a platform to explore and fine-tune these customizations effortlessly.

Conclusion

By leveraging the inference optimization toolkit in Amazon SageMaker, businesses can achieve up to ~2x higher throughput and cost savings of up to ~50% for generative AI inference. This empowers organizations to accelerate generative AI adoption, drive better business outcomes, and unlock new opportunities in the realm of artificial intelligence.