Cisco accomplishes 50% decrease in delay with Amazon SageMaker Inference rapid scaling option on AWS Machine Learning Blog.

Introduction to Webex by Cisco

Webex by Cisco is a leading provider of cloud-based collaboration solutions encompassing various features like video meetings, messaging, events, and customer experience solutions, specializing in contact center services and purpose-built collaboration devices. The company focuses on delivering inclusive collaboration experiences through innovative AI and Machine Learning technologies, ensuring security and privacy in all solutions.

Enhancing Collaboration with AI

Cisco’s Webex AI (WxAI) team plays a crucial role in integrating AI-driven features and functionalities, leveraging Large Language Models (LLMs) to improve user productivity and experiences within Webex solutions. Their work extends to Webex Contact Center, enhancing capabilities such as intelligent virtual assistants, natural language processing, and sentiment analysis for personalized customer support.

Optimizing AI Infrastructure with Amazon SageMaker

As the complexity of LLM models increased, the WxAI team faced challenges in efficiently allocating resources and starting applications. They successfully migrated LLMs to Amazon SageMaker Inference, enhancing speed, scalability, and price-performance, thereby improving the overall efficiency of AI/ML infrastructure at Cisco.

Improving Inference Auto Scaling with SageMaker

Cisco collaborated with Amazon SageMaker to improve inference auto scaling times, resulting in faster detection of scaling needs and reduced latency. The addition of new pre-defined metric types significantly enhanced autoscaling capabilities, leading to up to a 50% improvement in end-to-end inference latency for Generative AI workloads like Llama3-8B.

Future Outlook and Collaboration

With continuous efforts to optimize AI inference performance and advance generative AI capabilities, Cisco looks forward to leveraging Amazon SageMaker to enhance its Webex portfolio further. Collaboration between Cisco’s Webex AI team and Amazon SageMaker will continue to drive innovations in AI-driven collaboration, catering to the evolving needs of customers across regions.

Contributors

The article features insights from Travis Mehlinger, Karthik Raghunathan, Praveen Chamarthi, Saurabh Trikande, and Ravi Thakur, who are experts from Cisco and Amazon Web Services dedicated to advancing AI/ML technology and providing innovative solutions for customers globally.