Streamline the machine learning model validation procedure using Amazon SageMaker Model Registry and Amazon SageMaker Pipelines | AWS Machine Learning Blog

Innovations in AI and ML

Innovations in artificial intelligence (AI) and machine learning (ML) are causing organizations to take a fresh look at the possibilities these technologies can offer. As you aim to bring your proofs of concept to production at an enterprise scale, you may experience challenges aligning with the strict security compliance requirements of their organization.

Challenges in ML Production

Organizations often want to introduce additional compliance checks that validate that the model aligns with their organizational standards before it is deployed. These frequent manual checks can create long lead times to deliver value to customers.

Transition to Automated Monitoring

This post illustrates how to use common architecture principles to transition from a manual monitoring process to one that is automated. You can use these principles and existing AWS services such as Amazon SageMaker Model Registry and Amazon SageMaker Pipelines to deliver innovative solutions to your customers while maintaining compliance for your ML workloads.

Model Approval Automation

Deployment of an ML model to production generally requires at least two artifacts to be approved: the model and the endpoint. Traditionally, organizations have created manual review processes to keep updated code from becoming available to the public through mechanisms such as an Enterprise Review Committee (ERC), Enterprise Review Board (ERB), or a Change Advisory Board (CAB). Just as mechanisms have evolved with the rise of continuous integration and continuous delivery (CI/CD), MLOps can reduce the need for manual processes while increasing the frequency and thoroughness of quality checks.

Automated Model Approval Workflow

This section details the code required for the automated monitoring workflow, including steps for evaluating model quality, bias, and feature importance metrics and updating the model status accordingly.

Complexities with LLMs

We explore how the complexities introduced by Large Language Models (LLMs) change the automated monitoring workflow, particularly focusing on monitoring metrics such as memorization and copyright, disinformation, bias, and toxicity.

Concluding Remarks

In this post, we discussed a sample solution to begin automating compliance checks for models going into production. By embedding expertise as code and running automated checks against models, organizations can increase both the speed and quality of models while scaling in the era of generative AI. If you have any thoughts or questions, please leave them in the comments section.