\u201cModel selection and evaluation is not just done at the beginning, but is something that\u2019s repeated periodically,\u201d Sivasubramanian said. \u201cWe think having a human in the loop is important, so we are offering a way to manage human evaluation workflows and metrics of model performance easily.\u201d<\/p>\n<\/div>\n
\n
Sivasubramanian told The Verge<\/em> in a separate interview that often some developers don\u2019t know if they should use a larger model for the project because they assumed a more powerful one would handle their needs. They later find out they could\u2019ve built on a smaller one. <\/p>\n<\/div>\n\n
Model Evaluation has two components: automated evaluation and human evaluation. In the automated version, developers can go into their Bedrock console and choose a model to test. They can then assess the model\u2019s performance on metrics like robustness, accuracy, or toxicity for tasks like summarization, text classification, question and answering, and text generation.\u00a0Bedrock includes popular third-party AI models like Meta\u2019s Llama 2, Anthropic\u2019s Claude 2, and Stability AI\u2019s Stable Diffusion.<\/p>\n<\/div>\n
\n
While AWS provides test datasets, customers can bring their own data into the benchmarking platform so they\u2019re better informed of how the models behave. The system then generates a report.\u00a0<\/p>\n<\/div>\n
\n
If humans are involved, users can choose to work with an AWS human evaluation team or their own. Customers must specify the task type (summarization or text generation, for example), the evaluation metrics, and the dataset they want to use. AWS will provide customized pricing and timelines for those who work with its assessment team.\u00a0<\/p>\n<\/div>\n
\n
AWS vice president for generative AI Vasi Philomin told The Verge<\/em> in an interview that getting a better understanding of how the models perform guides development better. It also allows for companies to see if models don\u2019t meet some responsible AI standards \u2014 like lower or too high toxicity sensitivities \u2014 before building using the model.<\/p>\n<\/div>\n\n
\u201cIt\u2019s important that models work for our customers, to know which model best suits them, and we\u2019re giving them a way to better evaluate that,\u201d Philomin said.<\/p>\n<\/div>\n
\n
Sivasubramanian also said that when humans evaluate AI models, they can detect other metrics that the automated system can\u2019t \u2014 things like empathy or friendliness.<\/p>\n<\/div>\n
\n
AWS will not require all customers to benchmark models, said Philomin, as some developers may have worked with some of the foundation models on Bedrock before or have an idea of what the models can do for them. Companies that are still exploring which models to use could benefit from going through the benchmarking process.\u00a0<\/p>\n<\/div>\n
\n
AWS said that while the benchmarking service is in preview, it will only charge for the model inference used during the evaluation.\u00a0<\/p>\n<\/div>\n
\n
While there is no particular standard for benchmarking AI models, there are specific metrics that some industries generally accept. Philomin said the goal for benchmarking on Bedrock is not to evaluate models broadly but to offer companies a way to measure the impact of a model on their projects.\u00a0<\/p>\n<\/div>\n<\/div>\n
\n
Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Amazon wants users to evaluate AI models better and encourage more humans to be involved in the process.\u00a0 During the AWS re: Invent conference, AWS vice president of database, analytics, and machine learning Swami Sivasubramanian announced Model Evaluation on Bedrock, now available on preview, for models found in its repository Amazon Bedrock. Without a way […]<\/p>\n","protected":false},"author":1,"featured_media":57845,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":{"0":"post-57844","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-tech"},"_links":{"self":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/posts\/57844","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/comments?post=57844"}],"version-history":[{"count":0,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/posts\/57844\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/media\/57845"}],"wp:attachment":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/media?parent=57844"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/categories?post=57844"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/tags?post=57844"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}