{"id":81068,"date":"2024-03-07T15:25:07","date_gmt":"2024-03-07T15:25:07","guid":{"rendered":"https:\/\/entertainment.runfyers.com\/index.php\/2024\/03\/07\/we-tested-anthropics-new-chatbot-and-came-away-a-bit-disappointed-techcrunch\/"},"modified":"2024-03-07T15:25:07","modified_gmt":"2024-03-07T15:25:07","slug":"we-tested-anthropics-new-chatbot-and-came-away-a-bit-disappointed-techcrunch","status":"publish","type":"post","link":"https:\/\/entertainment.runfyers.com\/index.php\/2024\/03\/07\/we-tested-anthropics-new-chatbot-and-came-away-a-bit-disappointed-techcrunch\/","title":{"rendered":"We tested Anthropic’s new chatbot — and came away a bit disappointed | TechCrunch"},"content":{"rendered":"


\n<\/p>\n

\n

This week, Anthropic, the AI startup backed by Google, Amazon and a who\u2019s who of VCs and angel investors, released a family of models \u2014 Claude 3<\/a> \u2014 that it claims bests OpenAI\u2019s GPT-4<\/a> on a range of benchmarks.<\/p>\n

There\u2019s no reason to doubt Anthropic\u2019s claims. But we at TechCrunch would argue that the results Anthropic cites \u2014 results from highly technical and academic benchmarks \u2014 are a poor corollary to the average user\u2019s experience.<\/p>\n

That\u2019s why we designed our own test \u2014 a list of questions on subjects that the average person might ask about, ranging from politics to healthcare.<\/p>\n

As we did with Google\u2019s current flagship GenAI model, Gemini Ultra<\/a>, a few weeks back, we ran our questions through the most capable of the Claude 3 models \u2014 Claude 3 Opus \u2014 to get a sense of its performance.<\/p>\n

Background on Claude 3<\/h2>\n

Opus, available on the web in a chatbot interface with a subscription to Anthropic\u2019s Claude Pro plan and through Anthropic\u2019s API, as well as through Amazon\u2019s Bedrock<\/a> and Google\u2019s Vertex AI<\/a> dev platforms, is a multimodal model. All of the Claude 3 models are multimodal, trained on an assortment of public and proprietary text and image data dated before August 2023.<\/p>\n

Unlike some of its GenAI rivals, Opus doesn\u2019t have access to the web, so asking it questions about events after August 2023 won\u2019t yield anything useful (or factual). But all Claude 3 models including Opus do\u00a0<\/em>have very large context windows.<\/p>\n

A model\u2019s context, or context window, refers to input data (e.g. text) that the model considers before generating output (e.g. more text). Models with small context windows tend to forget the content of even very recent conversations, leading them to veer off topic.<\/p>\n

As an added upside of large context, models can better grasp the flow of data they take in and generate richer responses \u2014 or so some vendors (including Anthropic) claim.<\/p>\n

Out of the gate, Claude 3 models support a 200,000-token context window, equivalent to about 150,000 words or a short (~300-page) novel, with select customers getting up a 1-milion-token context window (~700,000 words). That\u2019s on par with Google\u2019s newest GenAI model, Gemini 1.5 Pro<\/a>, which also offers up to a 1-million-token context window \u2014 albeit a 128,000-token context window by default.<\/p>\n

We tested the version of Opus with a 200,000-token context window.<\/p>\n

Testing Claude 3<\/h2>\n

Our benchmark for GenAI models touches on trivia, medical and therapeutic advice and generating and summarizing content \u2014 all things that a user might ask (or ask of) a chatbot.<\/p>\n

We prompted Opus with a set of over two dozen questions ranging from relatively innocuous (\u201cWho won the football world cup in 1998?\u201d) to controversial (\u201cIs Taiwan an independent country?\u201d). Our benchmark is constantly evolving as new models with new capabilities come out, but the goal remains the same: to approximate the average user\u2019s experience.<\/p>\n

Questions<\/h2>\n

Evolving news stories<\/h3>\n

We started by asking Opus the same current events questions that we asked<\/a> Gemini Ultra not long ago:<\/p>\n