{"id":33336,"date":"2023-08-21T22:04:37","date_gmt":"2023-08-21T22:04:37","guid":{"rendered":"https:\/\/entertainment.runfyers.com\/index.php\/2023\/08\/21\/the-new-york-times-blocks-openais-web-crawler\/"},"modified":"2023-08-21T22:04:37","modified_gmt":"2023-08-21T22:04:37","slug":"the-new-york-times-blocks-openais-web-crawler","status":"publish","type":"post","link":"https:\/\/entertainment.runfyers.com\/index.php\/2023\/08\/21\/the-new-york-times-blocks-openais-web-crawler\/","title":{"rendered":"The New York Times blocks OpenAI\u2019s web crawler"},"content":{"rendered":"<p> <br \/>\n<\/p>\n<div>\n<p class=\"duet--article--dangerously-set-cms-markup duet--article--standard-paragraph mb-20 font-fkroman text-18 leading-160 -tracking-1 selection:bg-franklin-20 dark:text-white dark:selection:bg-blurple [&amp;_a:hover]:shadow-highlight-franklin dark:[&amp;_a:hover]:shadow-highlight-blurple [&amp;_a]:shadow-underline-black dark:[&amp;_a]:shadow-underline-white\"><em>The New York Times<\/em> has blocked OpenAI\u2019s web crawler, meaning that OpenAI can\u2019t use content from the publication to train its AI models. If you check <a href=\"https:\/\/www.nytimes.com\/robots.txt\" target=\"_blank\" rel=\"noopener\">the <em>NYT\u2019s<\/em> robots.txt page<\/a>, you can see that the <em>NYT<\/em> disallows GPTBot, the crawler that OpenAI introduced <a href=\"https:\/\/www.theverge.com\/2023\/8\/7\/23823046\/openai-data-scrape-block-ai\" target=\"_blank\" rel=\"noopener\">earlier this month<\/a>. Based on the Internet Archive\u2019s Wayback Machine, it appears <em>NYT<\/em> blocked the crawler as early as August 17th.<\/p>\n<\/div>\n<div>\n<p class=\"duet--article--dangerously-set-cms-markup duet--article--standard-paragraph mb-20 font-fkroman text-18 leading-160 -tracking-1 selection:bg-franklin-20 dark:text-white dark:selection:bg-blurple [&amp;_a:hover]:shadow-highlight-franklin dark:[&amp;_a:hover]:shadow-highlight-blurple [&amp;_a]:shadow-underline-black dark:[&amp;_a]:shadow-underline-white\">The change comes after the <em>NYT<\/em> updated its terms of service at the beginning of this month to prohibit the use of its content <a href=\"https:\/\/www.theverge.com\/2023\/8\/14\/23831109\/the-new-york-times-ai-web-scraping-rules-terms-of-service\" target=\"_blank\" rel=\"noopener\">to train AI models<\/a>. <em>New York Times<\/em> spokesperson Charlie Stadtlander spokesperson declined to comment. OpenAI didn\u2019t immediately reply to a request for comment. <\/p>\n<\/div>\n<div>\n<p class=\"duet--article--dangerously-set-cms-markup duet--article--standard-paragraph mb-20 font-fkroman text-18 leading-160 -tracking-1 selection:bg-franklin-20 dark:text-white dark:selection:bg-blurple [&amp;_a:hover]:shadow-highlight-franklin dark:[&amp;_a:hover]:shadow-highlight-blurple [&amp;_a]:shadow-underline-black dark:[&amp;_a]:shadow-underline-white\"><em><strong>Update August 21st, 7:55PM ET<\/strong>: <\/em>The New York Times<em> declined to comment.<\/em><\/p>\n<\/div>\n<p><br \/>\n<br \/><a href=\"https:\/\/www.theverge.com\/2023\/8\/21\/23840705\/new-york-times-openai-web-crawler-ai-gpt\" target=\"_blank\" rel=\"noopener\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The New York Times has blocked OpenAI\u2019s web crawler, meaning that OpenAI can\u2019t use content from the publication to train its AI models. If you check the NYT\u2019s robots.txt page, you can see that the NYT disallows GPTBot, the crawler that OpenAI introduced earlier this month. Based on the Internet Archive\u2019s Wayback Machine, it appears [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":33337,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":{"0":"post-33336","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-tech"},"_links":{"self":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/posts\/33336","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/comments?post=33336"}],"version-history":[{"count":0,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/posts\/33336\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/media\/33337"}],"wp:attachment":[{"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/media?parent=33336"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/categories?post=33336"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/entertainment.runfyers.com\/index.php\/wp-json\/wp\/v2\/tags?post=33336"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}