* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, October 4, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Al Roker Gives Olivia Dean an Unexpected ‘New Job’ on the ‘Today’ Show – Yahoo

    Al Roker Shocks Olivia Dean with an Exciting New Role on the ‘Today’ Show

    Books about the arts and some haunts for a Denton October – Denton Record-Chronicle

    Uncover Artistic Treasures and Spooky Adventures to Experience in Denton This October

    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    MAC Brings iPad Technology to Football Sidelines Across All 13 Member Schools – Sports Video Group

    MAC Brings iPad Technology to Football Sidelines Across All 13 Member Schools – Sports Video Group

    Technology Is Becoming More Important Than Humans In CX – No Jitter

    Technology Is Becoming More Important Than Humans In CX – No Jitter

    A Tech Expo Shows What China Can Make, but Not Who’ll Buy It All – The New York Times

    Inside China’s Tech Expo: Cutting-Edge Innovations Face Uncertain Demand

    Steampunk Metal Oval Technology Sense Sunglasses Personality Handmade Chain Multicolor Sunglasses UV400 – The San Joaquin Valley Sun

    Steampunk Metal Oval Sunglasses with Handmade Multicolor Chain – Bold UV400 Protection and Unique Style

    STELLA Automotive AI Appoints Fred Seidelman as Chief Technology Officer – Yahoo Finance

    STELLA Automotive AI Appoints Fred Seidelman as New Chief Technology Officer

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Al Roker Gives Olivia Dean an Unexpected ‘New Job’ on the ‘Today’ Show – Yahoo

    Al Roker Shocks Olivia Dean with an Exciting New Role on the ‘Today’ Show

    Books about the arts and some haunts for a Denton October – Denton Record-Chronicle

    Uncover Artistic Treasures and Spooky Adventures to Experience in Denton This October

    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    MAC Brings iPad Technology to Football Sidelines Across All 13 Member Schools – Sports Video Group

    MAC Brings iPad Technology to Football Sidelines Across All 13 Member Schools – Sports Video Group

    Technology Is Becoming More Important Than Humans In CX – No Jitter

    Technology Is Becoming More Important Than Humans In CX – No Jitter

    A Tech Expo Shows What China Can Make, but Not Who’ll Buy It All – The New York Times

    Inside China’s Tech Expo: Cutting-Edge Innovations Face Uncertain Demand

    Steampunk Metal Oval Technology Sense Sunglasses Personality Handmade Chain Multicolor Sunglasses UV400 – The San Joaquin Valley Sun

    Steampunk Metal Oval Sunglasses with Handmade Multicolor Chain – Bold UV400 Protection and Unique Style

    STELLA Automotive AI Appoints Fred Seidelman as Chief Technology Officer – Yahoo Finance

    STELLA Automotive AI Appoints Fred Seidelman as New Chief Technology Officer

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home General

Why publishers are questioning the effectiveness of blocking AI web crawlers

October 1, 2023
in General
Why publishers are questioning the effectiveness of blocking AI web crawlers
Share on FacebookShare on Twitter

This article is part of Digiday’s coverage of its Digiday Publishing Summit. More from the series →

A number of publishers — including Bloomberg and The New York Times — were quick to block OpenAI’s web crawler from accessing their sites, to protect their content from getting scraped and used to feed the artificial intelligence tech company’s large language models (LLMs). But whether this tactic is actually effective is debatable, according to conversations with five publishing executives.

“It’s a symbolic gesture,” said a senior tech executive at a media company, who requested anonymity to speak freely.

In August, OpenAI announced that publishers can now block its GPTBot web crawler from accessing their web pages’ content. Since then, 26 of the 100 most-visited sites (and 242 of the top 1,000 sites) have done so, according to Originality.ai.

However, publishers’ content distribution models might make the protective strategy moot. One publishing exec told Digiday their company publishes on eight different syndication apps and websites. Because the content is already so discoverable, it feels like the protective measure to block OpenAI’s web crawler was a futile effort, they said.

“I think it was kind of a wasted effort on my part. It’s an inevitability that this stuff is ingested and crawled and learned from,” the exec said during a closed-door session at the Digiday Publishing Summit in Key Biscayne, Fla. last week.

Publishers have had a hard time protecting against generative AI tools like OpenAI’s chatbot ChatGPT from bypassing their paywalls and scraping their content to train their LLMs. Though publishers can now block OpenAI’s crawler, some publishing execs aren’t convinced it’s enough to protect their IP.

“It’s a long-term problem, and there isn’t a short-term solution,” said Matt Rogerson, director of public policy at Guardian Media Group. “It’s a sign that publishers are taking back a bit more control and are going to start demanding more control over other folks that are scraping for different purposes.”

Google and Microsoft are listening

OpenAI is just one of the tech companies using web crawlers to feed their LLMs for AI tools and systems. Google and Microsoft’s web crawlers are essential for publishers’ content to get indexed and surfaced in search results on Google Search and Bing — but those crawlers also scrape content to train those tech companies’ LLMs and AI chatbots. The Guardian’s Rogerson called these “bundled scrapers.”

“They treat it all as one big search product,” the first tech exec said. “They’re like, ‘No, you don’t get the granularity choice. We give you the opportunity to opt out.’ But obviously, we don’t want to opt out of all web crawling.”

Those tech companies are listening to publishers’ concerns. In July, Google announced it was exploring alternatives to its robots.txt protocol — the file that tells search engine crawlers which URLs they can access — to give publishers more control over how their IP is used in different contexts. And just Thursday, Google released a new tool called Google-Extended that gives website owners the ability to opt out of having their sites crawled for data used to train Google’s AI systems and its generative AI chatbot Bard. (The execs interviewed for this story spoke to Digiday before that announcement.)

Microsoft has chosen to go another route. Last week, the company announced that publishers can add a piece of code to their web pages to communicate that the content should not be used for LLMs (a bit like a copyright tag). Microsoft is giving website owners two options: a “NOCACHE” tag that allows only titles, snippets and URLs to appear in the Bing chatbot or to train its AI models, or a “NOARCHIVE” tag, which prevents any usage in its chatbot or AI training.

“They are signaling that they will add more granularity,” Rogerson said. “We’re examining that in detail.”

The New York Times took matters into their own hands and added language to its Terms of Service last month prohibiting the use of its content to train machine learning or AI systems, giving the Times the ability to pursue legal action against companies using their data.

A negotiation tactic

So why are publishers blocking OpenAI’s web crawler at all, if the move doesn’t ensure protection of their content?

Execs told Digiday it’s a negotiation tactic.

“Putting the blocker in place is at least one… starting point for the inevitable negotiations that we’ll have as publishers with OpenAI and other companies. We’ll be able to have that as a point of leverage and say, we’ll take it off if we can reach a deal or an agreement,” said the publishing exec at the Digiday Publishing Summit.

Publishers’ protective actions are creating a “market for licenses for data mining,” with a potential for compensation for sharing their data, Rogerson said. OpenAI struck a licensing partnership with the Associated Press in July, wherein OpenAI is paying to license part of the AP’s text archive to train its models.

But not all publishers feel like they’re powerful enough to negotiate the use of their content with these large tech companies.

“We’re not big enough to flex our muscles and block it,” said a second publishing executive who asked to remain anonymous. The exec was also unsure if blocking OpenAI’s web crawler would affect their use of GPT, the AI technology ChatGPT is built on that OpenAI has made available for outside developers to license.

“If you start blocking the crawler, do they cut you off from using the tool? Does the tool stop working as well? It’s really unclear,” the publishing exec said. “There probably is a way to eventually figure it out, but not without a ton of detective work,” they added.

https://digiday.com/?p=519789

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : DigiDay – https://digiday.com/media/a-symbolic-gesture-publishers-question-the-effectiveness-of-blocking-ai-web-crawlers/?utm_campaign=digidaydis&utm_medium=rss&utm_source=general-rss

Previous Post

Why security, scalability and a data-driven mindset are crucial for enterprise analytics

Next Post

Actor Michael Gambon, Known For ‘Harry Potter’ Dumbledore Role, Passes Away At Age 82

How the gathering of Israel is accelerating around the world – Church News

The Global Gathering of Israel: How It’s Accelerating Like Never Before

October 4, 2025
United States Economic Forecast Q3 2025 – Deloitte

United States Economic Forecast Q3 2025 – Deloitte

October 4, 2025
Al Roker Gives Olivia Dean an Unexpected ‘New Job’ on the ‘Today’ Show – Yahoo

Al Roker Shocks Olivia Dean with an Exciting New Role on the ‘Today’ Show

October 4, 2025
Holding Commercial Health Insurers Accountable – American Hospital Association

Holding Commercial Health Insurers Accountable – American Hospital Association

October 4, 2025
Inside Iowa Politics: Ian Roberts deceived Des Moines School Board, chair says – KTIV

Inside Iowa Politics: Ian Roberts deceived Des Moines School Board, chair says – KTIV

October 4, 2025
‘This is unprecedented:’ Ecology restricts surface water use in the Yakima River Basin – Yakima Herald-Republic

Unprecedented Move: Ecology Imposes New Restrictions on Yakima River Basin Surface Water Use

October 4, 2025
Roane State, TCAT Knoxville mark milestone at new health science campus – Oakridger

Roane State and TCAT Knoxville Celebrate Major Milestone at New Health Science Campus

October 4, 2025
Science Hill blanks Morristown East – Johnson City Press

Science Hill Cruises to a Commanding Shutout Over Morristown East

October 4, 2025
Favorite Halloween candies in Missouri, Oklahoma, Kansas, Arkansas – Yahoo

Discover the Most Beloved Halloween Candies in Missouri, Oklahoma, Kansas, and Arkansas!

October 4, 2025
MAC Brings iPad Technology to Football Sidelines Across All 13 Member Schools – Sports Video Group

MAC Brings iPad Technology to Football Sidelines Across All 13 Member Schools – Sports Video Group

October 4, 2025

Categories

Archives

October 2025
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Sep    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (850)
  • Economy (871)
  • Entertainment (21,745)
  • General (17,402)
  • Health (9,914)
  • Lifestyle (883)
  • News (22,149)
  • People (872)
  • Politics (882)
  • Science (16,081)
  • Sports (21,371)
  • Technology (15,853)
  • World (854)

Recent News

How the gathering of Israel is accelerating around the world – Church News

The Global Gathering of Israel: How It’s Accelerating Like Never Before

October 4, 2025
United States Economic Forecast Q3 2025 – Deloitte

United States Economic Forecast Q3 2025 – Deloitte

October 4, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version