* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Wednesday, October 22, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

    Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

    AMC brings first new Dolby Experience to Gwinnett since 2017 – Wyoming News Now

    AMC Launches First New Dolby Experience in Gwinnett Since 2017

    Hetzel Design: blending architecture and entertainment – Blooloop

    Hetzel Design: Where Architecture and Entertainment Unite in Perfect Harmony

    Country music legend rushed to hospital year after heart surgery. Here’s what we know – PennLive.com

    Country Music Legend Rushed to Hospital One Year After Heart Surgery – What’s Happening Now?

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Placer County town of Loomis considers entertainment zone for downtown – CBS News

    Loomis Unveils Thrilling New Entertainment Zone to Revitalize Downtown

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 Technology Stocks to Buy Now – Yahoo Finance

    3 Must-Buy Tech Stocks You Can’t Afford to Miss Right Now

    ‘New frontier’: Austin leaders start discussions on air taxi technology – KXAN Austin

    Austin Leaders Ignite Exciting Conversations on the Future of Air Taxi Technology

    How a Gemma model helped discover a new potential cancer therapy pathway – blog.google

    How a Gemma Model Revealed a Breakthrough Pathway for Cancer Treatment

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

    Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

    AMC brings first new Dolby Experience to Gwinnett since 2017 – Wyoming News Now

    AMC Launches First New Dolby Experience in Gwinnett Since 2017

    Hetzel Design: blending architecture and entertainment – Blooloop

    Hetzel Design: Where Architecture and Entertainment Unite in Perfect Harmony

    Country music legend rushed to hospital year after heart surgery. Here’s what we know – PennLive.com

    Country Music Legend Rushed to Hospital One Year After Heart Surgery – What’s Happening Now?

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Placer County town of Loomis considers entertainment zone for downtown – CBS News

    Loomis Unveils Thrilling New Entertainment Zone to Revitalize Downtown

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 Technology Stocks to Buy Now – Yahoo Finance

    3 Must-Buy Tech Stocks You Can’t Afford to Miss Right Now

    ‘New frontier’: Austin leaders start discussions on air taxi technology – KXAN Austin

    Austin Leaders Ignite Exciting Conversations on the Future of Air Taxi Technology

    How a Gemma model helped discover a new potential cancer therapy pathway – blog.google

    How a Gemma Model Revealed a Breakthrough Pathway for Cancer Treatment

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home General

Why publishers are questioning the effectiveness of blocking AI web crawlers

October 1, 2023
in General
Why publishers are questioning the effectiveness of blocking AI web crawlers
Share on FacebookShare on Twitter

This article is part of Digiday’s coverage of its Digiday Publishing Summit. More from the series →

A number of publishers — including Bloomberg and The New York Times — were quick to block OpenAI’s web crawler from accessing their sites, to protect their content from getting scraped and used to feed the artificial intelligence tech company’s large language models (LLMs). But whether this tactic is actually effective is debatable, according to conversations with five publishing executives.

“It’s a symbolic gesture,” said a senior tech executive at a media company, who requested anonymity to speak freely.

In August, OpenAI announced that publishers can now block its GPTBot web crawler from accessing their web pages’ content. Since then, 26 of the 100 most-visited sites (and 242 of the top 1,000 sites) have done so, according to Originality.ai.

However, publishers’ content distribution models might make the protective strategy moot. One publishing exec told Digiday their company publishes on eight different syndication apps and websites. Because the content is already so discoverable, it feels like the protective measure to block OpenAI’s web crawler was a futile effort, they said.

“I think it was kind of a wasted effort on my part. It’s an inevitability that this stuff is ingested and crawled and learned from,” the exec said during a closed-door session at the Digiday Publishing Summit in Key Biscayne, Fla. last week.

Publishers have had a hard time protecting against generative AI tools like OpenAI’s chatbot ChatGPT from bypassing their paywalls and scraping their content to train their LLMs. Though publishers can now block OpenAI’s crawler, some publishing execs aren’t convinced it’s enough to protect their IP.

“It’s a long-term problem, and there isn’t a short-term solution,” said Matt Rogerson, director of public policy at Guardian Media Group. “It’s a sign that publishers are taking back a bit more control and are going to start demanding more control over other folks that are scraping for different purposes.”

Google and Microsoft are listening

OpenAI is just one of the tech companies using web crawlers to feed their LLMs for AI tools and systems. Google and Microsoft’s web crawlers are essential for publishers’ content to get indexed and surfaced in search results on Google Search and Bing — but those crawlers also scrape content to train those tech companies’ LLMs and AI chatbots. The Guardian’s Rogerson called these “bundled scrapers.”

“They treat it all as one big search product,” the first tech exec said. “They’re like, ‘No, you don’t get the granularity choice. We give you the opportunity to opt out.’ But obviously, we don’t want to opt out of all web crawling.”

Those tech companies are listening to publishers’ concerns. In July, Google announced it was exploring alternatives to its robots.txt protocol — the file that tells search engine crawlers which URLs they can access — to give publishers more control over how their IP is used in different contexts. And just Thursday, Google released a new tool called Google-Extended that gives website owners the ability to opt out of having their sites crawled for data used to train Google’s AI systems and its generative AI chatbot Bard. (The execs interviewed for this story spoke to Digiday before that announcement.)

Microsoft has chosen to go another route. Last week, the company announced that publishers can add a piece of code to their web pages to communicate that the content should not be used for LLMs (a bit like a copyright tag). Microsoft is giving website owners two options: a “NOCACHE” tag that allows only titles, snippets and URLs to appear in the Bing chatbot or to train its AI models, or a “NOARCHIVE” tag, which prevents any usage in its chatbot or AI training.

“They are signaling that they will add more granularity,” Rogerson said. “We’re examining that in detail.”

The New York Times took matters into their own hands and added language to its Terms of Service last month prohibiting the use of its content to train machine learning or AI systems, giving the Times the ability to pursue legal action against companies using their data.

A negotiation tactic

So why are publishers blocking OpenAI’s web crawler at all, if the move doesn’t ensure protection of their content?

Execs told Digiday it’s a negotiation tactic.

“Putting the blocker in place is at least one… starting point for the inevitable negotiations that we’ll have as publishers with OpenAI and other companies. We’ll be able to have that as a point of leverage and say, we’ll take it off if we can reach a deal or an agreement,” said the publishing exec at the Digiday Publishing Summit.

Publishers’ protective actions are creating a “market for licenses for data mining,” with a potential for compensation for sharing their data, Rogerson said. OpenAI struck a licensing partnership with the Associated Press in July, wherein OpenAI is paying to license part of the AP’s text archive to train its models.

But not all publishers feel like they’re powerful enough to negotiate the use of their content with these large tech companies.

“We’re not big enough to flex our muscles and block it,” said a second publishing executive who asked to remain anonymous. The exec was also unsure if blocking OpenAI’s web crawler would affect their use of GPT, the AI technology ChatGPT is built on that OpenAI has made available for outside developers to license.

“If you start blocking the crawler, do they cut you off from using the tool? Does the tool stop working as well? It’s really unclear,” the publishing exec said. “There probably is a way to eventually figure it out, but not without a ton of detective work,” they added.

https://digiday.com/?p=519789

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : DigiDay – https://digiday.com/media/a-symbolic-gesture-publishers-question-the-effectiveness-of-blocking-ai-web-crawlers/?utm_campaign=digidaydis&utm_medium=rss&utm_source=general-rss

Previous Post

Why security, scalability and a data-driven mindset are crucial for enterprise analytics

Next Post

Actor Michael Gambon, Known For ‘Harry Potter’ Dumbledore Role, Passes Away At Age 82

Grandmaster Daniel Naroditsky remembered for his impact on the chess world – NPR

Grandmaster Daniel Naroditsky: Honoring a Timeless Chess Legacy

October 22, 2025
Seattle’s Creative Economy: Building Talent, Culture, and Community – GeekWire

Seattle’s Creative Economy: How Talent, Culture, and Community Are Shaping the Future

October 22, 2025
Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

October 22, 2025
Self-efficacy mediates the effect of hope on health promotion intention in Chinese stroke patients – Nature

How Hope Boosts Health Motivation in Chinese Stroke Patients Through Self-Efficacy

October 22, 2025
Arizona AG sues to force House Speaker Johnson to seat Democrat Adelita Grijalva – NBC News

Arizona AG sues to force House Speaker Johnson to seat Democrat Adelita Grijalva – NBC News

October 22, 2025
Historic achievement: Governor Ferguson, Ecology celebrate nuclear waste officially being turned into glass at Hanford Site | Governor Bob Ferguson – Governor Ferguson (.gov)

Historic Breakthrough: Governor Ferguson and Ecology Celebrate Nuclear Waste Transformed into Glass at Hanford Site

October 22, 2025
Transcript: Mission to Mars — Bad science fiction – Financial Times

Mission to Mars: When Science Fiction Takes a Dark Turn

October 22, 2025
Science Hill, South Greene volleyball win TSSAA state tournament openers – WCYB

Science Hill and South Greene Dominate Opening Matches in TSSAA State Volleyball Tournament

October 22, 2025
GCC Influencer Market Jumps 75% As Lifestyle Creators Lead Growth – Net Influencer

GCC Influencer Market Soars 75% Fueled by Lifestyle Creators Leading the Boom

October 22, 2025
A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

October 22, 2025

Categories

Archives

October 2025
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Sep    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (880)
  • Economy (902)
  • Entertainment (21,773)
  • General (17,740)
  • Health (9,943)
  • Lifestyle (914)
  • News (22,149)
  • People (902)
  • Politics (912)
  • Science (16,112)
  • Sports (21,401)
  • Technology (15,881)
  • World (885)

Recent News

Grandmaster Daniel Naroditsky remembered for his impact on the chess world – NPR

Grandmaster Daniel Naroditsky: Honoring a Timeless Chess Legacy

October 22, 2025
Seattle’s Creative Economy: Building Talent, Culture, and Community – GeekWire

Seattle’s Creative Economy: How Talent, Culture, and Community Are Shaping the Future

October 22, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version