* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Friday, October 3, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

    The Police Made Chart History With This 1979 Hit Nearly 50 Years Ago – Yahoo

    How The Police Changed Music Forever with Their Iconic 1979 Hit Nearly 50 Years Ago

    Good Deed Entertainment Acquires Worldwide Rights To Liza Mandelup’s Documentary ‘Caterpillar’ – Deadline

    Good Deed Entertainment Lands Global Rights to Liza Mandelup’s Captivating Documentary ‘Caterpillar

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Technology Is Becoming More Important Than Humans In CX – No Jitter

    Technology Is Becoming More Important Than Humans In CX – No Jitter

    A Tech Expo Shows What China Can Make, but Not Who’ll Buy It All – The New York Times

    Inside China’s Tech Expo: Cutting-Edge Innovations Face Uncertain Demand

    Steampunk Metal Oval Technology Sense Sunglasses Personality Handmade Chain Multicolor Sunglasses UV400 – The San Joaquin Valley Sun

    Steampunk Metal Oval Sunglasses with Handmade Multicolor Chain – Bold UV400 Protection and Unique Style

    STELLA Automotive AI Appoints Fred Seidelman as Chief Technology Officer – Yahoo Finance

    STELLA Automotive AI Appoints Fred Seidelman as New Chief Technology Officer

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Four Strategic Signals Technology Leaders Are Tuning In To – SPONSOR CONTENT FROM ARM – Harvard Business Review

    Four Essential Strategic Signals Every Technology Leader Should Watch

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Toni Braxton Is Turning Her Biggest Hits Into Lifetime Movies – Yahoo

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Major airline to offer new in-flight entertainment options for passengers – PennLive.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

    Penn State-Themed Restaurant and Entertainment Spot Happy Valley Live Set to Open in State College – StateCollege.com

    The Police Made Chart History With This 1979 Hit Nearly 50 Years Ago – Yahoo

    How The Police Changed Music Forever with Their Iconic 1979 Hit Nearly 50 Years Ago

    Good Deed Entertainment Acquires Worldwide Rights To Liza Mandelup’s Documentary ‘Caterpillar’ – Deadline

    Good Deed Entertainment Lands Global Rights to Liza Mandelup’s Captivating Documentary ‘Caterpillar

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Technology Is Becoming More Important Than Humans In CX – No Jitter

    Technology Is Becoming More Important Than Humans In CX – No Jitter

    A Tech Expo Shows What China Can Make, but Not Who’ll Buy It All – The New York Times

    Inside China’s Tech Expo: Cutting-Edge Innovations Face Uncertain Demand

    Steampunk Metal Oval Technology Sense Sunglasses Personality Handmade Chain Multicolor Sunglasses UV400 – The San Joaquin Valley Sun

    Steampunk Metal Oval Sunglasses with Handmade Multicolor Chain – Bold UV400 Protection and Unique Style

    STELLA Automotive AI Appoints Fred Seidelman as Chief Technology Officer – Yahoo Finance

    STELLA Automotive AI Appoints Fred Seidelman as New Chief Technology Officer

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Saving Energy and Money with Smart Technology – Terms of Service with Clare Duffy – Podcast on CNN Podcasts – CNN

    Four Strategic Signals Technology Leaders Are Tuning In To – SPONSOR CONTENT FROM ARM – Harvard Business Review

    Four Essential Strategic Signals Every Technology Leader Should Watch

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

Search engines that don’t pay up can’t index Reddit content

July 24, 2024
in Technology
Search engines that don’t pay up can’t index Reddit content
Share on FacebookShare on Twitter

Will Shanklin

When Reddit said last month that it would block unauthorized data scraping from its site, everyone’s (rightful) first reaction was “AI, AI, AI.” However, now that the change has taken effect, chatbot makers aren’t the only ones being locked out. The widely used forum also appears to be blocking all search engines other than Google, which reportedly inked a deal earlier this year with Reddit worth $60 million annually.

404 Media reported on Wednesday (and Engadget confirmed in our queries) that searching for Reddit results from the past week on rival engine Bing (using “site:reddit.com”) returns empty results. The publication reported that DuckDuckGo produced seven links without any descriptions, only providing the note, “We would like to show you a description here but the site won’t allow us.” The engine now appears to have removed even those, as our test only produced an empty page, reading, “no results found.”

When Reddit said last month that it would update its Robots Exclusion Protocol (robots.txt) to block automated data scraping, it’s now apparent that it wasn’t only meant to thwart AI companies like Perplexity and its controversial “answer engine.” Currently, Google appears to be the only search engine allowed to crawl Reddit and produce results from “the front page of the internet.”

Ironically, part of the forum website’s robots.txt file reads, “Reddit believes in an open internet, but not the misuse of public content.” The file for Reddit now essentially says, “Do not scrape.” Apparently, it now considers search engines that don’t buy into exclusive deals to be misusing its content.

The ubiquitous robots.txt is the web standard that communicates which parts of a site can be crawled. Although many crawlers are known to ignore its instructions, Google’s standard procedure is to respect it. So, on the technical side, the companies in cahoots on the lucrative deal appear to have deployed some manual override.

Of course, the saga is a trickle-down effect of AI chatbots scraping the live web for results. With courts slow to determine how much of the open web is fair use to train chatbots on, companies like Reddit, whose bottom lines now depend on safeguarding their data from those who don’t pay, are building walls at the expense of the open web. (Although, given the integral role Microsoft has played in this AI era, cozying up with OpenAI early on, it seems ironic that Bing finds itself on the losing end of at least one aspect of the fallout.)

Colin Hayhurst, CEO of lesser-known “no-tracking” search engine Mojeek, told 404 Media that Reddit is “killing everything for search but Google.” In addition, the executive said his attempts to contact Reddit were ignored. “It’s never happened to us before,” he said. “Because this happens to us, we get blocked, usually because of ignorance or stupidity or whatever, and when we contact the site you certainly can get that resolved, but we’ve never had no reply from anybody before.”

Engadget asked Google and Reddit for comment and confirmation, but we hadn’t heard back by publication. 404 Media reported running into a similar wall of silence from the companies.

Reddit has made no secret of its desire to block AI companies from scraping its treasure trove of data in this burgeoning age of AI. Last year, CEO Steve Huffman risked alienating large portions of its user base by blocking third-party API requests, leading to the demise of beloved apps like Christian Selig’s Apollo. Despite widespread protests among moderators and forum-goers, the company only temporarily lost negligible numbers of users.

The gamble appeared to pay off, and Reddit recovered. It went public in March.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Engadget – https://www.engadget.com/search-engines-that-dont-pay-up-cant-index-reddit-content-172949170.html?src=rss

Tags: EnginesSearchtechnology
Previous Post

CrowdStrike offers a $10 Uber Eats card to say sorry before pulling the offer

Next Post

Meta takes down 63,000 Instagram accounts linked to extortion scams

Top-12 NFL running games: Which teams are helping our RBs score fantasy football points this season? – Yahoo Sports

Top-12 NFL running games: Which teams are helping our RBs score fantasy football points this season? – Yahoo Sports

October 3, 2025
FIRST LOOK: Official match ball of FIFA World Cup ‘26 – Yahoo Sports

Get an Exclusive First Look at the Official Match Ball of FIFA World Cup 2026!

October 3, 2025
Arts economy grows, but funding lags on Long Island – Long Island Business News

Long Island’s Arts Economy Thrives Despite Ongoing Funding Challenges

October 3, 2025
Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

Taylor Swift Releases New Album The Life of a Showgirl : Listen and Read the Full Credits – Yahoo

October 3, 2025
HIV pioneer Dr. William Valenti retires, leaving lasting impact on health community – 13wham.com

HIV Pioneer Dr. William Valenti Retires, Leaving an Enduring Impact on the Health Community

October 3, 2025
As the shutdown drags on, the threat of permanent cuts is mired in politics – NPR

As the Shutdown Continues, the Risk of Lasting Cuts Grows Amid Political Stalemate

October 3, 2025
West Seattle Ecology Fair 2025, report #2: Heat waves aren’t just a source of discomfort – West Seattle Blog…

West Seattle Ecology Fair 2025: Uncovering the Hidden Impacts of Heat Waves Beyond Our Comfort

October 3, 2025
Roam The Huntington after dark at an atmospheric ‘Strange Science’ night – NBC Los Angeles

Uncover the Secrets of The Huntington After Dark at the Enchanting ‘Strange Science’ Night

October 3, 2025
Viral apple cider vinegar weight loss study retracted for flawed science – ScienceDaily

Viral apple cider vinegar weight loss study retracted for flawed science – ScienceDaily

October 3, 2025
A love letter to the garden – The Inquirer and Mirror

A Heartfelt Ode to the Garden: Celebrating Nature’s Timeless Beauty

October 3, 2025

Categories

Archives

October 2025
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Sep    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (848)
  • Economy (869)
  • Entertainment (21,743)
  • General (17,381)
  • Health (9,912)
  • Lifestyle (881)
  • News (22,149)
  • People (870)
  • Politics (880)
  • Science (16,079)
  • Sports (21,370)
  • Technology (15,852)
  • World (852)

Recent News

Top-12 NFL running games: Which teams are helping our RBs score fantasy football points this season? – Yahoo Sports

Top-12 NFL running games: Which teams are helping our RBs score fantasy football points this season? – Yahoo Sports

October 3, 2025
FIRST LOOK: Official match ball of FIFA World Cup ‘26 – Yahoo Sports

Get an Exclusive First Look at the Official Match Ball of FIFA World Cup 2026!

October 3, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version