* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, August 23, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    How to watch ‘F1: The Movie’ on Prime Video – About Amazon

    Experience the Thrill: How to Stream ‘F1: The Movie’ on Prime Video

    FOX One is now available on Prime Video: Here’s everything to know – About Amazon

    FOX One is now available on Prime Video: Here’s everything to know – About Amazon

    What Are Our Predictions for Taylor Swift’s ‘Life of a Showgirl’ Based on What She’s Told Us So Far? – yahoo.com

    Uncover the Hidden Surprises in Taylor Swift’s ‘Life of a Showgirl’ – Can You Decode Her Clues?

    Bama Dining Serving Up New Food, Entertainment – UA News Center

    Bama Dining Launches Thrilling New Food and Entertainment Experiences

    The latest on Williams-Brice Stadium renovations, entertainment district – On3

    Discover the Thrilling New Upgrades Coming to Williams-Brice Stadium and Its Vibrant Entertainment District!

    1000+ Entertainment Leaders Release Open Letter Criticizing TIFF’s Handling of Film – Creative Community for Peace

    1000+ Entertainment Leaders Release Open Letter Criticizing TIFF’s Handling of Film – Creative Community for Peace

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Alcorn State awarded grant to boost STEM with VR technology – WJTV

    Alcorn State Secures Grant to Transform STEM Education Through Cutting-Edge VR Technology

    Hyundai: The Only Way To Beat China Is To Embrace Technology – InsideEVs

    Hyundai’s Bold Strategy to Outpace China with Cutting-Edge Technology

    Teaching older adults how to use technology – WWNY

    Empowering Older Adults to Master Technology with Confidence

    Denver man receives pacemaker using new technology – CBS News

    Denver Man’s Life Transformed by Revolutionary Pacemaker Technology

    Morgan County Rescue Squad is using new technology to recover people missing in the water faster – WAFF

    Morgan County Rescue Squad is using new technology to recover people missing in the water faster – WAFF

    YSU to celebrate launch of new radiology technology program – WKBN.com

    YSU Launches Exciting New Radiology Technology Program Celebration

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    How to watch ‘F1: The Movie’ on Prime Video – About Amazon

    Experience the Thrill: How to Stream ‘F1: The Movie’ on Prime Video

    FOX One is now available on Prime Video: Here’s everything to know – About Amazon

    FOX One is now available on Prime Video: Here’s everything to know – About Amazon

    What Are Our Predictions for Taylor Swift’s ‘Life of a Showgirl’ Based on What She’s Told Us So Far? – yahoo.com

    Uncover the Hidden Surprises in Taylor Swift’s ‘Life of a Showgirl’ – Can You Decode Her Clues?

    Bama Dining Serving Up New Food, Entertainment – UA News Center

    Bama Dining Launches Thrilling New Food and Entertainment Experiences

    The latest on Williams-Brice Stadium renovations, entertainment district – On3

    Discover the Thrilling New Upgrades Coming to Williams-Brice Stadium and Its Vibrant Entertainment District!

    1000+ Entertainment Leaders Release Open Letter Criticizing TIFF’s Handling of Film – Creative Community for Peace

    1000+ Entertainment Leaders Release Open Letter Criticizing TIFF’s Handling of Film – Creative Community for Peace

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Alcorn State awarded grant to boost STEM with VR technology – WJTV

    Alcorn State Secures Grant to Transform STEM Education Through Cutting-Edge VR Technology

    Hyundai: The Only Way To Beat China Is To Embrace Technology – InsideEVs

    Hyundai’s Bold Strategy to Outpace China with Cutting-Edge Technology

    Teaching older adults how to use technology – WWNY

    Empowering Older Adults to Master Technology with Confidence

    Denver man receives pacemaker using new technology – CBS News

    Denver Man’s Life Transformed by Revolutionary Pacemaker Technology

    Morgan County Rescue Squad is using new technology to recover people missing in the water faster – WAFF

    Morgan County Rescue Squad is using new technology to recover people missing in the water faster – WAFF

    YSU to celebrate launch of new radiology technology program – WKBN.com

    YSU Launches Exciting New Radiology Technology Program Celebration

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

Search engines that don’t pay up can’t index Reddit content

July 24, 2024
in Technology
Search engines that don’t pay up can’t index Reddit content
Share on FacebookShare on Twitter

Will Shanklin

When Reddit said last month that it would block unauthorized data scraping from its site, everyone’s (rightful) first reaction was “AI, AI, AI.” However, now that the change has taken effect, chatbot makers aren’t the only ones being locked out. The widely used forum also appears to be blocking all search engines other than Google, which reportedly inked a deal earlier this year with Reddit worth $60 million annually.

404 Media reported on Wednesday (and Engadget confirmed in our queries) that searching for Reddit results from the past week on rival engine Bing (using “site:reddit.com”) returns empty results. The publication reported that DuckDuckGo produced seven links without any descriptions, only providing the note, “We would like to show you a description here but the site won’t allow us.” The engine now appears to have removed even those, as our test only produced an empty page, reading, “no results found.”

When Reddit said last month that it would update its Robots Exclusion Protocol (robots.txt) to block automated data scraping, it’s now apparent that it wasn’t only meant to thwart AI companies like Perplexity and its controversial “answer engine.” Currently, Google appears to be the only search engine allowed to crawl Reddit and produce results from “the front page of the internet.”

Ironically, part of the forum website’s robots.txt file reads, “Reddit believes in an open internet, but not the misuse of public content.” The file for Reddit now essentially says, “Do not scrape.” Apparently, it now considers search engines that don’t buy into exclusive deals to be misusing its content.

The ubiquitous robots.txt is the web standard that communicates which parts of a site can be crawled. Although many crawlers are known to ignore its instructions, Google’s standard procedure is to respect it. So, on the technical side, the companies in cahoots on the lucrative deal appear to have deployed some manual override.

Of course, the saga is a trickle-down effect of AI chatbots scraping the live web for results. With courts slow to determine how much of the open web is fair use to train chatbots on, companies like Reddit, whose bottom lines now depend on safeguarding their data from those who don’t pay, are building walls at the expense of the open web. (Although, given the integral role Microsoft has played in this AI era, cozying up with OpenAI early on, it seems ironic that Bing finds itself on the losing end of at least one aspect of the fallout.)

Colin Hayhurst, CEO of lesser-known “no-tracking” search engine Mojeek, told 404 Media that Reddit is “killing everything for search but Google.” In addition, the executive said his attempts to contact Reddit were ignored. “It’s never happened to us before,” he said. “Because this happens to us, we get blocked, usually because of ignorance or stupidity or whatever, and when we contact the site you certainly can get that resolved, but we’ve never had no reply from anybody before.”

Engadget asked Google and Reddit for comment and confirmation, but we hadn’t heard back by publication. 404 Media reported running into a similar wall of silence from the companies.

Reddit has made no secret of its desire to block AI companies from scraping its treasure trove of data in this burgeoning age of AI. Last year, CEO Steve Huffman risked alienating large portions of its user base by blocking third-party API requests, leading to the demise of beloved apps like Christian Selig’s Apollo. Despite widespread protests among moderators and forum-goers, the company only temporarily lost negligible numbers of users.

The gamble appeared to pay off, and Reddit recovered. It went public in March.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Engadget – https://www.engadget.com/search-engines-that-dont-pay-up-cant-index-reddit-content-172949170.html?src=rss

Tags: EnginesSearchtechnology
Previous Post

CrowdStrike offers a $10 Uber Eats card to say sorry before pulling the offer

Next Post

Meta takes down 63,000 Instagram accounts linked to extortion scams

Crossings: How Road Ecology is Shaping the Future of Our Planet – Living on Earth

Crossings: How Road Ecology is Shaping the Future of Our Planet

August 23, 2025
Toyota Friday Night Rivals kicks off season with Elizabethton at Science Hill – WCYB

Toyota Friday Night Rivals Ignite the Season with Elizabethton vs. Science Hill Showdown

August 23, 2025
Star Trek as Refuge for the Science-Minded – The Provincetown Independent

How Star Trek Ignites Passion and Empowers the Science-Minded

August 23, 2025
Refresh your wardrobe at these 11 QTS award-winning fashion and lifestyle stores – Time Out Worldwide

Transform Your Look with These 11 Must-Visit Fashion and Lifestyle Stores

August 23, 2025
Alcorn State awarded grant to boost STEM with VR technology – WJTV

Alcorn State Secures Grant to Transform STEM Education Through Cutting-Edge VR Technology

August 23, 2025
Is Mastantuono in line for a start this Sunday? – Yahoo Sports

Is Mastantuono Set to Make His Starting Debut This Sunday?

August 23, 2025
‘Urgent action’ needed to protect workers from heat stress in warming world, UN says – Reuters

Urgent Action Needed to Shield Workers from Rising Heat Risks in a Warming World

August 22, 2025
The United States’ Biggest Vulnerability in the Trade War: Its Knowledge Economy – World Politics Review

The United States’ Greatest Weakness in the Trade War: Its Knowledge Economy

August 22, 2025
How to watch ‘F1: The Movie’ on Prime Video – About Amazon

Experience the Thrill: How to Stream ‘F1: The Movie’ on Prime Video

August 22, 2025
Willis Knighton Health in Louisiana joins Mayo Clinic Care Network – Mayo Clinic News Network

Willis Knighton Health Teams Up with Mayo Clinic Care Network to Elevate Patient Care in Louisiana

August 22, 2025

Categories

Archives

August 2025
MTWTFSS
 123
45678910
11121314151617
18192021222324
25262728293031
« Jul    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (785)
  • Economy (805)
  • Entertainment (21,684)
  • General (16,619)
  • Health (9,845)
  • Lifestyle (818)
  • News (22,149)
  • People (807)
  • Politics (814)
  • Science (16,018)
  • Sports (21,304)
  • Technology (15,786)
  • World (786)

Recent News

Crossings: How Road Ecology is Shaping the Future of Our Planet – Living on Earth

Crossings: How Road Ecology is Shaping the Future of Our Planet

August 23, 2025
Toyota Friday Night Rivals kicks off season with Elizabethton at Science Hill – WCYB

Toyota Friday Night Rivals Ignite the Season with Elizabethton vs. Science Hill Showdown

August 23, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version