* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Monday, September 1, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Country music star apologizes after drunken show ends with cops taking him down: ‘I’m not OK’ – PennLive.com

    Country Music Star Apologizes After Drunken Show Ends in Police Intervention: ‘I’m Not OK

    Comanche Nation Entertainment closes casino near Devol – KSWO 7News

    Comanche Nation Entertainment Closes Casino Near Devol in Surprising Move

    Erykah Badu Announces ‘Abi & Alan’ Album With The Alchemist Will Be Delayed – yahoo.com

    Erykah Badu Opens Up About the Delay of Her Highly Anticipated ‘Abi & Alan’ Album with The Alchemist

    If You Needed Some BTS Pics Of The Rock’s Tree Trunk Legs To Spice Up Your Day, We Got ‘Em – yahoo.com

    Need a Boost? Check Out These Epic BTS Shots of The Rock’s Tree Trunk Legs!

    Sabrina Carpenter’s pearl-clutching magic? Humor – CNN

    Sabrina Carpenter’s pearl-clutching magic? Humor – CNN

    South Park Is In Trouble And Needs To Change Course, Fast – yahoo.com

    South Park Faces Major Challenges and Must Adapt Fast to Survive

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    This Secret Technology Will Make The IPhone 17 Super Thin Air – VOI.ID

    How This Breakthrough Technology Will Make the iPhone 17 Incredibly Thin and Lightweight

    Gator football kicks off with excitement and new technology – WCJB | TV20

    Gator Football Kicks Off with Thrilling Action and Innovative Technology

    Marvell Technology (MRVL) Loses 18.6% as Growth Outlook Fails to Impress – Yahoo Finance

    Marvell Technology (MRVL) Loses 18.6% as Growth Outlook Fails to Impress – Yahoo Finance

    Propelling solar technology into a perovskite future – CORDIS

    Propelling solar technology into a perovskite future – CORDIS

    13 Top Technology Trends (2025) – Exploding Topics

    13 Game-Changing Technology Trends to Watch in 2025

    Japan’s legacy LCD and chip technology find new home in India – Nikkei Asia

    How Japan’s Breakthrough LCD and Chip Technologies Are Driving Innovation in India

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Country music star apologizes after drunken show ends with cops taking him down: ‘I’m not OK’ – PennLive.com

    Country Music Star Apologizes After Drunken Show Ends in Police Intervention: ‘I’m Not OK

    Comanche Nation Entertainment closes casino near Devol – KSWO 7News

    Comanche Nation Entertainment Closes Casino Near Devol in Surprising Move

    Erykah Badu Announces ‘Abi & Alan’ Album With The Alchemist Will Be Delayed – yahoo.com

    Erykah Badu Opens Up About the Delay of Her Highly Anticipated ‘Abi & Alan’ Album with The Alchemist

    If You Needed Some BTS Pics Of The Rock’s Tree Trunk Legs To Spice Up Your Day, We Got ‘Em – yahoo.com

    Need a Boost? Check Out These Epic BTS Shots of The Rock’s Tree Trunk Legs!

    Sabrina Carpenter’s pearl-clutching magic? Humor – CNN

    Sabrina Carpenter’s pearl-clutching magic? Humor – CNN

    South Park Is In Trouble And Needs To Change Course, Fast – yahoo.com

    South Park Faces Major Challenges and Must Adapt Fast to Survive

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    This Secret Technology Will Make The IPhone 17 Super Thin Air – VOI.ID

    How This Breakthrough Technology Will Make the iPhone 17 Incredibly Thin and Lightweight

    Gator football kicks off with excitement and new technology – WCJB | TV20

    Gator Football Kicks Off with Thrilling Action and Innovative Technology

    Marvell Technology (MRVL) Loses 18.6% as Growth Outlook Fails to Impress – Yahoo Finance

    Marvell Technology (MRVL) Loses 18.6% as Growth Outlook Fails to Impress – Yahoo Finance

    Propelling solar technology into a perovskite future – CORDIS

    Propelling solar technology into a perovskite future – CORDIS

    13 Top Technology Trends (2025) – Exploding Topics

    13 Game-Changing Technology Trends to Watch in 2025

    Japan’s legacy LCD and chip technology find new home in India – Nikkei Asia

    How Japan’s Breakthrough LCD and Chip Technologies Are Driving Innovation in India

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information

August 10, 2023
in Technology
OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information
Share on FacebookShare on Twitter

OpenAI Launches a Crawler Bot to Scrape Websites for Information

OpenAI, the company behind ChatGPT, recently launched its own web crawler GPTBot, to scrape websites for information. However, the company also released the crawler’s specifications so that website owners and publishers can block the bot from scraping their content.

In a technical document released by OpenAI, the company has described how to identify the crawler using its user agent token and string. The document also explains how to block the crawler by adding an entry to the server’s robots.txt file.

What Does GPTBot Do and How to Block It?

Just like any other web crawler, GPTBot crawls through websites, scanning the web pages and scraping information. However, it’s the purpose of the scraped information that sets GPTBot apart from search engine indexing crawlers – the gathered data would be used to train the company’s AI models. This is a part of OpenAI’s effort to develop the next generation of AI models, which reportedly include GPT-5.

Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.OpenAI

It adds that the web pages crawled using the bot may be filtered to remove sources. These include sources that contain text which violates OpenAI’s policies, collects personally identifiable information, or requires paywall access.

Of course, most website owners and publishers wouldn’t want to let the machine learning giant scrape their content and use them for its AI models. The document published by OpenAI details how to block GPTBot, and the process is rather simple.

To disallow the web crawler from accessing a website entirely, all you have to do is add its token to the site’s robots.txt file and use the “Disallow: /” command.

The bot can also be blocked from accessing certain pages on a website but allowed access to the rest. For this, site owners would have to use the “Allow: /directory-1/” and “Disallow: /directory-2/” commands and then customize as necessary.

Growing Concerns Over AI Companies Scraping Information From the Internet

The web crawler happens to be OpenAI’s latest acknowledgment that it trains its AI models based on public information from the internet. This coincides with the growing efforts by different organizations to restrict automated access to information via the web.

Companies like OpenAI make millions of dollars in revenue by training their models on all sorts of information gathered from the internet. Frustrated at not getting a share of the profits earned by AI companies using their content, business owners are taking a stand by closing off access.

Recently, four unidentified entities were sued by Twitter in order to prevent data on the website from being scraped and used to train AI models.

Reddit made changes to its API terms too, enabling the company to effectively monetize the content that’s created by its users for free of charge.

Not too long ago, OpenAI was also sued by award Sarah Silverman for training ChatGPT on their copyrighted works without their consent. Other companies such as Microsoft, Google, and its AI research arm Deepmind have faced similar lawsuits.

According to Israel Krush, CEO and co-founder of Hyro, the fact that publishers have to manually opt out of having their sites scraped by GPTBot raises a big concern. Hyro is the company behind an AI assistant used in the healthcare industry.

He went on to add that while his own firm scrapes data from the internet, it does so only with explicit permission and ensures the appropriate handling of personal information.

Companies like Adobe have also suggested marking information as “not for AI training” through legal means. It remains to be seen if any legal discourse would be taken to prevent GPTBot from scraping websites by default.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : TechReport – https://techreport.com/news/openai-launches-a-crawler-bot-named-gptbot-to-scrape-websites-for-information/

Tags: launchesOpenAItechnology
Previous Post

Zoom’s Data Usage Sparks EU Legal Questions over AI Training

Next Post

Shiba Inu Trillionaires Accumulate 11 Trillion SHIB in 2 Days As Shibarium Nears; New Alt Crypto Surges in Presale, Are Whales Set To Buy This Coin Next?

ECB’s Lagarde warns of risk to world economy via Trump interference on Fed – Reuters

ECB’s Lagarde warns of risk to world economy via Trump interference on Fed – Reuters

September 1, 2025
The Resilient Engine: Investing in Poland’s Outperforming Economy Amid Global Slowdown – AInvest

The Resilient Engine: Why Investing in Poland’s Booming Economy Is a Smart Move Amid Global Slowdown

September 1, 2025
Country music star apologizes after drunken show ends with cops taking him down: ‘I’m not OK’ – PennLive.com

Country Music Star Apologizes After Drunken Show Ends in Police Intervention: ‘I’m Not OK

September 1, 2025
Arts to promote mental health goal of new University of Michigan clinical program – MLive.com

Innovative University of Michigan Program Harnesses the Arts to Enhance Mental Health

September 1, 2025
Meet the man defending Trump’s political adversaries – ABC News

Meet the man defending Trump’s political adversaries – ABC News

September 1, 2025
Sky High Farm’s Biennial Blends Art, Agriculture and Ecological Urgency – observer.com

Sky High Farm’s Biennial Blends Art, Agriculture and Ecological Urgency – observer.com

September 1, 2025

After 150 Job Applications and Rescinded Offers, Computer Science Grads Face an Unforgiving Job Market

September 1, 2025
Proportion of US students taking chemistry fails to keep pace with other science disciplines – Chemistry World

Why Fewer US Students Are Choosing Chemistry Compared to Other Sciences

September 1, 2025
Going zero-waste for a month opened my eyes — here are 7 lessons I wish I knew sooner – VegOut

7 Eye-Opening Lessons I Learned from Living Zero-Waste for a Month

September 1, 2025
This Secret Technology Will Make The IPhone 17 Super Thin Air – VOI.ID

How This Breakthrough Technology Will Make the iPhone 17 Incredibly Thin and Lightweight

September 1, 2025

Categories

Archives

September 2025
MTWTFSS
1234567
891011121314
15161718192021
22232425262728
2930 
« Aug    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (801)
  • Economy (820)
  • Entertainment (21,700)
  • General (16,798)
  • Health (9,861)
  • Lifestyle (834)
  • News (22,149)
  • People (821)
  • Politics (828)
  • Science (16,030)
  • Sports (21,318)
  • Technology (15,800)
  • World (801)

Recent News

ECB’s Lagarde warns of risk to world economy via Trump interference on Fed – Reuters

ECB’s Lagarde warns of risk to world economy via Trump interference on Fed – Reuters

September 1, 2025
The Resilient Engine: Investing in Poland’s Outperforming Economy Amid Global Slowdown – AInvest

The Resilient Engine: Why Investing in Poland’s Booming Economy Is a Smart Move Amid Global Slowdown

September 1, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version