* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Tuesday, July 29, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, Boyd – CDC Gaming

    Top Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, and Boyd Take Center Stage

    Micro wrestling coming to NE Ohio – Cleveland.com

    Get Ready, NE Ohio: Micro Wrestling Is Making Its Exciting Debut!

    League City seeking proposals for 53-acre entertainment district on sportsplex land – galvnews.com

    League City Invites Proposals to Transform 53-Acre Sportsplex into Vibrant Entertainment District

    Top 5 entertainment news: Sandeep Reddy Vanga regrets trimming Animal’s runtime by 7-8 minutes, Akshay Ku – Times of India

    Top 5 Entertainment Highlights: Sandeep Reddy Vanga Reveals Why He Trimmed Animal’s Runtime by 7-8 Minutes, Plus Akshay Ku Updates

    Cote de Pablo reveals how Michael Weatherly used his soap opera roots to put her at ease in “NCIS” love scene – yahoo.com

    Cote de Pablo Reveals How Michael Weatherly’s Soap Opera Background Made Their “NCIS” Love Scene Easier

    City of Pelham announces entertainment district plans for former Oak Mountain Amphitheatre site – WVTM

    Pelham Unveils Exciting New Entertainment District Plans for Former Oak Mountain Amphitheatre Site

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    AI’s race in the dark with China – Axios

    The High-Stakes AI Race: Innovation and Competition in the Shadows

    Eagle Unveils Revolutionary X-Ray Technology at Pack Expo

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, Boyd – CDC Gaming

    Top Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, and Boyd Take Center Stage

    Micro wrestling coming to NE Ohio – Cleveland.com

    Get Ready, NE Ohio: Micro Wrestling Is Making Its Exciting Debut!

    League City seeking proposals for 53-acre entertainment district on sportsplex land – galvnews.com

    League City Invites Proposals to Transform 53-Acre Sportsplex into Vibrant Entertainment District

    Top 5 entertainment news: Sandeep Reddy Vanga regrets trimming Animal’s runtime by 7-8 minutes, Akshay Ku – Times of India

    Top 5 Entertainment Highlights: Sandeep Reddy Vanga Reveals Why He Trimmed Animal’s Runtime by 7-8 Minutes, Plus Akshay Ku Updates

    Cote de Pablo reveals how Michael Weatherly used his soap opera roots to put her at ease in “NCIS” love scene – yahoo.com

    Cote de Pablo Reveals How Michael Weatherly’s Soap Opera Background Made Their “NCIS” Love Scene Easier

    City of Pelham announces entertainment district plans for former Oak Mountain Amphitheatre site – WVTM

    Pelham Unveils Exciting New Entertainment District Plans for Former Oak Mountain Amphitheatre Site

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    AI’s race in the dark with China – Axios

    The High-Stakes AI Race: Innovation and Competition in the Shadows

    Eagle Unveils Revolutionary X-Ray Technology at Pack Expo

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information

August 10, 2023
in Technology
OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information
Share on FacebookShare on Twitter

OpenAI Launches a Crawler Bot to Scrape Websites for Information

OpenAI, the company behind ChatGPT, recently launched its own web crawler GPTBot, to scrape websites for information. However, the company also released the crawler’s specifications so that website owners and publishers can block the bot from scraping their content.

In a technical document released by OpenAI, the company has described how to identify the crawler using its user agent token and string. The document also explains how to block the crawler by adding an entry to the server’s robots.txt file.

What Does GPTBot Do and How to Block It?

Just like any other web crawler, GPTBot crawls through websites, scanning the web pages and scraping information. However, it’s the purpose of the scraped information that sets GPTBot apart from search engine indexing crawlers – the gathered data would be used to train the company’s AI models. This is a part of OpenAI’s effort to develop the next generation of AI models, which reportedly include GPT-5.

Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.OpenAI

It adds that the web pages crawled using the bot may be filtered to remove sources. These include sources that contain text which violates OpenAI’s policies, collects personally identifiable information, or requires paywall access.

Of course, most website owners and publishers wouldn’t want to let the machine learning giant scrape their content and use them for its AI models. The document published by OpenAI details how to block GPTBot, and the process is rather simple.

To disallow the web crawler from accessing a website entirely, all you have to do is add its token to the site’s robots.txt file and use the “Disallow: /” command.

The bot can also be blocked from accessing certain pages on a website but allowed access to the rest. For this, site owners would have to use the “Allow: /directory-1/” and “Disallow: /directory-2/” commands and then customize as necessary.

Growing Concerns Over AI Companies Scraping Information From the Internet

The web crawler happens to be OpenAI’s latest acknowledgment that it trains its AI models based on public information from the internet. This coincides with the growing efforts by different organizations to restrict automated access to information via the web.

Companies like OpenAI make millions of dollars in revenue by training their models on all sorts of information gathered from the internet. Frustrated at not getting a share of the profits earned by AI companies using their content, business owners are taking a stand by closing off access.

Recently, four unidentified entities were sued by Twitter in order to prevent data on the website from being scraped and used to train AI models.

Reddit made changes to its API terms too, enabling the company to effectively monetize the content that’s created by its users for free of charge.

Not too long ago, OpenAI was also sued by award Sarah Silverman for training ChatGPT on their copyrighted works without their consent. Other companies such as Microsoft, Google, and its AI research arm Deepmind have faced similar lawsuits.

According to Israel Krush, CEO and co-founder of Hyro, the fact that publishers have to manually opt out of having their sites scraped by GPTBot raises a big concern. Hyro is the company behind an AI assistant used in the healthcare industry.

He went on to add that while his own firm scrapes data from the internet, it does so only with explicit permission and ensures the appropriate handling of personal information.

Companies like Adobe have also suggested marking information as “not for AI training” through legal means. It remains to be seen if any legal discourse would be taken to prevent GPTBot from scraping websites by default.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : TechReport – https://techreport.com/news/openai-launches-a-crawler-bot-named-gptbot-to-scrape-websites-for-information/

Tags: launchesOpenAItechnology
Previous Post

Zoom’s Data Usage Sparks EU Legal Questions over AI Training

Next Post

Shiba Inu Trillionaires Accumulate 11 Trillion SHIB in 2 Days As Shibarium Nears; New Alt Crypto Surges in Presale, Are Whales Set To Buy This Coin Next?

Time to stop Dhaka from rolling to ecological disaster – New Age BD

Urgent Action Needed to Prevent Dhaka’s Ecological Disaster

July 29, 2025
Guest column | I’m a cardiologist. Here are 10 science-based ways to prevent heart disease. – The Washington Post

I’m a Cardiologist: 10 Science-Backed Tips to Prevent Heart Disease

July 29, 2025
Author talk with science, environmental journalist, ‘Strata: Stories from Deep Time’ – PenBay Pilot

Author Talk with Science and Environmental Journalist Behind ‘Strata: Stories from Deep Time

July 29, 2025
Alzheimer’s progression could be slowed by these changes to lifestyle – The Independent

Simple Lifestyle Changes That Could Slow Alzheimer’s Progression

July 29, 2025
Divya Deshmukh becomes third Women’s World Cup Champion, defeats Humpy Koneru in tiebreak – FIDE

Divya Deshmukh becomes third Women’s World Cup Champion, defeats Humpy Koneru in tiebreak – FIDE

July 29, 2025
It’s a gig economy for CEOs these days, too – Axios

It’s a gig economy for CEOs these days, too – Axios

July 29, 2025
Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, Boyd – CDC Gaming

Top Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, and Boyd Take Center Stage

July 29, 2025
US judge blocks Trump’s effort to defund reproductive health organisation – Al Jazeera

US judge blocks Trump’s effort to defund reproductive health organisation – Al Jazeera

July 29, 2025
New documents show how passport and Social Security rules would change to enforce Trump’s birthright citizenship order – CNN

New documents show how passport and Social Security rules would change to enforce Trump’s birthright citizenship order – CNN

July 29, 2025
Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

July 28, 2025

Categories

Archives

July 2025
MTWTFSS
 123456
78910111213
14151617181920
21222324252627
28293031 
« Jun    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (744)
  • Economy (768)
  • Entertainment (21,648)
  • General (16,166)
  • Health (9,805)
  • Lifestyle (776)
  • News (22,149)
  • People (770)
  • Politics (777)
  • Science (15,981)
  • Sports (21,265)
  • Technology (15,748)
  • World (751)

Recent News

Time to stop Dhaka from rolling to ecological disaster – New Age BD

Urgent Action Needed to Prevent Dhaka’s Ecological Disaster

July 29, 2025
Guest column | I’m a cardiologist. Here are 10 science-based ways to prevent heart disease. – The Washington Post

I’m a Cardiologist: 10 Science-Backed Tips to Prevent Heart Disease

July 29, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version