* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, October 18, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    CT Culture Corner: Robert Redford films to watch – CT Insider

    CT Culture Corner: Robert Redford films to watch – CT Insider

    Elmira’s New Entertainment Venue ‘Centertown Social’ Adding its Final Touches – WENY News

    Elmira’s New Entertainment Venue ‘Centertown Social’ Adding its Final Touches – WENY News

    Bella Thorne Shows Off Her Toned Abs in Crop Top – Yahoo

    Bella Thorne Shows Off Her Toned Abs in a Chic Crop Top

    Bricktown continues to thrive as OKC’s entertainment district – OKC VeloCity

    Bricktown Shines as Oklahoma City’s Premier Entertainment Hotspot

    This Haunting ‘60s Country Song Is Even More Tragic Than You Remember – Yahoo

    This Haunting ’60s Country Song Will Break Your Heart All Over Again

    Arts/Entertainment: ‘Wait Until Dark’ brings spooky season center stage – Times Herald Online

    Wait Until Dark’ Delivers Chilling Thrills Perfect for the Spooky Season

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Italian Technology in Manufacturing: Supporting North American Industries and Keeping Production Local – Thomasnet

    How Italian Technology is Revolutionizing North American Manufacturing and Boosting Local Production

    Guide to Proteomics Project Planning: Sample Preparation Strategies – Technology Networks

    Guide to Proteomics Project Planning: Sample Preparation Strategies – Technology Networks

    DLG−Agrifuture Concept Winners 2025 announced – Visionary Ideas in agricultural technology recognized – TheNewsMarket

    Meet the Visionary Innovators Revolutionizing Agricultural Technology in 2025

    Spanning the Gulf: Global Defence Technology 155 – Army Technology

    Spanning the Gulf: Exploring Cutting-Edge Global Defence Technology

    A Nobel Prize for explaining when technology leads to growth – NPR

    Nobel Prize Awarded for Unraveling How Technology Drives Economic Growth

    Erie Canal’s 200th anniversary: How a technological marvel for trade changed the environment forever – The Conversation

    Erie Canal’s 200th Anniversary: How a Trade Marvel Transformed the Environment Forever

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    CT Culture Corner: Robert Redford films to watch – CT Insider

    CT Culture Corner: Robert Redford films to watch – CT Insider

    Elmira’s New Entertainment Venue ‘Centertown Social’ Adding its Final Touches – WENY News

    Elmira’s New Entertainment Venue ‘Centertown Social’ Adding its Final Touches – WENY News

    Bella Thorne Shows Off Her Toned Abs in Crop Top – Yahoo

    Bella Thorne Shows Off Her Toned Abs in a Chic Crop Top

    Bricktown continues to thrive as OKC’s entertainment district – OKC VeloCity

    Bricktown Shines as Oklahoma City’s Premier Entertainment Hotspot

    This Haunting ‘60s Country Song Is Even More Tragic Than You Remember – Yahoo

    This Haunting ’60s Country Song Will Break Your Heart All Over Again

    Arts/Entertainment: ‘Wait Until Dark’ brings spooky season center stage – Times Herald Online

    Wait Until Dark’ Delivers Chilling Thrills Perfect for the Spooky Season

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Italian Technology in Manufacturing: Supporting North American Industries and Keeping Production Local – Thomasnet

    How Italian Technology is Revolutionizing North American Manufacturing and Boosting Local Production

    Guide to Proteomics Project Planning: Sample Preparation Strategies – Technology Networks

    Guide to Proteomics Project Planning: Sample Preparation Strategies – Technology Networks

    DLG−Agrifuture Concept Winners 2025 announced – Visionary Ideas in agricultural technology recognized – TheNewsMarket

    Meet the Visionary Innovators Revolutionizing Agricultural Technology in 2025

    Spanning the Gulf: Global Defence Technology 155 – Army Technology

    Spanning the Gulf: Exploring Cutting-Edge Global Defence Technology

    A Nobel Prize for explaining when technology leads to growth – NPR

    Nobel Prize Awarded for Unraveling How Technology Drives Economic Growth

    Erie Canal’s 200th anniversary: How a technological marvel for trade changed the environment forever – The Conversation

    Erie Canal’s 200th Anniversary: How a Trade Marvel Transformed the Environment Forever

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information

August 10, 2023
in Technology
OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information
Share on FacebookShare on Twitter

OpenAI Launches a Crawler Bot to Scrape Websites for Information

OpenAI, the company behind ChatGPT, recently launched its own web crawler GPTBot, to scrape websites for information. However, the company also released the crawler’s specifications so that website owners and publishers can block the bot from scraping their content.

In a technical document released by OpenAI, the company has described how to identify the crawler using its user agent token and string. The document also explains how to block the crawler by adding an entry to the server’s robots.txt file.

What Does GPTBot Do and How to Block It?

Just like any other web crawler, GPTBot crawls through websites, scanning the web pages and scraping information. However, it’s the purpose of the scraped information that sets GPTBot apart from search engine indexing crawlers – the gathered data would be used to train the company’s AI models. This is a part of OpenAI’s effort to develop the next generation of AI models, which reportedly include GPT-5.

Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.OpenAI

It adds that the web pages crawled using the bot may be filtered to remove sources. These include sources that contain text which violates OpenAI’s policies, collects personally identifiable information, or requires paywall access.

Of course, most website owners and publishers wouldn’t want to let the machine learning giant scrape their content and use them for its AI models. The document published by OpenAI details how to block GPTBot, and the process is rather simple.

To disallow the web crawler from accessing a website entirely, all you have to do is add its token to the site’s robots.txt file and use the “Disallow: /” command.

The bot can also be blocked from accessing certain pages on a website but allowed access to the rest. For this, site owners would have to use the “Allow: /directory-1/” and “Disallow: /directory-2/” commands and then customize as necessary.

Growing Concerns Over AI Companies Scraping Information From the Internet

The web crawler happens to be OpenAI’s latest acknowledgment that it trains its AI models based on public information from the internet. This coincides with the growing efforts by different organizations to restrict automated access to information via the web.

Companies like OpenAI make millions of dollars in revenue by training their models on all sorts of information gathered from the internet. Frustrated at not getting a share of the profits earned by AI companies using their content, business owners are taking a stand by closing off access.

Recently, four unidentified entities were sued by Twitter in order to prevent data on the website from being scraped and used to train AI models.

Reddit made changes to its API terms too, enabling the company to effectively monetize the content that’s created by its users for free of charge.

Not too long ago, OpenAI was also sued by award Sarah Silverman for training ChatGPT on their copyrighted works without their consent. Other companies such as Microsoft, Google, and its AI research arm Deepmind have faced similar lawsuits.

According to Israel Krush, CEO and co-founder of Hyro, the fact that publishers have to manually opt out of having their sites scraped by GPTBot raises a big concern. Hyro is the company behind an AI assistant used in the healthcare industry.

He went on to add that while his own firm scrapes data from the internet, it does so only with explicit permission and ensures the appropriate handling of personal information.

Companies like Adobe have also suggested marking information as “not for AI training” through legal means. It remains to be seen if any legal discourse would be taken to prevent GPTBot from scraping websites by default.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : TechReport – https://techreport.com/news/openai-launches-a-crawler-bot-named-gptbot-to-scrape-websites-for-information/

Tags: launchesOpenAItechnology
Previous Post

Zoom’s Data Usage Sparks EU Legal Questions over AI Training

Next Post

Shiba Inu Trillionaires Accumulate 11 Trillion SHIB in 2 Days As Shibarium Nears; New Alt Crypto Surges in Presale, Are Whales Set To Buy This Coin Next?

Navigating Autonomy, Energy and Ecology in Ladakh, India – resilience.org

Navigating Autonomy, Energy and Ecology in Ladakh, India – resilience.org

October 18, 2025
Can We Save the AI Economy? – Towards Data Science

Can We Save the AI Economy? – Towards Data Science

October 18, 2025
Scientists discover surprising health impact of simple walk in park: ‘More efficient and stronger’ – Yahoo

Discover How a Simple Walk in the Park Can Transform Your Health and Strength

October 18, 2025
These Are Officially the 3 Best Tequilas of 2025, According to Concours Mondial de Bruxelles – Yahoo

These Are Officially the 3 Best Tequilas of 2025, According to Concours Mondial de Bruxelles – Yahoo

October 18, 2025
Italian Technology in Manufacturing: Supporting North American Industries and Keeping Production Local – Thomasnet

How Italian Technology is Revolutionizing North American Manufacturing and Boosting Local Production

October 18, 2025
This week in HS Sports: Separation is looming but common sense could still prevail – AL.com

High School Sports This Week: Tensions Flare, but Common Sense Could Prevail

October 18, 2025
Ball State Alumnus Alex Call on Dodgers Team Heading to World Series – Ball State University Athletics

Ball State Alumnus Alex Call Celebrates Dodgers’ Thrilling Journey to the World Series

October 18, 2025
Javier Milei’s economic policy in Argentina – GIS Reports

Javier Milei’s Bold Plan to Transform Argentina’s Economy

October 18, 2025
CT Culture Corner: Robert Redford films to watch – CT Insider

CT Culture Corner: Robert Redford films to watch – CT Insider

October 18, 2025
A Look Behind Routine Checkups with Our Animal Health Team – New England Aquarium

Discover What Happens During Routine Checkups with Our Animal Health Team

October 18, 2025

Categories

Archives

October 2025
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Sep    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (874)
  • Economy (895)
  • Entertainment (21,767)
  • General (17,669)
  • Health (9,936)
  • Lifestyle (908)
  • News (22,149)
  • People (896)
  • Politics (905)
  • Science (16,106)
  • Sports (21,395)
  • Technology (15,875)
  • World (878)

Recent News

Navigating Autonomy, Energy and Ecology in Ladakh, India – resilience.org

Navigating Autonomy, Energy and Ecology in Ladakh, India – resilience.org

October 18, 2025
Can We Save the AI Economy? – Towards Data Science

Can We Save the AI Economy? – Towards Data Science

October 18, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version