• About
  • Advertise
  • Privacy & Policy
  • Contact
Friday, December 1, 2023
Earth-News
  • Home
  • Business
  • Entertainment
    “Godzilla Minus One” Lands Rave Reviews

    “Godzilla Minus One” Lands Rave Reviews

    TV News: River, Dying, Griselda, Criminal

    TV News: River, Dying, Griselda, Criminal

    Brooks, Hansen Join The “Minecraft” Movie

    Brooks, Hansen Join The “Minecraft” Movie

    “Doctor Who” 2023 Xmas Special Photos

    “Doctor Who” 2023 Xmas Special Photos

    Teaser Trailer: “Furiosa: A Mad Max Saga”

    Teaser Trailer: “Furiosa: A Mad Max Saga”

    Empress Of Shares New “Femenine” Video: Watch

    Empress Of Shares New “Femenine” Video: Watch

  • General
  • Health
  • News
    End of an era: Singapore’s first ever Jollibee at Lucky Plaza to move out after 10 years, Lifestyle News

    End of an era: Singapore’s first ever Jollibee at Lucky Plaza to move out after 10 years, Lifestyle News

    I’m only staying here temporarily, says man who pitched tent outside cluttered Boon Lay flat, Singapore News

    I’m only staying here temporarily, says man who pitched tent outside cluttered Boon Lay flat, Singapore News

    Fainted while frying noodles: Guan Kee Fried Kuay Teow owner reveals reason behind stall closure, Lifestyle News

    Fainted while frying noodles: Guan Kee Fried Kuay Teow owner reveals reason behind stall closure, Lifestyle News

    Spinelli Coffee chain to close remaining outlets by end of 2023: Business Times

    Spinelli Coffee chain to close remaining outlets by end of 2023: Business Times

    AHTC case: Workers’ Party leaders awarded costs for appeals, Singapore

    AHTC case: Workers’ Party leaders awarded costs for appeals, Singapore

    From Hitchcock to Loach and Lynch, directors go under the spotlight

    From Hitchcock to Loach and Lynch, directors go under the spotlight

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Tech
    The Best Notebooks for Photo and Video Editing Reviews

    The Best Notebooks for Photo and Video Editing Reviews

    Xiaomi POCO X6 Pro 5G to launch globally as re-branded Redmi K70E and not POCO F6

    Xiaomi POCO X6 Pro 5G to launch globally as re-branded Redmi K70E and not POCO F6

    Texas Instruments low-power GaN series releases to bring super compact power adapters

    Texas Instruments low-power GaN series releases to bring super compact power adapters

    Cybertruck orders open as range surpasses rumours, Tesla confirms US$79,990 AWD starting price alongside 250-mile RWD and 845 hp Cyberbeast

    Cybertruck orders open as range surpasses rumours, Tesla confirms US$79,990 AWD starting price alongside 250-mile RWD and 845 hp Cyberbeast

    New Dell UltraSharp 40 leaks as U4021QW successor

    New Dell UltraSharp 40 leaks as U4021QW successor

    €25,000 Renault 5 E-Tech Electric hatchback flaunts sharp retro looks alongside solid range and 2024 debut in latest teaser

    €25,000 Renault 5 E-Tech Electric hatchback flaunts sharp retro looks alongside solid range and 2024 debut in latest teaser

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    “Godzilla Minus One” Lands Rave Reviews

    “Godzilla Minus One” Lands Rave Reviews

    TV News: River, Dying, Griselda, Criminal

    TV News: River, Dying, Griselda, Criminal

    Brooks, Hansen Join The “Minecraft” Movie

    Brooks, Hansen Join The “Minecraft” Movie

    “Doctor Who” 2023 Xmas Special Photos

    “Doctor Who” 2023 Xmas Special Photos

    Teaser Trailer: “Furiosa: A Mad Max Saga”

    Teaser Trailer: “Furiosa: A Mad Max Saga”

    Empress Of Shares New “Femenine” Video: Watch

    Empress Of Shares New “Femenine” Video: Watch

  • General
  • Health
  • News
    End of an era: Singapore’s first ever Jollibee at Lucky Plaza to move out after 10 years, Lifestyle News

    End of an era: Singapore’s first ever Jollibee at Lucky Plaza to move out after 10 years, Lifestyle News

    I’m only staying here temporarily, says man who pitched tent outside cluttered Boon Lay flat, Singapore News

    I’m only staying here temporarily, says man who pitched tent outside cluttered Boon Lay flat, Singapore News

    Fainted while frying noodles: Guan Kee Fried Kuay Teow owner reveals reason behind stall closure, Lifestyle News

    Fainted while frying noodles: Guan Kee Fried Kuay Teow owner reveals reason behind stall closure, Lifestyle News

    Spinelli Coffee chain to close remaining outlets by end of 2023: Business Times

    Spinelli Coffee chain to close remaining outlets by end of 2023: Business Times

    AHTC case: Workers’ Party leaders awarded costs for appeals, Singapore

    AHTC case: Workers’ Party leaders awarded costs for appeals, Singapore

    From Hitchcock to Loach and Lynch, directors go under the spotlight

    From Hitchcock to Loach and Lynch, directors go under the spotlight

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Tech
    The Best Notebooks for Photo and Video Editing Reviews

    The Best Notebooks for Photo and Video Editing Reviews

    Xiaomi POCO X6 Pro 5G to launch globally as re-branded Redmi K70E and not POCO F6

    Xiaomi POCO X6 Pro 5G to launch globally as re-branded Redmi K70E and not POCO F6

    Texas Instruments low-power GaN series releases to bring super compact power adapters

    Texas Instruments low-power GaN series releases to bring super compact power adapters

    Cybertruck orders open as range surpasses rumours, Tesla confirms US$79,990 AWD starting price alongside 250-mile RWD and 845 hp Cyberbeast

    Cybertruck orders open as range surpasses rumours, Tesla confirms US$79,990 AWD starting price alongside 250-mile RWD and 845 hp Cyberbeast

    New Dell UltraSharp 40 leaks as U4021QW successor

    New Dell UltraSharp 40 leaks as U4021QW successor

    €25,000 Renault 5 E-Tech Electric hatchback flaunts sharp retro looks alongside solid range and 2024 debut in latest teaser

    €25,000 Renault 5 E-Tech Electric hatchback flaunts sharp retro looks alongside solid range and 2024 debut in latest teaser

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Tech

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information

August 10, 2023
in Tech
OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information
Share on FacebookShare on Twitter

OpenAI Launches a Crawler Bot to Scrape Websites for Information

OpenAI, the company behind ChatGPT, recently launched its own web crawler GPTBot, to scrape websites for information. However, the company also released the crawler’s specifications so that website owners and publishers can block the bot from scraping their content.

In a technical document released by OpenAI, the company has described how to identify the crawler using its user agent token and string. The document also explains how to block the crawler by adding an entry to the server’s robots.txt file.

Table of Contents

  • What Does GPTBot Do and How to Block It?
  • Growing Concerns Over AI Companies Scraping Information From the Internet

What Does GPTBot Do and How to Block It?

Just like any other web crawler, GPTBot crawls through websites, scanning the web pages and scraping information. However, it’s the purpose of the scraped information that sets GPTBot apart from search engine indexing crawlers – the gathered data would be used to train the company’s AI models. This is a part of OpenAI’s effort to develop the next generation of AI models, which reportedly include GPT-5.

Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.OpenAI

It adds that the web pages crawled using the bot may be filtered to remove sources. These include sources that contain text which violates OpenAI’s policies, collects personally identifiable information, or requires paywall access.

Of course, most website owners and publishers wouldn’t want to let the machine learning giant scrape their content and use them for its AI models. The document published by OpenAI details how to block GPTBot, and the process is rather simple.

To disallow the web crawler from accessing a website entirely, all you have to do is add its token to the site’s robots.txt file and use the “Disallow: /” command.

The bot can also be blocked from accessing certain pages on a website but allowed access to the rest. For this, site owners would have to use the “Allow: /directory-1/” and “Disallow: /directory-2/” commands and then customize as necessary.

Growing Concerns Over AI Companies Scraping Information From the Internet

The web crawler happens to be OpenAI’s latest acknowledgment that it trains its AI models based on public information from the internet. This coincides with the growing efforts by different organizations to restrict automated access to information via the web.

Companies like OpenAI make millions of dollars in revenue by training their models on all sorts of information gathered from the internet. Frustrated at not getting a share of the profits earned by AI companies using their content, business owners are taking a stand by closing off access.

Recently, four unidentified entities were sued by Twitter in order to prevent data on the website from being scraped and used to train AI models.

Reddit made changes to its API terms too, enabling the company to effectively monetize the content that’s created by its users for free of charge.

Not too long ago, OpenAI was also sued by award Sarah Silverman for training ChatGPT on their copyrighted works without their consent. Other companies such as Microsoft, Google, and its AI research arm Deepmind have faced similar lawsuits.

According to Israel Krush, CEO and co-founder of Hyro, the fact that publishers have to manually opt out of having their sites scraped by GPTBot raises a big concern. Hyro is the company behind an AI assistant used in the healthcare industry.

He went on to add that while his own firm scrapes data from the internet, it does so only with explicit permission and ensures the appropriate handling of personal information.

Companies like Adobe have also suggested marking information as “not for AI training” through legal means. It remains to be seen if any legal discourse would be taken to prevent GPTBot from scraping websites by default.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : TechReport – https://techreport.com/news/openai-launches-a-crawler-bot-named-gptbot-to-scrape-websites-for-information/

Tags: launchesOpenAItechnology
Previous Post

Zoom’s Data Usage Sparks EU Legal Questions over AI Training

Next Post

Shiba Inu Trillionaires Accumulate 11 Trillion SHIB in 2 Days As Shibarium Nears; New Alt Crypto Surges in Presale, Are Whales Set To Buy This Coin Next?

Live coverage: SpaceX to launch ride share mission with South Korean spy satellite, first Irish satellite

Live coverage: SpaceX to launch ride share mission with South Korean spy satellite, first Irish satellite

December 1, 2023
Russia launches space station cargo ship

Russia launches space station cargo ship

December 1, 2023
ESA Director General: Ariane 6 aiming for summer 2024 debut

ESA Director General: Ariane 6 aiming for summer 2024 debut

December 1, 2023
GAO report warns Artemis 3 landing may be delayed to 2027

GAO report warns Artemis 3 landing may be delayed to 2027

December 1, 2023
NASA researchers get permission to apply for China’s moon samples

NASA researchers get permission to apply for China’s moon samples

December 1, 2023
Australian startup develops dual-headed space camera

Australian startup develops dual-headed space camera

December 1, 2023
Intelsat to bring multi-orbit Wi-Fi to regional American Airlines jets

Intelsat to bring multi-orbit Wi-Fi to regional American Airlines jets

December 1, 2023
Ursa Major closes $138 million Series D and D-1 financings

Ursa Major closes $138 million Series D and D-1 financings

December 1, 2023
Personal Transformation and Holistic Naturopathic Healing

Personal Transformation and Holistic Naturopathic Healing

December 1, 2023
Elevating Skincare: The Certified Organic Difference at Sanctum Australia

Elevating Skincare: The Certified Organic Difference at Sanctum Australia

December 1, 2023

Categories

Archives

December 2023
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031
« Nov    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (6,709)
  • Entertainment (7,081)
  • General (5,196)
  • Health (3,177)
  • News (7,733)
  • Science (5,242)
  • Sports (7,068)
  • Tech (5,264)

Recent News

Live coverage: SpaceX to launch ride share mission with South Korean spy satellite, first Irish satellite

Live coverage: SpaceX to launch ride share mission with South Korean spy satellite, first Irish satellite

December 1, 2023
Russia launches space station cargo ship

Russia launches space station cargo ship

December 1, 2023
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information- OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information * OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information | | OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information |