• About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, September 23, 2023
Earth-News
  • Home
  • Business
  • Entertainment
    Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

    Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

    Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

    Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

    Governor Lawal Bans Illegal Mining In Zamfara

    Governor Lawal Bans Illegal Mining In Zamfara

    Popular Nigerian filmmaker Charles Grandsville dies

    Popular Nigerian filmmaker Charles Grandsville dies

    I’m Willing To Become Unpopular To Reform Nigeria – President Tinubu

    I’m Willing To Become Unpopular To Reform Nigeria – President Tinubu

    Squier Paranormal Jazzmaster XII Review

    Squier Paranormal Jazzmaster XII Review

  • General
  • Health
  • News
    September Market Trends Reports drills down into oil and gas sales, plus more

    September Market Trends Reports drills down into oil and gas sales, plus more

    National Youth Games: Delta athletes urged to win 7th edition as gift to Gov. Oborevwori

    National Youth Games: Delta athletes urged to win 7th edition as gift to Gov. Oborevwori

    Benue govt. distributes relief materials to IDPs

    Benue govt. distributes relief materials to IDPs

    Police begin search for missing Ekiti varsity female undergraduate

    Police begin search for missing Ekiti varsity female undergraduate

    Army commander charges soldiers to remain disciplined, professional

    Army commander charges soldiers to remain disciplined, professional

    My goal now is to help Tinubu to succeed: Yahaya Bello

    My goal now is to help Tinubu to succeed: Yahaya Bello

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Tech
    Access Bank fails to double profits in H1 2023, despite 52% growth

    Access Bank fails to double profits in H1 2023, despite 52% growth

    A Somali-based bootcamp is teaching code to young people in native languages

    A Somali-based bootcamp is teaching code to young people in native languages

    New SASSA SRD contact channels for quick help

    New SASSA SRD contact channels for quick help

    Even the most popular femtech apps are still leaving African women behind

    Even the most popular femtech apps are still leaving African women behind

    Latest on using the KPLC self service 2023

    Latest on using the KPLC self service 2023

    Meituan Delivery Takes Four Months to Expand From Kowloon to Hong Kong Island

    Meituan Delivery Takes Four Months to Expand From Kowloon to Hong Kong Island

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

    Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

    Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

    Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

    Governor Lawal Bans Illegal Mining In Zamfara

    Governor Lawal Bans Illegal Mining In Zamfara

    Popular Nigerian filmmaker Charles Grandsville dies

    Popular Nigerian filmmaker Charles Grandsville dies

    I’m Willing To Become Unpopular To Reform Nigeria – President Tinubu

    I’m Willing To Become Unpopular To Reform Nigeria – President Tinubu

    Squier Paranormal Jazzmaster XII Review

    Squier Paranormal Jazzmaster XII Review

  • General
  • Health
  • News
    September Market Trends Reports drills down into oil and gas sales, plus more

    September Market Trends Reports drills down into oil and gas sales, plus more

    National Youth Games: Delta athletes urged to win 7th edition as gift to Gov. Oborevwori

    National Youth Games: Delta athletes urged to win 7th edition as gift to Gov. Oborevwori

    Benue govt. distributes relief materials to IDPs

    Benue govt. distributes relief materials to IDPs

    Police begin search for missing Ekiti varsity female undergraduate

    Police begin search for missing Ekiti varsity female undergraduate

    Army commander charges soldiers to remain disciplined, professional

    Army commander charges soldiers to remain disciplined, professional

    My goal now is to help Tinubu to succeed: Yahaya Bello

    My goal now is to help Tinubu to succeed: Yahaya Bello

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Tech
    Access Bank fails to double profits in H1 2023, despite 52% growth

    Access Bank fails to double profits in H1 2023, despite 52% growth

    A Somali-based bootcamp is teaching code to young people in native languages

    A Somali-based bootcamp is teaching code to young people in native languages

    New SASSA SRD contact channels for quick help

    New SASSA SRD contact channels for quick help

    Even the most popular femtech apps are still leaving African women behind

    Even the most popular femtech apps are still leaving African women behind

    Latest on using the KPLC self service 2023

    Latest on using the KPLC self service 2023

    Meituan Delivery Takes Four Months to Expand From Kowloon to Hong Kong Island

    Meituan Delivery Takes Four Months to Expand From Kowloon to Hong Kong Island

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Tech

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information

August 10, 2023
in Tech
OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information
Share on FacebookShare on Twitter

OpenAI Launches a Crawler Bot to Scrape Websites for Information

OpenAI, the company behind ChatGPT, recently launched its own web crawler GPTBot, to scrape websites for information. However, the company also released the crawler’s specifications so that website owners and publishers can block the bot from scraping their content.

In a technical document released by OpenAI, the company has described how to identify the crawler using its user agent token and string. The document also explains how to block the crawler by adding an entry to the server’s robots.txt file.

Table of Contents

  • What Does GPTBot Do and How to Block It?
  • Growing Concerns Over AI Companies Scraping Information From the Internet

What Does GPTBot Do and How to Block It?

Just like any other web crawler, GPTBot crawls through websites, scanning the web pages and scraping information. However, it’s the purpose of the scraped information that sets GPTBot apart from search engine indexing crawlers – the gathered data would be used to train the company’s AI models. This is a part of OpenAI’s effort to develop the next generation of AI models, which reportedly include GPT-5.

Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.OpenAI

It adds that the web pages crawled using the bot may be filtered to remove sources. These include sources that contain text which violates OpenAI’s policies, collects personally identifiable information, or requires paywall access.

Of course, most website owners and publishers wouldn’t want to let the machine learning giant scrape their content and use them for its AI models. The document published by OpenAI details how to block GPTBot, and the process is rather simple.

To disallow the web crawler from accessing a website entirely, all you have to do is add its token to the site’s robots.txt file and use the “Disallow: /” command.

The bot can also be blocked from accessing certain pages on a website but allowed access to the rest. For this, site owners would have to use the “Allow: /directory-1/” and “Disallow: /directory-2/” commands and then customize as necessary.

Growing Concerns Over AI Companies Scraping Information From the Internet

The web crawler happens to be OpenAI’s latest acknowledgment that it trains its AI models based on public information from the internet. This coincides with the growing efforts by different organizations to restrict automated access to information via the web.

Companies like OpenAI make millions of dollars in revenue by training their models on all sorts of information gathered from the internet. Frustrated at not getting a share of the profits earned by AI companies using their content, business owners are taking a stand by closing off access.

Recently, four unidentified entities were sued by Twitter in order to prevent data on the website from being scraped and used to train AI models.

Reddit made changes to its API terms too, enabling the company to effectively monetize the content that’s created by its users for free of charge.

Not too long ago, OpenAI was also sued by award Sarah Silverman for training ChatGPT on their copyrighted works without their consent. Other companies such as Microsoft, Google, and its AI research arm Deepmind have faced similar lawsuits.

According to Israel Krush, CEO and co-founder of Hyro, the fact that publishers have to manually opt out of having their sites scraped by GPTBot raises a big concern. Hyro is the company behind an AI assistant used in the healthcare industry.

He went on to add that while his own firm scrapes data from the internet, it does so only with explicit permission and ensures the appropriate handling of personal information.

Companies like Adobe have also suggested marking information as “not for AI training” through legal means. It remains to be seen if any legal discourse would be taken to prevent GPTBot from scraping websites by default.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : TechReport – https://techreport.com/news/openai-launches-a-crawler-bot-named-gptbot-to-scrape-websites-for-information/

Tags: launchesOpenAItechnology
Previous Post

Zoom’s Data Usage Sparks EU Legal Questions over AI Training

Next Post

Shiba Inu Trillionaires Accumulate 11 Trillion SHIB in 2 Days As Shibarium Nears; New Alt Crypto Surges in Presale, Are Whales Set To Buy This Coin Next?

Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

September 23, 2023
Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

September 23, 2023
Governor Lawal Bans Illegal Mining In Zamfara

Governor Lawal Bans Illegal Mining In Zamfara

September 23, 2023
Popular Nigerian filmmaker Charles Grandsville dies

Popular Nigerian filmmaker Charles Grandsville dies

September 23, 2023
I’m Willing To Become Unpopular To Reform Nigeria – President Tinubu

I’m Willing To Become Unpopular To Reform Nigeria – President Tinubu

September 23, 2023
Squier Paranormal Jazzmaster XII Review

Squier Paranormal Jazzmaster XII Review

September 23, 2023
All-Pedal Unveils the Microdose Phaser

All-Pedal Unveils the Microdose Phaser

September 23, 2023
George Pajon—From the Black Eyed Peas to Cairo Knife Fight

George Pajon—From the Black Eyed Peas to Cairo Knife Fight

September 23, 2023
Save $45 on these open-ear conduction stereo wireless headphones

Save $45 on these open-ear conduction stereo wireless headphones

September 23, 2023
In photos: Airbus Helicopters’ fancy flying machine, the H160

In photos: Airbus Helicopters’ fancy flying machine, the H160

September 23, 2023
Partenaires
afric.info
europ.info
usa-news
lesmeilleureschaussures.fr
goodsforyoutoday.com
looktopbeauty.com
keto-diet.biz
soccernews.info
nba-news.net
sportsprblog.com compar.biz
the-best.top
bestchoices.biz
expert-plus.fr
travel-mag.info
pksportsnews.com
news-sports.org

Categories

Archives

September 2023
M T W T F S S
 123
45678910
11121314151617
18192021222324
252627282930  
« Aug    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (3,369)
  • Entertainment (3,425)
  • General (2,455)
  • Health (1,637)
  • News (3,866)
  • Science (2,657)
  • Sports (3,575)
  • Tech (2,617)

Recent News

Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

Cross pays touching tribute to Mohbad (Update) – Lifestyle Nigeria

September 23, 2023
Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

Imperial Governors On The Rampage — The PUNCH Editorial – Lifestyle Nigeria

September 23, 2023
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version

OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information- https://news7.asia/   https://www.animaleries.info   https://www.comptoirdubagage.biz/   https://www.wa-news.com/   https://www.magasinjouet.net/   https://teenoi.net   https://paschers.info   https://www.1nfo.net   -/- J’ai encore menti !   Bensimon femme Sport Bag Sac bandouliere Orange (Coquelicot)   Asus ExpertCenter PB63: 1.35 liter mini PC debuts with support for 65 W Intel processors   Wimbledon 2023: British number one Cameron Norrie beats Tomas Machac to reach round two   Douyin Boosts Self-Operated E-Commerce with Launch of ‘Imported Supermarket’   -*-$$Antoine Dupont blessé : le capitaine du XV de France a été opéré et va pouvoir rejoindre les Bleus   Lampe de Scène, Lumière Fête 9W 9 couleurs Éclairage Soirée Boule Disco Avec Bluetooth Président sans fil de connexion…   Honte à l’“anti-rugby” de l’Angleterre, et vive les flamboyants “petits poucets”   Stormsure Stormproof Durable Hydrofuge (spray étanche) – 1L   Meilleurs choix d’appareils photo : Olympus OM-D E-M10 Mark II   $$* James Betterave Blanche It Betterave Organique 1L De Jus   Secret des Chefs Arome Naturel Grenadine 50 ml – Lot de 2   OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information *OpenAI Launches a Crawler Bot Named GPTBot to Scrape Websites for Information