* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Friday, August 8, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Themed Entertainment Design – Purdue Polytechnic

    Innovative Themed Entertainment Design: Creating Immersive Experiences

    Rachael Leigh Cook and Brandon Routh ‘Happy to Have Found Each Other’ Following Respective Divorces – yahoo.com

    Rachael Leigh Cook and Brandon Routh ‘Happy to Have Found Each Other’ Following Respective Divorces – yahoo.com

    ‘Billie Jean’ – Hyde Park Herald

    The Enduring Magic Behind ‘Billie Jean’ Revealed

    Hank Hill returns to a changed world in new ‘King of the Hill’ episodes – New Haven Register

    Hank Hill Navigates a Bold New World in Thrilling New ‘King of the Hill’ Episodes

    Exclusive | Fox Takes Stake in IndyCar Owner Penske Entertainment – The Wall Street Journal

    Exclusive | Fox Takes Stake in IndyCar Owner Penske Entertainment – The Wall Street Journal

    Go-to entertainment: why gaming was made for the toilet – The Guardian

    Why Gaming Is the Ultimate Way to Pass Time in the Bathroom

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    MBU showcases student work at Occupational Therapy Technology Fair – WHSV

    Discover the Most Innovative Student Projects at the Occupational Therapy Technology Fair

    BlackSky Technology Inc. (BKSY) Reports Q2 Loss, Lags Revenue Estimates – Yahoo Finance

    BlackSky Technology Inc. Reports Q2 Loss, Misses Revenue Targets

    Improved Technology Access: A Key to Closing the Healthcare Gap for African Americans – BIOENGINEER.ORG

    LMI Expands Technology Org, Appoints New Leaders – GovCon Wire

    LMI Expands Technology Team with Dynamic New Leadership Appointments

    Midland Innovation and Technology Charter School closing down – CBS News

    Midland Innovation and Technology Charter School Closes Permanently

    Future Trends In HR Technology – Dataconomy

    Future Trends In HR Technology – Dataconomy

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Themed Entertainment Design – Purdue Polytechnic

    Innovative Themed Entertainment Design: Creating Immersive Experiences

    Rachael Leigh Cook and Brandon Routh ‘Happy to Have Found Each Other’ Following Respective Divorces – yahoo.com

    Rachael Leigh Cook and Brandon Routh ‘Happy to Have Found Each Other’ Following Respective Divorces – yahoo.com

    ‘Billie Jean’ – Hyde Park Herald

    The Enduring Magic Behind ‘Billie Jean’ Revealed

    Hank Hill returns to a changed world in new ‘King of the Hill’ episodes – New Haven Register

    Hank Hill Navigates a Bold New World in Thrilling New ‘King of the Hill’ Episodes

    Exclusive | Fox Takes Stake in IndyCar Owner Penske Entertainment – The Wall Street Journal

    Exclusive | Fox Takes Stake in IndyCar Owner Penske Entertainment – The Wall Street Journal

    Go-to entertainment: why gaming was made for the toilet – The Guardian

    Why Gaming Is the Ultimate Way to Pass Time in the Bathroom

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    MBU showcases student work at Occupational Therapy Technology Fair – WHSV

    Discover the Most Innovative Student Projects at the Occupational Therapy Technology Fair

    BlackSky Technology Inc. (BKSY) Reports Q2 Loss, Lags Revenue Estimates – Yahoo Finance

    BlackSky Technology Inc. Reports Q2 Loss, Misses Revenue Targets

    Improved Technology Access: A Key to Closing the Healthcare Gap for African Americans – BIOENGINEER.ORG

    LMI Expands Technology Org, Appoints New Leaders – GovCon Wire

    LMI Expands Technology Team with Dynamic New Leadership Appointments

    Midland Innovation and Technology Charter School closing down – CBS News

    Midland Innovation and Technology Charter School Closes Permanently

    Future Trends In HR Technology – Dataconomy

    Future Trends In HR Technology – Dataconomy

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

Hugging Face releases a benchmark for testing generative AI on health tasks

April 19, 2024
in Technology
Hugging Face releases a benchmark for testing generative AI on health tasks
Share on FacebookShare on Twitter

Generative AI models are increasingly being brought to healthcare settings — in some cases prematurely, perhaps. Early adopters believe that they’ll unlock increased efficiency while revealing insights that’d otherwise be missed. Critics, meanwhile, point out that these models have flaws and biases that could contribute to worse health outcomes.

But is there a quantitative way to know how helpful, or harmful, a model might be when tasked with things like summarizing patient records or answering health-related questions?

Hugging Face, the AI startup, proposes a solution in a newly released benchmark test called Open Medical-LLM. Created in partnership with researchers at the nonprofit Open Life Science AI and the University of Edinburgh’s Natural Language Processing Group, Open Medical-LLM aims to standardize evaluating the performance of generative AI models on a range of medical-related tasks.

New: Open Medical LLM Leaderboard! 🩺

In basic chatbots, errors are annoyances.
In medical LLMs, errors can have life-threatening consequences 🩸

It’s therefore vital to benchmark/follow advances in medical LLMs before thinking about deployment.

Blog: https://t.co/pddLtkmhsz

— Clémentine Fourrier 🍊 (@clefourrier) April 18, 2024

Open Medical-LLM isn’t a from-scratch benchmark, per se, but rather a stitching-together of existing test sets — MedQA, PubMedQA, MedMCQA and so on — designed to probe models for general medical knowledge and related fields, such as anatomy, pharmacology, genetics and clinical practice. The benchmark contains multiple choice and open-ended questions that require medical reasoning and understanding, drawing from material including U.S. and Indian medical licensing exams and college biology test question banks.

“[Open Medical-LLM] enables researchers and practitioners to identify the strengths and weaknesses of different approaches, drive further advancements in the field and ultimately contribute to better patient care and outcome,” Hugging Face wrote in a blog post.

gen AI healthcare

Image Credits: Hugging Face

Hugging Face is positioning the benchmark as a “robust assessment” of healthcare-bound generative AI models. But some medical experts on social media cautioned against putting too much stock into Open Medical-LLM, lest it lead to ill-informed deployments.

On X, Liam McCoy, a resident physician in neurology at the University of Alberta, pointed out that the gap between the “contrived environment” of medical question-answering and actual clinical practice can be quite large.

It is great progress to see these comparisons head-to-head, but important for us to also remember how big the gap is between the contrived environment of medical question answering and actual clinical practice! Not to mention the idiosyncratic risks these metrics can’t capture.

— Liam McCoy, MD MSc (@LiamGMcCoy) April 18, 2024

Hugging Face research scientist Clémentine Fourrier, who co-authored the blog post, agreed.

“These leaderboards should only be used as a first approximation of which [generative AI model] to explore for a given use case, but then a deeper phase of testing is always needed to examine the model’s limits and relevance in real conditions,” Fourrier replied on X. “Medical [models] should absolutely not be used on their own by patients, but instead should be trained to become support tools for MDs.”

It brings to mind Google’s experience when it tried to bring an AI screening tool for diabetic retinopathy to healthcare systems in Thailand.

Google created a deep learning system that scanned images of the eye, looking for evidence of retinopathy, a leading cause of vision loss. But despite high theoretical accuracy, the tool proved impractical in real-world testing, frustrating both patients and nurses with inconsistent results and a general lack of harmony with on-the-ground practices.

It’s telling that of the 139 AI-related medical devices the U.S. Food and Drug Administration has approved to date, none use generative AI. It’s exceptionally difficult to test how a generative AI tool’s performance in the lab will translate to hospitals and outpatient clinics, and, perhaps more importantly, how the outcomes might trend over time.

That’s not to suggest Open Medical-LLM isn’t useful or informative. The results leaderboard, if nothing else, serves as a reminder of just how poorly models answer basic health questions. But Open Medical-LLM, and no other benchmark for that matter, is a substitute for carefully thought-out real-world testing.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : TechCrunch – https://techcrunch.com/2024/04/18/hugging-face-releases-a-benchmark-for-testing-generative-ai-on-health-tasks/

Tags: Huggingreleasestechnology
Previous Post

Tesla still plans to build 1,800-mile charging corridor for semi trucks despite Biden funding snub

Next Post

Internet users are getting younger; now the UK is weighing up if AI can help protect them

Icing‐related injuries in polar bears (Ursus maritimus) at high latitudes – Laidre – 2024 – Ecology – ESA Journals

Frozen Peril: The Devastating Impact of Icing Injuries on Polar Bears in the High Arctic

August 8, 2025
Carnegie Science Center launching new name in September – CBS News

Carnegie Science Center Unveils Exciting New Name This September

August 8, 2025
Petersburg youth explore Coho Creek for science education – Petersburg Pilot

Petersburg Youth Dive into Science with Hands-On Exploration of Coho Creek

August 8, 2025
Is there a path to healthier aging? What the latest research shows | Bodyworks – The Oklahoman

Is there a path to healthier aging? What the latest research shows | Bodyworks – The Oklahoman

August 8, 2025
MBU showcases student work at Occupational Therapy Technology Fair – WHSV

Discover the Most Innovative Student Projects at the Occupational Therapy Technology Fair

August 8, 2025
Official | Evann Guessand completes €35m Aston Villa move – Yahoo Sports

Official | Evann Guessand completes €35m Aston Villa move – Yahoo Sports

August 8, 2025
Trailer: Netflix Animation Welcomes Viewers to the Whimsical World of Dr. Seuss! – Animation Magazine

Trailer Unveils the Whimsical World of Dr. Seuss in Netflix Animation!

August 8, 2025
Spending on AI data centers is so massive that it’s taken a bigger chunk of GDP growth than shopping—and it could crash the American economy – Fortune

Spending on AI data centers is so massive that it’s taken a bigger chunk of GDP growth than shopping—and it could crash the American economy – Fortune

August 8, 2025
SPC Health Programs Showcase: Featuring Nursing, Radiography, and Surgical Services Degrees – St. Petersburg College

Explore Exciting Career Paths in Nursing, Radiography, and Surgical Services at SPC Health Programs Showcase

August 8, 2025
Top Trump officials discussed Epstein at White House meeting Wednesday night – CNN

Top Trump officials discussed Epstein at White House meeting Wednesday night – CNN

August 8, 2025

Categories

Archives

August 2025
MTWTFSS
 123
45678910
11121314151617
18192021222324
25262728293031
« Jul    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (760)
  • Economy (783)
  • Entertainment (21,659)
  • General (16,347)
  • Health (9,822)
  • Lifestyle (793)
  • News (22,149)
  • People (784)
  • Politics (792)
  • Science (15,996)
  • Sports (21,280)
  • Technology (15,763)
  • World (765)

Recent News

Icing‐related injuries in polar bears (Ursus maritimus) at high latitudes – Laidre – 2024 – Ecology – ESA Journals

Frozen Peril: The Devastating Impact of Icing Injuries on Polar Bears in the High Arctic

August 8, 2025
Carnegie Science Center launching new name in September – CBS News

Carnegie Science Center Unveils Exciting New Name This September

August 8, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version