* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, October 25, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    General Hospital’s Rena Sofer Exits as Lois — But the Door Isn’t Closed – Yahoo

    General Hospital’s Rena Sofer Exits as Lois — But the Door Isn’t Closed – Yahoo

    CNN Launches New Show – What to Know About Host Elex Michaelson – Central Oregon Daily

    Get to Know Elex Michaelson: The Dynamic New Host Taking CNN by Storm

    Johnny Depp Set To Finally Make His Big Hollywood Comeback After Amber Heard Controversy – Yahoo

    Johnny Depp Set for a Triumphant Hollywood Comeback Following Amber Heard Controversy

    ‘Chainsaw Man — The Movie: Reze Arc’ Review: Hit Manga Gets an Ultra-Violent, Surprisingly Emotional Big-Screen Adaptation – Yahoo

    Chainsaw Man – The Movie: Reze Arc Review: A Brutal and Unexpectedly Emotional Big-Screen Adaptation

    Reba McEntire Details Personal Relationship With Late Stepson Brandon Blackstock – KNDU

    Reba McEntire Shares Emotional Tribute to Her Late Stepson Brandon Blackstock

    Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

    Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Ghost Tapping is exploiting tap-to-pay technology in order to steal your money; what your need to know – ABC7 New York

    Ghost Tapping: How Thieves Are Using Tap-to-Pay Technology to Steal Your Money and What You Need to Know

    New technology for grading and packing dates – FreshPlaza

    Revolutionary Technology Transforms Date Grading and Packing Process

    Project underway to upgrade technology on 911 towers in Kanawha County – WCHS

    Kanawha County Launches Major Upgrade to 911 Tower Technology

    Next steps: Technology opens new options for greater mobility – Missouri Independent

    Next Steps: How Technology is Opening Exciting New Doors to Greater Mobility

    Rydberg Technologies Inc. Announces Launch of Rydberg Photonics in Berlin – The Quantum Insider

    Rydberg Technologies Launches Exciting New Photonics Division in Berlin

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    General Hospital’s Rena Sofer Exits as Lois — But the Door Isn’t Closed – Yahoo

    General Hospital’s Rena Sofer Exits as Lois — But the Door Isn’t Closed – Yahoo

    CNN Launches New Show – What to Know About Host Elex Michaelson – Central Oregon Daily

    Get to Know Elex Michaelson: The Dynamic New Host Taking CNN by Storm

    Johnny Depp Set To Finally Make His Big Hollywood Comeback After Amber Heard Controversy – Yahoo

    Johnny Depp Set for a Triumphant Hollywood Comeback Following Amber Heard Controversy

    ‘Chainsaw Man — The Movie: Reze Arc’ Review: Hit Manga Gets an Ultra-Violent, Surprisingly Emotional Big-Screen Adaptation – Yahoo

    Chainsaw Man – The Movie: Reze Arc Review: A Brutal and Unexpectedly Emotional Big-Screen Adaptation

    Reba McEntire Details Personal Relationship With Late Stepson Brandon Blackstock – KNDU

    Reba McEntire Shares Emotional Tribute to Her Late Stepson Brandon Blackstock

    Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

    Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Ghost Tapping is exploiting tap-to-pay technology in order to steal your money; what your need to know – ABC7 New York

    Ghost Tapping: How Thieves Are Using Tap-to-Pay Technology to Steal Your Money and What You Need to Know

    New technology for grading and packing dates – FreshPlaza

    Revolutionary Technology Transforms Date Grading and Packing Process

    Project underway to upgrade technology on 911 towers in Kanawha County – WCHS

    Kanawha County Launches Major Upgrade to 911 Tower Technology

    Next steps: Technology opens new options for greater mobility – Missouri Independent

    Next Steps: How Technology is Opening Exciting New Doors to Greater Mobility

    Rydberg Technologies Inc. Announces Launch of Rydberg Photonics in Berlin – The Quantum Insider

    Rydberg Technologies Launches Exciting New Photonics Division in Berlin

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Science

How Do Machines ‘Grok’ Data?

April 13, 2024
in Science
How Do Machines ‘Grok’ Data?
Share on FacebookShare on Twitter

As a network trains, it tends to learn more complex functions, and the discrepancy between the expected output and the actual one starts falling for training data. Even better, this discrepancy, known as loss, also starts going down for test data, which is new data not used in training. But at some point, the model starts to overfit, and while the loss on training data keeps falling, the test data’s loss starts to rise. So, typically, that’s when researchers stop training the network.

That was the prevailing wisdom when the team at OpenAI began exploring how a neural network could do math. They were using a small transformer — a network architecture that’s recently revolutionized large language models — to do different kinds of modular arithmetic, in which you work with a limited set numbers that loop back on themselves. Modulo 12, for example, can be done on a clock face: 11 + 2=1. The team showed the network examples of adding two numbers, a and b, to produce an output, c, in modulo 97 (equivalent to a clock face with 97 numbers). They then tested the transformer on unseen combinations of a and b to see if it could correctly predict c.

As expected, when the network entered the overfitting regime, the loss on the training data came close to zero (it had begun memorizing what it had seen), and the loss on the test data began climbing. It wasn’t generalizing. “And then one day, we got lucky,” said team leader Alethea Power, speaking in September 2022 at a conference in San Francisco. “And by lucky, I mean forgetful.”

The team member who was training the network went on vacation and forgot to stop the training. As this version of the network continued to train, it suddenly became accurate on unseen data. Automatic testing revealed this unexpected accuracy to the rest of the team, and they soon realized that the network had found clever ways of arranging the numbers a and b. Internally, the network represents the numbers in some high-dimensional space, but when the researchers projected these numbers down to 2D space and mapped them, the numbers formed a circle.

This was astonishing. The team never told the model it was doing modulo 97 math, or even what modulo meant — they just showed it examples of arithmetic. The model seemed to have stumbled upon some deeper, analytical solution — an equation that generalized to all combinations of a and b, even beyond the training data. The network had grokked, and the accuracy on test data shot up to 100%. “This is weird,” Power told her audience.

The team verified the results using different tasks and different networks. The discovery held up.

Of Clocks and Pizzas

But what was the equation the network had found? The OpenAI paper didn’t say, but the result caught Nanda’s attention. “One of the core mysteries and annoying things about neural networks is that they’re very good at what they do, but that by default, we have no idea how they work,” said Nanda, whose work focuses on reverse-engineering a trained network to figure out what algorithms it learned.

Nanda was fascinated by the OpenAI discovery, and he decided to pick apart a neural network that had grokked. He designed an even simpler version of the OpenAI neural network so that he could closely examine the model’s parameters as it learned to do modular arithmetic. He saw the same behavior: overfitting that gave way to generalization and an abrupt improvement in test accuracy. His network was also arranging numbers in a circle. It took some effort, but Nanda eventually figured out why.

While it was representing the numbers on a circle, the network wasn’t simply counting off digits like a kindergartner watching a clock: It was doing some sophisticated mathematical manipulations. By studying the values of the network’s parameters, Nanda and colleagues revealed that it was adding the clock numbers by performing “discrete Fourier transforms” on them — transforming the numbers using trigonometric functions such as sines and cosines and then manipulating these values using trigonometric identities to arrive at the solution. At least, this was what his particular network was doing.

When a team at MIT followed up on Nanda’s work, they showed that the grokking neural networks don’t always discover this “clock” algorithm. Sometimes, the networks instead find what the researchers call the “pizza” algorithm. This approach imagines a pizza divided into slices and numbered in order. To add two numbers, imagine drawing arrows from the center of the pizza to the numbers in question, then calculating the line that bisects the angle formed by the first two arrows. This line passes through the middle of some slice of the pizza: The number of the slice is the sum of the two numbers. These operations can also be written down in terms of trigonometric and algebraic manipulations of the sines and cosines of a and b, and they’re theoretically just as accurate as the clock approach.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Quanta Magazine – https://www.quantamagazine.org/how-do-machines-grok-data-20240412/

Tags: ‘Grok’machinesscience
Previous Post

My Fantastic Voyage at Quanta Magazine

Next Post

Inhale at Your Own Risk: Even Brief Secondhand Smoke Exposure Increases Risk of Dangerous Heart Rhythm Disorder

The NBA Made a Big Bet on Sports Gambling—and It Just Blew Up – The Wall Street Journal

The NBA Made a Big Bet on Sports Gambling—and It Just Blew Up – The Wall Street Journal

October 25, 2025
The $500m slugger who is taking Canada to the World Series – BBC

The $500 Million Slugger Powering Canada’s Journey to World Series Glory

October 25, 2025
Russian Central Bank Cuts Key Rate to 16.5% as Economy Slows – The Moscow Times

Russian Central Bank Cuts Key Rate to 16.5% in Response to Economic Slowdown

October 25, 2025
General Hospital’s Rena Sofer Exits as Lois — But the Door Isn’t Closed – Yahoo

General Hospital’s Rena Sofer Exits as Lois — But the Door Isn’t Closed – Yahoo

October 25, 2025
Health Department Encourages Residents to Take Steps to Prevent Respiratory Viruses – | Larimer County (.gov)

Stay Healthy This Season: Must-Know Tips to Shield Yourself from Respiratory Viruses

October 25, 2025
Politics | Oct 25th 2025 Edition – The Economist

Politics | Oct 25th 2025 Edition – The Economist

October 25, 2025
Nutritional supplements boost baby coral survival – EurekAlert!

How Nutritional Supplements Are Boosting Baby Coral Survival Rates

October 25, 2025
Scientists Are One Step Closer to Letting Humans ‘Breathe’ Through Their Butts – Yahoo

Scientists Are One Step Closer to Letting Humans ‘Breathe’ Through Their Butts – Yahoo

October 25, 2025
Bizarre Crystals in Titan’s Lakes Could Break a Fundamental Rule of Chemistry – ScienceAlert

Strange Crystals Found in Titan’s Lakes Challenge a Fundamental Chemistry Rule

October 25, 2025
Mortgage Rates Inch Toward 6% as Fed Weighs October Cut – IslanderNews.com

Mortgage Rates Surge Near 6% Amid Speculation of October Fed Rate Cut

October 25, 2025

Categories

Archives

October 2025
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Sep    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (885)
  • Economy (907)
  • Entertainment (21,778)
  • General (17,796)
  • Health (9,948)
  • Lifestyle (919)
  • News (22,149)
  • People (907)
  • Politics (917)
  • Science (16,117)
  • Sports (21,407)
  • Technology (15,886)
  • World (890)

Recent News

The NBA Made a Big Bet on Sports Gambling—and It Just Blew Up – The Wall Street Journal

The NBA Made a Big Bet on Sports Gambling—and It Just Blew Up – The Wall Street Journal

October 25, 2025
The $500m slugger who is taking Canada to the World Series – BBC

The $500 Million Slugger Powering Canada’s Journey to World Series Glory

October 25, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version