* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Friday, June 27, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Susquehanna Raises Penn Entertainment Inc. (PENN) Price Target. – Yahoo Finance

    Susquehanna Raises Price Target for Penn Entertainment Inc. (PENN)

    George Lopez is coming to Spokane – KXLY.com

    George Lopez is coming to Spokane – KXLY.com

    Netflix unveils Dallas immersive venue for fans of hit shows like ‘Squid Game,’ ‘Stranger Things’ – Houston Chronicle

    Step Inside Netflix’s New Dallas Immersive Experience Featuring Hits Like ‘Squid Game’ and ‘Stranger Things

    ‘Puttin’ on the Ritz’: Civic Players bring ‘Young Frankenstein’ to life – Yahoo

    Civic Players Deliver a Hilarious and Unforgettable Performance of ‘Young Frankenstein

    ‘Wheel of Fortune’: Amputee Wins $60,000 After Breaking Incredible ‘Curse’ – Hastings Tribune

    Wheel of Fortune’ Amputee Breaks Incredible ‘Curse’ to Win $60,000!

    North Star Sports & Entertainment Network: Coming soon – KTTC News

    North Star Sports & Entertainment Network: Coming soon – KTTC News

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    St. Francis Medical Center brings advanced robotic surgery technology to Northeast Louisiana – KNOE

    St. Francis Medical Center brings advanced robotic surgery technology to Northeast Louisiana – KNOE

    Wayve Expands Engineering Leadership to Power Next-Gen Autonomous Driving Technology – Silicon Canals

    Wayve Boosts Engineering Leadership to Accelerate Next-Gen Autonomous Driving Innovation

    Frontdoor Announces Tech Expert Dr. Bala Ganesh as Chief Technology Officer – Business Wire

    Frontdoor Appoints Tech Visionary Dr. Bala Ganesh as New Chief Technology Officer

    Defense technology giant Northrop Grumman to host interviews in Iuka to fill technician roles – supertalk.fm

    Defense technology giant Northrop Grumman to host interviews in Iuka to fill technician roles – supertalk.fm

    China’s Military Introduces Mosquito-Sized Drones: A Game-Changing Surveillance Technology – Indian Defence Review

    China Unveils Mosquito-Sized Drones: Revolutionizing Surveillance Technology

    Marvell Technology Stock Rallies After AI Event Sparks Investor Optimism – Yahoo Finance

    Marvell Technology Stock Rallies After AI Event Sparks Investor Optimism – Yahoo Finance

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Susquehanna Raises Penn Entertainment Inc. (PENN) Price Target. – Yahoo Finance

    Susquehanna Raises Price Target for Penn Entertainment Inc. (PENN)

    George Lopez is coming to Spokane – KXLY.com

    George Lopez is coming to Spokane – KXLY.com

    Netflix unveils Dallas immersive venue for fans of hit shows like ‘Squid Game,’ ‘Stranger Things’ – Houston Chronicle

    Step Inside Netflix’s New Dallas Immersive Experience Featuring Hits Like ‘Squid Game’ and ‘Stranger Things

    ‘Puttin’ on the Ritz’: Civic Players bring ‘Young Frankenstein’ to life – Yahoo

    Civic Players Deliver a Hilarious and Unforgettable Performance of ‘Young Frankenstein

    ‘Wheel of Fortune’: Amputee Wins $60,000 After Breaking Incredible ‘Curse’ – Hastings Tribune

    Wheel of Fortune’ Amputee Breaks Incredible ‘Curse’ to Win $60,000!

    North Star Sports & Entertainment Network: Coming soon – KTTC News

    North Star Sports & Entertainment Network: Coming soon – KTTC News

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    St. Francis Medical Center brings advanced robotic surgery technology to Northeast Louisiana – KNOE

    St. Francis Medical Center brings advanced robotic surgery technology to Northeast Louisiana – KNOE

    Wayve Expands Engineering Leadership to Power Next-Gen Autonomous Driving Technology – Silicon Canals

    Wayve Boosts Engineering Leadership to Accelerate Next-Gen Autonomous Driving Innovation

    Frontdoor Announces Tech Expert Dr. Bala Ganesh as Chief Technology Officer – Business Wire

    Frontdoor Appoints Tech Visionary Dr. Bala Ganesh as New Chief Technology Officer

    Defense technology giant Northrop Grumman to host interviews in Iuka to fill technician roles – supertalk.fm

    Defense technology giant Northrop Grumman to host interviews in Iuka to fill technician roles – supertalk.fm

    China’s Military Introduces Mosquito-Sized Drones: A Game-Changing Surveillance Technology – Indian Defence Review

    China Unveils Mosquito-Sized Drones: Revolutionizing Surveillance Technology

    Marvell Technology Stock Rallies After AI Event Sparks Investor Optimism – Yahoo Finance

    Marvell Technology Stock Rallies After AI Event Sparks Investor Optimism – Yahoo Finance

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Science

How Do Machines ‘Grok’ Data?

April 13, 2024
in Science
How Do Machines ‘Grok’ Data?
Share on FacebookShare on Twitter

As a network trains, it tends to learn more complex functions, and the discrepancy between the expected output and the actual one starts falling for training data. Even better, this discrepancy, known as loss, also starts going down for test data, which is new data not used in training. But at some point, the model starts to overfit, and while the loss on training data keeps falling, the test data’s loss starts to rise. So, typically, that’s when researchers stop training the network.

That was the prevailing wisdom when the team at OpenAI began exploring how a neural network could do math. They were using a small transformer — a network architecture that’s recently revolutionized large language models — to do different kinds of modular arithmetic, in which you work with a limited set numbers that loop back on themselves. Modulo 12, for example, can be done on a clock face: 11 + 2=1. The team showed the network examples of adding two numbers, a and b, to produce an output, c, in modulo 97 (equivalent to a clock face with 97 numbers). They then tested the transformer on unseen combinations of a and b to see if it could correctly predict c.

As expected, when the network entered the overfitting regime, the loss on the training data came close to zero (it had begun memorizing what it had seen), and the loss on the test data began climbing. It wasn’t generalizing. “And then one day, we got lucky,” said team leader Alethea Power, speaking in September 2022 at a conference in San Francisco. “And by lucky, I mean forgetful.”

The team member who was training the network went on vacation and forgot to stop the training. As this version of the network continued to train, it suddenly became accurate on unseen data. Automatic testing revealed this unexpected accuracy to the rest of the team, and they soon realized that the network had found clever ways of arranging the numbers a and b. Internally, the network represents the numbers in some high-dimensional space, but when the researchers projected these numbers down to 2D space and mapped them, the numbers formed a circle.

This was astonishing. The team never told the model it was doing modulo 97 math, or even what modulo meant — they just showed it examples of arithmetic. The model seemed to have stumbled upon some deeper, analytical solution — an equation that generalized to all combinations of a and b, even beyond the training data. The network had grokked, and the accuracy on test data shot up to 100%. “This is weird,” Power told her audience.

The team verified the results using different tasks and different networks. The discovery held up.

Of Clocks and Pizzas

But what was the equation the network had found? The OpenAI paper didn’t say, but the result caught Nanda’s attention. “One of the core mysteries and annoying things about neural networks is that they’re very good at what they do, but that by default, we have no idea how they work,” said Nanda, whose work focuses on reverse-engineering a trained network to figure out what algorithms it learned.

Nanda was fascinated by the OpenAI discovery, and he decided to pick apart a neural network that had grokked. He designed an even simpler version of the OpenAI neural network so that he could closely examine the model’s parameters as it learned to do modular arithmetic. He saw the same behavior: overfitting that gave way to generalization and an abrupt improvement in test accuracy. His network was also arranging numbers in a circle. It took some effort, but Nanda eventually figured out why.

While it was representing the numbers on a circle, the network wasn’t simply counting off digits like a kindergartner watching a clock: It was doing some sophisticated mathematical manipulations. By studying the values of the network’s parameters, Nanda and colleagues revealed that it was adding the clock numbers by performing “discrete Fourier transforms” on them — transforming the numbers using trigonometric functions such as sines and cosines and then manipulating these values using trigonometric identities to arrive at the solution. At least, this was what his particular network was doing.

When a team at MIT followed up on Nanda’s work, they showed that the grokking neural networks don’t always discover this “clock” algorithm. Sometimes, the networks instead find what the researchers call the “pizza” algorithm. This approach imagines a pizza divided into slices and numbered in order. To add two numbers, imagine drawing arrows from the center of the pizza to the numbers in question, then calculating the line that bisects the angle formed by the first two arrows. This line passes through the middle of some slice of the pizza: The number of the slice is the sum of the two numbers. These operations can also be written down in terms of trigonometric and algebraic manipulations of the sines and cosines of a and b, and they’re theoretically just as accurate as the clock approach.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Quanta Magazine – https://www.quantamagazine.org/how-do-machines-grok-data-20240412/

Tags: ‘Grok’machinesscience
Previous Post

My Fantastic Voyage at Quanta Magazine

Next Post

Inhale at Your Own Risk: Even Brief Secondhand Smoke Exposure Increases Risk of Dangerous Heart Rhythm Disorder

Opinion: American Science Is in a Nosedive – and We’re All on Board – The Austin Chronicle

Opinion: American Science Is in a Nosedive – and We’re All on Board

June 27, 2025

Food contact articles as source of micro- and nanoplastics: a systematic evidence map | npj Science of Food – Nature

June 27, 2025
Michelle Dee is featured in Pride Month issue of digital lifestyle magazine – GMA Network

Michelle Dee is featured in Pride Month issue of digital lifestyle magazine – GMA Network

June 27, 2025
These are the best airlines of 2025 – and no US carrier made the list. Here’s why. – USA Today

These are the best airlines of 2025 – and no US carrier made the list. Here’s why. – USA Today

June 27, 2025
Britain’s industrial strategy is unlikely to boost its economy – The Economist

Britain’s industrial strategy is unlikely to boost its economy – The Economist

June 27, 2025
Ashley Jensen joins Superior Clinic to practice family medicine – Essentia Health

Ashley Jensen Joins Superior Clinic to Deliver Expert Family Medicine Care

June 27, 2025
Heads of state and political leaders gather at ‘Summer Davos’. Here’s what they are saying – The World Economic Forum

Heads of state and political leaders gather at ‘Summer Davos’. Here’s what they are saying – The World Economic Forum

June 27, 2025
St. Francis Medical Center brings advanced robotic surgery technology to Northeast Louisiana – KNOE

St. Francis Medical Center brings advanced robotic surgery technology to Northeast Louisiana – KNOE

June 27, 2025
NBA Draft 2025: Did the Hornets reach for Kon Knueppel? And why the Spurs did the right thing with Dylan Harper – Yahoo Sports

NBA Draft 2025: Did the Hornets Overreach for Kon Knueppel? Why the Spurs’ Pick of Dylan Harper Was a Brilliant Move

June 27, 2025
Fossil evidence of proboscidean frugivory and its lasting impact on South American ecosystems – Nature

Fossil evidence of proboscidean frugivory and its lasting impact on South American ecosystems – Nature

June 27, 2025

Categories

Archives

June 2025
MTWTFSS
 1
2345678
9101112131415
16171819202122
23242526272829
30 
« May    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (699)
  • Economy (720)
  • Entertainment (21,613)
  • General (15,593)
  • Health (9,759)
  • Lifestyle (725)
  • News (22,149)
  • People (722)
  • Politics (727)
  • Science (15,937)
  • Sports (21,216)
  • Technology (15,704)
  • World (700)

Recent News

Opinion: American Science Is in a Nosedive – and We’re All on Board – The Austin Chronicle

Opinion: American Science Is in a Nosedive – and We’re All on Board

June 27, 2025

Food contact articles as source of micro- and nanoplastics: a systematic evidence map | npj Science of Food – Nature

June 27, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version