* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Sunday, July 27, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Top 5 entertainment news: Sandeep Reddy Vanga regrets trimming Animal’s runtime by 7-8 minutes, Akshay Ku – Times of India

    Top 5 Entertainment Highlights: Sandeep Reddy Vanga Reveals Why He Trimmed Animal’s Runtime by 7-8 Minutes, Plus Akshay Ku Updates

    Cote de Pablo reveals how Michael Weatherly used his soap opera roots to put her at ease in “NCIS” love scene – yahoo.com

    Cote de Pablo Reveals How Michael Weatherly’s Soap Opera Background Made Their “NCIS” Love Scene Easier

    City of Pelham announces entertainment district plans for former Oak Mountain Amphitheatre site – WVTM

    Pelham Unveils Exciting New Entertainment District Plans for Former Oak Mountain Amphitheatre Site

    Black Box Players presents ‘The Three Musketeers’ – CBS 19 News

    Experience the Adventure: Black Box Players Bring ‘The Three Musketeers’ to Life!

    AP Entertainment SummaryBrief at 1:51 p.m. EDT – Channel 3000

    Entertainment Highlights: Key Updates You Can’t Miss

    ‘Devil Wears Prada 2’ casts Anne Hathaway’s love interest replacing Adrian Grenier’s Nate – Entertainment Weekly

    Devil Wears Prada 2′ Casts New Love Interest for Anne Hathaway, Replacing Adrian Grenier’s Nate

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology

    Eagle Unveils Revolutionary X-Ray Technology at Pack Expo

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Behind the Screens: The Impact of Technology on Real Estate – TRREB

    Behind the Screens: How Technology is Transforming the Future of Real Estate

    Sustainserv and Palau Announce Technology Partnership to Leverage Innovative AI Platform to Advance Sustainability Reporting – Business Wire

    Sustainserv and Palau Team Up to Transform Sustainability Reporting with Breakthrough AI Technology

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Top 5 entertainment news: Sandeep Reddy Vanga regrets trimming Animal’s runtime by 7-8 minutes, Akshay Ku – Times of India

    Top 5 Entertainment Highlights: Sandeep Reddy Vanga Reveals Why He Trimmed Animal’s Runtime by 7-8 Minutes, Plus Akshay Ku Updates

    Cote de Pablo reveals how Michael Weatherly used his soap opera roots to put her at ease in “NCIS” love scene – yahoo.com

    Cote de Pablo Reveals How Michael Weatherly’s Soap Opera Background Made Their “NCIS” Love Scene Easier

    City of Pelham announces entertainment district plans for former Oak Mountain Amphitheatre site – WVTM

    Pelham Unveils Exciting New Entertainment District Plans for Former Oak Mountain Amphitheatre Site

    Black Box Players presents ‘The Three Musketeers’ – CBS 19 News

    Experience the Adventure: Black Box Players Bring ‘The Three Musketeers’ to Life!

    AP Entertainment SummaryBrief at 1:51 p.m. EDT – Channel 3000

    Entertainment Highlights: Key Updates You Can’t Miss

    ‘Devil Wears Prada 2’ casts Anne Hathaway’s love interest replacing Adrian Grenier’s Nate – Entertainment Weekly

    Devil Wears Prada 2′ Casts New Love Interest for Anne Hathaway, Replacing Adrian Grenier’s Nate

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology

    Eagle Unveils Revolutionary X-Ray Technology at Pack Expo

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    Validea’s Top Information Technology Stocks Based On Peter Lynch – 7/25/2025 – Nasdaq

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    WhoFi: New surveillance technology can track people by how they disrupt Wi-Fi signals – Tech Xplore

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Google Cloud Announced as a Key Technology Partner for Odoo Connect 2025 in San Francisco – GlobeNewswire

    Behind the Screens: The Impact of Technology on Real Estate – TRREB

    Behind the Screens: How Technology is Transforming the Future of Real Estate

    Sustainserv and Palau Announce Technology Partnership to Leverage Innovative AI Platform to Advance Sustainability Reporting – Business Wire

    Sustainserv and Palau Team Up to Transform Sustainability Reporting with Breakthrough AI Technology

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Science

Robotic Grasp of Language: Unlocking an Open-Ended World for Automation

November 17, 2023
in Science
Robotic Grasp of Language: Unlocking an Open-Ended World for Automation
Share on FacebookShare on Twitter

Robot Open World Art Concept

MIT’s CSAIL introduced F3RM, a robotic system that combines visual and language features, allowing robots to grasp objects following open-ended instructions. This innovation, which supports task generalization from few examples, could significantly improve efficiency in warehouses and extend to various real-world applications, including domestic assistance.

By blending 2D images with foundation models to build 3D feature fields, a new MIT method helps robots understand and manipulate nearby objects with open-ended language prompts.

Imagine you’re visiting a friend abroad, and you look inside their fridge to see what would make for a great breakfast. Many of the items initially appear foreign to you, with each one encased in unfamiliar packaging and containers. Despite these visual distinctions, you begin to understand what each one is used for and pick them up as needed.

Inspired by humans’ ability to handle unfamiliar objects, a group from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) designed Feature Fields for Robotic Manipulation (F3RM), a system that blends 2D images with foundation model features into 3D scenes to help robots identify and grasp nearby items. F3RM can interpret open-ended language prompts from humans, making the method helpful in real-world environments that contain thousands of objects, like warehouses and households.

Robotic Adaptability and Task Generalization

F3RM offers robots the ability to interpret open-ended text prompts using natural language, helping the machines manipulate objects. As a result, the machines can understand less-specific requests from humans and still complete the desired task. For example, if a user asks the robot to “pick up a tall mug,” the robot can locate and grab the item that best fits that description.

Feature Fields for Robotic Manipulation (F3RM)

Feature Fields for Robotic Manipulation (F3RM) enables robots to interpret open-ended text prompts using natural language, helping the machines manipulate unfamiliar objects. The system’s 3D feature fields could be helpful in environments that contain thousands of objects, such as warehouses. Credit: Courtesy of the researchers

“Making robots that can actually generalize in the real world is incredibly hard,” says Ge Yang, postdoc at the National Science Foundation AI Institute for Artificial Intelligence and Fundamental Interactions and MIT CSAIL. “We really want to figure out how to do that, so with this project, we try to push for an aggressive level of generalization, from just three or four objects to anything we find in MIT’s Stata Center. We wanted to learn how to make robots as flexible as ourselves, since we can grasp and place objects even though we’ve never seen them before.”

Learning “What’s Where by Looking”

The method could assist robots with picking items in large fulfillment centers with inevitable clutter and unpredictability. In these warehouses, robots are often given a description of the inventory that they’re required to identify. The robots must match the text provided to an object, regardless of variations in packaging, so that customers’ orders are shipped correctly.

For example, the fulfillment centers of major online retailers can contain millions of items, many of which a robot will have never encountered before. To operate at such a scale, robots need to understand the geometry and semantics of different items, with some being in tight spaces. With F3RM’s advanced spatial and semantic perception abilities, a robot could become more effective at locating an object, placing it in a bin, and then sending it along for packaging. Ultimately, this would help factory workers ship customers’ orders more efficiently.

“One thing that often surprises people with F3RM is that the same system also works on a room and building scale, and can be used to build simulation environments for robot learning and large maps,” says Yang. “But before we scale up this work further, we want to first make this system work really fast. This way, we can use this type of representation for more dynamic robotic control tasks, hopefully in real-time, so that robots that handle more dynamic tasks can use it for perception.”

Application Across Environments

The MIT team notes that F3RM’s ability to understand different scenes could make it useful in urban and household environments. For example, the approach could help personalized robots identify and pick up specific items. The system aids robots in grasping their surroundings — both physically and perceptively.

“Visual perception was defined by David Marr as the problem of knowing ‘what is where by looking,’” says senior author Phillip Isola, MIT associate professor of electrical engineering and computer science and CSAIL principal investigator. “Recent foundation models have gotten really good at knowing what they are looking at; they can recognize thousands of object categories and provide detailed text descriptions of images. At the same time, radiance fields have gotten really good at representing where stuff is in a scene. The combination of these two approaches can create a representation of what is where in 3D, and what our work shows is that this combination is especially useful for robotic tasks, which require manipulating objects in 3D.”

Creating a “Digital Twin”

F3RM begins to understand its surroundings by taking pictures on a selfie stick. The mounted camera snaps 50 images at different poses, enabling it to build a neural radiance field (NeRF), a deep learning method that takes 2D images to construct a 3D scene. This collage of RGB photos creates a “digital twin” of its surroundings in the form of a 360-degree representation of what’s nearby.

In addition to a highly detailed neural radiance field, F3RM also builds a feature field to augment geometry with semantic information. The system uses CLIP, a vision foundation model trained on hundreds of millions of images to efficiently learn visual concepts. By reconstructing the 2D CLIP features for the images taken by the selfie stick, F3RM effectively lifts the 2D features into a 3D representation.

Open-Ended Interaction

After receiving a few demonstrations, the robot applies what it knows about geometry and semantics to grasp objects it has never encountered before. Once a user submits a text query, the robot searches through the space of possible grasps to identify those most likely to succeed in picking up the object requested by the user. Each potential option is scored based on its relevance to the prompt, similarity to the demonstrations the robot has been trained on, and if it causes any collisions. The highest-scored grasp is then chosen and executed.

To demonstrate the system’s ability to interpret open-ended requests from humans, the researchers prompted the robot to pick up Baymax, a character from Disney’s “Big Hero 6.” While F3RM had never been directly trained to pick up a toy of the cartoon superhero, the robot used its spatial awareness and vision-language features from the foundation models to decide which object to grasp and how to pick it up.

F3RM also enables users to specify which object they want the robot to handle at different levels of linguistic detail. For example, if there is a metal mug and a glass mug, the user can ask the robot for the “glass mug.” If the bot sees two glass mugs and one of them is filled with coffee and the other with juice, the user can ask for the “glass mug with coffee.” The foundation model features embedded within the feature field enable this level of open-ended understanding.

“If I showed a person how to pick up a mug by the lip, they could easily transfer that knowledge to pick up objects with similar geometries such as bowls, measuring beakers, or even rolls of tape. For robots, achieving this level of adaptability has been quite challenging,” says MIT PhD student, CSAIL affiliate, and co-lead author William Shen. “F3RM combines geometric understanding with semantics from foundation models trained on internet-scale data to enable this level of aggressive generalization from just a small number of demonstrations.”

Reference: “Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation” by William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling an Phillip Isola, 27 July 2023, Computer Science> Computer Vision and Pattern Recognition.
arXiv:2308.07931

Shen and Yang wrote the paper under the supervision of Isola, with MIT professor and CSAIL principal investigator Leslie Pack Kaelbling and undergraduate students Alan Yu and Jansen Wong as co-authors. The team was supported, in part, by Amazon.com Services, the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research’s Multidisciplinary University Initiative, the Army Research Office, the MIT-IBM Watson Lab, and the MIT Quest for Intelligence. Their work will be presented at the 2023 Conference on Robot Learning.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : SciTechDaily – https://scitechdaily.com/robotic-grasp-of-language-unlocking-an-open-ended-world-for-automation/

Tags: Grasproboticscience
Previous Post

Scientists Shed New Light on the “Dark Matter” of Cellular Biology

Next Post

Formula One race car cracks a drain cover during Las Vegas Grand Prix practice run

Energy landscapes direct the movement preferences of elephants – besjournals

Energy landscapes direct the movement preferences of elephants – besjournals

July 27, 2025
At United Nations Summer Program, Computer Science Student Examines AI Ethics – Georgetown University

At United Nations Summer Program, Computer Science Student Examines AI Ethics – Georgetown University

July 27, 2025
Return of wolves to Yellowstone has led to a surge in aspen trees unseen for 80 years – Live Science

Wolves’ Return to Yellowstone Sparks Unprecedented Aspen Tree Boom After 80 Years

July 27, 2025
‘Bluey’ Is Growing Up: BBC Plots All-Ages Expansion For The Franchise As A Full-Fledged Lifestyle Brand – Cartoon Brew

‘Bluey’ Is Growing Up: BBC Plots All-Ages Expansion For The Franchise As A Full-Fledged Lifestyle Brand – Cartoon Brew

July 27, 2025
Katie Ledecky, Summer McIntosh and a race for the ages at World Swimming Championships – NBC Sports

Katie Ledecky and Summer McIntosh Ignite an Epic Showdown at the World Swimming Championships

July 27, 2025
5 ways Trump has shaped the economy in 6 months – The Hill

5 Game-Changing Ways Trump Transformed the Economy in Just 6 Months

July 27, 2025
Flutter Entertainment Announces Pricing Of $1.272 Bln Of Additional Senior Secured Notes Due 2031 – Nasdaq

Flutter Entertainment Raises $1.27 Billion in Major Senior Secured Notes Offering Due 2031

July 27, 2025
KFF Health Tracking Poll: Public Views on Recent Tax and Budget Legislation – KFF

How the Public Really Feels About the Latest Tax and Budget Changes

July 27, 2025
Dollar falls against yen as markets weigh new trade deal, Japanese politics – Reuters

Dollar falls against yen as markets weigh new trade deal, Japanese politics – Reuters

July 27, 2025
Clinch County Schools prepare for students’ return with new policies, technology – WALB

Clinch County Schools Prepare for Students’ Return with Exciting New Policies and Cutting-Edge Technology

July 27, 2025

Categories

Archives

July 2025
MTWTFSS
 123456
78910111213
14151617181920
21222324252627
28293031 
« Jun    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (741)
  • Economy (765)
  • Entertainment (21,645)
  • General (16,133)
  • Health (9,803)
  • Lifestyle (773)
  • News (22,149)
  • People (767)
  • Politics (774)
  • Science (15,980)
  • Sports (21,262)
  • Technology (15,746)
  • World (748)

Recent News

Energy landscapes direct the movement preferences of elephants – besjournals

Energy landscapes direct the movement preferences of elephants – besjournals

July 27, 2025
At United Nations Summer Program, Computer Science Student Examines AI Ethics – Georgetown University

At United Nations Summer Program, Computer Science Student Examines AI Ethics – Georgetown University

July 27, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version