* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Monday, June 16, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    ‘Gangs of London’ Producer Explains Season 3 Deaths, Hypes Season 4 – Citizen Tribune

    Gangs of London’ Producer Reveals Shocking Season 3 Deaths and Teases Exciting Season 4

    The Iconic Missouri Diner That Gives You A Taste Of Live Entertainment With Your Meal – Yahoo

    Savor Delicious Meals While Enjoying Live Entertainment at Missouri’s Iconic Diner

    Keke Palmer Revealed How She Came Up With Her Son Leodis’ Name – Yahoo

    Keke Palmer Shares the Heartwarming Story Behind Her Son Leodis’ Name

    The Media and Entertainment Deal Machine Is Revving Up – WSJ

    The Media and Entertainment Deal Machine Is Gearing Up for Action

    Op-Ed: Data Storage and Protection in Today’s Media & Entertainment Industry – Sports Video Group

    How Data Storage and Protection Are Transforming the Media & Entertainment Industry

    The Analyst Verdict: PENN Entertainment In The Eyes Of 7 Experts – Nasdaq

    7 Experts Reveal the Ultimate Verdict on PENN Entertainment

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Further Upside For Aeries Technology, Inc (NASDAQ:AERT) Shares Could Introduce Price Risks After 27% Bounce – simplywall.st

    Further Upside For Aeries Technology, Inc (NASDAQ:AERT) Shares Could Introduce Price Risks After 27% Bounce – simplywall.st

    Editor’s Pick: 9 Books on Technology – The Gospel Coalition

    9 Must-Read Books That Will Completely Transform How You Understand Technology

    New Semiconductor Technology Could Supercharge 6G Delivery – SciTechDaily

    Revolutionary Semiconductor Technology Set to Turbocharge 6G Connectivity

    UTC To Host Quantum Technology Workshop June 23-25 – Chattanoogan.com: Breaking News

    Join the Quantum Technology Workshop This June 23-25!

    Rimac Technology Powers the Bugatti Tourbillon with Cutting-Edge Battery and Powertrain Tech – Rimac Newsroom

    Rimac Technology Drives the Bugatti Tourbillon with Revolutionary Battery and Powertrain Innovation

    “Co-creation” boosts commercial technology for dual-use defense applications – Breaking Defense

    “Co-creation” boosts commercial technology for dual-use defense applications – Breaking Defense

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    ‘Gangs of London’ Producer Explains Season 3 Deaths, Hypes Season 4 – Citizen Tribune

    Gangs of London’ Producer Reveals Shocking Season 3 Deaths and Teases Exciting Season 4

    The Iconic Missouri Diner That Gives You A Taste Of Live Entertainment With Your Meal – Yahoo

    Savor Delicious Meals While Enjoying Live Entertainment at Missouri’s Iconic Diner

    Keke Palmer Revealed How She Came Up With Her Son Leodis’ Name – Yahoo

    Keke Palmer Shares the Heartwarming Story Behind Her Son Leodis’ Name

    The Media and Entertainment Deal Machine Is Revving Up – WSJ

    The Media and Entertainment Deal Machine Is Gearing Up for Action

    Op-Ed: Data Storage and Protection in Today’s Media & Entertainment Industry – Sports Video Group

    How Data Storage and Protection Are Transforming the Media & Entertainment Industry

    The Analyst Verdict: PENN Entertainment In The Eyes Of 7 Experts – Nasdaq

    7 Experts Reveal the Ultimate Verdict on PENN Entertainment

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Further Upside For Aeries Technology, Inc (NASDAQ:AERT) Shares Could Introduce Price Risks After 27% Bounce – simplywall.st

    Further Upside For Aeries Technology, Inc (NASDAQ:AERT) Shares Could Introduce Price Risks After 27% Bounce – simplywall.st

    Editor’s Pick: 9 Books on Technology – The Gospel Coalition

    9 Must-Read Books That Will Completely Transform How You Understand Technology

    New Semiconductor Technology Could Supercharge 6G Delivery – SciTechDaily

    Revolutionary Semiconductor Technology Set to Turbocharge 6G Connectivity

    UTC To Host Quantum Technology Workshop June 23-25 – Chattanoogan.com: Breaking News

    Join the Quantum Technology Workshop This June 23-25!

    Rimac Technology Powers the Bugatti Tourbillon with Cutting-Edge Battery and Powertrain Tech – Rimac Newsroom

    Rimac Technology Drives the Bugatti Tourbillon with Revolutionary Battery and Powertrain Innovation

    “Co-creation” boosts commercial technology for dual-use defense applications – Breaking Defense

    “Co-creation” boosts commercial technology for dual-use defense applications – Breaking Defense

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

Ferret: An End-to-End MLLM by Apple

December 23, 2023
in Technology
Ferret: An End-to-End MLLM by Apple
Share on FacebookShare on Twitter

An End-to-End MLLM that Accept Any-Form Referring and Ground Anything in Response. [Paper]

Haoxuan You*, Haotian Zhang*, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang
[*: equal contribution]

Overview


Diagram of Ferret Model.

Key Contributions:

Ferret Model – Hybrid Region Representation + Spatial-aware Visual Sampler enable fine-grained and open-vocabulary referring and grounding in MLLM.
GRIT Dataset (~1.1M) – A Large-scale, Hierarchical, Robust ground-and-refer instruction tuning dataset.
Ferret-Bench – A multimodal evaluation benchmark that jointly requires Referring/Grounding, Semantics, Knowledge, and Reasoning.

Release

[12/14] 🔥 We released the checkpoints(7B, 13B).
[10/30] 🔥 We released the code of FERRET model and Ferret-Bench.

Usage and License Notices: The data, and code is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.

Contents

Install
Train
Evaluation
Demo

Install

Clone this repository and navigate to FERRET folder

git clone https://github.com/apple/ml-ferret
cd ml-ferret

Install Package

conda create -n ferret python=3.10 -y
conda activate ferret
pip install –upgrade pip # enable PEP 660 support
pip install -e .
pip install pycocotools
pip install protobuf==3.20.0

Install additional packages for training cases

pip install ninja
pip install flash-attn –no-build-isolation

Train

FERRET is trained on 8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly. Always keep the global batch size the same: per_device_train_batch_size x gradient_accumulation_steps x num_gpus.

Hyperparameters

We use a similar set of hyperparameters as LLaVA(Vicuna) in finetuning.

Hyperparameter
Global Batch Size
Learning rate
Epochs
Max length
Weight decay

FERRET-7B
128
2e-5
3
2048
0

FERRET-13B
128
2e-5
3
2048
0

Prepare Vicuna checkpoint and LLaVA’s projector

Before you start, prepare our base model Vicuna, which is an instruction-tuned chatbot. Please download its weights following the instructions here. Vicuna v1.3 is used in FERRET.

Then download LLaVA’s first-stage pre-trained projector weight (7B, 13B).

FERRET Training

The scripts are provided (7B, 13B).

Evaluation

Please see this doc for the details.

Checkpoints

We extracted the delta between our pre-trained model and Vicuna. Please first download weights of Vicuna following the previous instruction. Then download our prepared offsets of weights: 7B, 13B using wget or curl, and unzip the downloaded offsets. Lastly, apply the offset to the Vicuna’s weight by running the following script:

# 7B
python3 -m ferret.model.apply_delta
–base ./model/vicuna-7b-v1-3
–target ./model/ferret-7b-v1-3
–delta path/to/ferret-7b-delta
# 13B
python3 -m ferret.model.apply_delta
–base ./model/vicuna-13b-v1-3
–target ./model/ferret-13b-v1-3
–delta path/to/ferret-13b-delta

Notices: Apple’s rights in the attached weight differentials are hereby licensed under the CC-BY-NC license. Apple makes no representations with regards to LLaMa or any other third party software, which are subject to their own terms.

Please refer to the next section about how to set up a local demo with pre-trained weight.

Demo

To run our demo, you need to train FERRET and use the checkpoints locally. Gradio web UI is used. Please run the following commands one by one.

Launch a controller
python -m ferret.serve.controller –host 0.0.0.0 –port 10000
Launch a gradio web server.
python -m ferret.serve.gradio_web_server –controller http://localhost:10000 –model-list-mode reload –add_region_feature
Launch a model worker

This is the worker that load the ckpt and do the inference on the GPU. Each worker is responsible for a single model specified in –model-path.

CUDA_VISIBLE_DEVICES=0 python -m ferret.serve.model_worker –host 0.0.0.0 –controller http://localhost:10000 –port 40000 –worker http://localhost:40000 –model-path ./checkpoints/FERRET-13B-v0 –add_region_feature

Wait until the process finishes loading the model and you see “Uvicorn running on …”. Now, refresh your Gradio web UI, and you will see the model you just launched in the model list.


Example of Ferret Interactive Demo.

Citation

If you find Ferret useful, please cite using this BibTeX:

@article{you2023ferret,
title={Ferret: Refer and Ground Anything Anywhere at Any Granularity},
author={You, Haoxuan and Zhang, Haotian and Gan, Zhe and Du, Xianzhi and Zhang, Bowen and Wang, Zirui and Cao, Liangliang and Chang, Shih-Fu and Yang, Yinfei},
journal={arXiv preprint arXiv:2310.07704},
year={2023}
}

Acknowledgement

LLaVA: the codebase we built upon.
Vicuna: the LLM codebase.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Hacker News – https://github.com/apple/ml-ferret

Tags: end-to-endFerrettechnology
Previous Post

AI companies required to disclose copyrighted training data under new bill

Next Post

You Can Train Your Brain to React to Stressful Situations Better. Here’s the 3-Step Process.

FDA Grants Sarepta Therapeutics Platform Technology Designation to Expedite Gene Therapy Reviews – geneonline.com

FDA Accelerates Gene Therapy Reviews with Breakthrough Platform Technology for Sarepta Therapeutics

June 16, 2025
Grading Auburn sports in 2024-25: How we saw football stack up in latest athletic calendar – Montgomery Advertiser

Auburn Sports 2024-25: Breaking Down the Football Season’s Biggest Highlights

June 16, 2025
Brain development in a facultatively social allodapine bee aligns with caste, but not group living – Frontiers

Brain development in a facultatively social allodapine bee aligns with caste, but not group living – Frontiers

June 16, 2025
NASA’s Hurricane Science, Tech, Data Help American Communities – NASA Science (.gov)

How NASA’s Breakthrough Science and Technology Are Safeguarding American Communities from Hurricanes

June 16, 2025
Groundwater in the Colorado River basin won’t run out — but eventually we won’t be able to get at it, scientists warn – Live Science

Urgent Warning: Colorado River Basin’s Groundwater Is Still There-but Access May Vanish Soon

June 16, 2025
Review: The best, worst and most interesting of Trader Joe’s summer products – – The Daily Tar Heel

Review: The best, worst and most interesting of Trader Joe’s summer products – – The Daily Tar Heel

June 16, 2025
Steve Kornacki: New poll shows Trump approval underwater due to low marks on the economy – MSN

Steve Kornacki: New poll shows Trump approval underwater due to low marks on the economy – MSN

June 15, 2025
‘Gangs of London’ Producer Explains Season 3 Deaths, Hypes Season 4 – Citizen Tribune

Gangs of London’ Producer Reveals Shocking Season 3 Deaths and Teases Exciting Season 4

June 15, 2025
Top Health Care Execs to Watch in 2025: Guidehouse’s Sarah Garnett – WashingtonExec

Meet the Trailblazing Health Care Leaders to Watch in 2025: Spotlight on Sarah Garnett

June 15, 2025
How politics undermine Prince George’s County Public Schools success – Maryland Matters

How politics undermine Prince George’s County Public Schools success – Maryland Matters

June 15, 2025

Categories

Archives

June 2025
MTWTFSS
 1
2345678
9101112131415
16171819202122
23242526272829
30 
« May    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (688)
  • Economy (702)
  • Entertainment (21,605)
  • General (15,404)
  • Health (9,743)
  • Lifestyle (707)
  • News (22,149)
  • People (704)
  • Politics (709)
  • Science (15,920)
  • Sports (21,202)
  • Technology (15,690)
  • World (683)

Recent News

FDA Grants Sarepta Therapeutics Platform Technology Designation to Expedite Gene Therapy Reviews – geneonline.com

FDA Accelerates Gene Therapy Reviews with Breakthrough Platform Technology for Sarepta Therapeutics

June 16, 2025
Grading Auburn sports in 2024-25: How we saw football stack up in latest athletic calendar – Montgomery Advertiser

Auburn Sports 2024-25: Breaking Down the Football Season’s Biggest Highlights

June 16, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version