* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, May 24, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Elizabeth Hurley Will Be ‘Deceased’ on New Reality Show The Inheritance – Yahoo

    Elizabeth Hurley Takes a Bold Turn in New Reality Show ‘The Inheritance

    SRM Entertainment Announces $5 Million Private Placement – GlobeNewswire

    SRM Entertainment Secures $5 Million Investment to Fuel Growth!

    Embracer intends to spin off Coffee Stain Group by the end of 2025, with remaining business rebranded as Fellowship Entertainment – GamesIndustry.biz

    Embracer intends to spin off Coffee Stain Group by the end of 2025, with remaining business rebranded as Fellowship Entertainment – GamesIndustry.biz

    San Jose eyes creation of entertainments zones with FIFA World Cup, Super Bowl LX on the horizon – The Mercury News

    San Jose Sets Its Sights on Exciting Entertainment Zones Ahead of FIFA World Cup and Super Bowl LX!

    Wilmington’s future of fun: 5 recreation and entertainment spaces planned in the Port City – Wilmington Star-News

    Exciting Developments Ahead: 5 New Recreation and Entertainment Spaces Coming to Wilmington!

    Jason Momoa Is Done With Peace in Apple’s ‘Chief of War’ Teaser – Yahoo

    Jason Momoa Embraces Chaos in Gripping Teaser for Apple’s ‘Chief of War’

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Nike cuts some jobs in technology division – Reuters

    Nike Streamlines Workforce: Job Cuts in Technology Division

    InfiMotion Technology launches TL 300 integrated drive system – Automotive Powertrain Technology International

    InfiMotion Technology Unveils Game-Changing TL 300 Integrated Drive System!

    Aera Technology Debuts Decision Intelligence Skill to Navigate Shifting Tariff Dynamics Across Value Chains – Silicon Canals

    Unlocking Success: Aera Technology Launches Innovative Decision Intelligence Skill to Tackle Evolving Tariff Challenges in Value Chains

    Auditory Processing and Psychosocial Improvements with Remote Microphone Technology: An Evidence Review – The Hearing Review

    Unlocking Sound: How Remote Microphone Technology Enhances Auditory Processing and Boosts Psychosocial Well-Being

    Quadient and Nuvei Forge Strategic Technology Partnership – Finovate

    Quadient and Nuvei Forge Strategic Technology Partnership – Finovate

    Novotech Honored with Triple Win in 2025 Pharmaceutical Technology Excellence Awards – Morningstar

    Novotech Celebrates Triple Triumph at the 2025 Pharmaceutical Technology Excellence Awards!

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Elizabeth Hurley Will Be ‘Deceased’ on New Reality Show The Inheritance – Yahoo

    Elizabeth Hurley Takes a Bold Turn in New Reality Show ‘The Inheritance

    SRM Entertainment Announces $5 Million Private Placement – GlobeNewswire

    SRM Entertainment Secures $5 Million Investment to Fuel Growth!

    Embracer intends to spin off Coffee Stain Group by the end of 2025, with remaining business rebranded as Fellowship Entertainment – GamesIndustry.biz

    Embracer intends to spin off Coffee Stain Group by the end of 2025, with remaining business rebranded as Fellowship Entertainment – GamesIndustry.biz

    San Jose eyes creation of entertainments zones with FIFA World Cup, Super Bowl LX on the horizon – The Mercury News

    San Jose Sets Its Sights on Exciting Entertainment Zones Ahead of FIFA World Cup and Super Bowl LX!

    Wilmington’s future of fun: 5 recreation and entertainment spaces planned in the Port City – Wilmington Star-News

    Exciting Developments Ahead: 5 New Recreation and Entertainment Spaces Coming to Wilmington!

    Jason Momoa Is Done With Peace in Apple’s ‘Chief of War’ Teaser – Yahoo

    Jason Momoa Embraces Chaos in Gripping Teaser for Apple’s ‘Chief of War’

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Nike cuts some jobs in technology division – Reuters

    Nike Streamlines Workforce: Job Cuts in Technology Division

    InfiMotion Technology launches TL 300 integrated drive system – Automotive Powertrain Technology International

    InfiMotion Technology Unveils Game-Changing TL 300 Integrated Drive System!

    Aera Technology Debuts Decision Intelligence Skill to Navigate Shifting Tariff Dynamics Across Value Chains – Silicon Canals

    Unlocking Success: Aera Technology Launches Innovative Decision Intelligence Skill to Tackle Evolving Tariff Challenges in Value Chains

    Auditory Processing and Psychosocial Improvements with Remote Microphone Technology: An Evidence Review – The Hearing Review

    Unlocking Sound: How Remote Microphone Technology Enhances Auditory Processing and Boosts Psychosocial Well-Being

    Quadient and Nuvei Forge Strategic Technology Partnership – Finovate

    Quadient and Nuvei Forge Strategic Technology Partnership – Finovate

    Novotech Honored with Triple Win in 2025 Pharmaceutical Technology Excellence Awards – Morningstar

    Novotech Celebrates Triple Triumph at the 2025 Pharmaceutical Technology Excellence Awards!

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

How to run an LLM on your PC, not in the cloud, in less than 10 minutes

March 17, 2024
in Technology
How to run an LLM on your PC, not in the cloud, in less than 10 minutes
Share on FacebookShare on Twitter

Hands On With all the talk of massive machine-learning training clusters and AI PCs you’d be forgiven for thinking you need some kind of special hardware to play with text-and-code-generating large language models (LLMs) at home.

In reality, there’s a good chance the desktop system you’re reading this on is more than capable of running a wide range of LLMs, including chat bots like Mistral or source code generators like Codellama.

In fact, with openly available tools like Ollama, LM Suite, and Llama.cpp, it’s relatively easy to get these models running on your system.

In the interest of simplicity and cross-platform compatibility, we’re going to be looking at Ollama, which once installed works more or less the same across Windows, Linux, and Macs.

A word on performance, compatibility, and AMD GPU support:

In general, large language models like Mistral or Llama 2 run best with dedicated accelerators. There’s a reason datacenter operators are buying and deploying GPUs in clusters of 10,000 or more, though you’ll need the merest fraction of such resources.

Ollama offers native support for Nvidia and Apple’s M-series GPUs. Nvidia GPUs with at least 4GB of memory should work. We tested with a 12GB RTX 3060, though we recommend at least 16GB of memory for M-series Macs.

Linux users will want Nvidia’s latest proprietary driver and probably the CUDA binaries installed first. There’s more information on setting that up here.

If you’re rocking a Radeon 7000-series GPU or newer, AMD has a full guide on getting an LLM running on your system, which you can find here.

The good news is, if you don’t have a supported graphics card, Ollama will still run on an AVX2-compatible CPU, although a whole lot slower than if you had a supported GPU. And while 16GB of memory is recommended, you may be able to get by with less by opting for a quantized model — more on that in a minute.

Installing Ollama

Installing Ollama is pretty straight forward, regardless of your base operating system. It’s open source, which you can check out here.

For those running Windows or Mac OS, head over ollama.com and download and install it like any other application.

For those running Linux, it’s even simpler: Just run this one liner — you can find manual installation instructions here, if you want them — and you’re off to the races.

curl -fsSL https://ollama.com/install.sh | sh

Installing your first model

Regardless of your operating system, working with Ollama is largely the same. Ollama recommends starting with Llama 2 7B, a seven-billion-parameter transformer-based neural network, but for this guide we’ll be taking a look at Mistral 7B since it’s pretty capable and been the source of some controversy in recent weeks.

Start by opening PowerShell or a terminal emulator and executing the following command to download and start the model in an interactive chat mode.

ollama run mistral

Upon download, you’ll be dropped in to a chat prompt where you can start interacting with the model, just like ChatGPT, Copilot, or Google Gemini.

LLMs, like Mistral 7B, run surprisingly well on this 2-year-old M1 Max MacBook Pro

LLMs, like Mistral 7B, run surprisingly well on this 2-year-old M1 Max MacBook Pro – Click to enlarge

If you don’t get anything, you may need to launch Ollama from the start menu on Windows or applications folder on Mac first.

Models, tags, and quantization

Mistal 7B is just one of several LLMs, including other versions of the model, that are accessible using Ollama. You can find the full list, along with instructions for running each here, but the general syntax goes something like this:

ollama run model-name:model-tag

Model-tags are used to specify which version of the model you’d like to download. If you leave it off, Ollama assume you want the latest version. In our experience, this tends to be a 4-bit quantized version of the model.

If, for example, you wanted to run Meta’s Llama2 7B at FP16, it’d look like this:

ollama run llama2:7b-chat-fp16

But before you try that, you might want to double check your system has enough memory. Our previous example with Mistral used 4-bit quantization, which means the model needs half a gigabyte of memory for every 1 billion parameters. And don’t forget: It has seven billion parameters.

Quantization is a technique used to compress the model by converting its weights and activations to a lower precision. This allows Mistral 7B to run within 4GB of GPU or system RAM, usually with minimal sacrifice in quality of the output, though your mileage may vary.

The Llama 2 7B example used above runs at half precision (FP16). As a result, you’d actually need 2GB of memory per billion parameters, which in this case works out to just over 14GB. Unless you’ve got a newer GPU with 16GB or more of vRAM, you may not have enough resources to run the model at that precision.

Managing Ollama

Managing, updating, and removing installed models using Ollama should feel right at home for anyone who’s used things like the Docker CLI before.

In this section we’ll go over a few of the more common tasks you might want to execute.

To get a list of installed models run:

ollama list

To remove a model, you’d run:

ollama rm model-name:model-tag

To pull or update an existing model, run:

ollama pull model-name:model-tag

Additional Ollama commands can be found by running:

ollama –help

As we noted earlier, Ollama is just one of many frameworks for running and testing local LLMs. If you run in to trouble with this one, you may find more luck with others. And no, an AI did not write this.

The Register aims to bring you more on utilizing LLMs in the near future, so be sure to share your burning AI PC questions in the comments section. And don’t forget about supply chain security. ®

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : The Register – https://go.theregister.com/feed/www.theregister.com/2024/03/17/ai_pc_local_llm/

Tags: cloudMinutestechnology
Previous Post

In the rush to build AI apps, please, please don’t leave security behind

Next Post

Stardew Valley’s 1.6 update gives the biggest gift for serial organizers: foodstuff that’s “colored based on the ingredient”

China Unveils for Public Comment Its Draft Ecological and Environmental Code for Integrating the Country’s Existing Environmental Laws – JD Supra

China Introduces Groundbreaking Draft Ecological and Environmental Code for Public Feedback!

May 24, 2025
Marvel’s Spiderman exhibit swings into Chicago’s Museum of Science and Industry – NBC 5 Chicago

Swing into Adventure: Marvel’s Spider-Man Exhibit Lands at Chicago’s Museum of Science and Industry!

May 24, 2025
Raven’s R&B science fiction evokes a San Francisco that still could be – 48 Hills

Raven’s R&B science fiction evokes a San Francisco that still could be – 48 Hills

May 24, 2025
On Nutrition: Sugar from zero to 50 – The Union Democrat

Sweet Secrets: Understanding Sugar’s Journey from Zero to 50!

May 24, 2025

Insects may rule the world – Times Republican

May 24, 2025
WNBA franchises are a slam dunk in this iffy economy – marketplace.org

WNBA franchises are a slam dunk in this iffy economy – marketplace.org

May 24, 2025
‘Andor’ creator says Jyn Erso cameo would have been ‘lame’ and ‘disrespectful’ – Entertainment Weekly

Why ‘Andor’ Creator Believes a Jyn Erso Cameo Would Have Missed the Mark

May 24, 2025
Chile joins Alliance for Primary Health Care in the Americas to strengthen primary healthcare – News-Medical

Chile Takes Bold Step to Enhance Primary Healthcare by Joining the Alliance for Primary Health Care in the Americas

May 24, 2025

Elon Musk says he’s going to do ‘a lot less’ political spending – NBC News

May 24, 2025
Information and communication technology (ICT) – Buro Happold

Unlocking the Future: The Transformative Power of Information and Communication Technology (ICT)

May 24, 2025

Categories

Archives

May 2025
MTWTFSS
 1234
567891011
12131415161718
19202122232425
262728293031 
« Apr    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (636)
  • Economy (648)
  • Entertainment (21,559)
  • General (15,233)
  • Health (9,689)
  • Lifestyle (653)
  • News (22,149)
  • People (651)
  • Politics (655)
  • Science (15,871)
  • Sports (21,156)
  • Technology (15,635)
  • World (637)

Recent News

China Unveils for Public Comment Its Draft Ecological and Environmental Code for Integrating the Country’s Existing Environmental Laws – JD Supra

China Introduces Groundbreaking Draft Ecological and Environmental Code for Public Feedback!

May 24, 2025
Marvel’s Spiderman exhibit swings into Chicago’s Museum of Science and Industry – NBC 5 Chicago

Swing into Adventure: Marvel’s Spider-Man Exhibit Lands at Chicago’s Museum of Science and Industry!

May 24, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version