* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Thursday, October 23, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

    Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

    AMC brings first new Dolby Experience to Gwinnett since 2017 – Wyoming News Now

    AMC Launches First New Dolby Experience in Gwinnett Since 2017

    Hetzel Design: blending architecture and entertainment – Blooloop

    Hetzel Design: Where Architecture and Entertainment Unite in Perfect Harmony

    Country music legend rushed to hospital year after heart surgery. Here’s what we know – PennLive.com

    Country Music Legend Rushed to Hospital One Year After Heart Surgery – What’s Happening Now?

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Placer County town of Loomis considers entertainment zone for downtown – CBS News

    Loomis Unveils Thrilling New Entertainment Zone to Revitalize Downtown

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Rydberg Technologies Inc. Announces Launch of Rydberg Photonics in Berlin – The Quantum Insider

    Rydberg Technologies Launches Exciting New Photonics Division in Berlin

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 Technology Stocks to Buy Now – Yahoo Finance

    3 Must-Buy Tech Stocks You Can’t Afford to Miss Right Now

    ‘New frontier’: Austin leaders start discussions on air taxi technology – KXAN Austin

    Austin Leaders Ignite Exciting Conversations on the Future of Air Taxi Technology

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

    Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

    AMC brings first new Dolby Experience to Gwinnett since 2017 – Wyoming News Now

    AMC Launches First New Dolby Experience in Gwinnett Since 2017

    Hetzel Design: blending architecture and entertainment – Blooloop

    Hetzel Design: Where Architecture and Entertainment Unite in Perfect Harmony

    Country music legend rushed to hospital year after heart surgery. Here’s what we know – PennLive.com

    Country Music Legend Rushed to Hospital One Year After Heart Surgery – What’s Happening Now?

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Strictly Come Dancing results: Chris Robshaw is eliminated while drag queen La Voix escapes dance-off – Yahoo

    Placer County town of Loomis considers entertainment zone for downtown – CBS News

    Loomis Unveils Thrilling New Entertainment Zone to Revitalize Downtown

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Rydberg Technologies Inc. Announces Launch of Rydberg Photonics in Berlin – The Quantum Insider

    Rydberg Technologies Launches Exciting New Photonics Division in Berlin

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    A look into new technology at Columbia University that could help prevent a dangerous pregnancy complication – ABC7 New York

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    Office Technology: Dealers’ Managed IT Revenue up Nearly 30% – The Cannata Report –

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 E Network Technology Group Limited Closes $1.5 Million Convertible Promissory Note Offering – Quiver Quantitative

    3 Technology Stocks to Buy Now – Yahoo Finance

    3 Must-Buy Tech Stocks You Can’t Afford to Miss Right Now

    ‘New frontier’: Austin leaders start discussions on air taxi technology – KXAN Austin

    Austin Leaders Ignite Exciting Conversations on the Future of Air Taxi Technology

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

How to run an LLM on your PC, not in the cloud, in less than 10 minutes

March 17, 2024
in Technology
How to run an LLM on your PC, not in the cloud, in less than 10 minutes
Share on FacebookShare on Twitter

Hands On With all the talk of massive machine-learning training clusters and AI PCs you’d be forgiven for thinking you need some kind of special hardware to play with text-and-code-generating large language models (LLMs) at home.

In reality, there’s a good chance the desktop system you’re reading this on is more than capable of running a wide range of LLMs, including chat bots like Mistral or source code generators like Codellama.

In fact, with openly available tools like Ollama, LM Suite, and Llama.cpp, it’s relatively easy to get these models running on your system.

In the interest of simplicity and cross-platform compatibility, we’re going to be looking at Ollama, which once installed works more or less the same across Windows, Linux, and Macs.

A word on performance, compatibility, and AMD GPU support:

In general, large language models like Mistral or Llama 2 run best with dedicated accelerators. There’s a reason datacenter operators are buying and deploying GPUs in clusters of 10,000 or more, though you’ll need the merest fraction of such resources.

Ollama offers native support for Nvidia and Apple’s M-series GPUs. Nvidia GPUs with at least 4GB of memory should work. We tested with a 12GB RTX 3060, though we recommend at least 16GB of memory for M-series Macs.

Linux users will want Nvidia’s latest proprietary driver and probably the CUDA binaries installed first. There’s more information on setting that up here.

If you’re rocking a Radeon 7000-series GPU or newer, AMD has a full guide on getting an LLM running on your system, which you can find here.

The good news is, if you don’t have a supported graphics card, Ollama will still run on an AVX2-compatible CPU, although a whole lot slower than if you had a supported GPU. And while 16GB of memory is recommended, you may be able to get by with less by opting for a quantized model — more on that in a minute.

Installing Ollama

Installing Ollama is pretty straight forward, regardless of your base operating system. It’s open source, which you can check out here.

For those running Windows or Mac OS, head over ollama.com and download and install it like any other application.

For those running Linux, it’s even simpler: Just run this one liner — you can find manual installation instructions here, if you want them — and you’re off to the races.

curl -fsSL https://ollama.com/install.sh | sh

Installing your first model

Regardless of your operating system, working with Ollama is largely the same. Ollama recommends starting with Llama 2 7B, a seven-billion-parameter transformer-based neural network, but for this guide we’ll be taking a look at Mistral 7B since it’s pretty capable and been the source of some controversy in recent weeks.

Start by opening PowerShell or a terminal emulator and executing the following command to download and start the model in an interactive chat mode.

ollama run mistral

Upon download, you’ll be dropped in to a chat prompt where you can start interacting with the model, just like ChatGPT, Copilot, or Google Gemini.

LLMs, like Mistral 7B, run surprisingly well on this 2-year-old M1 Max MacBook Pro

LLMs, like Mistral 7B, run surprisingly well on this 2-year-old M1 Max MacBook Pro – Click to enlarge

If you don’t get anything, you may need to launch Ollama from the start menu on Windows or applications folder on Mac first.

Models, tags, and quantization

Mistal 7B is just one of several LLMs, including other versions of the model, that are accessible using Ollama. You can find the full list, along with instructions for running each here, but the general syntax goes something like this:

ollama run model-name:model-tag

Model-tags are used to specify which version of the model you’d like to download. If you leave it off, Ollama assume you want the latest version. In our experience, this tends to be a 4-bit quantized version of the model.

If, for example, you wanted to run Meta’s Llama2 7B at FP16, it’d look like this:

ollama run llama2:7b-chat-fp16

But before you try that, you might want to double check your system has enough memory. Our previous example with Mistral used 4-bit quantization, which means the model needs half a gigabyte of memory for every 1 billion parameters. And don’t forget: It has seven billion parameters.

Quantization is a technique used to compress the model by converting its weights and activations to a lower precision. This allows Mistral 7B to run within 4GB of GPU or system RAM, usually with minimal sacrifice in quality of the output, though your mileage may vary.

The Llama 2 7B example used above runs at half precision (FP16). As a result, you’d actually need 2GB of memory per billion parameters, which in this case works out to just over 14GB. Unless you’ve got a newer GPU with 16GB or more of vRAM, you may not have enough resources to run the model at that precision.

Managing Ollama

Managing, updating, and removing installed models using Ollama should feel right at home for anyone who’s used things like the Docker CLI before.

In this section we’ll go over a few of the more common tasks you might want to execute.

To get a list of installed models run:

ollama list

To remove a model, you’d run:

ollama rm model-name:model-tag

To pull or update an existing model, run:

ollama pull model-name:model-tag

Additional Ollama commands can be found by running:

ollama –help

As we noted earlier, Ollama is just one of many frameworks for running and testing local LLMs. If you run in to trouble with this one, you may find more luck with others. And no, an AI did not write this.

The Register aims to bring you more on utilizing LLMs in the near future, so be sure to share your burning AI PC questions in the comments section. And don’t forget about supply chain security. ®

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : The Register – https://go.theregister.com/feed/www.theregister.com/2024/03/17/ai_pc_local_llm/

Tags: cloudMinutestechnology
Previous Post

In the rush to build AI apps, please, please don’t leave security behind

Next Post

Stardew Valley’s 1.6 update gives the biggest gift for serial organizers: foodstuff that’s “colored based on the ingredient”

Forest cover enhances pest control by birds and bats independently of vineyard management intensity – besjournals

Forest cover enhances pest control by birds and bats independently of vineyard management intensity – besjournals

October 22, 2025
November Family Consumer Science Program – Alachua County

November Family Consumer Science Program – Alachua County

October 22, 2025
Correction: Anti-Science Movement story – The News-Gazette

Unraveling the Truth Behind the Anti-Science Movement: A Closer Look

October 22, 2025
Best Burgers 2025: 9 cult-favorite burgers off the beaten path – Yahoo

Best Burgers 2025: 9 cult-favorite burgers off the beaten path – Yahoo

October 22, 2025
Rydberg Technologies Inc. Announces Launch of Rydberg Photonics in Berlin – The Quantum Insider

Rydberg Technologies Launches Exciting New Photonics Division in Berlin

October 22, 2025
Forbes Unveils First-Ever “Most Powerful Women In Sports” List, Featuring 25 Of The Trailblazers Transforming The Industry, Including Gayle Benson, Caitlin Clark, Michele Kang, Serena Williams And More – Forbes

Meet the 25 Trailblazing Women Revolutionizing the World of Sports, Featuring Gayle Benson, Caitlin Clark, Michele Kang, Serena Williams and More

October 22, 2025
Grandmaster Daniel Naroditsky remembered for his impact on the chess world – NPR

Grandmaster Daniel Naroditsky: Honoring a Timeless Chess Legacy

October 22, 2025
Seattle’s Creative Economy: Building Talent, Culture, and Community – GeekWire

Seattle’s Creative Economy: How Talent, Culture, and Community Are Shaping the Future

October 22, 2025
Sacramento city leaders approve adding 2 entertainment zones in midtown – CBS News

Sacramento City Leaders Approve Two Thrilling New Entertainment Zones in Midtown

October 22, 2025
Self-efficacy mediates the effect of hope on health promotion intention in Chinese stroke patients – Nature

How Hope Boosts Health Motivation in Chinese Stroke Patients Through Self-Efficacy

October 22, 2025

Categories

Archives

October 2025
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  
« Sep    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (881)
  • Economy (902)
  • Entertainment (21,773)
  • General (17,746)
  • Health (9,943)
  • Lifestyle (915)
  • News (22,149)
  • People (903)
  • Politics (912)
  • Science (16,113)
  • Sports (21,402)
  • Technology (15,882)
  • World (885)

Recent News

Forest cover enhances pest control by birds and bats independently of vineyard management intensity – besjournals

Forest cover enhances pest control by birds and bats independently of vineyard management intensity – besjournals

October 22, 2025
November Family Consumer Science Program – Alachua County

November Family Consumer Science Program – Alachua County

October 22, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version