* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Saturday, September 27, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Cardi B Adds More Dates to Little Miss Drama Tour: ‘Y’all Making Me Work’ – Yahoo

    Cardi B Extends Little Miss Drama Tour: “Y’all Making Me Work

    ‘Today’: Sheinelle Jones Thanks Katie Couric for Support After Husband’s Death – CBS 19 News

    Sheinelle Jones Expresses Heartfelt Thanks to Katie Couric for Support After Husband’s Passing

    Sate your hunger at DBA’s Taste of Downtown – Bakersfield.com

    Indulge Your Cravings at DBA’s Taste of Downtown!

    Caesars Entertainment (CZR): Assessing Valuation After Times Square Casino Setback and Mounting Investor Concerns – simplywall.st

    Caesars Entertainment Faces Times Square Casino Hurdles as Investor Concerns Mount

    Why Hilaria Baldwin Has Found the ‘DWTS’ Process ‘Embarrassing’ At Times – WFXG

    Hilaria Baldwin Opens Up About the Embarrassing Moments on Her ‘DWTS’ Journey

    Harvest Fest 2025 – yadkinripple.com

    Celebrate the Bounty: Harvest Fest 2025 is Coming!

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Aurora police hope to add facial recognition technology to crime-fighting tools – CBS News

    Aurora Police Aim to Boost Crime-Fighting with New Facial Recognition Technology

    Autonomous Solutions shows off cutting-edge technology for the public – Cache Valley Daily

    Autonomous Solutions Unveils Cutting-Edge Technology for the Public

    Amazon to Pay $2.5 Billion in Prime Membership Settlement – The New York Times

    Amazon to Pay $2.5 Billion in Prime Membership Settlement – The New York Times

    What are we really gaining from technology? – Fast Company

    What Are We Really Gaining from Technology?

    TOMI Environmental Solutions, Inc. Expands SteraMist iHP Technology Services in Healthcare Sector with New Provider Partnership – Quiver Quantitative

    TOMI Environmental Solutions Accelerates SteraMist iHP Technology Expansion in Healthcare with New Provider Partnership

    Indiana County Technology Center’s Joint Operating Committee looks to the future as program plans began to take shape – Indiana Gazette Online

    Indiana County Technology Center’s Joint Operating Committee Charts an Exciting Path Forward as New Program Plans Take Shape

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Cardi B Adds More Dates to Little Miss Drama Tour: ‘Y’all Making Me Work’ – Yahoo

    Cardi B Extends Little Miss Drama Tour: “Y’all Making Me Work

    ‘Today’: Sheinelle Jones Thanks Katie Couric for Support After Husband’s Death – CBS 19 News

    Sheinelle Jones Expresses Heartfelt Thanks to Katie Couric for Support After Husband’s Passing

    Sate your hunger at DBA’s Taste of Downtown – Bakersfield.com

    Indulge Your Cravings at DBA’s Taste of Downtown!

    Caesars Entertainment (CZR): Assessing Valuation After Times Square Casino Setback and Mounting Investor Concerns – simplywall.st

    Caesars Entertainment Faces Times Square Casino Hurdles as Investor Concerns Mount

    Why Hilaria Baldwin Has Found the ‘DWTS’ Process ‘Embarrassing’ At Times – WFXG

    Hilaria Baldwin Opens Up About the Embarrassing Moments on Her ‘DWTS’ Journey

    Harvest Fest 2025 – yadkinripple.com

    Celebrate the Bounty: Harvest Fest 2025 is Coming!

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Aurora police hope to add facial recognition technology to crime-fighting tools – CBS News

    Aurora Police Aim to Boost Crime-Fighting with New Facial Recognition Technology

    Autonomous Solutions shows off cutting-edge technology for the public – Cache Valley Daily

    Autonomous Solutions Unveils Cutting-Edge Technology for the Public

    Amazon to Pay $2.5 Billion in Prime Membership Settlement – The New York Times

    Amazon to Pay $2.5 Billion in Prime Membership Settlement – The New York Times

    What are we really gaining from technology? – Fast Company

    What Are We Really Gaining from Technology?

    TOMI Environmental Solutions, Inc. Expands SteraMist iHP Technology Services in Healthcare Sector with New Provider Partnership – Quiver Quantitative

    TOMI Environmental Solutions Accelerates SteraMist iHP Technology Expansion in Healthcare with New Provider Partnership

    Indiana County Technology Center’s Joint Operating Committee looks to the future as program plans began to take shape – Indiana Gazette Online

    Indiana County Technology Center’s Joint Operating Committee Charts an Exciting Path Forward as New Program Plans Take Shape

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

How to run an LLM on your PC, not in the cloud, in less than 10 minutes

March 17, 2024
in Technology
How to run an LLM on your PC, not in the cloud, in less than 10 minutes
Share on FacebookShare on Twitter

Hands On With all the talk of massive machine-learning training clusters and AI PCs you’d be forgiven for thinking you need some kind of special hardware to play with text-and-code-generating large language models (LLMs) at home.

In reality, there’s a good chance the desktop system you’re reading this on is more than capable of running a wide range of LLMs, including chat bots like Mistral or source code generators like Codellama.

In fact, with openly available tools like Ollama, LM Suite, and Llama.cpp, it’s relatively easy to get these models running on your system.

In the interest of simplicity and cross-platform compatibility, we’re going to be looking at Ollama, which once installed works more or less the same across Windows, Linux, and Macs.

A word on performance, compatibility, and AMD GPU support:

In general, large language models like Mistral or Llama 2 run best with dedicated accelerators. There’s a reason datacenter operators are buying and deploying GPUs in clusters of 10,000 or more, though you’ll need the merest fraction of such resources.

Ollama offers native support for Nvidia and Apple’s M-series GPUs. Nvidia GPUs with at least 4GB of memory should work. We tested with a 12GB RTX 3060, though we recommend at least 16GB of memory for M-series Macs.

Linux users will want Nvidia’s latest proprietary driver and probably the CUDA binaries installed first. There’s more information on setting that up here.

If you’re rocking a Radeon 7000-series GPU or newer, AMD has a full guide on getting an LLM running on your system, which you can find here.

The good news is, if you don’t have a supported graphics card, Ollama will still run on an AVX2-compatible CPU, although a whole lot slower than if you had a supported GPU. And while 16GB of memory is recommended, you may be able to get by with less by opting for a quantized model — more on that in a minute.

Installing Ollama

Installing Ollama is pretty straight forward, regardless of your base operating system. It’s open source, which you can check out here.

For those running Windows or Mac OS, head over ollama.com and download and install it like any other application.

For those running Linux, it’s even simpler: Just run this one liner — you can find manual installation instructions here, if you want them — and you’re off to the races.

curl -fsSL https://ollama.com/install.sh | sh

Installing your first model

Regardless of your operating system, working with Ollama is largely the same. Ollama recommends starting with Llama 2 7B, a seven-billion-parameter transformer-based neural network, but for this guide we’ll be taking a look at Mistral 7B since it’s pretty capable and been the source of some controversy in recent weeks.

Start by opening PowerShell or a terminal emulator and executing the following command to download and start the model in an interactive chat mode.

ollama run mistral

Upon download, you’ll be dropped in to a chat prompt where you can start interacting with the model, just like ChatGPT, Copilot, or Google Gemini.

LLMs, like Mistral 7B, run surprisingly well on this 2-year-old M1 Max MacBook Pro

LLMs, like Mistral 7B, run surprisingly well on this 2-year-old M1 Max MacBook Pro – Click to enlarge

If you don’t get anything, you may need to launch Ollama from the start menu on Windows or applications folder on Mac first.

Models, tags, and quantization

Mistal 7B is just one of several LLMs, including other versions of the model, that are accessible using Ollama. You can find the full list, along with instructions for running each here, but the general syntax goes something like this:

ollama run model-name:model-tag

Model-tags are used to specify which version of the model you’d like to download. If you leave it off, Ollama assume you want the latest version. In our experience, this tends to be a 4-bit quantized version of the model.

If, for example, you wanted to run Meta’s Llama2 7B at FP16, it’d look like this:

ollama run llama2:7b-chat-fp16

But before you try that, you might want to double check your system has enough memory. Our previous example with Mistral used 4-bit quantization, which means the model needs half a gigabyte of memory for every 1 billion parameters. And don’t forget: It has seven billion parameters.

Quantization is a technique used to compress the model by converting its weights and activations to a lower precision. This allows Mistral 7B to run within 4GB of GPU or system RAM, usually with minimal sacrifice in quality of the output, though your mileage may vary.

The Llama 2 7B example used above runs at half precision (FP16). As a result, you’d actually need 2GB of memory per billion parameters, which in this case works out to just over 14GB. Unless you’ve got a newer GPU with 16GB or more of vRAM, you may not have enough resources to run the model at that precision.

Managing Ollama

Managing, updating, and removing installed models using Ollama should feel right at home for anyone who’s used things like the Docker CLI before.

In this section we’ll go over a few of the more common tasks you might want to execute.

To get a list of installed models run:

ollama list

To remove a model, you’d run:

ollama rm model-name:model-tag

To pull or update an existing model, run:

ollama pull model-name:model-tag

Additional Ollama commands can be found by running:

ollama –help

As we noted earlier, Ollama is just one of many frameworks for running and testing local LLMs. If you run in to trouble with this one, you may find more luck with others. And no, an AI did not write this.

The Register aims to bring you more on utilizing LLMs in the near future, so be sure to share your burning AI PC questions in the comments section. And don’t forget about supply chain security. ®

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : The Register – https://go.theregister.com/feed/www.theregister.com/2024/03/17/ai_pc_local_llm/

Tags: cloudMinutestechnology
Previous Post

In the rush to build AI apps, please, please don’t leave security behind

Next Post

Stardew Valley’s 1.6 update gives the biggest gift for serial organizers: foodstuff that’s “colored based on the ingredient”

New Delhi 2025: Preview, stars and how to watch the World Championships – Paralympic.org

New Delhi 2025 World Championships: Top Athletes to Watch and How You Can Catch the Action

September 27, 2025
Stock market exodus to Wall Street hits 20-year high – EL PAÍS English

Stock market exodus to Wall Street hits 20-year high – EL PAÍS English

September 27, 2025
October Prime Day TV Deals: Elevate Your Entertainment Space With These Early Savings – CNET

October Prime Day TV Deals: Upgrade Your Entertainment Space with These Early Savings

September 27, 2025
Georgia’s Medicaid work requirement program spent twice as much on administrative costs as on health care, GAO says – North Carolina Health News

Georgia’s Medicaid Work Requirement Program Spent Twice as Much on Administration as on Patient Care, GAO Finds

September 27, 2025
TGIF: Ian Donnis’ Rhode Island politics roundup for Sept. 26, 2025 – The Public’s Radio

TGIF: Ian Donnis’ Rhode Island politics roundup for Sept. 26, 2025 – The Public’s Radio

September 27, 2025
City Parks Initiative Launches Ecological Tracker for Bond Projects – Citizen Portal AI

Revolutionary Ecological Tracker Unveiled to Transform Monitoring of City Parks Bond Projects

September 27, 2025
Award-winning science writer leads student discussions at Eckerd – theonlinecurrent.com

Award-Winning Science Writer Inspires Student Discussions at Eckerd College

September 27, 2025
Human Head Transplants: Where the Science Stands, and Why the Ethics Are So Complicated – Discover Magazine

Human Head Transplants: The Science Behind the Procedure and the Complex Ethical Debate

September 27, 2025
New lifestyle brand TENŌRE set to open flagship store in Waikīkī – KITV

New lifestyle brand TENŌRE set to open flagship store in Waikīkī – KITV

September 27, 2025
Aurora police hope to add facial recognition technology to crime-fighting tools – CBS News

Aurora Police Aim to Boost Crime-Fighting with New Facial Recognition Technology

September 27, 2025

Categories

Archives

September 2025
M T W T F S S
1234567
891011121314
15161718192021
22232425262728
2930  
« Aug    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (839)
  • Economy (860)
  • Entertainment (21,734)
  • General (17,270)
  • Health (9,903)
  • Lifestyle (872)
  • News (22,149)
  • People (861)
  • Politics (870)
  • Science (16,069)
  • Sports (21,359)
  • Technology (15,842)
  • World (842)

Recent News

New Delhi 2025: Preview, stars and how to watch the World Championships – Paralympic.org

New Delhi 2025 World Championships: Top Athletes to Watch and How You Can Catch the Action

September 27, 2025
Stock market exodus to Wall Street hits 20-year high – EL PAÍS English

Stock market exodus to Wall Street hits 20-year high – EL PAÍS English

September 27, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version