* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Friday, August 1, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Sens. Blackburn, Warnock introduce CREATE Act to provide tax relief to music creators – Yahoo Home

    Sens. Blackburn and Warnock Launch CREATE Act to Deliver Tax Relief for Music Creators

    That’s (Political) Entertainment: When Theatre Meets Politics

    Future Script: How Generative AI Is Changing Collective Bargaining in the Entertainment Industry – Jackson Lewis

    Future Script: How Generative AI Is Transforming Collective Bargaining in Entertainment

    The SBA’s live-entertainment bailout was supposed to end two years ago. We still don’t know how $1.5 billion was spent. – Yahoo Home

    $1.5 Billion Live-Entertainment Bailout: Two Years Later, Where Did the Money Go?

    Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, Boyd – CDC Gaming

    Top Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, and Boyd Take Center Stage

    Micro wrestling coming to NE Ohio – Cleveland.com

    Get Ready, NE Ohio: Micro Wrestling Is Making Its Exciting Debut!

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Cognizant Technology Solutions Corp (CTSH) Q2 2025 Earnings Call Highlights: Strong Revenue … – Yahoo.co

    Cognizant Q2 2025 Earnings: Impressive Revenue Growth and Key Takeaways

    Revving Up The U.S. Technology Engine – Forbes

    Revving Up The U.S. Technology Engine – Forbes

    More than just a hockey player – Rochester Institute of Technology Athletics

    Beyond the Ice: The Inspiring Journey of a Remarkable Athlete from Rochester Institute of Technology

    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    AI’s race in the dark with China – Axios

    The High-Stakes AI Race: Innovation and Competition in the Shadows

    Eagle Unveils Revolutionary X-Ray Technology at Pack Expo

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Sens. Blackburn, Warnock introduce CREATE Act to provide tax relief to music creators – Yahoo Home

    Sens. Blackburn and Warnock Launch CREATE Act to Deliver Tax Relief for Music Creators

    That’s (Political) Entertainment: When Theatre Meets Politics

    Future Script: How Generative AI Is Changing Collective Bargaining in the Entertainment Industry – Jackson Lewis

    Future Script: How Generative AI Is Transforming Collective Bargaining in Entertainment

    The SBA’s live-entertainment bailout was supposed to end two years ago. We still don’t know how $1.5 billion was spent. – Yahoo Home

    $1.5 Billion Live-Entertainment Bailout: Two Years Later, Where Did the Money Go?

    Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, Boyd – CDC Gaming

    Top Wall Street Bets: Caesars, Golden Entertainment, Churchill Downs, GLPI, and Boyd Take Center Stage

    Micro wrestling coming to NE Ohio – Cleveland.com

    Get Ready, NE Ohio: Micro Wrestling Is Making Its Exciting Debut!

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Cognizant Technology Solutions Corp (CTSH) Q2 2025 Earnings Call Highlights: Strong Revenue … – Yahoo.co

    Cognizant Q2 2025 Earnings: Impressive Revenue Growth and Key Takeaways

    Revving Up The U.S. Technology Engine – Forbes

    Revving Up The U.S. Technology Engine – Forbes

    More than just a hockey player – Rochester Institute of Technology Athletics

    Beyond the Ice: The Inspiring Journey of a Remarkable Athlete from Rochester Institute of Technology

    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    Smart Logistics in Warehousing – From Legacy Protocols to Green IoT – How Technology Is Reshaping the Sustainable Supply Chain – Logistics Viewpoints –

    AI’s race in the dark with China – Axios

    The High-Stakes AI Race: Innovation and Competition in the Shadows

    Eagle Unveils Revolutionary X-Ray Technology at Pack Expo

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

Dependencies Belong in Version Control

November 26, 2023
in Technology
Dependencies Belong in Version Control
Share on FacebookShare on Twitter

I believe that all project dependencies belong in version control. Source code, binary assets, third-party
libraries, and even compiler toolchains. Everything.

The process of building any project should be trivial. Clone repo, invoke build command, and that’s it. It
shouldn’t require a complex configure script, downloading Strawberry Perl, installing Conda, or any of that
bullshit.

Infact I’ll go one step further. A user should be able to perform a clean OS install, download a zip of master,
disconnect from the internet, and build. The build process shouldn’t require installing any extra tools or
content. If it’s something the build needs then it belongs in version control.

Your gut reaction may be revulsion. That’s it not possible. Or that it’s unreasonable.

You’re not totally wrong. If you’re using Git for version control then committing ten gigabytes of
cross-platform compiler toolchains is infeasible.

That doesn’t change my claim. Dependencies do belong in version control. Even if it’s not practical today due
to Git’s limitations. More on that later.

Why do dependencies belong in version control? I’ll give a few reasons.

Usability
Reliability
Reproducibility
Sustainability

Committing dependencies makes projects trivial to build and run. I have regularly failed to build open source
projects and given up in a fit of frustrated rage.

My background is C++ gamedev. C++ infamously doesn’t have a standard build system. Which means every project
has it’s own bullshit build system, project generator, dependency manager, scripting runtimes, etc.

ML and GenAI projects are a god damned nightmare to build. They’re so terrible to build that there are
countless meta-projects that exists solely to provide one-click installers (example: EasyDiffusion). These installers are fragile and
sometimes need to be run several times to succeed.

Commit your dependencies and everything “just works”. My extreme frustration with trying, and failing, to build
open source projects is what inspired this post.

Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies
and that will never happen.

There’s a whole class of problems that simply disappear when depdendencies are committed. Builds won’t break
because of an OS update. Network errors don’t exist. You eliminate “works on my machine” issues because someone
didn’t have the right version of CUDA installed.

Builds are much easier to reproduce when version control contains everything. Great build systems are hermetic
and allow for determistic builds. This is only possible when your build doesn’t depend on your system
environment.

Lockfiles are only a partial solution to reproducibility. Docker images are a poor man’s VCS.

Committing dependencies makes it trivial to recreate old builds. God help you if you try to build a webdev
stack from 2013.

In video games it’s not uncommon to release old games on new platforms. These games can easily be 10 or 20
years old. How many modern projects will be easy to build in 20 years? Hell, how many will be easy to build in
5?

Commit your dependencies and ancient code bases will be as easy to rebuild as possible. Although new platforms
will require new code, of course.

To prove that this isn’t completely crazy I built a proof of life C++ demo. My program is exceedingly
simple:

#include

int main() {
fmt::print(“Hello world from C++ 👋n”);
fmt::print(“goodbye cruel world from C++ ☠️n”);
return 0;
}

The folder structure looks like this:

root
sample_cpp_app
– main.cpp
thirdparty
fmt (3 MB)
toolchains
win
cmake (106 MB)
LLVM (2.5 GB)
mingw64 (577 MB)
ninja (570 KB)
Python311 (20.5 MB)
– CMakeLists.txt
– build.bat
– build.py

The toolchains folder contains five dependencies – CMake, LLVM, Ming64, Ninja, and Python 3.11. Their combined size is 3.19 gigabytes. No effort was made to trim these folders down in size.

The build.bat file nukes all environment variables and sets PATH=C:WindowsSystem32;. This ensures only the included toolchains are used to compile.

The end result is a C++ project that “just works”.

Here’s where it gets fun. I wrote a Python that script that scans the directory for “last file accessed time” to track “touched files”. This let’s me check how many toolchain files are actually needed by the build. It produces this output:

Checking initial file access times… 🥸👨‍🔬🔬

Building… 👷‍♂️💪🛠️
Compile success! 😁

Checking new file access times… 🥸👨‍🔬🔬
File Access Stats
Touched 508 files. Total Size: 272.00 MB
Untouched 23138 files. Total Size: 2.93 GB
Touched 2.1% of files
Touched 8.3% of bytes

Running program…
Target exe: c:tempcodetoolchain_vcsbinmain.exe

Hello world from C++ 👋
goodbye cruel world from C++ ☠️

Built and ran successfully! 😍

Well will you look at that!

Despite committing 3 gigabytes of toolchains we only actually needed a mere 272 megabytes. Well under 10%! Even better we touched just 2.0% of repo files.

The largest files touched were:

clang++.exe [116.04 MB]
ld.lld.exe [86.05 MB]
llvm-ar.exe [28.97 MB]
cmake.exe [11.26 MB]
libgcc.a [5.79 MB]
libstdc++.dll.a [5.32 MB]
libmsvcrt.a [2.00 MB]
libstdc++-6.dll [1.93 MB]
libkernel32.a [1.27 MB]

My key takeaway is this: toolchain file sizes are tractable for version control if you can trim the fat.

This sparks my joy. Imagine cloning a repo, clicking build, and having it just work. What a wonderful and delightful world that would be!

I’d like to paint a small dream for what I will call Next Gen Version Control Software (NGVCS). This is my vision for a Git/Perforce successor. Here are some of the key featurs I want NGVCS to have:

virtual file system to fetch only files a user touches
copy-on-write file storage
system cache for NGVCS files

Let’s pretend for a moment that every open source project commits their dependencies. Each one contains a full copy of Python, Cuda, Clang, MSVC, libraries, etc. What would happen?

First, the user clones a random GenAI repo. This is near instantaneous as files are not prefetched. The user then invokes the build script. As files are accessed they’re downloaded. The very first build may download a few hundred megabytes of data. Notably it does NOT download the entire repo. If the user is on Linux it won’t download any binaries for macOS or Windows.

Second, the user clones another GenAI repo and builds. Does this need to re-download gigabytes of duplicated toolchain content? No! Both projects use NGVCS which has a system wide file cache. Since we’re also using a copy-on-write file system these files instantly materialize in the second repo at zero cost.

The end result is beautiful. Every project is trivial to fetch, build, and run. And users only have to download the minimum set of files to do so.

Hopefully I’ve convinced some of you that committing dependencies is at least a good idea in an ideal world.

Now let’s consider the real world and a few counter arguments.

Unfortunately I must admit that committing dependencies is not be practical today. The problem is Git. One of my unpopular opinions is that Git isn’t very good. Among its many sins is terrible support for large files and large repositories.

The root issue is that Git’s architecture and default behavior expects all users to have a full copy of the entire repo history. Which means every version of every binary toolchain for every platform. Yikes!

There are various work arounds – Git LFS, Git Submodules, shallow clones, partial clones, etc. The problem is these aren’t first-class features. They are, imho, second-class hacks. 😓

In theory Git could be updated to more properly support large projects. I believe Git should be shallow and partial by default. Almost all software projects are defacto centralized. Needing full history isn’t the default, it’s an edge case. Users should opt-in to full history only if they need it.

An alternative to committing dependencies is to use containers. If you build out of a container you get most, if not all, of the benefits. You can even maintain an archive of docker images that reliably re-build tagged releases.

Congrats, you’re now using Docker as your VCS!

My snarky opinion is that Docker and friends primarily exist because modern build systems are so god damned fragile that the only way to reliably build and deploy is to create a full OS image. This is insanity!

Containers shouldn’t be required simply to build and run projects. It’s embarassing that’s the world we live in.

Not all dependencies are authorized for redistribution. I believe MSVC and XCode both disallow redistribution of compiler toolchains? Game consoles like Sony PlayStation and Nintendo Switch don’t publicly release headers, libs, or compilers.

This is mostly ok. If you’re working on a console project then you’re already working on a closed source project. Developers already use permission controls to gate access.

The lack of redistribution rights for “normal” toolchains is annoying. However permissive options are available. If committing dependencies becomes common practice then I think it’s likely that toolchain licenses will update to accomdate.

Committing library dependencies to version control means they need to be updated. If you have lots of repos to update this could be a moderate pain in the ass.

This is also the opposite of how Linux works. In Linux land you use a hot mess of system libraries sprinkled chaotically across the search path. That way when there is a security fix you update a single .so (or three) and your system is safe.

I think this is largely a non-issue. Are you building and running your services out of Docker? Do you have a fleet of machines? Do you have lockfiles? Do you compile any thirdparty libraries from source? If the answer to any of these questions is yes, and it is, then you already have a non-trivial procedure to apply security fixes.

Committing dependencies to VCS doesn’t make security updates much harder. In fact, having a monorepo source of truth can make things easier!

One of Git’s claims to fame is its distributed nature. At long last developers can commit work from an internetless cafe or airplane!

My NGVCS dream implies defacto centralization. Especially for large projects with large histories. Does that mean an internet connection is required? Absolutely not! Even Perforce, the King of centralized VCS, supports offline mode. Git continues to function locally even when working with shallow and partial Git clones.

Offline mode and decentralization are independent concepts. I don’t know why so many people get this wrong.

Do I really think that every library, such as fmt, should commit gigabytes of compilers to version control?

That’s a good question. For languages like Rust which have a universal build system probably not. For languages like C++ and Python maybe yes! It’d be a hell of a lot easier to contribute to open source projects if step 0 wasn’t “spend 8 hours configuring environment to build”.

For libraries the answer may be “it depends”. For executables I think the answer is “yes, commit everything”.

NGVCS is obviously a dream. It doesn’t exist today. Actually, that’s not quite true. This is exactly how Google and Meta operate today. Infact numerous large companies have custom NGVCS equivalents for internal use. Unfortunately there isn’t a good solution in the public sphere.

Is committing dependencies reasonable for Git users today? The answer is… almost? It’s at least closer than most people realize! A full Python deployment is merely tens to hundreds of megabytes. Clang is only a few gigabytes. A 2TB SSD is only $100. I would enthusiastically donate a few gigabytes of hard drive space in exchange for builds that “just work”.

Committing dependencies to Git might be possible to do cleanly today with shallow, sparse, and LFS clones. Maybe. It’d be great if you could run git clone –depth=1 –sparse=windows. Maybe someday.

I strongly believe that dependencies belong in version control. I believe it is “The Right Thing”. There are significant benefits to usability, reliability, reproducibility, sustainability, and more.

Committing all dependencies to a Git repo may be more practical than you realize. The actual file size is very reasonable.

Improvements to VCS software can allow repos to commit cross-platform dependencies while allowing users to download the bare minimum amount of content. It’s the best of everything.

I hope that I have convinced you that committing dependencies and toolchains is “The Right Thing”. I hope that version control systems evolve to accomodate this as a best practice.

Thank you.

If you read it this far, thank you! Here are some extra thoughts I wanted to share but couldn’t squeeze into the main article.

The sample project can be downloaded via Dropbox as a 636mb .7zip file. It should be trivial to download and build! Linux and macOS toolchains aren’t included because I only have a Windows machine to test on. It’s not on GitHub because they have an unnecessary file size limit.

My dream NGVCS has first class support for all the features I mentioned and more.

Git LFS is, imho, a hacky, second class citizen. It works and people use it. But it requires a bunch of extra effort and running extra commands.

I have a related rant that not only should all dependencies be checked into the build system, but that deployments should also include all dependencies. Yes, deploy 2gb+ of CUDA dlls so your exe will reliably run. No, don’t force me to use Docker to run your simple Python project.

There are a handful of interesting Git alternatives in the pipeline.

Jujutsu – Git but better
Pijul – Somewhat academic patch-based VCS
Sapling – Open source version of Meta’s VCS. Not fully usable outside of Meta infra.
Xethub – Git at 100Tb scale to support massive ML models

Git isn’t going to be replaced anytime soon, unfortunately. But there are a variety of projects exploring different ideas. VCS is far from a solved problem. Be open minded!

Package managers are not necessarily a silver bullet. Rust’s Cargo is pretty good. NPM is fine I guess. Meanwhile Python’s package ecosystem is an absolute disaster. There may be a compile-time vs run-time distinction here.

A good package manager is a decent solution. However package managers exist on a largely per-language basis. And sometimes per-platform. Committing dependencies is a guaranteed good solution for all languages on all platforms.

Polyglot projects that involve multiple languages need multiple package managers. Yuck.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Hacker News – https://www.forrestthewoods.com/blog/dependencies-belong-in-version-control/

Tags: 'belong'Dependenciestechnology
Previous Post

Svix (YC W21) Is Hiring a Tech Lead (US Remote)

Next Post

Tokyo Regains Spot as Top APAC Investment Destination in ULI Poll

‘Enough is enough,’ say Global South bishops in climate letter – U.S. Catholic

Enough is Enough’: Global South Bishops Deliver Urgent Plea for Bold Climate Action

August 1, 2025
Science Central names new executive director – 21Alive

Science Central Welcomes New Executive Director to Lead Exciting Future

August 1, 2025
Citizen Scientists Are Accelerating Ecology Research, Study Suggests – The New York Times

Citizen Scientists Are Accelerating Ecology Research, Study Suggests – The New York Times

August 1, 2025
I tried Beyoncé’s brutal pre-Coachella vegan diet—and here’s the unfiltered truth – VegOut

I tried Beyoncé’s brutal pre-Coachella vegan diet—and here’s the unfiltered truth – VegOut

August 1, 2025
U17 World Wrestling Championships: Day 3 Live Results – United World Wrestling

U17 World Wrestling Championships Day 3: Live Action and Results

August 1, 2025
New GDP data leads Trump to change his mind about blaming Biden for the economy – MSNBC News

New GDP Figures Force Trump to Rethink Blaming Biden for Economic Woes

August 1, 2025
‘Soft summer’ in Las Vegas, says one of Strip’s largest operators – Las Vegas Review-Journal

Las Vegas Braces for a “Soft Summer,” Warns Leading Strip Operator

August 1, 2025
‘Industry turf war’ or ‘public health crisis’? Federal push to regulate kratom fuels debate in California. – Politico

Is California at a Crossroads: Industry Turf War or Urgent Public Health Crisis in the Kratom Debate?

August 1, 2025
Senate bill to ban lawmaker stock trading moves step forward – and Trump turns on Hawley – CNN

Senate Bill to Ban Lawmaker Stock Trading Gains Momentum Amid Trump’s Criticism of Hawley

August 1, 2025
Hungary to Gain Access to Cutting-Edge US Nuclear Technology – Hungarian Conservative

Hungary Poised to Unleash Cutting-Edge US Nuclear Technology

July 31, 2025

Categories

Archives

August 2025
MTWTFSS
 123
45678910
11121314151617
18192021222324
25262728293031
« Jul    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (748)
  • Economy (773)
  • Entertainment (21,652)
  • General (16,220)
  • Health (9,810)
  • Lifestyle (781)
  • News (22,149)
  • People (775)
  • Politics (782)
  • Science (15,986)
  • Sports (21,269)
  • Technology (15,751)
  • World (756)

Recent News

‘Enough is enough,’ say Global South bishops in climate letter – U.S. Catholic

Enough is Enough’: Global South Bishops Deliver Urgent Plea for Bold Climate Action

August 1, 2025
Science Central names new executive director – 21Alive

Science Central Welcomes New Executive Director to Lead Exciting Future

August 1, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version