Shipyard.rs: Private Crates as Easy as Using Cargo

⇒ Rust enjoys a first class package manager for open source crates.

⇒ Shipyard.rs is working to bring that same seamless experience to private code.

If you had to pick just one reason why Rust has been the most loved programming language in Stack Overflow's annual survey for six years running, you could do a lot worse than its beloved package manager, Cargo.

It's not just that adding dependencies is effortless, to the point that devs, oft-forgetting the days when code reuse meant hoarding a bloom filter, concurrent HashMap, and bespoke circular buffer in the "vendor" dir, wonder if Cargo might make it a little bit too easy, encouraging dreaded "micro-dependencies."

It's not just that the semantic versioning-based dependency resolution gives you fine-grained control over upgrading, with hardly any effort, such that cargo upgrade is generally a worry-free experience.

It's not just that many versions of the same crate can coexist peacefully across the same codebase, unlike, say, Python, where One True Version is enforced for every package, and circular dependency hell often lurks around the bend.

(Speaking of which, It's not just that code utilizing competing Rust editions can be used entirely interchangeably, skipping effortlessly over chasms that caused decades of pain in other ecosystems.)

And it's not just that extremely nice, generated documentation comes for free just by writing (and commenting) the code, and the docs for every published crate version are automatically generated and published at docs.rs, providing a hugely valuable information resource for anyone connected to the Internet.

No, it's all of those things together, the combination of which is a code reuse experience that may be unmatched by any other programming language.

And yet. For Rust developers working on larger, proprietary codebases, code reuse can often be a "Tale of Two Cities". Adding public, open-source dependencies from Crates.io is a breeze, of course. But private crates are relegated to a second-tier experience, suffering from many of the same problems that Cargo solves so wonderfully.

Cargo Dependencies: An Evolutionary Account

It all starts with path-based dependencies as a quickie solution when you're prototyping some new project:

# Cargo.toml

[dependencies.my-private-crate]
path = "../some-local-dir"

Instead of a range of semantic versions, from which Cargo will pull the best available candidate, we have instead instructed Cargo to use whatever happens to currently reside in ../some-local-dir.

It's a completely fine, even useful technique, when it's one crate. But soon it's five crates, then ten. Hope you're good at shell scripting! It becomes an ordeal to manage the state of a growing set of local directories.

(True confessions: I once worked on a project with 40 (!) path-based dependencies. The justfile to wrangle all those was ... involved. This was circa 2018, back then we just didn't know any better.)

Path-based dependencies are a pain. There must be a better way ... and there is!

Git-Based Dependencies

# Cargo.toml

[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "master"

Much better! Now Cargo can go fetch the code on its own if we don't have it yet. And when we push improvements to my-private-crate to origin, we can get them in the other crate with cargo update, too!

Except, very soon after pushing a breaking change to my-private-crate, you realize -- "master" isn't a version. Like path-based dependencies, "master" is just whatever's been pushed to a git branch. Soon you find yourself avoiding cargo update, knowing that you're not ready to handle the breaking changes that have been pushed -- or, in the worse case, not realizing it until your code breaks, requiring work to undo.

There must be a better way ... and there is!

Version Branches

# Cargo.toml

[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "v0.1.x"

Now we no longer use a single branch like a bunch of barbarians -- we keep separate histories, e.g. v0.1.x and v0.2.x, so that progress can steam ahead on the newest branch (v0.2.x ), all while backporting important fixes to the v0.1.x branch for those ancient, lolly-gagging crates still falling far behind the latest release.

A few weeks go by, and maybe even months. But pretty soon you realize: it's a lot of work to keep all these parallel histories intact, where whatever exists on the branch is a de facto version release. Git maintenance busy work grows tedious. You struggle to remember which branch has what bug fix, which has been pushed, which hasn't.

The work piles up, sufficiently so to prompt a nagging, pestering thought: is this actually more work than it would take to set up a proper registry server -- you know, the actual right and proper way to handle this problem?

I can't be certain what your answer will be. But mine was -- and is -- emphatically "yes!"

Private Registry

A private registry server solved problems I had forgotten -- over years -- I even had, problems I had slowly grown used to, Stockholm Syndrome-like, all to avoid the initial setup costs of a registry server.

But, once a registry server was in place, a truth emerged brightly from the fog, suddenly obvious:

Yes, of course a registry works wonderfully. It works ... just like ... Cargo does!

It's the same awesome experience you already had for open source crates, but now also for your private crates. It was a surprisingly welcome change, but one made only after I'd endured path-based and git-based deps for literally years.

This experience -- my experience -- is the primary motivation behind Shipyard.rs: to make private crates as easy as Cargo, so that Rust devs everywhere can enjoy the same seamless experience they do for open source crates on their proprietary or non-public crates.

- Jonathan Strong