Shipyard.rs: Private Crates as Easy as Using Cargo

⇒ Rust enjoys a first class package manager for open source crates.

⇒ Shipyard.rs is working to bring that same seamless experience to private code.

If you had to pick just one reason why Rust has been the most loved programming language in Stack Overflow's annual survey for six years running, you could do a lot worse than its beloved package manager, Cargo.

It's not just that adding dependencies is effortless, to the point that devs, oft-forgetting the days when code reuse meant hoarding a bloom filter, concurrent HashMap, and bespoke circular buffer in the "vendor" dir, wonder if Cargo might make it a little bit too easy, encouraging dreaded "micro-dependencies."

It's not just that the semantic versioning-based dependency resolution gives you fine-grained control over upgrading, with hardly any effort, such that cargo upgrade is generally a worry-free experience.

It's not just that many versions of the same crate can coexist peacefully across the same codebase, unlike, say, Python, where One True Version is enforced for every package, and circular dependency hell often lurks around the bend.

(Speaking of which, It's not just that code utilizing competing Rust editions can be used entirely interchangeably, skipping effortlessly over chasms that caused decades of pain in other ecosystems.)

And it's not just that extremely nice, generated documentation comes for free just by writing (and commenting) the code, and the docs for every published crate version are automatically generated and published at docs.rs, providing a hugely valuable information resource for anyone connected to the Internet.

No, it's all of those things together, the combination of which is a code reuse experience that may be unmatched by any other programming language.

And yet. For Rust developers working on larger, proprietary codebases, code reuse can often be a "Tale of Two Cities". Adding public, open-source dependencies from Crates.io is a breeze, of course. But private crates are relegated to a second-tier experience, suffering from many of the same problems that Cargo solves so wonderfully.

Cargo Dependencies: An Evolutionary Account

It all starts with path-based dependencies as a quickie solution when you're prototyping some new project:

# Cargo.toml

[dependencies.my-private-crate]
path = "../some-local-dir"

Instead of a range of semantic versions, from which Cargo will pull the best available candidate, we have instead instructed Cargo to use whatever happens to currently reside in ../some-local-dir.

It's a completely fine, even useful technique, when it's one crate. But soon it's five crates, then ten. Hope you're good at shell scripting! It becomes an ordeal to manage the state of a growing set of local directories.

(True confessions: I once worked on a project with 40 (!) path-based dependencies. The justfile to wrangle all those was ... involved. This was circa 2018, back then we just didn't know any better.)

Path-based dependencies are a pain. There must be a better way ... and there is!

Git-Based Dependencies

# Cargo.toml

[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "master"

Much better! Now Cargo can go fetch the code on its own if we don't have it yet. And when we push improvements to my-private-crate to origin, we can get them in the other crate with cargo update, too!

Except, very soon after pushing a breaking change to my-private-crate, you realize -- "master" isn't a version. Like path-based dependencies, "master" is just whatever's been pushed to a git branch. Soon you find yourself avoiding cargo update, knowing that you're not ready to handle the breaking changes that have been pushed -- or, in the worse case, not realizing it until your code breaks, requiring work to undo.

There must be a better way ... and there is!

Version Branches

# Cargo.toml

[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "v0.1.x"

Now we no longer use a single branch like a bunch of barbarians -- we keep separate histories, e.g. v0.1.x and v0.2.x, so that progress can steam ahead on the newest branch (v0.2.x ), all while backporting important fixes to the v0.1.x branch for those ancient, lolly-gagging crates still falling far behind the latest release.

A few weeks go by, and maybe even months. But pretty soon you realize: it's a lot of work to keep all these parallel histories intact, where whatever exists on the branch is a de facto version release. Git maintenance busy work grows tedious. You struggle to remember which branch has what bug fix, which has been pushed, which hasn't.

The work piles up, sufficiently so to prompt a nagging, pestering thought: is this actually more work than it would take to set up a proper registry server -- you know, the actual right and proper way to handle this problem?

I can't be certain what your answer will be. But mine was -- and is -- emphatically "yes!"

Private Registry

A private registry server solved problems I had forgotten -- over years -- I even had, problems I had slowly grown used to, Stockholm Syndrome-like, all to avoid the initial setup costs of a registry server.

But, once a registry server was in place, a truth emerged brightly from the fog, suddenly obvious:

Yes, of course a registry works wonderfully. It works ... just like ... Cargo does!

It's the same awesome experience you already had for open source crates, but now also for your private crates. It was a surprisingly welcome change, but one made only after I'd endured path-based and git-based deps for literally years.

This experience -- my experience -- is the primary motivation behind Shipyard.rs: to make private crates as easy as Cargo, so that Rust devs everywhere can enjoy the same seamless experience they do for open source crates on their proprietary or non-public crates.

- Jonathan Strong

Private Crate Models

When specifying Rust dependencies in a crate's Cargo.toml file, the normal case is adding an open source crate hosted at Crates.io:

# Cargo.toml

[dependencies]
uuid = "1.2"
rand = "*"
serde_json = "1.0.87"

These dependencies, which merely require a name and version range, are actually a form of "registry" dependency, with the registry (Crates.io) implied -- Cargo assumes Crates.io is the intended registry unless another registry is specified.

But for adding private crates as dependencies, there are several other approaches, each of which offer advantages and disadvantages, including:

• Path-based dependencies
• Git-based dependencies
• Private Registry Dependencies

Path-Based Dependencies

Path-based dependencies are the simplest way to add non-public crates as dependencies in your Cargo.toml:

# Cargo.toml
my-private-crate = { path = "../local-dir-with-my-private-crate" }

This technique is can be useful in situations where changes in two or more crates need to be evaluated together, because it provides the fastest feedback cycle for the change in one crate being "visible" in the a second crate which depends on the first.

However, path-based dependencies suffer from significant weaknesses for larger codebases, teams of developers, or other situations where fixed (published) versions and semantic versioning-based dependency resolution provides finer-grained control over upgrade paths.

Also, the more private crates you are using, the bigger a problem it is to keep the state of your local directories in sync and up to date with the latest changes. For example, you might use a script to overcome the need to manually enter each dependent crate directory and git pull, but if one of those directories has unstaged changes in it, you'll need to resolve that before you can build the crate you are (were) working on, which is annoying and, more importantly, poses a significant cost in terms of context switching.

Git-Based Dependencies

Git-based dependencies allow you to point a dependency to a git repo, including specifying a branch, ref or tag, and provide important benefits over path-based dependencies:

# Cargo.toml
[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "master"

One key benefit of git-based dependencies, compared to path-based dependencies, is that Cargo can fetch the source code from git on its own if it doesn't already have it. Also, cargo update will prompt the latest changes to be pulled from origin, providing a (relatively crude) form of acquiring updates.

Assuming, in the context of proprietary crates, that the git repository specified requires authentication, this method also poses more complex setup requirements compared to path-based dependencies. CI or other automated deployment tools can often present challenges for securely providing credentials to Cargo to be able to pull the code from the repository, for example.

"master" is not a version

The most significant drawback of git-based dependencies, when compared to using open source crates with Cargo, is being forced to choose between a "moving target" or a potentially stale fixed point in the code history.

For example, if a git dependency is anchored to a branch (say, "master"), whatever changes that are pushed to that branch will be pulled and used by Cargo, without any semantic versioning-based logic to determine whether those changes are breaking.

With a bigger team and many, frequent changes being pushed to primary branches, cargo update can become a pretty dangerous operation, at least when compared to the experience you get upgrading open source crates hosted on Crates.io. Often, this leads to a reluctance to perform cargo update and a growing gap between the versions in a crate's Cargo.lock compared to the newest available versions.

See also: Version Branches

Version Branches

Using version branches is a variation on git-based dependencies which entails keeping parallel, intact code revision histories in separate git branches, and anchoring the dependencies in Cargo.toml to those "version branches" instead of a "general" branch like "dev" or "master":

For instance, imagine there is a crate, my-private-crate, currently at version "0.1.7". You want to make a breaking change to the code, but allow another crate, private-crate-the-sequel to continue depending on the version "0.1.7" release.

A new branch will be created in the my-private-crate repository, i.e.

$ cd ~/src/my-private-crate
$ git checkout -b v0.2.x

Breaking changes to my-private-crate can be pushed to the v0.2.x branch, leaving the version "0.1.7" "release" intact on the v0.1.x branch. If there is an important bug- or hot-fix, it should be made to both v0.1.x and v0.2.x branches.

Now private-crate-the-sequel can depend on the old "0.1.7" version, while a third crate, private-crate-triple-threat, can depend on a newly "released" version "0.2.0-rc.1" of my-private-crate:

# Cargo.toml for private-crate-the-sequel

[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "0.1.x"

# Cargo.toml for private-crate-triple-threat

[dependencies.my-private-crate]
git = "ssh://my-repo.com/my-private-crate.git"
branch = "0.2.x" # <- note: newer branch

Allows Multiple Version Coexistence, at a Cost

The chief benefit of this technique is that different versions of a crate can be used simultaneously as dependencies, which is difficult or error-prone when anchoring the dependency to a "general" branch like "dev" or "master".

A downside of this approach is that it requires quite a bit of maintenance and care to cultivate useful version branches. Unlike publishing a crate version, which is a discrete, intentional act done at a moment when the code is verified (via cargo publish) to be in a good state, version branches are always subject to possible "errant" pushes, in which incomplete and/or non-compiling code is pushed to the branch, potentially leaving other crates themselves unable to build after pulling in those errant changes.

In general, my experience using this technique is, it's more work than setting up a private registry server, especially since the work never ends, yet works less well than a private registry in subtle but important ways.

Private Registry Server

A private registry server is "the Cadillac" of private crate reuse: a fully-realized solution that offers an experience parallel to seamless experience of adding open source crates hosted at Crates.io, but for your private code.

However, these benefits come at the cost of initial setup, configuration, and, possibly, ongoing server maintenance.

Private registries work like Crates.io, with crate versions published to the server, listed in a git-based metadata index, and available for download from the registry server.

Authentication: Keeping Proprietary Code Secure

A critical difference between how Crates.io works vs. Shipyard.rs is that Shipyard.rs has been carefully designed to require authentication both to publish crate versions to the registry server, as well as download published versions from the registry server.

In some cases, this creates additional configuration steps, e.g. to setup authentication to the crate index git repository, which is private, unlike the open source Crates.io registry index.

In other cases, Cargo is being actively improved to add authentication to categories of requests (namely crate downloads) that previously did not contemplate it.

Private Registry Dependencies: the Quick Version

An example of a private registry dependency:

# Cargo.toml

[dependencies]
krate-search = { version = "0.3", registry = "shipyard-rs" }

In this case, the registry (shipyard-rs) corresponds to an entry configured in ~/.cargo/config.toml (or other path in the Cargo configuration hierarchy).

Assuming authentication has been configured, Cargo will download this crate (krate-search) from the shipyard-rs registry server similar to how it downloads open source crates from the Crates.io registry server.

Open Source Registry Servers

There are several open source registry server projects, which can be used to self-host a private registry server.

Meuse

Meuse is an open source Rust registry server written in Clojure.

Shipyard.rs uses Meuse underneath the hood. It has proven to be a solid piece of code over time, in our estimation.

Potential downsides include the source code language, Clojure, which may not be familiar to Rust developers, as well as fairly expensive memory usage of JVM-based programs.

Other Projects

Other projects, which we are aware of but do not have extensive experience using, include:

• Ktra
• Alexandrie
• Estuary
• gitlab-cargo-shim: bare-bones, git-based registry server without Cargo compatibility

See also: rust-lang list of open source registry servers

Hosted Registry Services

In addition to open source registry server projects, there are several options for hosted registry services.

Shipyard.rs

Shipyard.rs provides a hosted private registry service designed for Rust developers.

Shipyard.rs is the only service that automatically generates (using rustdoc) and hosts crate documentation (aka docs.rs-like functionality for your private crates).

Pros:

• Cargo-compatible Registry Server (equivalent to crates.io)
• Only hosted service with automated Rustdoc builds (equivalent to docs.rs)
• Secure by default: end-to-end authentication, including crate downloads
• Integrated management of users, tokens, SSH keys
• Excellent value
• 100% Purpose-Built Service for Hosting Rust Crates

Cons:

• Relatively new service
• Rapidly iterating product with frequent feature releases

JFrog Artifactory

JFrog Artifactory, a comprehensive binary hosting service aimed at enterprise users, has added some functionality to host Rust crates to its software.

Pros:

• Hosting for many other binary artifacts other than Rust crates
• Ability to self-host, albeit at a steep price of up to $42k/year

Cons:

• No Rustdocs
• Some users have raised concerns about the service's authentication model as applied to its Rust implementation, which permits unauthenticated crate downloads
• Expensive

Cloudsmith

Cloudsmith offers comprehensive binary artifact hosting similar to JFrog Artifactory, which includes Rust functionality.

Pros:

• Hosting for many other binary artifacts other than Rust crates

Cons:

• No Rustdocs
• Expensive

See Also: rust-lang list of commercial registry server options

Semantic Versioning-Based Dependency Resolution

Cargo's semantic versioning-based dependency resolution is one of the most important parts of why it works so well as a package manager.

The below examples, from the Cargo Book, demonstrate how this dependency resolution logic works in concrete form.

1.2.3  :=  >=1.2.3, <2.0.0
1.2    :=  >=1.2.0, <2.0.0
1      :=  >=1.0.0, <2.0.0
0.2.3  :=  >=0.2.3, <0.3.0
0.2    :=  >=0.2.0, <0.3.0
0.0.3  :=  >=0.0.3, <0.0.4
0.0    :=  >=0.0.0, <0.1.0
0      :=  >=0.0.0, <1.0.0

This model enables robust, yet flexible version specification:

• Cargo will fetch upgrades beyond the literal stated version, but only up to a maximum range

• Versions ranges are variable by specificity. When a dependent crate's interface is tightly integrated into the code, a tight version range can be used, if the dependency is loosely integrated, a wider range can be used. This provides flexibility

• In practice, adherence to semantic versioning guidelines is consistent enough in the crates published to Crates.io that it provides meaningful tiers of change severity. In other words, You can generally rely on it.

Missing SemVer

Not all methods of specifying dependencies utilize the semantic versioning-based dependency resolution. For example, path-based and git-based dependencies do not have any built-in notion of version, so Cargo does not have any means of applying this logic to its upgrade decisions for those dependencies. This is a key drawback of path- and git-based dependencies that is sometimes neglected when considering the pros and cons of different approaches.

Fixed Crate Versions vs. Moving Targets

One thing that becomes evident after moving private crates to a registry server is how big of an advantage it is for a specific crate version, once published, to remain available at the registry server in perpetuity.

A lot of the problems that arise from the use of path- and git-based dependencies can be categorized as specifying a dependency to a "moving target" rather than a fixed version.

In the case of path-based dependencies, there is no version, there is only what code is currently residing in a local directory at build time. This is inherently unstable, and can easily lead to problems such as confusion about which version of the code was present in the directory at build time.

Git-based dependencies suffer from the same problem -- instead of the moving target being a local directory, it's a git branch. Changes, including possibly breaking changes, can be pushed to a branch at any time. What exists in master branch today is unlikely to be the same as what exists in master branch next week.

In contrast, a registry server makes each specific crate version -- a fixed point in the code history which was verified to build immediately before publishing -- remains available in perpetuity, without needing to perform any work (i.e. checking out the correct branch, etc.) for that version to be available.

In practice, choosing between a set of fixed crate versions rather than the moving targets of path-based or git-based dependencies solves a surprising number of problems relating to dependency management of private crates.

Upgrade Flexibility vs. Forced Synchronicity

Cargo allows many versions of a single crate to be used across a larger codebase without conflict.

As a Rust developer, this aspect of Cargo is so pervasive and painless it's easy to forget that not all package managers and/or language import semantics allow this. There are many languages, including major ones with many more users than Rust, which do not permit this paradigm. In other cases it is impossible in practice.

(Not only can multiple versions of the same crate be used, but code utilizing competing Rust editions can be used entierely interchangeably, which is amazing for anyone who endured certain difficult transitions between language versions.)

To demonstrate the idea of multiple version coexistence, consider the following example:

# Cargo.toml

[package]
name = "my-crate"

[dependencies]
reqwest = "0.11" # <- depends on base64 v0.13
base64 = "0.12"

The reqwest crate also depends on base64, but on a newer version, 0.13. But this poses no problem for Cargo, the two versions of base64 can coexist peacefully in the larger codebase.

This is so easy and common that you probably no longer give it a second thought.

However, things become more complicated when you step away from the semantic versioning-based dependency resolution of registry dependencies.

Consider this alternative scenario, using git-based dependencies:

[package]
name = "a"

[dependencies.b]
git = "https://my-git-server.com/proprietary-crate-b.git"
branch = "master"

[dependencies.c]
git = "https://my-git-server.com/proprietary-crate-c.git"
branch = "master"

And in the Cargo.toml for b:

[package]
name = "b"

[dependencies.c]
git = "https://my-git-server.com/proprietary-crate-c.git"
branch = "master"

This creates a problem which was not present for the registry-based dependencies above (reqwest using base64 version "0.13" while my-crate depends on base64 version "0.12").

Namely, it's not possible for b and c to depend on different versions of c, as configured. They both depend on "master". As far as has been specified to Cargo, they depend on the same thing. But if one of the crates (say, b) hadn't run cargo update in a while (perhaps to avoid the upgrade conflicts that arise from git-based dependencies, it might, in practice, have been depending on a much different point in "master" as compared to c.

  "b" was depending on "this" master
   |
   |
x--x--x--x--x--x--x--x--x-> master      
   ^           |
   |           |
   ----->     "c" was depending on "this" master
   |
   code at the b, c
   commits is not compatible


   ------ time ----->

Thus, when you attempt to build these three crates together, it won't actually build, and the reasons why are complex and confusing. Meanwhile, the headache could have been avoided entirely by utilizing a registry server and the semantic versioning-based dependency resolution model that is the norm for open source crates.

Creating a Shipyard.rs Account

Creating a new account at Shipyard.rs is free and takes less than 30s. Our free tier provides everything you need to see how a private crates registry works, and might even provide enough usage for small projects.

To create an account, navigate to the sign-up page.

You'll need to provide:

1. An email: this will be the primary identifier or your user account
2. An organization or registry name; this is a name that will be used to refer to your registry.
3. A password: choose wisely.

(Note: authentication via Google Oauth is also provided. In that case, sign-up and login will follow a slightly different path, which will discuss in the next chapter.)

The Shipyard.rs sign up form

Authenticating With Google Oauth

In addition to basic username/password-based registration, it's also possible to authenticate via Google Oauth instead.

At the sign-up page, look for the "Continue with Google" button:

Authenticate via Google

Clicking the button will forward you to Google's servers, which will ask for your permission to provide Shipyard.rs with your basic info.

Once you're authenticated, there's one more step: providing an organization or registry name.

After authenticating, there's one more step

If you create an account using Google Oauth, you won't provide a password, and you will only be able to login to the account via Google Oauth going forward. Logging in with email/password won't be possible for that account.

The same goes for creating an account with email/password: logging in via Google Oauth won't be available for those accounts, even if they share the same email address.

Launching Your New Registry Server

Upon creating an account, either using an email/password or via Google Oauth, the Shipyard.rs will go to work provisioning the infrastructure for your new registry server.

Meuse

Each organization account is provisioned a Meuse instance to provide the core registry server backend, as well as a Postgresql database instance for its storage.

Meuse, an open source registry server written in Clojure, has been very solid for us.

Gitea

A registry server can't function (at least currently) without a git repository for storing its crate metadata. Just as Crates.io has a massive index at Github with hundreds of thousands of crate versions, even the smallest registry starts with providing a git repo with a list of which crates can be downloaded.

At git.shipyard.rs we provide a private Gitea server which is used almost entirely to host the crate index repositories for Shipyard.rs accounts.

When you create a Shipyard.rs account, a sub-account is created on your behalf at git.shipyard.rs. Most of your interaction with the crate-index repository will be automated, but it's possible to log into the Gitea instance if you registered with an email/password. (That is not currently possible if you signed up via Google Oauth).

Docyard: Shipyard's Docs.rs Equivalent

As soon as you publish your first crate version at the new registry, our Docyard server will receive an assignment to build the Rustdocs for you, so you can access them at Shipyard.rs anytime you need.

Since building a Rust crate entails running arbitrary code (via a build.rs script) written in a systems langague, we make significant efforts, similar to the approach of Docs.rs, to ensure those builds are sandboxed and isolated.

Shipyard.rs Web Backend: the Glue that Keeps it all Together

Orchestrating the various components is a significant job. Our web backend is on it, tracking service health, routing API requests, and ensuring the security of your source code. Although we generally route registry server requests to your Meuse backend server, an increasing number of endpoints are served directly from Shipyard.rs Web Backend.

Many excellent Rust crates are powering the backend, but some notable mentions include actix-web, used as the primary web framework, and Tantivy, used for indexing and searching each account's crates.

Authentication Model

Authorization of HTTP(S) requests to a crates registry server is performed via an authentication token in the body/value of a HTTP "authorization" header, e.g.:

$ curl -H "authorization: ${MY_AUTH_TOKEN}" \
  https://registry-server.rs/api/v1/example-authenticated-request

In general, cargo appends the authentication headers to outgoing requests it generates as part of a build command.

For example, cargo publish will generate an HTTPS PUT request to https://crates.io/api/v1/crates/new (assuming the crate is being published at --registry crates.io) with an auth token in the "authorization" header for identifying whether the user has permissions to publish a crate version for the crate in question (i.e. is an owner of the crate).

Historical Limitations

For the public crates registry at Crates.io, only commands that involve publishing crates, such as login, publish, and yank perform and/or require authorization.

This poses a problem for a private crates registry, because under that model, it is possible to download the .crate artifacts (source code) for any crate hosted at the registry server, without any authorization, so long as the crate name and version are known or guessed.

Also, when performing cargo build, cargo check and other build commands that do not involve publishing crates, cargo does not (currently) include an "Authorization" header, so there is no way for a private crate registry server to determine whether the requester is authorized to perform the request.

Authenticated Downloads Via -Z registry-auth

As of the cargo nightly 2022-11-17, the -Z registry-auth unstable feature is available for fully authenticated downloads without the use of non-standard configuration.

How It Works

The way the -Z registry-auth feature works is by providing an optional auth-required setting in the registry index's config.json file.

When auth-required setting is true, Cargo then requires an auth token for crate downloads and includes an "Authorization" header with the auth token in its download requests to the registry server.

By default, Shipyard.rs sets auth-required to true in new registry index config.json files.

Example config.json with auth-required setting set to true:

{
  "dl": "https://crates.shipyard.rs/api/v1/crates",
  "api": "https://crates.shipyard.rs",
  "allowed-registries": [
    "https://github.com/rust-lang/crates.io-index",
  ],
  "auth-required": true
}

How to Use -Z registry-auth Feature

Via Configuration (Recommended)

The -Z registry-auth feature can be enabled via configuration:

# ~/.cargo/config.toml 
[unstable]
registry-auth = true

Via Command Line Flag

Adding -Z registry-auth command line flag to a cargo command will enable the feature, e.g.:

$ cargo check -Z registry-auth

Via Environment Variable

The environment variable CARGO_UNSTABLE_REGISTRY_AUTH can also be used:

$ CARGO_UNSTABLE_REGISTRY_AUTH=true cargo check

Background: Using Nightly

If you have not previously used Rust nightly, you may need to install the nightly channel with rustup:

$ rustup install nightly

Updating Rust Nightly to a Version with -Z registry-auth Available

Use the following command to update the nightly channel to the latest version:

$ rustup update nightly

Using Rust Nightly By Default

Use the following command to use rust nightly by default:

$ rustup default nightly

Troubleshooting

"Authenticated Registries Requre a Credential-Provider to be Configured"

See Rust 1.74 and credential-process Changes.

"Authenticated Registries Require -Z registry-auth"

Example error message:

$ cargo check
error: failed to download `my-private-crate v0.1.0 (registry `my-registry`)`

Caused by:
  unable to get packages from source

Caused by:
  authenticated registries require `-Z registry-auth`

This error message indicates that the -Z registry-auth feature was not enabled when the Cargo subcommand was invoked. See "How to Use -Z registry-auth Feature".

Rust 1.67 Compatibility Issue

The "authenticated registries require -Z registry-auth" error message can also result from using (stable) Rust version 1.67.

In version 1.67, Cargo will prevent downloads from registries configured with the auth-required setting to true, however, use of the -Z registry-auth feature is not possible using a stable version of rust. This produces a situation where Cargo will not download from the registry unless a mode is enabled that is not permitted to be enabled.

After discussions with the Cargo team, this change was reverted in 1.68. However, in order to use Shipyard.rs with Rust 1.67, you must use the "User-Agent" header-based authorization, and contact support@shipyard.rs to modify the configuration settings for your crate index repository. (Note: Shipyard.rs always requires authentication for all API calls, whether or not the auth-required setting is set to true in the crate index config.json file.)

This problem is not present using Rust 1.66 or below, or Rust 1.68 and above, just 1.67.

"Only Accepted on the Nightly Channel"

Example error message:

$ cargo check
error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel
See https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.

This error indicates that the Cargo subcommand was invoked using the stable version of Cargo/rustc; using Rust nightly is required to enable this feature. See "Using Nightly".

"Failed to Download" (401)

Example error message:

$ cargo check
error: failed to download from `https://crates.shipyard.rs/api/v1/crates/my-private-crate/0.1.0/download`

Caused by:
  failed to get 200 response from `https://crates.shipyard.rs/api/v1/crates/my-private-crate/0.1.0/download`, got 401

You might get this error if the -Z registry-auth feature was not enabled, and the stable version of Cargo was used to invoke the subcommand. In that case, the download request will not have included an "Authorization" header, and Cargo will not have known about the auth-required setting in the registry index's config.json file.

"Unknown -Z Flag"

Example error message:

$ cargo +nightly check -Z registry-auth
error: unknown `-Z` flag specified: registry-auth

This error indicates that the nightly version used to invoke the subcommand is from before the -Z registry-auth feature became enabled. To fix, update your nightly version with the rustup update nightly command.

-Z registry-auth Stabilization Process

• RFC 3139, proposing an auth-required setting that would prompt Cargo to send an authentication token with crate download requests, was approved in March, 2022

• A pull request by Arlo Siemsen implementing RFC 3139 was approved Nov. 16, 2022

• The -Z registry-auth unstable feature is available in cargo versions as of nightly 2022-11-17

Rust 1.74 and credential-process Changes

In nightly Rust as of 2023-09-19 (rustc v1.74.0), Cargo requires the [registry.global-credential-providers] key to be configured in ~/.cargo/config.toml to be able to use the registry-auth feature and perform authenticated downloads:

Example config:

# ~/.cargo/config.toml

[registry]
global-credential-providers = ["cargo:token"]

Without configuring global-credential-providers, you may get an error like this when Cargo tries to download a .crate file from the registry server:

$ cargo +nightly check
error: failed to download `my-private-crate v1.2.3 (registry `shipyard-rs`)`

Caused by:
  unable to get packages from source

Caused by:
  authenticated registries require a credential-provider to be configured
  see https://doc.rust-lang.org/cargo/reference/registry-authentication.html for details

Note: the URL in the error message is currently incorrect; the documentation can be found instead at:

https://doc.rust-lang.org/nightly/cargo/reference/registry-authentication.html

The current official documentation recommends also including cargo:libsecret in global-credential-providers on Linux. Including it may result in the following error message:

$ cargo +nightly check
error: failed to download `my-private-crate v1.2.3 (registry `shipyard-rs`)`

Caused by:
  unable to get packages from source

Caused by:
  credential provider `cargo:libsecret` failed action `get`

Caused by:
  failed to load libsecret: try installing the `libsecret` or `libsecret-1-0` package with the system package manager

Caused by:
  libsecret-1.so: cannot open shared object file: No such file or directory

To resolve this error, either remove cargo:libsecret from the list of credentials providers, or install libsecret using your package manager. cargo:token will allow you to use token-based authentication as before the 1.74 changes.

Stabilization of -Z registry-auth

With Rust 1.74.0, it looks like the registry-auth feature is on track to finally be stabilized!

When it is stabilized, it will no longer be necessary to include the [unstable] configuration in your ~/.cargo/config.toml, i.e. this section can be removed:

[unstable]
registry-auth = true

If you use multiple versions of Rust in your development work, it may be prudent to leave the unstable configuration in place, despite the warning message from Cargo that it will be ignored (appears in Rust 1.74 and newer).

Passing Auth Token Via User-Agent Header

Until the -Z registry-auth unstable cargo feature is stabilized, we are providing a temporary solution to perform authenticated crate downloads at Shipyard.rs.

Note: Using the "User-Agent" header to pass an auth token is not ideal. If you are using the nightly version of Cargo/rustc, a better solution is to enable the -Z registry-auth unstable feature.

Configuring User-Agent Header

There is a somewhat obscure configuration option to specify the "User-Agent" header that cargo sends as part of all of its requests, which defaults to the cargo version number being used.

Shipyard.rs will first check requests for the presence of an "Authorization" header, but will fall back to checking for a "User-Agent" header in a specific format, which will be used as a fallback.

For an example auth token with the dummy value AxzHJS6gWK=, the cargo "User-Agent" setting would be configured by adding this to ~/.cargo/config.toml:

# set a custom user-agent header value for cargo requests:
[http]
user-agent = "shipyard AxzHJS6gWK="

See the configuration docs for additional details.

Using the "User-Agent" header to pass an authorization token is less than ideal and temporary. However, for what it is worth, our servers have been configured not to log the "User-Agent" header for requests that might contain an authentication token in the header value.

Case Study: cargo publish --help

Explaining several flags from the --help output from cargo publish subcommand may help further clarify:

$ cargo publish --help
cargo-publish 
Upload a package to the registry

USAGE:
    cargo publish [OPTIONS]

OPTIONS:
        --token <TOKEN>             Token to use when uploading
        --index <INDEX>             Registry index URL to upload the package to
        --registry <REGISTRY>       Registry to publish to
        # ...

--token is an authentication token, used to identify your request to the registry server (via the "authorization" header, generally speaking, or the "user-agent" header, using the temporary Shipyard.rs solution.

--index is the url of a git repository with metadata about the crates and crate versions available at the registry server.

--registry is the name of the registry server, specified to cargo via configuration. Specifically, the [registries] setting allows one to associate a registry name with an git url (i.e. what you might otherwise pass using the --index flag):

[registries.my-registry-server]
index = "https://git.shipyard.rs/my-registry-server/crate-index.git"

With this configuration, the registry name, passed via --registry flag, allows cargo to identify the crate --index git url for use in retrieving metadata about the registry server.

Cargo and ~/.cargo/config.toml

cargo searches for its configuration files in a hierarchical set of paths.

$CARGO_HOME/config.toml is generally the best place to put global configuration settings, which typically would include a [registries] entry specifying the index url of your private crate registry. ("global" here means across all projects for a specific user on a given machine.)

Alternatively, cargo configuration can also be specified inside a crate's source directory, i.e. for a crate at ~/src/my-crate, configuration can be set in a file at ~/src/my-crate/.cargo/config.toml, which is useful for settings that are specific to a given crate/project.

Tip: A tricky problem to detect is when the cargo config settings in one file are overriding the settings in another file unexpectedly. Cargo's documentation about which paths it searches for configuration files is here.

Example Configuration

When you log in to Shipyard.rs, an example configuration file is generated for you, using your actual account details. See the Git Index page.

Generated example configuration using values specific to your account

Adding a [registries] Entry

Example registries entry - would be written to $CARGO_HOME/config.toml or other configuration path:

[registries.my-crate-registry]
index = "https://git.shipyard.rs/my-org/crate-index.git"

One this has been set, cargo will recognize the registry in various subcommands via the --registry flag, e.g.:

cargo publish \
--registry my-crate-registry

When a registries entry has not been set, Cargo will return a "no index set" error:

$ cargo update
error: failed to parse manifest at `~/src/my-private-crate/Cargo.toml`

Caused by:
  no index found for registry: `my-shipyard-registry`

Authenticated Downloads: Configuration

Shipyard.rs requires authorization for all crate downloads, unlike how crates.io works.

This page contains instructions for configuring ~/.cargo/config.toml. For additional discussion about cargo authentication, see the Authentication secion.

Unstable Option: -Z registry-auth

If you are using nightly, the unstable registry-auth feature is available, which prompts cargo to pass an auth token via the "Authorization" header in HTTP download requests.

To enable:

# ~/.cargo/config.toml

[unstable]
registry-auth = true

Additional details: Authenticated Downloads Via -Z registry-auth.

Via User-Agent

Example of using cargo's "user-agent" setting to pass an auth token (see Temporary Fix) - would be written to $CARGO_HOME/config.toml or other configuration path:

# set a custom user-agent header value for cargo requests:

[http]
user-agent = "shipyard AxzHJS6gWK="

# this setting, while facially unrelated and not strictly required for performing
# authentication # via the "user-agent" header, seems to mitigate various problems
# that arise from using the "user-agent" # header in a ... non-traditional way

[net]
git-fetch-with-cli = true

Git (Crate Index) Configuration

The "crate index" is a git repository hosted at git.shipyard.rs, which stores metadata about which crate versions are available at your organization's registry server.

In the case of Crates.io's github repository, no authentication is required, since the repository is publicly available. Shipyard.rs Crate Index repositories, intended for proprietary code reuse, are configured to be private by default.

Git Authentication

Unlike the Crates.io github repository, you must configure authentication to access the Shipyard.rs crate index, which is not publicly available.

Authentication for the Crate Index repository works the same as authentication for Github or other git hosts, either via HTTPS with username/password, or by adding an SSH public key to perform key-based authentication.

Adding SSH Keys

Adding SSH public keys for accessing your registry's crate-index repository hosted at git.shipyard.rs can be done using directly from Shipyard.rs (vs. logging into git.shipyard.rs) using the tools on the Git Index page. Full details are included in the SSH-based Authentication subchapter:

Form for adding an SSH public key

Logging into git.shipyard.rs

When a new Shipyard.rs user account is created, a parallel user account is created for the user at git.shipyard.rs, using the same email/password the user registered with for the main Shipyard.rs site.

Using those credentials, users can log in to the git server (running Gitea under the hood) to perform administrative actions such as adding SSH keys to their account.

Note: this functionality is not currently available for accounts using Google Oauth for authentication.

Logging into the Gitea account directly is not required, and generally not needed to use the registry server, but remains available for advanced configuration and account maintenance.

Git Index Privacy

Your organization, its users, and the contents of the crate-index git repository created for your account are not visible to other users of Shipyard.rs when they log in to git.shipyard.rs.

crate-index Must Exist

Please do not delete the crate-index repository or the Gitea organization account if you wish to continue using the Shipyard.rs registry server. The crate-index repository is the authoritative storage for metadata about which crates and crate versions have been published to your registry server.

SSH-based Authentication

In general, SSH-based authentication works better than HTTPS + username/password authentication for use with Cargo and private registries, and should be preferred.

Adding SSH Keys in Shipyard.rs

The Git Index page provides the ability to add and remove SSH public keys to your git.shipyard.rs account directly from Shipyard.rs.

To add an SSH public key, provide a title, paste the public key content, and decide if you want the key to have read-only permissions (recommended):

Form for adding an SSH public key

After an SSH public key has been added, it will be listed in the SSH Keys section:

List of SSH public keys associated with your account

Removing SSH Keys

To remove a SSH key, click the "trash" icon ( ) on the row corresponding to the SSH key you want to delete.

Adding a Public Key to Gitea

SSH key management can also be performed by logging into Gitea directly:

Logging into the Gitea account directly is not required, and generally not needed to use the registry server, but remains available for advanced configuration and account maintenance.

Cargo-friendly SSH Url

TLDR: the git index SSH url should have the format ssh://git.shipyard.rs/<org-slug>/crate-index.git for it to work with Cargo.

The example configuration generated for you on the Git Index includes the SSH url in the correct, cargo-friendly format.

Additional Details:

The SSH clone URL provided by Gitea in its UI needs to be tweaked slightly for Cargo to play nice with it.

Gitea gives the SSH url for Shipyard.rs's crate index repo as:

git@ssh.shipyard.rs:shipyard-rs/crate-index.git`

Also, passing that url to git clone works as expected:

$ git clone git@ssh.shipyard.rs:shipyard-rs/crate-index.git   # <- that works great

However, if you create an entry in ~/.cargo/config.toml that specifies that URL, it will not work:

# this will not work

[registries.shipyard-rs]
index = "git@ssh.shipyard.rs:shipyard-rs/crate-index.git"

Two changes to the format of the url must be changed for it to work with cargo:

1. The ssh:// scheme needs to be added to the beginning of the url
2. The colon character (:) following the host name must be changed to a slash (/)

With those changes, authentication will work using SSH url:

# correct format 

[registries.shipyard-rs]
index = "ssh://git@ssh.shipyard.rs/shipyard-rs/crate-index.git"

Cargo Security Advisory (CVE-2022-46176)

On Jan. 10, the Rust team announced that Cargo was being patched to fix a security vulnerability relating to how it cloned git repositories over SSH.

Specifically, a previous implementation had not verified the identity of SSH keys when performing clone, leaving it vulnerable to man-in-the-middle (MITM) attacks.

The issue was fixed in Rust 1.66.1.

One important aspect of the vulnerability is that it did not affect users who had set the git-fetch-with-cli setting to true, which prompts Cargo to use the local git binary to perform clones over SSH.

We strongly recommend setting the git-fetch-with-cli configuration setting, and all Shipyard.rs rustdoc builds are performed with the setting enabled.

HTTPS-based Authentication

Another possible way to authenticate to the repository is HTTPS with username and password using git's "credential helper" store.

In ~/.gitconfig:

[credential]
  helper = store

In ~/.git-credentials (note: @ (as %40) and other characters in a username or password must be escaped using url encoding):

https://user:pass@git.shipyard.rs

Note: remember to set the git-fetch-with-cli setting In ~/.cargo/config.toml (optional but highly recommended):

[net]
git-fetch-with-cli = true

git-fetch-with-cli: the Silver Bullet of Cargo Configuration Troubleshooting

Cargo can be configured to use the git executable to perform repository authentication, instead of the default, which uses libgit2:

# ~/.cargo/config.toml

[net]
git-fetch-with-cli = true

Changing this setting resolves a shockingly wide array of configuration problems.

The root issue is that Cargo uses libgit2 by default to interact with a registry index repository, which has known shortcomings as it relates to authentication. These shortcomings are fully acknowledged by the Cargo team, and modifying the git-fetch-with-cli setting is frequently put forward as a potential solution to users having trouble.

Honestly, do yourself a favor: for anything other than bog standard Cargo settings, set git-fetch-with-cli to true.

Here is an example error output from cargo login that was resolved by setting git-fetch-with-cli = true:

$ cargo login --registry shipyard-rs ${AUTH_TOKEN}

error: failed to update registry `shipyard-rs`

Caused by:
failed to fetch `ssh://git@ssh.shipyard.rs/shipyard-rs/crate-index.git`

Caused by:
failed to authenticate when downloading repository

* attempted ssh-agent authentication, but no usernames succeeded: `git`

if the git CLI succeeds then `net.git-fetch-with-cli` may help here
https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
no authentication available

Relation to CVE-2022-46176

On Jan. 10, the Rust team announced that Cargo was being patched to fix a security vulnerability relating to how it cloned git repositories over SSH.

One important aspect of the vulnerability is that it did not affect users who had set the git-fetch-with-cli setting to true, which prompts Cargo to use the local git binary to perform clones over SSH.

Creating an Auth Token

To create an authentication token, log in to Shipyard.rs using your email/password, then navigate to the tokens page.

Use the form to create an auth token, specifying a name for the token and the number of days it will be valid, starting from today:

Using the cargo login Command

Using the auth token generated in Create Auth Token, you can use the cargo login command to save the token in ~/.cargo/credentials for future reuse.

Note: please be aware the token will be saved in plain text (600/rw------- mode) in ~/.cargo/credentials.

To save the token using cargo login:

$ cargo login \
--registry my-crate-registry \
dOvx04Jm4lYGw=

In the above command, my-crate-registry should match the name given to the registry in the [registries] configuration entry.

$ cargo login --help
cargo-login 
Save an api token from the registry locally. If token is not specified, it will be
read from stdin.

USAGE:
  cargo login [OPTIONS] [token]

ARGS:
  <token>    

OPTIONS:
      --registry <REGISTRY>    Registry to use
      # ...

Successful login will produce a message like this:

$ TOKEN="x8P1UUELcY6SxRA3Ylsjlkjsdfoiew0239Lh7Gth5qrI46a3mZ47cjA0wDU+9cNmJMQ="
$ cargo login \
--registry my-crate-registry \
${TOKEN}

Updating `my-crate-registry` index
Login token for `my-crate-registry` saved

cargo publish Command

If you configured a registries entry in ~/.cargo/config.toml and saved your auth token with cargo login, you can publish by specifying the --registry flag:

$ cd ~/src/my-proprietary-crate
$ cargo publish --registry my-crate-registry

Alternatively, the auth token and also crate index url can be specified manually:

$ TOKEN="x8P1UUELcY6SxRA3Ylsjlkjsdfoiew0239Lh7Gth5qrI46a3mZ47cjA0wDU+9cNmJMQ="
$ INDEX="https://git.shipyard.rs/my-registry-server/crate-index.git"

$ cargo publish \
--index ${INDEX} \
--token ${TOKEN}

As discussed in Case Study: cargo publish --help, there are multiple ways to specify registry and/or authentication token using the cargo publish command:

cargo publish --help
cargo-publish 
Upload a package to the registry

USAGE:
  cargo publish [OPTIONS]

OPTIONS:
      --index <INDEX>             Registry index URL to upload the package to
      --token <TOKEN>             Token to use when uploading
      --registry <REGISTRY>       Registry to publish to
      # ...

Pre-Flight Checklist

Quick checklist for cargo publish success:

• [registries] entry in ~/.cargo/config.toml
• Crate index git authentication in place (i.e. you can git clone the repo)
• Auth token created
• Auth token saved or passing explicitly using --token
• Clean git status in crate directory (or --allow-dirty flag if you are barbarian)
• No compilation errors preventing the crate from building

cargo publish Success Output

Lets publish a crate version!

$ cd ~/src/my-proprietary-crate
$ cargo publish --registry my-crate-registry

Packaging my-proprietary-crate v0.1.0 (~/src/my-proprietary-crate)
Verifying my-proprietary-crate v0.1.0 (~/src/my-proprietary-crate)
Updating crates.io index
Downloaded libc v0.2.127
Downloaded 1 crate (595.0 KB) in 0.29s
Compiling libc v0.2.127

# ...

Compiling my-proprietary-crate v0.1.0 (~/src/my-proprietary-crate/target/package/my-proprietary-crate-0.1.0)
Finished dev [unoptimized + debuginfo] target(s) in 2.82s
Uploading my-proprietary-crate v0.1.0 (~/src/my-proprietary-crate)

Preventing Accidental Publishing to Crates.io

Prior to publishing, you may want to specify which registries a crate may be published to, using the [package.publish] key in Cargo.toml, to avoid accidentally publishing to Crates.io:

# Cargo.toml

[package]
name = "my-proprietary-crate"
publish = ["my-crate-registry"]

Depending on Registry Crates

Adding a crate from a Private Crate Registry in Cargo.toml is nearly identical to adding a Crates.io dependency, with the exception that you must specify the registry name:

# Cargo.toml

[dependencies]
# a Crates.io dep
uuid = { version = "1.1", features = ["v4"] }                               

# a private crate registry dep
my-proprietary-crate = { version = "0.1", registry = "my-crate-registry" }

Troubleshooting: 'I Can Publish, But Not Download'

One situation you can find yourself in is that you can publish crates, but not depend on them.

Publishing goes great, the .crate file gets uploaded to the server, and everything seems like it's finally in place.

But then you add the crate you published as a dependency in a second crate:

# Cargo.toml

[dependencies]
my-first-publish = { version = "*", registry = "shipyard-rs" }

...and it all falls apart:

$ cargo check
  Updating `shipyard-rs` index
error: failed to download from `https://crates.shipyard.rs/api/v1/crates/my-first-publish/0.1.0/download`

Caused by:
failed to get 200 response from `https://crates.shipyard.rs/api/v1/crates/my-first-publish/0.1.0/download`, got 403

The problem is likely created by not configuring cargo to send an auth token via the "user-agent" header (see Temporary Fix).

Once you configure cargo to send a Shipyard.rs auth token via the "user-agent" header:

# ~/.cargo/config.toml
#
# or
#
# ~/path/to/crate/.cargo/config.toml

[http]
user-agent = "shipyard sdk4fXksTjf2247example="

...it should work:

$ cargo check
Downloaded my-first-publish v0.1.0 (registry `shipyard-rs`)
Downloaded 1 crate (811 B) in 0.53s
  Checking my-first-publish v0.1.0 (registry `shipyard-rs`)
  Checking dep-test v0.1.0 (/home/jstrong/src/dep-test)
  Finished dev [unoptimized + debuginfo] target(s) in 0.65s

In the future, the config.json file in the crate index will specify auth-required as true, cargo will send the auth token for you without any fuss, and this problem will cease to exist (see Pending Improvements).

Crate Name Uniqueness

For a given registry server, a "crate" is identifiable by its name, which is unique to the crate registry. Each Shipyard.rs organization is its own registry server in this sense; if organization A publishes crate "abc", it will not prevent organization B from also publishing a crate that is named "abc".

Permitted Crate Names

Shipyard.rs enforces the same rules as Crates.io regarding valid crate names:

Cargo itself allows names with any alphanumeric, -, or _ characters. crates.io imposes its own limitations, including the following:

• Only allows ASCII characters.
• Only alphanumeric, -, and _ characters.
• First character must be alphabetic.
• Case-insensitive collision detection.
• Prevent differences of - vs _.
• Under a specific length (max 64).
• Rejects reserved names, such as Windows special filenames like "nul".

Crate Ownership

A given crate is owned by one or more users. The user who first publishes a crate version for a given crate (identifiable by its name) becomes its owner.

Only owners of a crate can publish new versions to that crate.

For the purposes of cargo owner -add and cargo owner --remove subcommands, a Shipyard.rs user should be identified by the email address associated with their account.

The email address must correspond to an existing Shipyard.rs user account which is part of the organization account which owns the crate.

See also: cargo owner docs.

Listing Crate Owners

To list current owners of a crate, navigate to the crate directory and run:

$ cargo owner --registry <registry-name> --list

scrooge.mcduck@fastmail.com (user-id = d4dddf5f-cb77-41dc-9635-1dfc072f40ae)

Note: passing the --registry flag is not required if you set a single registry in the publish field in the crate's Cargo.toml.

The subcommand also provides a --token option, for use if you have not configured your auth token via cargo login. If you get the error message

error: no upload token found, please run `cargo login` or pass `--token`

this indicates you need to pass an auth token using --token or configure the token using cargo login.

The output displayed by a successful cargo owner --list command lists each user who is an owner of the crate. The user's email is listed first, and his or her Shipyard.rs user id is listed in parenthesis.

Add/Remove Crate Owners

The cargo owner --add subcommand can be used to add owners to a crate, and the cargo owner --remove subcommand can be used to remove users from a crate.

To add a user as an owner of the crate:

$ cargo owner --registry <registry-name> \
  --add launchpad.mcquack@hotmail.com

Owner added user(s) 4c26f6c5-6aa0-4d36-8aa7-6c846c7acbeb as owner(s) of <crate-name>

Listing crate owners will now display the second user as an owner of the crate:

$ cargo owner --registry <registry-name> --list

scrooge.mcduck@fastmail.com (user-id = d4dddf5f-cb77-41dc-9635-1dfc072f40ae)
launchpad.mcquack@hotmail.com (user-id = 4c26f6c5-6aa0-4d36-8aa7-6c846c7acbeb)

To remove a user as an owner of the crate:

$ cargo owner --registry <registry-name> \
  --remove launchpad.mcquack@hotmail.com

Owner removing ["launchpad.mcquack@hotmail.com"] from crate <crate-name>

Listing crate owners will now display the removed user is no longer an owner:

$ cargo owner --registry <registry-name> --list

scrooge.mcduck@fastmail.com (user-id = d4dddf5f-cb77-41dc-9635-1dfc072f40ae)

Crate Owners and Account Permissions

An account with Admin permissions may add "itself" as an owner of a crate, using its own auth token, despite not being an owner of the crate.

$ cargo owner --registry <registry-name> \
  --add it-dept@ducks.org \
  --token ${ADMIN_ACCOUNT_AUTH_TOKEN}

# works

Neither Publish nor Read accounts have this ability.

If you try adding a user to a crate you do not own using an auth token from a Publish or Read account, you will receive the following error:

$ cargo owner --registry <registry-name> \
  --add huey.mcquack@msn.com \
  --token ${PUBLISH_OR_READ_ACCOUNT_AUTH_TOKEN}

error: failed to invite owners to crate `<crate-name>` on registry
at https://crates.shipyard.rs

Caused by:
the remote server responded with an error: user does not own the
crate <crate-name>

Alternate Ownership Models

When building the functionality around cargo owner, it seemed likely to us that the current ownership model would not be ideal for a smaller organization.

In the case of a small, high-velocity team, it seems like it would be useful to allow any user with Admin or Publish permissions to publish to any crate in the registry, avoiding the need to manage user-based ownership of each crate.

If you are interested in something like that (or find the existing ownership structure is not suited to your organization's workflow), please reach out to us via email or Zulip, we'd love to hear about it!

Yanking a Crate Version

"Yanking" a crate version is a way of marking that version as unsuitable for use, without deleting it (and potentially breaking other code that depends on it).

After a crate version has been yanked, Cargo will never choose that version for use when resolving dependencies. However, if the yanked version had previously been used, and is present in a crate's Cargo.lock file, that crate will still be able to download the yanked version (and use it).

In this way, yanking a version prevents its use going forward while not breaking existing code.

Yanking is reversible; a yanked version may be unyanked.

How to Yank a Crate Version via cargo yank

$ cargo yank \
    --registry ${REGISTRY_NAME} \
    --version ${VERSION_TO_YANK} \
    ${NAME_OF_CRATE}

Or using specific (dummy) names:

$ cargo yank \
    --registry my-private-registry \
    --version 0.2.1 \
    rust-deserts

Requires Auth Token

cargo yank command requires an auth token, which can either be provided using a stored token (saved to ~/.cargo/credentials via cargo login), or by using the --token command line option, e.g.:

$ cargo yank \
    --registry my-private-registry \
    --version 0.2.1 \
    --token ${AUTH_TOKEN} \
    rust-deserts

How to Unyank a Crate Version via cargo yank

To unyank a crate version, pass the --undo flag to the command:

$ cargo yank \
    --registry my-private-registry \
    --version 0.2.1 \
    --undo \
    rust-deserts

How to Yank/Unyank using Shipyard.rs

Shipyard.rs also provides a way to yank or unyank a crate version.

To yank a crate version, navigate to the crate version detail page, located at the (pseudo-) url:

https://shipyard.rs/<org-slug>/crates/<crate-name>/<version>

For example, the crate version detail page for the crate rust-deserts version 1.1.0 for the my-private-registry registry would be:

https://shipyard.rs/my-private-registry/crates/rust-deserts/1.1.0

Navigating to the page should be fairly intuitive starting from the Crates page.

The "Yank" section contains a button to yank the respective crate version:

Yank section from crate version detail page

When a crate version is currently yanked, clicking the "Unyank" button will unyank it:

Unyank button appears when crate version is yanked

Cargo Yank Case Study: Rust Deserts

Consider the my-private-registry registry, which hosts the crate rust-deserts.

rust-deserts has published the following versions:

• 0.1.0
• 0.2.0
• 0.2.1

A second crate in the registry, innocent-bystander, depends on rust-deserts:

# Cargo.toml (innocent-bystander)

[dependencies]
rust-deserts = { version = "0.2.0", registry = "my-private-registry" }

A developer working on innocent-bystander runs the cargo update command while working on the crate, and Cargo identifies rust-deserts version 0.2.1 as the highest available version based on the semantic versioning-based dependency resolution of the specifications declared in innocent-bystander's Cargo.toml.

The version of rust-deserts used by innocent-bystander is stored in its Cargo.lock file:

# Cargo.lock (innocent-bystander)

[[package]]
name = "rust-deserts"
version = "0.2.1"
source = "registry+https://git.shipyard.rs/my-private-registry/crate-index.git"
checksum = "0b943c9c44efe45f8384871a022e4eb728cfed28a5e752fa57f7cb1c89680160"

A Dish Served Cold

Unfortunately, the authors of rust-deserts had been rather cavalier in their use of unsafe (infamously, the rationale for one unsafe block included the line "a little ub never hurt anyone"), and discover that the changes in version 0.2.1 introduced a major security vulnerability.

To mitigate the security risk, rust-deserts' authors yank version 0.2.1:

$ # ~/rust-deserts

$ cargo yank --registry my-private-registry --version 0.2.1 rust-deserts

Since version 0.2.1 of rust-deserts has been yanked, Cargo will never choose that version again when resolving dependencies. However, since 0.2.1 appears in the Cargo.lock file for the innocent-bystander crate, Cargo will still download the 0.2.1 version for it to use even after 0.2.1 has been yanked:

$ # ~/innocent-bystander - post-yank

$ cargo fetch
    Updating `my-private-registry` index
  Downloaded rust-deserts v0.2.1 (registry `my-private-registry`)
  Downloaded 1 crate (680 B) in 0.01s

However, if the cargo update command is run for innocent-bystander, the rust-deserts dependency will be downgraded down to 0.2.0 (which is not yanked).

$ # ~/innocent-bystander

$ cargo update
    Updating `my-private-registry` index
    Updating rust-deserts v0.2.1 (registry `my-private-registry`) -> v0.2.0

The same is true if the Cargo.lock file is simply deleted from the directory, prompting Cargo to re-resolve the dependency graph.

$ # ~/innocent-bystander

$ rm Cargo.lock
$ cargo fetch
    Updating `my-private-registry` index
  Downloaded rust-deserts v0.2.0 (registry `my-private-registry`)
  Downloaded 1 crate (678 B) in 0.01s

A Harder Case: Patch vs. Minor Version

Life goes on, and rust-deserts reaches a v1.0 release. The crate has now published:

• 0.1.0
• 0.2.0
• 0.2.1
• 1.0.0
• 1.1.0

Alas, the 1.1.0 release includes another jaw-dropping security vulnerability, and innocent-bystander is caught up in the problem once again:

# Cargo.toml (innocent-bystander)

[dependencies]
rust-deserts = { version = "1.1.0", registry = "my-private-registry" }

Like before, innocent-bystander can still download version 1.1.0 of rust-deserts so long as it appears in its Cargo.lock file. However, this time the cargo update command results in a failed build, due to the semantic version-based dependency resolution of Cargo:

$ # ~/innocent-bystander

$ cargo update
    Updating `my-private-registry` index
error: failed to select a version for the requirement `rust-deserts = "^1.1.0"`
candidate versions found which didn't match: 1.0.0, 0.2.0, 0.1.0
location searched: `my-private-registry` index
required by package `innocent-bystander v0.1.0 (~/innocent-bystander)`

Cargo's dependency resolution rules allow downgrading from 0.2.1 to 0.2.0, since they are only a patch version away from each other.

But 1.1.0 and 1.1.0 are a minor version apart, and the Cargo.toml specifies the dependency to minor version granularity, preventing an automatic downgrade. This is just the reverse of the normal upgrade process performed during cargo update.

If innocent-bystander had specified a major version only, i.e.:

# Cargo.toml (innocent-bystander)

[dependencies]
rust-deserts = { version = "1", registry = "my-private-registry" }

...it could still downgrade from 1.1.0 to 1.0.0, even if it had previously used 1.1.0:

$ # ~/innocent-bystander - with `version = "1"` (vs "1.1.0")

$ cargo update
    Updating `my-private-registry` index
    Updating rust-deserts v1.1.0 (registry `my-private-registry`) -> v1.0.0

Deleting a Crate Version

Deleting a crate version permanently removes it from the registry.

Deleting crate versions is not part of the official registry API specified by Cargo. Instead, it is an additional tool provided by Shipyard.rs to give our users complete control over a registry which exists for their purposes (a different context from a public registry like crates.io for which the API was designed).

Words of Caution

In general, yanking should be preferred, for several reasons:

• Stability: Yanking won't break existing code. A yanked version can still be downloaded, a deleted version cannot be.

• Reversibility: A yanked version can be unyanked, a deleted version cannot be undeleted.

• Visibility/Transparency: A yanked version remains listed among other versions, providing discoverability to other developers that a problem was discovered in that version's code. It's documentation is still available.

How to Delete a Crate Version

To delete a crate version, navigate to the crate version detail page, located at the (pseudo-) url:

https://shipyard.rs/<org-slug>/crates/<crate-name>/<version>

For example, the crate version detail page for the crate rust-deserts version 1.1.0 for the my-private-registry registry would be:

https://shipyard.rs/my-private-registry/crates/rust-deserts/1.1.0

Navigating to the page should be fairly intuitive starting from the Crates page by clicking to the crate detail page, and then selecting a version and clicking through to the crate version detail page.

The Delete Crate Version section contains a button to initiate the delete:

Delete Crate Version section from crate version detail page

Clicking the "Delete" button will bring up a confirmation page:

Delete crate version confirmation page

Clicking the "DELETE" button (with a skull and crossbones icon) will delete the crate version.

All copies of the version's:

• .crate file (what Cargo downloads to build the dependency)
• Rustdoc (documentation) .html files
• Metadata entry in the registry's crate-index repository
• Crate metadata entries stored in databases

...will be permanently deleted.

Copies of that data may exist in backups for up to 60 days. It is not possible for a user to retrieve those copies for the purpose of reinstating a deleted crate version.

Deleting an Entire Crate

Deleting can only be applied to a single crate version. To delete an entire crate, delete all versions of the crate. When no versions of the crate remain, the crate itself will be automatically deleted.

After a crate itself has been deleted it no longer counts towards resource limits regarding number of crates that apply to some account plans.

Users

Managing users to your organization account can be performed at the Users page:

• Roles
• Adding a User
• Modifying a User's Role
• Deactivating a User
• Google Workspace Integration

Users page

User Roles

Shipyard.rs user accounts have one of three possible roles: "Admin", "Publish" or "Read".

Admin

"Admin" users have permissions to:

• publish crates
• modify crate ownership
• modify other users' roles
• deactivate and reactivate users
• modify account settings

Publish

"Publish" users may publish crate versions to the registry, but do not have permissions to modify crate ownership, modify other users' roles, deactivate/reactive users, or modify account settings.

Read

"Read" users may download crates from the registry and view crate documentation at Shipyard.rs, but may not publish crate versions or perform any other modifications.

Adding a User

To add a user, enter the user's email address in the Create Users Form on the Users page:

Create User Form

The Create User Form includes a toggle to select the user's Role: choose one of "Admin", "Publish", or "Read".

On form submission, a Pending User account is created, and a link to finish account setup emailed to the user:

Successful submission of Create User Form

Following the invite link brings up a form to complete account setup, which must be completed to finish creating the new account:

Complete Account Setup Form

Pending user invites may be deleted on the Users page by clicking the trash icon that corresponds to the entry in the Pending Users list:

Pending Users List

Modifying a User's Role

Admin users can modify other users' roles using the Edit User page.

To edit a user's role, click the "edit" icon corresponding to the user's entry on the Users page.

Users page with active, pending, and inactive user lists

The Edit User page provides the ability to edit:

• the user's role (Admin/Publish/Read)
• whether the user is active (see Deactivating a User)

Edit User page

Modification Edge Cases

Like deleting a crate version, modifying a user's role is a powerful tool with some potentially sharp edges.

For example:

1. Publish user publishes a crate, and becomes its owner
2. The same user is downgraded to Read
3. The user is now the owner of the crate, but cannot publish new versions to it

In the example, an Admin user would need to add another owner to the crate, at which point the new owner would be able to publish new versions to it.

In the current implementation, Admin users have the ability to modify their own role, after which they would not be able to reverse the course, having forfeited the ability to modify user roles. (It's not permitted, however, to downgrade the last remaining Admin user, stranding the registry without anyone left to change its settings.)

Feel free to contact us on Zulip or via email if you are encountering tricky issues.

Deactivating a User

Deactivating a user revokes that user's permissions to the registry, akin to "deleting" the user, while leaving the user's published crate versions and other data intact.

To deactivate a user, click the "edit" icon on the Users page to navigate to the Edit User page, toggle the "Active" field to "off", and save changes:

Deactivating a user

Deactivation Semantics

When a user is deactivated:

• All active sessions are immediately logged out
• All auth tokens are immediately revoked
• The user is no longer able to log in to Shipyard.rs

However, any crate versions published by the user, as well as other kinds of records, will remain in place.

Crate Ownership

A deactivated user will remain listed as an owner of any crates they previously had ownership over.

SSH Keys

Currently, a deactivated user's SSH public keys are not automatically disabled. This allows ongoing SSH-based access to the crate index metadata using keys that were adding to the users' Gitea account while they were active. Until a deactivated user's SSH keys are automatically disabled upon deactivation, please contact support if you need assistance manually disabling a user's SSH keys.

Reactivating a User

A deactivated user may also be reactivated via the Edit User page, by toggling the "Active" field to "on."

If reactivated, a deactivated user's previous auth tokens will begin working again.

Google Workspace Integration

Shipyard.rs' Google Workspace Integration enables you to grant access to your organization account to all users with an email address from your Workspace domain name.

When enabled, Shipyard.rs accounts will be created on demand as new users log in for the first time, without needing to have used the "Invite User" process to initiate account creation.

How It Works

The Google Workspace Integration relies on Google to authenticate users, using the Oauth2.0 protocol. Once authenticated, the domain name of the user's email address is checked against all Workspace Domain settings enabled for Shipyard.rs organization accounts. If there is a match between domain names, the user is granted access to the Shipyard.rs organization account with the matching domain name.

Hypothetical Example

1. The "Rust Evangelism Strikeforce" organization creates a Shipyard.rs account and subscribes to the Pro plan.

2. Separately, the organization uses Google Workspace to host its email for the domain name rust-evangelism-strikeforce.org.

3. Admin user segfaults_not_even_once@rust-evangelism-strikeforce.org sets the Workspace Domain setting of their Shipyard.rs account to "Enabled" for the domain name rust-evangelism-strikeforce.org.

4. A junior developer at "Rust Evangelism Strikeforce", borrowck4eva@rust-evangelism-strikeforce.org, visits Shipyard.rs for the first time, clicking the "Login With Google" button to authenticate.

5. Shipyard.rs servers match the domain name of borrowck4eva@rust-evangelism-strikeforce.org's email address to the Workspace Domain setting of the organization's account, create a new account for the user, and grant access to the registry.

Modifying the Workspace Domain Setting

The Workspace Domain setting can be modified using the Settings page.

First, click the "Edit" button to modify the setting:

Click "Edit" to begin modifying the Workspace Domain Setting

Initially, the form will be in the "Disabled" state:

Workspace Domain form in "Disabled" state. Clicking "Save" with the form in this state would disable the Workspace Integration for the account.

To enable the feature, toggle the switch button to "On" and set the domain name to your Google Workspace domain:

Workspace Domain form in "Enabled" state. Clicking "Save" with the form in this state would enable the Workspace Integration feature using the specified domain name.

Finally, click the "Save" button to make the change.

To disable the Google Workspace Integration when it had previously been enabled, perform the same process in reverse, toggling the switch to "Off" position and saving the form in a "Disabled" state.

Login and Account Creation

When the Google Workspace Integration feature is enabled for a given domain, new users may access the organization's Shipyard.rs registry without performing the "Invite User" step to initiate account creation.

To create a new account, go to the Login page, and click the "Login With Google" button:

Shipyard.rs Login form. Use the "Login With Google" button to access an organization's account as a Workspace user.

At the Oauth2.0 prompt, authenticate with Google for an email address with the same domain name as the organization's Workspace Domain setting:

Google Oauth2.0 Consent Prompt

New Accounts Given "Read" Role

Accounts created on demand for a new Workspace user will be given the "Read" role. "Read" users can download crates and view crate docs at Shipyard.rs, but cannot publish crates or modify account settings (more about roles). The user's role can later be modified.

Requires Pro

The Google Workspace Integration feature requires a subscription to Shipyard.rs Pro.

Domain Name Limitations

For security reasons, some categories of domain names are not permitted to be used as Workspace Domain values:

• the domain name is validated to have an ICANN-delegated suffix
• domain names from public email services (e.g. hotmail.com) are not permitted
• Google-owned domain names are not permitted
• only root domain names allowed (no subdomains)
• domain name must be unique across all Shipyard.rs organization's that have enabled the Google Workspace Integration

Account Settings

The Settings page can be used to modify organization name, user email, and other account settings:

Settings Page

About

Shipyard.rs is run by (and this documentation is authored by) Jonathan Strong, a developer in Washington, D.C.

Jonathan works as a Rust developer for Keyrock, a cryptocurrency market maker based in Brussels, Belgium.

Email

Jonathan Strong can be reached via email at jstrong-at-shipyard-dot-rs.

For general inquiries, please email support@shipyard.rs.

Zulip Server

Shipyard.rs hosts a Zulip Server for support questions, feedback, and announcements.