Clean more aggressively in CI
The Windows x86_64 gnu CI job is running dangerously low on disk space. This PR adds another step to delete test output more aggressively. The test output with x86_64-pc-windows-gnu is nearly 9.5GB. The benchmark step is adding about 1GB of space (unfortunately it is rebuilding cargo, which may be hard to avoid without a workspace).
Eventually we should probably look at figuring out how to reduce the amount of disk space used by the testsuite. Perhaps something like #9701 (see comments there). Or, making aggressive changes to the tests themselves. Many tests can probably be changed to use `cargo check` instead of `cargo build` (or maybe even `cargo tree`). We can default to not generating debuginfo. Or perhaps there are other changes to put the tests on a diet.
Fix git2 safe-directory disable
The call to `set_verify_owner_validation` was not getting called unless a network configuration was found. This means in the common case that `cargo new` will fail when there is a safe-directory error. This fixes the issue by making sure that `set_verify_owner_validation` is called before the early-exits in `init_git_transports`.
Fixes#11365
fix(publish): Block until it is in index
Originally, crates.io would block on publish requests until the publish
was complete, giving `cargo publish` this behavior by extension. When
crates.io switched to asynchronous publishing, this intermittently broke
people's workflows when publishing multiple crates. I say interittent
because it usually works until it doesn't and it is unclear why to the
end user because it will be published by the time they check. In the
end, callers tend to either put in timeouts (and pray), poll the
server's API, or use `crates-index` crate to poll the index.
This isn't sufficient because
- For any new interested party, this is a pit of failure they'll fall
into
- crates-index has re-implemented index support incorrectly in the past,
currently doesn't handle auth, doesn't support `git-cli`, etc.
- None of these previous options work if we were to implement
workspace-publish support (#1169)
- The new sparse registry might increase the publish times, making the
delay easier to hit manually
- The new sparse registry goes through CDNs so checking the server's API
might not be sufficient
- Once the sparse registry is available, crates-index users will find
out when the package is ready in git but it might not be ready through
the sparse registry because of CDNs
So now `cargo` will block until it sees the package in the index.
- This is checking via the index instead of server APIs in case there
are propagation delays. This has the side effect of being noisy
because of all of the "Updating index" messages.
- This blocks by default but there is an unstable `publish.timeout` config field that will disable blocking when set to 0. See #11222 for stablization
Blocking is opt-out as that is the less error prone case for casual users while those doing larger integrations are also likely to do the testing needed to make more complicated scenarios work where blocking is disabled.
Right now we block after the publish. An alternative would be to block until all dependencies are in the index which makes the blocking only happen when needed
- Blocking on dependencies can be imprecise to detect when to block vs propagate an error up
- This is the less error prone case for users. For example I recently publish a crate in one tab and immediately switched to another tab to use it and this only worked because `cargo-release` blocked until it was ready to use
In reviewing this change, be sure to look at the individual commits
- The first makes it possible to write the tests for this
- The second adds a test that shows the current behavior
- The third updates the test to the expected behavior, showing all of this works
In addition to the publish tests:
- We want to maximize the nightly-to-stable time to collect feedback
- We will put this in TWiR's testing section to raise visibility
Fixes#9507
Add Accept-Encoding request header to enable compression
### What does this PR try to resolve?
Cargo does not request compression from servers. Enabling compression can save bandwidth and improve performance.
### How should we test and review this PR?
I validated locally that the header was being sent using a local proxy (Fiddler). It sent `Accept-Encoding: deflate, gzip` on Windows.
Update contrib docs for highfive transition
As of #11293, this repo is now using triagebot (AKA `@rustbot)` for auto-assignment of PRs. This PR updates the contributor documentation to accommodate this change.
Fix confusing error messages when using -Zsparse-registry
Built-in replacements of crates.io have confusing a description:
> crates.io index (which is replacing registry `crates-io`)
This adds a special case for built-in source replacements of `crates-io`. User-defined replacements of crates.io continue to use the existing description.
Also fixes the test framework `__CARGO_TEST_CRATES_IO_URL_DO_NOT_USE_THIS` variable to strip the `sparse+` prefix when checking if a url `is_crates_io`.
Fixes#11277
Fix 410 gone response handling
This changes the sparse-registry support for the 410 "Gone" HTTP response code. This is out of sync with the [load function](8adf1df292/src/cargo/sources/registry/http_remote.rs (L375)) mentioned in the comment. I assume it is supposed to be 410 and not 401 since 401 is "Unauthorized", and that doesn't signify that the resource is "not found".
r? `@arlosi`
Fix inequality in "stale mtime" log messages
### What does this PR try to resolve?
If `CARGO_LOG=cargo::core::compiler::fingerprint=info` is set, cargo will print log messages that are useful for finding out why something is being recompiled. E.g. if the modification time (mtime) of a source file is newer than the cached build result, it will print something like:
```
stale: changed "/host//home/runner/.cargo/registry/src/github.com-1285ae84e5963aae/proc-macro2-1.0.47/build.rs"
(vs) "/host/home/runner/target/release/build/proc-macro2-45f04ea9067a46ed/output"
FileTime { seconds: 1666559031, nanos: 16426033 } != FileTime { seconds: 1666559080, nanos: 324117075 }
```
However, the `!=` in the log message is misleading, as equality is not the [criterion that's actually used](071eeaf210/src/cargo/core/compiler/fingerprint.rs (L1761)) to determine when to rebuild.
This PR fixes that by changing `!=` to `<`, corresponding to the actual criterion.
### How should we test and review this PR?
Look at the [criterion](071eeaf210/src/cargo/core/compiler/fingerprint.rs (L1761)) I linked and trace `stale_mtime` in the log message back to `path_mtime` in said criterion to see why it has to be `<`.
Improve the error message if `publish` is `false` or empty list
### What does this PR try to resolve?
close https://github.com/rust-lang/cargo/issues/11262
Improve the error message if `publish` is the false or empty list.
Say `publish` is set to `false` or an empty list in Cargo.toml and prevents publishing.
### How should we test and review this PR?
Unit test
### Additional information
If there was a way we could tell `publish` to be empty or `false`, I think it would get better. If you know an easy way to implement it, please feel free to comment.
Add test for deleted index entry
This adds a test for when an entry is deleted from the index. This is done on crates.io occasionally for things like copyright takedown requests.
This behavior was tripping a debug assert which this removes. I'm not entirely certain why the debug assert is there, but I think it is not correct, since this is obviously a scenario where there might be a cached file, but the index responds with NotFound.
Fix publishing with a dependency on a sparse registry
### What does this PR try to resolve?
On publishing, the `registry_index` field was being set to the registry URL without the `sparse+` prefix. During the verification build, Cargo would attempt to fetch the registry as a `git` registry. Caused by #11209.
Fixes#11263.
### How should we test and review this PR?
Added a test that fails without the change.
Add missing edition
When I read this doc, I found that we are missing the edition key in this manifest. I think we should add it because` cargo new` always does this.
fix(publish): Check remote git registry more than once post-publish
With `publish-timeout` on remote git registries, we were checking once and never checking again. There were 3 layers of guards preventing the cache from being updating
- `needs_update`, handled via `invalidate_cache`
- `config.updated_sources()`,. handled by removing from the HashSet
- `updated`, inaccessible
This change collapses `updated` into `config.updated_sources()`, allowing the cache to be updated
Tested by publishing `epage-publish-test`. After about 7 registry updates, it succeded. Before, it just hit the timeout of 60s. No tests are added as we don't have the plumbing setup to be able to control when the test remote git registry publishes. All of our tests focus on the remote http registry.
Fixes#11253
With `publish-timeout` on remote git registries, we were checking once
and never checking again. There were 3 layers of guards preventing the
cache from being updating
- `needs_update`, handled via `invalidate_cache`
- `config.updated_sources()`,. handled by removing from the HashSet
- `updated`, inaccessible
This change collapses `updated` into `config.updated_sources()`,
allowing the cache to be updated
Tested by publishing `epage-publish-test`. After about 7 registry
updates, it succeded. Before, it just hit the timeout of 60s. No tests
are added as we don't have the plumbing setup to be able to control when
the test remote git registry publishes. All of our tests focus on the
remote http registry.
Fixes#11253
Some tiny refactors around `ops::cargo_compile`
### What does this PR try to resolve?
Some tiny refactors I found during polishing documentations.
- Extract `CompileFilter` and `Packages` from `ops::cargo_compile` to their own modules.
- Remove `FilterRule::try_collect` as its intent is not clear, and we don't need this indirection.
- Remove `CompileOptions::local_rustdoc_args`, which is obsolete since 1ef954ea.
Remove sparse+ prefix for index.crates.io
### What does this PR try to resolve?
#11209 changed how sparse registry URLs are stored in a `SourceId` to remove the `sparse+` prefix, however the URL was not updated.
Fixes#11246
### How should we test and review this PR?
Run a command that requires sparse registry `cargo -Z sparse-registry update --dry-run`
Tests can't really be added for this case, since the `index.crates.io` URL was incorrect, and tests shouldn't access the network.
r? `@Eh2406`
Eric Huss
bors
Rageking8
Arlo Siemsen
Weihang Lo
hi-rustin
smheidrich
Ed Page
