skill-fedora-mirror-injections

Fedora Mirror Injections (AzL4 Stage 1 dependency gaps)

Diagnose BuildRequires resolution failures in AzL4 Stage 1 (*-bootstrap-mirror) builds, decide which are real prod-mirror gaps, and backfill the missing RPMs via azl4-stage1-injections.yaml in azl-infra.

Validated on the mirror_bootstrap0623 round (draft PR #27888; 34-package retest all passed).

Repos You Need

• azl-infra — holds the injections file and mirror tooling. Clone from: https://dev.azure.com/mariner-org/polar/_git/azl-infra (Azure DevOps org mariner-org, project polar). The relevant tree is scripts/fedora_mirror/.
• azurelinux — the package specs (for spec-fix cases). Clone from: https://github.com/microsoft/azurelinux.git (branch 4.0).

When to Use This Skill

Use when the user asks to:

• Triage a stage-1 / azl4-bootstrap build failure or triage summary.
• Find what RPMs are missing from the prod Fedora updates mirror.
• Add / verify entries in the injections file.
• Decide if a failure needs a mirror injection, a retest, or a spec fix.
• Open a PR adding entries to the injections file.

Mental Model — a resolution failure IS a mirror gap

Stage 1 (azl4-bootstrap-mirror) builds resolve all BuildRequires from the prod Fedora mirror alone (kojifedoramirror.blob.core.windows.net). Koji-built RPMs from the same build set are not used as build deps, so build ordering is irrelevant — a resolution failure means the mirror lacks that exact NVR/crate version, even if the provider built fine in the same run. Do not dismiss these as build-order races.

Repo split: fedora-43-everything (base release) is frozen and consumed live from upstream; fedora-43-updates drifts and is what gets mirrored into blob snapshots — so it is what needs injections.

The Eviction Pattern (root cause)

1. A build pulled NVR X from fedora-43-updates.
2. Fedora bumped to a newer major/minor Y; X demoted from updates repodata.
3. X never entered the frozen releases/43/Everything.
4. X is still recoverable from kojipkgs (Fedora's build archive).

The spec's BuildRequires upper bound (>= X < next~) can't accept Y → inject X back from kojipkgs.

Procedure

1. Scan the triage summary for resolution signatures

import json, re
d = json.load(open('triage-summary-*.json'))
pat = re.compile(r'nothing provides|No match for argument|Could NOT find|missing:', re.I)
for b in d['buckets']:
    for t in b['tasks']:
        if pat.search(t['shortSummary']):
            print(b['name'], t['taskID'], t['package'])

Check all buckets — gaps hide outside the obvious one.

2. Get EXACT pins from koji root.log (never trust summary ranges)

curl -sk "https://<koji-host>/koji/getfile?taskID=<childID>&name=root.log&offset=0" \
  | grep -iE "nothing provides|No match for argument|conflicting requests"

crate(icu_collections) >= 2.1.1 < 2.2.0~ proves the mirror has the name at 2.2.0 but not the needed version (the eviction signature).

3. Classify (not everything is injectable)

Signature	Class	Action
nothing provides ... < X~ / No match for argument: crate = X	Real mirror gap	Inject X from kojipkgs
Curl error (28): Timeout from the blob mirror	Transient infra	Retest, don't inject
Could NOT find <LIB> at %build, lib IS in mirror	Spec bug	Fix the azurelinux spec (BuildRequires)
Needs version requiring a whole-stack realignment	Version skew	Upstream owner fix

0623 exclusions: grpc (libpfm-devel current, just add BuildRequires: libpfm-devel), qt6-qthttpserver (qt6-qtbase-private-devel = 6.10.2 vs mirror qtbase 6.9.2 → qt6 stack realignment).

4. Find the NVR + full subpackage fan-out on kojipkgs

Rust: SRPM is rust-<crate>; binaries are rust-<crate>-devel, rust-<crate>+<feature>-devel — all noarch, one shared NVR. Inject every subpackage of the NVR.

# versions
curl -s "https://kojipkgs.fedoraproject.org/packages/<name>/" | grep -oE '>[0-9][^<]*/<' | sed -E 's/>//;s/\/<//' | sort -V
# releases for a version
curl -s "https://kojipkgs.fedoraproject.org/packages/<name>/<ver>/" | grep -oE '>[0-9][^<]*\.fc[0-9]+/<' | sed -E 's/>//;s/\/<//'
# noarch subpackages for the NVR
curl -s "https://kojipkgs.fedoraproject.org/packages/<name>/<ver>/<rel>/noarch/" | grep -oE 'href="[^"]+\.noarch\.rpm"' | sed -E 's/href="//;s/"//' | sort

5. Verify the transitive closure (avoid peeling the onion)

rpm -qp --requires <rpm> | grep -E 'crate\(' | grep '<'   # upper-bounded deps
rpm -qp --provides <rpm> | grep -E '^crate\('

Confirm each < X~ dep is satisfied by the injected set or current mirror (e.g. siblings bounded < 3.0.0~ are fine against a 2.2.0 mirror; only < 2.2.0~ pins need injecting). Iterate to a fixpoint. For masked gaps (downstream packages that never reached builddep), run a full-set offline depsolve against mirror + injections before rerunning. Note Rust %generate_buildrequires produces BRs at build time — a static depsolve misses them.

6. Add entries to the injections file

File: azl-infra/scripts/fedora_mirror/injections/azl4-stage1-injections.yaml

- {source: rust-icu_collections, name: rust-icu_collections-devel, version: 2.1.1, release: 5.fc43, arch: noarch}

• source = SRPM, name = binary RPM (set source whenever they differ — always for rust crates; binaries live under the SRPM dir on kojipkgs).
• noarch fans out to x86_64 + aarch64; arch-specific binaries get one entry per arch.
• File is sorted by (source, name, arch), +feature-devel before bare -devel. Insert in place; do not append. Keep entries comment-free (rationale goes in the PR/commit).

7. Pre-flight validation (offline, no Azure)

No standalone --dry-run; reuse the script's own functions:

import yaml, inject_packages as ip, logging
logging.disable(logging.CRITICAL)
inj = yaml.safe_load(open('injections/azl4-stage1-injections.yaml'))
repos = yaml.safe_load(open('repos.yaml'))
new = [p for p in ip.expand_packages(inj, repos) if p['source_name'] in {'rust-...'}]
print('HEAD failures:', sum(0 if ip.head_check(p['url'])[0] else 1 for p in new))  # want 0

8. Know how the run behaves

inject_packages.py is add-only + idempotent (dedup by dest path, SKIPs existing). No delete path. Storage: account kojifedoramirrorsa, container fedora-mirror; blob path snapshots/<snap>/<repo>/<arch>/Packages/<first-letter>/<file>.rpm. Snapshot caveat: the injection only helps if it lands in the snapshot the target consumes — confirm the file's snapshot: matches before running.

9. Open the PR

Commit the injection entries on a branch off azl-infra main and open the PR — this is the deliverable.

• Branch: <user>/<task>-<date> off latest origin/main.
• Diff: additions-only to azl4-stage1-injections.yaml (no existing lines touched), entries comment-free and in the correct alphabetical slot.
• PR description carries the rationale: the eviction pattern, exact NVRs, which failing package(s) each entry unblocks, closure notes, the target snapshot:, and any packages deliberately excluded (spec bugs / version skew) with why.
• Evidence: paste the Step 7 offline HEAD-check result (want 0 failures).
• Open as draft first if you want review before it's merge-ready.

After merge (optional): once the injection lands in the prod snapshot, retest the affected packages (include the transient-timeout ones; exclude spec/version-skew) with a body like:

{
  "repoUri": "https://github.com/microsoft/azurelinux.git",
  "branch": "4.0",
  "commitSha": "",
  "packageTarget": "azl4-bootstrap-mirror",
  "packages": ["rust-icu_normalizer", "rust-rkyv", "..."],
  "isScratchBuild": true
}

Transient timeout vs. real gap

A burst of Curl error (28): Timeout is usually a time-windowed blob outage, not a bad package. Confirm: (1) same RPM failing across many packages → systemic; (2) failures pack into a short window (0623: 29 in ~78 min) → transient; (3) most other builds in that window succeeded (0623: 97%) → mirror was up. If all three hold, retest — do not inject.

Gotcha — version float (pin the spec, don't delete)

When old + new both exist (old in frozen base, new in updates/mirror), dnf picks the highest and an old-API spec fails (e.g. rust-uucore 0.0.27 vs 0.7.0). Deleting from the mirror is the wrong lever (sync re-adds it, no delete tooling, diverges from upstream). Fix: pin the spec's crate(uucore) = 0.0.27 (+ lockstep siblings uucore_procs, uuhelp_parser). Since 0.0.27 lives in the frozen live base repo, the pin needs zero mirror changes.

Reference

• Repo azl-infra: https://dev.azure.com/mariner-org/polar/_git/azl-infra
• Injections file: azl-infra/scripts/fedora_mirror/injections/azl4-stage1-injections.yaml
• Schema/template: .../injections/example.yaml
• Tooling: .../fedora_mirror/{inject_packages,snapshot_repos}.py, repos.yaml, _common.py
• kojipkgs archive: https://kojipkgs.fedoraproject.org/packages/
• Prod mirror: https://kojifedoramirrorsa.blob.core.windows.net/fedora-mirror/
• Worked example: draft PR #27888 in azl-infra