# pkgatlas — SPEC v1 **One question, answered well:** given a build-time dependency signal, which package do I install on *(distro, release, arch)*? Signals: a linker flag (`-lcurses`), a missing header (`ncurses.h`), a pkg-config name (`ncurses`), or a bare file path. **Primary consumer: saggar** (auto-magic dependency detection). Secondary: any build agent. --- ## 1. Product shape Build once, serve forever. ``` pkgatlas-build ──> snapshot .sqlite ──┬──> pkgatlas-serve (HTTP) (ETL) (the artifact) └──> published file (local query) ``` The **artifact is the product**. Each build emits a versioned SQLite database per (manager, release, arch). It is served over HTTP *and* published as a downloadable file. Why this matters: a consumer can pin an exact snapshot for reproducible builds, and can query locally with zero network dependency mid-build. A resolver that cannot break your build is worth more than a fast one. **Non-goals (v1)** - No web UI. Human browsing can come later as a separate app over the same API. - No `apk` / `nix` (deferred — see §7). - No language-registry data (npm/cargo/pypi). That is deps.dev's job. - **No fuzzy/heuristic matching.** All three v1 sources are exact file→package lookups. If a query cannot be answered exactly, return empty; never guess. --- ## 2. Verified upstream sources Every fact below was fetched and inspected. Sizes are real observed responses. | Manager | Source | Observed size | |---|---|---| | deb | `Contents-.gz` | 12.7 MB (trixie amd64), 47.6 MB (noble riscv64), 18.4 MB (sid amd64) | | rpm | `filelists.xml.zst` | 39.7 MB (openSUSE TW), 48.9 MB (Fedora 43) | | pacman | `.files.tar.gz` | 1.5 MB (core), 51.0 MB (extra), 0.1 MB (multilib) | ### 2.1 deb — Debian / Ubuntu ``` Debian https://deb.debian.org/debian/dists/{release}/{component}/Contents-{arch}.gz Ubuntu amd64 https://archive.ubuntu.com/ubuntu/dists/{release}/Contents-{arch}.gz Ubuntu other https://ports.ubuntu.com/ubuntu-ports/dists/{release}/Contents-{arch}.gz ``` > **Trap:** Ubuntu non-amd64 is **not** on `archive.ubuntu.com`. > `dists/noble/Contents-riscv64.gz` returns 404 there; the same path on > `ports.ubuntu.com/ubuntu-ports/` returns 200. Ubuntu supports > `amd64 arm64 armhf i386 ppc64el riscv64 s390x` — so this affects most arches. Format: gzip-compressed text, one line per path: ``` /[,/...] ``` ``` usr/include/ncurses.h libdevel/libncurses-dev usr/lib/x86_64-linux-gnu/libcurses.so libdevel/libncurses-dev usr/lib/x86_64-linux-gnu/libcurl.so libdevel/libcurl4-gnutls-dev,libdevel/libcurl4-openssl-dev ``` - Paths have **no leading slash**. - `prefix` is a repo/component label (`main`, `universe`, `libdevel`). It does **not** indicate binary-vs-source. - The right-hand side is **binary package names** — directly installable via `apt-get`. ### 2.2 rpm — openSUSE / Fedora ``` openSUSE https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml Fedora https://dl.fedoraproject.org/pub/fedora/linux/releases/{rel}/Everything/{arch}/os/repodata/repomd.xml ``` `repomd.xml` lists content sections. Read `` → its `` → that is `filelists.xml.zst` (zstd-compressed). Verified structure: ```xml /usr/bin/0ad /usr/lib64/0ad /usr/lib64/0ad/libAtlasUI.so ``` - **The package name is inline** (`name="0ad"`) — no join against `primary.xml` needed. - `type="dir"` marks directories; skip them. - Paths **do** have a leading slash (unlike deb and pacman). - Names here are binary package names (`ncurses-devel`), verified against the real file. > **Note:** Fedora also publishes `filelists_db` (a prebuilt SQLite, 64.6 MB → 420 MB) and > `filelists_zck` (zchunk). The `_db` is a genuine SQLite but awkward: `packages(pkgKey, > pkgId)` where **pkgId is a SHA256 checksum, not a name** (needs a `primary_db` join), and > `filelist(pkgKey, dirname, filenames, filetypes)` stores filenames **slash-separated in a > single field**. v1 parses the XML instead. See §6 for zchunk. ### 2.3 pacman — Arch ``` https://geo.mirror.pkgbuild.com/{repo}/os/{arch}/{repo}.files.tar.gz ``` Repos: `core`, `extra`, `multilib` (plus `*-testing`). Format: tar.gz, one directory per package named `--/`, containing `files` and `desc`. The `files` member: ``` %FILES% usr/ usr/bin/ usr/bin/clear usr/include/ncurses.h usr/lib/libcurses.so usr/lib/libncurses.so.6 ``` - Paths have **no leading slash**; a trailing `/` means directory. - Verified to contain the unversioned devel symlink `usr/lib/libncurses.so`, which is exactly what `-lcurses` needs. > **Trap:** Arch's *web API* `?file=` parameter is a **no-op** — it returns unfiltered > results (`0ad` first for any query). Do not use it. The data above is the correct source. > (`?q=` does filter, but answers a different question.) --- ## 3. Normalization Applied to every source so downstream storage is uniform: 1. Strip any leading `/`. 2. Drop entries whose last character is `/` (directories). 3. `basename = path.Base(path)` — the hot lookup key. 4. Classify `kind`: | kind | rule | |---|---| | `lib` | `*.so` or `*.so.N`, `*.so.N.N.N` | | `static` | `*.a` | | `header` | `*.h`, `*.hpp`, `*.hh` | | `pc` | `*.pc` | | `cmake` | `*.cmake` | | `m4` | `*.m4` | | `bin` | path begins `usr/bin/` or `bin/` | | `other` | everything else | 5. **Keep only dev-relevant kinds** (`lib`, `static`, `header`, `pc`, `cmake`, `m4`, `bin`). Debian trixie `main` has 1,853,320 total entries; the dev-relevant subset is ~313,749, and just `.so`+`.pc` is ~68,509. The gzipped dev subset is ~2.3 MB. Do not store the full index — it is 6x the size for no benefit. --- ## 4. Identifier model A query is exactly one of: | kind | input | lookup | |---|---|---| | `lib` | `curses` (from `-lcurses`) | file with basename `libcurses.so` | | `header` | `ncurses.h` | file with basename `ncurses.h` | | `pc` | `ncurses` | file with basename `ncurses.pc` | | `file` | a path or basename | exact match on `path` or `basename` | The linker error names the exact flag (`-lpcre2-8`, never `-lpcre2`), so `lib` queries are reliable as given: strip the leading `-l`, prefix `lib`, suffix `.so`. ### Ambiguity must be surfaced, never resolved silently The same flag legitimately maps to different packages: | query | distro | result | |---|---|---| | `-lcurses` | debian trixie | `libncurses-dev` | | `-lncurses` | openSUSE | `ncurses-devel` (`/usr/lib64/libncurses.so`) | | `-lcurses` | openSUSE | `termcap-devel` (`/usr/lib64/curses/libcurses.so`) | | `-lcurl` | debian trixie | `libcurl4-gnutls-dev`, `libcurl4-openssl-dev` | Consequences: - Responses always contain an **array**, plus `ambiguous: bool`. - The same flag giving **different answers on different distros** is precisely why per-distro ingestion is mandatory. Derivatives cannot be assumed to inherit their parent's answers for file→package resolution. --- ## 5. Schema and API See [`schema.sql`](./schema.sql) for the full DDL. ### API ``` GET /v1/resolve?lib=curses[&distro=&release=&arch=] GET /v1/resolve?header=ncurses.h[&...] GET /v1/resolve?pc=ncurses[&...] GET /v1/resolve?file=libcurses.so[&...] GET /v1/snapshots GET /v1/healthz ``` Exactly one of `lib|header|pc|file` is required. Filters are optional and repeatable; **omitting them returns all known matches** (that is the point of a precomputed catalog). ```json { "query": {"kind": "lib", "value": "curses"}, "resolved": [ { "manager": "deb", "distro": "ubuntu", "release": "24.04", "arch": "riscv64", "path": "usr/lib/riscv64-linux-gnu/libcurses.so", "package": "libncurses-dev", "install": "apt-get install -y libncurses-dev" } ], "ambiguous": false, "truncated": false } ``` Contract: - `resolved` is always an array. Empty means **genuinely unavailable on that distro** — a real answer the consumer can act on, not an error. - `install` is always populated (per-manager command). - `ambiguous: true` signals the consumer must choose; it must not pick silently. - `truncated: true` when a result limit clipped the response. ### CLI ``` pkgatlas-build --source deb --release trixie --arch amd64 --out out/trixie-amd64.sqlite pkgatlas-build --source rpm --distro opensuse --release tumbleweed --arch x86_64 --out ... pkgatlas-build --source pacman --repo core --repo extra --arch x86_64 --out ... pkgatlas-serve --db out/ --addr :8080 ``` --- ## 6. Refresh strategy - **Conditional GET** on every upstream fetch (`If-Modified-Since` / `ETag`). Debian's `Release`/`InRelease` files give a cheap timestamp to compare before pulling a 47 MB `Contents` file. Verified reachable: `dists/trixie/Release` (138,612 B). - **zchunk** (RPM): Fedora publishes `*_zck` variants supporting ranged/delta downloads. If usable, a weekly refresh pulls only changed chunks instead of ~40-90 MB per repo. **Evaluate after v1 works**; do not block on it. - **Mirror etiquette:** descriptive User-Agent, prefer mirrors over primary hosts, back off on 429/503. This runs weekly forever — do not get rate-limited off Debian's CDN. --- ## 7. Deferred - **apk (Alpine):** `https://dl-cdn.alpinelinux.org/alpine/v{ver}/{repo}/{arch}/APKINDEX.tar.gz` (473 KB). Reachable and cheap. **Not yet verified to carry file-level paths** — confirm before implementing. - **nix:** needs the nix-index backend; `search.nixos.org` serves only a JS shell (1.7 KB). Unverified. - **zchunk delta refresh** (§6). --- ## 8. Implementation constraints - **Go 1.25.x.** Module path set to the final repo URL. - **stdlib `net/http` only** — no web framework. - **`modernc.org/sqlite`** (pure Go; `rusqlite`-style CGO deps are not acceptable). - **`github.com/klauspost/compress/zstd`** (pure Go) for `.zst` inputs. - **`CGO_ENABLED=0 go build ./...` must succeed.** - Unit tests must not touch the network; parse from small checked-in fixtures (`testdata/`). Real-source tests belong behind an opt-in build tag. - Streaming, not buffering: RPM `filelists` is 875 MB uncompressed. Use `xml.Decoder` token streaming and a batched SQLite writer. Never `io.ReadAll` a source. ### Interfaces ```go type Snapshot struct { Manager, Distro, Release, Arch, Repo string UpstreamDate, FetchedAt, SourceURL string } type FileEntry struct { Path, Basename, Kind, Repo, Package, Version string } type Source interface { Name() string // Versions lists ingestable (release, arch, repo) combinations. Versions(ctx context.Context) ([]Snapshot, error) // Fetch streams one snapshot's index. Caller closes. Fetch(ctx context.Context, s Snapshot) (io.ReadCloser, error) // Parse streams entries to emit. Must not buffer the whole document. Parse(ctx context.Context, r io.Reader, s Snapshot, emit func(FileEntry) error) error } ``` Adding a manager = one file implementing `Source` + a fixture + a case in the builder.