pkgatlas/docs/SPEC.md

pkgatlas — SPEC v1

One question, answered well: given a build-time dependency signal, which package do I install on (distro, release, arch)?

Signals: a linker flag (-lcurses), a missing header (ncurses.h), a pkg-config name (ncurses), or a bare file path.

Primary consumer: saggar (auto-magic dependency detection). Secondary: any build agent.

---

1. Product shape

Build once, serve forever.

pkgatlas-build  ──> snapshot .sqlite  ──┬──> pkgatlas-serve (HTTP)
   (ETL)              (the artifact)    └──> published file (local query)

The artifact is the product. Each build emits a versioned SQLite database per (manager, release, arch). It is served over HTTP and published as a downloadable file.

Why this matters: a consumer can pin an exact snapshot for reproducible builds, and can query locally with zero network dependency mid-build. A resolver that cannot break your build is worth more than a fast one.

Non-goals (v1)

• No web UI. Human browsing can come later as a separate app over the same API.
• No apk / nix (deferred — see §7).
• No language-registry data (npm/cargo/pypi). That is deps.dev's job.
• No fuzzy/heuristic matching. All three v1 sources are exact file→package lookups. If a query cannot be answered exactly, return empty; never guess.

---

2. Verified upstream sources

Every fact below was fetched and inspected. Sizes are real observed responses.

Manager	Source	Observed size
deb	Contents-<arch>.gz	12.7 MB (trixie amd64), 47.6 MB (noble riscv64), 18.4 MB (sid amd64)
rpm	filelists.xml.zst	39.7 MB (openSUSE TW), 48.9 MB (Fedora 43)
pacman	<repo>.files.tar.gz	1.5 MB (core), 51.0 MB (extra), 0.1 MB (multilib)

2.1 deb — Debian / Ubuntu

Debian        https://deb.debian.org/debian/dists/{release}/{component}/Contents-{arch}.gz
Ubuntu amd64  https://archive.ubuntu.com/ubuntu/dists/{release}/Contents-{arch}.gz
Ubuntu other  https://ports.ubuntu.com/ubuntu-ports/dists/{release}/Contents-{arch}.gz

Trap: Ubuntu non-amd64 is not on archive.ubuntu.com. dists/noble/Contents-riscv64.gz returns 404 there; the same path on ports.ubuntu.com/ubuntu-ports/ returns 200. Ubuntu supports amd64 arm64 armhf i386 ppc64el riscv64 s390x — so this affects most arches.

Format: gzip-compressed text, one line per path:

<path><whitespace><prefix>/<pkg>[,<prefix>/<pkg>...]

usr/include/ncurses.h	libdevel/libncurses-dev
usr/lib/x86_64-linux-gnu/libcurses.so	libdevel/libncurses-dev
usr/lib/x86_64-linux-gnu/libcurl.so	libdevel/libcurl4-gnutls-dev,libdevel/libcurl4-openssl-dev

• Paths have no leading slash.
• prefix is a repo/component label (main, universe, libdevel). It does not indicate binary-vs-source.
• The right-hand side is binary package names — directly installable via apt-get.

2.2 rpm — openSUSE / Fedora

openSUSE  https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml
Fedora    https://dl.fedoraproject.org/pub/fedora/linux/releases/{rel}/Everything/{arch}/os/repodata/repomd.xml

repomd.xml lists content sections. Read <data type="filelists"> → its <location href> → that is filelists.xml.zst (zstd-compressed).

Verified structure:

<?xml version="1.0" encoding="UTF-8"?>
<filelists xmlns="http://linux.duke.edu/metadata/filelists" packages="52882">
<package pkgid="df97ba73..." name="0ad" arch="x86_64">
  <version epoch="0" ver="0.28.0" rel="1.5"/>
  <file>/usr/bin/0ad</file>
  <file type="dir">/usr/lib64/0ad</file>
  <file>/usr/lib64/0ad/libAtlasUI.so</file>
</package>

• The package name is inline (name="0ad") — no join against primary.xml needed.
• type="dir" marks directories; skip them.
• Paths do have a leading slash (unlike deb and pacman).
• Names here are binary package names (ncurses-devel), verified against the real file.

Note: Fedora also publishes filelists_db (a prebuilt SQLite, 64.6 MB → 420 MB) and filelists_zck (zchunk). The _db is a genuine SQLite but awkward: packages(pkgKey, pkgId) where pkgId is a SHA256 checksum, not a name (needs a primary_db join), and filelist(pkgKey, dirname, filenames, filetypes) stores filenames slash-separated in a single field. v1 parses the XML instead. See §6 for zchunk.

2.3 pacman — Arch

https://geo.mirror.pkgbuild.com/{repo}/os/{arch}/{repo}.files.tar.gz

Repos: core, extra, multilib (plus *-testing). Format: tar.gz, one directory per package named <pkgname>-<pkgver>-<pkgrel>/, containing files and desc.

The files member:

%FILES%
usr/
usr/bin/
usr/bin/clear
usr/include/ncurses.h
usr/lib/libcurses.so
usr/lib/libncurses.so.6

• Paths have no leading slash; a trailing / means directory.
• Verified to contain the unversioned devel symlink usr/lib/libncurses.so, which is exactly what -lcurses needs.

Trap: Arch's web API ?file= parameter is a no-op — it returns unfiltered results (0ad first for any query). Do not use it. The data above is the correct source. (?q= does filter, but answers a different question.)

---

3. Normalization

Applied to every source so downstream storage is uniform:

1. Strip any leading /.

2. Drop entries whose last character is / (directories).

3. basename = path.Base(path) — the hot lookup key.

4. Classify kind:

   kind	rule
   lib	*.so or *.so.N, *.so.N.N.N
   static	*.a
   header	*.h, *.hpp, *.hh
   pc	*.pc
   cmake	*.cmake
   m4	*.m4
   bin	path begins usr/bin/ or bin/
   other	everything else

5. Keep only dev-relevant kinds (lib, static, header, pc, cmake, m4, bin).

   Debian trixie main has 1,853,320 total entries; the dev-relevant subset is ~313,749, and just .so+.pc is ~68,509. The gzipped dev subset is ~2.3 MB. Do not store the full index — it is 6x the size for no benefit.

---

4. Identifier model

A query is exactly one of:

kind	input	lookup
lib	curses (from -lcurses)	file with basename libcurses.so
header	ncurses.h	file with basename ncurses.h
pc	ncurses	file with basename ncurses.pc
file	a path or basename	exact match on path or basename

The linker error names the exact flag (-lpcre2-8, never -lpcre2), so lib queries are reliable as given: strip the leading -l, prefix lib, suffix .so.

Ambiguity must be surfaced, never resolved silently

The same flag legitimately maps to different packages:

query	distro	result
-lcurses	debian trixie	libncurses-dev
-lncurses	openSUSE	ncurses-devel (/usr/lib64/libncurses.so)
-lcurses	openSUSE	termcap-devel (/usr/lib64/curses/libcurses.so)
-lcurl	debian trixie	libcurl4-gnutls-dev, libcurl4-openssl-dev

Consequences:

• Responses always contain an array, plus ambiguous: bool.
• The same flag giving different answers on different distros is precisely why per-distro ingestion is mandatory. Derivatives cannot be assumed to inherit their parent's answers for file→package resolution.

---

5. Schema and API

See schema.sql for the full DDL.

API

GET /v1/resolve?lib=curses[&distro=&release=&arch=]
GET /v1/resolve?header=ncurses.h[&...]
GET /v1/resolve?pc=ncurses[&...]
GET /v1/resolve?file=libcurses.so[&...]
GET /v1/snapshots
GET /v1/healthz

Exactly one of lib|header|pc|file is required. Filters are optional and repeatable; omitting them returns all known matches (that is the point of a precomputed catalog).

{
  "query": {"kind": "lib", "value": "curses"},
  "resolved": [
    {
      "manager": "deb",
      "distro": "ubuntu",
      "release": "24.04",
      "arch": "riscv64",
      "path": "usr/lib/riscv64-linux-gnu/libcurses.so",
      "package": "libncurses-dev",
      "install": "apt-get install -y libncurses-dev"
    }
  ],
  "ambiguous": false,
  "truncated": false
}

Contract:

• resolved is always an array. Empty means genuinely unavailable on that distro — a real answer the consumer can act on, not an error.
• install is always populated (per-manager command).
• ambiguous: true signals the consumer must choose; it must not pick silently.
• truncated: true when a result limit clipped the response.

CLI

pkgatlas-build --source deb --release trixie --arch amd64 --out out/trixie-amd64.sqlite
pkgatlas-build --source rpm --distro opensuse --release tumbleweed --arch x86_64 --out ...
pkgatlas-build --source pacman --repo core --repo extra --arch x86_64 --out ...
pkgatlas-serve --db out/ --addr :8080

---

6. Refresh strategy

• Conditional GET on every upstream fetch (If-Modified-Since / ETag). Debian's Release/InRelease files give a cheap timestamp to compare before pulling a 47 MB Contents file. Verified reachable: dists/trixie/Release (138,612 B).
• zchunk (RPM): Fedora publishes *_zck variants supporting ranged/delta downloads. If usable, a weekly refresh pulls only changed chunks instead of ~40-90 MB per repo. Evaluate after v1 works; do not block on it.
• Mirror etiquette: descriptive User-Agent, prefer mirrors over primary hosts, back off on 429/503. This runs weekly forever — do not get rate-limited off Debian's CDN.

---

7. Deferred

• apk (Alpine): https://dl-cdn.alpinelinux.org/alpine/v{ver}/{repo}/{arch}/APKINDEX.tar.gz (473 KB). Reachable and cheap. Not yet verified to carry file-level paths — confirm before implementing.
• nix: needs the nix-index backend; search.nixos.org serves only a JS shell (1.7 KB). Unverified.
• zchunk delta refresh (§6).

---

8. Implementation constraints

• Go 1.25.x. Module path set to the final repo URL.
• stdlib net/http only — no web framework.
• modernc.org/sqlite (pure Go; rusqlite-style CGO deps are not acceptable).
• github.com/klauspost/compress/zstd (pure Go) for .zst inputs.
• CGO_ENABLED=0 go build ./... must succeed.
• Unit tests must not touch the network; parse from small checked-in fixtures (testdata/). Real-source tests belong behind an opt-in build tag.
• Streaming, not buffering: RPM filelists is 875 MB uncompressed. Use xml.Decoder token streaming and a batched SQLite writer. Never io.ReadAll a source.

Interfaces

type Snapshot struct {
    Manager, Distro, Release, Arch, Repo string
    UpstreamDate, FetchedAt, SourceURL   string
}

type FileEntry struct {
    Path, Basename, Kind, Repo, Package, Version string
}

type Source interface {
    Name() string
    // Versions lists ingestable (release, arch, repo) combinations.
    Versions(ctx context.Context) ([]Snapshot, error)
    // Fetch streams one snapshot's index. Caller closes.
    Fetch(ctx context.Context, s Snapshot) (io.ReadCloser, error)
    // Parse streams entries to emit. Must not buffer the whole document.
    Parse(ctx context.Context, r io.Reader, s Snapshot, emit func(FileEntry) error) error
}

Adding a manager = one file implementing Source + a fixture + a case in the builder.