pkh/plans/pkh-lint.md

pkh lint — Package linting — Spec

Status: phase 1 (wrapper + native skeleton) implemented; phases 2–4 proposal / discussion draft

Phase 1 implementation notes

What landed (all of it behind pkh lint):

• src/lint/: tag model, check trait + registry (check.rs), collectors (collect.rs: source name from control/changelog, git dirty-state under debian/), emitter + report (emit.rs), text/JSON rendering + exit-code decision (output.rs), lintian wrapper (wrapper.rs), and the first native check pkh-git (checks/pkh.rs, tag pkh-debian-changes-not-committed).
• CLI: every option of the table below; --list-tags prints the native catalog. Tests: parser goldens captured from real lintian 2.129 output (source, binary, overridden, --info shapes), registry lint, merge dedup, display/exit-code matrix, git-check fixtures.
• Measured deltas from the original draft, resolved during implementation:
  • lintian accepts only built package files, never trees — the wrapper prefers the pkh build output next to the tree when current (freshness-guarded), else packs fresh fast (-Zgzip -z1); see Wrapper mode. A tree with uncommitted quilt changes makes dpkg-source refuse; the error points at committing or --native.
  • lintian's own exit code is 2 both for "fail-on met" and "runtime error", so the wrapper never trusts it: the verdict comes from parsed findings, and pkh keeps its 0 (clean) / 1 (fail-on met) / 2 (runtime error) contract.
  • --display-info (lintian's -I) exists as its own flag: --info shows explanations (lintian's -i), it does not raise the display level.
• Licensing outcome (see risks): pkh will be dual-licensed GPL-2+ / MIT; any future lintian-derived files stay GPL-2+-only, file-level separated.

Problem

pkh covers the package lifecycle (pull, chlog, build, deb, put) but never validates the packaging itself: nothing catches a broken debian/control stanza, an unparsable changelog version, a missing copyright file, or a leftover UNRELEASED entry before a build or a PPA upload — and when the build does fail, the error often surfaces late and far from the cause. The standard tool, lintian, is a separate invocation with its own conventions, its own lab setup, and no integration with pkh's workflow.

The requirement is feature parity with lintian. Rewriting lintian outright is a multi-year project; ignoring it leaves no parity at all. The strategy below gets parity on day one (wrap lintian), then migrates checks to a native Rust engine incrementally, with a differential harness proving equivalence as checks move.

What lintian actually contains (measured, v2.129)

Numbers from the installed package, because they drive every decision:

Component	Size	Declarative?
Tag catalog	1,546 tag files (Tag/Severity/Check/Explanation/See-Also)	Yes — already data
Check logic	346 Perl modules, 47,586 lines total	No — code
— small modules (280 of 346)	< 150 lines (≈40–60 of those are GPL headers/boilerplate; real logic often 10–100 lines)	Partly expressible as predicates
— heavy modules (18)	400–1,089 lines (Debhelper, Scripts, Changelog, Dep5, Cruft, Menus, Debconf, InitD, Rules…)	No — mini-programs
Static data	110 files, 2.2 MB (spelling dictionaries, section lists, debhelper command tables…)	Yes
Infrastructure	unpackers, file index, ELF analysis (objdump integration), changelog/copyright parsers	No — code

The split that matters: 157 of the 346 checks walk the unpacked binary tree (visit_installed_files); the rest operate on source/control data. Binary-tree checks are the majority of the long tail and need unpack infrastructure before they can be native.

Goal

pkh lint [path] is the single linting entry point of pkh:

• Day-one parity via wrapping: by default, pkh lint runs the installed lintian with mapped flags and merges its findings with pkh's own. Every lintian check, tag, profile and override works through pkh.
• A native Rust engine growing underneath: pkh-native checks (git workflow, PPA conventions — things lintian cannot know) plus checks ported from lintian one batch at a time, each gated by a differential harness against real lintian. Parity never regresses because the wrapper keeps covering whatever is not yet native.
• New checks are cheap to add: one Rust file per check area, one registration line, fixture tests — plus a data-driven escape hatch for one-off, package-specific rules without writing Rust.
• Lintian-compatible where it matters: lintian's tag naming, override file formats and names, output letter conventions and exit codes, so muscle memory, docs, and in-tree overrides carry over. Where a check is equivalent to a lintian check, we use lintian's tag name (this also makes deduplication between wrapper and native findings trivial).

Non-goals

• No monolithic rewrite upfront. Native coverage grows check by check under the harness; the wrapper keeps full parity in the meantime.
• No YAML as a check-logic language. YAML carries the tag catalog, data tables and a thin predicate layer; real check logic is Rust (see Why not YAML for check logic).
• No lintian profile files, no Perl/Python plugin loading.
• No auto-fixing (--fix is a possible follow-up; checks report, they don't edit).

Proposed UX

$ pkh lint                        # lints the source tree in the current directory
$ pkh lint ../hello_2.10/         # or an explicit path
$ pkh lint --json                 # machine-readable (CI, editor integrations)
$ pkh lint --native               # skip the lintian wrapper, native engine only

Default mode: wrapper + native when lintian is installed (notice printed if it is absent, native-only fallback); --native forces native only. Native findings and parsed lintian findings are merged into one report; duplicates are dropped by tag name (works because we mirror lintian's tag names for equivalent checks).

Typical output, lintian-shaped:

E: hello source: broken-debian-changelog-line line 12: unterminated line
W: hello source: maintainer-name-missing John
I: hello source: out-of-date-standards-version 4.5.0 (current is 4.7.2)
W: hello source: pkh-debian-changes-not-committed  (native: uncommitted debian/ edits)
N: 2 overridden tags were shown as O:

Options (lintian names kept where equivalent; pkh common options where relevant):

Option	Meaning
[path]	Source tree to lint (default: cwd)
-d, --dist <dist>	Target distribution (debian/ubuntu); default: from changelog / host
-s, --series <series>	Target series; enables series-validity checks
--native	Native engine only, do not invoke lintian
--repack	Ignore existing pkh build output; pack the tree fresh for linting
--fail-on <list>	Severities that make the exit code 1, comma-separated (default: error)
--info / -i	Show long descriptions and references after each finding
--display-info / -I	Also display info-level tags
--pedantic	Also display pedantic tags
--experimental	Also display experimental tags
--show-overrides	Also display suppressed tags marked O:
--suppress-tags <list>	Ignore specific tag names for this run
--check <name>	Run only the named native checks (repeatable)
--list-tags	Print the merged tag catalog (name, severity, description) and exit
--json	Emit findings as JSON instead of text (same exit code)
--color <when>	auto (default: TTY without NO_COLOR), always, never

Exit codes, mirroring lintian: 0 no findings at/above --fail-on; 1 findings at/above the threshold (in wrapper mode: the merge of lintian's verdict and native findings); 2 runtime error. Default display level: error + warning (deliberately quieter than lintian, which also shows I:; info is one -i away).

Wrapper mode (the parity layer)

src/lint/wrapper.rs:

• Requires the lintian binary on PATH; without it, prints a notice and runs native-only (exit code unaffected by the missing wrapper).
• Artifact strategy — reuse pkh build output, never lint stale state. Lintian accepts only built package files, so the wrapper lints ../<source>_<version>.dsc (pkh build's own naming) when it matches the current changelog entry and no tree content is newer (mtimes, skipping .git/.pc). Current output → linted directly, no packing at all (the fast path: 2.7 s vs 12.5 s on a 111 MB tree). Missing, stale, or --repack → the tree is packed fresh with dpkg-source -b into a temp dir, using weak compression (-Zgzip -z1) since the artifact is ephemeral; quilt trees get their orig tarballs symlinked from the tree's parent (dpkg-source searches cwd, which is the temp dir). Each run prints an N: note stating which artifact was linted and why.
• Flag mapping: --info, --display-info, --pedantic, --experimental, --show-overrides, --suppress-tags translate 1:1; path, --repack, --check and --json are pkh-side.
• Output parsing is load-bearing here (unlike a convenience-only bridge): lintian's line format (X: <pkg> <type>: <tag> <details>) is parsed into pkh's LintReport model so both worlds merge, --json works uniformly, and native checks can be deduplicated against lintian's. The parser is tested against golden samples of real lintian output (the format is stable and version-pinned by the installed lintian).
• Exit code: max(lintian's verdict, native findings at/above --fail-on); lintian crash → exit 2 with raw output retained.
• The wrapper streams nothing to the terminal until merged: pkh renders the unified report (same renderer in all modes).

Native engine

New module src/lint/, library-first like every other subcommand; main.rs only parses flags and picks a view.

src/lint/
  mod.rs        engine entry point: run(options) -> LintReport
  tag.rs        Severity, Certainty, Tag metadata
  check.rs      Check trait, registry
  collect.rs    collectors producing LintData (see below)
  emit.rs       Emitter (dedup, override matching) + LintReport
  output.rs     text / JSON rendering, exit-code decision
  overrides.rs  lintian override file parsing and matching
  wrapper.rs    lintian invocation, flag mapping, output parsing, merge
  checks/
    control.rs       debian/control checks
    changelog.rs     debian/changelog checks
    copyright.rs     debian/copyright checks
    source_format.rs debian/source/format checks
    rules.rs         debian/rules checks
    watch.rs         debian/watch checks
    patches.rs       debian/patches/ checks (DEP-3)
    deps.rs          dependency-structure checks (uses debian::deps)
    pkh.rs           pkh-native checks (git workflow, PPA conventions)

Tag model

pub enum Severity { Error, Warning, Info, Pedantic }
pub enum Certainty { Certain, Possible, WildGuess }

pub struct Tag {
    pub name: &'static str,        // lintian-compatible kebab-case
    pub severity: Severity,
    pub certainty: Certainty,
    pub experimental: bool,
    pub description: &'static str, // shown by --info and --list-tags
    pub references: &'static [&'static str],
}

Certainty exists so gates can be tuned later (--fail-on error,certain is out of scope for v1, but the data is captured from day one).

Check trait and registry

pub trait Check: Sync {
    fn id(&self) -> &'static str;            // "control", "changelog", ...
    fn tags(&self) -> &'static [Tag];
    fn run(&self, data: &LintData, emit: &mut Emitter);
}

Checks are zero-sized structs registered in one explicit array in checks/mod.rs (pub static CHECKS: &[&dyn Check] = &[&control::Control, ...]), in line with the codebase's low-magic style (no proc macros, no inventory crate). A unit test walks the registry and fails on duplicate tag names, missing descriptions, or malformed tag casing — the registry cannot rot silently.

Emitter wraps the report and knows the tag registry; emit.tag("name", "detail") looks up metadata once and applies overrides immediately, so suppressed findings never reach the view. The engine shares the report.rs ports philosophy: the emitter produces a LintReport; rendering (text/JSON/quiet) is a separate concern.

Collectors: LintData

One struct built per run, lazily where expensive (source trees are small; lintian's on-disk lab is unnecessary at this stage):

pub struct LintData<'a> {
    pub root: &'a Path,
    pub dist: Option<String>,        // resolved from --dist / changelog
    pub series: Option<String>,
    pub control: Option<ControlInfo>,        // debian::control
    pub changelog: Option<...>,              // debian::changelog
    pub deps: Option<Deps>,                  // debian::deps (Build-Depends etc.)
    pub files: Vec<TreeEntry>,               // walked tree: path, mode, size
    pub source_format: Option<String>,
    pub copyright: Option<Dep5Copyright>,
    pub patches: Vec<PatchInfo>,             // series + DEP-3 headers
    pub read(&self, rel: &Path) -> Option<&str>,   // cached file content
}

Collector failures (e.g. unparsable control) are themselves findings: the collector records a tag and downstream checks that need that data are skipped — exactly lintian's behavior.

Overrides

Parse the formats and file names lintian already defines, so existing and shared trees work with both tools unchanged:

• debian/source.lintian-overrides — applies to the source package
• debian/<package>.lintian-overrides — applies to that binary package's tags (accepted in v1, enforced when binary tags land)

Syntax supported: # comments, bare tag-name, and tag-name [restrictions] with arch/type restrictions parsed but honored only as far as native tags go (source type only). Unknown tag names in override files emit an override-file-unknown-tag info tag — cheap guard against typos (lintian calls this out too).

Distro scoping (assessment + design)

Ground truth from the installed lintian 2.129: lintian is distro-aware via a vendor/profile mechanism, but Ubuntu's layer is thin. The ubuntu/main profile only disables 14 Debian-specific tags (NMU bookkeeping, bugs-field-does-not-refer-to-debian-infrastructure, upstart leftovers, ...) and the vendors/ubuntu/ data adds exactly one file (changes-file/known-dists). On an Ubuntu host, lintian auto-selects the ubuntu vendor via /etc/dpkg/origins/default. Consequences, visible on e.g. linux-riscv:

• unknown-field Ubuntu-Compatible-Signing fires even on Ubuntu hosts: lintian's derivative-field allowlist mechanism exists (data/fields/derivative-fields, format field ~~ regex ~~ explanation), Ubuntu just never shipped its fields in vendors/ubuntu/. The proper fix is upstreaming to lintian (bug/merge proposal against Ubuntu's lintian); pkh should carry a stopgap suppression meanwhile.
• malformed-debian-changelog-version 7.2.0-5.5.2 (for native) and binary-nmu-debian-revision-in-source: lintian's changelog logic rejects native-format-with-revision, which is how Ubuntu kernel packages are shaped by convention. Profiles can only disable whole tags, not reshape logic, so this class can't be fixed by lintian data — it is exactly the pkh-side scoping below (and eventually our own native checks, which know the dist).
• Ordinary Ubuntu backport versions (...~26.04 tildes) are fine: tilde is valid dpkg version syntax, and the ubuntu profile already disables the Debian upload-version bookkeeping checks.

Design (phase 2, needs sign-off on the tag lists):

• Distro → wrapper profile: done in phase 1 — -d ubuntu/-d debian passes --profile to lintian so the target distro's rules apply even on a foreign host (auto-detection already covers the same-host case).
• Distro-level suppression data (data/lint.yml, embedded): per-dist lists of tags to downgrade or drop for wrapper-origin findings, e.g. ubuntu: unknown-field once the field list is upstreamed — deliberately kept minimal; suppression is policy, not taste.
• Package-level suppression via quirks.yml (existing package-keyed pattern), e.g. for the kernel family:

  linux-riscv:
    lint:
      suppressed_tags: [malformed-debian-changelog-version,
                        binary-nmu-debian-revision-in-source]
  

  In-tree debian/source.lintian-overrides remains the package's own voice (shared with lintian, already honored); quirks are pkh's voice for conventions the ecosystem hasn't written down.

Checks — first native batch

~20 tags across 7 check areas, chosen for what bites pkh users before an upload. Severities below are the proposal; tuning happens in review.

control (debian/control):

• missing-package-field (E) — required source fields absent
• maintainer-name-missing / maintainer-email-missing (W) — malformed Maintainer/Uploaders
• unknown-priority (E), unknown-section (W) — validated against data/ lists (new small embedded data files)
• description-too-short (I) — Description under ~20 chars
• missing-tests-control (I) — no Testsuite: autopkgtest nor debian/tests/
• binary-field-mismatch (W) — Architecture/Section inconsistencies between stanzas

changelog (uses debian::changelog + debian::version):

• changelog-file-missing (E) — no parsable debian/changelog
• bad-urgent-value (W) — urgency outside low/medium/high/critical
• unreleased-changes-present (W) — top entry is UNRELEASED
• unknown-distribution (W) — distribution not in the dist's series list (reuses distro_info data!) nor a known alias
• version-syntax-error (E) — DebianVersion::parse failure
• version-native-mismatch (E) — native format with revision, or quilt format with native version

copyright:

• missing-debian-copyright-file (E)
• copyright-not-machine-readable (I) — no DEP-5 header
• missing-copyright-license-paragraph (W) — files referenced but no License stanza

source_format: missing-debian-source-format (W), non-3.0-source-format (I)

rules: debian-rules-not-executable (E), debian-rules-missing (E), debian-rules-no-required-target (W) — conservative static scan, no make invocation

pkh (native only, the differentiator — lintian can't know these):

• pkh-debian-changes-not-committed (W) — git tree dirty in debian/ (pkh's whole flow is git-centric; catching "edited control, forgot to commit" before pkh put)
• pkh-new-template-drift (I) — tree was scaffolded by pkh new and template files have drifted from the current templates in data/

Note: pkh new's existing structural self-checks should eventually be rebuilt on this engine so there is one definition of "a healthy tree".

Why not YAML for check logic

The question "can the 1,500 checks live in YAML?" decomposes into three different things, with three different answers:

1. Tag catalog: yes — and it is already data. Lintian stores all 1,546 tag descriptions as flat files (Tag/Severity/Check/Explanation/See-Also). Converting them to a YAML/JSON catalog is a mechanical script job — no agentic engineering required. The native registry should load or generate from this catalog so native tags and wrapper-parsed tags share one metadata source.
2. Static data tables: yes. The 2.2 MB under lintian's data/ (spelling dictionaries, section/priority lists, debhelper command tables) is exactly the kind of content data/*.yml + embed_data! exists for. License applies (see risks): copying these files makes pkh GPL-derived.
3. Check logic: no. Even the smallest lintian check is code — Files/LdSo.pm is one regex on a filename plus one condition on the package name, as a Perl method; most of the 280 small checks are similar ("visit file X, apply conditions, emit tag"), and 18 checks are 400–1,089-line mini-programs with loops, cross-file state, package relation graphs and version arithmetic. A DSL expressive enough for those is a scripting language with an interpreter you now have to maintain, debug, and document — worse than Rust, with no compiler safety net. YAML is a data format, and check logic is not data.

Where YAML does fit in logic: a thin predicate layer for the simple tail (field present/absent, file exists, glob match, regex on a field, version comparison) — this is what debian/pkh-lint.yml (below) exposes for user-specific rules. That covers a meaningful slice of the small checks, but pursuing 100% parity through it would mean maintaining a bespoke interpreter as big as the problem.

Agentic engineering changes the economics of porting, not the target representation. LLM agents are good at mechanical translation (Perl check → Rust check) and at writing the differential tests; the correctness gate is the harness below, not the faithfulness of the translation. Porting is per-check, priority-ordered, and reviewable — compatible with the wrapper keeping parity in the meantime.

Differential harness (the trust anchor for porting)

Before any ported check ships:

1. Corpus: a set of real source packages (the pkh pull flow already fetches them) plus lintian's own test suite fixtures (in the lintian source package).
2. Record: run real lintian over the corpus, store expected tag sets per package.
3. Diff: pkh lint --native over the same corpus; report tag-set deltas (missing / extra / severity-mismatch) per package. The CI gate for ported checks is an empty delta on the covered corpus subset.
4. Graduation: a check is "native" once its delta is empty; the wrapper continues covering everything else, and both-mode dedup by tag name means graduation is invisible to users.

Declarative rules (user-specific, no Rust)

The user-facing "add specific rules" path, mirroring the quirks.yml philosophy: a small YAML file linted against the tree.

debian/pkh-lint.yml:

rules:
  - id: keep-changelog-synced
    tag: pkh-changelog-sync-rule
    severity: warning
    when:
      control-field-present: Vcs-Browser
    expect:
      file-present: debian/salsa-ci.yml
  - id: no-embedded-js
    tag: pkh-embedded-javascript
    severity: error
    expect:
      no-file-matching: "*.min.js"

Phase 2; the predicates are the collector primitives (field present/absent, file present/absent, glob, regex on a field). Custom pkh-* tags get auto-generated catalog entries. This keeps the Rust registry for real checks while giving package maintainers one-off rules.

Output

• Text: lintian's <L:> <pkg> <type>: <tag> <detail> line format, colors only when TTY (reuse the ui.rs conventions; --info appends indented description + references). Summary line: N: N tags (3 E, 5 W, ...), M overridden.
• JSON (--json): array of {tag, severity, certainty, experimental, check, message, overridden, file?, line?, origin: native|lintian} plus a summary object — schema-versioned for CI use. Works identically in wrapper and native modes (wrapper findings are parsed into the same model).
• Exit: 0/1/2 as above; --fail-on decides the threshold (error default; error,warning for upload gating).

Testing

• Wrapper parser: golden tests against captured output of real lintian (several packages, all display levels, overridden/O: and C: lines). Version the fixtures by lintian version.
• Differential harness: as above; the CI gate for native checks.
• Fixture trees (tests/fixtures/lint/<case>/): minimal source trees exercising one tag each, with a manifest of expected tags; negative fixtures (clean trees, zero tags) live next to positive ones — same pattern pkh new verification uses.
• Registry lint test: duplicate/undocumented tag names fail the test suite.
• Override parser: table-driven tests against lintian's documented syntax, including the weird corners ([restrictions], inline comments).
• Golden text output: one snapshot test of the renderer so formatting changes are deliberate.

Phases

1. Wrapper v1 (parity day one) — src/lint/ skeleton, lintian invocation + flag mapping + output parsing, unified report (text + JSON, exit codes), native engine skeleton with the pkh-native checks (pkh-*), CLI wiring. Feature parity achieved here.
2. Native core checks + harness — collectors, first native batch (control/changelog/copyright/source_format/rules), overrides, tag catalog import from lintian's tag files, differential harness in CI, both-mode dedup. pkh new self-checks migrate onto the engine.
3. Porting track (agentic, incremental) — batches of lintian checks ported Perl→Rust, priority-ordered by pkh-user relevance (source checks first); each batch gated by the harness. Predicate layer + debian/pkh-lint.yml. Data-table import once licensing is decided.
4. Binary artifacts + unpack infra — lint .changes + built .debs (temp unpack): file index unlocks the 157 installed-files checks family (manpages, md5sums, maintainer scripts, init/systemd units, scripts). The heavy checks (Debhelper, Scripts, Dep5…) and their data land here, batch by batch under the harness. <pkg>.lintian-overrides enforcement; optional pkh deb --lint and a pkh put upload gate.

Risks & open questions

• Licensing is a blocking decision for the porting track. Lintian is GPL-2+. Porting its check logic or copying its data files produces a derivative work; pkh currently declares no license. Options: (a) pkh goes GPL-2+ — porting and data import are clean; (b) pkh stays unlicensed/ permissive — then only the wrapper plus independently-authored checks (the pkh-* set) are possible, and the porting track is off the table. The wrapper alone is always fine (subprocess boundary, no derivative). This decision gates phase 3, not phases 1–2.
• Wrapper parsing depends on lintian's output format. Stable for years and pinned to the installed version, but a lintian major change would break wrapper mode; mitigated by golden fixtures and the native track reducing dependency over time.
• False positives on PPA workflows. Lintian targets archive uploads; some checks are noise for quick PPA builds. Mitigation: quiet default display level, --fail-on error default, pkh-* severities tuned for the pkh flow.
• Porting velocity is the open question of phase 3. The harness makes each port verifiable, but 346 checks (and their data) is a campaign, not a sprint; the wrapper exists precisely so this never blocks users.
• Tag-name mirroring is a commitment. Reusing lintian names buys familiarity, shared override files, and trivial both-mode dedup — at the cost of renames breaking overrides. Accepted; lintian tag names are stable.
• Overlap with pkh new self-checks — migrating them onto the engine needs a small "run subset of checks programmatically" API; the library-first design allows it but it should be an explicit decision when touched.