TL;DR

#

The admins on a production Bitbucket Data Center said “HA is not supported, so we can’t run the official migration export.” That refusal was the entire motivation for hans: a tool that reconstructs a valid Bitbucket_export_<jobId>.tar using only read-only REST API access and git mirror clones — no admin privileges — such that GitHub Enterprise Importer (gh bbs2gh migrate-repo --archive-path) accepts it. Correctness wasn’t assumed: every format claim was reverse-engineered from a real admin export of a scripted “golden” repo, then gated four ways. Along the way we hit (and solved) a licensing hellscape, an activity-schema mismatch, a 35-minute git-object unpack, a 100x Rust rewrite, a KeyError: 'commit' crash on a real 10,031-PR repo caused by stacked PRs. Final state: all gates PASS, git SHAs migrated byte-identical, and the tool produced a 30,105-entry archive for a real 10K-PR production repo.

The Setup

#

• The problem: a production Bitbucket Data Center (9.4.18) where a GitHub migration is needed, but the admins won’t run the admin-only export. “HA not supported” was the stated reason.
• The goal: produce an archive the GitHub Enterprise Importer will accept, using only what a normal (read-only) user can see via REST and git.
• The lab: two containers bb-lab-a / bb-lab-b (atlassian/bitbucket:9.4.18-jdk21) run under nerdctl, exporting to each other for round-trip validation.
• The user: working from a Windows box (Rancher Desktop / nerdctl) at work, with the “home ox” running Linux. This split shaped a lot of the portability work.

The killer insight that made it possible: the export format has no checksums, no signatures, no byte-pinning. It’s a plain tar of bare git repos + JSON metadata, all derivable from data a normal user can read. So hans re-derives the exact same archive from a REST scrape + git mirror.

The Architecture: What Flows Where

#

Two loops run in parallel, and keeping them separate is the whole design. The pipeline turns production data into an archive; the ground-truth loop is what makes the pipeline trustworthy.

The end-to-end pipeline

#

production Bitbucket DC 9.4.18
(read-only user: REST + git clone)
                │
   bb-archiver scrape
                │
        ┌───────┴────────┐
        │                │
  scrape dir         git mirror
  rest/*.json        refs/heads, refs/tags,
  index.json         refs/pull-requests,
  warnings.jsonl     refs/stash-refs
        │                │
        └───────┬────────┘
                │
   bb-archiver assemble
                │
   Bitbucket_export_synth_N.tar
        │
   bb-archiver validate   (schema self-check)
        │
   gh bbs2gh migrate-repo --archive-path ...
        │
   GitHub Enterprise  (byte-identical git SHAs)

The scrape produces two complementary views of the repo: JSON (PR metadata, activities, comments — the things REST knows) and the git mirror (every commit object, author, and message — the things git knows). Assemble fuses them into the archive’s two halves: JSON metadata files and a bare-repo / loose-objects tar. GitHub attribution works precisely because the mirror carries the real author/committer emails that REST exposes only in embedded user objects.

The ground-truth loop

#

      fixtures/  (scripted golden repo on bb-lab-a)
            │
      ┌─────┴──────┐
      │            │
  REST dumps    real admin export
  (read-only)   (admin-only, the "golden")
      │            │
      └─────┬──────┘
            │
  corpus/  +  ground-truth/export-a/
            │   (the supervision signal)
            ▼
  FORMAT_SPEC.md
  (reverse-engineered format, byte-level)
            │
            ▼
  bb_archiver/emitter.py
  (implements the spec exactly)
            │
            ▼
  Gate 1:   assemble vs golden export   → NO GENUINE DIFFERENCES
  Gate 2:   import into bb-lab-b, rescrape, diff → PASS (round-trip)
  Gate 2B:  re-export bb-lab-b officially, diff vs golden → PASS
  Gate 3:   gh bbs2gh end-to-end → byte-identical SHAs

Every format claim the assembler makes is checked against a real admin export, never guessed. The method in one sentence: build a scripted golden repo, capture both views of it, write the spec from the capture, then prove the assembler reproduces the golden.

Phase goals at a glance

#

Phase 0 — Lab Bring-Up Was a Licensing Hellscape

#

What this phase is for. Every later claim depends on being able to generate two things: a real admin export (the ground truth) and a second instance that can import an archive (the round-trip target). The design is deliberately two containers on the same 9.4.18 image as production, scripted so the whole lab is disposable — because the eval license keeps expiring and re-creatability is the point. The reasoning for running it ourselves: without a local ground-truth generator, every format claim in the whole project would be unverifiable guesswork.

The nerdctl kickoff

#

The user: “actually, start from phase 0, there is nerdctl (sudo nerdctl).” So the lab was built with atlassian/bitbucket:9.4.18-jdk21 (uid 2003 inside), exposed on :7990/:7991.

The license gate

#

Bitbucket’s setup wizard gates on hasLicenseAndBaseUrl() — no license, no further setup. trial3h.txt is a Data Center evaluation license, but the whole bring-up kept stalling at FIRST_RUN. The wizard wouldn’t honor the setup.* properties on this build, so it had to be driven manually with curl. And even then the license POST failed.

The killer bug: curl -d form-encoding turns + into spaces. The trial license contains + characters (it’s a SEN-500 test artifact), so every curl -d license=... submission corrupted the license and got “The provided license is not valid.” The fix was --data-urlencode license=..., or sending the whole thing as a JSON body. That single encoding fact kept biting us across two container re-creates.

There was also a heavier hack explored: bytecode-patching the wizard gate (iconst_1; ireturn) so the “no license” check passed. It worked well enough to be noted, but the --data-urlencode path became the scripted one.

License-limit errors that looked like authentication

#

Once licensed, pushing a repo would occasionally fail with fatal: remote error: License limit exceeded / Pushing has been disabled — composed via i18n, and looking for all the world like an auth problem. It was the license window expiring mid-session (the eval license is relative to when it’s applied to an empty env — a fact the user supplied and that unblocked a later re-creation).

Readiness polling and field-name traps

#

• The readiness wait would stall at FIRST_RUN state and time out at 600s.
• The admin-form field names differed from expectations (fullname not fullName) — subtle 400s.
• The wizard’s stack maps were NOP-filled, producing VerifyErrors on a version with the bytecode patch.

Phase 0 deliverable: scripted lab/lab-up.sh / lab-down.sh, a SEED.md, and a hard-won list of license/wizard gotchas. Committed as 811f976 “Phase 0: Bitbucket DC 9.4.18 lab environment (scripted bring-up + unlicensed hack)” and 0ac2bb6 (“commit phase 0 too”).

Lesson: an evaluation-license lab that you must keep re-creating is its own recurring time-sink; script everything, including the manual wizard dance, and never submit a +-containing license through form-encoded curl.

Phase 1 — Scripting a “Golden” Repository (and 16 API Findings)

#

What this phase is for. Correctness can only be claimed if there is a corpus with known properties to check against. The design is a single golden repo that exercises every data shape the exporter might encounter — all PR terminal states, force pushes, tasks, unicode/emoji, a known-bad deleted-branch merge — so every later phase has a full-spectrum test subject. The reasoning is that a fixture that can’t be reproduced isn’t ground truth, so everything is scripted (fixtures/run.sh) and idempotent.

The whole correctness story rests on this fixture corpus: a scripted repo on bb-lab-a (FIX/golden), fully reproducible. Five pull requests (PR1–PR5), three users (ada/grace/alan with distinct emails), crafted git history with distinct author/committer pairs and forced dates, force pushes, tags (lightweight + annotated), unicode/emoji, tasks, and a deliberate “known-bad” case (merged PR whose source branch is deleted).

Building it surfaced a pile of live-verified Bitbucket DC 9.4.18 API facts:

1. POST /pull-requests/{id}/reviewers returns 404. Reviewers are managed via POST /participants with {"user":{...},"role":"REVIEWER"}.
2. A PR author cannot approve their own PR — ada’s self-approval failed silently.
3. pr_create dropped the credentials arg (positional-order bug) so PRs were authored by admin. Fixed.
4. The api helper ignored HTTP status — a failed squash merge logged “MERGED” while the PR stayed OPEN. Fixed by adding -f so non-GET failures die loudly.
5. Merge conflicts veto merges (PullRequestMergeVetoedException, HTTP 409). Fixed by making PR3 touch a new file.
6. diff_line looked at .destination.path — which doesn’t exist. The path is .destination.toString.
7. /tasks endpoint 404s on 9.4.18 (this becomes a whole saga later).
8. Merge-version races: merging PR2 while PR1 is open bumps PR1’s version → PullRequestOutOfDateException. Fixed with a retry helper that re-reads version.
9. Comment versions start at 0; DELETE /comments/{id} requires ?version=0.
10. Comment DELETE is a HARD delete — no trace remains in REST.
11. GET /commits/{sha}/comments REQUIRES ?path= and only returns path-anchored comments.
12. NEEDS_WORK is self-service — setting it as admin returns 403, as the reviewer 200.
13. Any PR mutation resets reviewer states — approvals and NEEDS_WORK are wiped by a push, another merge, even a title edit. Reviewer states must be set last, after a drift-settle poll.
14. Annotated-tag tagger = local git config at tag time; git -c user.name=... tag -a.
15. Merging does NOT auto-delete the source branch on 9.4.18 (the known-bad case).
16. DuplicatePullRequestException for re-running the fixture scripts per branch.

One fixture bug stood out in Phase 2: C2 was an empty commit. create.sh wrote src/util.py via heredoc but never ran git add src/util.py — so C2’s tree was identical to C1 and the “add” became a “modify” on a later commit. A missing git add that silently corrupted the entire fixture corpus.

Lesson: when you’re building ground truth, every silent API surprise is a landmine — -f on curls, explicit field names, and read-the-version-before-mutate all earn their keep.

Phase 2 — Paired Capture: REST Dumps Beside the Real Export

#

What this phase is for. Reverse-engineering needs a supervision signal: you cannot decode a format by staring at one side of it. The design captures both views of every entity — the raw REST JSON a normal user can read, and the extracted real admin export — indexed by a manifest, so any field in the archive can be traced to its REST source (or proven to have none). The reasoning: the pairing check is what turns “I think the exporter stores X” into “here is the archive field, here is its REST source field, both captured from the same repo at the same time.”

The supervision signal is a paired capture: raw REST dumps of every entity, beside the extracted real admin export of the same repo.

Finding the export endpoint

#

This took real archaeology. UI paths and REST guesses all 404’d. We pulled the bundled bitbucket-migration-plugin-2.6.38.jar and found it was the Cloud-migration plugin — a dead end. The real API, from Atlassian docs:

POST /rest/api/1.0/migration/exports
{ "repositoriesRequest": { "includes": [ { "projectKey": "FIX", "slug": "golden" } ] } }

Poll GET /rest/api/1.0/migration/exports/{jobId} until COMPLETED. The archive lands at <shared-home>/shared/data/migration/export/Bitbucket_export_<jobId>.tar — note it’s a plain tar, not gzip (the export tar’s .tar extension is literal; gzip is only per-entry).

The archive-vs-REST activity schema finding

#

corpus/compare.py pairing check matched PR metadata, but activity counts differed per the schemas. The headline discovery:

• REST activities use action ∈ {OPENED, UPDATED, COMMENTED, APPROVED, UNAPPROVED, REVIEWED, RESCOPED, MERGED, DECLINED} with commentAction/participantStatus.
• Archive activities use kind ∈ {COMMENT:ADDED, COMMENT:OTHER, ACTIVITY, REVIEWERS:UPDATED, RESCOPED, MERGED} with createdTimestamp, userId as "ada|Ada Lovelace||NORMAL" strings (the GEI mannequin keys), and nested comment objects with thread.anchor.

They’re not 1:1 — REST COMMENTED maps to archive COMMENT:ADDED and COMMENT:OTHER (edits); REST APPROVED maps to REVIEWERS:UPDATED + ACTIVITY. The archiver had to derive archive kinds from REST events. This became the heart of Phase 3.

Phase 3 — Reverse-Engineering the Format, Byte by Byte

#

What this phase is for. The archive format is the contract the assembler must implement, so it has to be written down before any tool code. The design decision was to produce FORMAT_SPEC.md early and iterate it, driven by the user’s warning that otherwise we’d run out of context. The reasoning for a written spec as the single source of truth: it makes every format claim checkable against the golden corpus, prevents the assembler from silently drifting, and gives Phase 4’s emitter a fixed target instead of “whatever the tool happens to produce.”

The user’s directive here was crucial: “start writing the format spec already, you can refine it in loops. otherwise you run out of context.” So FORMAT_SPEC.md was written incrementally and iterated, rather than exhaustively researched first.

The archive envelope

#

tar -tvf on the real export showed a plain tar, entries owned 0/0, mode 0644, mtime = export time, PAX format. Order: instance-details.json.atl.gz → project metadata → project perms → _/repository/hierarchy_begin/<id> (0-byte markers) → repo metadata/perms → git metadata/hooks/objects → git-lfs settings → per-PR metadata/activities/caches → hierarchy_end.

Two Jackson serializers — the byte-level battle

#

This was the deepest reverse-engineering pass. Raw bytes (xxd, cat -A) proved the archive uses two different JSON writers:

1. Compact, alphabetically-keyed, no trailing newline — project/repo/permissions/LFS JSON: {"description":"Golden fixture corpus","id":"1","key":"FIX","name":"Fixture Project","public":false,"type":0}.
2. Pretty, Jackson DefaultPrettyPrinter — PR metadata, activities, instance-details.json: 2-space indent, "key" : value (spaces around the colon), arrays of objects as [ { ... }, { ... } ] (consecutive }/{ on one line), empty arrays as [ ], no trailing newline.

And the string escaping, verified by cat -A:

• Control chars and non-BMP (emoji) → uppercase \uXXXX surrogate pairs (\uD83D\uDE80).
• BMP non-ASCII (ü, ñ, 汉字) → raw UTF-8 bytes, not escaped.

Why does any of this matter? Because nothing else pins the JSON bytes. We verified: no checksums, no manifest anywhere (archiveVersion: 2, version: 9.4.18, buildVersion: 9004018, nodeId = export job node). The only way to be byte-faithful is to reproduce the exporter’s exact serializer output. Matching the writer quirks is the correctness.

Gzip headers also verified: magic 1f8b, CM 8, FLG 0, MTIME zeroed, OS byte 255.

The stash-refs discovery

#

The git “bare-repo skeleton” tar carries refs/heads/*, refs/tags/*, refs/pull-requests/4/from for OPEN PRs only, and — the critical find — stash-refs/pull-requests/{1..5}/from for ALL PRs. Bitbucket’s old name was “Stash,” and the exporter preserves the historical PR source refs there even after the live refs/pull-requests/*/from entries were cleaned up on merge. No refs/pull-requests/*/merge refs exist — merge results live in PR metadata/activities instead. This directly answered a plan risk (whether GEI needs merge refs).

The merge-strategy reality check

#

The archive showed PR2 (intended ff) and PR3 (intended squash) both producing two-parent merge commits, contradicting our fixture’s claims. Instead of guessing, we probed live on a scratch repo:

• strategy: "ff" → merge commit 2697dae6 with two parents, tree == source tip tree (“content fast-forward recorded as a merge commit”).
• strategy: "squash" → merge commit with parents [prev, branchTip].
• strategy: "bogus" → silently accepted (HTTP 200), treated as merge_commit.

Conclusion: on 9.4.18 all three strategies yield two-parent merge commits; ff is not a true fast-forward. And the canonical merge-commit shape: author = PR author, committer = merger, with the standard Merge pull request #N in FIX/golden from <branch> to main message.

PR caches and rescopedTimestamp

#

cached-ancestor.txt = <fromTip>,<toTip>,<mergeBase>. rescopedTimestamp is always present; it equals createdTimestamp when never re-scoped, and a target-only advance bumps it without a RESCOPED activity (an internal refresh REST cannot see).

Phase 4 — Building bb-archiver

#

What this phase is for. Turn the spec into a working tool. The design separates concerns into three layers — scrape (collect raw data), model (map it into the archive’s ID/entity model), emitter (serialize byte-faithfully) — plus a dedicated jsonwriter, because matching Jackson’s exact serializer output is the only thing pinning the archive bytes. The reasoning: by making each layer independently testable against the corpus, a byte-level bug gets located fast instead of surfacing as a mysterious mismatch in the final tar.

Now the actual tool, in Python (per the user’s standing directive: “NOT create it as .sh bash scripts, but python”). Three components:

jsonwriter.py — the Jackson clone

#

The escaping bugs were caught by round-tripping against the real archive:

• Python iterates strings by full code point, so the surrogate branch never fired for emoji — needed explicit o >= 0x10000 → compute UTF-16 hi/lo surrogates.
• First diff: lowercase \ud83d\ude80 vs archive \uD83D\uDE80 → uppercase hex.
• First diff: },\n { vs archive }, { → arrays-of-objects render inline.

After fixes: PR1 metadata byte-identical: True, PR1-5 activities byte-identical: True, instance-details byte-identical: True.

scrape.py — the crawler

#

A series of bugs fixed in sequence: limit shadowing (limit_prs), a missing git_dir param, argument ordering, InvalidURL (the get() helper prepended base to already-absolute URLs), and TypeError: string indices must be integers (branches/tags endpoints wrap values in {"values":[...]}). Output: rest/*.json, users.json, index.json, warnings.jsonl, and a bare git mirror with refs/heads/*, refs/tags/*, refs/pull-requests/*, refs/stash-refs/*.

emitter.py — the assembler

#

Assembly details:

• Loose objects via git repack -adf then git unpack-objects -q -r (reading the pack from stdin) into a temp .git/objects dir — the exact format the real exporter’s objects.atl.tar contains.
• Bare-repo skeleton: HEAD, config, refs, stash-refs, reflogs, app-info/gc.pid.
• PR metadata from REST fields; activities derived from the REST stream per the §7 mapping table; caches with merge-bases.
• PAX tar building with mtime=0, uid=gid=0.

Gate 1 — golden-master diff

#

corpus/gate1.py semantic-diffs the tool’s archive against the real export, with documented normalizations (JSON semantics not bytes, allParticipants as a set, title-edit UPDATED as a known gap, inner-tar mtimes ignored, git objects by set-of-(path, content)).

Iteration revealed: nested replies must omit thread; thread.updatedTimestamp = max(top, replies); RESCOPED totals come from added.total (not .count). Final: Gate 1 PASS — no genuine differences, only the three documented benign divergences (inner-tar mtimes, title-edit UPDATED gap, rescopedTimestamp lazy refresh).

Phase 5 — The Gates, the Tasks Saga, and a Live GEI Migration

#

What this phase is for. Proof. The user’s standard was unyielding: every check had to be scripted, committed, and re-runnable — no manual eyeballing, no one-off verification. Four gates each answer a different question: Gate 1 does the assembler reproduce the golden?, Gate 2 does the archive survive a real import round-trip?, Gate 2B what does the official exporter preserve that REST can’t see?, Gate 3 does GitHub Enterprise Importer actually accept it?. The reasoning: a synthetic archive that has never been through the real importer is just a hypothesis.

The user’s contribution here was relentless demand for scripted, re-runnable, honest verification: “you did git refs/branches verification manually, needs to be scripted… also you skipped object-wise comparison… ive come up with gate 2B.”

Gate 2 and the invention of Gate 2B

#

• Gate 2: import the synthetic archive into bb-lab-b, rescrape, and scripted-diff PRs/activities/branches/tags and git objects object-wise (git cat-file --batch-all-objects). PASS — 54 objects identical, stubs expected.
• Gate 2B: re-export bb-lab-b with the official admin endpoint and diff that tar against the golden export — surfacing everything the official path preserves that REST can’t see. The normalization bugs were instructive: repo-id vs PR-id collision on a fresh target (both 1 — a global numeric rewrite corrupts PR metadata), userId→slug normalization, comment-id stripping, timestamps rounded to the second. PASS — the only genuine diff was the title/desc-edit UPDATED events.

The PR tasks saga — the user knew better

#

“did you say smth earlier that PR tasks are not supported?” — I’d recorded /tasks as a 404. The user pushed back: “i know that tasks can be added as comments with some kind of property! i’ve in fact done it… google the web, that’s how i found to create tasks.” Then the key: “checked my work scripts: it uses severity: ‘BLOCKER’ on the same level as the text attribute to create a task.”

Verified live: POST .../comments with {"text":"...","severity":"BLOCKER"} → HTTP 201 with "severity":"BLOCKER","state":"OPEN". Resolve via PUT with state:"RESOLVED". And in the official export, a resolved task comment carries resolvedTimestamp + resolverId — fields absent on normal comments. REST exposes the same under different names: resolvedDate/resolver. Emitter mapped resolvedDate → resolvedTimestamp, resolver.slug → resolverId. End-to-end: IDENTICAL vs the official export.

This spawned a subtle emitter rule: the resolve operation bumps the comment’s version to 1 and sets resolvedDate but not updatedDate — so the naive updatedDate != createdDate edit-detection missed it. The real rule is version > 0 → emit COMMENT:OTHER/EDITED, timestamp = resolvedDate if present else updatedDate.

The lab re-create (license renewal trick)

#

The 3-hour eval license expired again. The user: “you can recreate the lab containers, this gets another 3h of the license… then update the gate verifiers also to check that the tasks-as-comments reproduce w/ fidelity.” The re-create (PURGE, sudo rm -rf lab/home/*, up, manual wizard with --data-urlencode) worked — and surfaced a real emitter bug the fresh corpus caught: the resolved task’s COMMENT:OTHER/EDITED. After the fix: NO GENUINE DIFFERENCES, TASK COMMENT BYTE-IDENTICAL, both tasks reproduced exactly through Gates 1/2/2B.

Gate 3 — end-to-end GitHub Enterprise Importer

#

The user: “maybe this procedure also works on the unapplicable org? worth a try.” After a PAT-flag trial-and-error gauntlet (dummy --bbs-* flags required but unused, BBS_USERNAME/BBS_PASSWORD env demanded even with --archive-path, --use-github-storage needed), and the discovery that there is no migrate PAT scope (my earlier claim was wrong; required scopes are repo, read:org or admin:org, workflow, plus an Enterprise-plan org), the migration succeeded:

BBS_USERNAME=admin BBS_PASSWORD=... GH_PAT="$(cat /tmp/hans.gei.txt)" gh bbs2gh migrate-repo \
  --github-org unapplicable --github-repo hans-golden \
  --archive-path /tmp/opencode/synth-a4.tar \
  --target-repo-visibility private --queue-only --use-github-storage \
  --bbs-server-url http://localhost:7990 --bbs-project FIX --bbs-repo golden

~7 minutes later: SUCCEEDED, one benign warning (“Allow Forking”). Verification: all 5 PRs with correct states/merge flags, 6 branches, both tags, review states, comment bodies (unicode/emoji/@mentions), and — the headline — git SHAs byte-identical to the corpus. Gate 3 PASS.

Gate 3 at Scale — the Verifier the User Would Take to Work

#

“improve gate3 checker so that i can use it at work against a repo with 10K PRs and ~100K commits… progress reporting etc and no crashing inbetween.”

Two things made scale feasible:

1. The merged_at discovery: the PR list endpoint returns "merged": null even for merged PRs — but merged_at is populated for merged and null for declined. So state can be classified from the paginated list (closed + merged_at != null → merged) without ~10K per-PR GETs.
2. A rate-limit-aware client: preemptive sleep below a rate-limit headroom budget, 403 rate-limit body handling, exponential backoff on 429/5xx, ETag/If-None-Match caching, Link: pagination.

Plus a resumable --deep mode (per-PR reviews + comment bodies persisted to a state file — Ctrl-C safe, runs across rate windows), ISO timestamps on every line (“the output should begin each line with iso timestamp so can correlate what happened when”), and real comment-body comparison (“i notice you download PR comments from GH but dont compare them to BB, you just verify counts”). GH flattens BB reply threads into one issue comment with > quoting, so norm_comment() strips quote markers before substring-match-both-ways.

“Is the bb-archiver scale-proof?”

#

Blunt question, honest answer: no — the verifier was scale-ready but the archiver was a lab prototype. Concrete blockers: PR list capped at 100 (no pagination), branches/tags capped, main history truncated at 100 commits, per-comment REST unpaginated, 100K+ per-commit REST requests, no rate limiting/retry, no checkpoint/resume, and the emitter reading the whole pack into RAM (OOM on multi-GB repos).

The fix: drop every REST call the git mirror covers

#

The user’s instinct was the roadmap: “verify that nothing unnecessary is fetched from BB via REST, that can be gotten via the git mirror, e.g. why do you need to fetch commits via REST?” — right. The emitter/model only consume project/repo/users/branches/tags/PR list/PR activities + the mirror. So we dropped from production scraping: commit_{sha}, commit_{sha}_changes, commit_{sha}_comments_*, pr_{id}_commits, pr_{id}_diff, pr_{id}_comments_path_*. All commit content (author/committer/message/tree/parents/diff) is in the mirror. That killed the 100K-request problem.

The rewrite: paginated everything (start/limit/nextPageStart), rate-limit + retry client, checkpoint/resume (checkpoint.json with prs_done), streaming loose objects to tar (O(1) memory), merge-base from git. The lean scrape produced 17 REST files (was ~58) — and Gates 1/2/3 still PASSed. Branches/tags stayed REST-only for one good reason: the tags dump carries annotated-vs-lightweight hash semantics the archive’s refs/tags/* need, and deriving that from the mirror means peeling every tag.

The Windows Work Box and the Missing Progress

#

The user tested at work and immediately hit friction:

1. env: 'python3\r': No such file or directory — CRLF line endings from the clone. Fix: skip the shebang and run python bb-archiver ... explicitly (the shebang is a comment to Python anyway), or normalize with PowerShell.
2. Password to env from file: $env:BB_PW = (Get-Content -Path .\pw.txt -Raw).Trim().
3. Scrape output missing timestamps — added ISO-UTC _log() on every line.
4. Assemble with zero feedback — python bb-archiver assemble printed nothing for 35 minutes. Added timestamped phase logs + per-PR progress every 25. Also fixed a cosmetic bug where the “archive complete” line wrongly got the [scrape] prefix (it imported _log from the wrong module).

Then the real pain: git unpack-objects on 863,129 objects at ~200/s, ETA ~65 minutes, rate dropping (314→299→200/s — NTFS degradation writing millions of tiny files). The user’s question — “how long would that take?” — and the answer (~1.5–2.5h total) triggered the single biggest performance move of the project.

The 100x Speedup: bb-obj-tar in Rust

#

The user proposed the solution: “can you maybe create a small helper in RUST for this specific slice of work? should be lightning fast, if you use git libraries to read from the git and avoid creation billions of files on the disk, you can figure out how to create byte identical output, i’m sure. you could also parallelize this process and create like 16 chunks of the resulting tar which you can later join together.”

Byte-identity first: the zlib investigation

#

Before writing Rust, we had to prove what “byte-identical” means. Earlier I’d brute-forced level×windowBits×memLevel×strategy and found no match between Python zlib and git’s loose objects. That turned out to be three test bugs: wbits=-15 tried alongside +15; a f-string stringified the header bytes object (b'blob') into the payload; and git cat-file <sha> without --batch returns empty for trees. Fixed: loose object = zlib.level1("<type> <size>\0" + content), windowBits 15, Z_DEFAULT_STRATEGY — reproduced git’s bytes 54/54, confirmed git links system libz.so.1.

Design

#

• Enumeration: parse the pack .idx v2 in-process (magic \xfftOc, 256×4-byte fanout, 20-byte SHA-1 names).
• Content: per-chunk git cat-file --batch subprocesses (C-speed inflate).
• Recompress: in-process flate2::ZlibEncoder at Compression::new(1).
• Tar: hand-rolled 512-byte USTAR headers matching Python’s PAX output for short names (mode 0000400, mtime 0, ustar\0/00).
• Parallelism: one chunk per logical CPU (available_parallelism()), each writing a .part.NNN file, concatenated + 1024 zero bytes.

The debugging saga (the best parts)

#

• Pipe deadlock: writing 1MB blocks to a 64KB pipe deadlocks unless the reader drains git’s stdout while the writer feeds stdin. The fix is strict interleaving in one thread: write one sha, flush, read its response — the batch protocol’s contract.
• The reporter counter bug: each worker did done.store(n_done) — overwriting the shared atomic with its own count, so the progress thread never saw the total and reporter.join() blocked forever. Fix: done.fetch_add(1).
• The flate2 backend: default flate2 uses pure-Rust miniz_oxide, which emits different compressed bytes (0/54 byte-identical). Switching to the zlib feature (system libz — the same one git/Python use) → 54/54 byte-identical. This was the single most important correctness decision.

The numbers

#

[objtar] 863129 objects from scrape\SX\git\objects\pack\pack-79e1ec...idx
[objtar] unpacked 105036/863129 (12.2%, 52406/s, ETA 14s)
[objtar] unpacked 863129/863129 (100.0%, 47933/s, ETA 0s)
[objtar] wrote ...objects-qdxyds_8.tar (863129 entries, 20.3s)

863,129 objects in 20.3s at ~50–60k/s vs Python’s ~200/s and a ~3900s ETA — roughly 100x faster. git repack -adf (~2 min) became the new bottleneck.

Release engineering

#

Max-optimized cross-compiled releases (linux + windows amd64), binaries committed to tools/bb-obj-tar/dist/:

[profile.release]
lto = "fat"
codegen-units = 1
opt-level = 3
panic = "abort"
strip = true

Windows binary statically links libgit/zlib (imports only KERNEL32/msvcrt/ntdll) — no zlib1.dll dependency. Python autodetects the platform binary (_autodetect_obj_tar_bin()), falls back to pure Python. And a live-streaming fix: the initial subprocess.run(capture_output=True) swallowed the [objtar] progress until exit — switched to Popen + line logging.

The PR Loop and the Imaginary git merge-base --batch

#

Next slowdown: the PR write loop, PR 25/10031 done every ~8 seconds. “i know it’s hard to tell from afar, but what’s so slow here?” Diagnosis: git merge-base once per PR — 10,031 subprocess spawns on Windows, each re-walking the 863K-object graph ≈ ~1 hour. First fix: precompute all merge-bases before the write loop with a ThreadPoolExecutor (16 workers), deduped by (from,to) pair → ~40/s, ETA ~4 min.

The user pushed further: “can we similarly make a rust tool that uses a git library and works in batch mode, like the imaginary git merge-base –batch that does not exist?” And when I started reverse-engineering the algorithm: “isn’t there a impl of rust git library or a binding to the existing git library impl that we can use from rust - instead of trying to reverse-engineer and reimplement the merge base algo?” — correct again. git2 0.21 (libgit2 bindings) has Repository::merge_base(one, two), keeps the ODB open across calls → batch-friendly. (gix was rejected: its merge-base docs warn “unstable / order-dependent / best-effort” — too risky for byte-exact gates.)

bb-merge-base reads <from> <to> pairs on stdin, writes one base per line. Verified with a Python verifier the user runs at work (corpus/verify_merge_base.py, expected side = parallel git CLI with --jobs 48):

10031 PR pairs -> 10025 unique
expected: 10025/10025 (54/s, ETA 0s)   # ~3.5 min
actual:   done in 0.6s
ALL 10025 UNIQUE PAIRS MATCH

0.6s vs ~183s — ~300x. Wired into the emitter with autodetection and a git-CLI fallback.

The Crash: KeyError: 'commit' at PR 2875/10031

#

The real bake crashed:

File "D:\wrk\hans\bb_archiver\emitter.py", line 292, in pr_activities
    ("hash", a["commit"]["id"]),
KeyError: 'commit'

A MERGED activity with no commit at all. corpus/find_bad_activities.py (a diagnostic that scans every activity for every hard access the emitter makes) found 6 offenders across 10,031 PR files — all action=MERGED, autoMerge:false, and no commit key. The user’s RCA: “4859 shows: Ha, Da remotely merged the pull request… this happens when PR A starts from PR Bs branch and then PR A is merged. these weird effects then happen on PR B.” Stacked PRs. And the user insisted on a lab reproduction: “definitely you need to repro it in lab.”

Reproducing the phantom-merge

#

After a license re-create (the eval license is “relative to when it’s applied to an empty env”), the repro was scripted with curl:

base (main: B0, B1)
branch1 -> A1, A2      PR A: branch1 -> main
branch2 (built ON branch1) -> B2, B3   PR B: branch2 -> main
then merge PR B

After merging PR B, PR A’s activities showed a MERGED record with no commit — reproduced exactly. And the official export’s ground truth for that PR:

{ "kind": "MERGED", "createdTimestamp": 1787161620699, "userId": "admin|Lab Admin||NORMAL", "autoMerge": false }

The exporter omits the hash key entirely — not "", not null. My first fix (emitting hash:"") was wrong. Corrected to omit the field:

items = [("autoMerge", bool(a.get("autoMerge")))]
if commit:
    items.append(("hash", commit))

Lab validation: === EXACT MATCH: True ===, gate1 NO GENUINE DIFFERENCES.

Baked into the golden

#

The user: “incorporate the scenario into the scenarios things that are played into the lab and the golden stuffs.” So fixtures gained PR6 (stacked/base, stays open) and PR7 (stacked/dependent, merged) — which shifts all subsequent PR ids (PR4→6, PR5→7). The stacked pair is a gem: when its dependent PR merges, the base PR flips to a commit-less MERGED (“phantom-merge”). The fresh golden export captured it, and all gates were re-run. Gate 2 proved the phantom survives import: PR4 activities: match (3).

Round-Trip at Real Scale, on the Work Box

#

The SX bake finished: assembled scrape/Bitbucket_export_SX202608191427_1.tar, archive complete: 30105 entries for a 10,031-PR / 863K-object repo. The user: “after it finishes, what are my real chances to verify it somehow? import into locally running lab box?” — and, crucially, “i can’t bring work product (the SX) into my home ox :D”.

So verification had to run entirely on the work box. We generalized Gate 2 (PR-count-agnostic, batch git cat-file for 863K objects instead of one subprocess per object) and made the lab scripts portable to Rancher Desktop/WSL2:

SKIP_A=1 LAB_LICENSE_FILE=/path/to/trial3h.txt ./lab/lab-up.sh
./lab/wizard.sh bb-lab-b
./lab/import.sh ./scrape/Bitbucket_export_SX..._1.tar bb-lab-b
python3 corpus/gate2.py ./scrape/SX ./scrape/sx-lab-b

New scripts: lab/wizard.sh (drives the manual setup wizard — this build won’t honor setup.*), lab/import.sh (copy into shared/data/migration/import/, chown, POST import, poll). A notable gotcha: the import endpoint reads only from migration/import/, not migration/export/. Expected benign diffs at scale: stub users (displayName=slug), reallocated comment ids, activity ordering churn, lost title-edit UPDATED events.

Gate 2B also needed a fix: the re-exported target legitimately contains both hierarchy ids (the imported repo’s original and its own new one) — _/repository/hierarchy_begin/... markers are per-instance bookkeeping and must be excluded.

Verification: What Actually Holds It Together

#

The project’s credibility rests on four scripted gates, all reproducible:

• Gate 1 — golden master: assemble from REST + mirror, semantic-diff vs the real admin export. NO GENUINE DIFFERENCES.
• Gate 2 — round-trip: import into bb-lab-b, rescrape, diff everything including git objects object-wise. PASS (all 7 PRs incl. the phantom-merge, 65 git objects identical).
• Gate 2B — official re-export: export bb-lab-b with the official endpoint, diff vs golden. PASS — only genuine diff is title-edit UPDATED (round-trip loss).
• Gate 3 — end-to-end GEI: gh bbs2gh migrate-repo accepted the archive and migrated with byte-identical git SHAs, correct PR states/merge flags, branches, tags, reviews, comment bodies, and author identities.

Known Fidelity Gaps (Honest Ones)

#

• Title/description-edit ACTIVITY/UPDATED events are not recoverable — the real exporter records them; REST /activities does not. The tool’s archive has fewer UPDATED records. Benign — GEI reads final title/desc from PR metadata.
• Repo commit-level comments aren’t exported by the real exporter either — parity.
• Hard-deleted comments leave no trace in REST or the real archive.
• allParticipants array order is an internal DB position, compared as a set.
• rescopedTimestamp internal target-advance approximated from the last REST RESCOPED. Benign.
• Users import as stub accounts (displayName=slug). The only email fidelity is in git author/committer objects — which is exactly what GitHub attribution/mannequin reclaim keys off.
• Merged PRs whose head branch was deleted lose their git refs and can fail on the GitHub side (real exports too). Flagged, not fixed.
• PR tasks have no /tasks endpoint, but live as severity:"BLOCKER" comments and migrate correctly (resolvedDate/resolver → resolvedTimestamp/resolverId).

Lessons

#

1. “No admin access” is a spec, not a blocker. The export format has no signatures or checksums — it’s a plain tar of bare repos + JSON. If the content is derivable read-only, the archive is reconstructable. Reverse-engineer the real artifact as ground truth, then script the gates.
2. Byte-level fidelity is a property of the writer, not the data. Two Jackson serializers, uppercase surrogate escapes, no trailing newlines, zeroed gzip MTIMEs — matching the exporter’s serializer quirks is correctness, because nothing else pins the bytes. cat -A and xxd earn their keep.
3. REST and archive speak different activity schemas. REST action ≠ archive kind; one REST event maps to multiple archive kinds (COMMENTED → COMMENT:ADDED + COMMENT:OTHER; APPROVED → REVIEWERS:UPDATED + ACTIVITY). Never assume the exporter’s model mirrors the API’s.
4. Don’t fetch via REST what the git mirror already gives you. Per-commit REST was a 100K-request killer; all of it lives in the mirror. When the user asks “why do you need this?”, the answer is usually “you don’t.”
5. The user is a better source of domain truth than my first web search. Tasks = severity:"BLOCKER" comments was the user’s memory, verified empirically; stacked-PR remote merges were the user’s RCA. When the user says “reproduce it in the lab,” reproduce it in the lab.
6. For byte-identical output, match the compression backend. miniz_oxide ≠ system zlib ≠ git’s libz. 0/54 objects matched until flate2 switched to its zlib feature. Compression is part of the format.
7. Batch interfaces beat process-per-item — in any language. git merge-base --batch doesn’t exist, so we built bb-merge-base on libgit2’s open ODB: 10025 pairs in 0.6s vs 183s of git merge-base subprocesses. And pipe protocols deadlock if you don’t strictly interleave write/read.
8. A crash at PR 2875 is a data-shape bug, not a flake. The commit-less MERGED activity came from stacked PRs: when PR B (built on PR A’s branch) merges, PR A flips to a phantom commit-less MERGED. The exporter omits hash entirely — omit it too, don’t empty-string it.
9. Progress reporting is a feature, not decoration. 35 silent minutes looks like a hang; a timestamped PR 25/10031 done line turns anxiety into ETA. ISO timestamps on every line made the user’s work-box debugging possible.
10. Test at the scale you’ll actually use. The lab proved correctness; the 10,031-PR/863K-object bake found the real bugs. The verifier was designed for rate limits, resumability, and Ctrl-C from day one — and it survived a real 10K-PR repo.

The entire hans project — from the first container to the final gate — was built in a two-day OpenCode session assisted by DeepSeek V4 Flash, which performed the lab automation, format reverse engineering, Python tooling, Rust performance helpers, and validation gates described above. The user’s product knowledge (tasks-as-BLOCKER-comments, the stacked-PR mechanism, the license semantics) was decisive at three separate moments.