30 releases 32 tags

RSS feed

• v0.10.0 2f4f066920

  Compare

  v0.10.0 Stable

  joeyr released this 2026-07-31 08:54:18 -05:00 | 238 commits to main since this release

  Fixed

  • Classic counters were about to publish in the wrong id namespace, and it
    would have failed silently. crawlCounters is shared by both passes, and it
    keyed entries by whatever id the tier list gave it. op.gg indexes classic
    pages by BASE champion id — its classic tier list returns champion_id: 24
    for Jax and the page lives at /champions/jax, while jade_jax 307s — so the
    file would have been written with base ids while champ select and the live
    client both report 60xxx.

    That is the failure mode this repo spends most of its comments avoiding: the
    artifact validates, reads correctly, and diffs sensibly night to night, and
    not one lookup ever matches. The runes pass already shifted its keys; counters
    never got the same treatment because nothing exercised it.

    Three maps needed it, not one:

    • the entry keys (counters['60266']),
    • the opponent ids inside every matchup — op.gg names classic opponents by
      their base display name ("Jax"), so the directory resolves them to base ids
      while the consumer's board holds 60xxx, and
    • the tier list, which is a second consumer-facing id-keyed map.

    toQueueChampionId (new src/classic-ids.ts) is idempotent, so applying it
    at more than one point in the assembly path cannot double-shift an id into a
    range that exists for nobody. Six tests cover it, all four of which fail
    against the previous code; the ranked namespace is asserted alongside each,
    since the shift must happen for exactly one queue.

    No published file is affected — the classic pass is opt-in and has never run.

  • Classic counters recorded the rank band the crawl asked for, not the one
    op.gg served. Same false-provenance problem the classic runes schema already
    fixed: op.gg accepts tier on a classic page and discards it, so the data is
    all ranks whatever was requested. CLASSIC_TIER moved to common.schema.ts,
    where both classic artifacts can reach it, and the counters artifact now
    records all for that queue.

  Changed

  • The classic universe is 90 (champion, role) pairs, measured. A full sweep
    on 16.15 replaced the 81 an earlier estimate assumed — TOP 23, MIDDLE 22,
    JUNGLE 21, UTILITY 15, BOTTOM 9. The nightly op.gg budget is therefore ~732
    requests rather than 714, which at 12.5 s is ~2.54 h and still inside the
    three-hour window. The pacing is unchanged; the arithmetic behind it now
    matches reality.

  • The 0.9 coverage floors are kept for Classic Rift, on evidence rather than
    by default. The concern was that a proportional floor is harsher on a
    smaller universe. The same sweep returned 87/90 counters pages (96.7%) and
    90/90 runes pages (100%), so both clear it with real headroom. The three
    counters misses are stable rather than random — Fiddlesticks, Nunu and Gragas,
    all jungle, all serving a page with no data array — which is why the margin
    can be trusted. Split the floors only if a later measurement says so.

  • Dropped the dead runesArtifact: null branch from OpggPass and the comment
    claiming Classic Rift has no rune extractor. It has had one since 0.9.0.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.9.0 835bb28a20

  Compare

  v0.9.0 Stable

  joeyr released this 2026-07-31 07:07:46 -05:00 | 239 commits to main since this release

  Changed

  • The op.gg and Riot halves now run concurrently, so pacing op.gg no longer
    costs the Riot crawl its window. They are unrelated hosts with unrelated
    limits and separate rate limiters — serialising them only ever meant one sat
    idle while the other waited out its own pacing. That was affordable when the
    op.gg half finished in ~14 minutes. At 12.5 s a page it takes ~2.5 hours,
    which would have left the Riot crawl ~30 minutes of a three-hour window, and
    champion-damage and champion-benchmarks are both built from those rows.

    Run together, each half gets the entire window instead of a slice. The crawl
    window no longer needs widening — the recommendation in 0.8.0's notes is
    withdrawn.

    Promise.allSettled, not Promise.all: a rejection in one half must not
    cancel the other, which is the same isolation the per-artifact try/catch gives
    inside each.

    The two op.gg passes (ranked, then classic) stay sequential with respect to
    each other. They are the SAME host, and running them together would double the
    request rate op.gg sees and undo the pacing.

  Fixed

  • Concurrent halves could have raced on the Forgejo branch ref. Both publish
    to the same branch, and the contents API's concurrency guard is the per-file
    blob sha — it stops two writers clobbering one file and says nothing about two
    commits racing to move the ref, which is what two different artifacts
    publishing at the same instant would do.

    Commits now go through a one-at-a-time queue (publish-queue.ts). Publishing
    is seconds against crawls measured in hours, so serialising it costs nothing
    measurable. A failed publish does not wedge the queue: later artifacts still
    commit, and the caller still sees its own rejection. Validation happens
    before queueing, so a schema failure surfaces as that artifact's error
    rather than waiting behind another half's commit.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.8.0 d5e5b262b3

  Compare

  v0.8.0 Stable

  joeyr released this 2026-07-30 23:45:01 -05:00 | 240 commits to main since this release

  Changed

  • op.gg is now paced at one page per 12.5 s, taking the nightly crawl from
    ~14 minutes to ~2.5 hours. Derived rather than picked: the window makes 714
    requests (5 tier lists + 271 counters + 271 runes for ranked, plus 5 + 81 + 81
    for classic), and 9000 s / 714 ≈ 12.6 s.

    The old 1.5 s pacing walked the whole universe in a burst, and
    champion-counters was aborting at 55% coverage — the shape you get when
    op.gg starts serving its 403 challenge page partway through. A 403 is
    deliberately not retried, so once it starts, every remaining page fails
    instantly and coverage collapses to however far the crawl got. This does not
    prove that diagnosis, but bursting was the one thing under our control.

  • COUNTERS_MAX_MINUTES and RUNES_MAX_MINUTES raised 45 → 70, and they are
    coupled to the interval. At 12.5 s a 271-pair crawl needs ~57 minutes, so
    the old 45-minute cap would have stopped it two thirds through and then
    aborted it on coverage — turning a pacing change into a permanent outage.
    Raise the caps with the interval or not at all.

  • CLASSIC_ENABLED now defaults to OFF. Every other artifact is opt-out;
    this one is opt-in. It adds ~167 requests to a window whose ranked half is
    already failing for reasons nobody has established, and cumulative volume is
    the last thing that should grow by default while that is open.

  Notes

  The crawl window probably needs widening, and this release does not do it.
  CRAWL_START_HOUR=2 / CRAWL_END_HOUR=5 is three hours. An op.gg half taking
  2.5 h of that leaves the Riot crawl ~30 minutes where it used to have ~2h45 —
  and champion-damage and champion-benchmarks are both built from those rows,
  so they would thin out night over night. 02:00–07:00 gives each half ~2.5 h.
  That is a deployment decision, so it is flagged here rather than changed.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.7.0 73e0b261ae

  Compare

  v0.7.0 Stable

  joeyr released this 2026-07-30 23:40:15 -05:00 | 241 commits to main since this release

  Added

  • champion-runes-classic.json is published. v0.6.0 shipped the extractor
    with nothing wired to it; the classic pass now crawls rune pages alongside
    counters and publishes them.

    Its own schema, not the modern one with a flag. Classic Rift restores the
    pre-2017 rune system, which is a different data model rather than a variant:
    no keystone, no primary/secondary path, no shard rows, no selectedPerkIds —
    the things champion-runes.schema.ts is built around. A page is a flat
    multiset of four rune kinds held in quantities. Forcing it into the modern
    shape would mean inventing empty paths or making half that schema optional,
    which would weaken the validation currently catching drift on the ranked file.

    No perk dictionary either. The modern artifact hoists display strings because
    inlining measured 1.9 MB against 212 KB across ~271 pairs of ~36-option
    boards; classic is ~81 pairs carrying four runes each, so inlining costs a few
    KB and removes a rehydration step from the consumer.

  Fixed

  • The classic artifacts no longer claim a rank band they do not have.
    op.gg accepts the tier parameter for classic pages and discards it —
    iron_plus, challenger and a deliberately invalid value all return
    byte-identical rows. Recording the band the crawl asked for would be a false
    provenance claim, and anything comparing classic to ranked figures would
    silently compare a gold+ population against an everyone population. The files
    now record all.

  Notes

  Champion ids are re-keyed on the way out, and this is the subtle one. op.gg
  indexes classic CHAMPION pages by BASE id — the tier list returns 24 for Jax and
  the page lives at /champions/jax, while jade_jax 307-redirects — but champ
  select and the live client both report 60xxx, and the consumer looks entries up
  by the id it is holding. So the crawl takes base ids in and publishes classic ids
  out. A file keyed by base ids would validate, read correctly in review, and never
  match a single lookup. Rune ids need no such treatment: op.gg already publishes
  those in the 775xxx classic block, as it does items in 77xxxx. Champions are
  the exception.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.6.0 c154a17ced

  Compare

  v0.6.0 Stable

  joeyr released this 2026-07-30 21:35:39 -05:00 | 242 commits to main since this release

  Added

  • Classic Rift is crawled as its own pass, publishing
    champion-counters-classic.json. op.gg serves the mode under
    type=classic, and the pass reuses everything except that parameter and the
    output paths — same region, tier, coverage floors, pacing and time budgets —
    so the classic file describes the same population as its ranked counterpart.

    It is a separate file rather than extra rows, and that is forced. op.gg
    keys classic pages by BASE champion id: champion_id: 24 is Jax in both
    queues, measured on two different kits. Merged into one map they would collide
    on the key and one would silently win.

    Classic is the cheaper half by a wide margin — 81 (champion, role) pairs
    against ranked's ~272 — so it runs second, where a short budget costs the
    ranked artifacts nothing. Its outcomes get their own publish streaks, because
    its file goes stale independently of theirs.

  • A classic rune extractor (opgg/classic-runes-page.ts), not yet wired to
    an artifact. Classic Rift restores the pre-2017 rune system, which is a
    different data model rather than a variant: no paths, no keystone, no shards,
    just a flat multiset of Marks, Seals, Glyphs and Quintessences held in
    quantities. op.gg serves it from the same runes page under classic_runes,
    with 775xxx ids and a win COUNT rather than a rate, normalised to a percent
    on the way out to match the other artifacts. Verified against live pages.

  Fixed

  • buildTierListUrl omitted type, which would have made the classic crawl
    describe ranked champions. The tier lists are the index both op.gg crawls
    run off, and op.gg defaults the parameter to ranked when it is absent — so
    the ranked pass was correct by luck. A classic pass would have discovered the
    RANKED universe and then gone looking for those champions on classic pages,
    producing a classic-labelled artifact full of the wrong roster. That is a
    wrong file rather than an empty one, which is the failure this codebase is
    built to avoid. Found before the classic pass ever ran.

  Notes

  Two things op.gg does not publish for this mode, established by sampling every
  role: masteries (legacy_masteries is an empty array on every page) and
  synergies (play: 0, and the page itself says the sample is too small).
  Neither is crawlable, and neither is a gap on this side.

  Classic data is also not segmented by rank: the tier parameter is accepted
  and discarded, with iron_plus, challenger and a deliberately invalid value
  all returning byte-identical rows. The published file therefore describes all
  ranks, and a later release should stop labelling it gold_plus.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.5.0 ee5b20e8b2

  Compare

  0.5.0 — escalate a chronically stale artifact Stable

  joeyr released this 2026-07-30 09:07:11 -05:00 | 243 commits to main since this release

  Declining to publish is correct — the coverage floors exist so a thin crawl cannot replace a good file — and it is also completely silent. The night a failure becomes chronic logs exactly what the first night logged, and the window still reports success on the strength of the artifacts that did publish.

  Patch 16.15 is the worked example: counters aborted at 55% coverage two nights running while champion-counters.json kept serving 16.14 data, and the only trace was one WARN among 122 others.

  Added

  Each of the four artifacts now carries a miss streak in a new publish_state table.

  outcome	level
  missed, streak < STALE_AFTER_NIGHTS (3)	WARN
  missed, streak >= STALE_AFTER_NIGHTS	ERROR, naming the streak and daysSinceSuccess
  missed because something threw	ERROR on the first night
  published after a streak	INFO, naming the missed nights
  published normally	nothing — the commit line already said so

  • It counts nights, not runs. The counter advances at most once per local calendar date. RUN_ON_START and a redeploy both re-enter the window, so without that guard, deploying a fix would trip the alarm it was fixing.
  • Faults skip the streak. A coverage abort is an expected non-result; a schema regression or a 403 from Forgejo is wrong immediately, and giving it three nights would bury the one signal worth acting on.
  • The state is in Postgres, not beside the heartbeat, which lives in an ephemeral /tmp — a redeploy would reset every alarm, and a redeploy is what happens right after shipping a fix for the thing being tracked.
  • Nothing is recorded when PUBLISH_ENABLED=false. A smoke run has no publish outcome, and letting it write would leave the production streaks reading stale.

  Fixed

  • A damage publish failure no longer costs the benchmarks artifact. The two come from one crawl and were documented as publishing independently, but shared a single try/catch — so a schema regression in the first skipped the second, built from rows already paid for.
  • The heartbeat is no longer touched after a Riot half that published nothing. runRiotWindow returned void and the caller marked the window successful unconditionally, so a night that dedup'd to zero new rows still reported health to the container healthcheck.

  Upgrading

  No schema change to any published artifact; consumers need no update. The migration is additive and idempotent. One new optional env var, STALE_AFTER_NIGHTS (default 3).

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.4.1 7760c9cce5

  Compare

  0.4.1 — build identity in the boot line Stable

  joeyr released this 2026-07-29 09:17:41 -05:00 | 251 commits to main since this release

  Changed

  The boot line now identifies the build and the config that decides whether a night produces anything.

  version and commit are stamped into the image by CI via --build-arg, using the same values it tags the image with. This service sleeps 21 hours a day and Swarm pins :latest without ever re-resolving it, so a deploy that silently did not pull was indistinguishable from one that did until the next window opened — telling them apart meant resolving the running task's image digest against the registry by hand. A version of dev in a container means the build args never reached docker build.

  The line also carries what is otherwise invisible until too late: publishEnabled (false crawls all night and commits nothing), both coverage floors (what withheld counters and runes on 2026-07-28), lookbackHours, the target owner/repo@branch, and archiveDir.

  DATABASE_URL is never logged — it carries the password inline. It is reduced to host/database.

  Upgrade notes

  No consumer change; logging and build metadata only.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.4.0 b4c4d2df63

  Compare

  0.4.0 — canonical-name fix and patch archiving Stable

  joeyr released this 2026-07-29 09:06:04 -05:00 | 253 commits to main since this release

  Fixed

  Champion display names stopped being unique, and it took both op.gg artifacts down.
  Patch 16.15 added 60 Jade_* entries to Data Dragon (173 champions -> 233), each carrying its base champion's display name. ChampionDirectory built its name->id maps last-write-wins, so for the 18 names whose base alias sorts before the Jade_ block, the 60001+ id won. That id picks the op.gg slug, and jade_annie is not a page: 31 of 272 (champion, role) pairs failed, coverage fell to 88.6%, and the 90% floor correctly withheld both counters and runes on the night of 2026-07-28.

  Name lookups now keep the lowest id. The id->slug direction never needed a fix.

  Added

  A patch's final artifacts are archived instead of overwritten. The four canonical paths still hold whatever patch is live — lol-companion fetches one fixed URL forever, unchanged. On the first run of a new patch the outgoing file is copied to ARCHIVE_DIR/<old patch>/<same basename>, e.g. data/archive/16.14/champion-runes.json.

  The trigger is the committed file's own generatedFrom.patch differing from the one being published, so no schedule or stored state is involved and a failed copy retries the next night. The bytes come from the read the no-op check already performs. Existing archives are never rewritten, and an archiving failure is logged rather than costing the publish.

  Kept indefinitely: ~3 MB per patch across the four files.

  Upgrade notes

  No consumer change. ARCHIVE_DIR defaults to data/archive; the deployed stack sets it explicitly.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.3.1 8538db6dd7

  Compare

  v0.3.1 Stable

  joeyr released this 2026-07-28 09:03:03 -05:00 | 256 commits to main since this release

  Changed

  • Log lines name their severity — INFO, WARN, ERROR — instead of
    pino's numeric scale. The Portainer dashboard renders whatever the level
    field holds, so a nightly run showed a bare 30 where the severity should be,
    readable only if you already know that debug is 20 and error is 50. The
    LOG_LEVEL threshold is unaffected; it took names already.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.3.0 f3741794c8

  Compare

  v0.3.0 Stable

  joeyr released this 2026-07-28 08:56:18 -05:00 | 257 commits to main since this release

  Changed

  • The counters and runes artifacts are published readably instead of
    minified. Both were committed as a single line, which kept them small but
    made them unreadable in the Forgejo UI and gave git no line structure to delta
    between nightly rewrites. They now use a width-bounded printer: structure on
    its own lines, but any subtree whose minified form fits in 120 columns stays
    inline, so a matchup or a rune board row remains on one line. Measured on
    patch 16.14 — runes 398 KB → 539 KB (12,747 lines) against 1.47 MB at full
    indentation; counters 911 KB → 1.08 MB (16,220 lines) against 2.01 MB. The
    damage and benchmarks artifacts are unchanged, still fully indented.

    The JSON data is byte-for-byte equivalent, so this is not a schema change
    and consumers need no update. The first run after deploying rewrites both
    files once, then nightly diffs resume — and are now legible.

  • The publish option that controlled this is now an ArtifactFormat
    (INDENTED_FORMAT / COMPACT_FORMAT) rather than a bare indent number.

  Added

  • Images are tagged with the package version (:X.Y.Z) alongside :latest
    and the commit SHA, so a running container can be matched to the vX.Y.Z tag
    and release it was cut from.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads