30 releases 32 tags

RSS feed

• v0.18.0 84c29e9b08

  Compare

  0.18.0 Stable

  joeyr released this 2026-08-04 16:18:52 -05:00 | 188 commits to main since this release

  Changed

  • champion-counters.json is now built from Riot match-v5, not scraped from
    op.gg. generatedFrom.source reads riot. No schema change and no consumer
    change: source has always been z.string().min(1), "present so a future
    second source is a value, not a fork".

    Matchups come from a self-join of participant_stat — two participants in the
    same match, at the same role, on opposite sides of the win partition. It
    needed no capture phase, unlike runes and items: every column it reads has
    been recorded since the table existed, so the swap built a complete file from
    rows already on disk. Zero additional Riot requests; ~271 fewer op.gg pages a
    night.

    What changes downstream is the sample. op.gg's file was built from a
    population and a window nobody here could see — roughly 192,000 matches on
    16.15, against the ~12,000 this service had collected at the same point. Ours
    is smaller, it is the games this crawl actually observed in the seeded ladder
    band, and it grows for as long as the patch runs. Every cell carries its own
    games.

    Two differences worth expecting rather than debugging: games per matchup
    is roughly an order of magnitude lower and climbs across a patch instead of
    arriving whole, and mirror matchups are absent — the crawl is queue 420,
    where draft pick makes champion selection exclusive across both teams, so
    Yasuo never faces Yasuo.

  • tierList still comes from op.gg, and it is now the only thing that does
    for ranked. It is op.gg's {tier, rank} bucketing, which is a score;
    deriving one here by sorting champion-roles on win rate would be inventing a
    scoring function and publishing it as though measured. Five tier-list pages a
    night, down from 276.

    A night op.gg cannot be reached publishes five empty arrays, never
    yesterday's list — the file is judged as a whole on generatedFrom.patch, so a
    stale tier list inside a current-patch artifact reads as current. The matchups
    publish as normal, and lol-companion's TierListProvider already rescrapes a
    role it finds empty.

  • COUNTERS_ENABLED now gates both sources, ranked and classic, exactly as
    RUNES_ENABLED does. COUNTERS_MIN_COVERAGE and COUNTERS_MAX_MINUTES are
    classic-only from this release: they describe a page crawl, and the ranked
    build is a database self-join.

  • One op.gg client for the whole window. The ranked tier lists and the
    classic pass used to be sequential calls that each built their own; the ranked
    half is now consumed by the Riot half, so they can overlap. Sharing one client
    means one limiter and one pacing, rather than two limiters at twice the rate.

  Removed

  • rankedPass in index.ts. The only op.gg pass is classic — the mechanism,
    deliberately, rather than a flag that could put ranked counters back on op.gg.
    src/counters.ts and the op.gg page extractors are untouched and still live:
    queue 4310 is absent from match-v5, so op.gg remains classic's only source.

  Notes

  • No consumer change is required or expected. The schema is byte-identical
    and lol-companion's mirror parses source: 'riot' today.
  • The first published file after this deploys describes whatever the crawl has
    stored for the live patch. Mid-patch that is a full file; on patch-drop
    morning it is a thin one, correctly labelled — where the op.gg build used to
    abort on its coverage floor and leave yesterday's file live instead.
  • New index participant_stat_pair_idx (patch, match_id, role), applied at
    boot. It serves the self-join; the existing rollup index leads on champion and
    cannot.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.17.0 ecce9f0801

  Compare

  0.17.0 Stable

  joeyr released this 2026-08-04 16:04:09 -05:00 | 189 commits to main since this release

  Added

  • The crawl now records each participant's final inventory. item0–item6
    are in every match payload this service already fetches, so capturing them
    costs no additional Riot requests — the same argument that made benchmarks and
    runes cheap. New participant_stat.items INTEGER[] column, populated for all
    ten participants of every match.

    Nothing is published from this yet. It ships alone and first because it is
    the only part of the item work whose cost is wall-clock: retention keeps two
    patches, so an inventory not captured while its patch is live cannot be
    recovered afterwards. Deploy it the day it merges; the artifact that reads it
    follows later.

    All seven slots are stored raw, empty slots and trinket included. Which id
    is a legendary, a boot or a ward is a Data Dragon judgement made when an
    artifact is built, and Data Dragon moves — classifying at capture time would
    mean a re-crawl after every item rework, from rows that no longer exist. The
    zeros are also the sentinel: a captured row is always seven integers, so
    items = '{}' identifies a row collected before the column existed, exactly
    as primary_style = 0 does for runes.

  Notes

  • The migration is an ADD COLUMN IF NOT EXISTS with a default, applied at
    boot alongside the existing ones. No downtime, no backfill, and rows written
    before it carry the empty-array sentinel.
  • No consumer change. No published artifact reads this column yet.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.16.0 40825bb5b7

  Compare

  v0.16.0 — ranked runes from match-v5, not op.gg Stable

  joeyr released this 2026-08-04 14:30:41 -05:00 | 199 commits to main since this release

  Changed

  • champion-runes.json is now built from Riot match-v5, not scraped from
    op.gg. generatedFrom.source reads riot. No schema change: a combo was
    already a (primary tree, secondary tree) pair rather than an exact nine-perk
    selection, so the published shape is identical and consumers need no update.

    The rune page is in every match payload this service already pays for, so the
    artifact is now a third reading of the same rows that produce
    champion-damage.json and champion-benchmarks.json — alongside a rollup in
    SQL and two static CommunityDragon fetches. It costs no additional Riot
    requests and it costs ~271 op.gg pages fewer per night.

    What changes downstream: the sample is the games this service actually
    collected, on the current patch, from the seeded ladder band, and it grows for
    as long as the patch runs. The op.gg numbers came from a population and a
    window nobody here could see.

  • The ranked op.gg pass runs counters only. rankedPass carries no rune
    artifact or path at all, so op.gg cannot publish that file regardless of
    configuration. Counters stays on op.gg because the matchup tail needs roughly
    fifteen times the sample a night collects.

  • RUNES_ENABLED now gates both sources, ranked and classic — "publish rune
    data" is one decision. RUNES_MIN_COVERAGE and RUNES_MAX_MINUTES are
    classic-only from this release: they describe a page crawl, and nothing on
    the Riot half reads either.

  Removed

  • The op.gg-sourced ranked rune data, in the previous release's data
    commits: data/champion-runes.json and data/archive/16.14/champion-runes.json.
    Every *-classic rune file and every classic archive is untouched — queue
    4310 is absent from match-v5, so op.gg remains its only possible source.

  Notes

  • champion-runes.json has been absent since the op.gg crawl was removed and is
    recreated on the first nightly run after this deploys. Installs with no cached
    copy have no ranked runes until then.
  • The file grows across a patch. Every tree pair with at least one game is
    published and the count per champion-role climbs as games accumulate: measured
    at 532 KB with four combos each against 2.56 MB at all twenty. The
    compaction path if that becomes a problem is a boards dictionary keyed by
    primary:sub, which is a schema change and therefore a separate, ordered
    piece of work.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.15.0 0c5263afbd

  Compare

  v0.15.0 — tell an empty op.gg items page from a broken one Stable

  joeyr released this 2026-08-04 09:43:06 -05:00 | 204 commits to main since this release

  Separates "op.gg has no build for this pair" from "the parser is broken".
  They arrived at the crawl as the same event, and only one of them is a problem.

  Why

  The first production run of the classic items crawl covered 88 of 90 pairs. The
  two it missed — Nunu support and Kog'Maw mid — render on op.gg with no core
  build at all, and the crawl counted them as parse failures. They spent the 90%
  coverage floor, and championRoles: 88 against a 90-pair universe was
  indistinguishable from two pages having broken.

  Harmless at 2 of 90 and wrong later. This mode only thins out, and the first
  night it crossed the floor the crawl would have aborted on a run that was
  entirely correct — reporting a healthy crawl as a parser bug.

  How

  One page cannot fully separate the two: "op.gg has no build" and "op.gg renamed
  metaBuildItems" are both zero core builds. What a page can prove is that
  op.gg served the items route, because Next.js ships only the active route's i18n
  namespace. So the page answers what it can and the run settles the rest:

  • anchor present, no core build → empty; the pair still counts as
    answered toward coverage, because op.gg replied and the crawl read it;
  • anchor absent → thrown, counted as a failure exactly as before. A
    challenge page or a redirect is not evidence about any champion;
  • nearly every answered pair empty → abort, naming the section keys. That is
    a shape change, not a thin mode, and the coverage floor is blind to it: every
    page answers, so it reads a clean 100% and would publish an empty artifact.

  The new ceiling is ITEMS_CLASSIC_MAX_EMPTY (default 0.5). Last measured, 2 of
  90 answered pairs were empty; the failure it guards against sits near 100%.

  Added

  • generatedFrom.emptyPages on champion-items-classic.json — how many
    pairs op.gg rendered but published nothing for, so championRoles short of the
    universe is readable rather than ambiguous. Optional: absent means unknown, not
    zero, for artifacts written before the distinction existed.

  Verified against the real captured HTML for both empty pairs and a populated one,
  plus a truncated response to confirm it still throws. Gate green: typecheck,
  lint, 338 unit, 30 integration, build.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.14.0 6cdd57386a

  Compare

  v0.14.0 — classic runes, masteries and item builds Stable

  joeyr released this 2026-08-03 16:13:31 -05:00 | 213 commits to main since this release

  Publishes every Classic Rift loadout op.gg serves, rather than only the rows
  with a sample behind them.

  Added

  • champion-masteries-classic.json — the pre-2017 mastery trees, two pages
    per champion-role at 30 points each. Costs no additional requests: op.gg
    embeds them in the same page object the rune crawl already fetches.

    Nothing in this file is measured. op.gg publishes no mastery statistics for
    this mode, so every row is basis: "recommended" and a consumer must not
    present these as what players run. It is the first artifact here that is not a
    measurement, which is why the discriminator is mandatory rather than implied.

    This repo previously stated in three places that legacy_masteries was "an
    empty array for every champion sampled". True when written; op.gg has since
    populated it. The comments are corrected.

  • champion-items-classic.json — core builds, boots, starters and op.gg's
    situational list. Its own pass with its own floor and budget, since items are
    the one classic page that cannot ride on another fetch.

  Changed

  • champion-runes-classic.json now publishes all three rune pages, not just
    the measured one. The file held exactly one combo per champion-role where
    op.gg's page shows three.

    Breaking to the artifact's shape — a combo is now a discriminated union on
    basis, with the stat fields absent on a recommended page rather than
    zeroed, so an editorial build cannot sort or filter alongside a measured one.
    generatedFrom gains measured / recommended counts.

    No consumer read this file at the time of the change.

  All three extractors were verified against live op.gg pages, not fixtures alone.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.13.0 80f6258bfd

  Compare

  v0.13.0 — readable error lines Stable

  joeyr released this 2026-08-02 20:52:43 -05:00 | 220 commits to main since this release

  stack_trace was the last of the five keys Portainer's log viewer colours to go
  unused — and the one that mattered on the worst night.

  What an error used to look like

  err={"type":"Error","message":"op.gg returned HTTP 403","stack":"Error: op.gg
  returned HTTP 403\n    at crawlCounters (/app/dist/src/counters.js:316:11)\n
  at async runOpggWindow (/app/dist/src/index.js:466:7)"}
  

  Three separate problems, and the dashboard could not help with any of them.
  err is not one of the five keys it knows, so the whole error was demoted to a
  trailing key=value at the end of the line. That renderer flattens a nested
  object to JSON. And the stack inside it was a single string with its newlines
  escaped, so the frames printed as literal \n on one unbroken line — the least
  readable form of the most useful field.

  What it looks like now

  counters/classic > counters crawl or publish failed  queue=classic
    error=OpggHttpError: op.gg returned HTTP 403 for https://op.gg/…/counters/top
      at crawlCounters (/app/dist/src/counters.js:316)
      at runOpggWindow (/app/dist/src/index.js:466)
  

  error is rendered red — formatKeyValuePair special-cases exactly the key
  names error and ERR — and stack_trace gets the dedicated renderer that
  prints one indented at func (source:line) per frame, which is the shape a
  stack was always meant to have.

  The class name is kept when it adds something. OpggHttpError versus
  RiotApiError says which half of the window failed before the message has to,
  and CountersCrawlAborted says the crawl stopped itself rather than broke. A
  bare Error says only that something threw, so it is left off.

  No call site changed

  The transformation is a formatters.log hook, which runs before pino's own
  error serialiser and therefore still sees the Error instance rather than the
  {type, message, stack} object it would otherwise become. That ordering is the
  whole opportunity.

  Being a property of the logger rather than of fifteen call sites also means a
  new logger.error({ err }, …) gets this for free instead of silently
  regressing. The six call sites that pass a pre-stringified err.message rather
  than the error get the red error key too, without being touched — they simply
  have no stack to give.

  Details

  • Stacks are capped at 12 frames. An async throw in Node carries twenty-odd,
    most of them node:internal/… plumbing below the first line of this codebase,
    and Portainer prints every frame it is given. A cap rather than a filter on
    node:internal/: the interesting frame is occasionally in there, and a filter
    that hides the cause is worse than a list that is slightly too long.
  • Frame parsing is anchored on the trailing numbers, because a source is
    routinely C:\…\index.js:527:5 or file:///C:/…/index.js:527:5 — both carry
    colons of their own, and splitting on the first takes the drive letter for the
    path.
  • A wrapped error's cause is reported beside it. OpggParseError takes
    one, and without this the line named the wrapper and lost what it wrapped.
  • A thrown non-Error carries its contents rather than [object Object].
    throw accepts anything, and a line that reports a failure and then spends
    its one field saying nothing is barely better than no line at all.

  ---

  Minor rather than patch for the changed log output. No artifact, schema, or
  published file changes. This completes the Portainer work: all five keys the
  dashboard renders — time, level, caller, message, stack_trace — are
  now used.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.12.0 ba02af120f

  Compare

  v0.12.0 — a boot preflight that proves every dependency Stable

  joeyr released this 2026-08-02 18:05:04 -05:00 | 221 commits to main since this release

  The boot log now proves each dependency instead of asserting it. Five checks,
  one request each, concurrent, reported as a block:

  preflight/db      | ok  server=PostgreSQL 17.10
  preflight/riot    | ok  platform=na1 band=GOLD I entries=205
  preflight/patch   | ok  patch=16.15
  preflight/forgejo | ok  target=joeyr/lol-companion-data@main push=true
  preflight/opgg    | ok  query=na/gold_plus/ranked bytes=749242
  preflight/summary | every dependency answered checks=5
  

  Why at boot

  This service is asleep 21 hours a day and does everything that matters in one
  unattended burst, so every dependency it has was discovered broken at the worst
  possible moment. A rejected Forgejo token surfaced at the commit, after a full
  night of crawling had already been paid for in API requests. A bad
  RIOT_API_KEY surfaced at 02:00 with nobody watching. An op.gg block surfaced
  as a coverage abort two hours in. In each case the container had been running
  for hours reporting nothing wrong, because it had not yet tried anything.

  The Forgejo check

  The one that pays for the rest. It verifies three things with three different
  fixes, and a naive "does the token work" probe passes two of them:

  • a read-only token authenticates perfectly and then fails at the commit, so
    permissions.push is checked directly rather than inferred from a 200;
  • a token scoped to the wrong repo 404s in a way that reads like a deleted
    repo;
  • a missing dataBranch is a typo in one env var, so the branch endpoint is
    checked by name — the failure line quotes the branch it looked for.

  Data Dragon, listed for the first time

  It is the only dependency with no credential to get wrong, which is why it went
  unlisted. But every artifact is stamped with the patch resolved from it and the
  crawl filters matches on it, so an outage does not degrade a window, it ends
  one. The resolved patch earns its line anyway: it answers "which patch is
  tonight's data for?" hours before anything else would say.

  op.gg reports bytes, not a status

  Its failure mode is a 200. The challenge page is served with an OK status and no
  data — that is why page.ts sends a desktop User-Agent at all. Four figures
  where six are expected is that page, and it is visible at a glance.

  Nothing here is fatal

  A dead Forgejo token must not cost a night of Riot data that a fixed token could
  publish tomorrow; the two halves already fail independently by design. And a
  boot that exited would have Swarm restart it on a loop, churning away the very
  lines that say what is wrong. The checks report, the operator decides.

  They are probes rather than crawls: no retries, and a 10s cap each. A dependency
  that is merely slow at boot is not worth delaying the boot over, and the answer
  a retry would eventually get is the answer the window will get for itself.

  Also

  Store.ping(), reporting the server version. migrate() already proved
  connectivity, but only as a side effect of the DDL — it says the database was
  reachable, not which database. A pool aimed at the wrong Postgres migrates
  exactly as cleanly as one aimed at the right one, and an empty schema is the
  expected state of both.

  ---

  Minor rather than patch for the added boot output and the new Store.ping().
  No artifact, schema, or published file changes.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.11.0 a335621b62

  Compare

  v0.11.0 — phase-tagged logs and a legible boot banner Stable

  joeyr released this 2026-08-02 17:23:32 -05:00 | 222 commits to main since this release

  Every log line is now attributed to a phase, in Portainer's caller field —
  the fourth of the five keys the dashboard colours, and the one 0.10.2 named but
  left unused.

  Phase tags

  The two halves of a window run concurrently by design, so their lines
  interleave. Until now the only way to tell which crawl a line came from was to
  recognise its wording, and the only way to tell a classic failure from a ranked
  one was a queue= pair at the far end of the line. Portainer prints caller
  before the message and colours it distinctly, so it reads as a column:

  2026-08-03 02:14:07 INF riot/crawl >       crawl complete | rows=7478
  2026-08-03 02:41:22 WRN counters/classic > coverage below floor | coverage=0.87
  

  The vocabulary is riot/{crawl,damage,benchmarks}, opgg/{http,universe},
  counters/{ranked,classic} and runes/{ranked,classic}.

  Two segments, always. Portainer renders caller as the last two
  slash-separated segments — the field is meant for a Go pkg/file.go — so a
  third is dropped silently, which would make opgg/counters/classic and
  riot/counters/classic arrive identical. The list is closed (PHASES in
  logger.ts) so that holds by construction and a typo is a type error rather
  than a tag that quietly never appears.

  counters and runes split by queue rather than by half because that is the
  axis that actually distinguishes their lines — both run on the op.gg half. The
  shared HTTP client keeps its own opgg/http tag rather than borrowing whichever
  crawl was using it: one instance serves the universe, counters and runes crawls
  of a pass, so attributing its retry warnings to any single phase would be a
  guess.

  The boot banner

  It now reports the config it was silently omitting: runOnStart, classic,
  the op.gg query band (region/tier/queue), the op.gg pacing interval, and the
  per-artifact time caps.

  classic is what motivated this. It is the only opt-in artifact flag, and there
  was no way to tell from the outside whether the pass had run — an unset
  CLASSIC_ENABLED looks exactly like a pass that ran and published nothing.
  runOnStart is the reason a container that "did nothing" may in fact have
  crawled the moment it booted, outside the window. The query band decides which
  population every op.gg artifact describes, and three artifacts describing three
  different populations is a failure with no error attached to it.

  It is also three lines now — config/build, config/riot, config/opgg —
  because the single line wrapped to four in the viewer where it is read, and at
  that width the wrap lands mid-field. The split follows the service's own shape:
  the process and its destinations, then one line per half.

  Local window times

  next and deadline were the last clocks still rendering as
  2026-08-01T07:00:00.000Z, sitting beside a local timestamp with nothing on the
  line to say the two were five hours apart by definition rather than by elapsed
  time. schedule.ts does all its arithmetic in local time off the container's
  TZ, so a UTC field was the one value that had to be converted before it could
  be checked against CRAWL_START_HOUR — and being five hours out is exactly what
  a TZ misconfiguration looks like. next now reads 2026-08-03 02:00:00
  against a window=02:00-05:00 on the line above it.

  The sv-SE formatting 0.10.2 inlined into pino's timestamp is now the exported
  localTime, so the timestamp and the fields beside it cannot drift apart.

  ---

  Minor rather than patch for the added boot output and the new caller field,
  though this remains confined to the dashboard: no artifact, schema, or published
  file changes.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.10.2 97f2a40624

  Compare

  v0.10.2 — Portainer-readable log field names Stable

  joeyr released this 2026-07-31 19:57:51 -05:00 | 236 commits to main since this release

  The nightly logs now render in colour in Portainer. They were emitting under
  field names the dashboard does not read, so every line arrived as an
  undifferentiated wall: no colour on the level, and no subject until the end.

  Portainer's JSON log viewer recognises exactly five keys — time, level,
  caller, message, stack_trace — colours those, and prints everything else
  as trailing key=value. Two of ours were named past it:

  • level was uppercased to INFO/WARN/ERROR. The viewer matches it
    case-sensitively against debug/info/warn/error, so every level fell
    through to an uncoloured default. This amends 0.5.0, which uppercased it to
    stop a bare 30 rendering where the severity should be. That problem was
    real and stays fixed — the level is still a name — but the fix overshot by
    one character class and cost the colour. pino's own labels are already the
    lowercase names Portainer wants.
  • the log text sat under pino's default msg, an unknown key, so it sorted
    in among the fields rather than leading the line. It is now message.

  time also moves from epoch milliseconds to a local 2026-07-31 09:09:03
  string. Portainer renders a numeric time through Y/MM/dd hh:mmaa — 12-hour
  and with no seconds — which drops exactly the resolution this service paces
  itself in (2.5s between Riot calls, 12.5s between op.gg ones). Local rather
  than UTC because every window here is local already.

  The logger tests now assert the invariant rather than the literal strings: the
  level must be one of the four values the dashboard colours.

  Cosmetic, and confined to the dashboard: no artifact, schema, or published file
  changes, and docker service logs is unaffected either way.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.10.1 6292c9813f

  Compare

  v0.10.1 Stable

  joeyr released this 2026-07-31 09:00:31 -05:00 | 237 commits to main since this release

  Changed

  • CLASSIC_TIER is now all_ranks rather than all. It reads as a band name
    alongside the gold_plus it stands in for, and a consumer can print it
    verbatim in a caption instead of special-casing a bare adjective. Neither
    classic artifact has been published — the pass is opt-in and has never run —
    so nothing migrates.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads