lol-companion-data

Nightly crawler that publishes three static data artifacts for lol-companion to fetch at runtime:

Artifact	What it answers	Source
champion-damage.json	armor or MR? — each champion's physical / magic / true split	Riot Match-V5
champion-counters.json	who beats what? — matchup win rates, plus the role tier lists	op.gg
champion-benchmarks.json	am I behind? — CS, gold, KDA, vision per minute by rank	Riot Match-V5

Why this exists

Damage splits. lol-companion can't answer the question every champ select asks, and no existing source can. op.gg champion pages carry no damage-type split at all. Data Dragon tags which abilities deal each type but not what share of a champion's damage they represent, and it misses passives entirely — Amumu's Cursed Touch true damage appears nowhere in the tooltip markup.

Riot's Match-V5 API does have it, per game, per participant. A validation crawl over 700 ranked matches measured Amumu at 10.7% physical / 76.8% magic / 12.4% true — against a careful hand-estimate of 5 / 85 / 10. Being off by 2x on a simple, stable champion is what ruled out curating the roster by hand.

Counters and benchmarks existed already, but in the wrong place. Every lol-companion install was scraping op.gg during champ select, one rate-limited page per hovered champion, and harvesting its own benchmarks from the user's personal Riot key — competing with the scouting report for the same 100-requests-per-2-minutes budget. Crawling both once, centrally, makes the app's data instant, more complete than any single install's cache accumulates, and gentler on op.gg than N clients scraping ad hoc.

How it works

  nightly 02:00-05:00
          │
          ├─► op.gg pages ──────────────────────────►  champion-counters.json
          │   (~270, one per champion+role, 1.5s apart)
          │
          └─► Riot Match-V5 ──►  Postgres  ──┬──────►  champion-damage.json
              (~24 req/min)   (two-patch      │
                               window)        └──────►  champion-benchmarks.json
                                                            │
                                        committed to data/ in this repo
                                                            │
                                                            v
                                                      lol-companion

Damage and benchmarks cost one crawl between them. A match response carries CS, gold, KDA, vision and duration for all ten participants whether or not anyone reads them, so everything is recorded once and the two artifacts are two readings of the same rows. The benchmarks artifact is, in request terms, free.

The op.gg half runs first because it is the bounded one — a fixed page count under its own time cap — while the Riot crawl deliberately spends whatever is left of the window. Each artifact publishes independently: a night where op.gg serves a challenge page still ships the damage aggregate the Riot budget was already spent on.

The crawler is outbound-only — no ports, no ingress, nothing to reach it on. It dials Riot, Data Dragon, op.gg and Forgejo, and that is the whole of its network surface. The raw match rows stay in its private Postgres; the only thing that ever leaves is the aggregates below.

Each night the service seeds from ranked ladder players, walks their recent solo-queue matches, and records every participant of every usable match — one fetch yields ten champions, so filtering would waste 90% of what it paid for. Rows accumulate; coverage compounds across nights rather than restarting.

The rank band

Gold IV through Emerald I, NA, ranked solo/duo only — the same gold_plus band lol-companion already uses for op.gg.

Seeding is the cross product of TIERS x DIVISIONS — twelve buckets, two league-exp pages each, drawn at random from the first 120 pages rather than taken off the top. There is no API that hands out ranked games, so a crawler must start from players; drawing pages at random is what keeps it from starting from the same players every night. Gold IV alone runs past 500 pages, so fixed pages would sample well under 1% of the bucket forever.

Two further consequences worth knowing:

• The band is covered evenly, not proportionally. Every bucket contributes the same number of pages, so Emerald isn't buried under a much larger Gold population. Weighting by real population would make this mostly a Gold dataset.
• Seeds set the band; they don't bound the sample. The seeded player is ranked within the band, but all ten participants of their matches are recorded. Matchmaking keeps the other nine close, but they are not guaranteed to be in it. Read generatedFrom.tiers as "games played by players of these ranks", not "games between players of these ranks" — filtering to seed-tier participants would discard 90% of every match already paid for.

The whole pool is shuffled before crawling, so a window that ends early has still sampled the full band rather than grinding through Gold IV alone.

Both lists are validated at boot. A misspelled tier or a nonexistent division (there is no division V — Riot removed it in 2019) fails immediately, because league-exp answers an unknown one with an empty list and would otherwise seed nothing at all without raising an error.

Pacing

One Riot request per 2.5 seconds (24/min). The binding personal-key limit is 100 per 2 minutes — that's 50/min, not the 20/s people usually quote, so anything at or above 1 call/sec overruns the cap and stalls on 429s.

We deliberately use less than half the budget because the desktop app shares this key. A 429 therefore means the app needs the key, and the crawler yields a full minute rather than retrying tight.

That still collects a few thousand matches (tens of thousands of participant rows) a night, which is more than enough. Throughput is not the constraint — don't tune the interval down.

op.gg is paced separately at one page per 1.5 seconds, on its own limiter. There is no published limit there; this is politeness, and it is already far gentler than what it replaces.

The benchmark rank buckets

Benchmarks are bucketed by rank, and a match carries no rank of its own. The crawler attributes every participant to the tier of the seed player whose match list surfaced the game.

That is an approximation — matchmaking keeps the other nine players close but not identical — and it is the standard one, for a concrete reason: looking up ten players' ranks per match would cost more requests than the crawl itself. It buys ten rows per request instead of one.

Each bucket carries its own n, and tiers are never pooled together. A consumer that wants one number per champion can weight the buckets itself; one that wants the reader's own rank needs them apart, and pooling here would be irreversible.

The 48-hour lookback

Match-id lookups reach back two days, and both a wider and a narrower window cost more.

Wider wastes fetches. Only current-patch matches are kept, and that check happens after the fetch has been paid for. At 14 days roughly half of every night's requests bought a match that was then thrown away.

Narrower wastes lookups. At 24 hours, half of all seeded players had played no ranked game since yesterday — and asking costs a request whether or not the answer is empty. Measured on a real run: 162 lookups produced 84 matches.

Two days is where those meet. Games per lookup roughly double, while a patch runs about two weeks so only its first morning returns many off-patch matches.

A side benefit: consecutive nights now overlap by a day, so a night the service missed is picked up by the next one instead of being lost.

The two-patch window

Retention keeps the current patch and its predecessor, and nothing else.

"Current patch only" would zero the dataset the morning a patch ships — exactly when players most want to know what changed. Instead, any champion or role with no rows yet on the new patch is served from the previous one and flagged stale. Entries degrade instead of vanishing, and self-heal as the night's rows land.

RETAIN_PREVIOUS_PATCH=false narrows it to the current patch alone. The flag moves retention and the fallback together — pruning the rows while still naming the predecessor in generatedFrom.previousPatch would advertise a fallback that cannot fire — so with it off, a champion-role with no rows yet is omitted from damage and benchmarks rather than served stale. Only those two artifacts have a fallback to lose; counters, runes, synergies and builds are current-patch-only either way, and data/archive is permanent under both settings.

That trade got cheaper as the crawl grew. It was written when a night collected ~1,700 matches, so rollover morning genuinely had gaps; a wider window means most champion-roles have rows by the first morning and the fallback rarely fires at all. The cost is roughly one thinner morning per patch, against holding half the rows.

Nothing is filtered out

Every measurement reaches the artifact, down to n: 1. This service gathers and publishes; deciding a sample is too thin to draw belongs to the consumer, which knows what it is rendering and has n on every entry to judge by. Filtering here would discard that choice irreversibly — what is never published cannot be recovered downstream.

ROLE_MIN_SAMPLES / CHAMPION_MIN_SAMPLES / BENCHMARK_MIN_SAMPLES default to 1 and exist to drive the fallback above, not to withhold data.

The counters artifact follows the same rule from the other direction: it carries every matchup op.gg lists, including thin ones, and no derived rating. A 30-game matchup may well be too noisy to ban on — but that is a call the consumer makes with games in hand, not one this service makes by deleting the row.

Privacy

No player identifiers are ever stored. match_id exists solely so a match is never counted twice. Only anonymous aggregate statistics leave this service — championId, role, tier, and per-bucket means with a sample count.

The published artifacts

Committed nightly to data/ and served from this public repo — no token, no account, three stable URLs:

https://forgejo.jrendar.org/joeyr/lol-companion-data/raw/branch/main/data/champion-damage.json
https://forgejo.jrendar.org/joeyr/lol-companion-data/raw/branch/main/data/champion-counters.json
https://forgejo.jrendar.org/joeyr/lol-companion-data/raw/branch/main/data/champion-benchmarks.json

The patch is inside each file, not in the URL. That is deliberate: a consumer would otherwise have to resolve the current patch before it could build a URL, and would 404 on patch morning — precisely when the fallbacks were supposed to save it.

Committing rather than uploading also means git log data/ is a free history of how the meta moved, and a run that changes nothing makes no commit.

champion-damage.json

{
  "generatedFrom": {
    "publishedAt": "2026-07-25T05:00:00.000Z",
    "patch": "16.14",
    "previousPatch": "16.13",
    "matches": 41230,
    "tiers": ["GOLD", "PLATINUM", "EMERALD"]
  },
  "roleWeights": { "TOP": 0.82, "JUNGLE": 0.78, "MIDDLE": 0.95, "BOTTOM": 1, "UTILITY": 0.41 },
  "champions": {
    "32": {
      "all":   { "phys": 10.7, "magic": 76.8, "true": 12.4, "n": 2401 },
      "roles": {
        "JUNGLE":  { "phys": 9.8,  "magic": 78.1, "true": 12.1, "n": 1802 },
        "UTILITY": { "phys": 13.3, "magic": 72.4, "true": 14.3, "n": 551, "stale": true }
      }
    }
  }
}

Guaranteed invariants — schema/champion-damage.schema.ts is the source of truth, and the artifact is validated against it immediately before it is committed:

• Every triple sums to 100 ± 0.2.
• n is present and ≥ 1 everywhere.
• Champion keys are numeric strings; role keys are TOP / JUNGLE / MIDDLE / BOTTOM / UTILITY.
• stale: true marks a previous-patch fallback.

Two fields exist purely for the consumer. Both all and roles are emitted because champ-select enemy roles are guessed, not known, so lol-companion often has no trustworthy role and must fall back to the champion-level split. roleWeights ships with the data so a team aggregate can weight by role instead of counting a support the same as a hypercarry.

champion-counters.json

271 champion-role entries covering all 173 champions, 13,440 matchup rows, 875 KB — one crawl of 276 pages in six minutes. Committed minified: at that size a pretty-printed diff is unreadable anyway, and indentation would more than double a file rewritten nightly.

{
  "generatedFrom": {
    "publishedAt": "2026-07-26T05:00:00.000Z",
    "patch": "16.14",
    "source": "op.gg",
    "region": "na",
    "tier": "gold_plus",
    "queue": "ranked",
    "championRoles": 271,
    "matchups": 13440
  },
  // One entry per role, ascending by rank. `tier` is op.gg's bucket, where
  // 0 is the "OP" tier and 1..5 are numbered.
  "tierList": {
    "JUNGLE": [{ "championId": 233, "tier": 1, "rank": 1 }]
  },
  "counters": {
    "32": {
      "JUNGLE": {
        "games": 20496, "wins": 10113, "winRate": 49.34,
        // Descending by games, so a consumer that truncates keeps the solid rows.
        "matchups": [
          { "opponentChampionId": 234, "games": 1168, "wins": 605, "winRate": 51.8 }
        ]
      }
    }
  }
}

These are measurements, not scores. lol-companion derives a counter rating from a win rate via a gain constant, an optional shrinkage term and a game-count floor. None of that is applied here. Those constants are tuned against its recommender's threat threshold and covered by its tests — baking them in would mean a tuning change required a re-crawl, and would silently invalidate every artifact published under the old gain. Deciding how to use a matchup is the consumer's job; measuring it is this service's.

The tier list is not a bonus feature. op.gg has no endpoint that enumerates which champions are played at which role, so the five tier-list pages are what tells the crawler which counters pages exist. Having paid for them, publishing them costs five arrays and saves the consumer five fetches a patch.

A counters run refuses to publish unless all five tier-list pages parsed and at least 90% of the discovered champion-role pairs produced an entry. A published artifact replaces the previous one, so a half-broken run is worse than no run — on patch-drop morning, before op.gg has built the new patch's pages, both guards fire and yesterday's file stays live, labelled with the patch it describes.

champion-benchmarks.json

{
  "generatedFrom": {
    "publishedAt": "2026-07-26T05:00:00.000Z",
    "patch": "16.14", "previousPatch": "16.13",
    "matches": 41230,
    "tiers": ["GOLD", "PLATINUM", "EMERALD"],
    "region": "na1", "queue": "RANKED_SOLO_5x5"
  },
  "benchmarks": {
    "32": {
      "JUNGLE": {
        "GOLD": {
          "csPerMin": 5.12, "goldPerMin": 401.99, "damagePerMin": 812.4,
          "kdaRatio": 3.2, "killParticipation": 62.3, "visionPerMin": 1.1,
          "n": 210,
          // The same measurements over only this bucket's winning games.
          "wins": { "csPerMin": 5.9, "goldPerMin": 442.1, "damagePerMin": 901.2,
                    "kdaRatio": 4.8, "killParticipation": 66.1,
                    "visionPerMin": 1.2, "n": 118 }
        }
      }
    }
  }
}

Every figure is a mean of per-game rates, not a total over a total. That is the opposite of what the damage artifact does, and deliberately so: a damage split must not let a 600-damage blowout outvote a 24k game, while a benchmark answers "what does a typical game look like" and must not let one 50-minute game speak for three short ones.

killParticipation is the one optional measurement — a team that took zero kills has no share to divide, so those games are excluded from its mean and the field is omitted if every game in the bucket was. wins is absent when the bucket holds no wins, which is common at small n and is not an error. stale: true marks the same previous-patch fallback the damage artifact uses.

Median CS/min by role, from a 567-match sample: UTILITY 1.6, JUNGLE 6.5, MIDDLE 6.5, TOP 6.7, BOTTOM 7.0.

The published tables

Since 0.23.0 each artifact is also written into Postgres, split per champion and role, so a future query API can serve one champion's entry instead of a whole file. The committed files are still what lol-companion fetches — nothing reads these tables yet, and the database write happens only after the commit succeeds.

Table	Holds
published_artifact	provenance, the shared blocks, and the whole file brotli'd
published_entry	one row per (champion, role)
published_current	the patch each artifact is currently serving

champion-damage.json is stored whole and not split: its record has no role dimension, and the damage bars need all ten champions on the board anyway.

The split is checked, not assumed. Every publish rebuilds the file from its own rows and refuses to write unless the bytes match what was committed — a slice that quietly dropped a champion would otherwise be indistinguishable from a complete one. scripts/verify-published.mjs checks the other direction, that each row's sha256 still matches the file at its raw URL.

Set PUBLISH_DB_ENABLED=false to skip it. It is independent of PUBLISH_ENABLED, so either output can be turned off without the other.

The plan this is the first step of is in docs/query-api-plan.md.

The query API

Since 0.24.0 the same image carries a second entrypoint, a small read-only HTTP service over those tables:

node dist/src/api/index.js        # or: npm run start:api

The crawler itself still listens on nothing. This is a separate process with a separate Postgres role, and it is the only thing here that accepts a connection.

Route	Auth	Returns
GET /v1/health	none	liveness, and the age of the oldest artifact
GET /v1/manifest	key	patch, publishedAt, sha256 and size per artifact (~1 KB)
GET /v1/artifacts/{name}	key	the whole file, byte-identical, brotli, ETag

{name} is the artifact's short name — counters, synergies, counters-classic — not the committed filename.

Nothing consumes it yet. lol-companion still fetches the raw URLs; the entry-level endpoints that make this worth switching to are Phases 3-5 of the plan.

Keys

Every route except /v1/health needs Authorization: Bearer lolc_…. Keys are per install, never one shared key baked into the app — an Electron asar unpacks in minutes, so a bundled key is a public key, and a shared one cannot be revoked without breaking every install. What per-key issuance buys is revocation, per client rate limits and usage accounting; what it cannot buy is secrecy, and nothing here needs any: these are anonymised aggregates of public match data, world-readable at the Forgejo URLs today.

npm run key:create -- --label joey-desktop     # prints the key once
npm run key:create -- --list
npm run key:create -- --revoke <key_id>

Run it against the crawler's DSN, not the API's — issuing INSERTs, and the API's role deliberately cannot. In the homelab that means the crawler container:

docker exec <crawler container> node dist/src/api/create-key.js --label joey-desktop

A revocation takes effect within the auth cache TTL (60 s by default).

Verifying it

scripts/verify-api.mjs fetches every artifact from the API in both encodings and compares the bytes against the file committed at its raw URL. It needs API_URL and API_KEY beside the Forgejo settings, and it retires when the Forgejo publish does.

Development

npm ci
cp .env.example .env      # fill in RIOT_API_KEY and FORGEJO_TOKEN
npm run pg:up             # local Postgres on :55432 (two DBs, see below)
npm test                  # unit + integration
npm run pg:down

Gate, mirroring CI:

npm run typecheck && npm run lint && npm test && npm run build

RUN_ON_START=true makes the service crawl immediately instead of waiting for the window — useful for a short manual run. Set COUNTERS_ENABLED=false with it unless you are testing the op.gg half: that crawl runs first, ignores MAX_REQUESTS (which budgets the Riot key), and takes several minutes.

ARTIFACT_OUT / COUNTERS_OUT / BENCHMARKS_OUT write an artifact to a local path, so a run can be inspected without a token that can write to the repo.

Two local databases

pg:up creates both:

• lolc_damage — crawl data. Rows are only ever added to it.
• lolc_test — the integration suite's scratch space.

The suite asserts exact row counts, so it truncates between tests. It reads TEST_DATABASE_URL and pointedly ignores DATABASE_URL, so running npm test can never destroy a manual crawl's data — which is what happened when the two shared one database.

Deployment

Runs on the homelab Swarm alongside aufhocker. The image is built by CI; the compose file lives in the homelab stacks repo, not here.

Required secrets: RIOT_API_KEY, FORGEJO_TOKEN, and the Postgres connection string. TZ must be set explicitly — all window arithmetic is local-time, and the default UTC would shift the 02:00–05:00 window by several hours.

FORGEJO_TOKEN needs write:repository so the service can commit. That is its only write scope; it does not need package, admin or org access.

The crawler listens on no port — do not publish one for it, and do not put it behind a reverse proxy. The query API is a second service off the same image (command: ["node", "dist/src/api/index.js"]), and it is the only process here that accepts a connection.

Give the API its own Postgres role rather than the crawler's:

CREATE ROLE lolc_api WITH LOGIN PASSWORD '…';
GRANT CONNECT ON DATABASE lolc_damage TO lolc_api;
GRANT USAGE ON SCHEMA public TO lolc_api;
GRANT SELECT ON published_artifact, published_entry, published_current, api_key TO lolc_api;
GRANT UPDATE (last_used_at) ON api_key TO lolc_api;

No DDL and no INSERT, which is why the crawler creates api_key and the issuance script runs against the crawler's DSN. That grant — not the code — is what keeps participant_stat unreadable from a process serving HTTP.

The API refuses to start as any other role. src/api/preflight.ts asks SELECT current_user at boot and compares it against API_DB_ROLE, which defaults to lolc_api so a stack file that omits the variable is still checked. A DATABASE_URL pointed at the crawler's DSN would otherwise work perfectly and log identically — and be an HTTP surface with SELECT on participant_stat. Running against a development database means setting API_DB_ROLE=lolc.

Both processes print a preflight block at boot; the API's is six checks:

INF preflight/db         | ok  server=PostgreSQL 18.4
INF preflight/role       | ok  role=lolc_api
INF preflight/schema     | ok  tables=4
INF preflight/grants     | ok  select=true update=last_used_at
ERR preflight/keys       | FAILED reason=no active keys (0 revoked) …
INF preflight/artifacts  | ok  count=10 expected=10 patch=16.15 oldest=19h

role and schema are fatal; the rest are loud and non-fatal, because each is fixable while the process runs — add a grant, mint a key, wait for tonight's crawl. This is the one way it differs from the crawler's block, which may never stop a boot: a read service holds no work in progress, so refusing to start costs nothing that starting broken would not cost more of.

It is public, and the edge is part of the design

Exposed on 2026-08-07: https://api.jrendar.org fronts it through nginx-proxy-manager with a Let's Encrypt certificate. The app runs on a laptop, so "the only consumer is this household" and "the only consumer is on the home network" turned out to be different claims — see the exposure section of docs/query-api-plan.md for how that was decided.

The proxy lives in the homelab stacks repo, not here, and is written down here anyway for the same reason the CREATE ROLE block above is: it is a fact about the design rather than a preference, and the properties the service relies on are not visible from its own source. Three properties are load-bearing:

• The proxy is the only way in. The container must not publish a port on the host — 127.0.0.1 at the narrowest, and preferably no ports: at all, with NPM reaching it over a shared network. A published port is a plaintext path to a Authorization: Bearer scheme, which is the one thing TLS was mandatory for. scripts/verify-edge.mjs checks for it if you give it API_LAN_URL.
• /v1/health is open by design and does not need to be open to the internet. It reports the version, every artifact name, its patch and its publish time — a staleness oracle and a build string for anyone who asks. Nothing polls it; the only consumer is a person on the LAN checking why something looks stale, so it is restricted to RFC 1918 at the proxy. The container healthcheck uses 127.0.0.1:8080 and never crosses it.
• Nothing in this repo bounds unauthenticated volume. auth.ts reaches the token bucket only after a key resolves, deliberately — see its header. So a flood of garbage bearer tokens is cheap per request and unbounded in count, and only the proxy can bound it.

And for a day it could not, because nginx never saw a client IP. NPM published through Docker Swarm's ingress routing mesh, which SNATs every connection — every client on earth logged as 10.0.0.2. allow/deny was handed a constant, and limit_req_zone $binary_remote_addr would have been one shared bucket for the entire internet, refusing real users over traffic they had nothing to do with. Fixed 2026-08-07 by publishing NPM's ports in long-form mode: host with a node.hostname placement constraint, since host mode binds only on the node actually running the container. A short-form ports: entry reintroduces the mesh silently, which is the argument for the check rather than the note.

/v1/health is restricted as of 2026-08-07, and the verification is the lesson. The rule looked correct from the LAN for a full day while allowing everyone, and three separate checks agreed it worked — including a fetch believed to come from outside the network that the log showed originating on the developer's own laptop. A tool that runs on your machine is not an external vantage point. edge/health is graded only under EDGE_OFF_LAN=1, and it was a phone on cellular that finally produced the 403.

Do not put a proxy cache in front of the artifact routes. They carry Cache-Control: public, and public is exactly what overrides nginx's default refusal to cache a request bearing an Authorization header — so a HIT would be served without the service ever being consulted, and the key would stop being what gates the data. verify-edge.mjs asserts two unkeyed requests in a row are both refused, which is the shape that failure would take.

node scripts/verify-edge.mjs                # what the edge refuses
EDGE_OFF_LAN=1 node scripts/verify-edge.mjs # adds the checks only outside can make
node scripts/verify-edge.mjs --burst        # also measures both rate ceilings

Fatal checks are the ones where a request reaches data or a bypass reaches the service; everything else is advisory, and a skipped check is reported as distinct from a passing one.

Consuming the artifacts

lol-companion fetches the raw URLs at runtime rather than vendoring the files, so each night's aggregate reaches players without cutting a release. It keeps a mirror of each schema in schema/, strict-parses every response against it, and caches the last good artifact under userData — that cache is what keeps the app working offline and on a cold start, which is the job vendoring would otherwise have done.

Two consequences for anyone changing a schema here:

• Drift is silent. A response that fails the consumer's parse is discarded in favour of its cached copy. Nothing errors and nothing looks broken; the feature just stops advancing. Land the mirror in lol-companion before publishing a new shape.
• The consumer's contract tests are not an alarm on this repo. They parse captured copies of published artifacts, so they catch drift only when those fixtures are refreshed from the URLs.

The consumer also treats a patch mismatch as expected rather than as an error: a crawler needs a night of games on a new patch before it can publish for it, so on patch-drop day it shows the previous patch's aggregate, labelled, instead of hiding it.

---

This project is not endorsed by Riot Games and does not reflect their views. League of Legends is a trademark of Riot Games, Inc. Data is derived from the Riot Games API.