What this is, in one paragraph
Read this before the diagrams. It is the part that matters.
The signal is a relative comparison of five ratios, computed over whatever evidence exists for a person, and published only when enough of it exists. It is not a performance rating, it does not accumulate, and it is recomputed from scratch for every window you select. It deliberately scores no count of anything — not commits, not hours, not lines — because every count can be inflated by working more carelessly. Where a component has no data it is withheld and the remaining weights rescaled; it is never filled in with a zero, because a zero reads as “did none of this” when the truth is “this system cannot see it”.
1 · Where a number comes from
Provider → normalised table → component → signal → band. Nothing enters the score that does not pass through this path.
2 · The five components and their weights
Drawn from the same table the scorer reads. If this bar is wrong, the score is wrong — they cannot disagree.
| Component | What it measures | How it is scored | Why this shape |
|---|---|---|---|
| Craft 25% | The share of commits whose message says what the commit did. | 100 × described ÷ commits, over the trailing quality window | In an organisation where work reaches the default branch by direct push, the commit message is the only place intent is recorded. It is a RATIO, so 500 careless commits score worse than 40 careful ones — volume cannot buy it. |
| Completed work 22% | Tasks somebody marked done in a connected tracker. | 50 × √(tasks done ÷ team median) | The only delivery evidence that does not come from a commit, so it is the one term that can see server work, bug triage and coordination. Withheld — never zero — for anyone never assigned a task. |
| Change discipline 18% | The size of a typical change. | piecewise on the median changed lines; full marks between 30 and 250 | A median of a few lines is noise; a median in the tens of thousands is a generated or vendored dump, not authored work. Both ends are penalised, so neither padding nor bulk imports pay. |
| Infrastructure 15% | Weighted AWS/CloudTrail changes — deploys, config, infra provisioning — for people who do this work. | 50 × √(infra points ÷ team median), among the cohort with infra evidence | Real delivery that leaves no trace in git, which is the whole reason the AWS ingest exists. Withheld — never zero — for anyone with no infrastructure evidence, whether or not they hold an AWS account: absence of ops work is not a deficiency in someone whose job is code. Weighted lower than the code components because CloudTrail records the API call, not the intent, and cannot distinguish a considered change from one corrected a minute later. |
| Consistency 12% | Whether work is sustained or arrives in one burst. | 50 × √(days with code ÷ team median) | Measured against the team’s own working calendar rather than against tracked-time overlap, so people without time tracking are not scored on fewer components than everybody else. |
| Review 8% | Reviews given to other people’s work. | 0.7 × review-volume score + 0.3 × share of own merges reviewed | The one term that credits reading and checking rather than typing. If nobody in the organisation reviews anything it is withheld for everyone, never scored as 0. |
Not a component: first-pass CI is measured and displayed on every person, but it is not scored. CI exists on only a minority of repositories, so scoring it would rank people by which repository they happen to work in. Tracked hours, commit counts, pull-request counts and lines changed are likewise shown and never scored.
3 · Why the curves are not straight lines
Two transforms do most of the work of stopping volume from buying a score. Both are plotted from the real formula.
Square root against the team median
Matching the team median scores 50. Doing four times the median work scores 100, not 400 — the curve flattens hard, so out-shipping everyone cannot dominate the ranking, and being somewhat below the median is not catastrophic.
Change size: both extremes are penalised
A median of 1–2 lines scores badly because it is mostly noise; a median in the thousands scores badly because it is a lock file, a vendored dependency or a generated client, not authored work. Only the readable band in the middle earns full marks, so neither padding commits nor dumping generated code pays.
4 · What happens when a component has no data
The single most important rule here, and the one most dashboards get wrong.
A missing component is a statement about our data, not about the person. Scoring it 0 would publish “this person delivered nothing” when the truth is “no tracker describes their work”. So the component is dropped and the remaining weights are renormalised.
The cost is that two people can be scored on different evidence. That is why every person page states what share of the intended weight their score rests on, and why a score is withheld entirely below 25% of available weight.
5 · How “behind” is decided
The real distribution for 2026-06-28 → 2026-09-26. Bands are cut from this shape, never from fixed thresholds.
Because the cuts are percentiles, somebody is always in the bottom band — even on a team where everyone is doing well. “Behind” means “below most colleagues on these five ratios, this period”. It does not mean underperforming, and it carries no history: change the window and the band can change with it.
Two guards suppress the bands entirely. Fewer than 4 people producing code, and there is nothing meaningful to rank against. A spread narrower than 8 points between the cuts, and everyone genuinely is doing much the same thing — forcing that into thirds would manufacture a difference that is not there.
6 · When the system refuses to publish a number
Each of these is a different sentence, and each is shown instead of a score — never as a low score.
| Condition | Threshold | What is shown instead |
|---|---|---|
| Not enough of the score has data | available weight below 25% | components with the share of weight that exists — no composite |
| Too small a sample for a ratio | fewer than 8 commits in the quality window | craft and discipline withheld; a ratio over fewer commits is noise |
| No code account linked | no `identities` row for github | “delivery cannot be measured for this person”, plus a flag |
| No code evidence at all | 0 commits, merges and reviews, and 0 code days | a sentence naming what was and was not found — never a 0 |
| Role is not code delivery | `ranked_on_code` is false on the employee | tracked time and activity only, with the role named |
| Cohort too small to compare | fewer than 4 people produced code | scores shown, bands withheld |
| Evidence confidence below the floor | confidence under 0.60 | no composite score and no alert can fire for this person |
7 · Every flag that can fire
Flags never touch the score. They exist to stop a number being read as more than it is.
| Flag | Fires when | What it does not mean |
|---|---|---|
| ⚑ Tracked time with nothing landed time-without-output | ≥ 20 tracked hours and zero commits, merged pull requests and reviews | This does NOT mean no work happened. Work that is not code is invisible here, and so is code pushed under an unlinked account or to a repository outside GITHUB_ORG. Check what the work was before treating it as an absence. |
| ⚑ Commits that do not say what they changed undescribed-commits | below 40% described commits, over a sample of at least 8 commits | A commit counts as described if it uses a conventional prefix (feat/fix/docs…) or has a first line of at least 25 characters. Short messages on genuinely trivial commits trip this. It measures the record of the work, not the work. |
| ⚑ Median commit is very large oversized-commits | median commit above 3,000 changed lines | A median this size is usually a lock file, a vendored dependency, generated client code or a migration — committed, but not authored line by line. It is a reason to discount the volume, not evidence of anything about the author. |
| ⚑ Very high line churn high-churn | more than 500,000 lines added + deleted in the window | Churn is never scored, precisely because it is this easy to inflate. The flag exists to stop a large number being read as a large contribution. Check for a vendored directory or a bulk import. |
| ⚑ No code account linked identity-unmapped | no row in `identities` for provider `github` | Nothing here says this person wrote no code — it says nothing can be attributed to them. Until an identity is mapped, every code-derived figure on this page is an absence of evidence, not evidence of absence. |
| ⚑ Too few people to rank against cohort-too-small | fewer than 4 people produced code in the period | The component scores are still measured. What is withheld is the comparison: with a cohort this small, "leading" and "behind" are noise wearing a number. |
8 · The ranking conclusion on /ranking — a different thing from the signal above
Not a sixth component. A written synthesis of everything above, produced by Claude, with the same refusal rules applied first.
Sections 1–7 describe the signal: a deterministic, code-computed score. The Ranking page is a separate pipeline that asks Claude (claude-opus-5) to read real PR titles, commit subjects, a complexity score, tracked hours and the twelve KPIs, and write a ranking directly from that — per policy set 2026-09-22, it does not require a connected task tracker first (the signal above still does; see section 6). Evidence confidence is still shown as context on every person, just not used to withhold a rank.
What the model is told not to do
- ·Treat hours tracked or commit COUNT as quality — they are activity, and the prompt says so explicitly. It is told to read the actual PR titles and commit subjects instead.
- ·Give AI token usage any positive weight — cost diagnostic only, same rule as the signal above.
- ·Rank anyone with zero commits, PRs and tracked hours this period — there is nothing to read for them, so nothing to rank. That is the only bar; a low evidence-confidence figure goes into their caveats as context, not as a reason to omit them.
- ·Skip naming what this dataset cannot see for each person — DevOps, incident response, mentoring, collaboration — the same list as section 9, repeated per person where it plausibly applies to them.
Right now, live
- Last run: 2026-09-26 08:50 UTC, triggered by cron:ec2
- 22 of 24 active people had evidence to rank on
- Cost of that run: ~$0.834
Runs once a day (06:00 UTC, GitHub Actions — never on this deployment, which holds no AI credentials) and on demand via Sync all on Sources, rate-limited so a repeated click cannot run up the bill. Every run — including a failed one — is kept, never overwritten, so a bad or biased conclusion is a visible history rather than a silently corrected one.
9 · What this cannot see
The list that should be read aloud before anyone uses these numbers in a conversation about a person.
Invisible to the score
- ·Architecture and design decisions
- ·Planning, scoping and estimation
- ·Mentoring and unblocking other people
- ·Incident response and on-call
- ·Infrastructure and server work
- ·Code review that happens verbally or in a call
- ·Any work done outside the connected repositories
Structural limits, this period
- ⚑Completed work is measurable for only 0 of 14 scored people (22% of the intended weight).
- ⚑Infrastructure is measurable for only 10 of 14 scored people (15% of the intended weight).
- ·Ratios are measured over a trailing 45-day window, not the selected range — how carefully someone works does not change between Monday and Tuesday.
If a number here looks wrong, it may well be. Every figure on a person's page links to the rows behind it, and anything can be disputed on the record from that page — a correction is stored beside the figure it challenges. Provider coverage is on Sources, and the reasons confidence is capped are on Data quality.