Metrics Methodology

Methodology v1.14

Every score GuideMode puts on a dashboard is computed by a formula published on this page. Nothing here is a description of intent — each section mirrors a specific module in the codebase, and the unit tests pin the code to these definitions. If a number on a chart looks wrong, you should be able to reproduce it by hand from this page.

Principles

1. Exact formulas are published. Weights, thresholds, and band boundaries appear here in full, not as prose approximations.
2. One normalization convention. Every perceptual score is normalized to 0–100, min-subtracted, with higher always better after normalization.
3. Missing data is never imputed. An unanswered question produces null and is excluded from both numerator and denominator. It is never counted as zero.
4. Aggregates carry their sample size. Where a score depends on how many people answered, the response count and a confidence measure are exposed alongside it.
5. Measure names are stable. Cube measure names are additive-only — new scores are added under new names, existing names never change meaning without a version bump on this page.
6. This page is versioned. Every methodology change is recorded in the changelog with the version it shipped in.

---

Normalization conventions

Likert to 0–100

Every bounded ordinal answer is normalized with the same min-subtracted formula:

likertToPercent(x, min, max, reverse)
  = ((reverse ? max + min − x : x) − min) / (max − min) × 100

The raw answer is clamped into [min, max] first, so a malformed row cannot drag a group average outside [0, 100].

Scale	Question type	Floor →	Midpoint →	Ceiling →
1–5	likert-5	1 → 0	3 → 50	5 → 100
1–7	likert-7	1 → 0	4 → 50	7 → 100
0–10	nps	0 → 0	5 → 50	10 → 100

The scale floor maps to 0, not to some fraction of the ceiling. This is the correctness fix that defines v1.1 — see the changelog.

Reverse-scored items

Some questions are written so that a high answer is the bad outcome. Those answers are flipped to the opposite end of their scale before normalizing:

reverseOnScale(x, min, max) = max + min − x

For a 1–7 scale this is the familiar 8 − x. It is an involution: applying it twice returns the original answer, endpoints swap, and the midpoint is fixed.

This is the complete list of reverse-scored items:

Question id	Instrument	Scale	Why a high raw answer is bad
verification_frequency	Session assessment	1–7	Constantly re-checking the AI’s output means low trust
cognitive_load	Session assessment	1–7	The AI increased, rather than reduced, mental effort
user_control	Session assessment	1–7	The AI dominated the session; the developer lost control
cognitive_load	Team experience survey	1–7	High mental effort required to use AI tools
tech_debt_impact	Team experience survey	1–5	Technical debt is slowing delivery down

Polarity is declared once, in question-scales.ts, and a unit test asserts it matches the reverseScored flag in the question JSON. A polarity flipped in one place cannot silently invert a dashboard score.

Raw averages are not inverted. Measures named avg* (for example avgCognitiveLoad) report the untouched answer on its native scale, so higher is worse for reverse-scored questions. The *Score companions (cognitiveLoadScore, techDebtScore, userControlScore) are the inverted, normalized, higher-is-better versions. Only the *Score family is safe to place on a shared axis.

Averaging and missing data

Rule	Behaviour
Unanswered question	null — excluded from both numerator and denominator, never scored 0
Explicit non-answer (e.g. “Not sure”)	Treated as unanswered, not as a midpoint
Unrecognized choice value	Treated as unanswered
All components null	The score is null, not 0

Weighted composites renormalize across the components that actually have data:

weightedMean = Σ(wᵢ × vᵢ) / Σ(wᵢ)   over components where vᵢ is not null

So a half-and-half composite where one half was never asked degrades gracefully to the other half’s score, rather than collapsing to null or being silently halved.

Group-level composites average each component over rows first, then combine. This is deliberately not the same as averaging a per-respondent composite: a developer who answered cognitive_load but skipped the long-form-only mental_alignment still contributes to the first half of the Clarity Score.

Net Promoter Score

NPS uses the standard bands on the 0–10 recommendation question:

Category	Answer
Promoter	9–10
Passive	7–8
Detractor	0–6

NPS = promoter% − detractor%        (range −100 … +100)

The denominator for both percentages is the number of respondents who answered the NPS question, not the number of assessments or responses. Counting unasked rows would dilute every NPS toward zero.

Two distinct measures exist and are not interchangeable:

Measure	Range	Meaning
npsScore	−100 … 100	The Net Promoter Score proper (promoter% − detractor%, rounded to a whole number)
npsNormalizedScore	0 … 100	The mean raw recommendation answer, rescaled — an average, not an NPS

---

Session assessment composites

Session assessments are the short in-product survey attached to an AI coding session. Four composite measures are derived from them, all 0–100, higher better.

Measure	Formula
helpfulnessScore	mean(likertToPercent(task_helpfulness, 1..7))
accuracyScore	0.5 × likertToPercent(deployment_confidence, 1..5) + 0.5 × likertToPercent(verification_frequency, 1..7, reverse)
speedScore	mean(SPEED_COMPARISON_WEIGHTS[speed_comparison]) over answered rows
clarityScore	0.5 × likertToPercent(cognitive_load, 1..7, reverse) + 0.5 × likertToPercent(mental_alignment, 1..5)

Measure	Reading it
helpfulnessScore	How helpful the AI was on the task
accuracyScore	Trust in AI output: half deployment confidence, half not having to verify constantly
speedScore	Perceived speed versus working without AI
clarityScore	Cognitive clarity: half reduced mental load, half mental-model alignment
userControlScore	How much control the developer retained (user_control, inverted, normalized)

Speed comparison weights

speed_comparison is a choice question, not a Likert item, so it is scored from an explicit weight table:

Choice id	Score
much_faster	100
somewhat_faster	75
about_same	50
somewhat_slower	25
much_slower	0
not_sure	excluded — non-answer, scores null

Answers stored before the choice-id cutover (display text such as "Much faster") are mapped to the canonical ids before scoring.

---

Team experience survey scores

Team experience surveys report both raw averages on their native scales (avg*) and normalized 0–100 companions (*Score). Each *Score is mean(likertToPercent(question, scale, reverse?)) — the same primitive as everywhere else on this page.

Measure	Question	Scale	Reversed
productivityScore	overall_productivity	1–7
satisfactionScore	job_satisfaction	1–7
aiImpactScore	ai_tool_impact	1–7
taskHelpfulnessScore	task_helpfulness	1–7
cognitiveLoadScore	cognitive_load	1–7	✓
collaborationScore	team_collaboration	1–5
techDebtScore	tech_debt_impact	1–5	✓
deploymentConfidenceScore	deployment_confidence	1–5
npsNormalizedScore	nps_score	0–10

The raw avg* measures were kept, under their original names, because saved dashboards reference measures by name. They remain on their native scales and are not comparable with each other or with the *Score family.

---

Session telemetry scores

These are computed from session transcripts by the session processor, independently of anything the developer reported.

Process quality score

process_quality_score rewards observable good practice in how the session was run. It is a percentage of what the session could have demonstrated, not a flat sum.

Detection is by tool capability, never by a literal tool name. Names are normalised (lowercased, separators stripped), so TodoWrite, todowrite, todo_write and manage_todo_list all resolve to the same capability and every provider is scored on the same behaviour.

Component	Detection	Points	Counted when
Planning	A planning tool (ExitPlanMode and equivalents) or a written plan file	30	always
Progress tracking	A task-list tool (TodoWrite, TaskCreate/TaskUpdate, manage_todo_list) or a written plan file	20	always
Read before write	At least one read and at least one write	25	the session made ≥ 1 write
Test/check after write	At least one execute and at least one write	15	the session made ≥ 1 write
Incremental approach	Between 2 and 5 writes inclusive	10	the session made ≥ 1 write
Excessive searching	reads / max(writes, 1) > 2	−10	the session made ≥ 1 read

process_quality_score = clamp(0, 100, round(earned / applicable × 100) − penalty)

Why it renormalises. Components a session had no opportunity to satisfy leave the denominator instead of counting as failures. A read-only investigation never writes, so scoring it against read-before-write would mark it down for a question it was never asked.

This also protects the score from tool churn. Claude Code’s task-list tool has been TodoWrite, then TaskCreate/TaskUpdate, and current versions ship none at all; when it disappeared, a flat +20 became unreachable and every session silently capped at 80. Accepting a union of signals for planning and progress tracking means no client version is structurally penalised.

The penalty is the only negative term and applies after renormalising: a session that reads far more than it writes suggests the AI was hunting for context rather than being pointed at it.

Input clarity score

input_clarity_score measures the density of concrete, actionable detail in the developer’s own messages — technical terms, code, file paths and pointers, per word written.

messageScore = technicalTerms
             + codeSnippets      × 1.5
             + fileReferences
             + specificityMarkers × 2
             + atReferences       × 2
             + imageAttachments   × 3

messageDensity      = min(100, round(messageScore / words × 100))
input_clarity_score = round(mean(messageDensity over messages))

Term	What is counted
technicalTerms	Keywords from a fixed programming/framework vocabulary
codeSnippets	Fenced code blocks plus inline backtick spans
fileReferences	File extensions and path-like tokens
specificityMarkers	Line references (file.ts:123, line 45), function references (name(), obj.method(), camelCase/PascalCase identifiers
atReferences	@-prefixed file or resource references
imageAttachments	Image content blocks and image attachments on the message

It is a density, not a volume: a long vague message scores lower than a short precise one.

Scored per message, then averaged. Pooling the session (Σ messageScore / Σ words) inverted the metric, because the marker counts saturate while the word count does not: one long message dragged down every short precise one, so writing a thorough prompt lowered the score.

Two exclusions keep it measuring the developer:

• Harness-injected blocks are stripped — system reminders, command wrappers and background-task notifications. None of it was typed by the developer, and it distorts the score in both directions: notifications are dense with paths and identifiers, caveat banners are plain prose.
• Sub-agent prompts are excluded. They are machine-written and unusually specific, so counting them would flatter the sessions that delegate most.

Messages with no words after stripping are skipped. Sessions with no user messages score 0.

Read/write ratio

read_write_ratio (read operations ÷ write operations) is reported raw, with an interpretation band. Lower is better — it indicates the AI knew where to go.

Ratio	Quality
≤ 2	Excellent
≤ 5	Acceptable
> 5	Poor

Interruption rate

interruption_rate is the share of the developer’s own inputs that cut the AI off mid-work.

interruption_rate = round(interruptions / (humanPrompts + interruptions) × 100)

An interruption is detected only from the explicit marker the client writes when the developer presses ESC, in either of its two forms — [Request interrupted by user] in a message, or [Request interrupted by user for tool use] in a tool result when the interruption landed mid-tool. A single ESC that emits both is counted once.

The marker must be the entire message, not merely appear in it. A session that greps for the marker, or reads a file defining it, fills its own tool results with the phrase; one real session contained it 21 times and had been interrupted twice.

Earlier versions also inferred interruptions from consecutive user messages and from the substrings wait, stop, actually and cancel appearing anywhere. Measured across 36 real sessions those rules produced 179 detections against 46 genuine ones, firing on ordinary prose like “I’ll wait for the agents”.

Interruptions are added back into the denominator because they are themselves developer inputs; without them the rate can exceed 100%.

Iteration count

iteration_count is the number of conversational rounds after the first — prompts the developer sent once the AI had already produced work. Consecutive prompts sent before the AI replies count as one round. Interruptions are excluded; they are reported separately.

It is deliberately structural rather than lexical. It previously looked for refinement phrases anywhere in a message, which was almost pure noise: rather than and instead of alone produced 28 of 42 pattern hits across 36 real sessions, and every one was a specification (“use Drizzle rather than Prisma”), not a correction. Anchoring those phrases to the start of the message was measured at exactly zero hits on the same corpus — real corrections open with the substance (“the policy should be explicitly selected at setup”), not a connective.

Recognising a genuine course correction is a semantic judgement, and the LLM phase analysis (ai_model_phase_analysis) already labels correction phases. This metric counts what can be counted exactly: how many times the developer had to come back.

Average tokens per message

avg_tokens_per_message is the mean number of new tokens each API request adds to the context.

newTokens             = input_tokens + cache_creation_input_tokens + output_tokens
avg_tokens_per_message = round(mean(newTokens over distinct requests))

Two quantities it is deliberately not:

• input_tokens alone. Under prompt caching that is only the uncached delta — a handful of tokens once the conversation is warm. Measured across twelve real sessions it was the constant 2 in every one.
• The full context (+ cache_read_input_tokens). Every request re-sends the whole conversation, so that measures average window occupancy — it tracks session length and duplicates context_length and context_utilization_percent.

Requests are deduplicated by requestId, since one response is written across several transcript lines that each repeat the same usage block.

API-equivalent cost

api_equivalent_cost_usd is the list-price value of the tokens a session consumed: for every model the session used, that model’s token counts multiplied by its list price at the moment the session ended, summed across models.

api_equivalent_cost = Σ over models m
                        Σ over dimensions d
                          tokens(m, d) × price(m, d, sessionEndedAt) ÷ 1,000,000

Dimension	What is priced
Input	Uncached input tokens, at the model’s input rate
Output	Generated tokens, at the model’s output rate
Cache write (5-minute)	Cache-creation tokens at the default write rate
Cache write (1-hour)	Cache-creation tokens written with a 1-hour TTL, which Anthropic bills at a higher rate
Cache read	Tokens served from cache, at the cache-read rate
Reasoning	Only where a model prices reasoning tokens separately from output

Prices come from an effective-dated table sourced from LiteLLM’s model_prices_and_context_window.json, refreshed nightly, so a price change opens a new row rather than overwriting the old one. Pricing is anchored on the session’s end time, falling back to its start — never on processing time, so a session backfilled months later prices identically to one priced as it arrived. There is no variant fallback: claude-opus-5[1m] must have its own price row and is never priced as claude-opus-5, because a 1m-context request costs materially more and the fallback would under-price silently and for ever. A model that cannot be priced is recorded as unpriced rather than skipped, so it surfaces as a coverage gap instead of vanishing.

It is a floor, not a bill. The 1-hour cache-write premium is priced only on the tokens the transcript can evidence. Per-message TTL data covers between 7% and 485% of what the provider’s own summary record bills for the same model — resumed and compacted sessions bill requests whose messages live in another file — so the observed 1-hour count is capped at the authoritative total and never extrapolated past it. Measured against Anthropic’s own reported figure across 15 real sessions, the derived cost came in 4.0% below, and never above. Extrapolating instead scored better on average but over-stated 8 of those 15 by up to 8%, and the cost of the choice made here is stated rather than left to be discovered: on any single session the figure is a lower bound, not the bill.

Sessions predating the price seed are priced at seed-date prices. Price history accrues forward from the first sync rather than being reconstructed backwards, so every model’s first row is backdated to a sentinel epoch and sessions older than the table still price — at whatever that model cost on the day the table was first seeded. From the seed date forward the point-in-time lookup is exact. The cost is that a pre-seed session carries a rate that was not necessarily in force when it ran, and a cost from a table that has only ever been seeded is stamped as such rather than dated.

Models that tier above 200k tokens cannot be priced correctly. The Sonnet 4 family bills a premium on input above 200k tokens, and the stored per-model breakdown does not split tokens by their position in the context window, so there is nothing to apply the premium to. Those tokens price at the base rate. The cost is real and one-directional: a long-context Sonnet session understates, and no amount of coverage reporting will show it, because the session is priced — just not correctly.

---

AIVA aggregation

AIVA (AI Value Accelerator) assessments score organizational maturity across value-stream and capability dimensions on a 1–4 scale, aggregated across respondents in six roles: leadership, product, engineering, architecture, operations, domain expert.

The aggregation math is published in full below. The instrument itself — question text, question-to-dimension mappings, and question-level weights — is not published; the framework is public, the questionnaire is not.

Role-normalized mean

Different roles have genuinely different visibility into different dimensions: operations staff see release engineering that leadership does not, leadership sees governance that engineering does not. AIVA weights by role visibility, not headcount, so a dimension is not decided by whichever role happened to send the most respondents.

1. roleMean(r)        = mean of all responses from role r
2. roleNormalizedMean = Σ(roleMean(r) × w(dimension, r)) / Σ w(dimension, r)
                        over roles that responded

Roles that did not respond contribute nothing to either sum. A role with no weight listed for a dimension defaults to 0.2. Final scores are rounded to one decimal and clamped to [1, 4]; all intermediate arithmetic uses full precision. A dimension with no responses at all scores null — never a default or assumed maturity level.

The four value-stream parent phases (vsDiscovery, vsDelivery, vsValidation, vsFoundations) are computed as the unweighted mean of their sub-dimension final scores, unless explicitly overridden during calibration.

Role weights

Dimension	Leadership	Product	Engineering	Architecture	Operations	Domain Expert
Value stream — parent phases
vsDiscovery	0.8	1.0	0.6	0.4	0.2	1.0
vsDelivery	0.4	0.3	1.0	0.8	0.9	0.1
vsValidation	0.9	1.0	0.8	0.5	0.2	1.0
vsFoundations	0.7	0.2	1.0	0.9	0.6	0.1
Discovery
vsDiscoveryResearch	0.6	1.0	0.4	0.3	0.2	1.0
vsDiscoveryClarity	0.7	1.0	0.8	0.5	0.2	1.0
vsDiscoveryPrioritization	0.8	1.0	0.6	0.4	0.2	1.0
vsDiscoveryCollaboration	0.6	1.0	0.8	0.5	0.2	1.0
Delivery
vsDeliveryVelocity	0.3	0.3	1.0	0.6	0.7	0.1
vsDeliveryQuality	0.2	0.3	1.0	0.8	0.6	0.1
vsDeliveryRelease	0.3	0.2	1.0	0.8	0.9	0.1
vsDeliveryFlow	0.4	0.5	1.0	0.7	0.7	0.1
Validation
vsValidationFeedback	0.7	1.0	0.6	0.3	0.2	1.0
vsValidationExperimentation	0.6	1.0	0.8	0.5	0.2	1.0
vsValidationDecisions	0.9	1.0	0.7	0.5	0.2	1.0
vsValidationMetrics	0.8	1.0	0.7	0.5	0.2	1.0
Foundations
vsFoundationsDx	0.3	0.5	1.0	0.9	0.5	0.1
vsFoundationsKnowledge	0.5	0.3	1.0	0.7	0.4	0.1
vsFoundationsObservability	0.4	0.2	1.0	0.9	0.9	0.1
vsFoundationsGovernance	0.8	0.2	0.5	0.9	1.0	0.1
Strategy & culture
capScLeadership	1.0	0.6	0.8	0.7	0.2	0.5
capScAiVision	1.0	0.6	0.9	0.8	0.4	0.5
capScExperimentation	0.9	0.6	1.0	0.6	0.2	0.5
capScLearning	0.7	1.0	0.9	0.5	0.3	0.5
capScChangeReadiness	1.0	0.9	0.8	0.7	0.5	0.7
People & skills
capPsAiFluency	0.5	0.8	1.0	0.6	0.7	0.4
capPsPromptEngineering	0.3	0.8	1.0	0.5	0.6	0.1
capPsGrowthFrameworks	1.0	0.9	0.7	0.5	0.6	0.4
capPsRoleEvolution	1.0	0.8	0.7	0.5	0.6	0.4
capPsTalentStrategy	1.0	0.7	0.6	0.4	0.5	0.4
Ways of working
capWowTeamTopology	0.8	0.5	1.0	0.9	0.2	0.3
capWowRituals	0.7	0.9	1.0	0.4	0.3	0.3
capWowDecisionMaking	1.0	0.8	0.9	0.8	0.2	0.7
capWowCrossFunctional	0.6	1.0	0.9	0.7	0.2	0.8
capWowPrioritization	0.9	1.0	0.8	0.6	0.2	0.9
Technical platform
capTpDesignSystem	0.3	1.0	0.8	0.7	0.2	0.3
capTpGoldenPaths	0.3	0.5	1.0	0.9	0.4	0.1
capTpCiCd	0.2	0.2	1.0	0.8	0.9	0.1
capTpAiTooling	0.8	0.6	1.0	0.8	0.6	0.4
capTpObservability	0.5	0.2	1.0	0.9	0.9	0.1
Governance & enablers
capGeCompliance	0.9	0.2	0.5	0.8	1.0	0.1
capGeSecurity	0.5	0.2	0.9	0.9	1.0	0.1
capGeCostManagement	1.0	0.2	0.7	0.7	0.9	0.1
capGeQualityGates	0.3	0.2	1.0	0.9	0.6	0.3
capGeDataGovernance	0.9	0.2	0.6	0.9	1.0	0.1
External interfaces
capEiStakeholderLiteracy	1.0	0.8	0.2	0.5	0.3	0.9
capEiCustomerValidation	0.9	1.0	0.4	0.2	0.2	1.0
capEiVendorAlignment	0.9	0.2	0.6	0.7	1.0	0.1
capEiRegulatory	0.9	0.2	0.5	0.7	1.0	0.5
capEiDependencyManagement	0.2	0.3	1.0	0.8	0.9	0.1

Agreement: average deviation from the median

ADm answers “on average, how many points from the median is a respondent?”

ADm = Σ |xᵢ − median(x)| / n

ADm (1–4 scale)	Label
≤ 0.5	Strong consensus
0.5 – 1.0	Moderate consensus
> 1.0	Disagreement

A single respondent yields ADm = 0 by definition.

Consensus: Tastle–Wierman

Alongside ADm, an entropy-based consensus measure in [0, 1] where 1 is perfect agreement:

Cns(X) = 1 + Σ pᵢ × log₂(1 − |Xᵢ − μ| / d)

where pᵢ is the proportion of respondents in bin i, μ is the mean, and d = 3 (the range of the 1–4 scale). Scores are binned to the nearest 0.5 before computation, and the result is clamped to [0, 1]. A single respondent yields 1.0.

Confidence: precision of the estimate

Confidence measures how precisely we know the mean, not how much people agree. The two are deliberately decoupled: a dimension with genuine disagreement but many respondents still has a well-estimated mean, and should not be dismissed as unreliable.

SEM             = SD / √n
MOE             = 1.96 × SEM                     (95% margin of error)
confidenceScore = max(0, 1 − MOE / (R/2))        where R = 3, the scale range

Normalizing by half the scale range makes the score scale-consistent: the same proportional uncertainty gives the same confidence whether the scale is 1–4, 1–5 or 1–7.

confidenceScore	Level	Meaning on a 1–4 scale
> 0.7	High	Margin of error < 0.45 points
> 0.5	Medium	Margin of error < 0.75 points
≤ 0.5	Low	Margin of error ≥ 0.75 points

A dimension with one or zero responses scores 0 (low).

Calibration

After aggregation, an assessment can be calibrated: a facilitator may override a dimension’s score, and the override must carry a written rationale, which is stored with the assessment and shown alongside the score. Calibrated dimensions display the calibrated value; the underlying aggregated distribution, confidence, and agreement measures are retained and remain visible. Calibration changes the score, never the evidence behind it.

What is not published

Per the AIVA content strategy — framework public, instrument gated — the following are deliberately excluded from this page: question text, question-to-dimension mappings, and question-level weights. The dimension-level math above is complete and sufficient to reproduce any published AIVA score from the dimension inputs.

---

Targets

Teams set targets against any cube measure. Status and progress are computed from the target definition.

Target status

The default tolerance is 10% (0.1) unless a target sets its own.

Target type	On track	At risk	Off track
min (higher is better)	value ≥ target	value ≥ target × (1 − tol)	otherwise
max (lower is better)	value ≤ target	value ≤ target × (1 + tol)	otherwise
exact	within target × (1 ± tol)	within target × (1 ± 2·tol)	otherwise
range	within [min, max]	within [min − buffer, max + buffer], buffer = (max − min) × tol	otherwise

A null current value is off track, not “unknown” — an unmeasurable target is not a met target.

Suggested targets

When suggesting targets, GuideMode uses the team’s own history rather than an external ideal. Percentiles use linear interpolation between the two nearest ranks.

Suggestion	Higher-is-better metric	Lower-is-better metric
Conservative	median	median
Moderate	p75	p25
Aggressive	p90	minimum observed

Direction is inferred from the measure name: a measure whose name contains time, duration, error, failure, latency, wait, bug, defect, or incident is treated as lower-is-better.

Trend detection

Over a series (minimum 3 points), the slope of a simple linear regression is normalized by the series mean to give change per period. A change of more than ±2% per period is reported as improving or declining; anything smaller is stable. Point-to-point trend arrows in the UI use the same ±2% threshold on percentage change between the current and previous value.

---

Benchmarks

A dashboard number without context is not an answer. “4.2 days lead time” is only meaningful next to something. Two reference populations are shipped, and a third is planned:

1. Static industry bands — published DORA performance bands, held as code constants and rendered from them. Shipped. A unit test fails if the DORA Metrics tables and the constants disagree, so the prose cannot drift from the code.
2. Cross-tenant percentiles — where a metric is comparable across customers. Shipped, subject to the privacy floor below.
3. Open-source corpus — public open-source organizations analyzed through the same pipeline, providing a third reference population and a calibration set for change-value scoring. Shipped, as labelled context rather than a target — see The open-source corpus below.

The metric-key contract

A benchmark is addressed by a metric key. Keys are a public contract: additive only, and a key never changes meaning without a version bump on this page — the same rule as cube measure names.

Key	Unit	Direction	Source
dora.lead_time_hours	hours	lower is better	Merge to production deployment
dora.deploy_freq_per_week	deploys/week	higher is better	Successful production deployments
dora.change_failure_rate	%	lower is better	Failed / completed production deployments
dora.mttr_hours	hours	lower is better	First failure to first subsequent success
flow.pr_cycle_time_hours	hours	lower is better	Pull request opened to merged
change.effort_per_author	CEU	higher is better	Change Effort Units per active PR author
change.survival_rate	%	higher is better	Window-complete changes that survived
assessment.helpfulnessScore	0–100	higher is better	Session assessment composite
assessment.clarityScore	0–100	higher is better	Session assessment composite
survey.productivityScore	0–100	higher is better	Self-reported overall productivity

GET /api/benchmarks returns this table live. GET /api/benchmarks/{key} returns your organization’s value alongside the reference distribution.

Tenant first, then percentile

The single most important rule in the computation:

1. reduce each organization to ONE value for the metric for the period
2. take percentile_cont across those per-organization values

Percentiles are never taken over raw rows. If they were, an organization merging a thousand pull requests a month would define the distribution that an organization merging twenty is then compared against — the benchmark would describe the largest customers rather than the typical one.

Periods are whole UTC calendar months, half-open [start, end), and a period is only ever computed once it has closed — never partially. Whole months rather than a trailing window, so that every organization and every re-run describes exactly the same interval.

Cadence is separate from period. Cross-tenant bands are recomputed once a month, after the month they describe has closed. Open-source corpus bands are recomputed nightly, still for the last closed month: the cohort is re-read every night, so recomputing is what corrects a closed month when a project’s ingestion failed near the boundary and caught up afterwards. Neither cadence changes the interval being described.

The privacy floor

No cross-tenant statistic is exposed unless at least 5 eligible organizations contribute a value to it.

The floor is enforced twice: at write, so a thin distribution is never stored, and again at read, so a row written when the population was larger stops being served the moment the rule tightens. When a distribution is suppressed, any previously stored row for that period is deleted rather than left behind.

The count that is checked is the number of eligible organizations with a non-null value — not the number of rows, and not the number of organizations that merely exist.

Who is excluded from the population, and why

Cross-tenant percentiles are computed only over organizations whose data is real. One shared predicate governs the refresh job, the reported sample size and the read path, so an exclusion genuinely tightens the privacy floor instead of being papered over.

Excluded	Why
Organizations that have generated demo/seed data (seed_data_created = true)	Seed data is synthesized and then scored through the real pipeline, so it is statistically plausible — which is exactly what makes it dangerous in a benchmark. It would silently shift every percentile toward generated data rather than showing up as obvious noise.
The open-source benchmark corpus organization (is_oss_corpus = true)	Its repositories are public projects, deliberately ingested so there is a labelled reference population from day one. That population is published separately as oss_corpus. Letting it into the cross-tenant distribution would silently mix open-source review norms and part-time contribution patterns into the number you are told is your peer group.

This is deliberately conservative: an organization with real data that also generated seed data is excluded until it cleans up. Benchmark integrity beats sample size, and the static-DORA fallback still gives that organization context on its own dashboards. The flag is cleared by seed-data cleanup and by the nightly inactive-demo job, so an organization that purges its seed data rejoins the population automatically — no opt-in step.

Nothing about an individual organization is ever exposed. The only fact published about the population is its size, and only once it clears the floor.

Missing data, again

A benchmark is where imputing zero does the most damage, so principle 3 is applied strictly:

• An organization with no value for a metric contributes no row — it is absent from the population, not present at zero.
• Every ratio divides by NULLIF(denominator, 0), so an empty denominator yields null rather than a fabricated rate.
• Your own value comes back as null, never 0, when there is nothing measured. Zero on a lower-is-better metric would read as Elite.

Percentile rank

Five percentiles are stored per distribution (p10, p25, p50, p75, p90). A rank is linearly interpolated between them, and clamped to [10, 90] outside that range — with five knots there is genuinely no information about position inside the top or bottom decile, and reporting 0 or 100 would be fabricated precision.

The reported percentileRank is a performance percentile: higher is always better, already flipped for lower-is-better metrics. A fast lead time reports high.

Falling back to static bands

When no cross-tenant distribution clears the floor, the response falls back to the published DORA bands for the four keys that have them, labelled source: "static_dora". This is what makes the feature useful on day one, before cross-tenant density exists.

For a key with neither cross-tenant density nor static bands, source is null and the UI renders nothing. An empty benchmark is reported as empty rather than defaulted — a “Low” badge caused by missing telemetry is indistinguishable from a real one.

The open-source corpus

A curated, versioned cohort of public open-source organizations, ingested and scored through exactly the same pipeline customer data goes through — the same GitHub sync services, the same change-effort scoring, the same fact tables and cubes. The corpus lives in its own organization, flagged is_oss_corpus, and its repositories are ordinary repository records: nothing about it is a parallel implementation.

It exists for three reasons: benchmark density before enough customers exist for a cross-tenant percentile; a calibration set that exposes a degenerate change-effort weighting before a customer sees it; and, eventually, a publishable index.

It is context, not a target. This is not a disclaimer bolted on at the end — it is the reason the corpus band is a separate, explicitly labelled field in the API response rather than another value of source. The response says “vs. open-source corpus”, never “vs. GuideMode organizations”, and carries its caveats with it.

What the corpus can and cannot measure

Available	Unavailable
flow.pr_cycle_time_hours	dora.change_failure_rate
change.effort_per_author	dora.mttr_hours
change.survival_rate
dora.lead_time_hours
dora.deploy_freq_per_week

A published release is the corpus’s production deployment. Public repositories almost never publish GitHub Deployments, but for a library the release is the moment a change reaches its users — there is no separate deploy to observe. The corpus therefore records each published release as an ordinary production deployment and attributes to it exactly the pull requests whose merge commits it shipped, so lead time (merge to release) and deployment frequency (releases per week) are measured on the same path as a customer organization’s, not approximated.

Read both with the release cadence in mind: a project that batches a month of merged work into one release has a long lead time by construction. That is a true statement about how it ships, but it is not the same quantity as a continuously deploying team’s — which is one more reason the corpus band is labelled and caveated wherever it appears.

The remaining two are still genuinely unmeasurable rather than merely sparse, and no corpus band is offered for them. A release carries no public success or failure signal, so there is nothing to derive a change failure rate or a recovery time from, and a fabricated zero failure rate would read as Elite on the DORA bands for every open-source project. Revert and hotfix detection, by contrast, works fine on public repositories, so change survival is the honest measure of the same concern — and it is only measurable at all because the release gives each change a production moment for its survival window to start from.

The observation window is a calendar boundary

The corpus reads back to the start of the previous calendar year and no further. On any day in 2026 that floor is 1 January 2025, so a corpus band always describes a stated, checkable span rather than whatever happened to be left over from the last successful run.

The alternative — stopping a nightly pass once it stops finding changes — was considered and rejected. It is cheaper, and it silently truncates a backfill somebody deliberately asked for: the resulting cohort is quietly missing its older half, and nothing in any number computed from it says so. Bounding the window instead makes the same decision explicit and reviewable.

After a project’s first pass, each entity resumes from its own watermark, so an ordinary night re-reads only what has actually changed. Issues and pull requests carry separate watermarks: a failure reading one no longer discards a clean read of the other, which is what stopped the largest projects in the cohort ever finishing their first pass.

Both issues and pull requests are read oldest-first, in bounded chunks, resuming from where the last pass stopped. A first backfill that is too large for one night simply continues the next until it reaches the floor. For the largest project in the cohort — tens of thousands of pull requests inside the window — that takes a handful of nights, after which each night costs almost nothing. Every project in the cohort therefore reaches the stated floor; none of them is quietly left with a shallower history than the window advertises.

One honest limit remains, and it is small. Pull requests whose author has since deleted their GitHub account cannot be retrieved by the date-ranged search the backfill uses — they are absent from GitHub’s search index entirely. Measured across roughly two thousand pull requests on three projects while validating this, exactly one was affected, and it was the only one with no author. It is stated here rather than papered over; nothing is filled in to disguise it.

The cohort is fixed per version

Every corpus repository carries the corpus version at which it entered the cohort and, if it has left, the version at which it left. Membership at version N is:

added <= N AND (removed IS NULL OR removed > N)

Adding or removing a repository bumps the version, and every computed benchmark row is stamped with the version it was computed at, so only equal-version distributions are ever compared. Without this, a quarter-over-quarter movement would confound a change in output with a change in who was counted. Retired repositories are retained, never deleted, so a benchmark published at an earlier version stays reproducible.

Organization first, then percentile

The same rule as the cross-tenant computation, one grain down: each corpus organization is reduced to one value for the period, and the percentile is taken across organizations. A large project and a small one carry equal weight.

An organization with no value for a metric contributes no row and is absent from the population, exactly as elsewhere — never present at zero.

Sample size: representativeness, not privacy

The cross-tenant privacy floor of 5 exists to protect organization anonymity. Corpus organizations are public and named, so there is nothing to anonymize and no privacy floor applies at write: corpus rows are always stored, labelled with an honest organization count.

A separate minimum of 5 contributing organizations governs whether the corpus band is surfaced. That is a representativeness judgement — a distribution over two projects is not something to compare against — and it is deliberately a different constant from the privacy floor so the two cannot be conflated.

Contributors carry no email address, ever

Corpus contributors are imported so that aggregates are correct: without authors, change.effort_per_author inflates because its denominator counts distinct authors, and every people-dimension breakdown goes empty.

Only what is needed for that is fetched. Every issue and pull request already records its author’s GitHub identifier and login, so authorship is reconstructed from data the corpus has already ingested rather than by reading profiles one by one. Organization membership is not imported at all by default: nothing published here is derived from it, and fetching it would mean requesting a profile per member whose principal contents — an email address — the corpus discards on principle.

No corpus contributor is ever given a real email address, even where GitHub publishes one. This is unconditional, not a fallback for a missing address. Every corpus user is written with a synthesized {login}@users.noreply.github.com address flagged as a placeholder. Two things follow: a corpus contributor can never be matched onto a real customer user, because placeholder addresses are excluded from email matching; and a corpus that is later published carries no personal contact data.

Individual-level corpus metrics are never surfaced, ranked or published. The corpus is org-level only, and any eventual public index inherits that constraint.

Comparability caveats — mandatory, not optional

These are rendered beside every corpus band in the product and are held as constants so the copy and this page cannot drift.

• Open-source review norms differ from commercial ones. Public pull requests wait on volunteer reviewers, so cycle times are longer for reasons unrelated to delivery capability.
• Contributors are largely part-time and unpaid, so per-author effort is not comparable to a funded team.
• The cohort is selected for being active, permissively licensed and well maintained — a survivorship bias toward healthy projects.
• Change-effort scoring is calibrated on a different shape of codebase. The architecture-role weights were derived from GuideMode’s own layout (routes/, services/, cubes/, ui/), so most open-source files fall to the default weight and corpus effort scores are flatter than they would be on a codebase matching those patterns. This is a known calibration limitation, documented rather than tuned away: retuning the weights would require a SCORING_VERSION bump and a rescore of every existing organization.
• A published release stands in for a production deployment, as above, so lead time and deployment frequency move with a project’s release cadence as much as with its delivery speed. Change failure rate and mean time to recovery are absent entirely, because a release publishes no success or failure signal.

Ingestion

Public repositories are read with a token-authenticated client rather than a GitHub App installation, since there is no installation on a repository we do not own. Everything downstream is the shipped path.

Because the corpus reuses the shipped sync rather than a cut-down one, it reuses its GraphQL documents too — and those select fields GitHub gates on organization scopes even for public repositories. The corpus token therefore needs three read-only scopes, not one: public_repo for repository metadata, contributors, pull requests and per-file diffs; read:org for team reviewers on pull requests; and read:project for project items on issues. A token missing the latter two fails loudly (the pass reports errors and the watermark is not advanced) rather than silently syncing half the data.

Repositories are synced one at a time and incrementally: each carries a watermark, and a pass fetches from that watermark minus a one-hour overlap (a bounded 90-day backfill on first run). The watermark advances only after a clean pass — the sync deliberately swallows a per-repository failure so one bad repository cannot stall the cohort, so a rate-limited half-fetch would otherwise be indistinguishable from success and the next run would resume past the window it never read.

Bot-authored pull requests are excluded by the same filter customer data uses. Open source is dependabot- and renovate-heavy, and corpus statistics would be meaningless without it.

Releases are read after pull requests, so there are changes to attribute them to. Only published releases count: a draft has reached nobody and is not a deployment. A release is attributed the commits between it and the previous release on the same version line — not simply the previous release by date. Maintained projects ship more than one major at a time, and a patch to the older line frequently lands between two releases of the newer one; diffing across that boundary would credit one release with the entire divergence between the two lines, most of which shipped months earlier under its own releases. The first release on a line has no bounded range and is therefore attributed nothing, which is honest rather than convenient. Prereleases are recorded as releases too — they reach users, and skipping one would leave a gap that the next release would diff across and claim.

Release attribution is exact rather than time-based: a change belongs to a release only if its merge commit is in that release’s commit range. Where the full range cannot be retrieved, nothing is linked for that release and the repository’s pass is treated as incomplete, so it is re-read rather than left with a partial attribution that looks complete.

Composite productivity index

Four dimension scores and one overall score, 0–100, higher is better. The index exists because GuideMode’s three measurement systems — flow telemetry, surveys, and session assessments — otherwise never meet, leaving dozens of raw KPIs on incompatible scales and no top-level answer.

Computed daily into composite_index_snapshots over a trailing 30-day window, stamped with indexVersion (currently 1).

Dimensions and inputs

Each dimension is a quarter of the index. Within a dimension, input weights sum to 1.

Dimension	Input	Source	Weight
Speed (0.25)	Lead time for changes	benchmark	0.4
	Deployment frequency	benchmark	0.3
	PR cycle time	benchmark	0.3
Effectiveness (0.25)	Overall productivity	survey	⅓
	Flow state frequency	survey	⅓
	Job satisfaction	survey	⅓
Quality (0.25)	Change failure rate	benchmark	⅓
	Accuracy/trust composite	assessment	⅓
	Code quality confidence	survey	⅓
AI Leverage (0.25)	Session process quality	session telemetry	⅓
	AI tool impact	survey	⅓
	Task helpfulness composite	assessment	⅓

Normalization

Benchmark inputs become your percentile — your score is your standing in the cross-tenant distribution, which makes the number unit-free and self-explaining. The ladder is strict:

1. A cross-tenant percentile rank, when one exists.
2. Otherwise a static DORA band midpoint — elite 90, high 70, medium 45, low 20 — so the index works on a single-tenant install from day one, before any cross-tenant density exists.
3. Otherwise null.

A percentile is used only when it came from the cross-tenant population. The open-source corpus is labelled context, never a target, and can never move a tenant’s index.

Survey and assessment inputs use the Likert and composite conventions defined above. Session process quality is already 0–100 and is passed through, clamped defensively.

Missing data

The rules are the ones used everywhere else on this page, and they matter more here than anywhere because the index fuses unrelated sources:

• A missing input is excluded, and the remaining weights within its dimension renormalize over what is present. It is never imputed, never defaulted, never zero.
• A dimension with no present input is explicitly null, and that null propagates into the overall index as an absence — the remaining dimensions renormalize around it — not as a value.
• coverage is the weighted fraction of the declared inputs that were actually present, 0–1. It does not renormalize itself, because its entire job is to report how much of the intended measurement is missing. A tenant with one of twelve inputs gets an honest score and a coverage of 0.03, not a confident-looking index.

Read the two together. An overall score of 70 at coverage 0.25 means “the quarter of this we could measure looks like 70”, not “you scored 70”.

Explainability

Every snapshot stores an inputs breakdown recording, for each declared input, its raw value in its own units, its normalized 0–100 value, its weight, and — for benchmark inputs — which population the normalization came from. Absent inputs are recorded with an explicit null rather than omitted.

This is a contract, not debug output: recomputing a dimension from its stored breakdown alone reproduces the stored dimension score exactly, and an integration test asserts it. Any number on the index dashboard can be taken apart and checked by hand.

Scope

Index version 1 is org-level only. Per-team indices, per-tenant custom weights, and SEM-based statistical confidence are all deliberately deferred; coverage is the confidence indicator for v1. AIVA scores are not inputs — a quarterly instrument does not belong in a daily index.

Change value

GuideMode measures what shipped work was worth in two layers:

• Change Effort Units (CEU) — deterministic per-file scoring of merged pull requests, computed from the diff. Shipped, documented in full below.
• Confirmed Change Value (CCV) — CEU multiplied by an outcome factor. Shipped for delivery work, documented in full below. Value is provisional at merge and confirmed by what happens next: delivery work confirms by survival (not reverted, not implicated in an incident); discovery work confirms by decisions it changed, which is not yet shipped.

The distinction matters because a diff-grading model on its own measures how hard something was to build, not whether it was worth building.

What CEU is, and what it is not

CEU is an effort, cost and risk measure. It is not a value measure, and it is not a productivity ranking.

This is not a hedge. The factors that make a diff score highly — size, control-flow density, nesting, centrality of the touched code — are precisely the classic defect predictors (Zimmermann & Nagappan, ICSE 2008). A model built from them tells you what was expensive and risky to build. Calling that “value” would mean rewarding churn, and would make any team that simplified its system look less productive for having done so.

CEU is therefore published as a directional measure. It is designed for comparing the shape of work over time — how much of a quarter went to KTLO versus investment — not for comparing individuals. It is not a compensation input.

Pre-release

CEU and CCV are not released, and are used by no one outside the team building them. No decision, dashboard or report outside this project depends on the numbers on this page today.

That is stated because it is load-bearing and a later reader cannot infer it. Several caveats here — and several elsewhere about “visibly moving published numbers” — were written for a measure that is not yet published, so the cost of correcting a number is currently close to zero. That is precisely why corrections are being made now. When this ships, this note goes, and the cost does not.

Scope and exclusions

CEU is computed for merged pull requests only. Files are excluded, but keep a row so counts reconcile against the PR’s own file count:

Reason	Rule
lockfile	package-lock.json, pnpm-lock.yaml, yarn.lock, Cargo.lock, poetry.lock, Gemfile.lock, go.sum, composer.lock, bun.lockb
vendored	node_modules/, vendor/, third_party/
generated	__generated__/, *.generated.*, *.min.js, *.min.css, dist/, build/, .next/, out/, *.map, *.snap
binary	Images, fonts, archives, PDFs, and .svg
bot_author	The PR author matches [bot], dependabot, renovate, github-actions, snyk-bot, or a -bot suffix

Bot authorship excludes the whole PR regardless of what it touched. This is the only bot-filtering rule in the product, and it is applied in one place.

migrations/ is scored, not excluded — corrected in scoring version 2. It was previously in this table and in the architecture weighting at the top weight of 10. Exclusion runs first, so the two rules never met: every migration scored zero and no migration ever reached the fact table. Hand-written schema changes are among the hardest work to reverse, and scoring them as nothing was a defect rather than a policy.

.svg is excluded as binary, although it is text. Most SVGs in a repository are exported assets rather than hand-written markup, and a single exported icon can run to thousands of lines — scoring it would swamp the real work in the same pull request. The cost is real and stated here rather than left to be discovered: a hand-authored SVG scores nothing.

Work classification: intent and area

Every scored file is classified on two independent axes, computed separately and never allowed to compete.

• Intent — why the change was made: feature, fix, maintenance. Nullable.
• Area — what kind of artifact was touched: app, test, docs, config. Never null.

Until scoring version 3 these shared a single work_type enum, and because there was one slot the classifier had to choose. It chose area: a path pattern won at confidence 1.0 before any intent rule ran, so a test written to reproduce a bug was recorded as test and the fact that it was bug work was discarded — and because test bucketed as investment, that effort counted against KTLO rather than toward it. Over the benchmark corpus, 10.2% of files had an intent signal that never reached the row.

The separation is not an invention. ODC (Chillarege et al., 1992) makes documentation a value of its Target axis — the entity that was fixed — and never of its Defect Type axis; ISO/IEC/IEEE 14764 defines every maintenance type by purpose and has no test or docs category at all; Conventional Commits makes only feat and fix normative and provides scope as a separate slot. Our three intents map 1:1 onto ISO 14764’s additive, corrective and perfective; we keep the shorter names.

Intent — first match wins

Precedence	Rule	Confidence
1	Linked issue type via issue_pull_request_links — bug/incident → fix, feature/discovery → feature, chore/tech_debt → maintenance, other falls through	1.0
2	PR title conventional-commit prefix — feat → feature, fix/hotfix/bug → fix, chore/refactor/perf/build/ci/style/deps → maintenance	0.7
3	PR title leading imperative verb — add/introduce/implement → feature, fix/correct/resolve → fix, bump/refactor/remove/rename → maintenance	0.6
4	This file’s own patch adds an exported symbol → feature	0.55
5	Fallback on file status — added → feature, removed → maintenance	0.5
—	Nothing matched → intent is null	—

Two of these are worth explaining. Rule 4 is the only rule that can make a modified file a feature on its own evidence rather than its pull request’s, which is what makes feature reachable without a linked issue at all. And rule 3 outranks rule 4 deliberately: intent is the author’s purpose, and a title verb states it while an added export is circumstantial — a bugfix that extracts a helper adds one.

docs: and test: title prefixes are not intent rules. They name an area at pull-request grain, and area is a file-grain fact: a docs:-titled pull request routinely touches source files.

A modified file that reaches no rule has a null intent. It is not guessed into maintenance, which is what the old rule 4 did — and that single change is the difference between a measured KTLO split and an assumed one. Expect roughly 45% of effort to carry a null intent.

Area — first match wins

Precedence	Rule
1	Path pattern — tests → test, docs/markdown → docs, CI/config/Dockerfile/tsconfig → config
2	Patch-derived — a change confined to comments → docs; a manifest whose every changed line pins a version → config
3	app, the residual

There is no area confidence column, deliberately: a single-rule dimension does not need a confidence scale, and publishing one would imply a gradation that does not exist. Rule 2’s claims are exhaustive (“every changed line is a comment”), so they are suppressed on a truncated patch — the line that would refute them is exactly the one the 16 KB cap may have removed. Rule 1’s are not, so they always apply.

Area is a reporting dimension only and never enters the effort formula. architecture already carries “where in the system” as a risk weight; an area multiplier would amount to “docs work counts less”, the double penalty this methodology rejects.

KTLO is keyed on intent alone

Derived, never stored: KTLO = fix + maintenance; Investment = feature. The KTLO and investment shares on dashboards are proportions of effort, not of file count, and the denominator is all scored effort, not just classified effort.

Area does not enter the bucket. A test is KTLO or investment according to why it was written.

A file whose intent could not be determined belongs to neither bucket. It keeps its effort score and still counts toward total effort, but it is excluded from both the KTLO and the investment share, exactly as it is excluded from the per-intent shares. Bucketing it as investment by default would credit the investment side with work nobody has identified. It is published as unclassifiedShare, and it is large — reading the KTLO share without it beside it is reading half a number.

How much of the split is actually evidence

The version-2 measurement below is what motivated the two-axis split, and it is recorded because the defect it names is the reason the taxonomy changed. Measured on the open-source benchmark corpus — 105,929 scorable files across four organizations, one of which supplies 94.5% of them — the four version-2 rules divided as follows:

Rule (version 2)	Grain it reads	Files	Effort
Path pattern	the file	24.2%	25.1%
Linked issue type	the pull request	16.8%	18.2%
Title prefix	the pull request	7.6%	7.3%
Fallback on file status	nothing	51.4%	49.4%

Three consequences, which together are the argument for the split.

Only a quarter of files were classified by looking at the file. The two middle rules apply one label to every file of a pull request, however mixed that pull request is.

feature was unreachable by path. There was no feature path pattern, so a modified source file that failed both pull-request rules could only reach maintenance. In a mature codebase nearly every change is a modification, and the result was that 76.3% of all KTLO effort carried a maintenance label that no rule reached by examining the change. Version 3 addresses this directly: the exported-symbol rule is the first that can make a modified file a feature on its own evidence, and the fallback no longer guesses maintenance at all.

The rules described layout, not work. Deterministic coverage by path ranged from 22% to 71% across the four organizations. A rule set that varies that much by repository is encoding one project’s conventions — which is precisely the observation that those rules were describing area, and were never intent rules at all.

Version-3 intent coverage

Measured on the same corpus after the replay — now 118,436 scorable files across six organizations, still dominated by one:

Rule	Grain it reads	Files	Effort
Linked issue type (1.0)	the pull request	18.7%	20.7%
Title prefix (0.7)	the pull request	12.1%	11.5%
Imperative title verb (0.6)	the pull request	15.2%	15.0%
Exported symbol added (0.55)	the file	6.8%	7.2%
Fallback on file status (0.5)	nothing	3.3%	3.6%
Nothing fired — intent unknown	—	43.9%	42.0%

Coverage went down, and that is the improvement. Version 2 classified 100% of files because rule 4 always returned a label; roughly half of that was the fallback guessing maintenance on any modified file. Version 3 declines to guess, so 42% of effort now carries no intent — and the KTLO share is published over the work something is actually known about instead of over every file with a guess attached. KTLO is 29.4% of effort; unclassified is 42.0%. Read them together: the second number is the honest width of the error bar on the first.

The fallback’s collapse from 49.4% of effort to 3.6% is the other half of the same change. It now fires only on whole added or deleted files, where status really is evidence.

Only 6.8% of files are still classified by looking at the file — the exported-symbol rule, which did not exist before and is the only rule that can make a modified file a feature on its own evidence. The rest is pull-request grain, which applies one intent to every file of a pull request however mixed it is. That remains the largest known weakness.

And the rules still describe convention rather than work. The undecided share ranges from 13.7% to 48.3% across the six organizations, which is a rule set encoding commit habits: repositories that use conventional commits are well covered and repositories that do not are not. That variation is now visible as an unclassified share per organization rather than hidden inside a maintenance label.

The formula

effortScore = (complexity × 0.5 + engagement × 0.2 + architecture × 0.3)
              × decay
              × fixMultiplier

Complexity and architecture are on a 0–10 scale; engagement is on 0–6 since scoring version 2 removed the term that reached the top of its range. The weights sum to 1 and live in one exported constant, but see the note under engagement — its effective weight is lower than 0.2 implies.

Complexity (0–10) — cognitive-load proxy over the changed lines:

effectiveLOC = addedLOC + 0.5 × removedLOC
sampled      = added lines if any, otherwise removed lines
sizeScore    = min(10, log₂(effectiveLOC + 1) × 1.5)
density      = 0.6 + min(1, controlFlowKeywords(sampled) / sampledLOC) × 0.8
depth        = 0.7 + min(1, meanIndentColumns(sampled) / 2 / 6) × 0.6
complexity   = clamp(sizeScore × density × depth, 0, 10)

Size is log-scaled so a 500-line change is not ten times a 50-line change. Control-flow keywords are if|for|while|switch|case|catch|&&|||=>|?.|return.

Removed lines count at half an added line — corrected in scoring version 2. Before that, complexity read the added lines only, so a pure deletion scored zero on this path while the no-patch fallback below scored it on full churn: the same deletion scored differently depending on whether GitHub returned a patch, and scored higher when it did not. Deleting is real work — you have to read and understand code before you can safely remove it — but it is cheaper than writing the same volume, so it counts at half on both paths. Density and nesting are read off whichever side has content, because for a pure deletion the branching of what was removed is the honest proxy for what understanding it cost.

Engagement (0–6) — how spread out the change is:

engagement = clamp(min(6, log₂(hunks + 1) × 2.2), 0, 10)

The co-change term was removed in scoring version 2, and with it the only part of a file’s score that depended on anything outside that file. It added a bucketed count of the other files in the same pull request — 0 (≤1 file), 1 (2–3), 2 (4–9), 3 (10–24), 4 (≥25) — as a per-pull-request constant on every file, which the cube then summed per file. Total effort therefore grew with the square of a pull request’s width, and splitting one 30-file pull request into five 6-file ones changed its total CEU. That is a gameable formula defect, not a signal.

The consequence is stated here rather than left to be derived: engagement’s observable range is now 0–6, not 0–10, so its maximum contribution to the blend falls from 2.0 to 1.2. The three weights still sum to 1 and the constant is unchanged, but the effective weighting of engagement is lower than 0.2 implies. This is the price of invariance, and it is paid deliberately.

Architecture (0–10) — a risk/effort weight, not a value weight:

Role	Weight	Matches
db-schema	10	db/schema/, db/migrations/, migrations/, schema/
api-surface	8	routes/, services/, workflows/, webhooks/, api/
analytics	6	cubes/, analytics/, metrics/
shared	6	shared/, utils/, lib/, common/, types/
default	5	Anything unmatched
ui	4	ui/, components/, pages/
presentation	2	*.css, *.scss, *.stories.*, styles/, stories/, fixtures/

Changes to central code are weighted as harder and riskier because centrality predicts defects. We do not claim central code is intrinsically more valuable — under that reading, a team that successfully decoupled its system would deflate its own score for having made the code easier to change.

Decay (0–1) — a discount for mechanical noise only. Defaults to 1.0, meaning no discount:

Case	Factor
Pure rename (git reports a move, ≤2 content lines changed)	0.1
Formatting-only (identical content after whitespace normalization)	0.15
Generated content that slipped past the path filters	0.2
Everything else	1.0

What decay deliberately does not penalise, and why:

• Refactors. Penalising restructuring contradicts the empirical refactoring literature (Kim, Zimmermann & Nagappan, FSE 2012) and creates a direct anti-maintenance incentive. Refactors are classified as maintenance and scored at full weight.
• Self-rewrites. Reworking your own recent code is normal iteration, not waste.
• Deletions. Removing dead code is high-value maintenance. Zeroing it would repeat the LOC fallacy in reverse.

This is a deliberate departure from ETV-style models, where maintenance is penalised twice — once by a decay term that catches refactors, and again by KTLO framing that codes it as overhead.

Fix multiplier (1.0–1.5) — applies only to fix work with a known linked-issue age:

fixMultiplier = 1 + clamp(log₁₀(ageDays + 1) / log₁₀(366) × 0.5, 0, 0.5)

Fixing a long-lived issue costs more than fixing something written this morning, because the context has to be rebuilt first. Capped at 1.5 so age can never dominate. Non-fix work, and fixes with no known issue age, get 1.0.

Note this term is the one place CEU touches the firefighting incentive Kerr (1975) warns about — rewarding the fix more than the prevention. It is capped tightly for that reason, and outcome confirmation (CCV) is the structural answer.

Missing patches

GitHub omits the patch for binary files and very large diffs. Those files fall back to size-only scoring from the additions/deletions counts, using the same addedLOC + 0.5 × removedLOC churn as the main path so a file does not score differently according to whether a diff came back. Hunk count is assumed to be 1: a file that changed at all changed in at least one place, which is a floor rather than a guess.

Decay is evaluated on this path too, corrected in scoring version 2 — it was previously hardcoded to 1.0. Two of its three rules need no content: a pure rename is visible from the file’s status, and a generated path is visible from the path. Only the formatting-only rule is unreachable without a diff, and it fails closed — no discount — rather than guessing.

Versioning and rescoring

scoringVersion is stored on every row and must be bumped whenever any weight, formula, or clamp changes. Rescore jobs target WHERE scoring_version < N, so recalibrating the weights is a backfill rather than a migration. A PR is also re-scored when its merge commit SHA changes.

Current scoring version: 3.

---

Confirmed Change Value (CCV)

CEU grades a change at the moment it merges, which makes it outcome-blind. A sophisticated, well-placed, expensive change that gets reverted the following week scores identically to one that quietly runs in production for a year.

CCV supplies the missing half:

CCV = effort units (CEU) × outcome factor

Value is provisional at merge and confirmed by what happens next. For delivery work, confirmation means survival in production.

Outcome factors

Survival status	Factor	Meaning
pending	null	Deployed but still inside its observation window — outcome not yet known
confirmed	1.0	Survived the window with no negative signal
reverted	0.0	An explicit revert landed
hotfixed	0.5	A fix-typed follow-up touched the same files inside the window
flagged	null	Needs review — for example, the linked issue was reopened

pending and flagged are null, not zero. This follows directly from principle 3 on this page: an outcome that is not yet known is missing data, and missing data is never imputed. Scoring pending changes as zero would make every recent change look worthless, and would make the aggregate drift with how recently the confirmation job happened to run. confirmedValue is therefore null for a change still inside its window — not a low number, but an absent one.

The observation window

The window opens when a change first reaches production successfully, detected from the existing deployment linkage (pull_request_deployment_links by commit SHA, filtered to production environments via the configured environment mappings) and the deployment’s first successful status.

Default window: 14 days. The window length is stored per row when the row is created, so changing the default never retroactively rewrites the history of changes already confirmed under the old one.

A change that never reaches production stays pending indefinitely. It is not confirmed, and it is not penalised — it simply has no outcome to report.

Revert detection is not SZZ

Reverts are matched from explicit signals only: GitHub’s Revert "<title>" convention and git’s This reverts commit <sha> trailer. SHA matching is preferred over title matching, and a title match must be exact, so near-duplicate titles do not cross-link.

The SZZ family of algorithms — which traces a fix back to the commit that allegedly introduced the bug — is deliberately not used. Its mislabeling rates are well documented (Rosa et al., ICSE 2021; Herbold et al.), and a metric built on unreliable blame attribution is worse than no metric, because the errors are invisible to the people being measured.

A revert always wins, including after a change has already been confirmed and including outside the window. Being reverted three weeks later still means the change did not survive.

Survival rate excludes in-flight work

survivalRate = confirmed / (confirmed + reverted + hotfixed + flagged)
windowComplete = deployedToProductionAt + windowDays ≤ now

Both halves count window-complete pull requests only. Changes still inside their window are excluded from both. Including them would dilute the rate purely by recency: a team that shipped heavily last week would appear to have a collapsing survival rate, when in fact nothing has been observed yet.

The denominator also excludes pending — corrected in scoring version 2, where it previously divided by every window-complete pull request. A window closes on the clock, but a status only advances when the nightly outcome pass runs. Between the two, a change is window-complete and still pending: in the denominator, ineligible for the numerator, and depressing the published rate by an amount that depended on when the job last ran. That is exactly the drift this section says it was designed to avoid, so the published formula and the shipped SQL now agree.

flagged deliberately stays in the denominator. A change whose linked issue was reopened inside its window never auto-confirms, so counting it would be wrong; but its window has closed and something is known about it, so dropping it would flatter the rate. It is an observed non-survival, not an unobserved one.

revertRate was corrected the same way, so the two remain directly comparable. Both are cube measures whose SQL changed and whose names did not, per the measure-stability principle at the top of this page.

Observed engagement

ETV-style models infer cognitive effort from the shape of the diff. GuideMode can observe it directly instead, by joining AI coding sessions to the changes they produced (pull_request_session_links → session metrics): session duration, iteration count, interruption rate, and input clarity.

Note that all three of those session signals were redefined — see their sections above. Iteration count now measures conversational rounds rather than inferred refinements, interruption rate counts only explicit interrupts, and input clarity is averaged per message. Historical sessions are reprocessed from their stored transcripts so the series stays on one definition.

This is the era-correct attribution answer. As AI writes more of the code, diff-shape heuristics increasingly measure the model; session joins measure the human.

Caveat, stated plainly: these measures depend on session upload, and adoption is currently low. Where no session is linked, avgSessionMinutesPerChange is null, not zero, and dashboards render it as “no data”. Treat these numbers as covering the subset of work that was captured, not the whole.

What CCV is still not

CCV measures whether a change survived, not whether it was worth building. A feature nobody uses can survive in production indefinitely. Business-impact signals — feature usage, revenue — are a further leg that needs product analytics integration, and are not part of this definition.

CCV is reported at team level. Individual confirmed-value rankings are deliberately not exposed, per SPACE.

Validation status

Publishing a formula is not the same as validating it. This section records, honestly, what has and has not been checked about CEU and CCV, and is updated as each study lands.

Property	Status (v1.11)
Reproducibility	Deterministic. The last exception closed in scoring version 2: where a pull request links to several issues, the oldest now wins, fixed by ordering the lookup rather than taking whichever row the database returned first. The defect was wider than the fix multiplier it was found through — the same arbitrary row also supplied the issue type, which sets work type at confidence 1.0 and therefore decided whether the fix multiplier applied at all, and what classification source was recorded. Oldest-wins is stable under later linking, because an issue linked afterwards can only be younger.
Calibration	The blend weights (0.5 / 0.2 / 0.3) and the architecture table were set by hand and have never been checked against a labelled corpus. The architecture patterns match this codebase’s layout; on other layouts most files take the default weight, which makes that factor close to a constant.
Criterion validity	Untested. Nothing has yet shown that a high-CEU change is more likely to be reverted, hotfixed, or slower to review than a low-CEU change. This is the first study planned against the open-source corpus.
Incremental validity	Untested, and the most important open question. Because per-file scores are bounded and are summed per file, total CEU is expected to track file count and lines changed closely. Until the corpus study shows CEU carries information beyond size, read aggregate CEU as size-weighted output, not as effort.
Invariance	Passes, exactly rather than approximately. Removing the co-change term in scoring version 2 left nothing in a file’s score that depends on anything outside that file, so the per-file scores of a change are identical however it is split across pull requests. Pinned by test.
Uncertainty	No confidence intervals are published yet. A survival rate of 87% from 8 changes is shown with the same weight as one from 800. Bootstrap intervals are planned.
Additivity across work types	We sum across the three intents into totalEffort. ETV-style models deliberately do not — and the reason they refuse is now half-addressed: two of their five buckets (tests, docs) were never on the same axis as the other three, and scoring version 3 moved those onto a separate area axis that is never summed with intent. Per-intent headlines are still planned.

Reading AI impact from these measures

aiAssistedShare means “at least one agent coding session was uploaded on the same branch”. It is bounded by session-upload adoption, so a rising share is evidence of adoption, not of impact, and comparing CEU for AI-assisted against other work confounds the two. The randomized evidence in the literature (see References) also shows that output-volume measures cannot detect the case where AI increases output and the time spent producing it.

The design we recommend, and are building, is within-developer: compare each author’s CEU, merged pull requests and survival rate before and after their first linked session, using authors who have not adopted as the calendar-trend control. Until that ships, treat any AI-vs-non-AI comparison on the Change Effort dashboard as descriptive.

---

Discovery value

Delivery confirms by survival. Discovery confirms by decisions — and both decision branches are wins.

A spike that concludes “don’t build this” has produced exactly what it was commissioned to produce. Most delivery funnels record that as leakage, because they only count discoveries that converted into features. GuideMode records it as a success.

Declared vs observed outcome

Two vocabularies coexist deliberately, because they answer different questions.

Field	Question	Source
validationStatus	Did a link to a downstream feature appear?	Observed — derived from issue_links counts
resolution	What did the team decide?	Declared — set on the issue, from a label mapping

validationStatus is unchanged from v1.0 and keeps its original name and derivation. discoveryOutcome is the reconciled view:

discoveryOutcome =
  resolution                     if the team declared one
  else 'validated_build'         if a `validates` link exists
  else 'invalidated'             if an `invalidates` link exists
  else 'closed_undeclared'       if the issue is closed
  else 'in_progress'

Where the two disagree, the declaration wins. A discovery closed as invalidated with no downstream links reads as a validated kill rather than closed_unvalidated funnel leakage — which is the entire point.

closed_undeclared is the value to watch. It means a discovery ended without anyone recording what was concluded.

Resolution values

Value	Meaning
validated_build	Research validated the idea; build it
invalidated	Research killed the idea
abandoned	Closed without reaching a decision

Resolution is only meaningful for discovery issues, and it is sticky: once declared, removing the source label does not erase it. A later sync that no longer sees wontfix must not silently un-decide a decision.

Provider labels map to resolutions through label_mappings.maps_to_resolution, the same mechanism as blocked-status mapping. Absent a tenant mapping, conservative defaults apply (wontfix, not-planned, rejected, invalid → invalidated; validated, approved → validated_build; stale, obsolete, duplicate → abandoned). Configuring any resolution mapping replaces the defaults entirely.

Kill efficiency

killEfficiency = count(discoveries where killCostDays is not null)
                 / sum(killCostDays)
killCostDays   = days from discovery creation to close, for kills only

Kills per discovery-day spent reaching them. Higher is better: the team is reaching “no” cheaply.

killCostDays is null for anything that was not a kill, so non-kills never dilute the denominator. Discovery duration is the denominator today; per-issue session effort is a planned refinement and is not yet part of this definition.

Belief shift

The genuinely novel instrument, and the one that requires opting in.

A two-question probe is sent to whoever is doing a discovery, once when it starts and once when it closes:

1. Confidence (Likert-7) — “How confident are you that this is the right thing to build?”
2. Decision (choice) — “Has what you learned changed the decision?”

beliefShift(respondent) = |post − pre|        on the normalized 0–100 scale
avgBeliefShift(issue)   = mean over respondents who completed BOTH halves
learningVelocity        = avgBeliefShift / discovery duration in days

Movement in either direction is learning. A spike that destroyed confidence in an idea taught the team as much as one that built it, so the metric is an absolute difference. The signal worth chasing is near-zero shift on a long, expensive discovery: weeks spent, and nobody’s mind moved.

This is a tractable approximation of value-of-information. Measuring VOI properly requires a prior, a posterior and a decision model; measuring how far a stated belief moved is something a team can actually answer in two clicks.

Null, never zero. A respondent who answered the pre probe but not the post contributes nothing — an unanswered post probe is an absence of measurement, not a shift of zero. Issues with no completed pair report null and render as “no data”. For the same reason, a post probe is never sent to someone who did not complete the pre probe.

decisionChangedRate counts only changed_direction. Confirming a direction is a useful outcome, but it is not a change.

Insight shelf life

insightShelfLifeDays = days since close, where
  resolution = 'validated_build'
  AND no linked feature has started delivery

Research has a half-life. The longer a validated conclusion sits unbuilt, the more the world it described has moved on. This is where decay belongs — on unrealized insight, not on refactors, which is why CEU’s decay factor is confined to mechanical noise.

Null unless a validated build is actually sitting unstarted.

Scope

All discovery-value measures are reported at team level. There is deliberately no author dimension on belief shift or kill efficiency. Rewarding individuals for spike volume, or for “shifting beliefs”, is a textbook Goodhart trap.

The belief-shift schedule ships inactive and cannot be created active. It is the only instrument that contacts a person twice about a single item and bypasses the per-user survey budget to do it, so it requires an explicit opt-in.

---

Methodology changelog

v1.14 — current

Adds API-equivalent cost as a session telemetry measure. Not formula-changing and not a scoring-version bump: seven measures and two dimensions ship under new names on the Metrics cube, no existing measure changes meaning, and nothing is rescored. Per Principle 5, no stored dashboard configuration needs resetting.

Change	Effect
API-equivalent cost published — totalApiEquivalentCost, avgApiEquivalentCost, costPerAgentLine, pricedSessionCount, costCoveragePercent, partiallyPricedSessionCount and totalProviderReportedCost, with costSource and primaryModel as dimensions.	The list-price value of tokens consumed becomes readable per session, per agent line and per model. It is deliberately not spend: a Max, Pro or Copilot user pays a flat subscription and is billed none of it. Deriving it from one price table is what makes providers comparable at all, since only Claude reports a cost of its own.
Unpriced sessions are excluded from every cost total, never counted as zero.	The same rule as everywhere else on this page, applied where a zero would be most tempting and most wrong. It makes the headline a total over an unknown denominator, which is why costCoveragePercent ships beside it and why totalApiEquivalentCost should never be read alone.
Cost is anchored on the session’s end time, falling back to its start, and never on processing time.	A session backfilled months later prices identically to one priced as it arrived. Re-pricing after a price row is added is therefore idempotent, and a step change in a cost time series can only be a change in usage or a change in a dated price — not a change in when the pipeline happened to run.
A model variant never falls back to its base row. claude-opus-5[1m] must have its own price row.	An unknown variant is recorded as unpriced instead of being silently priced as its cheaper base model. Coverage falls where a variant is missing, which is visible; under-pricing would not have been.
The 1-hour cache-write premium is priced only on tokens the transcript can evidence, capped at the provider’s authoritative total and never extrapolated.	The figure is a floor. Across 15 real sessions it came in 4.0% below Anthropic’s own reported cost and never above; extrapolating scored better on average but over-stated 8 of the 15 by up to 8%. A reliably low bound was preferred to a sometimes-high estimate.
Price history accrues forward from the first sync, with every model’s first row backdated so older sessions still price.	Sessions predating the seed are priced at seed-date prices rather than left unpriced. Exact from the seed date forward; approximate before it, in a direction that cannot be known.
Models that tier above 200k input tokens price at the base rate. The Sonnet 4 family bills a premium there, and the stored breakdown does not split tokens by position in the context window.	A long-context Sonnet session understates, and coverage reporting will not reveal it because the session did price. Stated here rather than left to be found in a reconciliation.

v1.13

Splits work classification into two independent axes, intent and area, and re-keys the KTLO split onto intent alone. Ships as scoring version 3 with one rescore, because re-keying the fix multiplier changes a score. Cube measures were renamed and dropped, so every tenant’s stored dashboard configuration must be reset to its template.

Change	Effect
work_type is replaced by work_intent and work_area. One enum carried two facts that are not the same kind of fact — why a change was made, and what kind of artifact it touched — and with one slot the classifier had to choose. It chose area: a path pattern won at confidence 1.0 before any intent rule ran.	A test written to reproduce a bug is now area = test and intent = fix, where it used to be test alone. Over the benchmark corpus, 10.2% of files had an intent signal that the path pattern discarded. The two are computed independently and never compete, so the first-match-wins conflict between them no longer exists.
KTLO is keyed on intent alone: KTLO = fix + maintenance, Investment = feature. Area does not enter the bucket.	test and docs effort stops being credited to investment regardless of why it was written. Roughly 5.8% of corpus effort is test × fix or test × maintenance — KTLO support work that was counted as investment. The published KTLO share moves substantially, and the denominator is unchanged: it is still all scored effort.
A modified file with no evidence now has a null intent, where it was previously guessed into maintenance.	The difference between a measured KTLO split and an assumed one. unclassifiedShare becomes large — expect roughly 45% of effort — and it is published beside the KTLO share rather than folded into it. This is a coverage figure falling and honesty rising at the same time.
Two rules promoted from the shadow study: a leading imperative verb in the title (0.6), and an added exported symbol in the file’s own patch (0.55).	The exported-symbol rule is the first that can make a modified file a feature on its own evidence rather than its pull request’s, which is the direct answer to feature having been unreachable by path. The title-verb rule fires on 29% of corpus files where the conventional-prefix rule fires on 7.6%. issue_label and branch_prefix were measured and not promoted: they fire widely but almost never on the files that had no classification, which was the population that needed one.
The fix multiplier is gated on intent, not on work type. This is the only place classification enters the effort arithmetic, and the only reason this is a scoring-version bump.	A test written for a long-lived bug now receives the multiplier it was always entitled to; a file that reached fix through a title guess when the oldest linked issue was not a bug stops receiving one. Measured net effect on total effort is under 1%, in the positive direction.
Area never enters the effort formula.	Stated because the alternative is tempting and wrong: architecture already carries “where in the system” as a risk weight, and an area multiplier would amount to “docs work counts less” — a double penalty this methodology rejects. Area is a reporting dimension only.
classification_source / classification_confidence become intent_source / intent_confidence, and a new area_source joins them. There is no area confidence.	Provenance follows the axis it describes. A single-rule dimension does not need a confidence scale, and publishing one would imply a gradation that does not exist.

v1.12

Documentation-only release. No formula, measure or classification change: the classifier behaves exactly as it did in v1.11, and a regression diff against the committed corpus baseline confirms zero files moved.

Change	Effect
Work-type coverage published. The study artifact now carries classification_source and classification_confidence, and a new chapter reports which rule decided each file, per organization, in effort as well as file count.	The KTLO split can be read with its evidence for the first time: a quarter of files decided by the file itself, half by no evidence at all, and 76.3% of KTLO effort resting on the fallback.
The unclassified state documented as unreachable. The page described what happens to a file with no work type while the rule table said rule 4 always returns one.	The contradiction is resolved in favour of the truth: the machinery exists end to end, nothing produces the state yet.
KTLO charts made able to show an unclassified residual. Two charts were driven by the ktloBucket dimension, and one stacked them to percent.	An unclassified residual will appear as visible headroom instead of an unlabelled series or, worse, being renormalised away. unclassifiedShare is now a published measure.
Pre-release status recorded.	Caveats written about “moving published numbers” can be read for what they are.

v1.11

Fixes the five CEU and CCV defects the v1.10 statistical review found. All five change a score or a denominator, so they ship together as scoring version 2 with one rescore. No cube measure was renamed; the SQL behind two of them changed.

Change	Effect
The co-change term is gone from engagement. It added a bucketed count of the other files in the same pull request to every file in it, and the cube then summed per file, so total effort grew with the square of a pull request’s width.	CEU is now invariant to how work is split: the per-file scores of a change are identical whether it ships as one pull request or five. Engagement’s observable range drops to 0–6, so its effective contribution to the blend falls from 2.0 to 1.2 — a real reweighting, stated rather than left to be derived. Every multi-file pull request scores lower than it did under version 1.
Linked-issue selection is deterministic — the oldest linked issue wins, by explicit ordering. The lookup previously kept whichever row the database returned first.	A rescore now reproduces its own numbers. The defect reached further than the fix multiplier it was found through: the same arbitrary row set the issue type, which drives work-type classification at confidence 1.0, so work type, classification source and whether the multiplier applied at all could all move between runs.
The survival-rate denominator excludes pending — confirmed / (confirmed + reverted + hotfixed + flagged), in the cube, the per-tenant benchmark query and the per-organization corpus query, which must agree.	The published rate no longer drifts with how recently the nightly outcome pass ran. Survival rates rise slightly wherever window-elapsed changes were sitting unprocessed. flagged deliberately stays in the denominator: its window has closed and it is an observed non-survival.
revertRate corrected the same way, so it and survivalRate share one denominator.	The two measures stay comparable. Wider than the original defect report, which named only survival rate; fixing one without the other would have made them silently inconsistent.
migrations/ is scored rather than excluded as generated output. It was in both the exclusion list and the architecture table at the top weight of 10, and exclusion ran first.	The db-schema weight is reachable for the path that most deserves it. Every migration previously scored zero and never reached the fact table at all, so schema work — among the hardest work to reverse — was invisible to CEU. Effort rises on schema-heavy pull requests.
Deletion scoring is symmetric. Removed lines count at half an added line on both paths, and the no-patch fallback evaluates decay instead of assuming 1.0.	The same deletion no longer scores differently according to whether GitHub returned a patch — and no longer scores higher when it did not. Pure deletions score above zero for the first time.
.svg documented as excluded. No behaviour change; the rule was already shipped and undocumented.	A reader can now see that a hand-authored SVG scores nothing, and why.

Two rows of the validation-status table flip to passing: reproducibility and invariance. Calibration, criterion validity, incremental validity and uncertainty are untouched and still untested — the corpus study that addresses them is separate work, and nothing here should be read as having done it.

v1.10

Documentation-only release following a statistical review of CEU and CCV. No formula or measure changes.

Change	Effect
Validation status published. A table records what has and has not been checked about CEU/CCV: calibration, criterion and incremental validity, invariance, uncertainty.	Readers can see that aggregate CEU is, for now, best read as size-weighted output. Each gap has a planned study or fix, named in the table.
Survival-rate formula corrected to what runs. The page previously published a status-based denominator; the implementation divides by all window-complete pull requests, which can include pending rows between a window closing and the daily outcome pass.	The defect is documented rather than hidden; the fix ships with scoring version 2.
AI-impact guidance added. aiAssistedShare is described as an adoption floor, and the recommended within-developer design is stated.	Prevents the adoption-vs-impact confound from being read as an AI effect.
References section added. Every external claim on this page now has a citable anchor, and new research has a home.	Doubles as the reading log for the methodology.

v1.9

Corrects the KTLO bucket for unclassified work, and breaks confirmed value out by work type.

Change	Effect
A change whose work type could not be classified now belongs to neither KTLO nor investment. It previously fell through to investment, because NULL IN ('maintenance', 'fix') evaluates to NULL rather than false, so unclassified effort silently inflated the investment side.	ktloShare and investmentShare no longer count unclassified effort in either numerator. This is how featureShare…docsShare already treated it, so the two families of share measure now agree. Both shares shift on the next fact refresh wherever unclassified work exists.
Per-work-type Confirmed Change Value and survival rate published — featureConfirmedValue…fixConfirmedValue and featureSurvivalRate…fixSurvivalRate.	Confirmed value can be read by kind of work rather than only in aggregate. Additive: no existing measure changes meaning.
Per-work-type survival is a breakdown, not a partition. The grain is one row per file while survival counts distinct pull requests, so a pull request touching both a feature and a test is counted in both.	Stated rather than silently corrected. A dominant-work-type attribution per pull request would be needed to make these mutually exclusive, and does not exist.

v1.8

Gives open-source changes a production moment.

Change	Effect
A published release is treated as a production deployment for the open-source corpus. For a library that is when a change reaches its users; there is no separate deploy to observe.	Confirmed Change Value resolves for corpus changes instead of staying permanently unknown. Nothing about how a customer organization’s deployments are recorded changes.
Change survival becomes measurable for the corpus. A survival window starts at a change’s first production deployment, which public repositories previously never supplied, so every corpus change sat unresolved for ever.	The corpus survival-rate band exists at all, rather than being computed over an empty population.
Lead time (merge to release) and deployment frequency (releases per week) join the corpus metric keys.	Two of the DORA four gain an open-source context band. Both move with release cadence as well as delivery speed, and are labelled and caveated accordingly.
Change failure rate and mean time to recovery remain excluded. A release publishes no success or failure signal.	No fabricated zero failure rate, which would read as Elite for every open-source project.
Attribution is exact, not time-based. A change belongs to a release only if its merge commit is in the range between that release and the previous release on the same version line.	A patch to an older major is never credited to a newer one, and no change is counted by two releases.
An unretrievable commit range links nothing and marks the pass incomplete.	A partial attribution is never left behind looking like a complete one.

v1.7

Adds the composite productivity index.

Change	Effect
Four-dimension composite index shipped — Speed, Effectiveness, Quality, AI Leverage, each a quarter of one 0–100 overall score, snapshotted daily over a trailing 30-day window.	The first top-level number GuideMode publishes. No existing measure changes; the index is additive and stored in its own table.
Benchmark inputs normalize to your percentile, falling back to static DORA band midpoints (elite 90 / high 70 / medium 45 / low 20) below the cross-tenant privacy floor.	The index works on a single-tenant install from day one rather than waiting for peer density.
The open-source corpus can never move a tenant’s index. Only percentiles from the cross-tenant population are used; a corpus percentile falls through to the static band.	Public-project review norms cannot be mistaken for, or silently substituted into, a peer comparison.
coverage published alongside every score as the weighted fraction of declared inputs actually present, and deliberately not renormalized.	A score of 70 at coverage 0.25 reads as “the quarter we could measure looks like 70”, not as a confident 70.
Missing inputs excluded and weights renormalized; a dimension with no inputs is explicitly null and propagates as an absence, not a zero.	The same rule as everywhere else on this page, applied where fusing unrelated sources makes silent imputation most tempting.
flow_state_frequency given an explicit 0–100 choice weighting (daily 100 → almost never 0), pinned by test to the survey question bank.	A previously unscored question becomes usable without changing how any existing question is scored.
Full per-input breakdown stored on every snapshot, including nulls for absent inputs and which population each benchmark normalization used.	Recomputing a dimension from its stored breakdown reproduces it exactly — asserted by test.
Index version stamped on every row.	Mixed-methodology data is detectable rather than silently averaged across a weight change.

v1.6

Adds the open-source benchmark corpus.

Change	Effect
Open-source corpus shipped as a third benchmark source. A versioned cohort of public organizations, ingested and scored through the same pipeline as customer data.	New oss_corpus rows in metric_benchmarks, surfaced as an explicitly labelled context band. No existing measure changes.
The corpus band is labelled and caveated, never merged into the cross-tenant one. “vs. open-source corpus” is a separate field from source.	A reader cannot mistake open-source review norms for a peer-group comparison.
The corpus organization is excluded from the cross-tenant population. Same shared eligibility predicate as the demo/seed exclusion.	Public-project data cannot move the number you are told is your peer group.
Cohort fixed per corpus version, stamped on every computed row.	Quarter-over-quarter deltas reflect changes in output, not changes in who was counted.
Only three metric keys are offered for the corpus. The deployment-derived DORA four are unmeasurable on public repositories and are omitted rather than approximated.	No fabricated band where there is no signal.
Representativeness minimum of 5 organizations, distinct from the privacy floor. Applied at read only; corpus rows are always written with an honest count.	Public data is not anonymized against, but a two-project distribution is not shown either.
Corpus contributors never carry a real email address, whatever GitHub publishes.	The corpus holds no personal contact data and can never be email-matched onto a customer user.
Change-effort calibration limitation documented, not tuned. The architecture weights are GuideMode-shaped, so most open-source files score at the default.	Corpus effort distributions are flatter than a matching codebase would give; retuning would need a SCORING_VERSION bump and a full rescore.

v1.5

Adds benchmarks.

Change	Effect
Static DORA bands moved into code. The Elite/High/Medium/Low boundaries are constants, and a parity test pins the docs tables to them.	Every DORA number can be banded from day one, with no cross-tenant density required.
Lead-time bands corrected. The DORA page carried two contradictory band tables; the per-metric one (<1h / 1h–1d / 1d–1w / >1w) is correct and the summary table now agrees.	The summary table previously overstated every band boundary by roughly an order of magnitude.
Cross-tenant percentiles shipped. Ten metric keys, computed monthly, aggregated per organization before the percentile is taken.	New metric_benchmarks table and /api/benchmarks route. No existing measure changes.
Privacy floor of 5 published and enforced twice. Checked at write and again at read, over eligible organizations with a non-null value.	A distribution that thins out stops being served, and its stored row is removed.
Demo/seed organizations excluded from the population. Documented above, with the rejoining rule.	Synthesized-but-plausible data cannot silently move a percentile.
Percentile rank is a performance percentile. Already flipped for lower-is-better metrics, and clamped to [10, 90] outside the stored knots.	Higher always means better, and the tails are not given invented precision.
Absent benchmarks render as absent. source: null where there is neither density nor a static band.	Stops a missing-telemetry badge being mistaken for a poor score.

v1.4

Change	Effect
Discovery outcomes shipped. A declared resolution sits alongside the observed validationStatus; discoveryOutcome reconciles them, declaration first.	New resolution and discoveryOutcome dimensions. validationStatus is unchanged — no existing measure moves.
A validated kill is a success branch. invalidated is counted as a decision reached, not as funnel leakage.	New validatedKillCount, decisionThroughput and killEfficiency. Conversion-rate measures are untouched.
Belief shift published. Paired pre/post confidence probes, measured as absolute movement on the 0–100 scale.	New avgBeliefShift, decisionChangedRate and learningVelocity. Opt-in; null until enabled.
Unpaired probes contribute nothing. A pre with no post is an absence of measurement, not a zero shift.	Stops an unanswered probe reading as “this work changed nothing”.
Decay moved to unrealized insight. insightShelfLifeDays ages validated conclusions nobody has started building.	Makes explicit why CEU’s decay stays confined to mechanical noise.
Discovery value is team-level only. No author dimension on belief shift or kill efficiency.	Closes the obvious Goodhart route on spike volume.

v1.3

Adds outcome confirmation.

Change	Effect
Confirmed Change Value (CCV) shipped for delivery work. CEU multiplied by an outcome factor derived from production survival. Window, factors and revert-detection rules published above.	New confirmedValue, survivalRate, revertRate and confirmationLatency measures. No existing measure changes.
Unknown outcomes are null, not zero. pending and flagged changes have no outcome factor and therefore no confirmed value.	Prevents recent work reading as worthless, and stops the aggregate drifting with how recently the confirmation job ran.
Survival rate excludes in-flight work. Only window-complete changes count in either half of the ratio.	A team that shipped heavily last week no longer appears to have a collapsing survival rate.
SZZ explicitly rejected. Reverts are matched from explicit git/GitHub signals only.	Avoids the documented mislabeling rates of blame-tracing approaches.
Observed engagement replaces inferred effort. Session metrics join onto changes where available.	Null, never zero, where no session was captured — adoption is currently low.

v1.2

Adds the first measure of shipped output.

Change	Effect
Change Effort Units (CEU) shipped. Deterministic per-file scoring of merged pull requests, with work-type classification and a derived KTLO-vs-investment view. Formulas, weights and exclusion rules are published above.	New ChangeValue cube and dashboard. No existing measure changes.
Effort is named as effort. The headline metric is effortScore, not a “value” score. Its factors are the classic defect predictors, so it measures cost and risk.	Sets the contract for CCV, where an outcome factor supplies the value half.
Decay narrowed to mechanical noise. Only pure renames, formatting-only diffs and generated content are discounted. Refactors, self-rewrites and deletions are scored at full weight.	Departs from ETV-style models, which penalise maintenance twice. Deletions score positively.
Architecture documented as a risk weight. Centrality raises the effort estimate, not a claim about worth.	A team that decouples its system does not deflate its own score.

v1.1

Correctness and transparency release.

Change	Effect
Consistent zero point. All 0–100 composites are min-subtracted. v1.0 mixed three conventions: avg × 100/7 mapped a Likert-7 floor to 14.3, avg × 20 mapped a Likert-5 floor to 20, and only the ceiling was correct in every case.	Likert composite values shift down by roughly 4–9 points. Nothing about the underlying responses changed.
Reverse-scored items are inverted. reverseScored was declared in the question JSON but never applied in aggregation.	Scores derived from verification_frequency, cognitive_load, user_control, and tech_debt_impact now move in the correct direction. Previously they were inverted.
Speed Score fixed. v1.0 divided by the count of all assessments (including those that never asked the question) and matched choice values that have never existed in the data. The result was pinned near zero.	speedScore now counts only rows with a recognized answer, and treats “Not sure” as a non-answer rather than as 0. Values increase substantially.
Single shared scoring module. Formulas moved out of inline cube SQL into a single shared scoring module, with the SQL emitters generated from the same constants the tests and this page read.	Cube SQL, unit tests, and this page cannot drift apart.
Duplicated statistics helpers consolidated. Percentile, mean, and trend helpers now have one implementation.	Identical inputs give identical outputs everywhere.

If you are comparing dashboards across the v1.0/v1.1 boundary, treat the composite scores as a new series rather than a continuation of the old one.

v1.0

Original implementation. Formulas lived in cube SQL comments and processor source; superseded by v1.1.

---

References

The research behind the choices on this page, grouped by what each source bears on. Append-only; last reviewed 2026-08-30.

Causal evidence on AI coding impact — bears on Reading AI impact

• Becker et al., Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR 2025. Randomized trial, 16 developers, 246 tasks: 19% slower with AI while self-reporting 20% faster. https://arxiv.org/abs/2507.09089
• Cui, Demirer, Jaffe, Musolff, Peng & Salz, The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers, Management Science 2026. Three RCTs, 4,867 developers, +26% completed tasks; gains concentrated in less-experienced developers. https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535 (preprint: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566)
• AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2× Mandate, 2026. 802 developers, 196k pull requests; throughput 2.09×, reviewer load doubled, revert rate flat; declines causal attribution because adoption was not randomized. https://arxiv.org/abs/2607.01904
• Intuition to Evidence: Measuring AI’s True Impact on Developer Productivity, 2025. Staggered difference-in-differences on adoption cohorts. https://arxiv.org/abs/2509.19708
• Impact of the Availability of ChatGPT on Software Development: A Synthetic Difference in Differences Estimation using GitHub Data, 2024. https://arxiv.org/abs/2406.11046
• Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace, 2024. https://arxiv.org/abs/2410.18334
• The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study, 2026. Self-report; work shifts toward supervising and verifying AI output. https://arxiv.org/abs/2605.23135

Complexity-metric validity — bears on the complexity term

• Complementarity in software code complexity metrics, Journal of Systems and Software 2025. Halstead effort/difficulty correlate with EEG-measured cognitive load at ρ ≈ 0.35; no single metric captures perceived complexity. https://www.sciencedirect.com/science/article/abs/pii/S0164121225003486
• An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability, 2020. https://arxiv.org/abs/2007.12520
• The Correlation among Software Complexity Metrics with Case Study, 2014. Halstead, cyclomatic and LOC are strongly inter-correlated. https://arxiv.org/abs/1408.4523
• Shepperd, A Critique of Cyclomatic Complexity as a Software Metric, Software Engineering Journal 1988.

Refactoring, churn and defects — bears on decay, architecture and CCV

• Zimmermann & Nagappan, Predicting Defects using Network Analysis on Dependency Graphs, ICSE 2008. Centrality predicts defects — why architecture is a risk weight.
• Kim, Zimmermann & Nagappan, A Field Study of Refactoring Challenges and Benefits, FSE 2012. Why refactors are classified, not discounted.
• Rosa et al., Evaluating SZZ Implementations Through a Developer-informed Oracle, ICSE 2021; Herbold et al. on SZZ mislabelling. Why reverts use explicit signals only.
• Tornhill & Borg, Code Red: The Business Impact of Code Quality, 2022. https://arxiv.org/abs/2203.04374
• GitClear, Coding on Copilot / AI code-quality reports, 2024–2025. Rising two-week churn and copy/paste in AI-era code.

LLM-as-judge reliability — gates any future LLM grading of ambiguous files

• Rating Roulette: Self-Inconsistency in LLM-As-A-Judge, Findings of EMNLP 2025. https://aclanthology.org/2025.findings-emnlp.1361.pdf
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models, 2026. https://arxiv.org/abs/2606.19544
• Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0–5, 2026. https://arxiv.org/abs/2601.03444
• Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory, 2026. https://arxiv.org/abs/2602.00521

AI-agent pull requests — context for work type classification and outcome signals

• How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests, 2026. https://arxiv.org/abs/2601.17581
• Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study, 2026. https://arxiv.org/abs/2605.22534
• Quality and Security Signals in AI-Generated Python Refactoring Pull Requests, 2026. https://arxiv.org/abs/2605.21453

Frameworks and incentives

• Forsgren, Storey, Maddila, Zimmermann, Houck & Butler, The SPACE of Developer Productivity, ACM Queue 2021. Why individual rankings are not exposed.
• DORA, Accelerate State of DevOps reports 2024–2025. Industry bands used by the static benchmark fallback.
• Kerr, On the folly of rewarding A, while hoping for B, Academy of Management Journal 1975. Why the fix multiplier is capped.

Comparable methodologies

• Navigara, The 500 OSS Performance Index — Methodology (Engineering Throughput Value). The model CEU is adapted from, and where it diverges: ETV uses LLM classification and function-scope cognitive load, calibrates on a labelled corpus, publishes bootstrap intervals, and is non-additive across work types. https://500.navigara.com/methodology

---

Related

• Surveys & Assessments — the survey and assessment cubes these scores appear in
• Sessions & AI — session telemetry metrics
• DORA Metrics — DevOps performance metrics and industry bands
• SPACE Framework — multidimensional productivity framework
• Analytics Overview — how the analytics system fits together