Inter-coder Agreement

Last updated 2026-07-04

Agreement tracks one thing: how often you keep the AI's suggestion versus reject it, broken down by code. When you're rejecting a code far more than you're keeping it, that's not you making a mistake — it's a sign the AI's instructions for that code are unclear. Agreement catches that pattern automatically and asks you to fix the definition, right there, before it wastes more of your review time.

This measures you-vs-AI, not coder-vs-coder. Despite the sidebar label, the in-app screen is titled simply Agreement, and its own subtitle is explicit: “How often you accept Paideias's suggestions, by code.” It's an AI-agreement / codebook-quality signal, not a classical two-human inter-rater reliability score — though it uses the same underlying idea (agreement rate, Cohen's κ) that real inter-coder work depends on, and it's exactly the kind of check you'd want to run before handing your codebook to a second human coder.

What this screen is for, and the redefine loop

Below is a real recording: opening Agreement on a project where one code — “Ability” — has been over-applied, seeing the AI flag itself for that code, clicking through to fix the definition, and watching the fix save.

paideias.org

Recorded, not staged

The AI asks to be corrected

“Ability” sits at 40% agreement after 3 rejections — below the 70% threshold — so a callout appears: “I might be misunderstanding ‘Ability’. Would you like to rewrite the definition?” Clicking Rewrite definition jumps to the Codebook with that code's definition already open for editing; typing a tighter rule and clicking Save definition closes the loop.

The three rejected suggestions in this example were real edge cases: one was a general statement about how trust accumulates over time (not about a specific teammate's skill), one was about the participant not having enough signal yet (a timing issue, not a competence judgment), and one was the participant doubting their own ability to judge someone quickly. All three technically mention ability-adjacent language, which is exactly why the AI kept reaching for the code — the old definition didn't rule them out. The new definition does, explicitly.

Reading the stat strip

Four numbers summarize the whole project at a glance.

The κ shown is an estimate, not a textbook Cohen's kappa. True Cohen's κ needs a full contingency table from two independent raters coding the same items. Paideias only has one side of that (you, deciding on the AI's suggestions), so it derives an approximate κ from your agreement rate — close enough to communicate reliability at a glance, but don't cite the exact number in a methods section without computing real inter-rater κ separately if you bring on a second human coder.

How a code gets flagged

A code is flagged the moment two conditions are both true: it has at least 3 decisions (approve + reject + modify combined), and its agreement rate is below 70%. Both conditions matter — a single early rejection on a brand-new code won't trigger anything; it needs a real pattern.

Rewrite definition — jumps straight to the Codebook screen, selects the flagged code, and opens its definition textarea already in edit mode, cursor ready. No manual searching for the right code.
Dismiss — hides the callout for this session without changing the definition. Use it when the low agreement is expected (e.g., you're intentionally being strict early on) rather than a real definition problem.

Dismissing doesn't fix the underlying rate. If you keep rejecting suggestions for that code afterward, the callout comes right back next time you open Agreement — dismissal only clears it for the current session, not permanently.

The per-code table

Below the stat strip and any flagged callouts, every code with at least one decision gets a row: an agreement bar, the exact percentage, and raw accept/reject/modify counts. Flagged rows get a NEEDS DEFINITION badge and a pale amber background so they stand out while scrolling.

Codes with zero decisions yet don't appear in this table at all — there's nothing to measure until at least one suggestion for that code has been approved, rejected, or modified.

Exporting a report

Export report in the top-right downloads a .csv snapshot of the current table: code name, agreement rate, and the accept/reject/modify counts behind it.

How this actually helps you

It's easy to read Agreement as just another dashboard. In practice it changes three things about how a coding project runs:

It catches codebook problems while they're still cheap to fix

Without this screen, a vague definition just quietly generates bad suggestions document after document — you reject them one at a time in Focus Mode and the pattern is invisible unless you're keeping a mental tally. Agreement makes the pattern visible after as few as 3 decisions, so you fix the definition after one document instead of after ten.

It's a rehearsal for real inter-coder work

If this project will eventually have a second human coder, a tight, unambiguous codebook is the single biggest predictor of good human-to-human agreement. Getting your AI-agreement rate up first — by writing definitions precise enough that even the AI stops guessing wrong — does a lot of that hardening for free, before a second person's time is on the line.

It turns "the AI is bad at this code" into a concrete edit

The flagged callout doesn't just say a code has low agreement — it quotes the exact rejection count and rate, and hands you a pre-filled, focused editing view instead of a vague prompt to "go check the codebook." That's the difference between knowing something is wrong and knowing exactly what to change.

It's cumulative, not a one-time report

Because Agreement reads live from every document's suggestions, the numbers update as you review more documents — a code that looked fine after document one can still get flagged after document three if a new pattern of disagreement shows up. Re-check it periodically through a project, not just once at the end.

You're ready to tighten your codebook. Once flagged codes are cleared, move on to Retrieval to review the passages behind any code, Visualisations to see coding patterns graphically, or Export to download the full project.

← Retrieval Visualisations →