Brand Equity Validation Tool

Sampling strategy · Expert agreement metrics · All parameters auto-determined from corpus · Use CSV for large files (>20 MB)

What would you like to do?

Step 1

Review Corpus Manager

Upload your annotated dataset. All sampling parameters are auto-determined from corpus characteristics. The random seed is the only required input. Produces a reproducible sample + expert template.

Step 2

Expert annotation

Load the expert template and annotate reviews one by one — blind to LLM output. Sessions are saved to the server and can be resumed at any time from any device on this network.

Step 3

Compute agreement metrics

Upload the LLM reference file and one or more expert annotation files or saved sessions. The appropriate metric is auto-selected per dimension. Supports multi-expert comparison.

Datasets Load one or more files · CSV recommended for >20 MB

Add dataset
XLSX or CSV

Stratification columns Optional (each column added ensures proportional representation of its values in the sample, e.g. company, star rating, category, country; columns with many distinct values can optionally be grouped into size bins before adding them)

Language filter Optional (select a column then auto-detect the values present)

Exclusion filters Optional (rows matching these conditions are excluded)

Dimensions Auto-detected from columns matching {name}_n_mentions + {name}_sentiment, or manually configured

For each dimension you can choose which associated columns to include in the sample output. n_mentions and sentiment are always included (used for sampling and metrics).

Or map a custom dimension manually:

n_mentions column

sentiment column

Base sample size Auto-computed from this dataset (editable). Only used when no stratification columns are selected (otherwise Phase 2's quotas determine the sample size and composition instead — see the info icon for details)

Row identifier Optional (used only for the SHA-256 reproducibility checksum)

Unique row ID Auto-detected if present

Additional columns to include in sample output Non-dimension columns (e.g. review text the expert needs to read)

Run analysis in step 2 first.

Complete steps 1–3 first.

How this works

Step 1 — Upload files. Upload the LLM annotations file and one or more expert annotation files. Each expert file must have the same column structure as the LLM file, with the same review identifiers and filled-in {dim}_n_mentions / {dim}_sentiment columns.

Step 2 — Metrics are auto-selected per dimension based on the sentiment category structure in the LLM file: binary → Cohen"s κ · 3 ordered categories → Krippendorff's α ordinal · 4+ unordered categories → Krippendorff"s α nominal.

Step 3 — With multiple experts, all pairwise comparisons are computed: LLM vs each expert, and expert vs expert (human baseline). The human baseline tells you how close LLM performance is to human-level agreement.

Upload files

LLM annotations file (.xlsx or .csv)

LLM output file
The full dataset or just the validation sample — both work. Must contain {dim}_n_mentions and {dim}_sentiment columns.

Expert annotation files — one per expert

Add expert file
Same structure as LLM file. Expert fills in {dim}_n_mentions and {dim}_sentiment columns for each review.

Selected metrics per dimension

Agreement results

Internal consistency (LLM)

Internal consistency (Experts)

Test-retest reliability

Agreement between two independent annotation attempts by the same expert on the same sample — distinct from within-session consistency above (fragment-level self-contradiction) and from Inter-rater Agreement (different people). Only shown for experts with more than one loaded attempt.

Textual interpretation

Graphical interpretation

Intraclass Correlation Coefficient (two-way random, absolute agreement) on the number of positive/negative/neutral fragments per annotator pair, conditional on both annotators having the dimension active. Thresholds: <0.50 Poor · 0.50–0.74 Moderate · 0.75–0.89 Good · ≥0.90 Excellent (Koo & Mae, 2016).

Convergent validity check: relationship between LLM sentiment annotation and reviewer star rating. A high correlation reflects consistency between the reviewer's numeric rating and the LLM's semantic annotation — not a causal claim.

Load or create review session

Sessions are saved on the server and can be resumed at any time. Each expert should use a unique session ID.

Your name (expert identifier)

Sample Dataset (saved on server)

Password (to protect the review)

Load a session in step 1 first.

Complete annotation in step 2 first.