Reliability and evidence for score interpretation

Start with the intended interpretation

A measurement produces observations or scores. To evaluate it, ask what someone intends to infer from those scores, for which people and in what setting. A test is not simply “valid” for every conceivable purpose because it has a familiar name or a numerical result. Evidence supports particular interpretations and uses. Reliability concerns consistency under specified conditions. Validity concerns the evidence and reasoning supporting the proposed interpretation of the scores. These ideas are related, but they are not synonyms. Consistent results can still be interpreted incorrectly, and an instrument appropriate for one population or decision may need additional evidence before being used elsewhere.

Different consistency questions

Test–retest reliability concerns consistency across occasions when the underlying attribute is expected to remain sufficiently stable. Inter-rater reliability concerns agreement or consistency across scorers applying a scoring process. Internal consistency concerns relationships among items within an assessment, under assumptions about what they measure. The right question depends on the likely source of variation. If two scorers interpret an essay rubric differently, retesting the same student tomorrow does not directly resolve that scoring problem. If a characteristic changes naturally over time, differences across occasions do not automatically demonstrate a defective instrument. Name the comparison before assigning a reliability label.

Worked example: consistent but shifted

A fictional measuring device reports 104 units each time a stable reference known to be 100 units is measured under the same conditions. The repeated readings are consistent in this small demonstration, but they are shifted relative to the reference. Repetition alone has not established accuracy for the intended quantity. This physical analogy helps separate consistency from justified interpretation, but it is not a complete model of psychological measurement. Psychological constructs may not have a single physical reference standard. Evidence can instead involve content, response processes, relationships with other measures and consequences of the proposed interpretation or use.

Worked example: two scorers and a rubric

Two fictional scorers independently assign scores to the same five short responses using a common rubric. Their pairs are 2 and 2, 3 and 3, 1 and 2, 4 and 4, and 2 and 3. They exactly agree on three of five responses, or 60% in this simple descriptive count. That percentage is not a complete reliability analysis: it ignores features such as the score scale and agreement expected by chance. It does, however, identify the relevant comparison as inter-rater. A useful next editorial step is to inspect the two disagreements and clarify the rubric's interpretation, not to claim that all score uses are valid because most pairs matched.

Avoid turning one statistic into a guarantee

A high consistency coefficient does not prove that an assessment samples the intended content well, measures only the intended construct, treats groups fairly, or supports a particular decision. It also does not certify a person's result as error-free. Every reported statistic belongs to a sample, method and set of conditions. Another misconception is to infer that more items automatically make a test appropriate. Adding many near-duplicate items can increase some consistency measures while narrowing coverage. A defensible assessment needs attention to the intended domain and use, not merely a larger question count or a favorable single number. This matters for evaluating FreeReviewer.ph practice as well: useful learning questions are not a validated licensure predictor.

Sources & further reading