Validity and Reliability

Validity and Reliability

Chapter 1 of Evidence-Based Decision-Making introduces validity and reliability as separate questions about measurement. Consider political scientists estimating the chance of a politically destabilizing event in a particular region over the next 12 months. They first need to define what counts as destabilization. They then need observable indicators, such as anti-government demonstrations, and evidence that those indicators help measure the concept.

These tasks require both (1) a clear definition of the concept and (2) a defensible connection between that concept and its observable indicators. A latent construct is something we cannot observe directly, such as anxiety or political instability. We infer it from responses, behavior, or events. Giving an indicator a name does not establish that it measures the intended construct.

Validity concerns the evidence for interpreting and using a measure as intended. Different forms of validity address different parts of that argument. The following grouping separates questions about a concept’s coverage and distinctness from questions about relationships among measures.

1. Conceptual coverage and distinctness

• Face validity: Does the measure appear, on inspection, to capture what it claims to capture? This is a preliminary judgment, not strong evidence by itself.
• Content validity: Does the measure cover the relevant parts of the concept, rather than leaving important aspects out?
• Discriminant validity: Can the measure distinguish its intended construct from other constructs? Distinct concepts may still be related; they need not be wholly unrelated.

2. Relationships with other measures and criteria

• Convergent validity: Does the measure relate to other measures in ways the theory predicts?
• Predictive validity: Does it predict a relevant outcome measured later?
• Concurrent validity: Does it agree with a relevant criterion measured at roughly the same time? This concerns agreement with that criterion, rather than stability over time.

Reliability concerns consistency under comparable measurement conditions. If the underlying construct has not changed, repeated measurements should give similar results. In classical test theory, reliability is the ratio of true-score variance to total observed-score variance. It is not the ratio of a true score to an observed score. Since true scores are not directly known, reliability must be estimated from a suitable pattern of repeated measurements or related items.

1. Internal reliability

• Split-half method: Divide a multi-item measure into halves and compare the resulting scores. Agreement helps assess whether the items measure consistently; it does not require every item to contribute equally.

2. Reliability across occasions or raters

• Test-retest: Does the measure give similar results on different occasions when the underlying construct is expected to remain stable?
• Inter-rater: Do different observers or scorers reach similar results when assessing the same cases?

A consistently recorded protest count might be reliable yet provide an incomplete measure of political instability. Conversely, inconsistent recording can weaken an otherwise sensible indicator. When assessing evidence, ask both whether the measure captures the intended concept and whether the measurement process is consistent.