You can also measure intra-rater reliability, whereby you correlate multiple scores from one observer. Experts rating the severity of a problemįor example, we found that the average inter-rater reliability of usability experts rating the severity of usability problems was r =.Evaluators identifying interface problems.Kappa and the correlation coefficient are two common measures of inter-rater reliability. Where there’s judgment there’s disagreement.Įven highly trained experts disagree among themselves when observing the same phenomenon. The extent to which raters or observers respond the same way to a given phenomenon is one measure of reliability. These four methods are the most common ways of measuring reliability for any empirical method or metric.

But don’t let bad memories of testing allow you to dismiss their relevance to measuring the customer experience.

Here are the four most common ways of measuring reliability for any empirical method or metric:īecause reliability comes from a history in educational measurement (think standardized tests), many of the terms we use to assess reliability come from the testing lexicon. In fact, before you can establish validity, you need to establish reliability. Reliability is a measure of the consistency of a metric or a method.Įvery metric or method we use, including things like methods for uncovering usability problems in an interface and expert judgment, must be assessed for reliability.

Reliability and validity of a measure