Introduction to Measurement

levels of measurement

ratio, interval, ordinal, nominal

ratio

absolute zero exists

interval

on a scale, distance is meaningful

ordinal

measuring using ranking, attributes can be ordered

nominal

only named

quality of measurement

reliability + validity

reliability

consistency of measurement

true score thoery

observed score = true ability + random error

measurement erros

random error, systematic error

random error

randomly affect measurement

systematic error

consistently affect measurement

reducing error

pilot test, train interviewees, observers etc., double-check data, use statistical procedures to adjust for measurement error, triangulate

pilot test

feedback from respondents

triangulate

combining multiple independent measures

inter-rater or inter observer reliability

Agreement or correlation between the ratings or codings. Two independent raters or observers of the same phenomenon

test-retest reliability

the same test or measurement, two successive time periods

parallel-forms reliability

two versions of the same test or measure that were constructed in the same way

internal consistency reliability

the degree to which items on the same multi-item instrument are interrelated

internal consistency methods

inter-item correlation, average item-total correlation, split half correlation, Cronbach’s Alpha

construct validity types

translation, criterion-related

translation validity

how well is the idea of measure translated into operationalisation?

types of translation validity

face, content

face validity

“on its face” good translation of the construct

content validity

check organisation against the relevant content domain (traits of your construct)

criterion-related validity

based on its relationship to another independent measure as predicted by your theory

types of criterion-related validity

predictive, concurrent, convergent, discriminant

predictive validity

predict what it theoretically should be able to predict

concurrent validity

able to distinguish between groups it should theoretically be able to

convergent validity

how similar the operationalisation to ones it should theoretically be similar

discriminant validity

how not similar the operationalisation to ones it should theoretically not be similar

threats to construct validity

not defined well construct, mono-operation bias, mono-method bias, interaction of different treatments, interaction of testing and treatment, restricted generalisability across constructs, confounding constructs and levels of constructs

mono-operation bias

when you rely on only a single implementation of your independent variable, cause, program, or treatment

mono-method bias

because you use only a single method of measurement

interaction of different treatments

when participants take part in several programs

interaction of testing and treatment

does the testing make groups more sensitive to treatment?

restricted generalisability across constructs

side effects or negative consequences on other not measured outcomes

confounding constructs and levels of constructs

results found only for e.g. one dosage of drug and not for range of different doses

hypothesis guessing

participants in a study guess the purpose of the study and adjust their responses based on that

evaluation apprehension

many people are anxious about being evaluated - may perform poorly

researcher expectancies

researcher can bias the results both in conscious and unconscious ways

social threats to construct validity

hypothesis guessing, evaluation apprehension, researcher expectancies