Item types
...
Optimal performance
There is a single correct answer,
Invertory
There are no correct answers, but responses that correspond with a higher standing on the construct, get more points
Pair comparisions
2 options: a you a or b
semantic differential
5 point scale and persion
ranking
make a list, the amount of points assigned is based upon the deviation from expert responses
point allocation
give points to various things, problems: people can't add...
likert items anchors problem 1
them meanings of many anchors are ambiguous and mean different things in terms of frequency to different people. For example, how do you know if occasionally means the same to me as it does to another person
likert items anchors problem 1 solution
use anchors with clearer meaning so define it
Likert items anchors problem 2
for interval level measurement, we must have equal intervals between our anchors.
Likert items anchors problem 2 solution 2
pick anchors that do have eqaul interval via a stimulus centered rating study
composite variables
they're everywhere
two types of composite variables1.2
1. total test score e.g. ACT2. subtest scores e.g. WAIS
Notes about composites
Composites can be weighted or unweighted
item scoring
the result of any scheme for assigning numbers to an item
two types of item scoring- the result of any schme for assigning numbers to an item1.2.
1. dichotomous- 2. multi-level
1. dichotomous-
scores by only a 2-level set of numbers ( right/ wrong, agree/disagress/ endorsed/ not endorsed)even multiple choice items are dichotomous! because you only get a point
2. multi-level
multi-level: categorical meaning nominal2. ordered categories - continuous more than 2 categories like likert format or partial credit
p-value
percent answering item in keyed direction
p-value describes
the percent of people who answered the item in the keyed directione.g., if 10 of 50 peole answered item 1 correctly, then item one's p-value is .20
p-value is equal to
p-value is equal to item mean so high p-valeu -easy itemlow pvalue- hard item
composite score properties- mean:How does the mean of a sum ( being a types of composite score)related to the components of the composite( - the items?
The mean of a sum is equal to the sum of the meansmean of a sum = S (item mean)
why does this make sense
If you have a lot of difficult items (low p-values), then the total score across those items is going to be low for most people.
item variance
item variance is good because that means that the item discriminates
dichotomous item variance
p*qq= 1-pThis is the same as variance with the n demoninator
non-dichotomous item variance
or continuous itemsvariance of the sum = sum of item variances + 2 *(sum of covariances between each item paircovariance = r(sx)(sy) so more r = more c
Thus composite scores have a lot of variance when
Thus composite scores have a lot of variance when 1. The. Test items have a lot of variance2. The test items are highly correlated it's each other
Is it possible to have too much of a good thing with composite score variance?
See page yes< Final ThoughtsIt would appear that the ideal test would be composed of a large number of items average in difficulty and correlated perfectly with each other.
A negative correlation
A negative correlation masks the item variance< result in a zero composite
Ctt based item anaylsis
SeperTinf the wheat from the chaff
Item anaylsis
Selection from a pool to achieve a given purpose
Item
any individual question that produces a single response
Test
Collection of items assembled for a given purposed
Item anaylsis aproaches12AB3
Item anaylsis approaches:1. Difficulty2. Discrimination Internal consistencyEmpirical keying3. Content based
1. Difficulty
1. Difficulty based - a good item is one that matches the ability of the test taker
Discrimination based
2. Discrimination based - a good item is one that has a strong correlation. With some variable. Which variable?
3. Content
A good set of variances are a representative sample of the content domain