What is Reliability?
The ability to repeat or reproduce the measurements or observations that are made during a study.
What is Reliability?
a. A scale or instrument that consistently measures some phenomenon with accuracy is said to be reliable.
b. When and instrument in reliable it has a certain amount of predictability, then administering it to similiar groups under comparable conditions should yield similiar measurements.
What is Internal consistency?
a. Individual items correlate well with the full scale.
b. If items do not correlate well, the items have very little in commom.
c. Known as coefficient alpha or Cronbach's alpha.
d. It will also be less than 1.
e. The closer to 1 the more reliable the scale, and the better the items work well together to measure the same concept.
What is Split-half technique?
Sometimes researches measure internal consistency by dividing their longer scales in half (either top versus bottom or even versus odd items) and examining how well the two halves correlate with each other.
What is Parallel or Alternate Versions of the scale?
a. The researcher administers another scale and adminsters both forms to similiar groups.
b. The researcher hopes that both versions will correlate with each other -- the higher the correlation coefficient, the stronger the reliability.
What is Interobserver Reliability?
a. The amount of agreement between to observers.
b. Expressed as a percentage of agreement or a correlation.
c. The higher the reliability, the better.
d. Generally, one would hope to achieve agreement at least 75% to 805 of the time.
What is Interrater Reliability?
a. Should be computed when more than one person is involved in rating clients' behavior.
b. Is computed to determine the percentage of time the raters are in agreement.
c. Low correlation = criteria may not be well defined and subjective judgement could have a biasing effect.
d. High correlation = If .70 or above the researcher has evidence that her rating has succeeded in providing sufficiently reliable measurements.
What is the best way to establish interraer or interobserver reliability?
Probably it's best to do this as a side study or pilot study. And, if your study goes on for a long time, you may want to reestablish inter-rater reliability from time to time to assure that your raters aren't changing.
What are the two major ways to estimate interrater/interobserver reliability?
1. If your measurement consists of categories -- the raters are checking off which category each observation falls in -- you can calculate the percent of agreement between the raters. For instance, let's say you had 100 observations that were being rated by two raters. For each observation, the rater could check one of three categories. Imagine that on 86 of the 100 observations the raters checked the same category. In this case, the percent of agreement would be 86%. OK, it's a crude measure, but it does give an idea of how much agreement exists, and it works no matter how many categories are used for each observation.
2. The other major way to estimate inter-rater reliability is appropriate when the measure is a continuous one. There, all you need to do is calculate the correlation between the ratings of the two observers. For instance, they might be rating the overall level of activity in a classroom on a 1-to-7 scale. You could have them give their rating at regular time intervals (e.g., every 30 seconds). The correlation between these ratings would give you an estimate of the reliability or consistency between the raters.
What is test-retest reliability?
Used to determine stability, researchers will often administer a scale to the same group on more than one occasion to see, for example, if persons with high test scores at the first administration maintain high scores at the second administration.
What is Validity?
An instument is said to be valid when it measures the constuct is was designed to measure.
Example: an intelligence test should measure intelligence.
What is Internal Validity?
a. It is concerned with whether the intervention was the cause of the observed change.
b. Good internal validity = one can place more confidence in the finding that it really was the intervention and not some other extraneous variable (e.g., nutritional supplements or an exercise program) that made the difference.
What are Threat to Internal Validity of Single-System Research Designs?
1. History2. Maturation3. Instrumentation4. Testing 5. Mortality6. Statistical Regression7. Contamination/Diffusion
What is the definition of the internal threat of history?
Anything that might happen during the study period beyond the investigator's control which might have had an effect or influenced.
What is the definition of the internal threat of maturation?
The change occurred because the individual grew older, developed, or became more experienced with the passing of time.
What is the definition of the internal threat of instrumentation?
The measurement process changed during the course of the study.
What is the definition of the internal threat of testing?
Simply having the individual complete the tests or questionnaires, tally behavior, or reflect about it had an effect by sensitizing him or her to the occurrence of the behavior.
The behavior changed because one knows that one's behavior is being monitored.
What is the definition of the internal threat of mortality?
Some clients dropped out of the study with the result that the group at the end didn't closely resemble the group at the beginning.
What is the definition of the internal threat of statistical regression?
Extreme scores at the beginning of the study which might naturally be expected to be less extreme later -- even if there had been no intervention.
What is the definition of the internal threat of Contamination/Diffusion?
The possibility that the clients being studied benefited from learning from clients enrolled in other programs or from other sources.
What is External Validity?
a. An intervention that produces strong rates of change across various behaviors or with different clients is showing generalizibility.
b. These designs provide evidence that the intervention was responsible for the improvement.
c. Assumes that other social workers who employ the same intervention should also expect to find the same beneficial effects with their clients.
External Validity study note
Think generalizability or representativeness of a study.
What are major threats to External Validity?
1. Reactive/interactive effect of testing.2. Interaction effects of selection biases and any research stimulus.3. Reactive effects of arrangements.4. Multiple-treatment interaction.
What is the External Threat of Reactive (or interactive) effect of testing?
When a pretest affects the repondents' sensitivity or awareness to important variables associated with the study.
What is the External Threat of Interaction effects of selections biases and any research stimulus?
When there are problems getting a random sample.
It is not a problem in experiments with random assignment.
What is the External Threat of Reactive Effects of Arrangements?
The experiment is conducted in a setting that is different from the subjects usual experience.
Subject's behavior may change because of the different environment.
Subjects may be more productive, wary, or nervous.
What is the External Threat of Multiple-treament Interaction?
When there is more than one intervention.
The researcher needs to be sure that the same timing, sequence, or order is followed for each subject.
What are the ways to discuss the validity of an instrument/scale?
1. Content validity2. Criterion validity a. concurrent approaches b. predictive approaches3. Construct validity a. convergent b. discriminant c. factorial (structural)
What is Content validity?
Instrument needs to sample from the entire range of the concept that is was designed to measure.
Example: A scale measuring anxiety in adolescents may not have content validity if if it did not include behavior such a nailbiting, crying, stomachaches, and sleeplessness.
What is Criterion Validity?
The scale's ability to correlate positively with some external criterion assumed to be and indicator of the attitude, trait, or behavior being measured.
What are the Two approaches or subtypes of Criterion Validity?
1. Concurrent Validity - administering your scale simultaneously along with another scale that has documented validity to the same subjects. If the two scales correlate well in the direction expected, then your scale has demonstrated one form of concurrent validity.
2. Predictive Validity - When scores on a test or scale predict future behavior or attitudes.
Example: The Drug Attitude Questionnaire would have predicted validity if, years later, you find that within your study group of middle school students, those who had prodrug attitudes were suspended from school or arrested for drug possession, while those with antidrug attitudes were not suspended or arrested for drug possession.
What is Construct Validity?
An overarching or fundamental type of validity on which other types depend.
It is not a specific procedure, but a collection of evidence that allows a researcher to see patterns in the way instrument performs along expected theoretical lines.
It is the ability of an instrument to distinguish among individuals who have a lot of, some of, or none of the characteristic being measured that make the instrument useful.
What are the three approaches or subtypes of Construct Validity?
1. Convergent Validity - Obtained when theoretically relevant variables demonstrate a relationship with the measure.
Example: An instrument that you developed to measure children's self-report of fear of playground bullies could, for instance, be correlated with teachers' or parents' rankings of children's fear.
2. Discriminant Validity - The researcher hope to find no relationship between the instument's measurements and variables that should not, in theory, be related to the contruct.
Example: Self-report instuments of children who are fearful of playground bullies should not correlate with other self-reports indicating that these same children are best friends with the bullies.
3. Factorial (structural) Validity - When a large number of items have been created or can be drawn on to compose a scale, factor analysis can be used to reduce the number of items to a smaller group that are statistically related.