Module 4: Study Design Measures

How To Create a Research Methodology

When formulating methodology, it is critical to consider the types of methods that will most accurately and efficiently answer the research questions. For example, if one wishes to assess the efficacy of an educational intervention, the “change in knowledge” must be measured. This lends itself to a pre-test/post-test methodology in which the researcher will determine the knowledge of study participants on a specific topic prior to the intervention and then again after the educational intervention has been implemented. However, in order to determine the actual impact of an intervention, a pre-test/post-test methodology must always be compared with a control group. For more information about this, please see the section on Assessing Behavioral Changes: The Importance of Having a Baseline For Comparison.

Study Measures

Measures are the items in a research study to which the participant responds. Research measures include survey questions, interview questions, or constructed situations. When constructing interviews and surveys, it is important that the questions directly relate to the research questions. Furthermore, it is important that the surveys and interviews are not extremely time-consuming (ideally within a 20-30 minute limit). Simple questions that avoid ambiguity will lead to more accurate responses, particularly if a translator is being used. Lastly, before creating a survey, a thorough literature review should be conducted to determine if a similar study has already been conducted. If so, previous surveys should be used to yield standardized measures for comparison. Irrespective of the form that these measures take, there are several important design elements that are required to make the study effective.

Study measures should:

• Take into account the characteristics of the participant.

  • Use informal language that a layperson will understand.

  • Be respectful of the cultural context in which the participant has shaped his or her worldview.

• Provide neither too much nor too little information. Too much information can be an unnecessary distraction, while too little information leads to ambiguity and potential misinterpretation of the study measure.

• Be brief and specific.

• Avoid negations, as they can lead to mistakes and can be difficult to understand.

• Avoid double-barreled questions (questions that ask two questions in one, such as “Do you support the government’s decision to cut spending to police training and after school programs?”). If a participant would answer “yes” to one part but “no” to another, requesting a single answer of “yes or no” to two questions invalidates the measure.

• Use multiple questions to assess the same construct. For example, simply asking a person if they feel “good” about themselves as a measure of self-esteem is not specific and leaves room for individual interpretation. Instead, asking that person several questions about body image, self-worth, and self-evaluation can help paint a better picture of how they are really feeling.(1),(2),(3)

Self-Report

Self-report measures are essential in situations where the researcher is asking about a participant’s self-concept or seeking to study the specificities of a participant’s experience. Self-reporting may also be useful for logistical reasons, as it is often the simplest research method to implement and requires the fewest resources. Self-report measures, whether in survey form or in interview form, are susceptible to bias and therefore must be designed carefully by the researcher. The researcher also must be cautious about what kinds of conclusions are drawn from these measures. Some potential issues with self-report are:

Social desirability bias: Participants are usually uncomfortable or unwilling to share information that does not reflect well on them in their social environment, even if they know their responses are entirely anonymous. For example, participants may understate or overstate the extent to which they experience a certain feeling, depending on how socially appropriate or desirable they believe that feeling to be. Researchers must do their best to make it abundantly clear that anonymity will be preserved for the participant, and honesty must be encouraged. Researchers should also lead with less intimidating questions to make the participant feel more comfortable before asking anything that might be more difficult to answer honestly. Another option is to structure the question in such a way as to normalize the behavior: “As you know, many people do X… To what extent do you do X?”

Self-evaluation biases: Participants will sometimes bend their answers on self-report measures to better reflect how they “think they should be” rather than how they actually are. This is similar to the social desirability bias, but is more difficult to overcome because anonymity is not the issue. Instead, bias results from the participant’s evaluation of him or herself. The researcher’s best course of action is to encourage honesty and normalize the behavior or feeling as reviewed above.

Forgetfulness: Sometimes researchers ask participants about their past experiences or feelings without considering the fact that human memory is very plastic. People’s recollections may be inaccurate, and it is important for a researcher to consider this when designing study measures.(4)(5)

Footnotes

(1)  Eibach, R.  “Scale Questions: Simple Questions, Complex Answers”  Lecture at Yale University 10/2/08.

(2) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition.  Wadsworth Publishing (February 27, 2006).

(3) Trochim, W. M. K.  “Survey Research” Research Methods Knowledge Base 2nd Edition.

(4) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition.  Wadsworth Publishing (February 27, 2006).

(5) Eibach, R.  “Scale Questions: Simple Questions, Complex Answers”  Lecture at Yale University 10/2/08.