Sign In

Pilot Testing

Overview

Source: Laboratories of Gary Lewandowski, Dave Strohmetz, and Natalie Ciarocco—Monmouth University

In any experiment researchers have the challenge of creating experiences for participants that are consistent (i.e., reliable) and authentic (i.e., valid). Yet there are many ways to manipulate any one variable. For example, if you want participants to feel sad, you can have them think of their own sad memory, watch a sad video, or read a sad story.

Researchers must find the best way to operationalize a psychological construct in order to produce the most effective manipulation possible. Often, before running the main study, researchers will pilot test (i.e., try out) their manipulations to check their effectiveness.

This video demonstrates how to operationalize the same independent variable (acute stress) in three different ways. Specifically, this study seeks to identify the best sound (static, ticking clock, or crying baby) to play during a difficult task (solving complex math problems) to optimally manipulate stress.

Psychological studies often use higher sample sizes than studies in other sciences. A large number of participants helps to ensure that the population under study is better represented and the margin of error accompanied by studying human behavior is sufficiently accounted for. In this video, we demonstrate this experiment using just three participants, one for each condition. However, as represented in the results, we used a total of 120 (40 for each condition) participants to reach the experiment’s conclusions reflected in the Results section.

Procedure

1. Define key variables.

  1. Create an operational definition (i.e., a clear description of exactly what a researcher means by a concept) of stressful sound.
    1. For the purposes of this experiment, a stressful sound is any noise that creates a feeling of tension, immediacy, or anxiety within participants.
      1. This will be manipulated through three different sounds: static, ticking clock, and a crying baby.
  2. Create an operational definition (i.e., a clear description of exactly what a researcher means by a concept) of acute stress.
    1. For purposes of this experiment, acute stress is defined as the stress or feeling of tension and strain resulting from recent demands or pressures.
      1. In order to measure this accurately, ask participants about their own stress levels using a straightforward question.

2. Conduct the study.

  1. Meet the student/participant at the lab.
  2. Provide participant with informed consent, a brief description of the research (concentration on a task), a sense of the procedure, an indication of potential risks/benefits, the right of withdrawal at any time, and a manner to get help if they experience discomfort.
  3. Run the static condition.
    1. Tell the participant “I’m going to give you a series of math problems that should be easy to solve. Your job is to complete as many as possible in the 2-min time limit. Please try to concentrate and ignore any sounds you may hear.”
    2. Give the participant the math problem sheet (Figure 1), start the timer (set for 2 min), play the static sound, and say “you may start.”
      Figure 1
      Figure 1: Math task. Examples of math problems given to participants
  4.  Give the participant the dependent variable.
    1. Give the participant a measure that asks him/her to indicate how he/she currently feels (Figure 2). The item stressed appears embedded within several other distractor items (i.e., items not related to the present study but included to make the true purpose of the study less obvious). Feelings will be rated on a 1-7 scale, ranging from ‘not at all’ to ‘very much.’
      Figure 2
      Figure 2: Survey on feelings. Participants rates their feelings on a scale from 1-7 (not at all to very much)

3. Debrief the participant.

  1.  Tell the participant the nature of the study.
    1. “Thank you for participating. In this study I was trying to determine what type of sound led participants to experience the most stress. There were three conditions. Everyone worked on the same math problems for two min, but each group did so while hearing static, a ticking clock, or a crying baby. We hypothesized that the group who listened to the crying baby would report the most acute stress.”
  2. Explain explicitly why deception was necessary for the experiment.
    1. “We want to tell you about the deception we used in this study. We used deception by telling participants that the study was about concentration, which wasn’t true, but we didn’t want participants knowing that the study was actually about stress because it may have led to unnaturally increasing stress levels. We also indicated that the math problems were easy, when in fact they were not. We chose difficult problems to increase demands on participants, which generally creates a sense of stress. In both cases, deception was necessary because we wanted to get participants’ natural reactions. If participants were to know the true reasoning and hypothesis behind the study, they may perform in an unnatural way by trying to purposefully disprove the experimenter’s hypothesis. Because of the nature of the deception, it is quite natural for participants to not realize that they were being deceived.”

4. Run steps 2 & 3 above two additional times—once for ticking clock and once for crying baby.

  1. Each condition has a unique participant with everything being exactly the same except for the sound played.

Results

The researcher used 40 participants per condition, and as a result, collected data from 120 participants overall. Numbers above reflect the mean reported stress levels that participants indicated on the 1-7 scale for the stressed item in each condition. This multi-group experiment showed how researchers can operationalize the same construct in multiple ways.

A large number of participants is necessary to ensure that the results are reliable. If this research were conducted using just a few participants, it is likely that the results would have been much different and not reflective of the greater population. 

After collecting data from 120 people, an analysis of variance (ANOVA) comparing the static, ticking clock, and crying baby conditions was performed to see how they influenced stress level. As shown in Figure 3, the crying baby condition reported the most stress as hypothesized. 

Figure 3
Figure 3: The effects of different noises on stress levels. Shown are the average stress levels reported by condition.

Application and Summary

The use of a pilot test helps researchers determine the most effective way to manipulate stress. With this knowledge, researchers can use the best manipulation in their future study.

For example, researchers manipulated stress by having participants do easy or difficult math problems to determine how stress influenced relationship behaviors.1 The results indicated that those under stress were more likely to pay attention to alternate partners and were less likely to give their own partner compliments.

Another study of stress used an entirely different method for manipulating stress.2 In this study, researchers induced stress by having participants immerse their arm in cold water to see how stress influenced long-term memory. Results indicated that exposure to stress led to worse performance on long-term memory tasks. 

References

  1. Lewandowski, G. W., Jr., Mattingly, B. A., & Pedreiro, A. Under pressure: The effects of stress on positive and negative relationship behaviors. Journal of Social Psychology. 154, 463-473. doi: 10.1080/00224545.2014.933162 (2014).
  2. Trammell, J. P., & Clore, G. L. Does stress enhance or impair memory consolidation? Cognition and Emotion. 28 (2), 361-374. doi:10.1080/02699931.2013.822346 (2014).

Tags

Pilot TestingPreliminary AnalysisPilot StudyExperimental ProcedurePsychological ExperimentsManipulation Of VariablesPilot Test EffectivenessSound ManipulationAcute Stress InducersOperational Definitions

Skip to...

0:00

Overview

1:16

Experimental Design

3:08

Running the Study

4:57

Results

5:45

Applications

6:46

Summary

Videos from this collection:

article

Now Playing

Pilot Testing

Experimental Psychology

10.3K Views

article

From Theory to Design: The Role of Creativity in Designing Experiments

Experimental Psychology

18.3K Views

article

Ethics in Psychology Research

Experimental Psychology

29.1K Views

article

Realism in Experimentation

Experimental Psychology

8.3K Views

article

Perspectives on Experimental Psychology

Experimental Psychology

5.6K Views

article

Observational Research

Experimental Psychology

13.0K Views

article

The Simple Experiment: Two-group Design

Experimental Psychology

73.7K Views

article

The Multi-group Experiment

Experimental Psychology

22.8K Views

article

Within-subjects Repeated-measures Design

Experimental Psychology

23.0K Views

article

The Factorial Experiment

Experimental Psychology

73.2K Views

article

Self-report vs. Behavioral Measures of Recycling

Experimental Psychology

11.8K Views

article

Reliability in Psychology Experiments

Experimental Psychology

8.5K Views

article

Placebos in Research

Experimental Psychology

11.6K Views

article

Manipulating an Independent Variable through Embodiment

Experimental Psychology

8.5K Views

article

Experimentation using a Confederate

Experimental Psychology

17.8K Views

JoVE Logo

Privacy

Terms of Use

Policies

Research

Education

ABOUT JoVE

Copyright © 2025 MyJoVE Corporation. All rights reserved