Quantitative Research Designs PDF

CHAPTER SEVEN Quantitative Research Designs Chapter Checklist After reading this chapter, you should be able to: 6. Conduct manipulation checks of independent variables. 1. Select and develop the appropriate research design for yo...

CHAPTER SEVEN Quantitative Research Designs Chapter Checklist After reading this chapter, you should be able to: 6. Conduct manipulation checks of independent variables. 1. Select and develop the appropriate research design for your hypotheses or research 7. Interpret findings from experimental and quasi- questions. experimental designs with respect to cause– effect relationships. 2. Understand the strengths and limitations of each design form as it relates to research 8. Appropriately interpret findings from findings, and argue for your design choices. descriptive research designs. 3. Explain the benefits of experimental forms over 9. Consider the appropriateness of using online quasi-experimental and descriptive forms. survey software. 4. Facilitate appropriate random assignment of 10. Develop a research protocol to limit researcher participants to treatment and control groups. effects and procedural bias when conducting research studies. 5. Manipulate independent variables according to their theoretical foundation.  THE EXPERIMENTAL FRAMEWORK 123 There are three types of quantitative research design: the public speaking example, one treatment group experimental forms, quasi-experimental forms, and would receive the same information in an interactive descriptive forms. These forms differ in fundamental face-to-face lecture on public speaking; a second treat- ways on two characteristics: manipulation of indepen- ment group would receive the same information in an dent variables and random assignment of participants online self-guided study of public speaking. to treatments or conditions (Pedhazur & Schmelkin, Other groups are labeled as experimental groups, 1991). The first characteristic on which quantitative or treatment groups, with each group receiving a differ- research designs differ—manipulation of independent ent treatment or level of the independent variable. This variables—occurs when the researcher intentionally characteristic is absent from both quasi-experimental varies or changes how the independent variable is and descriptive forms. Figure 7.1 demonstrates how presented to participants. This fundamental charac- these two characteristics differ for the three quantita- teristic must be satisfied to locate a research study in tive research designs. the classic experimental framework. Manipulation of Initially developed for study in the physical sciences, ­independent variables also occurs in quasi-experimental experimental forms can be found in all disciplines research designs but is absent from descriptive forms. ­related to communication—education, management, The second characteristic on which quantitative psychology, and sociology. Due to their widespread research designs differ—random assignment of par- acceptance, the essential characteristics of the experi- ticipants to treatments or conditions—is unique to mental framework have become the standard by which ­experimental forms. After being selected to participate quasi-experimental and descriptive forms of research are in the experiment, the researcher randomly assigns evaluated. All three types of quantitative research design individuals to one of at least two groups. One group are explored, and relevant communication ­examples are is the control group; this group serves as a baseline presented in this chapter. against which the treatment groups are evaluated. The control group can be used in two ways. First, the con- trol group can receive no treatment. For example, in THE EXPERIMENTAL FRAMEWORK a study evaluating the effectiveness of public speaking instruction, the control group would not receive any When researchers are curious about causes, they often type of instruction. Second, the control group can re- turn to experimental research, which has a long tradi- ceive the standard treatment. Using the same example, tion in the social sciences, including communication. the control group would receive a routine face-to-face This type of research is most often conducted in the classroom lecture on public speaking. In either case, laboratory or in other simulated environments that the control group is the baseline of comparison for the are controlled by researchers. Alternatively, research- experimental, or treatment groups. Continuing with ers can conduct experiments in the field, or naturally Experimental Quasi-experimental Descriptive forms forms forms Manipulation of independent Present: Present: variables researcher natural Absent controlled variation Random assignment of Present Absent Absent participants to conditions FIGURE 7.1 Three Forms of Quantitative Research Designs 124 CHAPTER 7 / Quantitative Research Designs occurring environments. This type of experiment is more than 1 hour, with all of the interaction under the very popular with communication scholars who study control and observation of the researcher. applied communication problems. When the word experiment is used, most people Most often, in experimental and quasi-experimental think of laboratory experiments. The primary ­defining designs researchers test hypothesis, which follow the characteristic of laboratory experiments is that the logic of scientific inquiry. Quantitative research as- ­researcher structures the environment in which the sumes that human behavior and communication is investigation takes place and in which the data are col- patterned, and that those patterns can be identified lected (Weaver, 2008). Conducting research in the lab or uncovered through research. For this reason, a null environment serves several purposes. First, it physically hypothesis, designated by H0, technically represents a isolates the research process from the day-to-day and failure to find a relationship or difference between vari- routine interaction of participants. This isolation gives ables when a hypothesis is supported. The outcome of a researcher greater control over what participants are hypothesis testing is to either reject the null hypothesis and are not exposed to. By limiting and controlling (i.e., reject the assumption that the world is random) ­exposure in this way, a researcher is attempting to elim- or fail to reject the null hypothesis (i.e., accept the as- inate extraneous variables and influences that are not sumption that the world is random). In other words, central to the investigation (Kerlinger, 1986). Second, a ­researcher has developed a hypothesis that asserts exploring communication in the lab allows ­researchers that one variable, the independent, causes change in to confine and examine theoretical relationships that a second variable, the dependent. Experimentation would be more difficult to do in the field. allows a researcher to evaluate hypotheses that have Burgoon et al. (2008) study of how people rely been developed from theories in the literature. In this on mental shortcuts to discriminate truths from case, results from previous research studies generate lies demonstrates how the lab environment controls new questions, and the researcher wants to answer the extraneous influences. To set the study up, partici- ­questions “Why?” and “How?” Experimentation is also pants in another study were randomly assigned to the chosen when researchers want to test a new method or role of thieves or innocent bystanders. Participants as- technique. This is especially true of instructional tech- signed the thief role were asked to take a wallet from a niques that are believed to have potential application classroom on a pre-assigned day and then to make de- in the classroom. Conducting an experiment allows the ceptive statements about that act during an interview researcher to test one technique against another to see about the theft. Participants assigned to the innocent if differences between techniques exist. In other cases, bystander role were simply told that a theft would take experiments are conducted to explore the specific con- place in their classroom; they were asked to respond ditions under which a phenomenon occurs. By varying truthfully during the interview. These ­interviews be- the conditions of the experiment, researchers can iden- came the stimuli for the main study. Next, participants tify which environmental conditions, for example, are for the main study came to a computer lab and were most likely to make speakers nervous. seated in front of a computer. They were told they When research is identified as experimental, the would see, hear, or read about an interviewee being goal of the researcher is to establish or explain what questioned about the theft of a wallet. They were caused a person’s behavior, feelings, or attitudes to also told that the interviewee would plead his or her change. Because this is the goal, certain characteris- innocence. The participants were told that they should tics must be satisfied. First, the research design must make a determination about whether the interviewee have a temporal component, with one element occur- was telling the truth about being innocent or being ring before another. For something to cause something ­deceptive. else, the causal agent must precede the change in Without lab experiments, how could researchers ­behavior, feelings, or attitudes. In this way, an experi- carefully make the distinctions between those who tell ment provides control of one variable, or the indepen- the truth and those who do not? How could they care- dent variable, to test its effect on another variable, the fully study the effects of those truths or lies? dependent variable. Second, there are comparisons To meet these research goals, the research team between at least two groups. Finally, the entire experi- created two laboratory experiments—the first to cre- ment is conducted within a limited time frame, seldom ate real instances of truth and deception; the second  THE EXPERIMENTAL FRAMEWORK 125 to test the influences of those truth-telling and decep- of the communication issues studied can seldom be tive messages. By working in the lab environment and represented so simply. using the interviews from the first study, the research In a classical experiment, the researcher controls the team could control what the truth-telling and deceptive treatment or the manipulation of the independent vari- messages were about. Being in the lab controlled the able by randomly assigning participants to treatment interaction environment. Thus, the results of the study or control groups. A treatment, or manipulation, is one can be assumed to result from the conditions of the of the ways in which the researcher varies the type of study rather than from any number of other influences stimuli or the amount or level of stimuli presented to (such as familiarity with the person telling the truth or research participants. This fundamental characteristic lies, different motivations for stealing the wallet, theft must be satisfied to locate a research study in the clas- of a variety of objects). sical experimental framework. Many universities and communication departments Experiments allow researchers to test two types of have rooms equipped as communication laboratories. hypotheses—those that predict differences and those Some are set up with one-way mirrors so that research- that predict relationships. Recall from Chapter 4 that ers can view what is going on, although participants hypotheses test differences and relationships between cannot see the researchers. Some are equipped with and among variables, not the variables themselves sophisticated digital recording equipment and other (Kerlinger, 1986). For the first type of hypothesis—the communication technologies. Although some labs are difference hypothesis—the experiment is designed so relatively sterile in appearance, others are designed to that the independent variable precedes the dependent simulate comfortable interaction environments like a variable in temporal order. Thus, the corresponding living room, waiting area, or computer lab. Research- hypothesis predicts that the independent variable ers without these types of laboratory environments of- causes changes, or effects, in the dependent variable. ten use traditional classroom space or meeting rooms, For the second type of hypothesis—the relational and temporarily transform it into a lab for research hypothesis—the experiment is designed so that two purposes. variables, the independent and dependent, occur close Using manipulation and random assignment, together. The hypothesis predicts that the two variables research­ers design and conduct a study, evaluate the exist together in some type of relationship where the evidence, and then develop a causal explanation for value of the independent variable is causing a change what has occurred. Experimental designs are deliber- in the value of the dependent variable. See Chapters 10 ate, standardized, and used as the research protocol and 11, respectively, for the statistical tests that accom- in many disciplines. Their strength lies in the control pany these hypotheses. they provide to researchers, which in turn helps them The researcher also controls the order of variables eliminate rival explanations for the changes they ob- in an experiment. One element, the independent vari- serve and record. Several of the experimental designs able, cannot be considered the cause of another ele- more commonly used in communication research are ment, the dependent variable, if the independent ­described in the following sections. variable occurs after the dependent variable (Selltiz et al., 1959). Simply put, the independent variable can The Classical Experiment be considered the cause of changes in the dependent only if the independent precedes the dependent or if Researchers devised the classical, or true, experimen- the two occur close together. tal form as a technique to help them in assigning cau- With this level of control, experiments are de- sation. Many researchers believe that the experiment signed to demonstrate a cause–effect relationship. is the most powerful research form for testing cause– Still, an experiment may not be able to provide a effect relationships. The clear logic of its design helps complete explanation of why an effect occurs. Re- researchers eliminate many of the other explanations member that researchers rely upon a theoretical base that can be given for the results found. In its simplest to provide a foundation for the variables in the study form, an experiment would be designed to test the and from which to develop hypotheses. Thus, for the ­influence of one independent variable on one depen- experiment to reflect a causal relationship, the cause, dent variable. But the social and practical significance or independent variable, must come before the effect, 126 CHAPTER 7 / Quantitative Research Designs or dependent variable. The gap between the two vari- any ­considerable attitude change toward Israelis” ables can vary greatly—from just a few minutes to ­(Alhabash & Wise, 2012, p. 364). National attitude was years—but the variables must have a time order. The the ­dependent variable; students evaluated both Israe- dependent variable must also be capable of change. lis and Palestinians on the same seven characteristics Some variables cannot be changed in experiments. (e.g., sympathy; belief about the national group’s inten- Your sex, for example, will not be changed by a re- tion for peace). searcher’s intervention. But your attitude toward a In this experiment, participants were both ran- public relations campaign can be. Finally, the causal domly selected and randomly assigned to different relationship between and among the variables must conditions. These procedures maximize the chances have theoretical plausibility. In other words, the pre- that individuals in the treatment and control groups sumed reason for the causal ­effect must make sense are relatively similar. Thus, any differences between the (de Vaus, 2001). groups on the dependent variable are said to be caused by the manipulation of the independent variable and Random Assignment of Participants In any experi- not by differences in individual participants. These ment in which the researcher wants to compare two or ­aspects of the classical experiment give the researcher more groups, the underlying principle is that individu- greater assurance that other alternative explanations als in the groups are equivalent before the treatment. for the results are eliminated (Sapsford & Jupp, 1996). To achieve equivalency, participants are randomly ­assigned to treatment or control groups that represent Creating Treatment and Control Groups The treat- the independent variable. This means that each partici- ment group is the group of participants who receive a pant has an equal chance of being assigned to either stimulus—anything that the researcher is interested in group. Selecting a random sample from an appropriate studying. Alternatively, if a participant is assigned to a population is not the same as randomly assigning indi- control group, no treatment or a baseline treatment is viduals to treatment and control groups. The two pro- offered. In the experimental framework, creation of the cedures together help eliminate any true differences treatment and control groups provides the opportunity between individuals in the groups before the treatment for the manipulation of the independent variable. It is is applied. Thus, the researcher can argue that dif- important to note here that identification of what con- ferences that result after the treatment is applied are stitutes a treatment, or condition, is driven by theory caused by the independent variable. (Boruch, 1998); not just any treatment will do. For example, to test the effects of role playing in Returning to Burgoon et al.’s (2008) study of detect- a video game about the Palestinian-Israeli conflict, ing deception, the research team formulated its hypoth- student participants were recruited from an American esis on the basis of an extensive literature review and university (Alhabash & Wise, 2012). They were ran- analysis of previous research on the truth bias. From domly assigned to one of two roles—that of Palestinian this they hypothesized “observers err in the ­direction president or Israeli prime minister; thus, nationality of judging more messages as true than the base rate of ­assignment had two conditions and was an indepen- truthful and deceptive stimuli being judged” (p. 576). dent variable. Participants played the game for about That is, people observing others telling truths and lies 20 minutes. For this study, it is important to know that are more likely to believe that others are telling the playing either role does not change anything about the truth. Type of message (truth or lie) was the indepen- structure or the content of the game. Because the stu- dent variable manipulated by the research team. Those dents were from an American university, the research- who took the wallet were told to lie and say that they ers hypothesized that the nationality of the role played did not. Those who did not take the wallet were told would have dissimilar effects on students’ attitudes of to tell the truth. In this case, the control group would Israelis as compared to Palestinians. be those who did not take the wallet and the treatment The hypothesis the researchers tested was the fol- group would be those who did take the wallet and lowing: “Participants who play the role of Palestin- then lied about doing so. By comparing participants’ ian president will exhibit unfavorable attitude change evaluations of the interviewees’ truthfulness, the re- toward Israelis, compared to those playing the search team provided support for the ­hypothesis—that role of Israeli prime minister, who will not exhibit is, when people make deceptive statements about an  THE EXPERIMENTAL FRAMEWORK 127 action, others are more likely to believe they are tell- emphasis was one independent variable, and use of ing the truth. As you can see, the conditions of the gestures was another independent variable. independent variable chosen by the research team were Why did the researchers select these treatment inextricably linked to the theory of interest. groups? Two dual processing theories, like elaboration In some communication experiments, multiple likelihood model and heuristic systematic model, sug- treatment groups are used without a control group. gest that two different kinds of persuasion ­processes Sometimes one group is considered a control group exist. One persuasion process is based on the ­receiver’s even though participants in it receive some form of ability to carefully consider the arguments made and the stimulus or treatment. Usually, the more stan- the receiver’s ability to focus on the quality of the dard, common, or traditional form of the stimulus content in the presentation. Thus, nonverbals of the serves as the control group. In other instances, the speaker are less important in evaluating the presenta- treatment groups serve as a control for one another, tion. Alternatively, other receivers process informa- and the ­researcher forgoes the use of a pure control tion and make speaker evaluations based upon the group. ­presenter’s attractiveness or the presenter’s use of non- Jackob et al. (2011) demonstrate this use of treat- verbals. By manipulating vocal emphasis and gestures, ment groups in their exploration of how different vo- the ­research team was able to control the presentation cal presentations and use of gestures influence the performance. persuasiveness of a presentation. In their experiment, After watching the video of their condition, par- the researchers created three treatment groups by hav- ticipants responded to interval rating scales to assess ing a presenter make three videos in which vocal em- the dependent variables: the perceived performance of phasis and gesture use was manipulated while making the speaker (e.g., vividness, liveliness), the perceived a presentation on globalization. In the first treatment ­characteristics of the argumentation (e.g., factual accu- group, participants viewed the professional presenter racy, thoroughness), and the perceived characteristics deliver the speech without vocal emphasis and of the speaker (e.g., credibility, competence). without gestures. The second treatment group saw the Thus, treatment groups are chosen by the researcher same presenter make the same presentation with vocal based on theory. Researchers must have meaningful emphasis but no gestures. The third treatment group reasons for selecting the treatment groups—just trying saw the same presenter make the same presentation different things does not uphold the scientific tradi- with both vocal emphasis and gestures. Thus, vocal tion on which experimentation is based. Withholding AN ETHICAL Would You Ask Participants To... ? ISSUE The ethical issues of experimental design are primarily focused on developing the treatment and control conditions to which participants are assigned. What would be the ethical objec- tions to creating manipulations for the following scenarios? For example, in the first instance, would it be ethical to knowingly assign engaged couples to a treatment group in which they received counseling even though you doubted its effectiveness? Testing different types of relational maintenance counseling for couples about to be wed? Assigning women with breast cancer to one of two group communication strategies for discussing their health concerns without knowing if the strategies were theoretically sound? Designing different persuasive appeals for nurses to use in dealing with difficult patients? Testing different visual strategies for making empathic appeals to solicit donations to a children’s charity from television viewers? 128 CHAPTER 7 / Quantitative Research Designs treatment or the stimulus creates the control group. two coders unaware of what the confederates were in- In cases in which it would not make sense to have a tra- structed to do and unaware of the study’s hypotheses ditional control group, multiple treatment groups can viewed videotapes of the group discussions and evalu- act as controls for one another, or the researcher may ated the confederates’ smiling behavior, friendliness, designate one treatment group as the standard against persuasiveness, talkativeness, and the degree to which which the other treatments will be evaluated. the confederates appeared to like the participant. Sta- tistical tests indicated that the confederates’ behaviors Manipulation Checks When an independent ­variable were not significantly different on these dimensions, is manipulated, a researcher should conduct a manipu- suggesting that the confederates were not acting differ- lation check. This test, or check, verifies that partici- ently except with respect to the mirroring behavior of pants did, in fact, regard the independent variable in the one. Manipulation checks should be conducted prior various ways that the researcher intended. This check to the statistical testing of any hypothesis in which the is conducted prior to statistical analyses of the hypoth- manipulated variable is used. eses. Why is this necessary? First, researchers need to confirm that participants were sensitive to the different Types of Experimental Design treatments. Second, researchers need to confirm that differences in treatments as perceived by participants Several types of experimental design meet the funda- were in line with the differences the researcher in- mental criteria of random assignment of participants tended. Without this confirmation, a researcher might and researcher control of the manipulations of the assume that differences in the ­dependent variable were independent variable. Three of the basic designs due to differences in the independent variable when commonly used in communication research are de- they were, in fact, not. scribed here—posttest only, pretest–posttest, and facto- In their study of students and working adults, rial ­design. More complex experimental designs exist ­Westerman and Westerman (2013) randomly assigned (e.g., see Campbell & Stanley, 1963; de Vaus, 2001). participants to one of four message conditions to Starting with these basic designs, however, will in- ­examine how participants responded to feedback about crease your ability to comprehend complex designs, a work project. Two independent variables, positive or because they are embedded variations of the basic negative valence, and private or public delivery, were designs ­presented here. combined to create four conditions: positive/private, positive/public, negative/private, and negative/public. Posttest Only After randomly assigning the sample to The manipulation check revealed that participants treatment and control groups, researchers need to mea- viewed the positive messages as more positive while sure the dependent variable either during or after the the negative messages were perceived as more negative. participants’ exposure to the independent variables. Likewise, evaluations by participants revealed that the This type of research design—the posttest only, or e-mail was perceived as more private while a company simple comparison—allows a researcher to conclude website was perceived as more public. that any significant differences found are due to the Another type of manipulation check should be fact that the treatment group received some different used when research designs include a confederate. stimuli or that the treatment group received a stimulus To examine how participants in a group discussion that participants in the control group did not. Thus, ­allow nonverbal cues from other members to influ- this type of design answers the questions, “Do treat- ence them, van Swol (2003) used two confederates ment groups differ based on different manipulations posing as participants with one naïve participant. of the stimulus?” or “Do the two groups differ after the ­Together, the three individuals, role-playing as man- stimulus is presented to only one group?” agers in a pharmaceutical company, were to decide This research design is fairly common in commu- which cholesterol-lowering drug they would market. nication research. As an example of a posttest design, During the discussion, only one confederate mirrored shown in Figure 7.2, Lee and Chen (2013) recruited teen- the nonverbal behavior (posture, hand gestures, facial agers between 14 and 20 years old to participate in a lab expressions, adaptors, and head movements) of the experiment. Of these participants, 18 self-categorized naïve participant. To assess the confederates’ behaviors, as moderate drinkers and 45 self-­categorized as binge  THE EXPERIMENTAL FRAMEWORK 129 Drinking Behavior Random Participants assignment Humorous anti-alcohol abuse TV ads with Perceptions of risk of Binge drinkers proactive-nonrestrictive excessive drinking message Humorous anti-alcohol abuse TV ad with Moderate drinkers negative-restrictive message Intention to change drinking behavior Control group; did not Does not drink view TV ad FIGURE 7.2 Posttest-Only Design drinkers; 22 indicated they did not drink alcohol. As change drinking behavior (example item: “I would very the first step, participants completed a survey about much like to change the fact that I drink excessively”). risky behaviors. Next, participants were randomly In designing this type of experiment, the research team assigned to one of three groups: (a) viewing humor- was interested in discovering how ads influenced teen- ous anti-­alcohol abuse television ads characterized by agers’ perceptions of their risk of drinking excessively proactive-­nonrestrictive messages (e.g., “Drink respon- and their intention to change their drinking behavior. sibly”), (b) viewing ­humorous anti-alcohol abuse tele- The objective of this type of experimental research is vision ads characterized by negative-­restrictive ­slogans to demonstrate a cause–effect relationship by looking (e.g., “Don’t be a loser. Don’t drink”), or (c) the con- for differences among scores of participants in the two trol group in which ­participants did not view either treatment groups and participants in the control group type of ad. Thus, this posttest-only experiment had two on the dependent variables. treatment groups that manipulated the way in which the television ad used humor to gain attention and de- Pretest–Posttest By adding one step to the posttest- ter excessive drinking, and one control group. Finally, only design, a researcher achieves the pretest–­posttest all participants filled out surveys about their percep- experimental form. Here a researcher measures the tions of risk of excessive drinking and their intention ­dependent variable before the treatment group is to change their drinking behavior. Participants in the ­exposed to the stimulus. After the stimulus is given, two conditions that viewed the television ads also re- the dependent variable is measured again in exactly the sponded to items about their interest in the ads, how same way with the same participants. In some written humorous they found the ads, and their perceptions research reports, the researcher refers to these measure- about the effectiveness of the ads in highlighting the ments as Time 1 and Time 2. Time 1 is the measurement risks of excessive drinking. before any stimulus is given to the treatment group, and Thus, two independent variables were manipulated. Time 2 is the measurement after the stimulus. First, the researchers manipulated the type of ad, as Adding the pretest, or Time 1, measurement allows they randomly assigned participants to one of the the researcher to determine the degree of change in the two treatment groups. Second, participants self- dependent variable from one measurement to the next categorized their drinking status as binge, moderate, or and to make more definitive assessments about the in- do not drink. The dependent variables were interest in fluence of the treatment. Although many researchers the ads (example item: “These ads truly held my inter- agree that this experimental form is more powerful, est”), perceived humor of the ads (example item: “I en- there is one caveat. Because measurements at Time 1 joyed the humor used in these ads”), risk perception of and Time 2 are conducted in the very same way, there ­excessive drinking (example item: “I consider myself to is some danger that participants become overly sen- be at risk of becoming an alcoholic”), and intention to sitized to the measurement, especially when data are 130 CHAPTER 7 / Quantitative Research Designs collected through questionnaires. Two different effects dramatized the point that a sexual encounter helped can occur. In the first, participants may try to respond to rekindle passion between the characters, thereby at Time 2 as they responded at Time 1, or, in other enhancing their relationship. In the negative condi- words, try to be consistent even though a change has tion, one show depicts a character who realizes that occurred. In the second, participants assume that the she became intimate with another character too soon researcher is looking for differences and may try to and now feels remorse for her behavior; the characters make sure they answer differently at Time 2 than at terminate their relationship. In the second show of the Time 1. In either case, the participants’ motivations or negative condition, one character, who admitted that expectations can confound the research design. he did not have sex with his girlfriend, finds himself in As an example of a pretest-posttest design, Eyal an unexpected romantic moment with his female room- and Kunkel (2008) recruited undergraduate freshmen mate. After their sexual encounter, he expresses guilt to participate in an experiment. As the first step, par- and is upset by his behavior and its implications. Im- ticipants completed a questionnaire, which ­included mediately after viewing the shows, participants filled a measure of attitudes toward premarital sexual out the measure of attitudes toward premarital sexual ­intercourse. At least one day later, participants were intercourse. Two weeks later, participants again filled randomly assigned to one of two types of viewing ses- out the same measure. sions. Each session included two episodes of 1-hour Using this research design, the researchers could television dramas that were popular with young adults compare the change in participants’ attitudes toward (Beverly Hills 90210, Dawson’s Creek, Party of Five). All premarital sexual intercourse by comparing first the shows included portrayals of young adults engaging in pretest and immediate posttest scores and then com- sexual intercourse; shows also depicted the emotional paring the pretest with the two-week follow-up scores. consequences experienced by the characters. In the This pretest–posttest experimental design is presented positive condition, one show ended with a character in Figure 7.3. The objective of this type of experimen- coming to new conclusions about relationships and tal design is to demonstrate a cause–effect relationship about feeling better about himself; the other show of attitude change from pretest to immediate posttest Treatment: Immediate Two–week Television posttest of posttest dramas with attitudes toward of attitudes positive premarital towards consequences sexual premarital of premarital intercourse sexual Pretest of sexual intercourse intercourse attitudes Random toward assignment premarital sexual Treatment: Immediate Two–week intercourse Television posttest of posttest dramas with attitudes toward of attitudes negative premarital towards consequences sexual premarital of premarital intercourse sexual sexual intercourse intercourse Comparison between pretest and posttest scores FIGURE 7.3 Pretest-Posttest Experimental Design  THE EXPERIMENTAL FRAMEWORK 131 and from pretest to a two-week follow up by looking for packaging of a fictional laundry detergent. After be- differences in attitudes between the two types of view- ing randomly assigned to one of six conditions, partici- ing groups. The researchers found that participants pants were brought in small groups to a lab and given held more negative attitudes of premarital sexual 1 minute and 40 seconds to view a detergent bottle. ­intercourse after viewing the shows in the negative con- While the design of the product label was the same sequence condition both immediately after the viewing across the six conditions, each condition had different and two weeks later. No effect was found for those par- combinations of label text and images of the product ticipants who viewed the shows in the positive conse- seal. This procedure controlled two independent vari- quence condition, as their attitudes toward premarital ables: type of ­argument presented in the text of the sexual intercourse did not change. label, and whether an ­image with the words “Green Seal ­Certified” was present or absent. With these two Factorial Design In a factorial design experiment, variables, or factors, participants were in one of six treatment groups are based on two or more indepen- treatment conditions, as ­demonstrated in Figure 7.4. dent variables. Researchers use this type of research In this case, the factorial design allowed research- design to investigate complex cause–effect relation- ers to test for the main effects of each independent ships that cannot be adequately tested with only one variable. In other words, how did participants’ per- independent variable. A factorial design allows the ceptions of the arguments on the label influence their researcher to test for the effects of each independent perceptions of the greenness of the product and their variable. In addition, he or she can test for the interac- intent to purchase? Figure 7.4 illustrates that sig- tion effect, or how the independent variables can com- nificant differences based on the argument (strong, bine to influence the dependent variable. weak, or no argument) would show up as differences Because more than one independent variable is of among rows. Scores for participants in boxes 1 and 4 interest, random assignment is necessary on only one. would be similar to one another, yet different from Natural variation on the personal attributes of partici- scores for participants in boxes 2 and 5, while scores pants may create assignment for the other independent for participants in boxes 3 and 6 would be similar variable. This is fairly common in communication and different from participants in boxes 1 and 4, and ­research in which participants’ sex is one of the inde- boxes 2 and 5. pendent variables of interest. Obviously, a researcher This design also allowed researchers to test for the cannot randomly assign sex to participants. Thus, he main effects of the presence of the label. Here the or she must rely on the participants’ natural variation question would be, “How did participants’ perceptions on this independent variable. But then participants of the seal image influence their perceptions of the must be randomly assigned by the researcher to one greenness of the product and their intent to purchase?” of the treatment groups or to the control group on the Figure 7.4 illustrates that significant differences would other independent variable. show up as differences between the columns. Scores Factorial designs allow researchers to test for the for participants in boxes 1, 2, and 3 would be similar to treatment effects of each independent variable sepa- one another, yet different from scores for participants rately, as well as for the joint effect of the two inde- in boxes 4, 5, and 6. pendent variables together. In research reports, the Testing for the interaction effect could answer the simple influence of one independent variable by itself question, “Did viewing the laundry product with a is referred to as a main effect. In other words, the influ- strong argument and no label produce different per- ence of one independent variable is examined without ceptions of the product’s greenness and their intent to considering the influence of the other independent buy than when participants viewed the laundry prod- variable. A researcher can also examine the interaction uct with a weak argument and the green seal image?” effect, or how the dependent variable is jointly affected If an interaction effect existed, differences would be by the two or more independent variables. between cells in the table, not between rows or col- Spack et al. (2012) used a factorial design to exam- umns. Findings from this study did show an interac- ine the effects of two independent variables—type of tion effect. Participants’ scores for the greenness of argument and presence of an image—on participants’ the product rose across the no argument, weak argu- perceptions of the environmental claims on product ment, and strong argument conditions except when 132 CHAPTER 7 / Quantitative Research Designs Seal Image Present Absent 1 4 Strong Seal image and strong No seal image and strong argument argument Type 2 5 of Weak Seal image and weak No seal image and weak Argument argument argument 3 6 No Argument Seal image and No seal image and no argument no argument FIGURE 7.4 Main Effects and Interaction Effects in a Factorial Design no image of the seal was present. This is called an their research in the discussion section of the written interaction because the differences are not solely journal report. ­dependent on the strength of the argument or solely How much time must there be between measure- ­dependent on the presence of the seal image. How- ments for a design to be considered longitudinal? That ever, with regard to participants’ intent to purchase depends on the nature of the communication phe- the product, only a main effect for presence of the seal nomenon under study. In other words, it is relative to image was significant. Strength of arguments on the the topic. Days, weeks, and sometimes even years could product label did not influence participants’ intent to occur between measurement points. Communication purchase. scholars frequently use longitudinal ­experimental de- In a factorial design, the minimum configuration signs to test the influence of message design strate- requires two independent variables with at least two gies to identify which strategies are more effective in levels each. But the design can be extended to consist changing participants’ attitudes and behaviors. For of several factors with as many levels as necessary example, Van Raalte et al. (2021) used a longitudinal and feasible for each one. Generally, no more than design to examine the over-time effects of cuddling four ­independent variables are built into one factorial on relational quality for male-female married couples. ­design. Likewise, seldom are more than four levels of The hypothesis for the study was: an independent variable considered. When a factorial H: Individuals in the treatment group report higher design is extended to more than two variables, the relational satisfaction (1), higher investment (2), number of interactions increases as well. higher commitment (3), and lower quality of alter- natives (4) than individuals in the comparison and control groups. (p. 65) Longitudinal Designs In the beginning of the research project, 80 adults When experiments of the types just described are (and their partners) were randomly assigned to one designed so that there are multiple measurements of of three conditions: increased cuddling (treatment), the dependent variable, the design is also labeled as increased time spent together (comparison), or no longitudinal. Researchers often call for longitudinal change in behavior (control). Every Wednesday of ­experimental designs as they discuss the limitations of the 4-week study, participants were sent an e-mail to  THE EXPERIMENTAL FRAMEWORK 133 remind them of their conditional assignment. By de- control groups, any significant difference found is signing the study this way, couples’ romantic behav- ­assumed to be the cause of variations in the inde- ior, including cuddling, could be studied—something pendent variable (de Vaus, 2001). Other primary ad- that otherwise would be difficult to study. vantages of experimentation are that the design and Longitudinal designs are also particularly helpful in researcher control over variable manipulation allows determining the degree to which training or instruc- testing of extremes and multiple replications. Opera- tion has been effective. In one study (Byrne, 2009), tionalizations and measuring techniques are decided 156 fourth- and fifth-grade students participated in a on in advance, and a researcher repeats the procedures research study over a 10-week period. The study was and methods in exactly the same way for each partici- designed to evaluate two media literacy interventions, pant or each group of participants. This leads to preci- both of which were designed to combat the negative sion (Kerlinger, 1986). A less obvious benefit is that effects of media violence. Students were randomly when the lab environment remains static, experimen- ­assigned to one of three groups. Both treatment tation can be cost effective and more convenient for groups received a media literacy lesson that included the researcher. After a lab environment is secured and information on violence in the media and violence in outfitted for a particular experiment, the researcher the real world, media effects, how to avoid negative uses it over and over. Seldom can all participants of an effects, and ways to critically evaluate characters that experiment be tested at the same time. use violence. These lessons were developed around ­PG-rated violent film clips. In addition, one of the Limitations of Experimentation treatment groups also completed a writing activity about what they learned; then they were videotaped Despite its strengths, experimentation has several reading aloud what they wrote. The control group limitations. Not all communication scholarship can did not receive instruction; instead they received a be conducted using one of the experimental forms. lesson on jobs in the film industry and completed a In some cases it would be immoral or unethical to short media production exercise in which they wrote conduct the research because subjecting participants a scene, acted it out, and videotaped it. Measures of in one group to some stimuli could be negative or the dependent variable, willingness to use aggression, hurtful (de Vaus, 2001). Legal standards might also were taken at weeks 1, 2, 4, and 10. By designing the make experimentation impossible. Thus, moral, ethical, study in this way, the research team conducted a rigor- and legal issues prevent a researcher from assigning ous examination of participants’ reactions to two dif- participants to some categories or from otherwise ma- ferent instructional models. This longitudinal design nipulating an independent variable. In other cases, it allowed the research team to examine both the short- is impossible to manipulate the independent variable and long-term effects of two different interventions. because these qualities are fixed in the participants. Results of this study demonstrated that children in the For instance, variables such as sex, socioeconomic treatment group that included the writing and reading class, and age cannot be manipulated. These character- aloud activity reduced their willingness to use aggres- istics and ­attributes are fixed within participants; the sion. Despite this finding, the aggression level of the researcher has no control over them. Thus, the first two treatment groups was not significantly lower than limitation is that experimentation by the nature of its the aggression level of the control group. This type of design characteristics will be inappropriate or impos- finding calls into question the use of violent film clips sible to use in all situations. in media literacy instruction. The second major limitation is that even with its stringent design, experimental forms cannot guarantee Strengths of Experimentation that some factor other than the treatment factor did not produce the significant effect (Sapsford & Jupp, The primary advantage of experimentation is that 1996). Researchers can control or minimize the influ- ­researchers can manipulate the independent variable ence of variables they know about. But researchers can to observe changes in the dependent variable. Because never know if they succeeded completely, and they cer- the researcher controls the manipulation, and because tainly cannot control the influence of a variable that is participants are randomly assigned to treatment and ­unknown to them. 134 CHAPTER 7 / Quantitative Research Designs TRY THIS! Designing an Experiment What social media do you use? Do you know people who are not on social media? Or use dif- ferent social media? Using social media as an independent variable, how would you design a pretest experimental design about social media use? A pretest-posttest design? A longitudinal design? What will you have participants do in the lab? How will that vary depending on the type of experiment you are designing? What will you, the researcher, manipulate as the independent variable? What are the various treatments or conditions? On what dependent variable do you expect the manipulation to have an effect? Why? What arguments can you give for the research design choices you have made? What are the limitations of your designs? Third, laboratory experiments rely on a research- QUASI-EXPERIMENTS er’s manipulation of the independent variable. In some cases, such pure manipulation may not exist in Because of the limitations mentioned earlier, sometimes reality. Thus, participants may react to the artificial- researchers rely upon natural variations in the inde- ity of the lab environment as well as to the potentially pendent variable. Called quasi-experiments, or natural artificial manipulation of the independent variable. experiments, this alternative form of research design Another limitation to experimentation is derived is possible because some variation in the independent from one of its strengths. Experimentation is hailed variable exists naturally. In other words, participants are because it allows researchers to test several variables not assigned randomly to treatment and control groups. or combinations of variables at once. Although the Lacking this assignment opportunity, variation in the in- variables can be manipulated and measured in the lab, dependent variable is not under the control or direction this activity does not always equate with how those of the researcher. The three basic designs of posttest only, variables exist and interact in natural communication pretest–posttest, and factorial, can still be used as long settings. as natural variation of the independent variable can be The most frequent complaint about experimen- substituted for manipulation of the independent variable. tation is the lack of reality in experimental designs. Longitudinal designs can also be incorporated Conducting research in sterile laboratories, available with quasi-experimentation. Recall that the distin- classrooms, or other environments unusual to partici- guishing characteristic of this research design is pants can influence how participants will respond or multiple measurements of the dependent variable. react, and that may be different from how they would Brandtzaeg’s (2012) longitudinal study of the use of behave in more natural surroundings or environments. social networking sites (or SNS) is an example of In general, then, experimental research, especially that a longitudinal quasi-experiment. Overall, the goal conducted in laboratories, may be investigating commu- of the project was to demonstrate the social impli- nication behaviors that are less complex than they are cations derived from using SNS, as well as the dif- in the communication environments in which they are ferences between people who use SNS and those found day to day (Miller, 1970). Other scholars, how- that do not. In this example, let’s focus on the us- ever, suggest that lab experiments should not attempt ers of SNS who were recruited and invited to par- to re-create natural interaction. Rather, artificiality ticipate in a study over three years. In the first is necessary to reduce or eliminate all the confound- year, 2,000 people participated; in the second year ing variables that exist in the real world (Pedhazur & 1,372 participated; and in the third and final year, Schmelkin, 1991). 708 people participated. This study clearly shows  QUASI-EXPERIMENTS 135 that longitudinal designs are likely (and expected) acquaintances. In these cases, SNS use (yes, no) was to lose a significant number of participants across a also an independent variable that varied naturally. long term of involvement. This study characterizes features of quasi-­ Brandtzaeg (2012) predicted that SNS use would experimental design because both independent vari- ­facilitate face-to-face contact with users’ close friends ables (SNS use and time) were based on natural and increase over time—so time (or year in this variation rather than researcher assignment of partici- case) is an independent variable. Time varies, but pants to treatments or conditions. This longitudinal the ­researcher does not control it; it varies naturally. quasi-experimental design is presented in Figure 7.5. Another hypothesis in the study predicted that the Scholars who study the efficacy of research meth- number of offline acquaintances of SNS users would ods differ in the degree to which they support quasi- increase over time. Yet another hypothesis predicted experimentation. The primary limitation, of course, that loneliness among SNS users would decrease stems from the lack of control the researcher has over from year one to year two. In each year, or wave, of manipulating the independent variable. Although the study participants responded to surveys. What natural variation can be said to be a substitute for did the researcher find? SNS users did not report an this manipulation, there are some concerns about the ­increase in face-to-face interaction over time, but they quality, or the purity, of the manipulation. For exam- did show an increase in the number of acquaintances ple, in communication research, sex is frequently used they had; they also reported an increase in loneliness. as an independent variable, which obviously is not Thus, this longitudinal study demonstrated that social under the control of the researcher. Thus, communica- implications of SNS time were varied. Users of SNS tion researchers rely on individuals to self-identify as reported both positive (more acquaintances) and females or males. These self-identifications are used negative ­implications (fewer friends, more ­loneliness). to classify participants with respect to sex. It seems However, when compared to participants who did simple, but there are inherent problems. Whereas we not use social networking sites, SNS users reported could agree that there are obvious differences between more face-to-face interactions and a greater number of women and men, there are also differences among Collect data Collect data Collect data Year 1 Year 2 Year 3 (baseline) Face-to-face Face-to-face Face-to-face Increase Increase interaction interaction interaction from year 1 from year 2 with close with close with close to year 2 to year 3 friends friends friends Number of Increase Number of Increase Number of offline from year 1 offline from year 2 offline acquaintances to year 2 acquaintances to year 3 acquaintances Decrease Degree of Degree of from year 1 loneliness loneliness to year 2 FIGURE 7.5 Hypothesized Differences for Users of Social Networking Site Users in Longitudinal Quasi-Experimental Design 136 CHAPTER 7 / Quantitative Research Designs women and among men. These differences within making sure categories are exhaustive, mutually exclu- categories can confuse any results researchers find. sive, and equivalent. Think of the problem in this way: Just because women When treatment groups are based on natural varia- and men differ with respect to physical sex does not tion, some researchers refer to these as nonequivalent mean that women and men will differ as distinctly on comparison groups. They are nonequivalent because the communication aspects of gender, such as com- participants are not randomly assigned. Thus, the way municating with feminine, masculine, or androgynous in which researchers use existing natural variation styles. to assign participants to treatment groups should be Thus, the goal for researchers using quasi-­ carefully considered. The goal is to use groups that experimental forms is to create and communicate are as equivalent as possible, and achievement of this clarity about the differences desired in the indepen- goal relies primarily on the procedures chosen by the dent variable. Additionally, Clark and Shadish (2011) ­researcher. recommend that when possible, to collect data over Another example of the way in which quasi-­ time and to not rely solely on independent variables experiments rely on natural variation was demon- that are dichotomous (e.g., sex). In many quasi-­ strated when Romero-Canyas et al. (2019) tested the experimental studies, participants are assigned to treat- effect of a television campaign (six, 30-second video ment groups by how they respond to that variable on clips advocating for local environmental awareness) the ­questionnaire. that emphasized the impacts of global warming on eco- In essence, participants self-select into the category systems and business activity. The campaign aired on of the independent variable to which they belong by one cable provider in a defined geographic area. Thus, labeling themselves. Other variables on which this type the program was the independent variable, as some of self-selection is common include marital status, po- residents could view that channel, and other residents litical affiliation, race or ethnicity, and university class could not. Potential participants were the subscribers level. Although researchers can never completely over- of the cable provider, who aired the programming, and come problems associated with self-selection, they competing providers in the same zip-codes. can reduce the potential error by providing full and Respondents exposed to the campaign were more complete descriptions of any categories used and by likely to believe that global warming is happening, DESIGN Is One Study Ever Enough? CHECK Imagine the following: In doing an assignment for one of your communication courses, you find a journal article that addresses the very constructs in which you are interested. Even though you use several electronic databases to search for more articles like the one you have, you are not successful. So, with one article in hand, you set out to complete the assignment. As you read about the hypotheses and the quasi-experimental design used in the study, several questions arise: Is there anything about the way in which the research was designed that would detract from its findings, even though the study examines exactly the variables in which you are interested? To what degree did the researcher’s procedures create control over the natural variation in the independent variables? Are the research procedures reported in enough detail and to your satisfaction? Could there be explanations for the findings other than what the researcher offers? Would the results from one study, even if it were an experiment, ever be enough?  QUASI-EXPERIMENTS 137 to accept the scientific consensus, to be more con- experiments are better suited for observing and cerned about impacts, and more supportive of policy evaluating complex and unfolding interaction events. solutions. Thus, the variation on this independent vari- But, in field experiments, researchers also lack the able—viewed the program, or not—existed naturally. ­degree of control they have in true experiments Respondents in the treatment group, those who sub- (Weaver III, 2015). scribed to a particular cable channel and more likely to Clearly, as a researcher moves from the lab into a see the campaign on cable TV, showed “higher levels of natural environment, some control is lost. Although acceptance that global warming was happening, height- researchers can control nearly all outside interferences ened concern, and were more likely to understand the (e.g., people interrupting the research process, intrud- scientific consensus on it” (p. 751). Respondents who ing noises) in the lab, they will be less likely to control did not subscribe to this particular cable channel were outside interferences in the field because they are not the control group, as they were less likely to see the in charge of the environment in which the ­research global warming campaign. However, there were no dif- is taking place. Although some control is lost, real- ferences between treatment and control groups on the ism is increased (Kerlinger, 1986). When realism is causes of global warming. In summary, the treatment ­increased, findings have greater generalizability. and control groups differed significantly along many Field experiments are particularly effective for study- of the dependent variables. The authors concluded, ing communication in context, especially the ­effect of “while other characteristics of the treatment group media or political campaigns. For example, Shaw and may explain the treatment effect, this seems unlikely. Gimpel (2012) used a field experiment to examine the The control and the treatment group did not differ in effectiveness of personal appearances of an incumbent their levels of education, household income, political governor in his campaigning for re-election. While ideology or political party affiliation, known predictors popular wisdom suggests that candidate appearances of people’s environmental attitudes and their beliefs have positive effects, the researchers designed a field about global warming” (Romero-Canyas et al., 2019). experiment to test if candidate appearances would But the question remains, could the results of the ­attract ­local news media attention, as well as increase study be based on preexisting differences? Or were volunteer signups to help with the campaign. Using the the results based on respondents’ participation in the incumbent governor’s campaign schedule, the research program? To answer these questions, the researchers team searched local newspaper and television news conducted closer examinations of the ways in which coverage before and after the governor’s public appear- respondents were different or similar. After carefully ances. Thus, public campaign appearance was the inde- guiding readers of the research article through these pendent variable; in this case, the research team used differences and similarities, the research team provides the city of the appearance as one category of the in- compelling arguments and data to demonstrate that ef- dependent variable and another city as a control (the fects of the race relations program were positive for governor did not make an appearance that day) as a sec- both existing members and new members. But overall, ond category of the independent variable. The number participants who had been part of the program longer of news stories was the dependent variable. Most local had more positive attitudes toward race relations. newspapers and television stations did not focus on the candidate for ­re-election the day prior to his visit. But Field Experiments after his visit, local news coverage was “pervasive after a visit” (p. 144). Thus, the research team concluded that As the name implies, field experiments are like experi- campaign visits help to drive news coverage. A similar ments in terms of researcher control over the manipula- effect was found for volunteer sign-ups, a dependent tion of independent variables and random assignment variable in a separate hypothesis. As predicted, volun- of participants. However, the research environment is teer sign-ups rose from 2.6 before the campaign appear- realistic and natural. Participants are not asked to come ance to 6.8 after the appearance. to a laboratory environment that is used exclusively for Obviously, the research questions addressed in experimentation. Rather, the research is conducted this study could not have been studied in a laboratory in environments with which participants already environment. Using a field experiment was practical are ­familiar. This provides another advantage—field because the results help document the effect of this 138 CHAPTER 7 / Quantitative Research Designs politician’s campaign visits. By designing the project to These differences in descriptive designs decrease compare news coverage in cities where the candidate the degree to which causation can be ensured. made appearances with news coverage in cities where ­Because a researcher has less control, variables other the candidate did not make appearances on the same than those selected as independent variables may be day, the research team could assess the effectiveness of responsible for changes in the dependent variable the candidate’s visits. Thus, this campaign strategy was ­(Kerlinger, 1986). This characteristic makes it diffi- studied in its natural environment without researcher cult for inferences to be made from the independent interference. Thus, the results from this research ­design variables to the dependent variables (Pedhazur & had high generalizability because researcher ­intrusion ­Schmelkin, 1991). Because of this issue, some was minimized to the greatest extent. ­researchers substitute the terms predictor variable and criterion variable for independent variable and depen- Strengths and Limitations of dent variable, respectively. Using these two terms is more appropriate because they do not imply causal- Quasi-Experimental Research Designs ity. Nevertheless, the tradition of naming variables as The obvious strength of quasi-experimental designs is independent and dependent is very strong, and many that the variation that communication scholars are ­often researchers use these terms even when their research interested in exists naturally. Thus, these research de- design is non-experimental. Although some researchers signs can address the real world in a way that experimen- believe that experimental research is always superior tation cannot; indeed it could be difficult, impossible, or to other research designs (Kerlinger, 1986), descrip- unethical for a researcher to re-create these conditions. tive research can paint a picture of a communication Quasi-experimental designs build on differences that phenomenon as it naturally occurs without relying naturally exist rather than rely on researcher-controlled on an intervention or a designed condition (Bickman differences through random assignment. et al., 1998). But the lack of random assignment brings up the One type of descriptive communication research question: Are the groups studied equivalent? Re- occurs when researchers want to demonstrate that search designs that are carefully thought through and categorical differences exist on some dependent vari- those which collect data on other variables that could able. When a researcher limits interpretation of results ­potentially confound the results may help alleviate that from this type of study to a description of differences, concern. So, while a quasi-experimental design does descriptive forms can be successful in identifying not allow researchers to infer cause and effect, a well- variables that may later be used to build a theoretical designed and theoretically grounded quasi-experiment explanation of the relationship between the indepen- does enable researchers to demonstrate that rival inter- dent variable and the dependent variable. However, pretations are implausible. researchers overstep the boundaries of descriptive ­research if they try to offer the differences they find as an explanation. DESCRIPTIVE DESIGNS For example, finding the answer to the research question “What politeness strategies do individuals Descriptive designs, sometimes called cross-sectional report using in this context? (i.e., snubbing a rela- or non-experimental designs, differ from experimental tional partner by using a cell phone in his/her pres- and quasi-experimental research designs in three fun- ence” (Kelly et al., 2019, p. 545), is a question for damental ways. Although the study is still systematic, which a descriptive answer can be found. But knowing the researcher does not have direct control over the what strategies participants gave for phubbing friends independent variables, nor are participants randomly does not explain the influence of that strategy (i.e., assigned. Additionally, data are collected in such a way phubbing friends). that temporal order cannot be determined. ­Despite In this descriptive study, 404 college students re- these differences, the logic is basically the same—­ sponded to a four-part, anonymous survey that asked researchers want to demonstrate that differences participants to estimate how much they used their or relationships exist between the independent and cell phone/smartphone for calling, texting, video chat- ­dependent variables. ting, and posting/checking social media. Two more  ONLINE EXPERIMENTS 139 measuring instruments captured participants’ percep- (p. 151). This type of research design cannot explain tions about perceived positive and negative face threat why relationships or differences occur, but it can posed by eight cell phone behaviors. Participants were ­describe their occurrence. not randomly assigned to conditions, nor did the re- searchers ­manipulate any variables. Rather, partici- Strengths and Limitations of Descriptive pants reported on their perceptions of phone use and Research Designs behaviors associated with phone use. And, as in most descriptive research, the data were collected simulta- Simply, some of the research issues that exist in the neously through written self-report questionnaires. study of communication do not lend themselves Descriptive statistics revealed the two most common to ­experimental or quasi-experimental designs. So, responses to a potential face threat: (a) to say/do noth- clearly, descriptive research has merit. The most ob- ing and (b) start using their own phone so their friend vious advantage is that descriptive studies are most would not feel bad about on his/her phone. often conducted in more realistic environments with As another example, Rosaen and Dibble (2008) participants who are more like the individuals to whom investigated the question, “Is there a relationship the researcher wants to generalize. Also important, de- ­between social realism and parasocial interaction?” scriptive studies can be used in an exploratory manner (p. 149). Children between the ages of 5 and 12 were to provide an orientation to a topic (Henry, 1998). asked to identify their favorite television characters. You have already seen that the differences ­between Over 70 characters were identified; Lizzie ­McGuire experimental and descriptive research—inability to and SpongeBob SquarePants were most frequently ­manipulate the independent variable and lack of mentioned. Children also provided responses to a power to randomly assign participants—create two of questionnaire of parasocial interaction items. “I would the fundamental limitations of this type of research invite SpongeBob SquarePants to my birthday party” ­design. These two weaknesses lead to a third limita- is an example of how the researchers measured para- tion, and that is the risk of improper interpretation social interaction, or the imaginary friendships that of the ­research findings. With significant findings, viewers form with media characters. To get a measure researchers are eager to accept these as meaningful of social realism, three coders made a coding judgment and real, despite the possibility that other explanations for each character on appearance and behavior. In for the findings may exist. Thus, descriptive research other words, did the character look like a real person? designs can be more powerful when researchers offer Did the character act like a real person? The research- several alternative hypotheses as explanations and test ers used a ­statistical test to answer their question; they each one. When one hypothesis is supported and other found that, yes, there is a relationship between social ­alternatives are not, the findings of the significant realism and parasocial interaction. It was a positive ­hypothesis are strengthened (Kerlinger, 1986). and weak relationship, but it was significant. Thus, parasocial interaction is greater when social realism of the character is stronger. ONLINE EXPERIMENTS “Although finding that two variables are correlated does not establish cause it does mean that a causal Some of the problems inherent in conducting all types ­explanation is possible” (de Vaus, 2001, p. 178). Thus, of experimental design can be overcome by conducting it would be inappropriate to claim that the social real- studies online instead of having participants come to a ism of characters caused children to have parasocial research lab or other controlled environment. Gener- relationships with the characters or that having a para- ally, an online, or web-based, study can be developed social relationship with a media character caused the for experimental, quasi-experimental, and descrip- character to seem more real. In the discussion section tive designs, and include longitudinal and factorial of the research article, Rosaen and Dibble (2008) ­aspects. Of course, online studies can also be designed explore the nature of their findings. The researchers as a field experiment—if online behavior is the natural admit that the strength of the relationship found was ­environment being examined. weak, and that the study did “not directly address Some researchers use software provided by their what a child will select as his or her favorite character” universities (e.g., Qualtrics™); other researchers use 140 CHAPTER 7 / Quantitative Research Designs software available to the public (e.g., ­SurveyMonkey™). can be embedded easily in the online environment. A These software programs can help you move your participant can read, listen to, or watch the stimuli and ­research design from a face-to-face environment to the then move directly to the items to evaluate the stimuli. web. In some unique cases, programming skills are Web-based surveys and other technology can now self- needed to design your own website and online experi- administer experimental manipulations that previously ment. But, generally, software programs like the ones took several people to administer in the lab. Moving mentioned above allow you to select or design the tem- experiments online is exciting, but it does not resolve plate (or look) for your study, and provide standard all problems. Sampling remains problematic as online options for presenting questions and responses to par- users are more representative of some demographic ticipants (e.g., Likert-type scales, categorical choices). groups than others (Iyengar, 2011). If your experimen- Online software can be valuable for research ­designs tal study is to be conducted online, be sure to think in which you want a participant to respond to a cer- carefully through how to promote your study to reach tain set of items based on how they answered a previ- the population and sample you desire. ous item. Known as skip logic, this technique allows you to determine, for example, that participants who watch television shows with violence will read your survey items with “violent” as the adjective in the item. On the RESEARCHER EFFECTS other hand, participants who watch situation comedies AND PROCEDURAL BIAS on television can read your survey items with “funny” as the adjective in the item. Skip logic is customized based Regardless of the degree of control offered by experi- on the research design and hypotheses, and it is con- ment and its alternative forms, researchers or other trolled by rules you define as you set up in the program. members of the research team can introduce bias Han and Brazeal (2015) used an online experi- and error into the research process through their ment to test the hypothesis, “When exposed to civil ­interaction with research participants. Researchers can discussions online, participants will emulate the mode influence participants’ behaviors during the ­research of civility in their own discourse” (p. 22). To enhance session, or they can create effects on the data itself. All the realism of reading news online, the researchers se- types of quantitative research designs—experimental, lected a news article from CNN.com. Then, relying on quasi-experimental, and descriptive forms—are suscep- an accepted definition of civil political discourse, the tible because each requires a researcher or assistant to researchers searched for comments from various news conduct or facilitate the process. sites. Examples of civil discourse include respectful It is easy to see that researchers and research par- and polite disagreements, whereas uncivil comments ticipants interact in a social environment. Who the included rude comments and name-calling. After read- researcher is (e.g., age, sex, and race or ethnicity) and ing the news article and comments about it online how the researcher communicates with participants from the civil or uncivil condition, participants typed or other members of the research team can create in their own comments. Interestingly, the research- ­influences on participants’ responses. Imagine the dif- ers found that participants exposed to civil discourse ference in your reaction as a research participant if did model their discourse in that way. However, par- you are met warmly and cordially by the researcher ticipants exposed to uncivil discourse also made civil as contrasted to being met in a hostile and rejecting comments. Thus, the hypotheses about civil discourse manner. How would you respond if you heard two was confirmed whereas the hypothesis about uncivil research assistants arguing over research procedures discourse was not supported. Using an online research before your experimental session began? Likewise, design made sense for this quasi-experiment given that the degree of experience or expertise demonstrated the researchers were interested in examining online by a researcher can influence research participants’ ­political discourse. ­responses to ­stimuli. Online experiments offer several advantages Researchers’ expectations can also create unwanted ­(Iyengar, 2011), such as greater reach to more diverse effects on participants’ responses (Abelson, 1995). populations. It is also efficient and effective to pres- ­Remember that researchers develop hypotheses before ent text, audio, and video stimuli because these stimuli a study is conducted. As a result, a researcher may  COMPARING RESEARCH DESIGNS 141 u­ nknowingly encourage participants to respond in the the study is conducted face to face, protocols should be way that supports the predictions. Even subtle non- practiced with individuals similar to the research par- verbal cues may be perceived by participants and, as ticipants and evaluated for their effectiveness before data a result, influence their responses and measurements. are ­collected. For online studies, a protocol should be Each of these biases can occur in quasi-experimental developed before creating the online research environ- and field designs as well. ment. Then, individuals similar to the research partici- One way to overcome researcher expectancy is for pants should test the online study to be sure it presents individuals other than the researcher who designed the stimuli and questions to participants as you expected. study to conduct the experiment without knowing any- In either case, if pre-testing the ­research ­design leads thing about the hypotheses or the conditions to which you to change something about the protocol, be sure to participants are assigned (Abelson, 1995). When this document those changes. An example of a research pro- occurs, the research report will indicate that the study tocol is available at the ­author’s website for this chapter: facilitator or research assistant was blind to, or did not www.joannkeyton.com/research-methods. Standardizing know, the hypotheses or conditions of the experiment. how the researcher and his or her ­assistants interact with Expectancy bias can also occur when research- participants and how research procedures are adminis- ers use participants with whom they are acquainted. tered ensures more control. This can occur when researchers conduct studies on student populations with whom they are familiar or in field settings where familiarity with potential partici- COMPARING RESEARCH DESIGNS pants eased their access into the field location. Even if the researcher does not know the students, it is very For the study of relationships among and differ- likely that students in the same department or univer- ences between communication phenomena that can sity know of, or know something about, the researcher. be quantified, experimental research designs are the These types of acquaintanceship between researcher most powerful. However, in many cases they simply and research participant can influence the way are impractical or impossible. Thus, researchers turn ­research participants react or respond. As a research to quasi-experimental and descriptive forms for their participant, would you knowingly or unknowingly alter quantitative research. your behavior if the researcher were your advisor? If Ideally, the authors of research reports would you were going to take a class from that professor next ­always explicitly describe the research design they semester? If he or she were the department chair? used, explain why that design was selected, and then Students as research participants are also sensitive identify the design by name. Figure 7.6 can help you to demand characteristics, a situation that creates a decipher the method section of a research report if bias based on participants’ perceptions and interpreta- these issues are not made explicit. tions of the research procedures. Students, particularly After you have identified the type of quantitative those with knowledge of research methods and com- research design, the next step is to identify what was munication theory, may try to second-guess what the measured. You should be able to identify a specific researcher is looking for—the demand—and then try to operationalization of each variable included in the provide answers to fulfill it. ­research hypotheses and research questions. This Demand characteristics are also created when the ­information should be included in the method section research topic has some socially desirable element. For of the research report. Finally, you should examine how example, in conducting research on topics like sexual the research procedures were carried out, particularly harassment, date rape, or deceptive practices, research- with respect to the temporal order of measurement. ers need to develop procedures that encourage partici- By independently checking these issues, you can pants to answer honestly. Honesty is encouraged when critically assess the results the researcher presents and research procedures do not create the perception for draw your own conclusions. You should be satisfied participants that they can be identified as victims or that the researcher used a design that was appropriate perpetrators of these socially undesirable acts. for t