Observation Oriented Modeling PDF - Teaching & Research

COMPREHENSIVE PSYCHOLOGY 2014, Volume 3, Article 3 ISSN 2165-2228 DOI 10.2466/05.08.IT.3.3 © James W. Grice 2014 NOTICE Attribution-NonCommercial- NoDerivs CC-BY-NC-ND Received October 14, 2013 Accepted January 23, 2014 Published Febru...

COMPREHENSIVE PSYCHOLOGY 2014, Volume 3, Article 3 ISSN 2165-2228 DOI 10.2466/05.08.IT.3.3 © James W. Grice 2014 Attribution-NonCommercial- NoDerivs CC-BY-NC-ND Received October 14, 2013 Accepted January 23, 2014 Published February, 2014 Re-licensed December 9, 2015 This Open Access ing, published following pages CITATION Innovative Grice, J. W. (2014) Observation be published Oriented Modeling: preparing students for research in the 21st century. Comprehensive Psychology, With their 3, 3. Commons license Psychology. and cited in Publishing Inc. This article in the format DOI has not changed. Ammons Scientific www.AmSci.com INNOVATIVE Observation TEACHING 2014, Volume 3, Article 3 ISSN 2165-2236 DOI 10.2466/05.08.IT.3.3 © James W. Grice 2014 Attribution-NonCommercial- NoDerivs CC-BY-NC-ND Abstract Observation conceptualization grated, Received October 14, 2013 is thus Accepted January 23, 2014 correlations, Published February, 2014 based will to undergraduate psychology, traditional adapted CITATION Grice, J. W. (2014) Observation Oriented Modeling: prepar- Overview ing students for research in Observation the 21st century. Innovative ceptualizing Teaching, 3, 3. instance, studying can be made such as height can be directly inferred from construct an to seeking the the observation and qualities in consistent Of course structured and ing toys or and the subsequent features or must be derived her model captures scientist must making observations. ality psychologists scores. This tities and can assumption (Michell, tion. Either els of traits would require trait?” and To accomplish structure of Ammons Scientific 1 Address correspondence www.AmmonsScientific.com State University, from the modern paradigm of statistical modeling. This paradigm is perhaps epitomized by structural equa- tion and path diagrams that connect constructs and variables in networks of associations. The variables in these diagrams are often assumed to represent continu- ous quantities and their analysis is consequently based on aggregate statistics, such as means, variances, and covariances. Their efficacy is moreover almost uni- versally assessed via the estimation of abstract popu- lation parameters in the assumption-laden context of null hypothesis significance testing (i.e., via p values). In contrast, the observation-oriented scientist must be willing to get beyond these methods and re-orient his or her attention to the observations being made and to the individuals being studied. Rather than compute aggregate statistics and estimate population parame- ters, patterns of observations are instead examined in light of a causal model using analysis techniques that are relatively free of assumptions and yield results that are both transparent and interpretable at the level of the individuals in the study. The overall analysis objective is thus to assess the accuracy of a model as an explana- tion of patterns of observations. As a novel approach for conceptualizing and ana- lyzing data, the use of Observation Oriented Model- ing requires something of a Gestalt shift on behalf of the scientist trained in contemporary experimental statis- tics. The purpose of this paper is to facilitate this shift by providing instructional information and class projects to teach Observation Oriented Modeling to graduate and advanced undergraduate students in the life sciences. The information and projects presented below are likely most suitable for an existing course on experimental statistics or research methods, and the example analyses can be taught alongside traditional methods of data analysis. In fact, the example analyses below compare and contrast traditional methods with Observation Oriented Modeling. As will be made plain below as well, however, a basic knowledge of the different philosophical schools of thought (e.g., positivism, realism, idealism) is required of the instructor. Accordingly, important concepts and ideas that must be taught to students are discussed gen- erally, and they are also specifically presented before each example project. The projects themselves are based on genuine or contrived studies and are designed to demonstrate: (1) the application of key concepts; (2) the development of integrated, causal models; and (3) how data can be analyzed with the Observation Oriented Modeling (OOM) software.2 The examples are drawn from the psychological literature but students from other life sciences (e.g., biology, sociology, education) should find them accessible and meaningful as well. 2 The OOM software can be downloaded for free at http://www.id iogrid.com/OOM. Additional information regarding and the Observation Oriented Modeling book (Grice, found at this website. Innovative Teaching for the psychological attributes they study. The conse- quent being, “There is no evidence that the attributes that psychometricians aspire to measure (such as abili- ties, attitudes and personality traits) are quantitative” (Michell, 2011, p. 245). If Michell's assessment is correct, then psychologists have been using and advocating methods that often presume continuous quantities (e.g., ANOVA, SEM, IRT, etc.) where no evidence exists that the qualities or attributes they study are in fact continu- ous. Again, this is an instance of psychologists allowing their methods to drive their metaphysics. The premise underlying the development of Obser- vation Oriented Modeling and the OOM software is that the arguments of methodolatry and statisticism are essen- tially correct; and they are correct because psychology has built its research methods foundation primarily upon the tenets of philosophical positivism, which is a viewpoint steeped in phenomenalism and long ago abandoned by philosophers due to its many additional inadequacies (see Costa & Shimp, 2011). What is currently needed, then, is a way of thinking about research that does not necessar- ily entail training in contemporary statistical methods or psychometric theory, but instead emphasizes model building and model evaluation from the perspective of philosophical realism. Observation Oriented Modeling is one such alternative in which students are challenged to confront human nature and the qualities people possess while building causal models that emphasize the integra- tion of structures and processes rather than the construc- tion and analysis of networks of variables in a structural equation or path diagram. A General Change in Perspective As mentioned above learning about Observation Ori- ented Modeling involves something of a Gestalt shift or change in perspective that can be described as a move from phenomenalism to moderate realism. Phenom- enalism is a catchall term that encompasses important aspects of positivism and the philosophies of Descartes, Hume, Kant, and others. The most important common feature of these philosophies is that the essences of things cannot be known; in other words, as Kant would say, the “things-in-themselves” cannot be known. When discussing nature through phenomenalism, human- ity will always be limited to the appearances of things as they are represented in consciousness. This attitude, advocated by Stanovich (2007, pp. 35–52) in his popu- lar book How to Think Straight about Psychology, remark- ably prevents psychologists from asking “what is” types of questions. Questions such as “What is intelligence?” are regarded as misguided and scientifically useless because psychologists work with numeric representa- tions of concepts that can be manipulated and analyzed for their consistency and coherence. Operational defini- tions are furthermore put forth as the solution to “what is” types of questions so that a person's Innovative Teaching Page Phenomenalism 4 Hume's Causation 7 Means, Variances, Covariances 7 Aggregates 10 Effect Sizes 10 Estimating Population Parameters 10 Assumption-Laden p-values 16 Assumed Continuous Quantities 16 Variables 16 Operational Definitions 22 Statistical Replication Note Page numbers refer will be discussed or demonstrated or ‘right.’” Further inquiry, however, reveals that while some people write exclusively with one hand, they may brush their teeth, swing a ping-pong paddle, or throw a ball with the other hand. Of course, a person who writes with the left hand may also swing a baseball bat or use a broom like a right-handed person. What if a person who writes with the left hand kicks a ball with the right foot? Does this matter in developing an understand- ing of handedness? Students should also be encour- aged to search for handedness questionnaires that are freely available on the Internet and to discuss the vari- ous strengths and weaknesses of operational definitions. The goal of asking “what is” types of questions is to begin the move from the phenomenalism inherent in modern research practices to a position of moderate real- ism. Table 1 presents a list of additional concepts that can be compared and contrasted when making this shift in perspective. The concepts are not presented in order of importance, and they will in turn be discussed quent sections of this paper. As indicated by in Table 1 some concepts will be discussed and demon- strated together. From Causation to Causality To begin the move from phenomenalism to moderate realism, students must be taught to think more richly about causality. This will entail a review of Aristotle's four causes (material, formal, efficient, and final) discussed in his Physics (Book II, Chapter 3) and Metaphysics (particu- larly Books I–VII). Although students should always be encouraged to read primary sources, a number of second- ary sources may be referenced (e.g., Rychlak, 1988; Wal- lace, 1996; Falcon, 2012). Fundamentally, students should understand that Aristotle sought to explain nature; con- sequently, when asking “what is” and “why” types of questions the material, formal, efficient, and final causes will be employed. For instance, a response to the question “What is deoxyribonucleic acid (DNA)?” will refer to the matter of which it is made. The response may be at a very Innovative Teaching son sits in direct sunlight to deepen his tan (final cause), the body will change in ways to release the added heat (e.g., perspiration will increase; efficient cause), but it will not release too much heat in moving toward its balanced point (final cause). Another example involves growth; for instance, red oaks normally grow to approximately 90 feet whereas pinon pines normally grow to approximately 30 feet in height. These homoeostatic points or natural end states towards which things move or change are therefore also entailed in Aristotle's final cause (Wallace, 1996). Biochemical pathway models are perhaps the simplest and most effective models to use when introducing stu- dents to Aristotle's four causes (e.g., an image of the Krebs cycle can be used: http://en.wikipedia.org/wiki/Citric_ acid_cycle). The atoms in the model are the material causes, the structures of the different molecules in the model are the formal causes, the arrows in the model showing how com- pounds influence each other or change their patterns are the efficient causes, and the rhythmic, stable nature of the structures and processes in the model (viz., it is in a normal “holding pattern”) is the final cause. A simplified model of an atom can also be discussed. The protons, neutrons, and electrons are the material cause, the electrons moving about in time in their orbits are efficient causes. Students can also imagine energy being used to change the orbits of the elec- trons as an efficient cause. Formal cause is the “whatness” of the atom, it's pattern or configuration of matter that makes it to be a particular atom. Lastly, the structures and processes in the atom are in balance. Again, students can imagine adding or subtracting energy to change the orbits of the electrons, and in normal circumstances the electrons will settle into a stable holding pattern (final cause). Once students appreciate the rich explanations of nature afforded by the four causes, they can apply them to psychological theories. A critical point must be made, however, either prior to or during this transition. The point is this: causes are bound to the natures of the things themselves. An unfortunate turn in history is that psy- chologists adopted a positivistic view of causation popu- larly found in the writings of J. S. Mill and David Hume. In this view, cause is reduced to efficient cause only, and moreover causes are regarded as consistent associations formed in the mind. In other words, the causal struc- ture of nature itself cannot be known; rather, the consis- tent associations of sense impressions are all that can be known, and these consistently associated impressions are regarded as causation. This metaphysical standpoint not only reduces Aristotle's causality to Hume's causa- tion, but it is highly subjective (viz., it fits with phenom- enalism) as well and serves as the impetus for viewing the search for causes as largely a statistical exercise, as with mediation or structural equation modeling, or still worse solely as “equation surgery” (Pearl, 2009, p. 417). From the perspective of moderate realism, however, the search for causes is not necessarily statistical or math- ematical and is instead similar to the sleuthing of Sher- Innovative Teaching An Exercise in Causal Modeling Zealure Holcomb's (1997) Real Data workbook is an in stance of an accessible resource for students that re- ports numerous studies with actual data. One example (p. 123) reports data for 135 women who visited a hos- pital seeking evaluation for self-identified symptoms of breast cancer, such as a lump or discharge (see Lauver & Youngran, 1995 for the original article). Among oth- er things, the goal of the study was to identify the rela- tionship between optimism and any delay in seeking the evaluation. The authors expected a negative linear relationship between the optimism and delay values as shown in the following variable-based model: A negative relationship was expected partly because optimism has purportedly been shown to correlate posi- tively with better future planning and with positive atti- tudes of acceptance (Lauver & Youngran, 1995, p. 203). The optimism scores ranged from 0 to 4 and were computed from responses to the original Life Orienta- tion Test, an 8-item self-report questionnaire. The women responded to statements such as “In uncertain times I usually expect the best” and “I’m always optimistic about my future” using a 5-point rating scale. The numerical ratings on the scale were then averaged. The delay val- ues were also reported by the women and ranged from 1 to 2,538 days (i.e., days between initial detection of the symptom and the visit to the hospital). Over half of the delay values were less than 21 days, and several extreme cases were noted, which will be discussed below. Students should be encouraged to write down their own thoughts and questions about the posited link between optimism and delay in visiting a doctor. The following types of questions may emerge: “Are optimistic women necessarily better planners, and why isn't planning in the model?” “Once an optimistic woman has identified what she thinks is a cancerous lump, would she necessarily feel a sense of panic and want to immediately see a doctor, or would she be less worried?” “Wouldn't an optimistic woman actually be more likely to judge the lump as non-cancerous and therefore delay a visit to the doctor?” The last question would in fact lead to an opposite expected correlation, but the more general and important point is that the students' questions will likely not fit any variable-based model and Pearson correlation analysis. Why? Because to be perfectly consistent, models like the one above only permit statements about variables, such as “Relatively high values on the optimism scale are expected to be associated with lower reported delay values,” or “Optimism scores are expected to be negatively correlated Innovative Teaching if Lump else (P)articipant Woman senses lump in her breast Fig. 1. Integrated model for optimism as a cause averaging items somehow improve the scores? Students can be encouraged to ask additional questions about the questionnaire in an attempt to determine what the opti- mism values actually mean in the context of the nature of optimism itself. They will quickly realize that theoretical, not statistical, answers are required, and they will learn that making relevant observations about optimism requires a richer theoretical understanding of exactly what it is as a quality ostensibly inhering in persons. Proceeding with the example, nonetheless, and pre- suming that optimism is an abstraction drawn from the sensory and intellectual “data” of one's self, it is logi- cally related to judgments about the lump discovered in the breast. If the woman forms an abstraction of herself as optimistic, then she will judge the lump as cancerous; whereas if she forms an abstraction of herself as pessi- mistic, then she will judge the lump as not cancerous. judgment is distinguished from an abstraction by repre- senting the former as a regular pentagon (see Fig. 1), and again exactly how the woman makes a judgment is not spelled out by the model. Once either judgment is made, then it acts as an efficient cause (arrows in the model) for a judgment regarding whether a visit to the doctor is war- ranted or the lump can be ignored. It can be seen in the model, however, that the lump is judged as “could be” or “is probably not” cancerous. Are these the expected judg- ments? Would it be an important difference if the woman judged the lump as “most certainly cancerous” or “abso- lutely not” cancerous? Would such judgments be an effect of optimism or some other judgment about oneself, such as fatalism? Students may also ponder if, as suggested above and contrary to the study authors' expectations, the pessimistic woman should actually be the one who judges the lump as cancerous and who rushes to make an appointment with her doctor; after all, bad things like cancer are expected to happen. All of this is left open to debate for the students, because the point is that the inte- grated model in Fig. 1 is much more rich and complex Innovative Teaching An Initial Exercise in OOM Data Analysis In the variable-based model, optimism and delay were assumed to be continuous quantities, hence a Pearson's correlation was computed and reported by the study au- thors. The correlation was significant (r = –.18, p <.05, two- tailed). It is also important to point out that the authors transformed the delay responses due to radical skewness and a number of salient outliers. The p value of course in- dicates that, assuming the null hypothesis is true, and a host of other assumptions have been met (including ran- dom sampling, bivariate normality, and homoscedastici- ty), the probability of obtaining results as extreme or more extreme than –.18 or.18 is less than.05. The conclusion to be drawn in the standard null hypothesis significance test- ing procedure is therefore that the two assumed continu- ous variables are linearly related in the population. The observed correlation serves as the estimated magnitude of the linear association, and at –.18 is rather anemic. In the OOM software, students must first define the units of observation for the optimism and delay values. Rather than refer to optimism and delay as variables, in OOM they are referred to as “orderings” to reflect the understanding that nature is integrated and ordered in par- ticular ways. The integrated model is supposed to reflect this ordering, both in terms of how the optimism and delay observations are ordered or structured within themselves and how they are ordered to one another. As derived by the study authors, the optimism observations are ordered into 33 units (0, 0.125, 0.250, 0.375,… 3.875, 4.00). Recall these values were computed by averaging responses on a 0–4 scale to eight items on the Life Orientation Test ques- tionnaire. A simpler less precise ordering should be used to make the task of building and testing a pattern, as described below, more manageable for students. Twenty units work well, with scores ordered as 0–0.20, 0.21–0.40, 0.41–0.60,… 3.81–4.0. The delay observations can similarly be ordered into 20 units as shown in Fig. 2. The task here is similar to creating a grouped frequency histogram, but the overall Fig. 2. Delay and optimism observations grouped are reported in square brackets. Innovative Teaching Fig. 3. Expected patterns (darkened cells) for Using the imprecise pattern in Fig. 3b, the OOM software has an option for showing the observations in the figure as well. As can be seen in Fig. 3d, only about one quarter of the observations fit the expected pattern. With such a simple and elegant visual tool the students immediately realize using only an “eye that the prediction is a bust. Of the 135 women, only 33 observations (24.44%) were located in the expected crossed units. Without any equations, without any p values, without any statistical assumptions, and with- out any attempt to conjure an imaginary population, the students are also here reminded of the importance of accuracy in science. The current cause-effect is clearly inaccurate, and since the pattern is derived from the integrated model developed by the students, the integrated model is deemed inaccurate as well. This is the primary question to be answered in Observation Oriented Modeling: Is the model accurate? As an inten- tional representation of the causes and effects inhering in the women in the current example study, the model is largely a failure. Beyond the simple and elegant graph, the OOM software provides summary information regarding the accuracy of the model. Innovative Teaching is in most instances in OOM derived from a randomiza- tion test, which is becoming more widely recognized by quantitative experts as superior to traditional p val- ues (see Manly, 2006; Howell, 2007, Chapter 18) despite being lauded as early as 1969 (Winch & Campbell). The c-value for the breast cancer data is computed by randomly pairing the optimism observations with the delay observations. Specifically, the optimism val- ues for the 135 women are randomly shuffled. The pre- dicted pattern is then applied to the randomly paired optimism and delay observations and the PCC index computed. If this PCC index equals or exceeds the PCC index for the actual data (here, 24.44), then randomized data performed just as well as the actual data in the con- text of the model, which is not a desirable occurrence. This process of randomizing the data and comparing PCC indices is repeated a set number of times (1000 trials in the output above) and the results tallied. c-value is the proportion of instances in which the PCC index from the randomized data equaled or exceeded the PCC index for the actual data. As with the tradi- tional p value, then, a low value is desirable, although no rational person would put forth or adopt a univer- sal convention to determine “significance” (e.g., c <.05). The c-value is entirely secondary, whereas the inte- grated model and PCC index are primary. The c-value in the shown output for the breast cancer data is.35, indicating that for 355 of the 1000 trials, a PCC index of 24.44 or higher could be achieved by simply randomly shuffling the data. Not only is the PCC value of 24.44 unimpressive, but it is fairly ordinary or usual as well. Lastly, in the move from aggregates to persons, stu- dents can be asked to compare their interpretation of the PCC index to the Pearson's correlation coefficient. Recall the correlation between optimism and delay for the breast cancer data was –.18. Students can be challenged to interpret the correlation coefficient in a way that can be applied to the 135 women or to any given woman in the study. The correlation can be squared as well to convert it to a proportional statistic. Again, students can be chal- lenged to explain how accounting for 3% of the variance in one variable by the second variable applies to the 135 women or any given woman in the study. The point of asking students to interpret the correlation in this way is of course to remind them that their interpretation must be confined to the variables; i.e., the variances and covariances of the variables. There is simply no clear way to cross the bridge from the aggregate statistic (the Pearson's r) to the scores, or observations, themselves in a way that conveys the meaningfulness of the result for these women. By contrast, the PCC index is tied directly to the observations and how they are ordered, both in terms of their units and in terms of the expected pat- tern. Whether or not any given woman's observations are consistent with the integrated model and pattern analysis can be determined. Indeed, using options in Innovative Teaching batteries don't start, people who contract the influ- enza virus grow ill, and uncommon people cannot stay romantically tied for life. But are these statements true? Do they really express the way nature is? Can- not an old car without electronics be physically pushed and started by popping the clutch? Aren't some people immune to certain strains of influenza? With respect to romantic relationships, is it not also true that some- times “opposites attract?” Clearly, gaining an under- standing of nature will require more than the students’ day-to-day informal thoughts of causes and effects. Still, each question raised implies that an answer might be found, thus bringing students to the doorstep of sci- ence; for as Aristotle understood well, the business of science is causal explanation. With a statement such as “The effect of the SAD light in combating depression was significant,” the psy- chologist is also intending to express a causal relation; namely, the SAD light causes a reduction in depression. Such statements are the “stuff” of science, but in psy- chology these causal-sounding expressions are almost invariably statistical expressions. While students may naturally pair effects to their causes in the class exercise described above, in psychology the opposite of “effect” is not typically “cause” but rather “null statistical dif- ference.” In other words, an effect is said to be found when the null hypothesis has been rejected; in other the words, an effect is a statistically significant finding. The magnitude of the effect or the size of the effect moreover often refers to the average difference between groups or the proportion of variance shared by two variables. A useful exercise to drive home these points is to again rely on published data, contrived data, or data from workbooks or textbooks. These data can be used “as is” or manipulated in a way to shed light on what is meant by a statistical effect. An Exercise on Effect Sizes in the OOM Software In his chapter on t tests, David C. Howell (2007) reports an example data set of weight gain (or loss) for anorexic girls who received therapy for their eating disorder and those who did not receive therapy. The outcome variable is the amount of weight gain, in pounds, from the beginning to the end of the treatment or therapy. In the parlance of modern research design, the grouping variable (Therapy vs Control) is the independent variable and cause, while weight gain is the dependent variable and effect. Using a standard variable-based model, the relationship between the two variables can be shown as follows: Howell's original data were altered here so that the computed effect size would be medium according to Innovative Teaching Weight Gain Data, Example Control Group Weight Gain Clear One Clear Two −7.5 −7.5 −7.5 −7.0 −7.0 −7.0 −5.7 −5.7 −5.7 −5.1 −5.1 −5.1 −4.5 −4.5 −4.5 −4.1 −4.1 −4.1 −3.8 −3.8 −3.8 −3.8 −3.8 −3.8 −3.7 −3.7 −3.7 −3.4 −3.4 −3.4 −3.1 −3.1 −3.1 −3.0 −3.0 −3.0 −2.5 −2.5 −2.5 −2.2 −2.2 −2.2 −1.8 −1.8 −1.8 0.3 −0.3 −0.3 0.8 −0.3 −0.3 0.2 −0.2 −0.2 0.2 0.0 0.0 0.5 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.1 0.0 0.0 1.3 0.0 0.0 2.0 0.0 0.0 2.1 0.0 0.0 3.5 0.0 −6.0 3.6 −0.5 −0.5 3.9 −0.8 −0.8 2.9 −1.0 −1.0 point of this challenge is to remind students that psy- chologists are almost always working with arbitrarily defined, imaginary populations. The population means are therefore almost always non-empirical; i.e., they are values that cannot be obtained in actuality. The essen- tial lesson to be learned from discussing assumptions and populations is that null hypothesis significance testing is an extremely abstract, mathematical-statisti- cal routine in which the primary goal is to pursue (esti- mate) population parameters that rarely have any basis in lived reality. Turning to effect size, by declaring the t test “sta- tistically significant” and rejecting the null hypothesis, the conclusion is that the population means differ by some magnitude. Cohen's d represents a standardized estimate of that magnitude based on the sample means and pooled standard deviation. For the current data the value is computed as, Innovative Teaching 12% of their variance. Conveying what this effect size index means for the actual effectiveness of the ther- apy for the individual women in the study is impos- sible, which is why psychologists eschew the problem by using Cohen's arbitrary conventions to describe their effects as small, medium, or large. By contrast, analyzing the same data in the OOM software will remind students that causes inhere in the persons in the study, and that any discussion of effects must be relevant to these persons. The analysis begins by first defining the units of observation. The women are observed to be in one of two groups: the Therapy group or the Control group. These two units of obser- vation represent the cause, or they at least carry infor- mation about the causal forces effecting weight gain or loss (in Observation Oriented Modeling, an integrated model must eventually be worked out to truly under- stand the causes). The changes in weight are continuous quantities representing a change in this quality (heavi- ness) over time. The values in Table 2 range from –7.5 to 7.5 pounds for the two groups of women, and most of the women have unique values. In OOM, continuous quantities must often be grouped so that each unit (val- ue) is represented by more than one observation. Be- cause a population parameter is not being estimated, the notion of statistical power is meaningless in OOM. Observations can consequently be grouped without fear of losing power. While modern methodologists, for in- stance, advise strongly against dichotomizing data (Mac- Callum, Zhang, Preacher, & Rucker, 2002; Irwin & Mc- Clelland, 2003), such alterations are welcome in OOM if they advance the goal of identifying clear and meaning- ful patterns in the observations. The weights groups of women were thus organized into 8 units to –6.1, –6.0 to –4.1,… 4.0 to 5.9, 6.0 be encouraged to try different groupings after analysis and explore how they affect the results, be shown below. The Build/Test Model option in the OOM software is then used to attempt to bring the 8-unit weight ordered observations into conformity with the 2-unit group ordered observations using binary Procrustes rota- tion, a procedure that does not rely on means ances but is instead based on the observations them- selves (see Grice, 2011, Chapter 3). The analysis compared to the a priori pattern defining methods used with the breast cancer data above. Specifically, rithm examines the relative magnitudes of frequencies of observations and determines how every observation should be classified based on the predominant the data. Each observation is then evaluated according this pattern and judged as either correctly or classified. The results can be presented in a referred to as a “multigram.” To help students interpret multigrams, instructors should prepare at least two con- trived sets of observations, like those reported in Table Innovative Teaching Fig. 5. Multigram of clear pattern example. This pattern does not match expectation perfectly, as nine women apy group lost weight. These nine women were classified correct by the ad hoc algorithm used in the analysis they were distinct from the other observations. as misclassified by the algorithm even though they were in the Control group and lost relatively large amounts of weight. This example thus demonstrates that the algo- rithm works on the pattern of observations (data), not on the researcher's predicted pattern of results as in the breast cancer study above. Nine women in the Therapy group lost 6.1 or more pounds, which is entirely con- trary to expectation; yet, the algorithm counted them as correctly classified. Only two women in the Control group lost this much weight, and all of the other women in the Therapy group lost more weight than all of the women in the Control group. Based on the clear sepa- ration between groups of observations, the algorithm determined that all women who lost 6.1 pounds or more should be classified as belonging to the Therapy group, which unfortunately contradicts expectation. The anal- ysis is entirely ad hoc, however, making such outcomes possible. A high PCC index in the Build/Test Model analy- sis therefore only indicates that the observations a clear pattern. It does not mean that the pattern fits the scientist's expectation. The multigram must be examined – preferably in light of an integrated model – to decide whether the pattern is consistent with expectation or theory, and students will quickly realize that interpret- ing the effects (and causes) embodied in a multigram is clearly different from computing and interpreting tical effects embodied in d and η2. The extreme multigrams and their accompanying data in Table 2 can also be used to help students under- stand the chance value. Specifically, each student can be asked to shuffle randomly the weight change val- ues (“Clear One” in Table 2) that generated the mul- tigram in Fig. 4. Values can simply be changed in the Innovative Teaching Fig. 6. Multigram of randomized data with no clear The OOM software has detected a pattern that classifies an overwhelming majority of the women correctly based on the observations themselves…but what is the mean- ing of the result? Because population parameters estimated, the three-valued logic hypotheses above are not relevant. Without an integrated model to guide their thinking, students must examine the multigram in Fig. 7 and interpret the pattern. As can be seen, it appears the effect of the therapy is dichotomous. Sixteen women have lost weight, which is a terrible outcome given they are anorexic, while 14 women have gained weight, which is a positive outcome. The Control group as well appears to be made up of two groups, those who lost approximately 1 to 8 pounds and those who gained approximately 1 to 4 pounds. Cohen's d and η2 offered no information about these patterns, and to speak of d as indicating a “typi- cal” standardized difference between women in the two groups is recognized as misleading. A majority of women in the Therapy group actually lost weight, contradicting the t test conclusion supportive of the efficacy of the ther- apy – a disaster for the anorexia therapy. If the weight observations are represented (–0.1 to –8.0, weight loss; 0.1 to 8.0, weight meaning of the results for the women in the study even more clear. As can be seen in the multigram 8, the two groups of women appear highly similar with respect to the 2-unit weight observations; and again, a small majority of women in the anorexia therapy group actually lost weight. These 16 women in the Therapy group were therefore classified correctly by the a result entirely opposite of expectation. This shows, however, no ability to discriminate clearly the women in terms of simple gain or loss in weight, as indicated by a PCC index of 51.67 and c-value of 1. In every instance, randomized versions of the data yielded a PCC index of 51.67 or higher, indicating that the pattern in Fig. 8, although opposite of expectation, is not at all distinct. Innovative Teaching all data to a common binary form, referred to as deep structure, upon which the analyses are based. More- over, the goal in OOM is not to estimate abstract popu- lation parameters, but to accurately classify the obser- vations based on some a priori or ad hoc pattern (ideally, of course, an integrated model would be available for explaining the pattern). Accuracy is therefore the key judgment, and the PCC index is readily understood by students, other scientists, and laypersons as well. The c-value in OOM is also free of assumptions and rela- tively concrete compared to the traditional p value in null hypothesis significance testing, and is based on a firm foundation of reasoning dating to at least the 1960s (Winch & Campbell, 1969). Nonetheless, the inference to cause and effect is only aided by a high PCC index and low c-value. The basis and force of such an infer- ence ultimately rest upon the structures and processes in an integrated model. From Continuities and Variables to Entities and Their Qualities With regard to measurement, one of the three domains of quantitative training reviewed by Aiken, et al. (2008), students of psychology are universally taught to mem- orize a NOTICE article originally appeared in Innovative Teach- by Ammons Scientific LTD. It is reproduced in the in that form. Teaching was sold to SAGE Publishing Inc., and will not after December 31, 2015. permission, the authors are hereby issued a new Creative for the article to be included in Comprehensive In this manner, the article can continue to be accessed an active Open Access journal, operated now by SAGE should be cited as a part of Comprehensive Psychology listed in the side bar of this cover page. The original Oriented Modeling: Preparing students for research in the 21st century1 James W. Grice Oklahoma State University Oriented Modeling is an alternative to traditional methods of data and analysis that challenges researchers to develop inte- explanatory models of patterns of observations. The focus of research shifted away from aggregate statistics, such as means, variances, and and is instead directed toward assessing the accuracy of judgments on the observations in hand. In this paper a number of example data sets be used to demonstrate how Observation Oriented Modeling can be taught and graduate students. While the examples are drawn from the method of contrasting Observation Oriented Modeling with methods of research design and statistical analysis can easily be to examples from other sciences. Oriented Modeling (Grice, 2011) is a novel and intuitive approach for both con- and analyzing data in the social and biological sciences. Imagine a scientist, for a mother and child interacting in a laboratory. Countless observations regarding the mother and child in this artificial setting. Physical characteristics and weight can readily be measured, behaviors such as talking and touching observed, and qualities such as parenting style or child temperament may be other observable behaviors. The goal of the observation-oriented scientist is to explanatory model for the observations being made, and this goal is equivalent causal structure of the natural system under investigation. In other words, oriented scientist assumes that the persons, physical features, behaviors, directly or indirectly observed are organized or oriented toward one another and knowable ways because of nature's causal structure. no scientist is a passive observer or tabula rasa; therefore, observations are ordered the way they are in part because of the scientist's judgments. Plac- magazines in the room, for instance, may affect the mother-child interaction observations. Decisions must also be made regarding which physical behaviors are to be observed and how they are to be recorded, and methods for assessing qualities that are not directly observable. To insure that his or the causal structure of nature accurately, then, the observation-oriented also be oriented toward the many assumptions and decisions that go into As an example, consider the commonplace strategy among person- of summing responses from multiple-item inventories to obtain trait procedure assumes that personality traits are structured as continuous quan- be measured as such, even though no scientific evidence exists for this latter 2011). For the observation-oriented scientist this is an untenable posi- evidence must be gathered to support the continuity claim or the various mod- must be altered to provide a more accurate view of reality. The latter approach personality psychologists to return to the question “What is a personality to devise novel methods of observation in conjunction with their new models. the goal of constructing a model that accurately captures the causal nature, the observation-oriented scientist must furthermore steer away to James W. Grice, Ph.D., Department of Psychology, 116 North Murray, Oklahoma Stillwater, OK 74078, or e-mail ([email protected]). Teaching OOM / J. W. Grice We begin with an explanation of the impetus behind the development of Observation Oriented Modeling. Why Observation Oriented Modeling? The majority of students today are taught that psycho- logical research can be subdivided into three areas: sta- tistical analysis, research design, and measurement. As revealed in a recent review of doctoral programs in the United States by Aiken, West, and Millsap (2008), students receive more training in statistics than either research design or measurement, although Aiken, et al. are quick to point out that training in all three areas is inadequate. They also imply in their review that insuffi- cient training – coupled with declining numbers of quanti- tative psychologists – will have a decidedly negative effect on the overall quality of future research in psychology. It could in fact be argued that the predicted decline in qual- ity has already occurred, as recent scholarly critiques have highlighted a general lack of understanding among mod- ern researchers with regard to fundamental issues of sta- tistics (particularly the p value; Gigerenzer, 2004; Lambdin, 2012), research design (Lebel & Peters, 2011), and mea- surement (Michell, 2008, 2011). Highly publicized cri- tiques have moreover spotlighted psychologists' general neglect of fundamental principles of scientific investiga- tion, particularly the importance of exact replication and the value of critical discourse among colleagues (Lehrer, 2010; Alcock, 2011; Yong, 2012; Abbott, 2013). While these recent critical appraisals may seem to support Aiken, et al.'s call for more quantitative train- ing, other scholars have drawn attention to a deeper set of issues that must first be addressed. Joseph Rychlak (1985, 1988) argued nearly thirty years ago that psychol- ogists have a tendency to permit their methods to drive their metaphysics (their general view of human nature), whereas in proper scientific investigation methods must be devised according to a suitable metaphysics. David Bakan (1967) referred to this error more forcefully as “methodolatry” – the irrational belief that nature should conform to research methods rather than the other way round. In the domain of statistics, David Freedman, author of one of the most highly regarded textbooks on statistics (Freedman, Pisani, & Purves, 2007), argued for a return to simpler statistical methods with greater emphasis placed on sound theorizing and sleuth-like detective work (1991; see also Mason, 1991). Psychol- ogists' tendency to idolize complex statistical methods at the expense of sound reasoning was more recently dubbed “statisticism” by James Lamiell (2013), which fits well with other descriptions of modern statistical analy- sis as “sorcery” (Lambdin, 2012), “mindless” (Gigeren- zer, 2004), and “inevitable and useless” (Toomela, 2010). With regard to measurement, Joel Michell has argued persuasively that psychologists have, since the early the software 2011). can also be 1900s, avoided the fundamental issue of measurement; namely, the establishment of additive units of measure 2 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice is regarded as his or her score on a psychometrically sound intelligence test. The circularity of this alleged solution is obvious and unfortunate, but it is also neces- sary if the things themselves—as they truly are—cannot be known to us on some level. In contrast, moderate realism posits that things have essences (or natures) and that these natures are indeed knowable. Knowledge of essences may at times be intu- itive or even confused, but it is nonetheless a power of the human intellect that makes science possible (Wallace, 1996; Dougherty, 2013). In the simplest terms, imagine a scientist who wishes to study trees. Presupposed in said study is the capability of discerning trees from shrubs, various grasses, and ferns, to name a few plants, not to mention the millions of animals and other objects of nature that are not trees. Students can walk around cam- pus and examine one hundred different trees without ever once confusing a tree for a shrub or a tree for a per- son. Their ability to abstract from the countless individ- ual, unique objects the universal essence “tree” is what makes this exercise possible. Now, a student may encoun- ter a shrub, such as a crape myrtle, and wonder if it is a small tree; but this uncertainty in no way removes the stu- dent's certainty about all of the other trees encountered. With the crape myrtle, the student investigates further, moving from what is well known into a part of the world in which his or her knowledge is less certain. Through critical thought, careful investigation, and research the student may come to realize the crape myrtle is best clas- sified as a shrub because it normally has multiple stems. In moderate realism essences, or the “whatness” of things, are distinguished from their various qualities (or accidents). Whereas trees and persons are integrated wholes existing of themselves, qualities exist within trees and persons, e.g., some trees are evergreens whereas oth- ers lose their leaves in winter, and some people have blue eyes whereas others have brown eyes. Empirical psychol- ogy often busies itself with the study of qualities, includ- ing determining whether or not a given quality, like intelli- gence, is structured continuously and measurable as such. It is therefore critical to ask “what is” types of questions. If intelligence is not in fact a continuous quality, for instance, then all current tests of intelligence are misguided. While this statement may be dramatic, the premises of moderate realism require students to ask such difficult questions to push their knowledge further and further toward confor- mity with the ways things are truly arranged in nature. A useful class exercise in light of this discussion is to require students to answer the question, “What is hand- edness, and how do I observe (or measure) it?” Hand- edness is an excellent choice for this question because it is something all students understand and experience in their daily lives. Students may indeed offer an initial simple answer based on their experience, such as “hand- edness is the dominant hand a person uses to write intelligence, e.g., with, and it can be observed as ‘left,’ ‘ambidextrous.’ 3 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice TABLE 1 Components of Phenomenalism and Moderate Realism Moderate Realism Aristotle's Causality (Four Causes) Patterns of Observations Individuals Accuracy Generalizing via Theory & Replication Randomization & Permutation Tests Discrete or Continuous Quantities Entities and Qualities Natures of Things and their Qualities Systematic Model Building & Testing to the sections in the manuscript in which the concept via example analyses. low level in terms of atoms, or at the level of the chem- ical compounds, Adenine, Cytosine, Guanine, and Thy- mine (ACGT). Speaking generically, a cause is that which is necessary for an effect, and in this sense ACGT are the necessary material causes of DNA. Of course DNA is per- haps known best – or at least most readily recognizable – through its double-helix structure. It is primarily the dis- covery of this structure for which Watson and Crick were awarded the Nobel Prize in 1962. Speaking of structure, pattern, or shape incorporates Aristotle's formal cause in the explanation of DNA. In a deeper sense, referred to as substantial form, formal cause may also refer to the “what- ness” of the thing, or the essence of the thing that makes it to be what it is rather than something else. Efficient cause will most likely be familiar to stu- dents in the form of the classic S  R or S  O  R models or in the common variable-based models found in published papers, like the one shown here: in subse- the integers This model presents the efficient cause (personal hap- piness) as a source of change or production that precedes the effect (subjective happiness) in time. Although not rel- evant to material or formal cause, time is therefore criti- cal when invoking efficient cause. Time is also relevant, but not necessary, to understanding final cause, which refers to a thing's end. When theorizing about persons, final cause can refer to the goals or purposes that explain behavior. For instance, a person enrolls in a psychology class to learn more about himself or herself, or enrolls in a Ph.D. program with the design of becoming a professor in the future. Final cause also entails natural end states as well that are important in explaining processes. For instance, the human body maintains a fairly steady over- all temperature. When factors in the environment warm the body (efficient cause), such as when a Caucasian per- 4 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice lock Holmes or Father Brown, requiring skills of careful observation, patience, right reasoning, and intuition to solve the mysteries of nature (cf., Freedman, 1991). A simple mental exercise that will help students realize the distinction between Aristotle's causality and Hume's causation is to consider a scientist who develops a new type of fertilizer for corn. The scientist works with a com- puter program and with a molecular model kit to build models of the chemical compounds that will comprise the fertilizer. She then goes to the laboratory and manufactures the compounds and places a quantity of the fertilizer in a container. She writes instructions on the container detail- ing how the fertilizer is to be added to the soil to insure maximum plant growth and health. Next, consider a stu- dent working in the scientist's laboratory. Does the stu- dent need to know the molecular formulas for the com- pounds to test the fertilizer's efficacy on samples of corn crops? The answer is clearly, “no.” The student need only follow the instructions on the container to deliver the fer- tilizer in appropriate proportions to the samples. The stu- dent does not need to know the formal and material causal structure of the compounds. As the scientist begins work- ing in her laboratory the causes exist intentionally in her mind. Once the fertilizer is created the causes inhere within it. The point is that causes are something more than sim- ple Humean associations of sense impressions in the scien- tist's mind, they are instead “wrapped up” in the fertilizer's nature. This is what was meant above in stating that causes are bound to the natures of the things themselves. Biochemical models, atomic models, computer models of the mind, and similar types of models used to under- stand the structures and processes of nature are referred to generally as analogical models (Haig, 2013); and when such models are developed to be faithful representations of specific natural systems (usually in visual form) they are referred to as iconic models (Harré, 1976). In Observation Oriented Modeling, these visual or diagrammatic models are referred to as integrated models to highlight the meta- physical position that nature is integrated and intelligible (i.e., explainable; see Doroughty, 2013). Perhaps the most challenging aspect of Observation Oriented Modeling, and the most challenging prospect for future psychologists as well, is the development of integrated (iconic) models. Examples can be found in Grice (2011) and Grice, Barrett, Schlimgen, and Abramson (2012), and students should be encouraged to develop such models on their own or as part of a common class exercise. It is most effective, how- ever, to build a model for which actual observations (data) are available or can be obtained by the students. Students can collect data through an individual, approved project or through a common class project. Existing data can be provided by the instructor, requested from colleagues, obtained from open sources, or taken from printed mate- rial such as textbooks or instructional manuals. Integrated, iconic models are not widely used in psychology, hence the goal is to build such a model for the data. 5 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice with delay values (estimated r = –.20).” Again, compare these statements carefully to the ways students will natu- rally think about the problem. When thinking about opti- mists and the time it takes them to report to the doctor the students' natural focus is on the persons in the study, which is proper for discussing causes that firstly inhere in the women themselves, not in the variables and variable labels (e.g., “optimism”) constructed by the researchers. This statement is of course in line with philosophical realism, and the capa- bility of creating consistency between how students will think naturally about the world and how they will analyze their observations is one of the main advantages of Obser- vation Oriented Modeling and the OOM software. By focusing on the persons in the study, the integrated whole under consideration by the students is any given woman detecting an anomaly in her breast. The inte- grated model (a type of iconic model) therefore begins with a person, as shown in Fig. 1. In this case, a simple stick figure is drawn to represent an individual woman in the study. Every integrated model must begin with a judgment about the integrated whole or the system under investigation, whether it be a person, a baseball team, another type of animal, an atom, a bio-chemical process, etc. As shown in Fig. 1, a mind-body distinction is made by denoting that the woman senses the lump in her breast, presumably by using her hands and fingers. In her mind an abstraction is made on the basis of what is reported by the senses. This abstraction is denoted in the figure by the circle enclosing the word “lump” and a pic- ture of the lump. How abstraction occurs and whether or not it includes a visual component is not spelled out by the model in its current form. The woman must, how- ever, know that her fingers have detected an internal lump rather than, for instance, an external skin tag. With the lump identified, the model next incorporates optimism, and the question is raised, “What is optimism?” On the one hand, it may be considered a quality inhering in the woman and may therefore be subject to observation by another person. On the other hand, the actual observa- tions relevant to optimism in the study are self-report. It is more appropriate, therefore, to represent optimism as an abstraction made by the woman. Similar to “this is a lump,” the abstraction is “this (myself) is a pessimist.” The problem with this part of the model, however, is that it does not fit the optimism observations which are numbers computed by averaging values attached to a scale for judg- ments about the self (e.g., “I’m always optimistic about my future” rated on a scale with values from 0 to 4). With scores ranging from 0 to 4, is it presumed that optimism is a continuum? Moreover, is it presumed that the woman somehow erects a crude “optimism ruler” in her mind upon which she demarcates herself? How would such a process actually occur? By contrast, why cannot one state- ment, such as “Generally speaking, I consider myself to be an optimist,” be used and code it as “Yes” or “No?” Why use a 5-point scale (0 to 4) rather than a 9-point scale? Does 6 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice Lump Visit Optimistic Could be Doctor then Cancerous Lump is Pessimistic Probably Not then Ignore Cancerous P Visits Doctor or P Does Not Visit Doctor of planning a visit to the doctor. than the simple variable-based model above. Asking stu- dents to construct an integrated model will require them to think about the structures and processes under inves- tigation; in other words, to think in terms of causality rather than simply causation. Finally, outside of the woman's mind in the model, whether or not she visited the doctor is observed. Again, to be truer to the observations in hand, the model needs to be adjusted so that the number of days between detecting the anomaly and the first visit to the doctor is the “output” from the internal features of the model. Whether the structures and processes are directly observable or not, the observations acquired must reflect some commitment to them, which is partly why this overall approach is referred to as Observation Oriented Modeling. A From Statistics and Aggregates to Patterns and Individuals As Fig. 1 shows, the proper level of reasoning about causes and effects is at the level of the individual women in the breast cancer study. If the causal model is accurate, then a woman who self-identifies as an optimist should not delay in seeking an evaluation from a doctor, and a woman who self-identifies as a pessimist should delay seeking an evaluation. The actual data from this study were obtained, however, under the phenomenalism (see Table 1) inherent in the variable-based model shown above. Some ambiguity or difficulty must therefore be expected when attempting to analyze the data with the OOM software (Grice, 2011). Nonetheless, students will learn important lessons when they attempt to analyze the data from the breast cancer study.3 3 All of data files in both SPSS and OOM formats can be found on the Observation Oriented Modeling website: http://www.idiogrid. com/OOM/InnovativeTeaching. In addition, instructional videos can be found on the website for the examples discussed in this pa- per. These videos demonstrate how the analyses are conducted and interpreted using the OOM software. 7 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice goal is to build a model for explaining how the optimism and delay observations are to be ordered to one another. The groupings into units here is entirely ad hoc, whereas in original research how the observations are ordered will be determined by an integrated model. With the observations grouped into their respective units, the next goal for the students is to devise a way of ordering one set of observations to the other, and the simplest and most powerful way to introduce them to this idea is through the Pattern Analysis – Crossed Obser- vations procedure of the OOM software. By crossing the two orderings a matrix is created, as shown in Fig. 3. The question now is, if the optimism observations rep- resent the cause, and the delay observations represent the effect, how are the effect observations ordered to (or conformed to) the cause observations? In simpler terms, what is the expected cause-effect pattern? Here the stu- dents are working visually rather than with equations, although simple equations could be used to determine a predicted pattern for discrete or continuous values. Fig- ure 3a shows a pattern the students might suggest. The units are matched, in order, on a one-to-one basis with units ostensibly representing high optimism matched with units representing short delays in choosing to visit the doctor. Every woman in the study is expected to be found in one of the grey squares denoted in the pattern. The students can be further pressed to explain exactly why the observations (all of the women) should fall exactly in the grey units of the pattern. They may wish to build some imprecision or “wiggle room” into the expected pattern, like that shown in Fig. 3b, or they may opt for a more radical dichotomizing as shown in Fig. 3c. As debate ensues, students should be encouraged to return to the integrated model, which will again require them to take greater responsibility in thinking clearly and deeply about optimism itself (what it is, and how it must be observed) and exactly how it affects a decision to seek evaluation for an anomaly in the breast. into 20 units. Each asterisk indicates two persons, and observed frequencies 8 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice optimism and delay orderings (a–c), and actual observations for 135 women (d). Classification Results Pairs of Observations Classified According to the Defined Pattern(s) Classifiable Pairs of Observations : 135 Correct Classifications : 33 test” Percent Correct Classifications : 24.44 Randomization Results Observed Percent Correct Classified : 24.44 Number of Randomized Trials : 1000 Minimum Random Percent Correct : 13.33 Maximum Random Percent Correct : 34.07 Values >= Observed Percent Correct : 355 Model c-value : 0.35 pattern The Percent Correct Classification (PCC) index, as noted, is only 24.44%, and this is the primary numeric result to be considered. The PCC index is computed or derived for most all of the analyses available in the OOM software, and students should be encouraged to look for it first in the program's output. As can also be seen, a probability statistic can be requested and used in an entirely secondary role. This probability statistic is referred to as a chance value, or c-value, in OOM in order to avoid confusion with the p value commonly encoun- tered in null hypothesis significance testing. The c-value 9 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice the OOM software students can explicitly identify the 17 women whose observations fit the expected pattern. The readily interpretable PCC index can also be com- pared to the correlation value in this example to remind students that “statistical significance” does not indicate practical, clinical, or theoretical significance (Thompson, 2002), nor does statistical significance indicate that even a simple majority of observations fit an hypothesized model. Lastly, students can contrast this rich exercise of thinking about what the scores on the Life Orienta- tion Test really mean and how they are causally related to observations of delay, with the relatively sterile exer- cise of checking statistical assumptions and interpreting the magnitude of the observed correlation coefficient according to some arbitrary convention (e.g., “according to Cohen's conventions, a correlation of –.18 represents a small effect size”). The From Effect Sizes, p-values, and Parameters to Accuracy, c-values, and Chance From the perspective of Observation Oriented Model- ing, the word “effect” has been hijacked by phenom- enalism (see Table 1), or more specifically by statisti- cism (Lamiell, 2013). Because psychology has primarily adopted statistics, particularly null hypothesis signifi- cance testing, as its main approach to working with quantities (discrete or continuous), the word “effect” has taken on an esoteric meaning not generally under- stood by scientists, philosophers, or the general pub- lic. The word and its derivatives are of course power- ful and influential, as can be seen in statements such as: “The effect of the SAD light in combating depression was significant,” “The effectiveness of the psychoana- lytic treatment was negligible,” “The bystander effect is reliable,” “This effect has been observed across numer- ous studies,” and “The size of the effect was rather large.” These statements convey a sense of importance, certitude, and scientific understanding, even though no details of the studies from which they may have come are provided. The meaning of effect in modern psychol- ogy, however, typically has very little to do with certi- tude and is instead almost always embedded in a prob- abilistic or statistical point of view. An introductory class exercise regarding the mean- ing of the word “effect” requires students to write about any event in their lives and then explain why the event happened. Students may respond: “My car wouldn't start today because the battery was dead,” “I felt sick last week because I contracted the flu from my brother,” “My girlfriend and I broke up because we discovered we have little in common.” Normally, the students' statements will attribute a cause to each event and will express an understanding of nature that seems to run deep while also transcending the particu- lar events of their lives. Such thoughts express the way things truly are…the way nature is. Cars with dead 10 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice popular conventions and in the expected direction (i.e., the mean for the Therapy group will be greater than the mean for the Control group); yet, when examined at the level of the individuals, most anorexic women would actually keep their existing weight or lose weight, con- trary to prediction. The altered data are reported in Table 2, and the exercise to be performed by students is to analyze the data using the traditional null hypoth- esis significance testing approach and the OOM soft- ware. Weight is a continuous quantity, and the observa- tions are independent both between and within groups. An independent samples t test is therefore suitable for these data, and the hypotheses can be written in three- valued logic form (Harris, 1997), H 0 : µ therapy = µ control H A1 : µ therapy > µ control < − predicted H A2 : µ therapy < µ control. The μ's of course represent population weights in pounds. Students will find that the Therapy group gained an average of 1.35 pounds (SD = 3.72) compared to the Control group who lost an average of 1.23 pounds (SD = 3.25), thus supporting the predicted difference (95%CI = 0.77 ≤ μT – μC ≤ 4.38) and efficacy of the thera- py. The t test is statistically significant (t58 = 2.86, p =.006), at the.05 level, and the magnitude of the effect (d = 0.74, η2 = 0.12) is nearly large according to Cohen's widely used conventions (d = 0.80 is a large effect). Examina- tion of the data also reveals no salient outliers, but the distributions appear to be somewhat bimodal, particu- larly for the therapy group, when examined with his- tograms. Boxplots, however, will show only moderate skewness. Students should be required to list and evaluate or discuss the various assumptions of the independent samples t test: (1) random sampling or random assign- ment to groups, (2) continuous dependent variable, (3) null hypothesis is true, (4) p ≤.05 is a reasonable cut- point for statistical significance, (5) normal population distributions, (6) observations are independent both within and between groups, and (7) homogeneity of population variances. Many of these assumptions of course underlie the validity of the observed p value, a probability the students should be expected to under- stand completely. An excellent resource for checking students' understanding of the p value is Oakes' (1986; see also, Gigerenzer, 2004) 6-item quiz. In the context of discussing the p value students should also be chal- lenged to define the populations under consideration. Are they all women? Are they women from certain ethnic groups or regions? Are they women of certain ages? Are they women now living, or do the popula- tions include women from the past and future? The 11 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice TABLE 2 Data, and Randomized Data For Control and Anorexia Therapy Groups Therapy Group Random Weight Gain Clear One Clear Two Random 3.8 −3.2 3.2 −7.0 −4.1 −7.0 −3.1 3.1 −7.4 −4.5 0.0 −2.5 2.5 4.0 0.0 6.2 −2.4 2.4 2.4 6.4 7.0 −2.2 6.2 −6.2 −5.1 −0.2 −2.0 5.0 5.0 3.2 3.0 −2.0 7.0 −7.0 −5.7 −0.3 −2.0 4.0 4.0 −7.5 5.5 −1.9 5.0 5.0 2.2 −3.8 −1.9 2.0 2.0 3.4 2.4 −1.9 2.0 2.0 5.5 5.9 −1.8 3.8 3.8 2.0 4.1 −1.4 2.2 2.2 −3.0 4.8 −1.3 2.2 2.2 2.0 6.1 −0.2 3.0 3.0 4.0 −3.7 −0.1 4.1 4.1 2.0 −2.2 1.7 2.0 2.0 −2.5 0.0 2.3 3.4 3.4 0.0 0.0 4.1 4.1 4.1 −1.8 6.1 4.1 6.1 −6.1 7.2 7.5 4.2 4.2 4.2 −0.3 4.2 4.8 4.8 4.8 −3.1 3.1 5.1 5.1 5.1 −3.4 4.1 5.5 5.5 5.5 −0.8 2.2 5.5 5.5 5.5 0.0 −0.5 5.9 5.9 5.9 0.0 5.0 6.1 6.1 −6.1 0.0 −1.0 6.4 6.4 −6.4 0.0 2.5 7.5 7.5 −7.5 5.0 −3.8 7.2 7.2 −7.2 5.1 Xtherapy − Xcontrol 1.35 − ( −1.23 ) d= = = 0.74. 2 Spooled 12.117 The formula incorporates aggregate statistics and consequently has no direct bearing on any one of the women in the study or on any pair of women from the two groups. As has been made abundantly clear above, however, causes and their effects inhere in the things or entities under investigation – in this instance, the women in the two groups. Students should be chal- lenged to relate the value of d to the women in the study. Another popular index of effect size is η2, which the students should compute and again attempt to relate to the women in the study: η2 = t2/(t2 + df) = 2.862/ (2.862 + 58) = 0.12. This task will prove even more diffi- cult as the interpretation of η2 is one based on shared overlap between variables. The value of 0.12 indicates that the independent and dependent variables share 12 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice 2, showing clear patterns. Consider 60 weight observa- tions, for instance, in which all of the women in therapy gain two or more pounds while all women in the con- trol group lose weight or maintain their current weight (column labeled “Clear One” in Table 2). The resulting multigram is shown in Fig. 4. As can be seen, a multi- gram graphs the two frequency distributions side by side, and in this extreme example the two distributions do not overlap at all. A clear pattern is present in which the weight values can be separated into two groups. The Percent Correct Classification (PCC) index for these data is 100%, and the pattern clearly supports the effective- ness of the therapy. Every woman in the therapy group gained weight, and moreover gained more weight than every woman in the control group. As another extreme example, consider data (“Clear Two” in Table 2) that generate the multigram in Fig. 5. Again, the 8-unit weight observations can be classified into the 2-unit group orderings with remarkable accu- racy (PCC = 96.67%). The pattern in the multigram shows that: (1) all of the women who gained 2 or more pounds were in the Therapy group; (2) all who gained up to 1.9 pounds or lost as much as 6 pounds were in the Control group; and (3) most of the women who lost 6.1 or more pounds were in the Therapy group, contrary to expecta- tion. Based on the relative frequencies both within the 8 weight units and the 2 group units, the algorithm deter- mined that all of the observations should have fit this pattern and were therefore classified as such. As can be seen in Fig. 5, then, two observations (viz., two Control women who lost 6.1 or more pounds) were inconsis- tent with this overall pattern and were hence counted for the two (–8.0 to 8.1). Students can the initial as will or covari- is ad hoc the algo- pattern in to incorrectly simple graph Fig. 4. Multigram of clear pattern example. This pattern matches expectation with the individuals in the Therapy group gaining weight. 13 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice OOM software, making certain the students randomize a large number of the 60 values across both groups. In this way, some weights for the women in the Therapy group will be given to women in the Control group and vice versa. Once the weights are randomly shuffled, the students run the binary Procrustes rotation and examine the multigram and PCC index. Fig. 6 shows an exam- ple multigram from the randomized data (labeled “Ran- dom”) in Table 2. The clear pattern in Fig. 4 has now been lost, and the PCC index for the randomized data, 61.67, is much smaller than the PCC index for the “Clear One” data (100%). The randomization test in OOM sim- ply repeats this process a set number of times (e.g., 1000 trials) and determines the proportion of trials in which the PCC index equals or exceeds the original value. This proportion is the chance value, or c-value, and for the data in Figs. 4 and 5 it is less than.001, indicating that not one time in 1000 trials did the randomized data yield PCC indices as high as 100 or 96.67, respectively. A low in the Ther- c-value indicates that randomized versions of the obser- as because vations do not readily produce a pattern as clear or dis- criminating as the one obtained for the actual data. It informs the students of the distinctiveness or unusual- ness of the observed PCC index and accompanying pat- tern, and it does so without any assumptions. Recall from above the list of assumptions necessary for the accuracy of the p value for the independent samples t test (e.g., normality of population distributions, homo- geneous population variances, independence of obser- vations). None of these assumptions are required for the randomization test (Manly, 2006), and students will con- sequently find the c-value to be concrete, simple, and intuitive in comparison to the traditional p value. Turning students to the OOM results for the actual weight data in Table 2, the multigram in Fig. 7 reveals that a single discrimination point between the two frequency distributions cannot be made. Nonetheless, a pattern of bimodal, non-overlapping units is evident in the mul- tigram, permitting a large number of the observations to be correctly and distinctively classified, PCC = 86.67, c-value <.001. The OOM results are as follows: formed Classification Results Conforming (Effect) Observations Classified to Target (Cause) Observations Classifiable Observations : 60 Ambiguous Classifications : 0 Correct Classifications : 52 Percent Correct Classifications : 86.67 statis- Randomization Results Observed Percent Correct Classifications : 86.67 Number of Randomized Trials : 1000 Minimum Random Percent Correct : 28.33 Maximum Random Percent Correct : 78.33 Values >= Observed Percent Correct : 0 Model c-value :less than (1/1000); that is, < 0.001 14 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice pattern. Fig. 7. Multigram of weight gain data for Control and Therapy groups. This pattern partly matches expectation, but 11 women in the anorexia Therapy group lost weight and were classified correctly by the ad hoc algorithm because of their distinctiveness. are not Effect size indices accompanying results for most modern statistical analyses are understood to be aggre- gate-based or variable-based values that are difficult to interpret without the aid of established conven- tions, like Cohen's conventions. As such, they are not effects that are logically tied to the causes that inhere in the people in the study. Effect sizes are statistical, aggregate summaries of the data, and their magnitudes have no meaning for any given individual in the study. Through this particular class exercise, students realize that OOM analyses are not based on aggregate statis- tics. Not a single mean, median, or standard deviation has been invoked. The output across analyses is also highly similar, allowing students to focus on the pri- mary information; viz., the patterns and PCC indices. This streamlining is possible because OOM reduces as two units gain), then the becomes in Fig. algorithm, analysis also between Fig. 8. Multigram of weight gain data for Control and Ther- apy groups. Weight gain has been ordered into two units, and the pattern is opposite of expectation with a slight majority of women in the anorexia Therapy group having lost weight. 15 2014, Volume 3, Article 3 Teaching OOM / J. W. Grice An Exercise in “Measurement” Following modern psychometric theory, students are instructed at the beginning of this exercise that Need for Speed is a continuous latent variable that cannot be measured without error. More or less time can be spent on discussing the modern conceptualization of latent variables (e.g., see Bollen, 2002; Borsboom, 2008), but the pragmatic point is that multiple items must be con- structed from which scores (values) can be obtained. These scores are expected to vary from person to per- son for two reasons: (1) variation due to true individ- ual differences in Need for Speed, and (2) variation due to unknown factors (errors). The latter errors are often undifferentiated and expected to be normally distrib- uted and independent. This is of course the simple clas- sical true score model of psychometrics. Eight items are written to include in the question- naire, as shown in Fig. 9. Data are obtained from rat- ings on a Likert-type scale, i.e., arbitrarily numbered with values from 0 (highly disagree) to 6 (highly agree). The fictitious data for this exercise are reported in the Appendix. Students next conduct a standard item anal- ysis, examining the inter-item correlations, corrected item-total correlations, and Cronbach's α. The results

Observation Oriented Modeling PDF - Teaching & Research

Document Details

Tags

Related

Summary

Full Transcript