An Introduction to Second Language Research Methods (Griffee) - PDF

CHAPTER NINE DATA FROM OBSERVATION S uccessful observation requires something more than just sitting and watching. (Lynch, 1996, p. 108) In this chapter you will learn the difference between ordinary observation and research observation, the advantages and disadvantages of classroom observation, what observer roles can be taken, and some observation techniques. When addressing readers, I use the term TREE, she, or you. Preview Questions 1. Do you believe that “seeing is believing?” (Or, is its converse, “believing is seeing” true sometimes as well?) 2. Have you ever seen something that turned out not to be what you thought? 3. What are some aspects of the second language classroom that can be observed? 4. What are some aspects that cannot be observed? 5. Have you ever observed any aspect of teaching and learning that you thought was interesting and potentially worth researching? Introduction Even though observation is a basic source of human knowledge going back to Aristotle and Herodotus (Adler & Adler, 1998), ordinary observation is not always a reliable source of information. Eyewitnesses are often wrong when observing action, even at close range and in good light. Here is a well-known experiment demonstrating this point: A researcher is lecturing a class (often on the topic of observation), when suddenly two people making a lot of noise enter the room wearing dark clothes. In one scenario, a man enters the room with a knife in his hand being chased by a woman with a gun. After a brief time with much yelling, the man runs out of the room followed by the woman. Immediately, the researcher asks the class to take out a piece of paper and write down what just happened. In many cases, the gun-carrier is changed to the man, and it is the woman who is being chased. This experiment is taken to illustrate that not only are our observations not reliable, but that basic facts can be changed to fit our cultural and social biases. We often see what we expect to see. In spite of this shortcoming, observations can be used for research data collection (Nunan, 1992), and also for teacher training (Day, 1990; Wang & Seth, 1998; Williams, 1989). However, these two uses of observation (research vs. teacher training) are not closely related (Cook, 1989, p. 72). This chapter describes observation for research data collection, drawing from four literature sources: applied linguistics, evaluation, education, and anthropology. Applied linguistics literature discusses observation from a classroom research point of view; evaluation literature _________________________________________________________________________________ 177 Data from Observation looks at educational programs, but also other types of programs such as medical, drug treatment, business training, and government programs; education literature is primarily concerned with conducting K-12 curriculum research; and anthropology observes people in their local situations. Classroom observation defined Ordinary observation tends to be sporadic, random, and accidental. For example, we are walking down the street and see something happening in the park across the street; we are teaching a class and notice something interesting; we walk by the open door of our colleague’s classroom and hear something that gives us pause. These are all examples of observation, but they are random and unplanned. On the other hand, research observation—including but not limited to classroom research—must be systematic, intentional, and theoretical. Systematic means the observation is not occasional, but must be principled so that it covers the area or time of interest (Genesee & Upshur, 1996, p. 85; Hitchcock & Hughes, 1995, p. 234). Intentional means the observer has a reason for observing. These reasons may be specific or vague, but the observer must have something in mind when observing (Adler & Adler, 1998, p. 80). The reason or reasons for observing can, in principle, be thought of as research questions or emerging research questions. Theoretical means the observer is working with or looking for underlying principles. In fact, Long (1980, p. 12) asserts that an observational instrument implies a theoretical claim about second language learning and teaching. Stern (1989, p. 208) opines that by ignoring theory, we run the risk of being overwhelmed by details that we can’t explain. Putting these ideas together, we can define observation as the systematic, intentional, and principled looking, recording, and analysis of the results of our observation for the purpose of research. Consider again the example of the observation experiment mentioned above. Students were sitting in a class when the incident happened. Their reports were not reliable, and thus not valid, because the observers were not paying attention to the incident. Therefore, they could not have any principled awareness, much less a systematic way of observing. But let’s imagine that at least one of those students is again sitting in the classroom or maybe even teaching the class. The chase happens. During the next class, it happens again. The observer thinks, “I should pay attention to this.” The next time it happens, our observer takes careful notes, interviews others to see what they believe happened, and maybe prepares to video the class hoping to record the event. Now we can argue that these observations have, or are beginning to have, a fairly high degree of reliability and validation, and could constitute observation data for research purposes. What form might classroom observation take? Classroom observation for research generally takes three forms. First, the TREE can observe her own class or the class of another colleague. If the observation takes place in a class taught by another teacher, the observer has more time to observe and record the results of the observation. If the observation takes place in a class taught by the teacher herself, then there may be less time and opportunity to record the events using paper and pencil for notetaking. This means the physical recording of the data may occur at a later date. Second, the observation items may be open or closed. Open means the observation item does not specify in advance what to look at or record. Closed means the items are specified in advance. Open means the TREE is interested in _________________________________________________________________________________ 178 Data from Observation what is happening, but has not determined exactly what she or he is looking for. Closed means the TREE has decided what she or he is looking for. Third, the data gathered may be quantitative, for example, frequency counts, or qualitative, for example, verbal descriptions. These questions can help you decide which observation technique to use: Are you observing another teacher’s class or are you observing your own class? Do you know what you are looking for (closed), or are you starting with no predetermined categories (open)? Do you prefer data in numbers or words? Table 1 shows how the answers to these questions provide eight possible categories of observation. Table 1. Eight forms classroom observation might take ____________________________________________________________________________________________________________ Orientation Observation Collecting data consisting of category Numbers Words ____________________________________________________________________________________________________________ Observing Open 1. Things you notice 2. Initially you were not another you can count, but were looking, but as the course went teacher’s not looking for when on, you decided to look class you started your at this more closely. research. ____________________________________________________________________________________________________________ Closed 3. This is structured 4. Noting and describing high observation, for example inference categories such as, a checklist with are teaching tasks communicative? predetermined categories. You observe and check the number of times you see an example of the category. ____________________________________________________________________________________________________________ Observing Open 5. Noticing something 6. Observing a class with your own countable or fixed in no predetermined class space or time. category. For example, For example who audio recording comes in late and by a class, and analyzing how many minutes? it later. ____________________________________________________________________________________________________________ Closed 7. This involves 8. Using preset categories recording your data to observe students and using numbers or recording what you observe ticking a box or using words. An example category every time you could be noting how students notice a predetermined perform a certain task. event. ____________________________________________________________________________________________________________ _________________________________________________________________________________ Griffee 179 What are the advantages of observation? First, a researcher can directly observe a program (Rossi, Freeman, & Lipsey, 1999, p. 183) which means she can focus on areas of interest including specific issues, materials, activities which can be observed and evaluated (Fradd & McGee, 1994, p. 43). For example, when I was teaching in Japan, some of the local Japanese high school teachers claimed they used English extensively in their classrooms. I observed a class and found that the only use of English was about 8 seconds when the teacher read a few sentences from the board. In addition, if one teacher is observing another teacher’s class, observation data allows an understanding of the program not possible by relying only on interviews (Patton, 1987, p. 12). Observation can give a “you-are-there” point of view to readers not possible from other types of data (Patton, 1990, p. 203). Second, a TREE can search for categories of research and evaluation that make sense to participants (Alder & Alder, 1998, p. 89), or can be more focused by using predetermined categories (Allwright & Bailey, 1991, p. 3). Predetermined categories can result in numbers that can be treated statistically and analyzed (Hitchcock & Hughes, 1995, p. 235). Third, since observation can be done by a teacher of her own class, students are not threatened by outside observers, and can do their best work (Fradd & McGee, 1994, p. 43). Finally, we have an advantage combined with a possible disadvantage. While observation data can be rigorous when combined with other types of data (Alder & Alder, 1998, p. 89), to understand what tallies, numbers, or checkmarks mean, it may be necessary to interview the teacher and maybe the students to understand what they tallies or checkmarks mean (Hitchcock & Hughes, 1995, p. 238; Stern, 1989, p. 211). This is because a list of numbers derived from observation do not interpret themselves. Observation may show us what is going on in a classroom, but not why. What are the disadvantages? There are several disadvantages to observation. First, not everything can be observed (Patton, 1987, p. 12). We can see only the surface of things (Richards, 1998, p. 142). Second, maintaining observer openness is difficult because TREEs are familiar with teaching and classrooms (Bernard, 1994, p. 149). In anthropological fieldwork, for example, often the anthropologist visits the village as an outsider and observes. Teachers, on the other hand, are not outsiders; rather, we are villagers observing our own village. Because of their familiarity with classrooms, teachers have to be trained, or train themselves, to note and question the obvious rather than take it for granted and perhaps for that reason, not see it. Third, observer education is not easy to learn and difficult to implement. Gebhard, Hashimoto, Joe, and Lee (1999) note that in graduate level courses, when observing a class, students trained to make specific comments such as “I saw the teacher write new vocabulary words on the board,” regularly fell back to using vague and general comments such as, “it was a good class.” As a result, if observers are not well trained, they tend to overestimate student performance (Fradd & McGee, 1994, p. 43). _________________________________________________________________________________ 180 Data from Observation Fourth, if the observer is not the usual classroom teacher, the threat of research effect is always present (Allwright & Bailey, 1994, p. 3). This means that an outsider observer can cause usual classroom behavior to change. In addition, it is often hard for a teacher to find somebody to observe her class (Fanselow, 1990, p. 191). Fifth, observer bias is always present, since we all see and interpret events through the lens of our own experience, assumptions, and interests. Another observer may notice something we do not, or give a different interpretation. In addition, there is a high level of dependence on observer articulation (Alder & Alder, 1998, p. 104; Allwright & Bailey, 1991, p. 3; Hitchcock & Hughes, 1995, p. 235). In other words, some teacher-observers may be quite clear and articulate about what they see whereas other teachers may see the same thing and not be able to discuss it well. Data from observation does not explain itself. Sixth, interpretation based on observation data and analysis is not adequate for most research purposes. In estimating student improvement, observation alone is usually not sufficient, and other data collection instruments, such as tests, will be needed (Fradd & McGee, 1994, p. 216). What are some of the key issues to consider? One key issue is the role the TREE plays in observation. Long (1980) describes two situations an observer may take: unstructured participant observation and nonparticipant observation. In unstructured participant observation, often known as ethnography, the observer does not set out to test a particular hypotheses, but takes a regular part in the activities of the group. A nonparticipant observer, on the other hand, sits in the back of the room taking notes, and does not take part in the activities. Vierra and Pollock (1992, p. 224) also describe two roles they call participant observation and nonparticipant observation. Again, a participant observer takes an active role in the activities whereas the nonparticipant sits in the back of the room and takes notes. Patton (1990, p. 208) draws a continuum with detachment at one end and full participation at the other. I accept Patton’s continuum and adopt Bernard’s (1994, p. 137) categories: detached observer, participant observer, observer participant, and full participant. More detached More involved Detached Observer Participant Observer Observer Participant Full Participant Figure 1. Observation participation options from detached to involved. Detached observation would entail either observation through a one-way mirror, or having somebody video the classroom in order to watch the video later. Participant observation occurs when the observer is not a student and not the teacher. The participant observer would visit _________________________________________________________________________________ Griffee 181 the class several times so that her presence was not disturbing, but sit at the back of the room taking notes. Observer participation occurs when the evaluator is much closer to the class. An example of observer participant would be where the evaluator/researcher is also the teacher. The fourth category is full participant, or what Bernard (1994) calls going native. An example of full participation would be the researcher/evaluator being a student in the same program, for example a graduate student in a seminar evaluating the seminar for the class project. Patton (1990, p. 208) says deciding which role we play should be based on what is possible and what is helpful. The point is that a TREE has several options when it comes to observer roles. Another issue is the validation of observation techniques. TREEs in their teacher roles are accustomed to many of the observation techniques, but may forget that when using observation data for research, they must also present evidence that some care has gone into the collection and analysis of the data collected. To assist TREEs in the validation of observation data, each technique has validation suggestions. Following are eleven observation techniques. Each is described, positive and negative aspects are noted, and a validation plan is suggested. 1. Audio Recording Audio recording is the recording of all or part of a regular class. Procedures for recording may depend on the level of technology available as well as the number of students. An audio recorder with a microphone could be placed in a central location. For example, the audio player could be placed on the floor and only the microphone attached by long cord placed on a table or stand. Alternatively, small lapel mikes (called lav mikes) can be pinned on each student or placed around their neck on lanyards. Another option is to place a small hand-held recorder in a central location (Burns, 1999; Day, 1990). Positive aspects: While some recording equipment can be bulky, a small hand held recorder is very convenient. Once the recorder is switched on and functioning, one can concentrate on teaching, knowing that data is being collected. A recording picks up linguistic details that otherwise may go unnoticed. You can decide to listen to the recording only, or you can transcribe it, resulting in a transcription others can read and study. Because audio recording produces a permanent record, you may be able to count certain details and analyze them statistically. Negative aspects: Audio recording may work for a class size up to 20, but what about a class size of 40? Whole class activity is easier to record than group or pair work, although this could be dealt with by providing one recorder to each group. Equipment of any kind, even a small hand held recorder, is intrusive and may disconcert students so that what is recorded does not represent typical behavior. Also, audio recording alone does not show us body language or other contextual features. Transcription is time consuming, and when listening to a recording, it can be hard to determine who is speaking. Finally, most recording devices have a limited recording range, and anything out of that range is muffled or not understandable. Validation plan: Describe your recording schedule so the readers of your report can see how often and what days of the week you recorded. Tell them your recording procedures. What did you do first, second? Describe your equipment. Where did you put the microphone? How did your students seem to respond to being recorded? Did you tell them in advance? How did they _________________________________________________________________________________ 182 Data from Observation respond when you told them? About how many minutes did the recording last each time? What were you looking for? Is there any reason to believe that your students knew what you were looking for? If there were, you run the risk of their responses not being typical. On the other hand, maybe it didn’t make any difference if they knew or not. If you think it didn’t make any difference, tell us why. How did you define what you were looking for? Were the data expressed in numbers or words? If the data can be counted, what kind of reliability are you prepared to report? If you used raters, tell us how you selected them. Generally, you don’t have to go into detail--just a sentence or two like, “Two fellow teachers not directly associated with this evaluation were selected as raters.” 2. Checklist A checklist is a form with predetermined or closed categories, usually listed down one side of the page. Space is provided (often in little boxes) to mark the presence or absence of the predetermined category. The resulting data are frequency data. The observation task can be to check a yes/no category, or it can be to check or tick a box on a continuum (Day, 1990, p. 47; Fradd & McGee, 1994, p. 272; Genesee & Upshur, 1996, p. 86; Richards, 1998, p. 144; Rossi, et al., 1999, p. 226. Positive aspects: A checklist is simple to make and easy to use. It can capture details of the lesson you are evaluating. Low inference data, meaning the action observed by the evaluator does not require much inference, can give reliable data. Data from checklists work well with other data collection methods. For example, you could audio record group work activity, and later use a checklist to collect data, providing a written record that can be easily stored for retrieval and analysis. This way, the observed data are organized and easy to analyze. Negative aspects: A checklist works well with low inference categories but not as well with high inference categories. As mentioned, low inference means the action the evaluator is observing is clear and easy to understand, and not much inference is required (e.g., How many questions did the teacher ask?). Its opposite, high inference, means the action requires considerable interpretation (e.g., Is the native language of the students valued by the instructor?). High inference categories are hard to determine with a simple yes/no checklist item, and require corroborating data. Low inference data may be reliable, but may lack a clear connection between what is observed and what is measured. Unless a TREE records what is observed, she has no way of going back to the original situation to verify the observer’s decision. In other words, readers have to accept the researcher’s word. The key to using a checklist is being able to establish a clear link between the predetermined category, the observed activity, and research questions. Validation plan: It’s important to report how many categories were used, and what they were. In an appendix, you can give the actual checklist form so the reader can see how it was arranged. Tell readers why you decided to use those categories. How do you define each one? Did you show your categories and definitions to a colleague for his/her critique? How many colleagues? Did you ask them if any of the categories overlapped? How did they respond? If they offered suggestions, what did you do? How many times did you administer the checklist? It could be important to mention this, because if you administered it five or six times, it would probably give more consistent results than if you did it only once. What happened when you first used your checklist? Was it easy? Do you think you left anything out? If you think that happened, tell _________________________________________________________________________________ Griffee 183 your readers why. Did you conduct a pilot study with a group of students similar to those in your study to familiarize yourself with how the checklist worked? This pilot data can be reported as validation information. Piloting always strengthens your validation position. Report what you did for your pilot, what you learned, and what changes you made as a result. Were your final results supported by any other data collection techniques? If so, state which ones. 3. Teacher Hand Held Digital Recorder Hand held digital recorder (or the older technology but sometimes available tape recorder) means holding a small, battery powered recorder, and speaking directly into it. The purpose is to record observations during or soon after the time of observation. The digital or taped record can be listened to or transcribed later (Burns, 1999, p. 88). Positive aspects: These electronic devices are available and are relatively inexpensive to buy. They are small enough to fit into pocket or bag. They use batteries, allowing freedom from cords and plugs. Some tape recorders use a standard size cassette that can be played back on a larger set when listening or transcribing. By holding it close to your mouth, the resulting recording can be loud and clear. Your recorded notes are private--for your use only, or they can be shared. Audio recording allows a permanent record to be made at the time of observation or soon after, which means the audio notes may contain details that might have been forgotten as time passed between the classroom observation and recording your observation. The recording can be played back anytime, for example, in your office, or listening in a car while driving home. You can use the recording as the basis for writing a more detailed, written record. Negative aspects: Some teachers are not comfortable talking into a recorder, even in private. This discomfort is magnified by using the recorder in public, for example, during class, after class, or while walking down the hall. Also, sooner or later, recording equipment can fail, or batteries may die and you won’t have extra batteries available. Validation plan: The raw data is made up of the recording of yourself talking about what you observed in class. This means you have anecdotal or unanalyzed data. Your words may contain a mixture of descriptive data (what you saw) and evaluation or interpretation data (how you felt about what you saw). Behind your evaluation/interpretation lie values. To identify a value, ask yourself, “What would I have to value in order to feel about X the way I do?” Behind your values is your teacher theory. You can use your raw data to unravel description from interpretation, and then from the interpretation, to reveal your values and your personal theory. This would enable you to understand not only how you feel, but why. Another validation approach is to listen to your recording and take notes, or listen and do a whole or partial transcription. A transcript provides raw data, which may be read and coded. Coding is reading the transcript and deciding what themes are present, also known as units of analysis. List the relevant data under each code. This process is called data reduction because any raw data not directly relevant to a code is disregarded. Working with your codes and data, list your interpretations. Writing up the process of going from your transcription to your interpretation allows your readers to trace your steps, and this constitutes validation evidence. Again, the idea is to document how you went from the raw data to an interpretation. _________________________________________________________________________________ 184 Data from Observation A third validation approach is to listen to your recording several times and reach conclusions. These conclusions are considered weak and unsupported because the readers have to take your word for it, and can reasonably ask, “Why we should believe you--couldn’t you be mistaken?” You can strengthen your position by stating how many times you listened to the recordings, that you were able to get a peer to also listen to them, and that your peer either agreed with your interpretations or came up with similar results on her own. 4. In-Class Observation Notes In-class observation notes refer to any type of written documentation made by an observer other than the teacher while the class is meeting. Notes written after the event are referred to as a Teacher Diary. Much has been written to describe the strong and weak aspects of this type of observation documentation, which is a mainstay of ethnographic observation (Adler & Adler, 1998; Allwright & Bailey, 1991; Bernard, 1994; Evertson & Green, 1996; Fradd & McGee, 1994; Genesee & Upshur, 1996; Gebhard, 1999; Guba & Lincoln, 1989; Lynch, 1996; Patton, 1987, 1990; Rossi, et al., 1999; Vierra & Pollock, 1992). Positive aspects: The evaluator can observe an authentic educational environment, for example, a classroom. Course goals and objectives can be verified. The observer does not have to rely solely on handouts or secondhand reports of what is happening. Specific materials, activities, and procedures can be observed and evaluated. Aspects of the course the teacher might miss, ignore, or take for granted can be observed and noted. Observation data can be words or countable items of interest, such as who speaks or where people sit. Nonverbal communication and behaviors, for example, the way people dress or the way they use physical space can be observed. Aspects of the classroom that might be meaningful, but are often overlooked--the bulletin board or room decor--might be observed and noted. Negative aspects: Observation using in-class notes is labor intensive. In addition, not everything can be observed in a clear and obvious way. For example, high-inference categories, such as student attitudes and feelings have to be carefully defined, and what you accept as evidence carefully spelled out. Any instrument gives just one view. Observation data by itself is probably not enough, and you may also need numerical, quantitative data, or qualitative interview data. It is hard to learn how to be an objective observer and not mix descriptive observation with evaluation and opinion. There are many sources of error, and Evertson and Green (1986, p. 183) list seventeen of them, including problems with both rating scales and ethnographic observation. Another consideration is that you need to negotiate an observation schedule with the teacher that allows for observation on a systematic basis. Observing only a single class leaves you open to charges of unreliability, that is, the claim that what you observed was an unusual occurrence and only happened one time that semester, which happened to be the day you were there. If you observe more than once, but you observe on the same day of the week, you run the risk of skewing results because Friday classes may be different from Monday classes in small but important ways. You also have to decide what role you want to play while observing. For example, while observing, what would you do if the teacher or a student turned to you and asked a question? Validation plan: If you have numerical data, see the validation plan for the observation technique called Checklist. Assuming your data consists of words, ask yourself if your readers can follow the actual sequence of how the data were collected, processed, and transformed for specific _________________________________________________________________________________ Griffee 185 conclusion drawing. Look at the validation strategy for Teacher Hand Held Digital Recorder to see what you can apply. Tell your readers the context of your observation: How often and when did you observe? Where did you sit? What was your level of class participation? What were the students like? What did your notes look like? How did you organize them? How did you go from observation to interpretations? What was your role in the class? In relation to the teacher, were you a peer, a supervisor, trainer, friend, or some other role? 5. Peer Observation Peer observation refers to two teachers observing each other’s classes. Both teachers typically adopt a participant observation stance, which means they sit in the back of the room, observe, and take notes. A feedback session may be held later. Since they are peers, evaluation of each other is not an issue. The purpose of peer observation from a course research perspective is to gather evidence from a viewpoint not otherwise possible. Peer observation is typically discussed in the context of teacher education (see for example Cosh, 1999; Fanselow, 1990; Gebhard, et al., 1999; Gebhard & Ueda-Motonaga, 1992; Richards, 1998, p. 147; Wang & Seth, 1998; and Williams, 1989). Positive aspects: Peer observation makes collegial research and evaluation possible. This means that two teachers working together can research the same phenomenon, but in each other’s class. Also, it may be possible to use another class as a mirror to evaluate an issue. For instance, Teacher A can have an issue of interest and use observation of Teacher B’s class to illuminate that issue. Teacher B may or may not even be aware of what Teacher A’s issue is. Negative aspects: Peer observation is not common, so it may be hard to find a willing partner because some teachers might see it as potentially threatening. It can also be difficult to arrange schedules. The key to participating in peer observation is to have a clear idea of your purpose, and for you and your peer to agree as to what you will do, when you will do it, and how you will share information. More than likely, your data will take the form of written notes. Validation plan: Refer to In-Class Observation and Teacher Hand Held Digital Recorder validation plans. 6. Proformas As seen in Figure 2, a proforma is a grid or a form to fill in that can hold a type of class performance, including descriptive as well as interpretative data (Burns, 1999; Day, 1990). Time Period Category 1 Category 2 Category 3 Category 4 Figure 2. Example of a Proforma _________________________________________________________________________________ 186 Data from Observation Categories of interest are written across the top of a piece of paper or card. Examples of column headings could be Date, Question or area of interest, Descriptive Notes, and Interpretation. Descriptive notes can be entered during or after class. If entered after class, the proforma functions as a teacher diary, journal, or log. Interpretative comments could also be written after class. Positive aspects: A proforma can be an all-purpose data collection instrument. Using just this one instrument, you can decide on a category, collect descriptive data, and engage in interpretation. The key is the categories, which ultimately have to be explained. Deciding which categories fit your situation may require trial and error to find the ones that give you helpful data. Questions or categories can be predetermined; this preparatory activity helps to decide what to note. Alternatively, categories can be relatively open and vague, depending on how clear you are about what you are looking for. Proformas can be put on cards for easy filing. Proformas can be taken into class, and there is space to jot notes on areas of interest. Proformas may take less time than the Teacher Diary because notes are shorter and more to the point than diary or log entries. Negative aspects: One problem is that if you are observing your own class, it may be difficult to take notes while teaching. Another potential problem is forgetting to date the proforma, or making notes so cryptic you can’t understand them later. Yet another problem is not interpreting your data, so that later you have data that you don’t know what it means because much of the context for the data is forgotten. A potential weakness of proformas is a tendency to use subjective and ill-defined categories. Validation plan: You should be sure to define the categories or at least explain what they mean. If you are collecting verbal, descriptive data, use validation strategies similar to the ones discussed in In-Class Observation and the three strategies discussed in Hand Held Digital Recorder. 7. Scribbles These are short and quick observation notes jotted down while teaching (Burns, 1999, p. 85). Positive aspects: Notes made during the class have an authenticity of being made by someone who was there and experienced a behavior or activity firsthand. Scribbles can be used as the basis for recalling and writing more complete diary or proforma entries. Negative aspects: It is difficult to teach (an active undertaking) and take notes (a reflective undertaking) at the same time. It may difficult to keep scribbles in a systematic way. If you don’t use your scribbles as the basis for more complete notes right after class, they may be hard to read or understand later. Some teachers have trouble taking notes while teaching because they tend to be peripatetic, moving around the classroom as they teach. On the other hand, it might be possible to schedule brief periods of time for writing. Writing could be done in a small pocket notebook, index cards, sticky notes, or in a text file on a mobile phone, tablet, or laptop. Validation plan: Keeping a record of each time you gather scribbles could show your observations to be intentional and systematic, two qualities that increase validity. As in Proformas, if you have predetermined categories, you should tell your readers what they are, why you are interested in them, how you selected them, how you define them, and how they are related to your evaluation _________________________________________________________________________________ Griffee 187 purpose and research questions. Probably the best idea would be to use scribbles as the basis for another observation technique such as Teacher Diary. 8. Seating Chart A seating chart can be made either by an observer sitting in the back of the room or by the teacher. It could be a single sheet of paper with boxes for each student showing where each student is sitting. If you allow students to sit wherever they please, you might find revealing patterns (Burns, 1999, p. 106; Day, 1990, p. 49). Positive aspects: A seating chart could be helpful to your research if you are investigating a topic such as student social relationships. While you are making a seating chart, it would be easy to use the chart to incorporate other features of the room including the number and location of all objects, such as windows and chalkboards, and perhaps the measurements and description of the classroom. This information is helpful when it comes to writing your report because you can describe the site with more detail. A seating chart can be combined with a video record to show what happened and where. Negative aspects: There are two kinds of data commonly provided by a seating chart: Where students are sitting and a record of what students are doing, for example, how many times each student initiates a speaking turn or other action. While sitting in the back of the room, an observer can not only make a seating chart, but also have the time to indicate student interaction. A classroom teacher can also make a seating chart, but may not be able to also conduct a class while using such a chart to record interaction. Validation plan: To validate seating chart data, you can create seating charts on multiple occasions to show that seating patterns were consistent. Students tend to sit in the same place, but if for some reason there is no consistency in where students sit, they this observation tool should be not be used. If you draw conclusions about what certain seating patterns mean; theory, principles, or prior research would be needed for explanatory purposes. 9. Structured Observation Structured observation is classroom observation using previously defined categories. In some cases, an observation form is given to the observer with instructions to note when, how often, or examples of classroom activities that in the observer’s opinion exemplify the category. This implies rater training, a theoretical basis for the categories, and instrument validation similar to that used by questionnaires and tests (Allwright, 1988; Chaudron, 1988, 1991; Cook, 1989; Fanselow, 1987; Galton, 1995; Long, 1980; Moskowitz, 1967; Nunan, 1992; Spada, 1990; Spada, & Lyster, 1997; Weir & Roberts, 1994). Positive aspects: There are many structured observation instruments available, for example Long (1980) reviews 22 such instruments. Two of the better known are: FOCUS (Foci for Observing Communications Used in Settings) (Fanselow, 1987); and COLT (Communicative Orientation of Language Teaching) (Spada, 1990; Spada & Lyster, 1997). Despite criticism, there remains continued interest in this type of data collection. When using low inference categories, relatively objective and reliable data can be collected. Data collection may be recorded using audio or video or in real time, and resulting data can be analyzed. Negative aspects: The main danger is lack of validation evidence, that is, the chosen categories _________________________________________________________________________________ 188 Data from Observation may exclude critical features from the data. The categories may appear relatively simple, but even seemingly simple categories are often ambiguous and capable of multiple interpretations. In addition, while one might find patterns, it does not follow that what is infrequent is insignificant. Quantification of data into numbers cannot explain what those patterns meant to the participants. Coding categories tend to focus on what a teacher says and does, which assumes that the teacher controls classroom interaction. This may or may not be the case, for examplea structured observation instrument may not be able to capture what goes on in pair or group work. Concentrating on the teacher may also limit the usefulness of data from observation restricted in the sense that nonverbal communication is not checked. Also, raters must be trained and retrained from time to time. If an observer has a strong stake in the outcome, observer bias is possible; the presence of an observer has some effect. Although observer effect can be overcome to some extent by having neutral observers, being unobtrusive as possible, and having a sufficient number of observations. Since the data gathered are typically a count of certain features that provide frequency data, some sort of statistical analysis and presentation is necessary. Those who aren’t trained in statistics will consider this a weakness, while those who appreciate statistics will consider this a strong point. The purpose of your research is key, and any observation instrument must serve the research purpose. Much of the criticism of structured instruments comes from ethnographically- oriented researchers who point out the problems of using predetermined categories. These same criticisms can be made of any predetermined category instrument as well, such as tests or questionnaires. For example, one charge is that using a predetermined category tends to focus the observer’s attention in a specific direction, and in doing so, the observer may miss something important that falls outside the category. The trouble with this criticism is that while it is true, in fact, the point of focusing your attention is precisely that, to focus your attention. If you don’t know what you are looking for, you shouldn’t use this type of instrument. But it is unfair to criticize a predetermined instrument for being predetermined. It is like criticizing an apple for being an apple and not an orange. Structured observation instruments are the Rolls Royce of the observation world, and as such, they are mostly used for research. Regarding the famous car, most of us have never driven-- let alone owned--one, and probably something similar can be said about using a structured observation instrument. First, you have to find or adapt the one you want to use, then, you have to be trained or train yourself to use it. Finally, you have to work out a validation plan. This type of observation instrument may require finding a mentor experienced in its use, and also requires familiarity with the literature. Be prepared to spend time in preparation, and be prepared to conduct one or more pilots, in which you not only train yourself in the use of your instrument, but also gather data for its validation. Essentially, you have three choices in using a structured observation instrument: You can make your own, you can use an existing instrument, or you can modify an existing instrument to suit your needs. Validation plan: Before you even begin, create a log to record your procedures. Include as much detail as possible; most teachers don’t gather enough of this information. The rater training should be documented in detail, including who was selected, why, and how they were trained. _________________________________________________________________________________ Griffee 189 This is true even if you are the rater. Be prepared to deal with the charge of observer bias, since the presence of an observer will have some effect. Inter-rater reliability is the most common as well as the recommended form of reliability measurement. Another major issue is validation of the categories. Validation involves establishing a link between the category, a theory or body of knowledge that describes and defines the category, and your research purpose. You may want to consider advanced statistical procedures such as factor analysis or structural equation modeling as well as case studies to validate your data. See Weir and Roberts (1994, p. 173), who discuss six validation strategies. 10. Teacher Diary A teacher diary, as used in this chapter, is a log or journal written primarily by a teacher after a class session is over. This document is based on observations made during class (see Bailey,1990; Bartlett, 1990, p. 209). When evaluating another teacher’s class, a teacher diary serves primarily as a chronology of events and a repository for reflection. When evaluating the TREE’s own class, this diary includes those two functions plus it takes the place of in-class field notes. In this way, a teacher diary functions as a source for descriptive data, and as a source for teacher reflective data. Positive Aspects: A teacher diary is a term made popular in the applied linguistics literature for ethnographic field notes, and as such, comes from a long and accepted history. It provides a written document of what happened. If you don’t write it down, your recollection of events will blur and merge with other memories until you can’t remember any details; details give your report credibility. Negative Aspects: Teacher diary data is recalled, rather than actually observed data. That means the sooner you make your diary entry after the class, the better. It is easy to put off writing (especially when you feel you have no time, tired, nothing special happened in class, or too many other things to do when you get home). It is also hard to separate descriptive from evaluative comments. Comments such as “Good class today” are evaluative rather than descriptive in that they don’t tell readers what happened to make the class good. You have to train yourself to write the details of what actually happened, as well as how you feel about it. Teacher feelings are important, but they are not the same as detailing what caused them. For a teacher evaluating another teacher’s class, a diary is helpful; for a teacher evaluating her own class, a diary is indispensable. Without a diary, you run the risk of using only quantitative data gathering instruments such as tests and questionnaires, which can give reliable and valid data, but will make interpreting that data difficult. All data from your diary, even your reflections, have to be considered raw data. That means all entries have to be analyzed and interpreted. Validation plan: If you are observing your own class, your teacher diary can become a valuable observation instrument, perhaps your only observation instrument. For that reason validation must be carefully considered. It might be helpful for you to arrange your diary into four sections: history, emotional reflection, descriptive data, and reflective data. To offer validation evidence for your history, you should make frequent diary entries, even if you have to force yourself. At the end of the evaluation period, write a history of the course as completely as you can. Consider collecting documents, such as your local or school newspaper, to _________________________________________________________________________________ 190 Data from Observation look for external events that might have influenced your students during the observation period. To offer validation evidence for your emotional reflection, keep a record of your high and low moods, what made you happy, depressed, elated, satisfied, and so forth. This can be an indication as to your bias, or to put it more positively, your belief system. Try to match high emotional states to events that caused or at least seem related to your high mood. Similarly, try to match events to low moods. Can you state what might be your bias? If so, you can report this as part of your validation data in that it helps your readers understand and interpret your findings. Do the same thing for your teaching beliefs. How do events and bias indicate your beliefs? Your personal theory is a filter through which all observations pass. It is helpful to report your understanding of this so readers can take it into account. To offer validation for your descriptive data, if you used Scribbles, Checklist, or Proforma, use the validation strategies mentioned there. Describe in detail how you collected your data. When did you write up your observations? Take readers through your process step by step so they can follow and understand your data collection process. Search your data for themes. Code the themes, that is, give each theme a name, and see if you can relate your themes to your research questions. If you don’t have satisfactory research questions yet, try to use your themes to create new ones or sharpen your present ones. Ask colleagues to look at your data, your themes, and your research questions to see if they can follow your logic. If they can, report this as validation data. If they can’t, work with your themes and RQs until they can. To offer validation for your interpretations, ask a colleague to look at your interpretations (or hypotheses). Are they plausible? If not, what additional evidence would you need? Find a critical colleague who is willing to look at your data to see if they can find fault with your interpretation and give you an “alternative hypothesis.” Don’t pick your best friend or someone who tends to agree with you or share all your values. Finally, see if you can match your findings (your interpretations) with findings from other sources of data, for example tests or interviews. This data triangulation strengthens your findings. 11. Video This refers to the video recording of all or a portion of a class. The purpose of video recording a class is to provide data to answer a research or evaluation question. For discussion on the use of video see Burns (1999, p. 94), Day (1990, p. 46), Galton (1995, p. 502) and Woods (1996, p. 37). Positive Aspects: Small high-quality video cameras are now relatively inexpensive and available, and many cell phones now include a video function. Video data can reveal things we might not otherwise notice. Video can give a detailed naturalistic view of life in a classroom, and a sense of being there. Negative Aspects: Cameras don’t see everything, only what they are pointed at, and they can’t be pointed at everything in a classroom all the time. You may have to buy the recording media, get the camera, and aim the camera. Video gets harder as the number of students increases. If you video a class for an hour, then you have to spend an hour watching the video, and perhaps several hours transcribing the sound track, which may not always be of high quality. In addition, video can be intrusive, and we run the risk of student reluctance to behave in a normal way in the _________________________________________________________________________________ Griffee 191 presence of a video camera, including students playing to the camera. There are certain ethical considerations given the fact that students may be recognizable on the tape. You may have to justify to an institutional research board (IRB) why you need to video students. If your students are under 18 years of age, videoing without parental or guardian permission is likely illegal. And yet, video is a technology that is here, can be used, and for some kinds of research, may be required. For example, if interpretation of body language is part of a research question, a visual record may be necessary. A school or department with video equipment, a budget, and staff to operate the camera would be helpful. You might want to try a pilot study to see if video is helpful, worth the trouble, and provides the kind of data you want. Validation plan: If your data are countable, adapt the validation plan from Audio Recording. If your data takes the form of words, look at how this type of raw data are validated in technique number two, Teacher Hand Held Recorder. In conclusion, the purpose of this discussion of observation techniques is to demonstrate that many observation techniques are available, that they can be used alone or in conjunction with others. Regardless of how they are used, they all must be validated. Observation techniques grouped by use In Table 2, the eleven observation techniques discussed in this chapter are grouped according to the eight categories described in Table 1. The number in parenthesis behind each technique is the number of its listing. Each of the eight groups of observation techniques is situated in a particular research situation. If you are playing an outsider role, consult sets one through four. If you are playing an insider role, evaluating or researching your own class, consult sets five through eight. Decide if you have a predetermined category (you know what you are looking for), here called closed, or if you do not have a predetermined category (you aren’t sure what exactly you are looking for), called open. Decide which observation techniques fit your situation and best answer your evaluation or research questions. Observation techniques, both in the classroom and in other research situations, are time-honored data collection instruments. However, observation of the type necessary for research and evaluation requires a certain amount of attention that goes beyond what teachers typically do. Not every observation technique fits every situation. A major difference seems to occur whether you are observing another teacher’s class or your own. Sometimes validation reporting takes the form of counting and statistical analysis, but often it does not. In many classroom research studies, validation data of the data collection instruments are not reported. Whether we use quantitatively or qualitatively oriented instruments, we are obligated to report how we got the data, how the data were analyzed, how the data were interpreted, and what steps we took to investigate the integrity of our data-gathering process. _________________________________________________________________________________ 192 Data from Observation Table 2. Placement of Observation Techniques in Eight Categories ________________________________________________________________________________________________________ Use Observation Data collected in terms of category Numbers Words ________________________________________________________________________________________________________ Observing Open 1. 2. another Seating Chart (8) In-Class Observation (4) teacher’s Peer Observation (5) class ________________________________________________________________________________________________________ Closed 3. 4. Checklist (2) In-Class Observation (4) Seating Chart 8) Peer Observation (5) Structured Observation (9) ________________________________________________________________________________________________________ Observing Open 5. 6. your own Audio Recording (1) Audio Recording (1) class Seating Chart (8) Handheld recorder (3) Video Recording (11) Proformas (6) Scribbles (7) Teacher Diary (10) Video Recording (11) ________________________________________________________________________________________________________ Closed 7. 8. Audio Recording (1) Audio Recording (1) Checklist (2) Handheld recorder (3) Seating Chart (8) Proformas (6) Video Recording (11) Scribbles (7) Teacher Diary (10) Video Recording (11) ________________________________________________________________________________________________________ Where can I read and find out more about observation? Brian Lynch, in his book Language Program Evaluation (1996), includes a section on observation as data gathering starting (p. 108). He also provides an example of a structured observation instrument, the Communicative Orientation of Language Teaching (COLT) scheme. He describes fieldnotes in detail, with a helpful discussion on the difference between descriptive as opposed to evaluative comments. Anne Burns, in a book on action research, Collaborative Action Research for English Language Teachers (1999), describes in detail many observation techniques, including a discussion of validation. Weir and Roberts in Evaluation in ELT (1994), discuss observation techniques. Finally, Genesee and Upshur, in Classroom-Based Evaluation in Second Language Education (1996), devote an entire chapter to observation and provide a framework for classroom observation. _________________________________________________________________________________ Griffee 193 DISCUSSION QUESTIONS Write some questions you had while reading about observation. Task 1. Describe a time when you observed something about your teaching or something that happened in your class that you thought was interesting. Task 2. How might your observation be framed as a research project? _________________________________________________________________________________ 194 Data from Observation Task 3. Using the situation you first noticed in Task 1 and reframed as a research project in Task 2, what observation technique(s) described in this chapter could you use to gather research data? Task 4. What additional data collection instruments might be helpful? _________________________________________________________________________________ Griffee 195 References for Data from Observation Adler, P. A., & Adler, P. (1998). Observational techniques. In N. K. Denzin & Y. S. Lincoln (Eds.). Collecting and interpreting qualitative materials (pp. 79-109). Thousand Oaks, CA: Sage. Allwright, D. (1988). Observation in the language classroom. London: Longman. Allwright, D., & Bailey, K. M. (1991). Focus on the language classroom: An introduction to classroom research for language teachers. Cambridge: Cambridge University Press. Bailey, K. M. (1990). The use of diary studies in teacher education programs. In J. C. Richards & D. Nunan (Eds.). Second language teacher education (pp. 215-226). Cambridge: Cambridge University Press. Bartlett, L. (1990). Teacher development through reflective teaching. In J. C. Richards & D. Nunan (Eds.). Second language teacher education (pp. 202-214). Cambridge: Cambridge University Press. Bernard, H. R. (1994). Research methods in anthropology: Qualitative and quantitative approaches (2nd ed.). Walnut Creek, CA: Altamira Press. Burns, A. (1999). Collaborative action research for English language teachers. Cambridge: Cambridge University Press. Chaudron, C. (1988). Second language classrooms. Cambridge: Cambridge University Press. Chaudron, C. (1991). Validation in second language classroom research: The role of observation. In R. Phillipson, E. Kellerman, L. Selinker, M. Sharwood Smith, & M. Swain (Eds.). Foreign/ Second language pedagogy research (pp. 187-196). Clevedon: Multilingual Matters. Cook, V. (1989). The I-language approach and classroom observation. In C. Brumfit and R. Mitchell (Eds.). Research in the language classroom (pp. 71-77). London: Modern English Publications in association with the British Council. Cosh, J. (1999). Peer observation: A reflective model. English Language Teaching Journal 53(1), 22-27. Day, R. R. (1990). Teacher observation in second language teacher education. In J. C. Richards & D. Nunan (Eds.). Second language teacher education (pp. 43-61). Cambridge: Cambridge University Press. Evertson, C. M., & Green, J. L. (1986). Observation as inquiry and method. In M. Wittrock (Ed.), Handbook on research and teaching (pp. 162-213). New York, NY: MacMillan. Fanselow, J. F. (1987). Breaking rules: Generating and exploring alternatives in language teaching. New York, NY: Longman. Fanselow, J. F. (1990). “Let’s see”: Contrasting conversations about teaching. In J. C. Richards & D. _________________________________________________________________________________ 196 Data from Observation Nunan (Eds.). Second language teacher education (pp. 182-199). Cambridge: Cambridge University Press. Fradd, S. H., & McGee, P. L. (1994). Instructional assessment: An integrative approach to evaluating student performance. Reading, MA: Addison-Wesley. Galton, M. (1995). Classroom observation. In L. W. Anderson (Ed.). International encyclopedia of teaching and teacher education (2nd ed.). (pp. 501-506). New York, NY: Pergamon. Gebhard, J. G. (1999). Seeing teaching differently through observation. In J. G. Gebhard & R. Operand (Eds.). Language teaching awareness: A guide to exploring beliefs and practices (pp. 35-58). Cambridge: Cambridge University Press. Gebhard, J. G., Hashimoto, M., Joe, J., & Lee, H. (1999). Microteaching and self-observation: Experience in a preservice teacher education program. In J. G. Gebhard & R. Oprandy (Eds.). Language teaching awareness: A guide to exploring beliefs and practices (pp. 172- 194). Cambridge: Cambridge University Press. Gebhard, J. G., & Ueda-Motonaga, A. (1992). The power of observation: “Make a wish, make a dream, imagine all the possibilities!” In D. Nunan (Ed.). Collaborative language learning and teaching (pp. 179-191). Cambridge: Cambridge University Press. Genesee, F., & Upshur, J. A. (1996), Classroom-based evaluation in second language education. Cambridge: Cambridge University Press. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage. Hitchcock, G., & Hughes, D. (1995). Research and the teacher: A qualitative introduction to school- based research (2nd ed.). New York, NY: Routledge. Long, M. H. (1980). Inside the “Black Box”: Methodological issues in classroom research on language learning. Language Learning, 30(1), 1-42. Lynch, B. (1996). Language program evaluation. Cambridge: Cambridge University Press. Nunan, D. (1992). Research methods in language learning. Cambridge: Cambridge University Press. Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage. Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage. Richards, J. C. (1998). Through other eyes: Revisiting classroom observation. In Beyond training. Cambridge: Cambridge University Press. Rossi, P. H., Freeman, H. W., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.). Newbury Park, CA: Sage. _________________________________________________________________________________ Griffee 197 Spada, N. (1990). Observing classroom behaviours and learning outcomes in different second language programs. In J. C. Richards & D. Nunan (Eds.). Second language teacher education. Cambridge: Cambridge University Press. Spada, N., & Lyster, R. (1997). Macroscopic and microscopic views of L2 classrooms. TESOL Quarterly 31(4), 787-792. Stern, H. H. (1989). Seeing the wood and the trees: Some thoughts on language teaching analysis. In R. K. Johnson (Ed.). The second language curriculum. Cambridge: Cambridge University Press. Vierra, A., & Pollock, J. (1992). Reading educational research. Scottsdale, AZ: Gorsuch Scarisbrick. Wang, Q., & Seth, N. (1998). Self-development through classroom observation: Changing perceptions in China. English Language Teaching Journal, 52(3), 205-213. Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell. Williams, M. (1989). A developmental view of classroom observations. English Language Teaching Journal 43(2), 85-91. Woods, D. (1996). Teacher cognition in language teaching. Cambridge: Cambridge University Press. _________________________________________________________________________________ 198 Data from Observation CHAPTER TEN DATA FROM DIARIES AND JOURNALS In this chapter you will learn three ways of looking at diaries and journals, which topics diaries and journals have been used to investigate, and how you can use a diary or journal for your own research purposes. While the first accounts of diary studies date from the 1970s (Schumann & Schumann, 1977), the use of written reflection on personal learning has only been popular in ESL classrooms from the 1980s onward (Bailey, 1991). Journals and diaries have been used to facilitate written dialogue between students and teachers, to assist self-assessment by teachers in training, and enable individual researchers to investigate their own learning processes. At the same time, the terminology to describe these instruments has grown increasingly confusing. Researchers variously refer to these instruments as diaries, journals, letters, logs, or some combination of these terms, and these instruments, by whatever name, have been used to investigate various topics such as evaluation (Parkinson & Howell-Richardson, 1989), learner characteristics (Bailey, 1995), learning context (Schumann & Schumann, 1977), learning processes (Schmidt & Frota, 1986), student self-assessment (Griffee, 1997), teacher reflection (Thornbury, 1991), and the writing process (Holmes & Moulton, 1995). What is a general description or definition of a diary? The most popular term in the literature is some form of the word diary—here is a list of its variants: § Diary studies (Bailey, 1995; Campbell, 1996; Jones, 1994; Matsumoto, 1989, 1996; Schmidt & Frota, 1986) § Learner diary (Lowe, 1987; Parkinson & Howell-Richardson, 1989) § Language learning diaries (Peck, 1996; Schumann, 1980) § Teacher diaries (McDonough, 1994) The next most popular term is some form of the word journal: § Dialogue journal (Carrell, 1990; Casanave, 1995; Holmes & Moulton, 1995; Meath- Lang, 1990) § Student journals (Casanave, 1994) § Journal studies (Numrich, 1996) § Working journal (Spack & Sadow, 1983) § Journal (Cummings, 1996; Bailey, 1980) § Personal journals (Meath-Lang, 1990) § Classroom journals (Lucas, 1990) _________________________________________________________________________________ 199 Data from Diaries and Journals A third term is log: § Listening log (Kemp, 2010) § Teacher research log (Griffee, 1995) § Teaching practice logs (Thornbury, 1991) If we look at these studies from the point of view of who was writing to whom, certain patterns emerge. For example, Bailey (1991) divides diary studies into two categories: in the first category, the diarists and the analyst are the same person, in the second, the diarists and the analyst are different people. Matsumoto (1989) calls the first group introspective and the second group non- introspective. Casanave (1995) prefers journals to diary studies, but continues the dichotomy. Her first category (corresponding to Bailey’s diarists and analyst as the same person and Matsumoto’s introspective writing), Casanave calls learning logs. Her second category (corresponding to Bailey’s second category where the diarists and analyst are different and which Matsumoto calls non-introspective), Casanave calls dialogue journals. I propose the global term journal writing, which would be composed of three categories: dialog journals, defined as student-to-teacher writing; teacher journals, defined as teachers (or teachers in training) writing to a senior teacher or to each other; and diary journals defined as someone writing to herself. Figure 1 shows these relationships. The advantage of this model is that it simplifies the terminology, maintains the use of common terms associated with this type of writing, and relates them in ways that are familiar to language teachers. Figure 1. A model of journal writing showing three types In this chapter, I will discuss journal writing using the three categories mentioned in Figure 1, namely dialog journals, teacher journals, and diary journals. For each type, I will discuss _________________________________________________________________________________ 200 Data from Diaries and Journals advantages and disadvantages, key issues, and how you can do each type. I will also discuss data analysis, reliability, and validity for journal writing in general. Finally, I will conclude with a discussion of how diaries and journals can be used in research. A word about the spelling of dialogue or dialog: Webster’s Student Dictionary (1992) lists both spelling; the Longman Dictionary of Contemporary English (1995) lists dialogue as British English and dialog as American English. I will use the simpler American version, unless I am quoting another researcher who uses the more traditional spelling. DIALOG JOURNALS (DJs) Genesee and Upshur (1996) define a journal as a written conversation between students and teachers, while Ogane (1997) defines a dialog journal as a private written conversation between two persons. I would accept both definitions; however, the first definition applies to all journals and the second definition emphasizes the writing and personal aspect. In fact, a dialog journal could be audio recorded and DJs could be shared. What form might a Dialog Journal take? Many published reports of studies using dialog journals as data collection instruments don’t report the physical aspect of the journal. Since Dialog Journals are typically a form of communication between one teacher and several students, convenience is important. For example, for Griffee (1997) and Ogane (1997), who were teaching large classes in Japan, the Dialog Journals were single sheets of paper turned in at the end of class. In a diary study I conducted in 2008, I passed out a piece of paper with short headings or questions followed by blank spaces for students to write in. On advantages and disadvantages of journal writing I will discuss advantages and disadvantages of each type of journal writing, but this discussion should not be interpreted as reasons for keeping or not keeping any of the types of journals. All data collection instruments have strong points (advantages) but also weak points (disadvantages). To use a traffic signal as a metaphor, an advantage can be interpreted as a green light saying go ahead, a disadvantage can be interpreted as a flashing yellow light saying be careful, slow down, think carefully about what you are doing because there are problems here, but then go ahead. There are no red lights in data collection, only green go-ahead lights and yellow caution lights. What are the advantages of dialog journals? Because Dialog Journals are between student and teacher, it can be argued that authentic communication can take place. It can also be argued that writing reduces stress (Ogane, 1997) and increases communication between student and teachers. Genesee and Upshur (1996) state that journals provide for increased communication, and thus provide feedback from students to teachers. Some students may feel reluctant to express their feelings about classroom learning publically in class, so journal writing may allow them to discuss these issues more privately. In this sense, journals provide an additional channel of communication between teacher and _________________________________________________________________________________ Griffee 201 student. Genesee and Upshur (1996) also argue that journal writing allows for the possibility that students may increase their interest in their learning through the reflection that journaling involves as well as allowing teachers to get to know their students better. What are the disadvantages of dialog journals? Ogane (1997) says there are no empirical data that writing in dialog journals causes improvement in reading and writing. This lack of empirical studies may be because to the extent that data from a diary or journal is introspective, it’s hard to verify (Hilleson, 1996). Genesee and Upshur (1996) point out another possible disadvantage, namely that both students and teachers might need some orientation when first using dialog journals. In fact, some students may not participate in the journal process, or may participate against their will (Holmes & Moulton, 1995; Ogane, 1997). What are some of the key issues to consider? Since any type of journal or diary can be used for a variety of purposes (evaluation, pedagogy, increased reflection, self-assessment, or research), you have to decide why you might want to use a dialog journal. Following are some potential issues and the decisions they raise: 1. If you collect the dialog journals and use their contents to support your research, do you plan to correct or edit their contents in any way? 2. Do you plan to use the diary data with other forms of data, such as interviews and questionnaires? How do they mix and match? Does each form of data exhibit a unique feature or point of view, making it necessary? Can you articulate what each unique feature is? 3. In which language can the students write (Hilleson, 1996)? If they write in their L1, do you plan to translate it into another language for publication purposes? 4. Do you need to secure permission from your school to use data from DJs? If so, how should you go about doing that? 5. Do you know how will you identify various diary journalists? Will you use their first names, an initial or initials, or a pseudonym? 6. Will you assess the DJs in any way? 7. Does the teacher respond to the Dialog Journal, initiate, give information and opinions, or just ask questions (Ogane, 1997)? 8. When will Dialog Journals be collected, and who will collect them? For example, researchers Griffee (1997) and Parkinson and Howell-Richardson (1989) collected them at the end of each day. 9. Do you consider the Dialog Journal a private document between student and teacher or can DJs be shared? After each journal writing, Fedele (1996) told students they could share what they wrote. She had them (4th graders) share with a partner. About halfway through the year, she corresponded with the students to keep them writing about a topic rather than abandoning it. _________________________________________________________________________________ 202 Data from Diaries and Journals 10. Genesee and Upshur (1996) raise the issue of regular times for writing entries, collecting journals, and reading them. Would writing take place at a regular time, and if so, what time? 11. How will you deal with ethical concerns if student express feelings such as sadness or depression? If students write something in their journals that indicates to you they may harm themselves or others, what would you do? Genesee and Upshur (1996) recommend knowing school policy on confidentiality and notification of authorities. If no such policy exists, decide on one of your own. TEACHER JOURNALS (TJs) A Teaching Journal requires not only a teacher but also students (otherwise it would be classified as a Diary Journal), and the students may be other teachers, graduate students, graduate-level or late-stage undergraduate native English speakers or non-native English speakers (otherwise it would be classified as a Dialog Journal). A typical example of what I am calling a Teacher Journal is produced when a teacher, often a university or college professor, engages students in a training project. What form might a Teacher Journal take? The physical form that a TJ might take is any convenient writing form, such as sheets of paper kept in a folder or a spiral bound notebook. Another way of thinking about the form of a TJ might take is its organizational form. For example, Francis (1995) had students keep journals in terms of four categories: teaching plans, reflective writing on the class, recording events that impacted their practice teaching, and a critical summary of their reading. Thornbury (1991) gave his students seven categories for their diaries: the aim of the lesson, assumptions and anticipated problems, description of lesson, how the trainee felt, assessment of the effectiveness of the lesson relative to its aim, suggestions for improvement, and personal objectives for the next lesson. What are the advantages of Teacher Journals? One advantage of TJs is that they can help teachers investigate issues from a subjective or insider point of view (Bailey, 1990; Numrich, 1996). This means that TJ data can complement data from instruments, such as tests and questionnaires that investigate issues from more of an outsider point of view. Another advantage is that TJs, like all diaries and journals, can engender reflection, which may in turn produce change. This may be because TJs can uncover processes by which novice teachers learn (Numrich, 1996). Teachers in training may even use TJs to ask questions, and then use the TJ to increase reflection, which may produce answers to those very questions (Porter, et al., 1990). A third advantage is that TJs are low tech and easy to do. All that is required is the assignment, _________________________________________________________________________________ Griffee 203 something to write on, and perhaps some organizing questions or categories. A fourth advantage is that TJs can allow students to tell their instructor where they are having trouble because the TJ is a safe place to ask questions (Porter et al., 1990). What are the disadvantages of Teacher Journals? Bailey (1990) mentions that writing in a journal involves editing. This means that since we are not recording events as they happen, we are selectively remembering, or editing. Journal writers have to work hard to include objectively what happened as well as reflect on what they learned. For this reason, giving TJ writing categories as Francis (1995) and Thornbury (1991) did allows for the separation of what happened, what the journal writers felt about it, and what they learned from it. What are some key issues for Teacher Journals? Bailey (1990) mentions that confidentiality of participants must be taken into account. Increasingly, researchers are required by institutional review boards (IRBs) to submit their research plan for review and approval; one of the main concerns of IRBs is protection of research participants. Bailey (1990) also mentions participant resistance to keeping a diary. Researchers using Dialog Journals or Teacher Journals would be wise to have a policy in place for dealing with students who do not want to keep a journal. DIARY JOURNALS (DJs) You will recall that a diary journal is a document maintained by an individual writing a report to himself or herself on some topic area, such as learning a language or teaching a course. What form might a Diary Journal take? Some teachers keep a separate journal for each class in spiral-bound standard-sized notebooks (Isakson & Williams, 1996). It is also possible to keep a small note pad in pocket or purse for quick entries during the day, and then transfer a fuller account later. Others write on single sheets of paper that they number and keep in a folder, and of course diaries may be digital. Why keep a Diary Journal? There are two broad reasons for initiating a Diary Journal. One is a discovery or inductive reason, namely that we want to explore and learn something. For example, we are teaching a course, maybe for the first time, and we are wondering about the curriculum, our pedagogical practice, or our students’ learning. We decide to keep a Diary Journal to record our thoughts and feelings that otherwise might be forgotten or overlooked when it comes time to evaluate and revise the course. On the other hand, we might have a specific problem we want to examine. This is more a deductive reason. For example, we want to investigate a particular problem or issue that has been on our mind, or that we have been reading about. Ellis (1989) used diaries in this way. He asked two teachers to keep diary journals to reflect on their learning. He then compared the results of the diary journals with the results of a questionnaire to investigate learning styles completed by the same two teachers. _________________________________________________________________________________ 204 Data from Diaries and Journals What are the advantages of keeping a Diary Journal? One advantage is that, as noted by Bailey (1995), being a diarist is somewhat like being a participant observer in an ethnographic study. If someone wants to be a participant observer, that is to say to be both participant in a situation and at the same time to be able to observe what happens, one has to either ask for permission to be there, or one has to somehow sneak into the situation, hopefully unnoticed. But as a teacher in a course, you are perfectly situated because you are already there. You already have access to one of the main participants who, in the case of a diary journal, is you. Another advantage of a diary journal is that there is a growing consensus that writing creates reflection, and reflection in and of itself is helpful (Cooper, 1991; Holly, 1989; Hoover, 1994). Journaling of any kind can be a rich source of data for whatever use we are seeking (McDonough, 1994). This can be helpful in research, especially extended research such as working on a thesis or dissertation. Borg (2001) documents how he used a diary to examine his own research and writing problems, and by examining and becoming aware of certain tendencies he was able to change them. What are the disadvantages of keeping a Diary Journal? Diary Journal entries, because they are about the self, have the potential to be personal or even painful (Bailey, 1995). Of course, no one knows about the diary except the person keeping it, and as McDonough (1994) points out, the diary data must be analyzed. The uncomfortable aspect of the diary might arise at this point simply because we tend to be more accustomed to holding our students up for evaluation than we are holding up ourselves for critical analysis. A second disadvantage of the diary journal is that, as Campbell (1996) notes, the original diary is not published. Usually only excerpts to support points are revealed. This makes comparisons and resulting conclusions difficult. Finally, and more importantly, we are limited observers (Schmidt & Frota, 1986). We don’t see or notice everything. In addition, as Fry (1988) says, diary data are retrospective data, in other words, we make diary entries after the event. That means there is some memory decay or lapse between the time the event occurred and the time we write about it. In addition, we have no way of knowing if we are typical observers. We know that we are influenced (biased) by our backgrounds, concerns, and interests. Would others notice the same things we did? It is for this reason that data from diaries of any kind must be considered data to be validated. It is not the case that diary data are valid simply by virtue of being written and recorded by us. How can I begin my own Diary Journal? Here are some possible suggestions to consider: First, just start. Find something with which to record your thoughts and begin. If a small audio recorder is handy and you don’t mind listening to the sound of your own voice, press the ‘record’ button. If you want to use your computer, open a new document and start writing. If you prefer paper, find any notebook with some unused paper and begin writing. The essence of this tip is that starting something new is half the battle. _________________________________________________________________________________ Griffee 205 The downside of this approach is that “haste makes waste” and what you do not consider may turn out to be a problem. For example, if in your rush to begin a diary you forget to list categories that you might want to comment on, then later it may be difficult to include them. On the other hand, how will you know what the problems are if you don’t begin? One way to deal with the “just do it” approach vs. the “think about it first” approach is to say to yourself, this is just a pilot. Do a pilot diary journal for the purpose of finding out how it works and what the problems might be. On the other hand, if you are a “let’s think about this” type of person, here are some things to consider. Are there any upcoming opportunities, such as beginning a new course or a trip you are about to take, that could be an opportunity for diary-keeping? What physical format such as type of notebook do you like? Are there any persistent problems that you have been wondering about that you could investigate with your diary? Where do you like to write? Do you have a special place to write, a special time, a special pen? Write out your questions and your answers to those questions. In doing so, you have just begun your diary. What kind of data typically results from Diaries and Journals? Diaries and Journals are virtual vacuum cleaners in that they can suck up anything. Data from journals can include daily events and related feelings (Schumann, 1980), impressions of people and circumstances in the environment of interest (Bailey, 1995), and critical inquiry focusing on practice (Holy, 1989). The data can take the form not only of words, but also of frequency counts, pictures, drawings, and even pieces of conversation overheard or transcribed from recordings. How is Diary data typically analyzed? The most common analysis advocated in the literature reviewed here is to read the journal entries and find patterns (Bailey, 1990, 1995; Borg, 2001; Hoover, 1994; Matsumoto, 1989; McDonough, 1994; Schumann 1980). Their advice is to read entries and notice themes, especially themes that reoccur. Then list these themes, perhaps with the number of times you notice them. This process, by the way, is similar to the statistical correlation procedures known as Factor Analysis, in which a factor would be the same thing as a theme. The only difference is that your mind, instead of a statistical procedure in a computer, is doing the sorting, but the results are similar: a list of identified themes, and in the case of journal data, a series of quotes to back up each theme. A second approach to journal data analysis is illustrated by Schumann and Schumann (1977). They report that they asked one question: What did the data reveal that was not known before? They decided this question was not answerable. So they changed the question to: What did the data tell them about their language learning? This question was answerable, and it formed the basis for their results, which they labeled as themes, and later they called them personal variables. These variables stated the conditions under which each of them best learned a foreign language. A third approach is to connect the data to theory. Although Bailey (1995) suggests rereading and looking for trends by frequency or saliency, she also says that data from journals can relate to theory without saying exactly how. She might be suggesting that one has a theory and can use data from journal entries to confirm or disconfirm the theory, or she might be suggesting that one can use data from journal entries to create a theory. Fry (1988) also allows for theory, in the form of a _________________________________________________________________________________ 206 Data from Diaries and Journals framework, as useful for understanding data from journals. Fry (1988) raises three possible uses for data from journals: 1) to generate hypotheses, 2) to provide insights into learner strategies, and 3) to give insights into second language acquisition. He agrees that diary data can be used for the first use, generating research hypotheses. If the second use is desired, investigation of learner strategies, then he thinks is appropriate for the categories to emerge from the data. This is essentially the first approach to analysis. If the third use is desired, namely research into second language acquisition, then he wonders if journal data are appropriate. However, if that is the desired use, it will be necessary to describe the categories and framework carefully so other researchers can use them and verify them. Describing categories and framework amounts to describing the theory used. A fourth approach to analyzing data from diaries and journals is to analyze the data in terms of categories derived from discourse analysis. For example, Ogane (1997) used categories based on Peyton and Seyoum (1989), such as topic initiation and responses. How can I calculate reliability and validity? Reliability and validation evidence is seldom provided in diary study research. Perhaps such evidence tends to be ignored because subjective self-report data are assumed to be both reliable and valid. The reasoning is, if someone reports his or her own observations, thoughts, and ideas, how can that person be mistaken? I have listed many reasons in the disadvantages section of this chapter that suggests otherwise. The main argument for investigating and reporting reliability and validity is that all data collection instruments have to account for it, and diaries and journals are not exempt. Because reliability and validity are seldom if ever mentioned in studies using data from diaries and journals, there is not much to draw from. What follows are some possible attempts that can be developed: 1. Triangulate diary data and recorded data of a similar type. Schmidt (Schmidt & Fronta, 1986) wondered how the occasional data he was gathering from his conversations in coffee shops would compare to more systematically collected data. He tape-recorded long stretches of conversation between him and his teacher (Fronta) so he could analyze and compare the two types of data. Another approach to validating journal data through triangulation would be to gather data from two different data collection instruments. For example, we could create a questionnaire that asked about learning styles, and also gather data from student journals on how they best learned in class. We could then compare interpretations from each of the instruments. If the conclusions we reached from one instrument matched or supported those of the other instrument, we could claim validation. 2. Stability of results over time can point to reliability. If the analysis of diary data consistently shows the same or similar results, these results could be an argument for reliability and thus validity of the data. 3. Bachman (1990) describes construct validity as the extent to which what we find in our data matches what we expect to find based on our theory. In other words, we can argue for construct validity of journal data if first, we define what we are looking for (our construct) _________________________________________________________________________________ Griffee 207 and second, show that our data and analysis are consistent with what we found. The key is to define the construct we are using. 4. Use peer review to establish validity (Miles & Huberman, 1994, p. 277). Peer review means finding a person with sufficient knowledge to understand the study; this person cannot be directly involved in the study, however. This person (the peer) can be asked to comment on their ability to trace a path from raw data to your conclusions. A series of three questions can be asked of a peer reviewer: Are my conclusions plausible? Can you think of alternative conclusions? Do you think my conclusions are not only plausible, but correct? These results can be offered as validity evidence. How can diaries and journals be used in research? Researchers conducting diary studies often wonder about validity, and their concern frequently focuses on the issue of generalizability (Bailey, 1980, 1991; Hilleson, 1996; McDonough, 1994; Matsumoto, 1989). One reason that generalizability is mentioned as lacking or impossible to obtain in diary studies, is because of the low N-size, that is, the number of persons involved in the study (Bailey, 1991). As a result, it is often lamented that the findings from diary studies do not, indeed cannot, generalize beyond the diary study itself. This is a misplaced argument because generalizability is a characteristic of a research design, not that of a data collection instrument, and a diary or journal is a data collection instrument, not a design. Understanding that a diary is a data collection instrument and not a research design relieves diarists from the burden of showing causality and generalizability of their data, but it increases the burden of showing reliability and validity of their interpretations based on that data. At the risk of being tedious, I will expand on this argument. Generalization is the ability to apply the findings of a study or investigation to other situations that are reasonably similar to the original study. It is sometimes called external validity in contrast to internal validity, and is a concern of design—the blueprint of relationships within a research study. For example, survey research design shows the relationship of a sample to a population while an experimental design spells out the relationship of variables of interest in two (or more) groups or at two (or more) times within the same group. No matter what the design, there is a concern with showing how causality and generalizability are developed in research. A data collection instrument (DCI), on the other hand, is a way of collecting data. Examples of data collection instruments include questionnaires, interviews, and classroom tests. Unlike designs, which are concerned with causality and generalizability, DCIs are concerned with the quality of the data they collect, better known as reliability and validity. To use a metaphor of space, designs are like suns, some burning brightly, some in the process of dying, and some being born; DCIs are like planets, humbly revolving around suns. Journals are data collection instruments, not designs Keeping a diary is a form of collecting data, not a way of designing a research project, which would have to show causality and generalize results. As stated earlier in the Introduction to Research Design, any design can accommodate any type of data. A design without data is an empty shell, and data without a design is meaningless. _________________________________________________________________________________ 208 Data from Diaries and Journals Data must be analyzed, as Bailey (1991) rightly points out, but data alone do not give a purpose or reason for collection, do not explain themselves (you need a theory for that), do not show causality, and do not generalize to other situations. Therefore, we can’t compare data from diaries with the experimental research design because that would be to compare apples (a form of data collection) with oranges (a design). How should diary studies be used? We have to recognize that diary data by alone are weak. This is not an attack on diary data as an inferior form of data; the same weakness can be said for any single form of data, including that from standardized tests or interviews. Given current research practices, it is doubtful that data from diaries and journals should be used alone. Probably, data from a journ

An Introduction to Second Language Research Methods (Griffee) - PDF

Document Details

Tags

Related

Summary

Full Transcript

Upgrade to continue