article 4 week 1

Overview

• This paper analyzes data on 11,600 students and their teachers who were randomly assigned to different size classes from kindergarten through third grade while using statistical methods are used to adjust for nonrandom attrition and transitions between classes.

Main Conclusions

• On average, performance on standardized tests increases by four percentile points the first year students attend small classes.
• The test score advantage of students in small classes expands by about one percentile point per year in subsequent years.
• Teacher aides and measured teacher characteristics have little effect.
• Class size has a larger effect for minority students and those on free lunch.
• Hawthorne effects were unlikely.

Introduction

• The effect of school resources on student achievement generally finds ambiguous, conflicting, and weak results.
• Most estimates of the effect of school inputs on student achievement are statistically insignificant.
• There appears to be no strong or systematic relationship between school expenditures and student performance.
• Data are more consistent with a pattern that includes at least some positive relation between dollars spent on education and output, than with a pattern of no effects or negative effects.
• Much of the uncertainty in the literature derives from the fact that the appropriate specification—including the functional form, level of aggregation, relevant control variables, and identification of the "education production function" is uncertain. Estimates that use cross-state variation in school resources typically find positive effects of school resources, whereas studies that use within-state data are more likely to find insignificant or wrong-signed estimates
• Specification issues arise because of the possibility of omitted variables, either at the student, class, school, or state level.
• Functional form issues are driven by concern for omitted variables, as researchers often specify education production functions in terms of test-score changes to difference out omitted characteristics that might be correlated with school resources (although such differencing could introduce greater problems if the omitted characteristics affect the trajectory of student performance).
• A classical experiment, in which students are randomly assigned to classes with different resources, would help overcome many of these specification issues and provide guidance for observational studies.
• This paper provides an econometric analysis of the only large-scale randomized experiment on class size ever conducted in the United States, the Tennessee Student/Teacher Achievement Ratio experiment, known as Project STAR.
• Project STAR was a longitudinal study in which kindergarten students and their teachers were randomly assigned to one of three groups beginning in the 1985-1986 school year: small classes (13–17 students per teacher), regular-size classes (22-25 students), and regular/aide classes (22-25 students) which also included a full-time teacher's aide.
• After their initial assignment, the design called for students to remain in the same class type for four years.
• 6000-7000 students were involved in the project each year, over all four years, the sample included 11,600 students from 80 schools.
• Each school was required to have at least one of each class-size type, and random assignment took place within schools.
• Students were given a battery of standardized tests at the end of each school year.
• Project STAR has been examined extensively by an internal team of researchers who found students in small classes tended to perform better than students in larger classes, while students in classes with a teacher aide typically did not perform differently than students in regular-size classes without an aide.
• Past research consists of comparisons of means between the assignment groups, and analysis of variance at the class level, while little attention has been paid to potential threats to the validity of the experiment or to the longitudinal structure of the data.
• As in any experiment, there were deviations from the ideal experimental design in the actual implementation of Project STAR.
• Students in regular-size classes were randomly assigned again between classes with and without full-time aides at the beginning of first grade, while students in small classes continued on in small classes, often with the same set of classmates which was done to placate parents of children in regular classes who complained about their children's initial assignment.
• Approximately 10 percent of students switched between small and regular classes between grades, primarily because of behavioral problems or parental complaints which could compromise the experimental results.
• Because some students and their families naturally relocate during the school year, actual class size varied more than intended in small classes (11 to 20) and in regular classes (15 to 30).
• Sample attrition was common, with half of students who were present in kindergarten were missing in at least one subsequent year.
• Some students may have nonrandomly switched to another public school or enrolled in private school upon learning their class-type assignments with these limitations that have not been adequately addressed in previous work.
• The paper has goals to probe the sensitivity of the experimental estimates to flaws in the experimental design as well as to use the experiment to identify an appropriate specification of the education production function to estimate with nonexperimental data and also to use the experimental results to interpret estimates from the literature based on observational data to make a rough attempt to compare the benefits and costs of reducing class size from 22 to 15 students.

Background on Project Star and Data

Design and Implementation

• Project STAR was funded by the Tennessee legislature, at a total cost of approximately $12 million over four years.
• The Tennessee legislature required that the study include students in inner-city, suburban, urban, and rural schools.
• The research was designed and carried out by a team of researchers at Tennessee State University, Memphis State University, the University of Tennessee, and Vanderbilt University.
• To be eligible to participate in the experiment, a public school was required to sign up for four years and be large enough to accommodate at least three classes per grade, so within each school students could be assigned to a small class (13-17), regular class (22–25 students), or regular plus a full-time aide class.
• The statewide pupil-teacher ratio in kindergarten in 1985–1986 was 22.3, so students assigned to regular classes fared about as well as the average student in the state.
• Schools with more than 67 students per grade had more than three classes.
• One limitation of the comparison between regular and regular/aide classes is that in grades 1–3 each regular class had the services of a part-time aide 25-33 percent of the time on average, so the variability in aide services was restricted.
• The cohort of students who entered kindergarten in the 1985-1986 school year participated in the experiment through third grade.
• Any student who entered a participating school in a relevant grade was added to the experiment, and participating students who repeated a grade, skipped a grade, or left the school exited the sample.
• Entering students were randomly assigned to one of the three types of classes (small, regular, or regular/aide) in the summer before they began kindergarten. Students were typically notified of their initial class assignment very close to the beginning of the school year.
• Students in regular classes and in regular/aide classes were randomly reassigned between these two types of classes at the end of kindergarten, while students initially in small classes continued on in small classes.
• Kindergarten attendance was not mandatory in Tennessee at the time of the study, many new students entered the program in first grade, so students were added to the sample over time because they repeated a grade or because their families moved to a school zone that included a participating school.
• 2200 new students entered the project in first grade and were randomly assigned to the three types of classes while another 1600 and 1200 new students entered the experiment in the second and third grades respectively, while newly entering students were randomly assigned to class types, although the uneven availability of slots in small and regular classes often led to an unbalanced allocation of new students across class types.
• A total of 11,600 children were involved in the experiment over all four years as after third grade, the experiment ended, and all students were assigned to regular-size classes with though data have been collected on students through ninth grade, the present study only has access to data covering grades K-3.
• Data were collected on students each fall and spring during the experiment where the class type is based on the class attended in the fall so all students who attended a STAR class in either the fall or spring are included in the database.
• The STAR data set does not contain students' original class type assignments resulting from the randomization procedure because only the class types that students actually were enrolled in each year are available which could mean it is possible that some students were switched from their randomly assigned class to another class before school started or early in the fall.
• Only 0.3 percent of students in the experiment were not enrolled in the class type to which they were randomly assigned in kindergarten.
• Baseline test score information on the students is not available, so one cannot examine whether the treatment and control groups "looked similar" on this measure before the experiment began.
• Students assigned to small- and regular-size classes look similar along other measurable dimensions at base line.

Data and Standardized Tests

• Students were tested at the end of March or beginning of April of each year in tests that consisted of the Stanford Achievement Test (SAT), which measured achievement in reading, word recognition, and math in grades K-3, and the Tennessee Basic Skills First (BSF) test, which measured achievement in reading and math in grades 1-3.
• The tests were tailored to each grade level where because there are no natural units for the test results, the test scores were scaled into percentile ranks.
• In each grade level the regular and regular/aide students were pooled together, and students were assigned percentile scores based on their raw test scores, ranging from 0 (lowest score) to 100 (highest score) with a separate percentile distribution was generated for each subject test.
• For each test it was determined where in the distribution of the regular-class students every student in the small classes would fall as then the students in the small classes were assigned these percentile scores.
• To summarize overall achievement, the average of the three SAT percentile rankings was calculated where if the performance of students in the small classes was distributed in the same way as performance of students in the regular classes, the average percentile score for students in the small classes would be 50.
• An examination of the correlations among the tests indicates that the strongest correlations typically are between tests of the same subject matter and tests of the same subjects tend to have a higher correlation from one grade to the next than tests of different subjects.
• The SAT and BSF tests are also highly correlated with each other, with the SAT exam is the primary focus of study because this test has been used on a national level for a long period of time.
• The main findings are similar for the BSF test, however, there are limitations in that the average student in small classes performed better on this summary test measure than did those in regular or regular/aide classes while there does not seem to be a very strong or consistent effect of the teacher aide.

Statistical Models

• To see the advantage of a randomized experiment in estimating the effect of school resources on student achievement, consider the general model: Yij = aSij + bFij + ɛij
  • Yij is the achievement level of student i in school j
  • Sij is a vector of school characteristics
  • Fij is a vector representing the family background of the student
  • ɛij is a stochastic error component
  • Sij and Fij include information cumulated over the student's life
• If a school characteristic such as class size is determined by random assignment it will be independent of the omitted variables.
• With random assignment, a simple comparison of mean achievement between children in small and large classes provides an unbiased estimate of the effect of class size on achievement.
• STAR data is analyzed by estimating the following regression equation for students in each grade level: Yics = β0 + β1SMALLcs + β2REG/Acs + β3Xics + αs + Eics
  • Yics is the average percentile score on the SAT test of student i in class c at school s
  • SMALLcs is a dummy variable indicating whether the student was assigned to a small class that year
  • REG/Acs is a dummy variable indicating whether the student was assigned to a regular-size class with an aide that year
  • Xics is a vector of observed student and teacher covariates
  • Independence between class-size assignment and other variables is only valid within schools, because randomization was done within schools with a separate dummy variable being included for each school to absorb the school effects
  • Error term &ics is modeled in a components-of-variance framework: &ics = μες + &'ics
  • μες is a class-specific random component that is common to all members of the same class
  • ɛ'ics is an idiosyncratic error term
• Equation (1) was also estimated using dummies indicating students' initial assignment the first year they entered the program, rather than their actual assignment each year because several students were reassigned to different classes after their initial random assignment, in part based on their performance.
  • Models including initial assignment are labeled “reduced-form" models, because one can think of initial assignment as an excluded variable that is correlated with actual class size
• Regression results are presented with students in small classes tending to perform better than those in regular and regular/aide classes with the gap in average performance being about 5 percentile points in kindergarten, 8.6 points in first grade, and 5–6 points in second and third grade.
• White and Asian students tend to score eight percentile points higher than black students in kindergarten, and this gap is about six points in third grade.
• Estimates of the effect of being in a small class which use initial assignment are only slightly smaller than the estimates which use the actual class assignment and are always statistically significant suggesting that possible nonrandom movement of students between small and regular classes was not a major limitation of the experiment.

Effects of Attrition

• If the students originally assigned to regular classes who left the sample had higher test scores, on average, than students assigned to small classes who also left the sample, then the small class effects will be biased upwards.
• One reason why this pattern of attrition might occur is that high-income parents of children in larger classes might have been more likely to subsequently enroll their children in private schools over time than similar parents of children in small classes. Adjusting for possible nonrandom attrition is a matter of imputing test scores for students who exited the sample by assigning the student's most recent test percentile to that student in years when the student was absent from the sample.

Two-Stage Least Squares (2SLS) Models

• Students in the Project STAR experiment who were assigned to small classes had varying numbers of students in their classes because of student mobility and enrollment differences across schools while students in the regular-size classes had variable class sizes.
• A natural model for this situation is a triangular model of student achievement in which the actual number of students in the class is included on the right-hand side, and initial assignment to a class type is used as an instrumental variable for actual class size. Estimated Model:
  • CSics = πο + π₁Sios + π₂Rios + π3Xics + δε + Tics
  • Yics = βο + B1CSics + β2Xics + αs + Eics
    • CSics is the actual number of students in the class
    • Sios is a dummy variable indicating assignment to a small class the first year the student is observed in the experiment
    • Rios is a dummy variable indicating assignment to a regular class the first year the student is observed in the experiment
  • In this setup, only variation in class size due to initial assignment to a regular or small class is used to provide variation in actual class size in the test score equation where because of the random assignment of initial class type, one would expect that this excluded instrumental variable is uncorrelated with &ics , as required for 2SLS to be consistent.
  • If attending a small class has a beneficial effect on students' test scores, β would be negative
• According to the 2SLS estimates, a reduction of ten students is associated with a seven-to-nine point increase in the average percentile ranking of students, depending on the grade.
• There is no obvious trend over grade levels in the effect of class size in these data.
• Those attending smaller classes tend to score higher on the standardized test by the end of the first year they entered the experiment leading to believe If assignment to small or regular classes was somehow nonrandom, then the initial assignment would have to have been skewed in the direction of producing higher test scores in the small classes for each wave of students who entered the program.
• For students entering the experiment in first or second grade, the test score gap between those in small- and regular-size classes grows as students progress to higher grades.

Models with Pooled Data

• The cumulative effects of having been in a small or regular class can be explored through estimating several models with the data pooled with the general model of the form: [Y.sub.ig] = [beta.sub.o] + [[beta].sub.1][S.sub.io] + [[beta].sub.2][REG/A.sub.io] + [[beta].sub.3][N.sup.S.sub.ig] + [[beta].sub.4][N.sup.A.sub.ig] + [[beta].sub.5][X.sub.ig] + [[alpha].sub.g] + [[alpha].sup.f] + [[alpha].sub.s] + [epsilon.sub.ig] where g indicates grade level (K, 1, 2, or 3) and i indicates students, [Y.sub.ig] is the test score, [S.sub.io] and [REG/A.sub.io] are dummy variables indicating a student's class type in the first year he or she participated in the program, [N.sup.S.sub.ig] and [N.sup.A.sub.ig] are the cumulative number of years (including the current grade) the student has spent in a small or regular/aide class, [X.sub.ig] is a vector of student, teacher, and class characteristics, [alpha.sub.g] is a set of three current grade dummies, [gamma.sub.f] is a set of three dummies indicating the first year the student entered the STAR sample, and [alpha.sub.s] is a set of school fixed effects. Achievement of students in small classes jumps up by about four percentile points the first year a student attends a small class, and increases by about one percentile point for each additional year that the student spends in a small class thereafter with both the initial effect of being in a small class and the cumulative effect being statistically significant in these models.

• Students in small classes were more likely to remain with their classmates in first grade because students in regular classes were randomly reassigned between regular classes with and without full-time aides so two variables are included to control for the impact of the constancy of one's classmates where if a student is new to the school in a particular grade, this variable will have a value of 0 and if a student attends a class that consists only of students who were in that student's class the preceding year, the variable will have a value of 1 along with the average of this variable over all the other students in the class.

• This variable might influence achievement because the extent to which other students in a class know each other could influence one's adjustment to the class: initial jump in test scores associated with attending a small class. -Also attendance in classes with a higher proportion of classmates who attended kindergarten has a large, positive effect on a student's own achievement.

• The value-added model only identifies the cumulative effect of time spent in a small class, and the estimated value-added specification indicates that students gain from attending small classes, the benefit is less than the full effect that accounts for the discrete gain that occurs the first year students are in a small class.

Heterogeneous Treatment Effects

• The effect of being in a small class:
  • May vary for students with different backgrounds where smaller classes tend to have:
    • A larger initial effect, but a smaller cumulative effect, for boys as compared with girls.
    • Students on free lunch and black students tend to have:
    • Both a Larger Initial Effect, and a Larger Cumulative Effect.
    • Inner-city students tend to have:
      • A move beneficial effect of attending a small class in the first year they attend one than students from other areas, and a sharper gain over time from remaining in a small class.
        • Generally the lower achieving students benefit the most from attending smaller classes.

Hawthorne and John Henry Effects

• It has been suggested by some that the effectiveness of small classes found in the STAR experiment may have resulted from "Hawthorne effects," in which teachers in small classes responded to
• "John Henry" effect, in which they could overcome the bad luck of being assigned more students. Both of these situations could limit the external validity of the results of the STAR experiment. There isnt provide much evidence of either Hawthorne or John Henry effects

This studies the findings of Project STAR. It covers experimental findings, validity, and data analysis methods. It also includes the model $Y_{ij} = aS_{ij} + bF_{ij} + e_{ij}$ and equation (2).