A surprising conclusion from some recent research on the impact of public vs. private schooling:
Students do better in private schools, according to common wisdom -- and some well-regarded data now more than two decades old. But a recent study of standardized math scores in more than 1,300 public and private schools says the opposite may be true, according to Sarah and Christopher Lubienski, education professors at the University of Illinois at Urbana-Champaign.Public school students from similar social and economic backgrounds tested higher in a national math achievement test than their peers in private schools, the Lubienskis say in an article to be published in the May issue of Phi Delta Kappan, an influential education journal. They also are presenting their findings at the annual meeting of the American Educational Research Association (AERA), being held April 11-15 in Montreal.
The research uses fourth- and eighth-grade NAEP scores from 2000. When the private school students are compared as an overall group to the public school students, the private schools have higher math means, but when each group is broken down into one of four SES quartiles, the researchers saw higher means for the public schools within each quartile.
How is that possible, you may ask? Well, I haven't seen their data, but I can easily think of one way this could happen. The situation of getting opposite results when using aggregated vs. disaggregated data has been researched for about 40 years and is known as Simpson's Paradox.
Let's start with the assumption that more of the private school kids are wealthy, and more of the public school kids are poor. Thus, if we had 285 kids from each type of school, the quartile sample sizes might break down as follows:
| SES Quartile | Public N | Private N |
|---|---|---|
| High SES | 50 | 100 |
| High-Middle SES | 60 | 75 |
| Low-Middle SES | 75 | 60 |
| Low SES | 100 | 50 |
What we see here is that we have more of the private school students in the high SES, and many fewer in the lower two SES groups. So let's say that these are the means we see:
| SES Quartile | Public N | Public mean | Private N | Private Mean |
|---|---|---|---|---|
| High SES | 50 | 96 | 100 | 95 |
| High-Middle SES | 60 | 94 | 75 | 93 |
| Low-Middle SES | 75 | 90 | 60 | 89 |
| Low SES | 100 | 87 | 50 | 86 |
Note that, at each quartile, the public schools do better. But because there are more private school students at the high SES quartiles that have higher mean scores, if I just aggregate across all 285 students in each school group, I end up with a public school mean of 90.84, and a private school mean of 91.63. Thus, at the aggregate level, it looks like private schools do better because private schools have more of the higher performers. The problem here is the disparate sample sizes in the SES groups; the overall picture doesn't reveal that the SES breakdowns within school type are very different.
Even though my example above is just one possible way that the research results could be explained, in general, disaggregating the data are a good idea in this type of analysis. It's also good that the authors caution that this is not a longitudinal study, nor does it tell you what would happen with any particular student who switched schools. It does suggest that the public schools might not be doing as poor a job as some have thought.
(P.S. - If anyone has any idea how to get rid of the gap that's appearing before each table, let me know. I couldn't get rid of it.)
Posted by kswygert at April 12, 2005 11:08 AM