Does more spending help student performance?
Activists and groups like the NEA tell us that we need to spend more money on schools in order to improve performance. Does more public school spending really improve student outcomes? Since different states spend different amounts, we can find out.
This is the first of a series of blog posts on school performance.
I went into this project believing that the NEA and friends were dead wrong, that increasing school spending beyond current levels would be very unlikely to have any effect on school performance. The truth turns out to be a bit more complicated, but ultimately I think that drastic increases in school spending will not by themselves greatly improve performance.
The Bottom Line
There are three bottom lines (which really means, there’s room for more interesting analysis and future blog posts).
The gold standard for student performance in the United States is the NAEP, National Assessment of Education Progress, administered by the National Center for Education Statistics (a research arm of the federal Department of Education). Why do I call it the gold standard? Because NAEP is the only assessment administered to representative groups of students in multiple grades (4th and 8th) across all 50 states. The NCES publishes a ton of data relating to NAEP.
By contrast, state-specific measures are only given in one or a few states. Some may be as good as NAEP or even better (in terms of accurately assessing performance), but since they are state-specific they aren’t useful for cross-state comparisons.
NAEP results let us compare states to see how educational practices—including spending policies—and other factors impact student performance.
Raw Data: Per-student Instructional Spending
I have used per-pupil insructional spending data gathered by the US Census Bureau. The link http://www.census.gov/govs/www/school.html has spending data from all fifty states and Washington, D.C., arranged in various ways. The data are available in CSV and Excel formats, making it easy to do your own analyses.
The census school spending data files also have per-pupil total spending, adminstrative spending, and other interesting information. I have chosen to use instructional spending because the arguments made for increased school spending generally center around getting more money “into the classroom.”
Data and Tools
For today’s introductory blog post on education, I have compared NAEP 4th grade math scores, from the 2003 administration of the test, to per-student spending in the school year 2002-2003. This is a very simplistic model. For example, a fourth-grader’s performance depends not only on his school environment during fourth grade, but also his educational experience in grades one, two, and three.
A better analysis would use a three year average of spending data with the third year being the same as an NAEP test administration year. This averaging would hopefully capture the time-based effects of spending on student performance, and minimize the effects of year-by-year spending changes at the state level. I’ll try to do that in a future post, and also look at other NAEP data.
Charts and statistical calculations courtesy of Microsoft Excel 2004 for Macintosh.
A Picture is Worth a Thousand Words
Let’s start with a picture (click to see the large version):
As you see, NAEP scores range from 0 to 500 possible. The state averages from the year 2003, however, cluster much more closely. The 50 states range in value from 223 to 243; the outlier at the left, with 205, is the District of Columbia. An individual student scoring at least 282 would be considered Advanced; one between 249 and 282 would be Proficient; one between 214 and 249 would have Basic ability; and one below 214 would show "below Basic" ability with math. Since very few people are either "Advanced" or "below Basic" we should expect the statewide averages to be in the "Basic" and "Proficient" ranges; as it turns out, all states average "Basic" (we'll discuss that another time).
Spending is fairly diverse, ranging from $3123 for Utah to $8375 for New York.
One interesting point is that D.C. has the third highest per-pupil expendture (nearly $7000 per year on instructional spending) while showing by far the lowest scores. Clearly, D.C. does not seem to suggest that more spending improves educational results. However, D.C. is so different from the 50 states in so many ways that I hesitate to draw too many conclusions.
Here is the same chart, without Washington, D.C. and shortened to show only the actual performance (not the cut scores). Click to see the large version.
No Trend? Trend?
Just glancing at the first chart, spending doesn’t seem to have much to do with performance. There’s a slight effect; 6 of the 8 highest performing states spend more than $5000 per year, while 7 of the 8 lowest performing states spend less than $5000 per year (7 of the 9 lowest if you include Washington, D.C.).
On the other hand, the middle-performing “clump” includes both the highest and lowest spending states. Overall, the results are messy in the first chart.
The second chart zooms in on the state data and shows a pattern more clearly; higher spending states seem to do somewhat better than lower spending states.
In statistics, a calculated number called Pearson’s r shows the amount of correlation between two sets of data. If r = 1, then the two data sets are perfectly correlated; that is, an increase in one will automatically increase the other, and the change in one is sufficient to account for all change in the other. The closer r is to 0, the less correlated (or in other words, the less closely related) the two data sets are. Thus an r value of 0.8 means a strong correlation, and an r value of 0.1 means a very weak correlation.
The r-value for the data set with Washington, D.C. is .17, a weak correlation. However, if we discard Washington, D.C., then the correlation (r-value) among the 50 states jumps to .45. This is a moderate to strong correlation, and indicates that increased spending is likely (but not certain) to improve student performance measurably.
But note that the second chart, the 50 states-only data set, shows a much narrower range of scores (and all of them are in the "Basic" area). Even with the r-value of .45, we are still faced with the fact that the difference between the lowest and highest performing states is only 10% of the highest state’s score. A state could drastically increase student spending and find itself with only slight increases in performance. For example, Utah (the lowest spending state) could double its spending, but given these data it seems unlikely that Utah’s test results would exceed the performance of New Hampshire, and there might well be much less of an improvement.
An increase of 8 or fewer points out of 243, with no movement of the statewide average from Basic to Proficient, sounds like very little gain for doubling expenditures. In other words, a measurable improvement is not necessarily the same thing as an impressive one.
Summary
Increased spending (above current state spending levels) seems to have only a small scale effect on student performance. A state increasing its education budget might raise its rank relative to other states, but is unlikely to see a large statewide increase in absolute terms.
A few questions raise themselves:
(1) Do these results hold up across a broader data set?
(2) What drives the variations we do see in state performance?
(3) What does Washington, D.C. tell us?
(4) What does the narrow range of scores tell us?
The Data
For the mathematically minded, here is a table of the source data. After the table you’ll find links to the raw data sources.
State NAEP math score Per-pupil instructional spending
Alabama 223 $3,868
Alaska 233 $5,728
Arizona 229 $3,168
Arkansas 229 $3,924
California 227 $4,666
Colorado 235 $4,212
Connecticut 241 $6,458
Delaware 236 $6,002
District of Columbia 205 $6,976
Florida 234 $3,793
Georgia 230 $4,916
Hawaii 227 $4,833
Idaho 235 $3,720
Illinois 233 $5,052
Indiana 238 $4,782
Iowa 238 $4,474
Kansas 242 $4,401
Kentucky 229 $4,068
Louisiana 226 $4,161
Maine 238 $5,826
Maryland 233 $5,467
Massachusetts 242 $6,336
Michigan 236 $4,889
Minnesota 242 $5,178
Mississippi 223 $3,462
Missouri 235 $4,422
Montana 236 $4,589
Nebraska 236 $4,967
Nevada 228 $3,807
New Hampshire 243 $5,217
New Jersey 239 $7,085
New Mexico 223 $3,870
New York 236 $8,375
North Carolina 242 $4,197
North Dakota 238 $4,326
Ohio 238 $4,881
Oklahoma 229 $3,458
Oregon 236 $4,407
Pennsylvania 236 $5,739
Rhode Island 230 $5,704
South Carolina 236 $4,169
South Dakota 237 $3,913
Tennessee 228 $4,002
Texas 237 $4,286
Utah 235 $3,123
Vermont 242 $6,438
Virginia 239 $4,814
Washington 238 $4,270
West Virginia 231 $5,044
Wisconsin 237 $5,532
Wyoming 241 $5,507
The following links were valid as of July 6, 2007:
http://nces.ed.gov/nationsreportcard/mathematics/results2003/stateavgscale-g4.asp
http://www.census.gov/govs/www/school.html
This is the first of a series of blog posts on school performance.
I went into this project believing that the NEA and friends were dead wrong, that increasing school spending beyond current levels would be very unlikely to have any effect on school performance. The truth turns out to be a bit more complicated, but ultimately I think that drastic increases in school spending will not by themselves greatly improve performance.
The Bottom Line
There are three bottom lines (which really means, there’s room for more interesting analysis and future blog posts).
- Among the 50 states, more school spending correlates moderately well to improved performance. r=.45
- However, the performance differences among states are so small that “improved performance” may not be worth the additional money necessary to get it.
- If you include Washington, D.C., the data get really skewampus and the correlation between spending and performance drops like a rock. r=.17
The gold standard for student performance in the United States is the NAEP, National Assessment of Education Progress, administered by the National Center for Education Statistics (a research arm of the federal Department of Education). Why do I call it the gold standard? Because NAEP is the only assessment administered to representative groups of students in multiple grades (4th and 8th) across all 50 states. The NCES publishes a ton of data relating to NAEP.
By contrast, state-specific measures are only given in one or a few states. Some may be as good as NAEP or even better (in terms of accurately assessing performance), but since they are state-specific they aren’t useful for cross-state comparisons.
NAEP results let us compare states to see how educational practices—including spending policies—and other factors impact student performance.
Raw Data: Per-student Instructional Spending
I have used per-pupil insructional spending data gathered by the US Census Bureau. The link http://www.census.gov/govs/www/school.html has spending data from all fifty states and Washington, D.C., arranged in various ways. The data are available in CSV and Excel formats, making it easy to do your own analyses.
The census school spending data files also have per-pupil total spending, adminstrative spending, and other interesting information. I have chosen to use instructional spending because the arguments made for increased school spending generally center around getting more money “into the classroom.”
Data and Tools
For today’s introductory blog post on education, I have compared NAEP 4th grade math scores, from the 2003 administration of the test, to per-student spending in the school year 2002-2003. This is a very simplistic model. For example, a fourth-grader’s performance depends not only on his school environment during fourth grade, but also his educational experience in grades one, two, and three.
A better analysis would use a three year average of spending data with the third year being the same as an NAEP test administration year. This averaging would hopefully capture the time-based effects of spending on student performance, and minimize the effects of year-by-year spending changes at the state level. I’ll try to do that in a future post, and also look at other NAEP data.
Charts and statistical calculations courtesy of Microsoft Excel 2004 for Macintosh.
A Picture is Worth a Thousand Words
Let’s start with a picture (click to see the large version):
As you see, NAEP scores range from 0 to 500 possible. The state averages from the year 2003, however, cluster much more closely. The 50 states range in value from 223 to 243; the outlier at the left, with 205, is the District of Columbia. An individual student scoring at least 282 would be considered Advanced; one between 249 and 282 would be Proficient; one between 214 and 249 would have Basic ability; and one below 214 would show "below Basic" ability with math. Since very few people are either "Advanced" or "below Basic" we should expect the statewide averages to be in the "Basic" and "Proficient" ranges; as it turns out, all states average "Basic" (we'll discuss that another time).
Spending is fairly diverse, ranging from $3123 for Utah to $8375 for New York.
One interesting point is that D.C. has the third highest per-pupil expendture (nearly $7000 per year on instructional spending) while showing by far the lowest scores. Clearly, D.C. does not seem to suggest that more spending improves educational results. However, D.C. is so different from the 50 states in so many ways that I hesitate to draw too many conclusions.
Here is the same chart, without Washington, D.C. and shortened to show only the actual performance (not the cut scores). Click to see the large version.
No Trend? Trend?
Just glancing at the first chart, spending doesn’t seem to have much to do with performance. There’s a slight effect; 6 of the 8 highest performing states spend more than $5000 per year, while 7 of the 8 lowest performing states spend less than $5000 per year (7 of the 9 lowest if you include Washington, D.C.).
On the other hand, the middle-performing “clump” includes both the highest and lowest spending states. Overall, the results are messy in the first chart.
The second chart zooms in on the state data and shows a pattern more clearly; higher spending states seem to do somewhat better than lower spending states.
In statistics, a calculated number called Pearson’s r shows the amount of correlation between two sets of data. If r = 1, then the two data sets are perfectly correlated; that is, an increase in one will automatically increase the other, and the change in one is sufficient to account for all change in the other. The closer r is to 0, the less correlated (or in other words, the less closely related) the two data sets are. Thus an r value of 0.8 means a strong correlation, and an r value of 0.1 means a very weak correlation.
The r-value for the data set with Washington, D.C. is .17, a weak correlation. However, if we discard Washington, D.C., then the correlation (r-value) among the 50 states jumps to .45. This is a moderate to strong correlation, and indicates that increased spending is likely (but not certain) to improve student performance measurably.
But note that the second chart, the 50 states-only data set, shows a much narrower range of scores (and all of them are in the "Basic" area). Even with the r-value of .45, we are still faced with the fact that the difference between the lowest and highest performing states is only 10% of the highest state’s score. A state could drastically increase student spending and find itself with only slight increases in performance. For example, Utah (the lowest spending state) could double its spending, but given these data it seems unlikely that Utah’s test results would exceed the performance of New Hampshire, and there might well be much less of an improvement.
An increase of 8 or fewer points out of 243, with no movement of the statewide average from Basic to Proficient, sounds like very little gain for doubling expenditures. In other words, a measurable improvement is not necessarily the same thing as an impressive one.
Summary
Increased spending (above current state spending levels) seems to have only a small scale effect on student performance. A state increasing its education budget might raise its rank relative to other states, but is unlikely to see a large statewide increase in absolute terms.
A few questions raise themselves:
(1) Do these results hold up across a broader data set?
(2) What drives the variations we do see in state performance?
(3) What does Washington, D.C. tell us?
(4) What does the narrow range of scores tell us?
The Data
For the mathematically minded, here is a table of the source data. After the table you’ll find links to the raw data sources.
State NAEP math score Per-pupil instructional spending
Alabama 223 $3,868
Alaska 233 $5,728
Arizona 229 $3,168
Arkansas 229 $3,924
California 227 $4,666
Colorado 235 $4,212
Connecticut 241 $6,458
Delaware 236 $6,002
District of Columbia 205 $6,976
Florida 234 $3,793
Georgia 230 $4,916
Hawaii 227 $4,833
Idaho 235 $3,720
Illinois 233 $5,052
Indiana 238 $4,782
Iowa 238 $4,474
Kansas 242 $4,401
Kentucky 229 $4,068
Louisiana 226 $4,161
Maine 238 $5,826
Maryland 233 $5,467
Massachusetts 242 $6,336
Michigan 236 $4,889
Minnesota 242 $5,178
Mississippi 223 $3,462
Missouri 235 $4,422
Montana 236 $4,589
Nebraska 236 $4,967
Nevada 228 $3,807
New Hampshire 243 $5,217
New Jersey 239 $7,085
New Mexico 223 $3,870
New York 236 $8,375
North Carolina 242 $4,197
North Dakota 238 $4,326
Ohio 238 $4,881
Oklahoma 229 $3,458
Oregon 236 $4,407
Pennsylvania 236 $5,739
Rhode Island 230 $5,704
South Carolina 236 $4,169
South Dakota 237 $3,913
Tennessee 228 $4,002
Texas 237 $4,286
Utah 235 $3,123
Vermont 242 $6,438
Virginia 239 $4,814
Washington 238 $4,270
West Virginia 231 $5,044
Wisconsin 237 $5,532
Wyoming 241 $5,507
The following links were valid as of July 6, 2007:
http://nces.ed.gov/nationsreportcard/mathematics/results2003/stateavgscale-g4.asp
http://www.census.gov/govs/www/school.html
posted by Steve Setzer | 10:13 PM
0 Comments:
Post a Comment
<< Home