First Approximation: Schools + Testing = Confusion


An ongoing series of articles which shares the results of one parent’s quest to understand the local school system and to provide an education for his children.

A simple question

It would seem to be a simple question. How well are our schools doing in educating our children?

You can look at your own child’s scores but that doesn’t really tell you what it seems at first to tell you. Is your kid a B-plus student where the average is a C or a B-plus student where the average is an A-minus?

Likewise with the schools themselves. If you want to know how well a school or a school system is doing you cannot just look at its scores in isolation. You need to see them in context. See them compared with its own scores from the past, see its scores compared with other school systems within the same state, see our state compared with other states, with the nation as a whole, with other countries.

You may have seen the graphs published earlier here in Montara Fog comparing our school district to other school districts in San Mateo County and comparing California to other states. The focus was on spending per pupil and those graphs showed that California ranks very low. At the national level California is an astonishing thirty-third.

Although I didn’t display a chart about achievement scores you may have heard that California ranks low when compared to other states.

The message here seems (at first) clear enough: California schools (including ours) are on the whole under-funded and under-performing.

The national test

Last week the American Institutes for Research published the results of a study comparing achievement scores from each of the states with other nations. You can download the entire study here.

The data in this study did not use the state achievement tests. It used the results of a national test, administered by the federal government.

I didn’t know there was a national test!

The difference here (and it is a biggie) is that the federal government uses the same test from state to state. Apples to apples, to use an old phrase. This much better than trying to compare widely divergent state tests.

This first pair of charts show California (note the orange arrow!) and the results of the federal math and science tests.

Now, adding further to the gloom, the math and science tests as compared to other nations.

Clear as an unmuddied lake? Clear as an azure sky of deepest summer?

Always read the footnotes

There are serious issues with all of these tests, so serious that you may find that what at first seemed clear becomes a hopeless muddle on closer inspection.

Take the state tests, for starters. Every state runs its own testing program and every state test is different. Right away you can see that this presents a great difficulty. How do you compare tests with different questions?

It gets worse. You might be surprised to discover that when states started testing some states where unhappy with the results. They did not score well as they wanted to, they did not score as well as other states. What to do?

The hopelessly optimistic among you may suggest that states scoring low in the tests would focus their energies and resources into building better schools, training teachers, providing computers, textbooks, and supplies. As if fueling an under educated, gear hungry mob is going to produce great national race to the top where even the last place state would be a winner.

I’ll pause a minute to let the laughter subside.

In the real world, what many states did was to dumb down the tests so they would have higher scores. Gasp. (Reality check: The race-to-the-top dream costs a zillion dollars and superhuman political will. Changing the tests’ difficulty may have cost as little as fifty bucks and a cup of coffee at Starbucks–and as you did this you would be showered with praise. Which would you choose in the end?)

Some schools, however, stood their ground. Some schools did not dumb down their tests. That seems especially courageous as time goes on because the schools with the more difficult tests would fall in the school rankings. Those schools start to look bad.

So, how to identify a state with a good test? Luckily for us the Thomas B. Fordham Foundation has provided a web page with a lively interactive map. As you mouse over each state a text box gives you a summary evaluation of that state’s achievement tests–not how well the students do on the tests but how well the state designs and implements the tests. Click on a state for a detailed review.

The site is highly interesting and worth an hour of your time to explore but for the moment note that only three states get straight A’s: Massachusetts, Indiana and…wait, here it comes…California.

A good idea

One way to get around the problem of dumbed-down tests and to allow us to make real comparisons between states is to have students in each state take the same test.

And that is just what the National Assessment of Educational Progress does. The charts above are derived from NAEP data. Testing with a single, national test gets around many of the problems described above and has the potential to offer us a clear picture of the state of education in the United States.

Sounds great. What’s the catch?

[Note, in an earlier version of this text I said that the NAEP used only 100 schools in the nation for their sample. This was based upon text at their web site that in hindsight was ambiguous. A reader pointed out a different, clearer explanation of the NAEP sampling elsewhere on the site–the actual samples are about 100 schools per state according to that text. Another part of the site says “100 schools per jurisdiction” and claims that 45-55 such jurisdictions typically participate in a survey.]

If the NAEP uses one hundred schools to represent a state the size of California–is that enough?

One education professional expressed to me the worry that the samples in California, in an effort to meet the needs of matching the demographic of the sample to the demographic of the state, ended up being at schools and in school districts that are more urban and low-performing.

The NAEP claims, however, that the selection is random (subject to various constraints).

Which leaves parents in a muddle.

Test your own kids

One way out of the muddle might be to decide what exactly is the question you want to ask. Do you want to know how well the school is doing–or are you really more interested is how well your child is doing? Those are very different questions.

To help answer the second you might want to test your child with the same test used by the NAEP (the national test). They have a top-notch web page which includes questions from prior tests and an extraordinary answer key, complete with the answers, sample answers from children in their own handwriting, a grading guide, and a graph indicating how well students did on each question.

The other day my kids were playing school, so I gave them a print out of the 4th grade science test and they tested each other. Fun for them, highly informative for me.

This is a good place to start.

PDF Files:

4th Grade Math Q’s (Questions only, for printing)

4th Grade Math Answers (Questions and detailed key, for viewing on-screen)

4th Grade Science Q’s (Questions only, for printing)

4th Grade Science Answers (Questions and detailed key, for viewing on-screen)

8th Grade Math Q’s (Questions only, for printing)

8th Grade Math Answers (Questions and detailed key, for viewing on-screen)

8th Grade Science Q’s (Questions only, for printing)

8th Grade Science Answers (Questions and detailed key, for viewing on-screen)

Get more tests (Civis, Economics, History, etc) at the NAEP Questions Page.


Photograph and Charts by Darin Boville

Comments are closed.