Testing: How Many Questions Should I ask?

This year I have switched to asking only short-answer questions on my American Government exams, instead of using multiple choice questions (and I have only rarely used long essay questions for that class), primarily to push the students to learn instead of memorize and to develop their writing skills (two areas that have suffered from NCLB).

It seems to be working, but I have a practical question:

How many short-answer questions do I need to ask on a midterm exam to get an accurate assessment of a student’s learning?

All the questions are in the syllabus, right beside the day’s reading, because A) I want to make it clear to the students just what I actually expect them to know, and B) I select one from the day’s reading for a start-of-class quiz. There were 29 questions on the syllabus prior to the midterm, and I semi-randomly selected 12 of them and required students to answer 10. I am now thinking that 10 may not be optimal, for several reasons.

  1. One is simply the time I invest in grading–I would like to minimize that without degrading the quality of the evaluation.
  2. The second is that a couple of students only answered 9. They weren’t doing too badly, so I don’t think inability to write at least something halfway plausible about the others was the problem. Rather, I think it’s just easier to lose track or miscount with a higher number than a lower number, so there’s a question of whether my approach had the effect of setting up some students to perform more poorly. It would be easy to criticize them for not counting properly, but that’s now what I think I should be grading them on in an American Government class.
  3. Finally, there’s the simple question of efficiency. If a lower number of questions is sufficient to assess student learning, what point is there in asking more? The test itself is not a part of the learning process–it is the studying for the test that is part of the learning process. If I thought it would result in sufficient study effort to promote learning and would be fair, I’d happily assign only one answer.
  4. Given that, how many questions should I assign?

Advertisements

About James Hanley

James Hanley is Associate Professor of Political Science at Adrian College and a Fellow of the Institute for Social Policy and Understanding. The views expressed here do not reflect the views of either organization.
This entry was posted in Uncategorized. Bookmark the permalink.

8 Responses to Testing: How Many Questions Should I ask?

  1. AMW says:

    I like to allow for an endogenous selection of exam length. On my final exam (though not the mid terms – yet) I give them 2.5 hours to answer 4 problems out of 5, or 1.5 hours to answer 3 problems out of 5. I find that cuts down the grading time considerably while retaining pretty much the same grade distribution.

    You could do something similar. Pick, say, 3 questions that everyone has to answer. Put that in Section 1. Section 2 contains 9 questions. Give them multiple tracks. Track 1 answers a total of 10 (all in Section 1 and 7 of 9 in section 2), but gets the whole class period (75 minutes?). Track 2 answers a total of 7 (all in section 1 and 4 of 9 in section 2), but has to turn in the exam 30 minutes early. You could even have a track 3: 5 questions in 30 minutes.

  2. Dr X says:

    Okay, my test-design and construction class was years ago and I don’t do anything with stats anymore. I understand some principals, but I leave the judgments of test-worthiness to test-makers and reviewers. So take this with a grain of salt (though I don’t think I’m too far off).

    First, drop the choose-10-out-12 business. This does nothing to reduce standard error or improve reliability (i.e., smaller confidence intervals for scores obtained) and these are the numbers I believe you want to know. Moreover, as you note, the option to choose a subset of the questions may actually increase error and reduce reliability. There’s no upside, except perhaps the student’s false perception that you’re giving them some sort of beneficial wiggle room.

    Once you have the numbers, it boils down to: how much increased error and decreased reliability you’re willing to trade for your incremental investment of time? The latter is entirely subjective, but you could pick a minimum reliability and maximum standard error you consider acceptably fair to your students and go with that. Next get to a statistics geek because I don’t think you or I can reasonably come up with answers just by eyeballing this problem. I think that anyone giving you an eyeball estimate, unless they have a lot of experience doing these analyses, is just shooting blanks in the dark.

    I think there are a number of ways to approach this problem in the neighborhood of Chronbach’s alpha and something or other Spearman calculation applied to that (I’m mostly guessing, I really don’t remember). Or maybe you can develop other measures of error and reliability over the course of several sections of teaching the course and collecting data.

  3. lancifer666 says:

    How much time is allotted for the exam?

  4. James Hanley says:

    Short answe: plenty of time.

    Long answer: the whole 100 minute class period. Nobody had trouble finishing on time, neither the Fs nor the As.

  5. lancifer666 says:

    Then I think you should be creative and institute a bold testing regimen that you feel will accurately reflect the student’s assimilation of the core principles and facts of the curriculum. Make them perform interpretive dance, build abstract mobiles or write all or nothing term papers in Esperanto.

    I work in a huge state sponsored university and I have about as much freedom and influence over the curriculum as a flea has over the path of an African elephant. But, to be depressingly honest, I can pretty much assign grades after the first ten minute, three question quiz.

    Be bold my friend! Do it for us powerless little cogs in the big education machine.

  6. James Hanley says:

    Interpretive dance? What did I ever do to make you hate me, Lance? Did I take your last bottle of beer that day or something?

    (There was some kind of running gag between my friend and I at an Oregon football game back in the late ’90s that had something to do with me doing an interpretive dance at half time. Why, or what it was responding to, I no longer remember. I remember only that it was just a running gag, and I did not dance, nor did I have any intention of ever dancing, nor did my friend at any time expect me to dance.)

    (But that does remind me of the time John Glenn went up in the space shuttle, and there were jokes (even in those primitive internet days, the jokes were burning up the internet) about all wearing chimp masks when he returned. Sometime during a football game they announced his safe return to earth, and the same friend turned toward me in the stands and yelled, “Hanley, where’s your chimp mask?” Without a pause I yelled back, “You were supposed to get the masks; I was up all night burying the Statue of Liberty!” That’s the only time I’ve ever gotten a laugh from an actual crowd of people. Precious memories. Precious memories, indeed.)

  7. lancifer666 says:

    I love those movies.

    “Take your sticking paws off of me you damn dirty ape!”

    Chuck Heston was the perfect ham for those sci-fi classics. I also enjoyed his over the top performances in The Omega Man and Soylent Green.

    “IT’S PEOPLE! SOYLENT GREEN IS PEOPLE!”

  8. James Hanley says:

    Yes, but the best was “Planet of the Apes, the Musical.” (Part 1; Part 2.)

Comments are closed.