(copy from internet, by Ron Kirkley)
In this two part study Berg and Smith of the University of Wisconsin-Milwaukee
examined the effectiveness of measuring student's ability to construct and interpret graphs
by using a free response instrument. Because of the great importance of graphing in
understanding science concepts and the use of graphs to convey information in the popular
media graphing skills are essential. Therefore an accurate assessment of graphing skills
and an accurate assessment of strategies used to teach graphing skills is important.
Since most studies involving graphing skills rely on multiple choice formats for
assessment then its important to know how effective multiple choice is for assessing
graphing skills. They found students could answer the free-response questions correctly
significantly more often than the multiple choice questions. They attribute this to the fact
that students were forced to think through the free response questions and that the multiple
choice format does not allow for "wrong answers with correct reasoning". Because of the
differences in success of students on the two formats of questions Berg and Smith
question the validity of studies that use multiple choice instruments to measure gains in
In the first study Berg and Smith modified three multiple choice questions used in
previous MBL studies (Barclay, 1986 and by Mokros & Tinker, 1987) into free-response
questions. They then used clinical interviews to administer the three questions to 72
seventh, ninth and eleventh graders balanced for sex, grade level and academic ability as
indicated by math and science grade point averages. The students were given the
situations and asked to draw a graph on a prelabeled/scaled graph. They were then
interviewed to determine the reasoning for the graph they constructed. Berg and Smith
controlled for interviewer affects by videotaping the interview sessions and analyzing the
tapes for interviewer influences and also rescoring to check for interviewer reliability.
In this first study Berg and Smith found the students were more successful giving correct
responses than students in the previous MBL studies on their multiple choice pretests.
They attribute this to the multiple choice format eliciting a superficial first reaction
response. Also the multiple choice format does not allow for different interpretations of the
given situations. Often the students would interpret a given situation slightly differently
which would in turn change their graph. This difference in reasoning only becomes
In the follow up study Berg and Smith directly compared the use of free-response
questions to multiple choice questions. They studied 1416 subjects from urban and
suburban, public and private schools, grades 8 through 12 balanced for sex and ability.
They used the same three questions used during the first study giving half of the subjects a
multiple choice instrument and the other half a free response instrument. The only
difference in the two instruments was that the students were required to draw in their
response on a prelabeled graph for the free-response instrument. No interviewer was used
this time; instead the items were scored based on the results of the various categories of
responses identified during the first study.
Berg and Smith found students scored as much as 19% better on the free-response than
on the multiple choice. They also found this difference was not consistent for grade levels
or ability levels but was consistent for sex. From the two studies Berg and Smith
determined there are enough questions about the use of the multiple choice instrument to
create questions about the previous MBL studies. Were the gains merely an increased
ability to recognize a correct graph or did the subjects gain an increased understanding of
graphing? Also if MBL is to be used to construct student understanding shouldn't we be
using an instrument that measures this constructed understanding?
This study seems to be well planned and executed even if somewhat broad. I was
surprised by the level of success students had in the first study in supplying a correct
response for each of the situations. I would like to know more about the students' previous
experience with graphing. The numerous graphs, tables and examples of questions did a
good job providing information but were often several pages from the associated text. The
use of the first somewhat qualitative study in formulating the free-response questions for
the second study would be a good model for developing more appropriate, more effective
instruments for future studies involving strategies for improving the instruction of graphing