Berg, C.A. & Smith, S.

Assessing students' abilities to construct and interpret line graphs: disparities between multiple-choice and free-response instruments.

Science Education, 78, 6, 527-554, 1994
Abstract

(copy from internet, by Ron Kirkley)

In this two part study Berg and Smith of the University of Wisconsin-Milwaukee

examined the effectiveness of measuring student's ability to construct and interpret graphs

by using a free response instrument. Because of the great importance of graphing in

understanding science concepts and the use of graphs to convey information in the popular

media graphing skills are essential. Therefore an accurate assessment of graphing skills

and an accurate assessment of strategies used to teach graphing skills is important.

Since most studies involving graphing skills rely on multiple choice formats for

assessment then its important to know how effective multiple choice is for assessing

graphing skills. They found students could answer the free-response questions correctly

significantly more often than the multiple choice questions. They attribute this to the fact

that students were forced to think through the free response questions and that the multiple

choice format does not allow for "wrong answers with correct reasoning". Because of the

differences in success of students on the two formats of questions Berg and Smith

question the validity of studies that use multiple choice instruments to measure gains in

graphing comprehension.

In the first study Berg and Smith modified three multiple choice questions used in

previous MBL studies (Barclay, 1986 and by Mokros & Tinker, 1987) into free-response

questions. They then used clinical interviews to administer the three questions to 72

seventh, ninth and eleventh graders balanced for sex, grade level and academic ability as

indicated by math and science grade point averages. The students were given the

situations and asked to draw a graph on a prelabeled/scaled graph. They were then

interviewed to determine the reasoning for the graph they constructed. Berg and Smith

controlled for interviewer affects by videotaping the interview sessions and analyzing the

tapes for interviewer influences and also rescoring to check for interviewer reliability.

In this first study Berg and Smith found the students were more successful giving correct

responses than students in the previous MBL studies on their multiple choice pretests.

They attribute this to the multiple choice format eliciting a superficial first reaction

response. Also the multiple choice format does not allow for different interpretations of the

given situations. Often the students would interpret a given situation slightly differently

which would in turn change their graph. This difference in reasoning only becomes

apparent during interviews.

In the follow up study Berg and Smith directly compared the use of free-response

questions to multiple choice questions. They studied 1416 subjects from urban and

suburban, public and private schools, grades 8 through 12 balanced for sex and ability.

They used the same three questions used during the first study giving half of the subjects a

multiple choice instrument and the other half a free response instrument. The only

difference in the two instruments was that the students were required to draw in their

response on a prelabeled graph for the free-response instrument. No interviewer was used

this time; instead the items were scored based on the results of the various categories of

responses identified during the first study.

 

Berg and Smith found students scored as much as 19% better on the free-response than

on the multiple choice. They also found this difference was not consistent for grade levels

or ability levels but was consistent for sex. From the two studies Berg and Smith

determined there are enough questions about the use of the multiple choice instrument to

create questions about the previous MBL studies. Were the gains merely an increased

ability to recognize a correct graph or did the subjects gain an increased understanding of

graphing? Also if MBL is to be used to construct student understanding shouldn't we be

using an instrument that measures this constructed understanding?

 

This study seems to be well planned and executed even if somewhat broad. I was

surprised by the level of success students had in the first study in supplying a correct

response for each of the situations. I would like to know more about the students' previous

experience with graphing. The numerous graphs, tables and examples of questions did a

good job providing information but were often several pages from the associated text. The

use of the first somewhat qualitative study in formulating the free-response questions for

the second study would be a good model for developing more appropriate, more effective

instruments for future studies involving strategies for improving the instruction of graphing

skills. (Ron Kirkley)

Annotatie