BACKGROUND

Of course, assessment work is done in the discipline of political science, and the pretest/posttest approach is not uncommon. Kelly and Klunk (Reference Kelly and Klunk2003) report that about 10% of 213 departmental surveys returned to them use the pretest and posttest method of assessment. Presumably, this is for political science courses, although the article does not state that explicitly.

The previous efforts demonstrate that learning facts and learning principles do indeed take place in political science classes in a variety of contexts. Anderson and Harsell (Reference Anderson and Harsell2005) administer pretests and posttests over 15 semesters to students in the Quantitative Methods in the Social Sciences course at the Maxwell School at Syracuse University. Students are asked to rate knowledge and comfort levels with computers, spreadsheets, equations, variables, and so on. The authors also randomly sample alumni and ask them whether they have had an opportunity to use information that was learned in the course. The analyses yield positive results on both counts.

Harkness (Reference Harkness2005) administers pretests and posttests to students in four sequential political science courses—America Government, Legislative Process, Research Skills, and Research Methods—over four semesters. Students in the courses show not only evidence of increased knowledge, but also evidence of retention of key concepts as they progress.

Alperin (Reference Alperin2007) describes how members at the University of Wisconsin–River Falls use pretests and posttests containing 65 multiple-choice questions to measure the government department's assessment goals for the introductory American government course. A portion of those questions are then inserted into traditional exams over the course of the semester, and students leave the class with “substantially more knowledge” than when they entered (Alperin Reference Alperin2007, 4). Similarly, the present analysis, as indicated below, employs 50 multiple-choice knowledge questions.

Omelicheva and Avdeyeva (Reference Omelicheva and Avdeyeva2008) measure student learning after exposure to six class periods of debate, on the one hand, and six class periods of lecture, on the other. They find that students score somewhat higher on factual knowledge after the lectures, but higher on comprehension after the debates. Comprehension is measured by asking students to explain what a concept means. Although exposure to debates results in students seeing how their knowledge can be used in situations not related to class is somewhat greater than in exposure to lectures, neither lectures nor debates excel in promoting application. The same is true of critical evaluation. Application is measured with hypothetical scenarios and critical evaluation is measured with ability to make a reasoned argument.

In connection with knowledge of current events, Huerta and Jozwiak (Reference Huerta and Jozwiak2008) investigate whether using the New York Times in general-education political science classes at Texas A&M University increases student engagement inside and outside the classroom. Students in four courses are given a pretest and a posttest containing questions on civic engagement. In a United States government course and a state and local government course, the New York Times is used. In a biology course and a United States government course, it is not. They find that exposure to the New York Times does not have a positive impact on the statement “I have a positive attitude toward politics,” which tends to be consistent with some of the other findings on political efficacy noted below. However, it does have a significant positive effect on community involvement measured as an index containing attitudes and actual activities.

The previous efforts yield mixed results with respect to stimulating political efficacy among students. In their study of traditional vs. Web-based American government classes, Botsch and Botsch (Reference Botsch and Botsch2001) administer a pretest and posttest on knowledge gained by students and administer a demographic survey. Consistent with the pattern of other studies, the value added for student knowledge of facts and principles is virtually identical in both versions of the course. The authors note, however, that GPA is a strong predictor of the knowledge gained in the traditional classes. Other factors, such as gender, have no impact. And students in both versions show increased political interest and political efficacy. Political interest is defined by asking (Botsch and Botsch Reference Botsch and Botsch2001, 137): “Would you say that you follow what is going on in government and public affairs (4) most of the time, (3) some of the time, (2) only know and then, or (1) hardly at all?” Political efficacy is defined by asking (137): “Sometimes politics and government seem so complicated that a person like me can't really understand what's going on. Do you (1) strongly agree, (2) agree, (3) have mixed feelings, (4) disagree, or (5) strongly disagree?”

Crawford (Reference Crawford2007) reports the results of pretests and posttests for seven introductory political science sections at a private midwestern university. Employing a 2–10 scale on internal efficacy, she compares sections emphasizing a rational-choice perspective to those that do not. The scale is based on the students' perceptions of how well qualified they are to participate in public affairs and whether they feel they are better informed than most. There are similar, statistically significant improvements for internal efficacy in both varieties of sections.

Strachan (Reference Strachan2006), however, does not find that political efficacy increases after employing a pretest and posttest for a one-credit course at the University of Albany designed to stimulate political efficacy, convince students that political participation can influence political outcomes, and make them feel more comfortable in engaging in such activities. Again, Likert scales are used to tap these dimensions. Efficacy is measured with a 5-point scale asking students to indicate their agreement that “my vote doesn't matter” and “it is difficult for someone like me to make a difference” (913).

The present analysis, as indicated below, includes 10 scaled questions regarding the extent to which students believe that knowledge of the United States government is important. Although somewhat different from the above indicators, this scale may also be used to tap political efficacy. The battery of questions is in appendix A.

DATA AND METHOD

We used a Solomon Four-Group Design to assess student learning outcomes in United States government classes at Sauk Valley Community College in Illinois and The University of Scranton in Pennsylvania (a private comprehensive institution). At each, a pretest and posttest was administered to students enrolled in United States government classes and students who had never enrolled in a college-level political science or government class, the latter selected to be representative of a variety of majors outside political science. In addition, the posttest was administered to a group of students who completed the United States government course, but were not given the pretest, and a group of students who have never enrolled in a college-level political science or government class and were not given the pretest. Hence, the four-group design: pretest/class/posttest (experimental group), pretest/posttest (control group), class/posttest, and posttest only.

The instrument is comprised of four parts. One contains demographic questions, such as gender, year, and major. Another contains 50 multiple-choice questions designed to measure knowledge of the United States government, previously employed at Sauk Valley Community College. The third contains 20 multiple-choice questions designed to measure knowledge of current events over the two-week period preceding administration. The fourth contains 10 scaled questions from the U.S. government attitudinal survey and the 10 scaled questions from the U.S. government self-confidence survey, both previously employed at Valencia Community College.

UNITED STATES GOVERNMENT LEARNING OUTCOMES

In the experimental group, background knowledge went from an average of 61% correct responses to an average of 72% correct responses, while current events went from 50% to 58%. Both are statistically significant at the .05 level, using the t statistic for difference of means. The control group went from 52% to 54% and 43% to 47%, respectively, and neither is statistically significant.

The United States government posttest only for background knowledge is 66% (statistically significant compared to the initial 61% in the experimental group). For current events, it is 54% (not statistically significant compared to the initial 50%). This suggests a modest possibility that the pretest may have impacted current-event performance. The control posttest only for background knowledge is 54%, very similar to the initial 52%. Somewhat surprisingly, the control posttest only for current events is 52% (statistically significant compared to the initial 43%). We have no explanation for this anomaly.

For both the attitudinal and self-confidence batteries, there are 10 questions arrayed on a scale of 1 to 5. Therefore, for each respondent, the lowest possible score across each set of 10 items is 10, and the highest 50. Low scores indicate more self-confidence and better attitudes. In the experimental group, for self-confidence, the pretest averages 25 and the posttest averages 19. This is statistically significant. On attitudes, the pretest and posttest are both right at 19. Apparently, self-confidence improves among students in the United States government classes, but their conviction that knowledge is important does not increase (although starting at a fairly healthy level). This is strikingly similar to Strachan's (Reference Strachan2006) finding that students' political efficacy did not improve significantly as a result of the one-credit course used in her study. She suggests that political efficacy is more likely based on long-term socializing agents. Our control group went from 21 to 23 on attitudes, while self-confidence was stable at 28. Neither is statistically significant.

The United States government posttest only for attitudes is 20, very similar to the experimental group. For self-confidence it is 21, similar to the posttest in the experimental group. The posttest only for the control group was remarkably similar to the other numbers the for control group.

A CLOSER LOOK

The previous section warrants the general conclusion that students in the United States government courses, compared to the control group, improve in their knowledge of the United States government and of current events. They also perceive their increased knowledge, although do not attach any more importance to it at the end of the semester than at the beginning.

However, it may be that the United States government courses are populated with students who are more likely to be receptive to increasing their knowledge. In particular, this might be true of political science majors, and those with higher GPAs. The literature, and our previous research, suggest it may also be true of males. A gender gap has been noted with respect to both interest in, and knowledge of, public affairs and current events (Ford Reference Ford2002).

Table 1 reports our investigation of these factors. As was the case with the initial analysis, for background knowledge and current events, the mean is the mean percentage of correct responses to the questions. Background knowledge improves from 61% to 72%. The degree of improvement is the same for males and females, although males start with a higher level of background knowledge. The pattern of improvement is also similar for high/low GPA and for political science majors vs. others. As with gender, political science majors begin with higher levels of knowledge.Footnote ¹ Although not listed in the table, it is worth noting that female political science majors improve very substantially, from 57% to 73% (male political science majors go from 75% to 81%). Apparently, exposure to a United States government course has a very positive impact on female majors who start with less knowledge.

Table 1. Before and After Means in the Experimental Group

* Significant at the .05 level (t statistic for difference of means)

Knowledge of current events improves from 50% to 58%. The degree of improvement is the same for males and females, although, again, males start with higher levels of knowledge. Those with higher GPAs improve more than those with lower GPAs. In addition, political science majors improve more than other majors.

Self-confidence improves from 25 to 19. The improvement is remarkably similar across genders and GPAs. Political science majors start with more self-confidence and do not improve quite as much. (21 to 17). There is virtually no improvement in attitudes (20 to 19) and the pattern is consistent across genders, GPAs, and majors.

The improvement in the United States government classes in background knowledge, knowledge of current events, and self-confidence is not limited to those students who might come into the courses predisposed toward improvement. It is also true, however, that males and those with higher GPAs come into these courses with somewhat higher levels of background knowledge. Clearly, all students gain self-confidence in their knowledge of government, while political science majors bring more initial self-confidence. However, the students' perception of the importance of having such knowledge does not appear to improve at all. This may be a source of some concern.

We also look at the pattern in the United States government courses within categories of father's education, within categories of mother's education (at least some college and no college for each), and reported household income (divided at the median). For background knowledge and self-confidence, there are statistically significant improvements in all groups. However, for those whose fathers had at least some college, and for those whose mothers had at least some college, there is not a statistically significant difference for current events. In both instances, knowledge of current events starts relatively high and remains there. Therefore, for current events, parents' educational backgrounds may be as important as exposure to a United States government course.