from Inquiry, Volume 1, Number 2, Fall 1997, 64-66
© Copyright 1997 Virginia Community College System
Abstract
This locally-developed
instrument provides student biographical data, insight on
students’ self-esteem, and a means of measuring
students’ knowledge in the eight areas of general education.
In 1987, public colleges and universities began measuring the general education knowledge of their students under a mandate from the State Council for Higher Education in Virginia (SCHEV). In 1989, the Virginia Community College System (VCCS) appointed a task force to develop a definition of general education. The resulting definition included eight elements which were to “embody the essence of general education.”
While many colleges adopted the use of national, standardized instruments, Mountain Empire Community College decided to develop its own instrument. The first instrument involved questions submitted by its arts and sciences faculty members; however, the questions primarily focused on the disciplines that made up the general education component of the associate degree programs. The questions were administered within classes by the professors.
Schoch and Tucker presented to the MECC Director of Planning the proposal of developing one instrument that focused on the eight elements of the VCCS definition. Questions from the original instrument and additional questions submitted by the faculty were evaluated, edited, and assembled into a testing instrument containing 100 questions.
This instrument was given to six large social science classes, and their responses were analyzed. The data provided for item analysis, reliability, and for construct/content validity. Results indicated discriminatory items and areas needing improvement and/or inclusion. Further input from faculty members and administrators helped to construct a shorter, more inclusive instrument. This instrument became the Schoch-Tucker Assessment of General Education (STAGE).
The obvious criterion of this project was whether or not this test had reliability and validity. While statistical evidence indicated reliability and validity for the first sample group, more data from a larger and more varied sample of students were needed.
In the spring semester of 1995, a test consisting of 88 items, with some additional biographical data, was administered to 193 graduating students. This resulted in rich statistical information about a graduating population in the following areas: students’ biographical data, measure of students’ self-esteem, and students’ knowledge in the eight elements of general education. These data provided the basis for conducting item analysis and tests of discrimination. Also, biographical data were used to establish a preliminary profile of socioeconomic and demographic information about graduating community college students. Arrangements were made with officials at Wytheville Community College (WCC) to administer STAGE to a group of incoming freshmen in the fall of 1995. STAGE was administered to 214 incoming students and provided comparative data. Item analyses and tests of discrimination were conducted with the freshman group. As with the group of graduating sophomores, biographical data were used to establish a preliminary profile of socioeconomic and demographic information about incoming community college students.
Furthermore, these two populations provided a basis for pre-post tests comparison. Item analyses were conducted, reliability and validity were assessed, tests of discrimination were performed, and group norms were established. This pre-post test array will help colleges gauge the extent to which students can be expected to grow academically, and otherwise, as they progress through the community college.
Researchers should always be concerned whether or not their instrument is really measuring what they set out to measure. This was an obvious concern of these researchers. While statistical evidence indicated that this instrument has a high degree of reliability and relatively high validity, a larger population would be needed to provide greater confidence.
The use of faculty and staff to provide input to develop questions was a positive aspect of this project. Involvement of faculties from other community colleges in the area to provide feedback and engage in question development also enhanced the validity of this test.
While the authors do not want to jeopardize the integrity of the STAGE instrument by providing examples of actual questions, a description of the instrument may be of interest to the reader. The current STAGE instrument consists of 80 objective questions comprised of groups of ten questions corresponding to each of the eight elements of the VCCS definition of general education: communication skills, learning skills, critical thinking, interpersonal and human relations, computational and computer skills, understanding of culture and society, understanding science and technology, and wellness.
STAGE also includes a student inventory which ascertains biographical and socio-behaviorial information, including a ten-item, self-esteem measurement. Currently, a listening exercise is available which has students respond to six objective questions after they listen to a short taped passage. Student information can also be obtained from each college’s Student Information System (SIS) and added to the STAGE data for analysis.
The benefits of an assessment of general education will provide colleges with a data base to determine change (growth) in the several designated areas. Soon, colleges and faculties will have data from incoming students to compare with graduating students. Where and how much change occurs, in what areas, and for whom, will begin to emerge. Then, faculty and administrators will have some empirical evidence to help make decisions concerning the curriculum. Eventually, student scores may be used as predictors of college success and become tools for advising and counseling.
Postscript
At the time of this publication, seven of Virginia’s community colleges are using the STAGE instrument. Following Tucker and Schoch’s presentation at the 1995 VAG Conference, interest in a new instrument was shown by other colleges. Eventually, Southwest Virginia Community College and Virginia Highlands joined MECC and WCC in the testing of their Spring 1996 graduates.
The four colleges agreed to obtain an external evaluation of the instrument’s reliability and validity. Dr. Donna Sundre, Research Associate at James Madison University, conducted various analyses and provided a comprehensive evaluation of the instrument and recommendations for improvement. Representatives of the participating colleges provided a panel discussion at the 1996 VAG Conference, detailing their collaborative effort in the adoption of this instrument. Based upon recommendations of Dr. Sundre, five colleges joined MECC in test revision.
Currently, the seven colleges administering the STAGE instrument in the Spring of 1997 are collaborating in a process for efficiently scoring and reporting test results. Future plans include adapting the instrument for administration via computer. Discussion is also underway for the development of a question bank and the development of a criterion-referenced test.
The advantages of using a locally-developed general education test far outweigh any disadvantages. STAGE measures the eight elements of the VCCS definition of general education, student-specific scores are being correlated with numerous variables on the SIS system, colleges have ownership of the instrument, and the instrument was produced at a minimal cost. Assurance of a reliable and valid instrument and the collaboration of the community colleges will ensure that general education is being assessed in a manner that results in strengthening the educational experience of our students.
These projects were funded in part by two VCCS professional development research grants, with additional support from the assessment programs of the participating colleges. The authors are Kendall Tucker, Associate Professor of Sociology; Dr. Eugene Schoch, Professor of Psychology (retired); and Dr. Sharon Fisher; Director of Planing and Community Relations, at Mountain Empire Community College.