Study of Academic Support Yields Mixed Results

In the spring of 1994, my colleague Jane Rosecrans and I cited Morris’ study in a position paper we presented to John Tyler Community College’s Dean of Academic and Student Services. Our goal was to establish a point-of-contact academic support service for our students. In the Fall 1994 semester, we piloted the Academic Support Center (ASC) on our Midlothian Campus. Although we recorded only 64 client contacts, faculty and administrators believed the ASC held promise for improving success and retention rates of John Tyler students. Since 1994, the ASC has grown tremendously, serving students in some twenty subject areas, as well as providing general study and motivational skills and computer literacy. In Fall 1997, we recorded 850 client contacts.

At the beginning of the semester, Associate Dean Dr. Wendy Weiner asked me what I thought I would find. I honestly answered that I wasn’t sure. Of course, I hoped the study would show that students who visit the ASC perform better than students who do not, for this would justify, in part, my existence.

Although my gut feeling was that I probably would not be able to demonstrate a correlation between ASC visits and course success rates for Fall 1997, I believed then, and still do, that the benefits of academic support would become more apparent further on in students’ course work. For one thing, my study focused on incoming developmental writers, students who are far more likely to have learning disabilities, to possess few overall academic skills, to drop out during their first semester, to fail their courses, and not to return in the next semester than students who do not have to enroll in developmental courses. Can a support center ameliorate all that in one semester? Perhaps not. Yet, if academic support personnel can introduce these students to the idea of collaboration with peers and the importance of seeking consultation as they proceed in their work, then over time they should achieve greater success in their courses and be more likely to stay in school.

Methodology

I took Morris’ study as my conceptual model, but I wanted to change the methodology. Although Morris demonstrated greater success for students who used the writing center, because he merely compared grades of students who voluntarily visited the center with the grades of students who did not visit, he could not determine whether the visiting students were better motivated and possessed stronger academic skills and were thus more likely to succeed with or without the intervention of academic support.

I designed my study so that two classes of developmental writing (ENG 03) would be required to visit the ASC every week and two classes could visit as best they saw fit. All four classes were scheduled during the day and would be instructed by me: my idea was to try to discount for dissimilarity in student populations or instructional methodology. Unfortunately, my final project did not follow this methodology because of scheduling changes. Instead, I required my two ENG 03 classes to visit the ASC each week while a different instructor’s students were not required to visit the center. This change in plans, I believe, significantly affected the final results. At the end of the semester, I compared placement test scores, ASC visits, and course success rates for the treated and untreated groups. I also compared the Fall 1997 treated group with two classes taught on the same days that I had instructed in Fall 1996.

In addition, I distributed a survey of the ASC to all developmental writing classes (ENG 01 and ENG 03) on the Midlothian Campus. While this was not part of my planned methodology, in many ways it proved to be the most instructive element of the research, yielding information both for the purposes of this study and to help improve the services provided by the ASC.

Results: Success and Retention Rates

ENG 03 is graded on a pass/fail basis with "S" indicating satisfactory progress, such that a student is ready for first semester freshman composition, "R" indicating that a student needs to repeat the course, and "U" indicating that a student’s performance (attendance, turning in work, etc.) is unsatisfactory. Tables 1 and 2 list the number of students ("N") who passed or failed, the average number of ASC visits for each grade group, and the most and least ("High/Low") ASC visits for members of each grade group in the treated and untreated classes for the fall of 1997. Because these students enroll in so many pass/fail courses their first semester, a comparison of the groups’ GPA’s wasn’t possible.

From this data alone, I would have to conclude that not only did the ASC not help students in the treated group, but that students who were not required to visit the ASC were much more likely to pass ENG 03 and to register for the next semester. While students in the treated group were more likely to pass the course the more they visited the ASC, it’s possible that weak ASC attendance is a co-factor rather than a cause of failure.

As noted above, my methodology for Fall 1997 could not ensure reliable results, mostly because the untreated group had a different instructor who employed different pedagogical strategies than the instructor for the treated group, so it’s possible that the untreated group experienced greater success because they had a much better teacher or because their teacher had much lower evaluation standards.

So what can I conclude? I’m tempted to say that students who visited the ASC regularly were obviously more likely to succeed in their writing courses than students who did not visit, and I am sure I will present this evidence to administrators, faculty, and students. However, the sample size of the two groups was so small that it’s hard to determine whether or not the difference in pass rates is truly significant, and so I am hesitant to make too strong a claim on these grounds.

When I compared retention rates for the untreated group from Fall 1996 with the treated group from Fall 1997, I noticed no significant differences or trends. In the untreated group, 72.5% returned the following semester, regardless of their grade in ENG 03. In the treated group 71% returned this spring.

Results: ASC Survey

A survey was distributed to all developmental writing students at the end of the Fall 1997 semester and filled out by them anonymously. I want to highlight a few of the results, for they point to significant contributions the ASC and this study have made to students.

I was quite pleased that all 47 respondents had heard of the ASC and that many learned of it from more than one source. Of those who visited the ASC, I think it’s important that the majority indicated they did so because they were motivated to and because their instructor required them to. For support centers to be effective, then, I believe instructors should incorporate them into the required work of the class. Perhaps students should not have to go every week, as in my study, but maybe as in the sociology class instructed by Mr. Beverly at John Tyler they should be required to work on one or two specific projects with peer consultants, or as in Mr. Fuschini’s computer class students who are experiencing difficulty should be personally brought to the ASC and introduced to a consultant who can engage both the instructor and student in a conversation about the kind of support that is required.

Forty-three students responded to the question "Do you plan to use the Academic Support Center next semester? Why or why not?" The overwhelming majority (32) said they would visit the ASC in the following semester, while eight said "maybe" and three said "no." Students who were unsure whether they would use the ASC again indicated that their decision would depend on the difficulty of their courses. I should mention here that more students said they would visit or consider visiting the ASC in the following semester than actually used it in the semester in which they were surveyed. The students who plan to start using the ASC said this was because of poor performance in their current classes. Of the three students who said they would not visit the ASC again, two indicated they would not be attending the Midlothian Campus, and one said he or she could get better help from friends and family.

For me, Question #6, "What subject(s) did you receive help with?" provides the greatest testament to the value of the ASC in supporting student achievement. While my students were required to visit the ASC to work on English assignments, and while this survey was conducted in English classes, students indicated they also used the ASC for help in math, psychology, and computers. It’s clear that many students consulted with the ASC for more than one subject.

Discussion

I learned many things from this process, perhaps most importantly what Dr. Carmon Kiah, Director of Assessment, Research and Planning at John Tyler, has often said to me: "There’s no such thing as a perfect study. The goal of research is to get answers that lead you to new questions."

By undertaking this research, I learned how difficult it is to construct a reliable and valid study that measures the effect of the ASC on GPA and retention. It’s quite difficult to set up treated and untreated groups large enough and similar enough to make strong conclusions about their differences. In addition, it’s probably unwarrantable to attribute any differences to one kind of experience, in this case, academic support.

I also learned that perhaps I do not have to generate this elusive data to make my case to the John Tyler community that the ASC is a valuable contributor to our program. While I do intend to follow these two groups next year to compare their GPA’s and retention rates to see if there is any long term benefit from the ASC, right now I have what I consider to be credible evidence that the ASC benefits our students. First, student responses to the survey were overwhelmingly positive. They say they like the center and that it helps them. There’s an old saying about teaching which goes: "If students think they are learning, they probably are." On the other hand, this survey may not be without bias, that is, even though the survey was filled out anonymously, students still may have anticipated the responses we were looking for and couldn’t help but please the teacher.

Second, and more importantly, a comparison of students from the Fall 1997 untreated and treated groups who visited the ASC in the following semester yielded startling results. Not one of the untreated students has used the ASC thus far (week 13) in the Spring Semester, while 26% of the treated group have visited the ASC, most of them more than once. This supports one of my earliest suspicions about academic support: instructors must require students to become acquainted with collaboration and consultation so that students may then take advantage of these resources on their own.

I shall continue to look at ways to evaluate the effectiveness of the ASC and to increase student and instructor involvement with it, especially by following the academic progress of the students selected for this study. I’m also confident that others will construct studies which shed light on this important subject, for we have to know if what we are doing as we allocate resources to provide support to students works, and if so, why it works.

Notes

1. I wish to thank the VCCS for funding this study as part of the Professional Development Initiative, Dr. Kiah and Crystal Hill for collaborating with me as I constructed the study and analyzed the results, Patty Williams for providing the tracking data on the study population, and Dr. Drinkwater, Dr. Garren, and Dr. Weiner for their commitment to the Academic Support Center.

2. Students who withdrew from the class were not included in the study because it is impossible to determine whether they withdrew for academic or personal reasons. This is a further limitation of this study.

3. I often urged students from the untreated group from Fall of >96 to visit the ASC, so some of these students did have contact with this support service.

4. Computerized Placement Test, Sentence Skills.

5. Students who enrolled in an English class after taking tthe placement test but before enrolling in ENG 03 were not included in this average.

6. The mission of the ASC is to develop students as self-critical learners. For this reason, we will not proofread and correct texts. Instead, we encourage students to read their work out loud so they can catch many of their most egregious errors; we review proofreading and editing strategies; we provide consultation on grammatical issues, and we will engage a student in an error analysis of their text.

References

Kevin Morris. "Assessing Two Writing Centers’ Effectiveness." Paper presented at SCETC Conference, Charleston SC, 1/31/ 94.