from VCCA Journal, Volume 4, Number 1, Spring/Summer 1989, 25-32
© Copyright 1989 VCCA Journal
During the first few months of 1987, the Blue Ridge Community College faculty and administration were faced with a formidable challenge. We learned that we would be required to show that our instructional programs were effective. But we had little idea of what that meant.
Our search for assistance quickly produced the information we needed. Frank Luth of James Madison University was kind enough to visit us, introduce us to the core concepts, and provide documentation of a variety of strategies and tactics. Elmo Roesler, the VCCS Assistant Vice Chancellor for Research and Planning, also met with us and has been only a phone call away when we needed help.
Our discussions with Elmo and Frank opened to us a growing network of people whose ideas we could apply freely, and whose patience has been boundless. In retrospect, we could not have developed a workable system without them. Unfortunately, they could provide only information. The nature of the task is such that each college must make its own decisions and then live with them. The remainder of this paper describes the nature of Blue Ridge, the decisions we have made, where we are in terms of living with those decisions, and how we expect this to affect college governance.
Blue Ridge is a small, rural, community college in the central Shenandoah Valley of Virginia. Our mission is to provide education, training, and community service. We employ 38 full time faculty, as well as about 100 adjuncts, and have annualized FTES of about 1100.
For many years, Blue Ridge has had components of outcomes assessment in place. Each occupational curriculum has a lay curriculum advisory committee of employers and former students; students evaluate faculty and courses; we survey graduates on an annual basis; and a variety of outside organizations review our programs. But these pieces do not a system make. Most of these components were imposed on faculty by administrators with little integration with the other components. Thus, curricular adaptation was oriented toward either internal crises or external funding.
That has all changed. Now we have in place a system which is endorsed by the administration and the faculty that allows us to measure curricular effectiveness and to improve curricula in an orderly and productive manner. A brief description of how this system evolved follows.
The development of the system has been influenced by Blue Ridge's culture, which is atypical among community colleges. The average faculty member has been at BRCC for 12 years and takes more than a modicum of pride in the college. "Entrenched" may be pejorative, but it is certainly descriptive of our faculty. Both administrators and faculty agree that faculty should be responsible for curriculum and should have an influential role in college governance. Both administrators and faculty agree that faculty should show more initiative, taking a grip on problems and solving them, rather than relying on an understaffed bureaucracy to answer difficult questions.
Neither the faculty nor the administration was comfortable with a model imposed on the faculty by the administration. Neither the faculty nor the administration wanted to place at unnecessary risk the comfortable working relationship between the two groups. And neither the faculty nor the administration wanted to be associated with something that was ineffective, disreputable, or the source of daily drama.
We all wanted something that would help the college do an even better job for its service area. These agreements on faculty role set the stage for the college's response to the challenge of assessment.
We created a set of policies from which the remainder of the system could be generated. Some of those follow.
1. Rather than merely certify, the purpose of the outcomes assessment system is to improve our curricula, evaluate faculty, compare curricula, and compare our performance to that of other schools.
2. Since the purpose of the system is curricular, and since the curricular experts in a college are faculty, the system is designed, implemented, and managed by faculty, both at the program and at the college level.
3. Because the assessment system does not bring with it new fiscal or human resources, it must compete for resources in a zero sum game. In order to reduce conflicts with present operations, the system is inexpensive to operate and to update, and requires as little effort as we believe possible.
4. The more complicated and technically sophisticated the methodology, the less likely that it will be understood, be applied, and be used. Therefore, the system is both low-tech and straight-forward.
5. A useful assessment system is unlikely to be a useful tool for basic research. Assessment is inherently applied, and does not require either high levels of precision or the services of "Tom Swift and his electrical factor analysis machine." From the perspective of the faculty and the administration, qualitative information is likely to be more valuable than quantitative information if our primary focus is on curricular improvement. For our purpose, assessment is and should be less science than art.
Upon receipt of the VCCS documents requiring that we generate an assessment plan, the Dean of Instruction and Student Services appointed an Assessment Committee. The selection of the Committee members proved crucial. Of eight members, two were the College's division chairmen, a third was the Registrar (who doubled as the institutional researcher), and the remaining five were teaching faculty members whose mean tenure at BRCC was 17 years. Of the five faculty members appointed to the Assessment Committee, one was the chairman of the faculty senate, one was a former chairman of the faculty senate, and all five were the senior faculty in their respective disciplines. As at most other colleges, primus inter pares is the key to curricular change.
The five faculty members were all sufficiently experienced that none felt inhibited by the presence of the three administrators. That the faculty members formed a majority of the Committee was important for form, not substance. At no point in the operation of the Committee has there been a faculty-administrator split in voting--in fact very few formal votes have been cast. The Committee views itself as analogous to a council of elders providing guidance based on experience and scholarship. Others outside the Committee perceive it as a group wrestling with an arcane challenge, one that we would all rather forget, but which will not go away. The Committee has made decisions and recommendations, and then marketed them to both faculty and administrators. Never during the course of the Committee's operation has either a recommendation or a decision been rejected by either the general faculty or administration.
The Assessment Committee charge was to do what seemed necessary and desirable in the way of outcomes assessment, both in planning and in implementation. The Committee wrote the College Assessment Plan, presented it for review and comment, and gained approval for the Plan from the VCCS. That was the easy part.
Selling the assessment concept and the Plan to the faculty at large was a major challenge. Outcomes assessment sharply contrasted with Blue Ridge's tradition of live and let live. It meant that accountability was on the horizon, and that, in turn, created considerable anxiety.
The workload of the typical faculty member had already increased substantially during the preceding eighteen months. We had acquired a new President who made a mandate for increased productivity, stating that financial stability (and positions) depended on increased productivity. Faculty felt threatened, overloaded, put upon, and generally down-trodden. Some of our faculty had never been expected to lead or change anything, but it seemed that everybody was being expected to change everything at once.
It was into this less than ideal situation that student outcomes assessment was cast. The Committee's first attempt at involving the general faculty was a failure--as previously mentioned, the review and comment was fruitless. The Committee decided that a different strategy was called for. One (and later two) of its faculty members was granted released time to work with other faculty on an individual peer basis. The idea of a faculty member being paid by a college to teach other faculty has its interesting points, but one on one communication was far more effective than posting notices, distributing copies, and making announcements.
Since the Assessment Plan required additional work of each faculty member, it became the task of the released-time faculty members to clarify what was required and to provide encouragement. More importantly, there was a need to overcome worries about the use of outcomes data to evaluate faculty, about the perceived possibility of a vendetta against faculty by administrators, and about the need for evaluation skills not possessed by the faculty member. BRCC's strategy made it relatively easy for most faculty to perceive both the inevitability and the desirability of assessment, but there still remained some concerns about workload excesses and skills deficits, as well as some residual institutional paranoia.
The released-time faculty members' efforts were facilitated by presentations at two consecutive general faculty meetings. At one, the members of the Committee presented the Plan, described its virtues, and endured the slings and arrows of an outraged faculty. At a second meeting, limited to teaching faculty, the Committee members provided more information and faced somewhat less opposition. It seemed a case of gradually wearing down the opposition while lobbying in corridors and offices for support. But it worked.
Although Blue Ridge's assessment system includes evaluations of placement activities, general education (both in transfer and in vocational curricula), and other components, we decided to focus most of our early efforts on assessment in the occupational majors because it was in the majors that we saw the most obvious need and the greatest potential for a quick payoff for faculty who had reservations about the utility of assessment. Here was our opportunity to make a difference.
We designed a system for assessment in the majors which is an unusual merging of administrative and instructional roles, but seems to be working. We enter the system with the individual faculty member who is assigned responsibility by his division chairman for an instructional program, for example, business management.
The faculty member is told that the Committee has written the BRCC Assessment Plan, that the Plan has been approved by the VCCS, and that certain requirements must be met. Those requirements are as follows.
1. The faculty members within each program must design a system for evaluating their own programs, with the following constraints. (a) The faculty members must clearly state what the goals of each program are. (b) There must be at least one measure upon matriculation, one during the student's tenure at Blue Ridge, one at the point of graduation, and one at some point during the subsequent five year period. There is no requirement that any particular measure or type of measure be used, but each program goal must be measured at some point in time. (c) No measure used for assessment purposes may be used for grading purposes, and vice versa. Grades are of little use in improving curricula, and allowing use of grades in our assessment system would create the potential for diluting the otherwise independent assessment system. If grades were admitted as measures, inevitably there would be rumors of self-serving grading practices. Faculty and administrators also might be tempted by the law of least effort and thus use grades more and more as a focal measure, creating in student assessment a new application of Gresham's Law. (d) Every measure must be documented. Existing data bases should be used if they meet the need. The low-tech, concrete, qualitative, inexpensive, and simple are preferred.
2. The faculty members associated with each program were required to submit their plans with supporting documents (e.g., survey forms, essay questions) to the Assessment Committee according to a set calendar and to gain approval of their respective plans.
3. Those faculty members must then (on a four year cycle) implement the plan and then provide their data and a written report to the faculty Curriculum Committee at the end of the following academic year. The report must evaluate the data and describe what changes are indicated, what changes the faculty members will make, and what changes the faculty members are requesting that the Curriculum Committee make.
4. The same faculty members must gain approval from the Curriculum Committee for their end of year report.
The only point at which the administration enters into the assessment process is through enforcement by the division chairmen of a requirement that program faculty gain timely approval of their plan from the Assessment Committee, and of their end of year report from the Curriculum Committee. Thus the administrative role is restricted to timeliness, not substance.
In fact, the system as a whole provides only a little form and allows nearly untrammeled diversity. The faculty members associated with each program must name their own poison. And that is not easy for some to do. They must select their own measures and report on what they mean. And they must gain peer approval.
But the key feature is that faculty must either adapt program content according to what they find out or explain to their peers why changes should not be made.
Three rationales were used to communicate the reason that we were bothering with assessment. First, the process would lead us to improve the quality of the education we provide. Second, it would allow faculty members to seize control of their own fate. And third, it could be a useful student recruiting device. The Assessment Committee made quite clear that while some assessment effort was required by outside agencies, we were taking it seriously because of its utility at Blue Ridge.
After a few weeks of broad-based discussions, the Assessment Committee assigned each of its members to serve as liaison with several programs. They visited faculty from each of the programs, offering advice and support, and also offering to help shepherd the draft program plan through the Assessment Committee. They provided concrete examples of how various problems could be solved with a minimum of work. Offers of assistance, predictably, were seldom refused.
Current Status
All of Blue Ridge's occupational programs now have documented assessment plans approved by the Assessment Committee, and most have developed their survey materials. Resistance to the concept of assessment has faded.
Many program assessment plans were rejected upon their first submission to the Assessment Committee. Almost all of the rejections were due to communication failures of an evolving system, rather than to faculty foot-dragging. The Assessment Committee required that some plans be revised as many as three times prior to approving them. There is now no question that faculty are holding faculty accountable, while providing them with the support they need to attack a formidable task of assessment.
Faculty members have chosen a wide variety of variables to measure and a wide variety of data sources. Most have elected to use ad hoc surveys of some or all of the following: students, graduates, local advisory committees, employers, and field internship supervisors. The most commonly selected standardized tests are the Assessment and Placement Service tests in reading, writing and mathematics, which are required of all matriculating students. Several program faculty--e.g., veterinary technology, nursing, automotive, and secretarial science--have elected to use certification or licensing examinations administered by external agencies as circum-graduation measures.
Except for the tests and examinations, the data collection devices are generally straight-forward. Most faculty have decided to abandon subtlety and simply ask people the questions that need to be answered. The modal questions are on the order of "What are the strong (or weak) points of the BRCC graduates you've hired?" and "How should the College improve this program?"
There is no shortage of mensuration choices. SACS has been quite obliging in listing a multitude of data sources. Our faculty members have chosen to use some on the SACS list (and some not on the list), but have rejected some options entirely. For example, no faculty members have elected to use a rating scale. From their perspective (and ours) rating scales, like grades, are difficult to translate into curricular change. Few faculty have elected to use standardized tests. No program faculty have restricted their assessment plans to the minimum required of them. Some program plans go far beyond what the Assessment Committee requires. In each program, the assessment plan reflects the unique attributes and resources of the program; while there is some overlap in methodology, no two program assessment plans are alike.
Getting faculty to commit to the assessment process has been a fascinating experience. As previously indicated, we used both traditional formal and peer methods. The peer methods were fundamentally a combination of teaching and reassurance with some guidance.
The mere advent of the assessment process has caused most faculty to rethink their programs and to take a serious look at what their program goals really should be. For the first time in a long while, many are taking a look at the population they are trying to serve and how they can best do so.
Implications for Governance
The advent of assessment has led the faculty to reconsider its role in governance. Until assessment was made a part of our culture, the faculty role in governance was mostly passive unless some administrative activity rankled them; then the faculty role became adversative. But faculty did not have a continuing role in the management of the college. We have decided to use assessment as a lever to change that.
The Assessment Committee has dropped its function of reviewing occupational-technical program assessment plans and has passed that function to the Curriculum Committee. The Assessment Committee is restricting its role to development of new functions, both within assessment and otherwise. For example, the Assessment Committee is now working on tracking in developmental studies and general education both in occupational-technical and in transfer programs. When these functions are operational, they too will be passed to other committees.
The Assessment Committee is also moving toward the development of a college-wide strategic planning process. Within the next year or so the Committee will be reconstituted as the steering committee for the SACS process. Faculty are taking their proper roles as leaders in the College, rather than as production line workers. And administrators are enjoying increased cooperation, a reduction in unproductive hassles, and a more productive College.
Prospects
Conflict is on the horizon. Some of the conflict will be between education and training. Some will be conflict over competition for limited resources. Some will be between program faculty and the Curriculum Committee as they try to sort out what the data really mean. But these inevitable conflicts are only healthy signs of a college in the process of change.
We expect that assessment will tell us how well we are doing, and we expect that not all the news will be good. We also expect that the assessment process will give us guidance on how we can improve. Faculty will be more likely to see that resource distribution is related to program quality and needs. And the general public will be more aware of what Blue Ridge can do.
If we had it to do over again, we would know more before we got started, but the model we have used would be essentially the same, even though its limitations have been recognized and acknowledged. It may not generalize well to other colleges. It does not provide uniform data which would allow comparisons between programs and colleges. It is decidedly low-tech and home-brewed. But it does the job, creates a minimum amount of conflict among faculty, demands little in the way of added resources, and reasserts faculty control over curriculum. It also promises significant faculty influence in the general governance of the college. It is an exciting time to be at BRCC. That suits us fine.
Inventory of Information Sources for Planning and Evaluation:
Bernard H. Levin is Professor of Psychology at Blue Ridge Community College in Virginia.
Metro Lazorack, a former professor of mathematics, is Dean of Instruction and Student Services at Blue Ridge Community College.
James C. Sears is President of Blue Ridge Community College.