Among the most significant changes in recent years at the University of Illinois at Springfield (UIS) has been the implementation of a comprehensive, mission-specific general education curriculum based on lifelong learning and engaged citizenship. The Engaged Citizenship Common Experience (ECCE), a 10-hour set of interdisciplinary courses designed to be taken by all UIS undergraduates, emphasizes social responsibility, the practice of diversity and experiential learning.
The UIS Assessment Academy Project initially focused on the Global Awareness (GA) category of the ECCE core and has been undertaken as the university’s Quality Initiative for the Open Pathway and as part of its membership in the Higher Learning Commission’s (HLC) Assessment Academy. The ultimate goal of the project is to create an effective and sustainable assessment process that has widespread support from faculty members and other campus stakeholders.
GA courses are designed to help students function in an increasingly interdependent and globalized environment and to foster awareness of non-U.S. and, especially, non-Western cultures, polities or natural environments, past or present. All ECCE course work, including the GA category, is mapped to six competencies in the UIS Baccalaureate Goals and Outcomes under Goal 5: Engaged Citizenship (University of Illinois at Springfield 2015b):
- Distinguish the possibilities and limitations of social change.
The research question governing this stage of the Assessment Academy Project focuses on whether students are learning what faculty members wanted them to learn when they created the GA category of the ECCE curriculum. To answer that question, the UIS Assessment Academy team developed a rubric that combined elements of the Association of American Colleges and University (AAC&U) VALUE Rubrics with the six Engaged Citizenship competencies identified above and category-level learning outcomes identified in the GA course approval criteria developed by the UIS General Education Council (AAC&U, n.d.; University of Illinois at Springfield 2015a). The GA rubric contains five dimensions: (1) Cultural Awareness, (2) Global Interconnectedness, (3) Global Power Inequalities, (4) Interdisciplinarity, and (5) Problem Analysis (University of Illinois at Springfield 2015c). Problem Analysis was considered a stretch category, as relatively few faculty members take a problem-solution approach to the topics of their GA courses. A three-point scale (novice-intermediate-advanced) was adopted to evaluate student work on each dimension.
Once the rubric was developed, the team engaged in course-embedded assessment. Members of the research team, Collaborative Research Faculty (CRFers), evaluated sample student work products from GA courses.
Data reported in this paper derive from student work products from the 28 unique GA courses taught in spring 2015. Instructors were invited to submit either (1) a syllabus, assignment prompt and one complete set of student work products related to that assignment or (2) all assignments and student work products from a single course.
Instructors in 21 percent of the GA courses taught in spring 2015 submitted sets of student work products from 200-, 300-, and 400-level GA courses. Ninety students were represented in the study. Two of the sets consisted of work done throughout the semester: a set of discussion board postings from an online course for the whole semester and a set of three short, five-page essays from a single course. The other four sets were single assignments from individual courses. Five separate departments from humanities, social sciences and STEM (science, technology, engineering and mathematics) disciplines are represented.
Each student work product was scored by two readers using version 10 of the rubric. When raw scores on an individual student work product differed between the two readers by more than one point, a third reader was asked to score the student work. After consulting with our HLC Assessment Academy mentors, the CRFers decided to use consensus to assess the reliability of the rubric. To achieve consensus, CRFers normed with six randomly selected student work products from four courses. Mean scores were calculated for each rubric dimension.
CRFers immediately noticed that the nature of the assignments and assignment prompts played a strong role in the outcomes data. Most of the assignment prompts did not guide students to consider the dimensions identified in the rubric, and some prompts emphasized one over the others. Ultimately, CRFers decided to remove two sets of student work products as outliers. Both were from a single course in which assignment prompts were unrelated to rubric dimensions and students scored very low on all five dimensions, except interdisciplinarity, which had a slightly higher mean.
When these two sets were removed, students in the sample demonstrated competency at the intermediate level in the rubric categories of Cultural Awareness (mean of 1.94 out of 3) and Global Interconnectedness (mean of 1.98 out of 3), less competency in Power Inequalities (mean of 1.71 out of 3) and Interdisciplinarity (mean of 1.61 out of 3), and the lowest competency in Problem Analysis, which was to be expected of the stretch category (mean of 1.41 out of 3).
As CRFers evaluated student work products from GA courses and analyzed the findings of that process, they uncovered a number of implications for effective teaching and learning in an Engaged Citizenship core. These implications are organized under two main categories: (1) benefits and challenges of an institutionally developed rubric, (2) course approval process and assignment building and (3) faculty development.
Benefits and Challenges of an Institutionally Developed Rubric
One of the benefits of using an institutionally developed rubric is that the rubric reflects the values of the institution and its goal to prepare students to function effectively in a diverse, globalized society. The scale used in the rubric (novice-intermediate-advanced) allows the research team to consistently apply the same set of criteria across 200-, 300-, and 400-level GA courses. It also allows the team to look across levels of undergraduates (sophomores, juniors and seniors), across course content areas and across sets of student work products.
One of the key challenges the team faced was integrating the language of the AAC&U VALUE Rubrics for Global Learning, Intercultural Awareness, and Integrative Learning with ECCE baccalaureate learning outcomes and the campus course approval criteria for GA. Also, achieving consensus on the language in the rubric was difficult. Terms like phenomena, systems, action, interconnected, interdependent, inequalities and diversity had different connotative values for different readers. The dimension descriptions were substantially longer and more complex in earlier drafts, making it difficult for evaluators to determine whether a student’s work was successful in meeting the components of the dimension. As a result, the wording of the five dimensions in the rubric changed over time.
Implications for Course Approval Process and Assignment Building
The course approval process developed by the General Education Council did not emphasize equally all aspects of the five concepts included in the rubric dimensions. Even today, faculty members are not asked to fulfill all GA learning outcomes. That choice means that some faculty members may emphasize cultural awareness but not power inequalities; others may emphasize power inequalities but not interconnectedness. One of the implications of this project is that the course approval process and the expectations for GA course syllabi and assignments need to more consistently address the dimensions of the rubric. Although CRFers and the General Education Council do not expect that every course will address every dimension equally, they are recommending changes to the course approval criteria to simplify learning outcomes and emphasize the five dimensions. Existing courses would need to be revised to be consistent with the altered expectations. Through faculty development workshops, faculty members could familiarize themselves with the revised criteria for course approval. Then, all courses would need to be recertified by the General Education Council on a rotating schedule.
This project has led CRFers to realize that when student learning outcomes are not aligned with or included in the prompts, they are less likely to be found in student work products submitted for evaluation. Although faculty members may have integrated GA learning outcomes into the course content, they may not have included them specifically in the assignments. In other words, discussion and class activities related to learning outcomes may have taken place, but no measurable product (assignment) was utilized to demonstrate the learning that occurred as a result. If prompts clearly identify the learning outcomes students are expected to achieve, faculty members will find it easier to evaluate how well students are learning what faculty members intended them to learn.
Another of the key findings of this Assessment Academy Project is that a single set of student work products from a GA course may not fully reflect the learning that took place in each of the five dimensions. When faculty members submitted multiple sets of student work products for the same course, as two did for the summer 2015 evaluation workshop, rubric scores were higher. CRFers believe scoring multiple assignments from the same course will yield a richer, more comprehensive understanding of student learning in GA.
Findings and Implications for Faculty Development
Throughout this project, one of the goals of faculty members and administrators has been to engage GA and other ECCE faculty members, as well as faculty members across the institution, in the assessment process. CRFers have held two GA faculty retreats where the project was introduced, rubric elements was discussed, faculty members were allowed to score student work products using rubric, and feedback was solicited for rubric development and further faculty development. CRFers used the feedback from those discussions to refine the rubric, and one more retreat is planned for spring 2016. During the upcoming retreat, CRFers plan to focus on ways faculty members can emphasize the five rubric dimensions in their assignment prompts, as well as in the course syllabi.
Association of American Colleges and Universities. n.d. VALUE Rubrics. http://www.aacu.org/value-rubrics.
University of Illinois at Springfield. 2015a. Engaged Citizenship Common Experience. General Education. http://www.uis.edu/generaleducation/curriculum/engagedcitizenshipcommonexperience/.
_____. 2015b. Goals and learning outcomes for baccalaureate education (as approved by the UIS Campus Senate). General Education. http://www.uis.edu/generaleducation/about/goals/.
_____. 2015c. UIS Quality Initiative Project. Institutional Accreditation. http://www.uis.edu/accreditation/quality/uis-quality-initiative-project/.