Office of Curriculum, Learning Design, and Academic Assessment

Purpose of the Proficiency Profile 

The Proficiency Profile, developed by the Educational Testing Service, measures college-level general education skills. The assessment provides data that can be used to compare Northern Arizona University (NAU) students’ performance to the performance of students at other institutions as well as to gauge how well NAU students attained various levels of proficiency in general education skills. The assessment helps NAU to improve student learning by providing faculty with information about freshman and senior general education skills to make modifications to curriculum and learning design, recommend improvements to assessments, and ask future assessment questions. The Proficiency Profile fulfills the Voluntary System of Accountability requirement to measure general education outcomes, providing important accountability information to the public.

Key Findings:

Overall, NAU students improved from freshman to senior year as demonstrated by their performance on the multiple-choice Proficiency Profile (mean total score increased from 441 to 456 on a 400-500 score range). 

NAU students demonstrated the most growth from freshman to senior year in critical thinking (mean subscore increased from 111 to 115 points on a 100-130 score range) and reading (mean subscore increased from 117 to 121 points on a 100-130 score range).

Compared to students at other institutions, NAU students exhibited the most growth from freshman to senior year in natural sciences (NAU freshmen scored better than 50% of freshmen in the comparison group, whereas NAU seniors scored better than 67% of seniors in the comparison group).

NAU students made the greatest gain in percent of students classified as proficient at the middle and highest levels in reading and critical thinking (from 35% for freshmen to 75% for seniors).

NAU students demonstrated the least growth from freshman to senior year in writing on the multiple-choice Proficiency Profile (mean subscore increased 113 to 115 points on a 100-130 score range), although results of the essay assessment were mixed (29% of freshmen scored in the upper third of the 1-6 score range compared to 42% of seniors, whereas 8% of freshmen scored in the lower third of the 1-6 score range compared to 16% of seniors).

Compared to other institutions, NAU students exhibited the least growth from freshman to senior year in social sciences (NAU freshmen scored better than 78% of freshmen in the comparison group, whereas NAU seniors scored better than 72% of seniors in the comparison group).

NAU students made the smallest gain in percent of students classified as proficient at the middle and highest levels in writing (from 21% for freshmen to 42% for seniors). 

This report provides background information regarding Northern Arizona University (NAU), the Voluntary System of Accountability (VSA), and the Proficiency Profile general education assessment as well as describes fall 2011 and spring 2012 student participation in the Proficiency Profile at NAU. Additionally, the report presents results of the Proficiency Profile assessment and suggests questions for discussion and future study. Details regarding Proficiency Profile methods can be found in the Appendix of this report. Additional information regarding the fall 2011 and spring 2012 administration of the Proficiency Profile at NAU can be found in the full Proficiency Profile at NAU 2011-2012 Report.

NAU, the VSA, and the Proficiency Profile

NAU joined the VSA in 2008. The VSA is an initiative by public four-year colleges and universities to provide prospective students, parents, and other interested stakeholders with clear, accessible, comparable information regarding students’ collegiate experience. As a member of the VSA, NAU is required to report on student learning outcomes using one of three standardized instruments—either the Collegiate Assessment of Academic Proficiency, the Collegiate Learning Assessment , or the Proficiency Profile—on the College Portraits website (http://www.collegeportraits.org/) and  to update this information every three years. Most recently, NAU reported on the results of the Proficiency Profile, an assessment of college-level general education skills.  

The Proficiency Profile was selected by NAU for VSA reporting because of sampling design and administrative flexibility as well as affordability. Most importantly, the Proficiency Profile provides score information regarding the general education skills of critical thinking, reading, writing, and mathematics and context areas of the social sciences and natural sciences that are aligned to NAU’s Liberal Studies Program student learning outcomes. As a result, faculty can use Proficiency Profile results to make modifications to curriculum and learning design, recommend improvements to assessments, and ask future assessment questions.     

The VSA requires reporting of the results of the multiple-choice Proficiency Profile only. In addition to administering the multiple-choice Proficiency Profile, NAU chose also administer an optional essay writing assessment for greater insight into students’ writing skills. The essay assessment asks students to respond to a college-level writing prompt in a 20-minute essay. 

Proficiency Profile Student Participation 

Fall 2011: Freshmen Participation

In fall 2011, all students enrolled in ENG 105 were required to participate in the Proficiency Profile. Students participated in both the multiple-choice and the essay component of the Proficiency Profile.

1,480 students received results for the multiple-choice Proficiency Profile. (27 students were excluded from the results because they answered fewer than75% of the questions.) Of these students, 1,174 were entering freshmen, representing 30% of 3,872 entering freshmen enrolled in fall 2011. 1,548 students received results for the essay Proficiency Profile. (29 students were excluded from the results because their essays were blank or too brief to evaluate, not relevant to the topic, or not written in English.) A demographic analysis of students by class level for the essay was not available from ETS.

Spring 2012: Senior Participation

In spring 2012, specific senior capstone courses were selected and capstone instructors invited to participate in the Proficiency Profile in an effort to obtain a representative sample of capstone students in each college.  Participation was voluntary for the capstone instructors but mandatory for the capstone students whose instructors agreed to participate. Capstone students participated in either the multiple-choice or the essay component of the Proficiency Profile.

Students in 15 senior capstone courses in five colleges across the university—Arts and Letters, Engineering, Forestry, and Natural Sciences, Health and Human Services,  Social and Behavioral Sciences, and the W. A. Franke College of Business—participated in the multiple-choice Proficiency Profile.

Students in 8 senior capstone courses in four colleges across the mountain campus—Arts and Letters, Engineering, Forestry, and Natural Sciences, Health and Human Services, and Social and Behavioral Science—as well as NAU Yuma participated in the essay Proficiency Profile.

319 students received results for the multiple-choice Proficiency Profile. (Nine students were excluded from the results because they answered fewer than 75% of the questions.) Of these students, 264 were seniors, representing 4% of 6,153 seniors enrolled in spring 2012. 133 students received results for the essay Proficiency Profile. (Two students were excluded from the results because their essays were blank or too brief to evaluate, not relevant to the topic, or not written in English.) A demographic analysis of students by class level for the essay was not available from ETS. 

The table below presents the mean scores and standard deviations for freshmen and seniors by Liberal Studies Essential Skills outcomes and corresponding Proficiency Profile score category.The mean score is “simply the average score computed by adding up the scores of all the students and dividing by the number of students” (Educational Testing Service, 2010, Educational Testing Service User’s Guide, p. 8). The standard deviation “describes the extent to which students’ scores are spread widely throughout the score scale, rather than being bunched closely together” (Educational Testing Service, 2010, Educational Testing Service User’s Guide, p. 8).

Overall, the mean scores went up from freshman to senior year in all score categories. The increase in the total score was relatively small—15 points on a 400 to 500 score range. The increases in the subscores were small as well—two to four points on a 100 to 130 score range. Students’ subscores increased the most in critical thinking and reading and the least in writing.

* Overall score range is 400-500.

** Subscore range is 100-130.

Percentile Comparisons

The following table presents the percent of a selected comparison group of Doctoral/Research Universities I and II scoring below NAU for entering freshmen and seniors by Liberal Studies Essential Skills and corresponding Proficiency Profile score category.  Overall, NAU students gained in percentile rankings for four of eight Proficiency Profile score categories. NAU students gained the most in the natural sciences (+17 percentiles).  NAU students lost the most in percentile rankings in the social sciences (-6 percentiles).

*Comparison Group Institutions: Clark Atlanta University, GA; Clemson University, SC; Florida Agricultural and Mechanical University, FL; Florida International University, FL; Indiana State University, IN; Long Island University-CW Post, NY; Mississippi State University, MS; Morgan State University, MD; Nova Southeastern University – Main Campus, FL; Oklahoma State University, OK; Spalding University, KY; University of Akron, The, OH; University of Alabama at Birmingham, AL; University of Colorado – Denver, CO; University of Delaware, DE; University of North Carolina at Greensboro, NC; University of Southern Mississippi, MS; University of Tulsa, OK.

Institutional Proficiency Comparisons

The table below presents proficiency classifications for NAU freshmen and seniors as well as for the selected comparison group of freshmen and seniors by Liberal Studies Essential Skills outcomes and Proficiency Profile skill dimensions of reading and critical thinking, writing, and mathematics. Within each of these skill areas, the specific skills tested by the Proficiency Profile are classified into three proficiency levels, labeled level 1 (lowest), level 2, and level 3 (highest).  Each proficiency level is “defined in terms of a set of specific competencies expected of students” (Educational Testing Service, 2010, Educational Testing Service User’s Guide, p. 9). At each level, students are classified as “proficient,” “marginal,” or “not proficient.” The table below presents results for those students classified as “proficient.”

Overall, NAU students gained from freshman to senior year in the percent classified as proficient at levels 2 and 3 on all skill dimensions. The greatest gain was in reading and critical thinking (from 35% for freshmen to 75% for seniors), while the smallest gain was in writing (from 21% for freshmen to 42% for seniors). NAU students made greater gains overall than the comparison group. Similar to NAU, the greatest gain for the comparison group was in reading and critical thinking (from 30% for freshmen to 53% for seniors), while the smallest gain was in writing (from 23% for freshmen to 36% for seniors).

*Comparison Group Institutions: Clark Atlanta University, GA; Clemson University, SC; Florida Agricultural and Mechanical University, FL; Florida International University, FL; Indiana State University, IN; Long Island University-CW Post, NY; Mississippi State University, MS; Morgan State University, MD; Nova Southeastern University – Main Campus, FL; Oklahoma State University, OK; Spalding University, KY; University of Akron, The, OH; University of Alabama at Birmingham, AL; University of Colorado – Denver, CO; University of Delaware, DE; University of North Carolina at Greensboro, NC; University of Southern Mississippi, MS; University of Tulsa, OK.

Results of the Proficiency Profile can be used to discuss in what Essential Skills areas NAU should celebrate student achievement as well as in what areas we can improve. Campus discussion among faculty could begin with a review of the key findings.

For example, one of the findings was that compared to other institutions, NAU students exhibited the most growth from freshman to senior year in the natural sciences. Where in the curriculum are we offering courses emphasizing scientific reasoning as an Essential Skill? What are students’ learning experiences in these courses? Why are students experiencing significant growth in this area when compared to other institutions?

Another finding was that students demonstrated the least growth from freshman to senior year in writing on the multiple-choice Proficiency Profile, although results of the essay assessment were mixed. Why is student performance on these two writing assessments different? Where in the curriculum are we offering courses emphasizing writing as an Essential Skill? What are students’ learning experiences in these courses?

As faculty discuss Proficiency Profile results, they should keep in mind that this is only one indicator of students’ general education skills. Good assessment practice encourages using many different measures and many different kinds of measures when assessing student learning outcomes (Suskie, 2000).

Assessment Improvements

Can our assessment of general education skills be improved? Obviously, much of our assessment efforts in this area are guided by VSA requirements. However, we have many options as far as administering these assessments on our campus.

For example, we currently “embed” or link the Proficiency Profile to ENG 105 and senior capstone courses. Although this strategy worked well for achieving a large sample size in ENG 105, it was not as successful in senior capstone courses. Why weren’t we able to test more students in the senior capstone courses? Would another administration/sampling strategy work better in the future? Perhaps we should consider an NAU Assessment Day dedicated to general education testing outside of class time.

Alternatively, we might want to consider using a different assessment instrument to measure general education skills. Of note, the VSA recently expanded its list of approved instruments to include the use of the AAC&U VALUE rubrics with samples of freshman and senior artifacts. Student work such as signature assignments and assignments collected in a portfolio would be assessed by campus faculty raters trained to use the AAC&U VALUE rubrics. This assessment option allows for a more flexible administration and can be tailored to the needs of the institution (Voluntary System of Accountability Administration and Reporting Guidelines: AAC&U VALUE Rubrics-Demonstration Project, 2012).

Assessment Questions

Proficiency Profile results can be used by faculty to ask additional questions about student learning. Other questions we might ask in the future can be guided by four helpful analytical strategies (Pieper, Fulcher, Sundre, & Erwin, 2008): differences, relationships, competency, and change. The differences strategy asks the question, “Do students learn or develop more if they participate in a course or program compared to students who did not participate?” The relationships strategy asks the question, “What is the relationship between student assessment measures and course grades?” The competency strategy asks the question, “Do students meet our expectations?” Finally, the change question asks, “Do students change over time?”

For example, we might want to focus on the differences strategy.  This strategy basically asks about the impact of an educational “treatment.” We would expect that students who complete more courses related to a student learning outcome of interest will perform better on an assessment of that outcome. We might ask the following question related to the completion of NAU coursework and performance on the Proficiency Profile:

Did seniors who took more courses in the aesthetic and humanistic inquiry distribution block score better on the critical thinking subscale of the Proficiency Profile than, for example, seniors who completed fewer courses?

All of these next-step questions should generate lively discussion among faculty regarding curriculum, learning design, and assessment. Going forward, faculty should be encouraged to develop their own assessment questions, reflecting on, discussing, and clearly articulating what they want to know about student learning. These questions will help to guide future assessment of general education skills at NAU.