Learning Assessment

How do we define the process of academic process?

The process of assessment, as it is understood in this guide, recommends that academic units

  • undertake activities to clarify the needs of their students and faculty, relevant administrators, community persons, and others involved with the outcomes of student learning,
  • enable the unit to decide what academic goals they value and thus what they expect students to learn,
  • determine and implement the best approaches to methods and measures that will evaluate the degree to which student learning outcomes meet these expectations, and finally
  • agree on ways to use this evidence to support improved student learning as well as an improved process for its assessment.

A comprehensive definition of assessment that portrays this process was proposed to the American Association for Higher Learning by Thomas Angelo, AAHE Bulletin, November 1995, p.7. The arrowed statements by ACAA following each component of the definition specify, broadly, actions to be taken.

“Assessment is an ongoing process aimed at understanding and improving student learning. It involves

  • making our expectations explicit and public;
    • All academic units should articulate clearly their learning goals as measurable objectives of their programs and ultimately of their courses, involving and sharing the development of these objectives with students, administrators, and other publics.
  • setting appropriate criteria and high standards for learning quality;
    • Standards of excellence are required when establishing learning expectations and the criteria for evaluating learning outcomes to assure the assessment process has as its primary goal the continuous improvement of the quality of student achievement.
  • systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards;
    • Faculty members in academic units should use valid and reliable ways to assess student learning, employ multiple measures on a regular basis, and ascertain the degree to which learning outcomes coincide with objectives and standards agreed on by the unit.
  • and using the resulting information to document, explain, and improve performance.”
    • These results, then, provide the evidence to document and support explanations of student performance. Using these results, the unit has an opportunity to re-examine objectives, methods and measures as feedback to help students to improve their learning.

What are the six steps to guide an assessment process?

1. Identify in broad terms what educational goals are valued

2. Articulate multiple measurable objectives for each goal

3. Select appropriate approaches to assess how well students are meeting the articulated objectives

4. Select appropriate measures that can be administered, analyzed, and interpreted for evidence of student learning outcomes

5. Communicate assessment findings to those involved in the process of assessment

6. Use feedback to make changes and inform curricular decisions and reevaluate the assessment process with the intent to continuously improve the quality of student learning.

These six steps can be applied to different levels or areas of student learning such as that of a program (e.g. graduate program or a program designed for a Learning Community), a major field of study (e.g. music), or a course (e.g. English 10001). The steps are useful for programs in academic units that are degree granting as well as for the wide range of programs and processes that influence learning, such as distance learning, library resources or liberal education courses.

Who is responsible for developing an assessment process for academic units?

  • Every academic unit will determine for itself the process most effective for the assessment of student learning in their programs.
  • Faculty are in the best position to determine educational values, to define measurable objectives to assess, to select methods and measures, and use findings to improve student learning outcomes.
  • Faculty need to assure that the educational requirements of others, such as students, employers, or other knowledgeable persons, are considered in developing this process.
  • This effort is a university wide one and should be consistent with our institutional mission and strategic directions

What is the role of Advisory Committee for Academic Assessment?

The Office of the Provost appointed the Advisory Committee for Academic Assessment (ACAA) in Spring 2000 for the purpose of

  • collaborating with the Office of Academic Assessment to support and coordinate a process to assess student academic achievement.

  • serving as one resource to academic units for the continuous improvement of the quality of education at Kent State University.

  • assisting in the development of materials such as this guide, workshops, and surveys to assist academic units in their preparation, implementation, and review of assessment plans.

This guide is a resource in support of assessment as are workshops and other events sponsored by ACAA.

Step One: Identify Goals

A goal is a statement expressing what ideals are to be achieved. Goal statements tend to be broadly philosophical, global, timeless and not readily amenable to measurement. They capture the knowledge, skills, and values that students should acquire in a program by a course.

This first step in identifying goals requires faculty and others to reflect on questions such as the following:

  • what is the mission of this unit that guides and encapsulates the essence of learning - the knowledge, the skills, the values or attitudes to be achieved by students?
  • are these goals compatible with the mission of the university and its strategic plan?

The characteristics of goal statements should be the same whether the focus is at the level of the undergraduate major or minor, a specialized program, the graduate program, courses, or an entire unit. Several examples of broadly stated, philosophical, and hopeful goal statements follow.

  • “The mission of the School of Architecture and Environmental Design is to encourage development of inquiring, responsible persons who will dedicate themselves to the improvement of the quality of life, the enhancement of the physical environment and the protection of the public welfare as related to architecture and urban design. The values to be developed are spiritual as well as physical, social as well as economic, and aesthetic as well as technical.” (Extrapolated from the Undergraduate Catalog, 2001-2002, pg. 243)

  • “The mission of the English Department is to foster literacy in the broadest sense through appreciation of the written word.” (From the Academic Program Review: Self-Study Report, the B.A. in English, Literature and Creative Writing Options, 1995)

  • “The goal of the School of Art is to provide graduates with the ability to develop works of art that express ideas and personal feelings as well as analyze and interpret works of art made by others.” (Submitted as an example for this guide by a faculty member of the School of Art, 2001)

Learning goals allow us to share with others the ideals of student learning we hope to achieve and to indicate the consistency of these goals with the mission of the university and its strategic planning. The above goal statements reflect the characteristics of such aims: they are general, they are ideals hoped for, they are not time-bound, and, unfortunately, they are not amenable, as stated, to being measured.

Herein lies the rub. Because the intent of academic assessment is to support continuous improvement of student learning, we must derive from these goals elements that can be measured. If this does not occur, we cannot evaluate how well students are learning what we expect of them. For this reason, the second task in the assessment planning requires the redefinition of goal statements as measurable objectives.

Step Two: Identify Objectives

The task at step two is to redefine broad, global goal statements by specifying them in terms that allow for evaluation of how well students are meeting these learning goals.

List the student learning objectives for this major (course). Learning objectives should specify the activities, products, or performances to be measured and evaluated and the criteria they must meet for success. Learning objectives state what students will know, understand, and be able to do when they complete this major (course).

Defining objectives requires faculty and others to reflect on the questions below:

  • how can the learning goals be stated as an activity, product, or performance that can be measured?
  • what will students know, understand, and be able to do when they complete studies within this academic unit?
  • will the specified learning objectives provide direction for educational activities in the unit and inform students about the expectations of the faculty?

An example of the transformation of learning goal statements to learning objectives can be demonstrated using the examples of learning goals presented in Step One.

Goal: To develop “responsible persons who will dedicate themselves to the . . . enhancement of the physical environment.
Learning Objective: Students will be able by their junior or senior years to critique various ethical and legal policies that impact the physical environment and defend, in both verbal and written work, their choices as to those that benefit this environment.

Goal: “To foster literacy through the appreciation of the written word.”
Learning Objective 1: Students will be able to master interpretive and analytical skills in writing about literature.
Learning Objective 2: Students will be able to critique and revise their own material.

Goal: “To design and develop works of art that express ideas and personal feelings” . . .
Learning Objective: Students will provide a body of their artwork, accompanied with narrative, that demonstrates independent artistic development and self-reflection.

Restating learning objectives with as much specificity as possible by defining the criteria by which knowledge, performance, or values will be evaluated assures objectivity and makes required standards apparent. For example, in the second objective for English above, the faculty need to make explicit the level of performance students would attain for acceptable or unacceptable work. These revisions of objectives occur with discussion of what approaches or methods will be used as well as how objectives will be measured . . . the next steps in this process of assessment

Step Three: Specify Approaches

Approaches define the procedures by which information is gathered; whereas, measures (in Step Four) are the specific instruments used to provide data. Some typical approaches (methods) used to gather information on student learning include portfolios, capstone courses, standardized achievement tests, external reviews, internship performances, focus groups, and so on. Multiple approaches (methods) and administration times are essential to ensure that students who may perform poorly with one method or at one time have other opportunities to demonstrate their learning.

More than one approach should be used to evaluate an objective. A valuable way to avoid the possible bias of using one method is to employ alternative methods at different points in time. For example, with regard to the learning objective for the School of Art on page 6, an approach might be student portfolios examined by external reviewers to judge if the artwork meets the criteria specified. An additional method is a student survey at graduation to evaluate to what degree students feel competent to perform the specified elements or criteria that reflect artistic development and self-reflection.

When selecting approaches to use, the following are some questions that need careful consideration.

  • How many faculty are willing to participate in the methods selected?
  • Will all students in a program or course be evaluated or a sample of students?
  • How much time is involved? Determine how this will affect faculty, staff, and student work.
  • How much useable information already exists and is available to the unit?
  • What are any cost constraints? Are there university resources that can be used for support?
  • Will a study at one point in time (cross-sectional method) provide the information needed or will following individual students or groups of students over several points in time (longitudinal method) be a more useful approach?
  • Does the unit prefer to use methods that rely primarily on numerical analysis (quantitative method) or on observations (qualitative method)?

Multiple approaches and times are essential to ensure that students who may perform poorly with one approach or at one time have an opportunity to demonstrate learning through multiple approaches and measures.

Step Four: Specify Measures

The task at this step is to identify and use measures appropriate for assessing the level at which students have achieved desired learning objectives. Needed now is agreement among faculty as to what evidence will assure that students are achieving the skills, knowledge, and values important to the academic unit. This moves the assessment process from a focus on intended results expressed as learning objectives to the level of achieved results.

Many measures can evaluate the objectives for learning, but it is important not to depend on a single measure to provide data about what and how well students are learning. Doing so can result in misinformation. Just as students learn in different ways, students respond differently with various evaluation tools. Using varied measures over time, including performance measures, more accurately affirms change and growth in learning. This allows greater confidence when recommending changes in the learning and assessment processes. Multiple measures to evaluate the learning objective for the School of Architecture (page 6) are offered below as examples of several ways to provide data about the level of student learning, each related to the same learning objective.

Capstone Experience Evaluation measures through explicitly defined criteria the competency level at which students have mastered the knowledge, skills, and values that define the major. To evaluate the level at which students have mastered one facet of this course, responsibility for enhancing the physical environment, seniors would write a paper. Faculty would establish a scoring guide of the essential elements used to judge this work. The elements might include knowledge of environmental policies, their historical development, their level of environmental impact, and students’ understanding and valuation of policies that benefit or threaten the environment. Each criterion/element would have a subscale defined possibly as percentages or in specific terms that are agreed to measure the competency of student work.

Internally Developed Tests are established by consensus of members of the faculty to measure, as above, the level of knowledge students have about legal and ethical policies regarding the physical environment, how they would use policies to enhance the environment, and their personal values regarding this issue. To ensure questions focus on the objectives of the program, elements should be evaluated that nearly all faculty in the program agree should be known, applied, and show commitment to the learning objective.

Surveys garnered often from alumni, employers, and students indirectly measure through self-report the competency of student learning at various times during their academic career or after completion of their studies. The School might choose a survey of graduates three years out to evaluate what responsibilities they are taking with regard to enhancing the environment, the degree to which the believe their studies fostered the direction they have taken, and the value of this objective to their job opportunities or community service.

Some questions that need to be considered when selecting measures include the following.

  • What schedule can be established to ensure an ongoing process of evaluation of objectives?
  • Will an externally developed test measure the specific goals and objectives of interest?
  • Are there faculty who can prepare internally developed tests or performance measures that are valid and reliable?
  • How can students be motivated to do their best on any measures assessing learning?

Step Five: Evaluate and Share Results

The task at this step is three-fold. First is to collect the information from measures that have been chosen to provide evidence of how close students’ actual learning comes to meeting the expected outcomes faculty and others have for a course, a major, or a program. Second is to evaluate what is found. Third is to share the findings.

Some questions that need consideration when planning this step follow.

In connection with the collection of measures

  • Are faculty or other knowledgeable personnel available to supervise the collection of information intended to measure student learning?
  • Is space available to maintain information securely so as to assure confidentiality?
  • If measures are to be repeated on a regular basis, or longitudinal studies are to be done, is staff available to assemble the information in a useable fashion? Are there software programs available that can assist in the storage of data such as is to be used?
  • Are students made aware of faculty advocacy of the assessment process and the agreed on measures to assess their learning? Faculty attitude and contribution are known to have an effect on student participation and motivation.

In connection with the interpretation of measures,

  • Are faculty available to score, analyze, and interpret the findings of the measures to be used? If not, are there university or college resources available to assist in these tasks?
  • If externally developed (standardized) tests are used, is there assurance that the interpretations from these tests will provide the information required?

In connection with the communication of results,

  • Who will have access to the findings from any studies of student learning?
  • Who are the audiences to whom the findings will be reported? How will communications differ for different audiences, e.g. for annual reports to the faculty, to the college, to ACAA, to students or for marketing purposes?
  • Who will have responsibility for writing reports?
  • How will the unit assure a timely and accurate flow of information to those who will use this information? How will findings be used and by whom?

Sharing results leads to the reevaluation of the assessment process to improve student learning.

Step Six: Make Changes

Academic assessment is an ongoing process that requires continuous reevaluation as to whether teaching and learning processes achieve the goals and objectives defined by faculty in the academic unit. When students succeed in achieving those goals and objectives, one might assume that the teaching and learning processes are functioning well. When students do not achieve those goals and objectives, changes should be made in teaching and learning processes. Reevaluation after changes are made will suggest if those changes were helpful to student learning. In this way, assessment creates a continuous cycle through these six steps in the assessment program and teaching/learning processes.

In making changes, faculty should consider the following two questions:

What elements of the teaching/learning processes should be added, deleted, or changed to improve student success?

Did the assessment plan for the academic unit produce results that have face validity? If not, why not?

Making changes to enhance student success requires reflection and thoughtful analysis foreshadowed in the actions suggested in the section on “How do we define the process of academic assessment?”

  • for academic units “to agree on ways to use this evidence to support improved student learning as well as an improved process for assessment”, and
  • to make use of findings to “provide the evidence to document and support explanations of student performance” and as “an opportunity to re-examine objectives, methods and measures as feedback to help students to improve their learning.”

Some believe when the words ‘improvement” or “enhancement” are used that something is wrong. That is not the case. Most faculty, for example, are accustomed to reviewing and looking to improve what occurs during class time, at the end of a course, or in committees that discuss curriculum, pedagogy, and other educational matters. The intent of step six is the same -- to plan, often with others, new ways to accomplish their goals for students.

Some questions that need consideration at this juncture follow.

  • Do the objectives and findings define as well as answer the questions that are important to understanding and enhancing student learning?
  • Are faculty and students motivated to participate in the assessment process? If not, why not?
  • Has thought been given to the use of benchmarks based on comparable student groups?
  • Are there resources available to assist in areas of assessment that are found problematic?
  • Is there adequate support from the university to allow for continuous implementation and evaluation of the assessment plan?

Glossary

In order to develop this guide, it was necessary to define and explain terms as they seemed appropriate for the purposes of this guide. ACAA recognizes that some terms are understood in ways different from that used here, concepts such as assessment, methods, measures, goals, objectives, and values to cite a few. In an effort to be helpful, not arbitrary, the following explanations of terms as used in this guide are offered.

Approaches are the procedures used to gather the information needed to assess how well students have met the learning objectives. They are the course of action through which evidence about courses, programs, majors and the like will be gathered. To provide quality information, multiple approaches should be used.

Assessment refers to a continuous process instituted to understand and improve student learning. While academic units may find alternative pathways to arrive at this goal, this process needs to begin with articulation of educational goals for all programs and courses. These goals should be expressed as measurable objectives followed by the selection of reliable and valid methods and measures. After collecting, interpreting, and sharing findings, the aim is to use these learning outcomes to better understand how and what students learn, how well students are meeting expected objectives, as well as to develop strategies to improve the teaching and learning processes.

Benchmark is the actual measurement of group performance against an established standard or performance, often external.

Criterion is the standard of performance established as the passing score for a performance or other measures such as a test. The performance is compared to an expected level of mastery in an area rather than to other students’ scores.

Cross-Sectional Studies provide information about a group of students at one point in time.

Evaluate and Evaluation are terms used in this guide to indicate the interpreting of findings and are used as synonymous to the term assess and assessment. ACAA is aware that many make a distinction between evaluation and assessment with the difference that assessment is a process predicated on knowledge of intended goals or objectives while, in contrast, evaluation is a process concerned with outcomes without prior concern or knowledge about goals. That distinction is not used in the guide.

Goals are statements about the general academic aims or ideals to which an educational unit aspires. Goal statements allow us to share with others our hopes in regard to the learning achievements of our students. Further, goals at the unit level should align with the mission of the university. Goal statements are not amenable, as stated, to measurement.

Longitudinal studies provide information from the same group of students at several different points in time.

Measures are the specific instruments or performances used to provide data about learning. They are the tools that are to provide information as to the level of achieved results or outcomes. To avoid systematic bias in findings, multiple measures are required.

Methods - see approaches.

Objectives are the redefinition of learning goals in a way that permits their measurement. Objectives express the intended results or outcomes of student learning and clearly specify the criteria by which student knowledge, performance, or values will be evaluated.

Process is a method generally involving steps or operations that are ordered and/or interdependent.

Qualitative and Quantitative Research describe two research methods. Both are valuable as a means to assess student learning outcomes. In a practical and somewhat philosophical sense the difference is that quantitative research tries to make use of objective measures to test hypotheses and to allow for controlling and predicting learning. Qualitative research makes use of more subjective observations of learning.

Reliability is the extent to which studies or findings can be replicated.

Sampling consists of obtaining information from a portion of a larger group or population. When the selection of a sample is randomly chosen there is greater likelihood that the findings from the sample will be representative of the larger group.

Validity depends on demonstrating that a measure actually measures what it is purported to measure.

Resources

There are many sources of information and assistance available to help with the tasks suggested in this guide. A full listing of these with brief comments as to their purpose will be available at the ACAA website. This page suggests a limited number of resources that may be valuable to be aware of immediately.

The Advisory Committee for Academic Assessment, the Office of Research Planning and Institutional Effectiveness, and the Faculty Professional Development Center are all resources on campus that can assist in certain areas of academic assessment planning.

The Office of Academic Assessment has a small library focused on academic assessment, with books, articles, and journals as well as various sourcebooks regarding methods and measures.

From a multitude of internet resources, a few that may be valuable sources to begin with follow.

North Carolina State University - University Planning and Analysis
California Academic Press - Student Outcomes Assessment: Opportunities and Strategies
American Association for Higher Education
The Higher Learning Commission

Local Links

Utility Links

Text Only Options

Top of page


Text Only Options

Open the original version of this page.

Usablenet Assistive is a UsableNet product. Usablenet Assistive Main Page.