Evaluation of Cultural Competence: Program, Students, and Community Effects
Linda G. Allison, MD, MPH
There are three components to evaluate a program in cultural competence: program outcomes, student learning, and the effect on the community.
Program outcomes are measured in terms of numbers:
- students who participated in the curriculum
- students who completed the curriculum
- students who improved from baseline in knowledge, attitude, and skills
- numbers and types of activities completed
- community partners and other collaborators
- service learning projects and number of community members served
- underrepresented minority or disadvantaged students recruited into program
- underrepresented minority faculty members recruited to program
- students who completed rotations in an underserved community
- students who went into practice in an underserved community upon program completion
Collecting and tracking this data is not complicated, but does take a little regular maintenance. Set up a basic spreadsheet and assign data collection to specific persons. For example, the clinical coordinator would be assigned to collect the information on number of students who completed a rotation in an underserved community, the academic coordinator would be assigned to collect information on students who participated in and completed the curriculum and their achievements, and the program director would be assigned to collect information on graduates. Service learning and other activities data would be assigned to the faculty member who coordinated those activities.
Set a specific beginning and ending date for data collection, and request all data be submitted to the person actually completing the report. The report should then be reviewed by the entire faculty and any staff members and any corrections or revisions made. This should be done at least annually, but quarterly reviews can provide information that can help in planning for the next academic term on a more timely basis than a yearly review can.
A program may choose to set benchmarks for students and faculty to meet, or it may choose to simply follow the numbers to determine whether there is a positive trend and compare the trend to regional and/or national numbers. Setting benchmarks requires the program to be thoughtful and purposeful in developing the goals and objectives for outcomes; these should be consistent with the program’s and the institution’s values and mission. For example, the Le Moyne College Cultural Competence Initiative set a benchmark of placing 100% of second year students in a clinical setting that was underserved, rural, or served a high percentage of non-white population; the clinical coordinator developed sites that met these criteria and placed all second year students in one of these sites.
The Association of American Medical Colleges has developed the Tool for Assessing Cultural Competence Training, which is available for free download at http://www.aamc.org/meded/tacct/start.htm. This instrument was designed to help medical schools identify gaps in their curriculum related to cultural competence, and can be used by most health professions programs for the same purpose. It can be used with both traditional and problem-based curriculums.
Student Learning Assessment: Knowledge, Attitude, and Skills
Purpose of Assessment
Assessment serves many needs, ranging from providing immediate or formative feedback to students as they are learning skills, to providing summative feedback to let students know they have or have not achieved the skills or gained the knowledge expected of them. Evaluations may have a positive, negative, or neutral connotation, depending on how it is formulated and implemented by instructors, and how it is interpreted by the learners. Students often see examinations as stressful, intimidating, or humiliating if they are struggling with the material or do now know what the teacher expects them to learn; they may see them as the “golden ring”, the trophy, or reward for doing well or at least doing what the teacher expects; they may see them as a waste of time; they may see them as a ticket to getting to the courses they are “really” interested in. Faculty members see assessments as
- Identify students who are not meeting expectations
- Identify students who are at the bottom
- Identify role models
- Identify offenders
- Reward good behavior
- Punish unacceptable behavior
- Continuous quality improvement of individuals
Overview of process of evaluation of learning outcomes
- Student learning assessment encompasses both qualitative and quantitative evaluation of learning.
- Assessment strategies should be consistent with the learning objectives, including the degree of attainment of knowledge, attitudes, and skills.
- List specific, observable/measurable behaviors representing cultural competence
- List specific, observable/measurable behaviors representing lapses in cultural competence
- State scope and components of cultural competence and lapses in cultural competence
- Define a vocabulary and develop criteria for assessing these behaviors
- Instruction and assessment related to cultural competence should be integrated and continuous
- Assessment methods should provide recurrent feedback so that students have an opportunity to improve
- Cultural competency behaviors change with time and acceptability changes across time; both depend on environment and situation
- Students should be expected to improve their behaviors if provided appropriate feedback
- We should avoid labeling students
- When should punitive action be taken?
- Formative for continuous quality improvement
- Formative or Summative
- Positive reinforcement (reward and recognition)
- Negative reinforcement (punishment)
- Use numerous approaches to assure adequacy and fairness of evaluation
Challenges of Evaluating Cultural Competence
Cultural competency refers to the set of acquired skills that allow for increased insight into and understanding of cultural differences. Evaluation of the development of these skills has been difficult, and there are no validated assessment tools to date. There has been significant work done in the area of evaluation of professionalism, which has some similarities to, and indeed overlaps, the evaluation of cultural competence.
Betancourt has outlined a conceptual approach to cultural competence education, including the awareness/sensitivity approach which focuses on provider attitudes, the multicultural/categorical approach which focuses on knowledge, and the cross-cultural approach which focuses on developing tools and skills for providers.
Further complicating evaluation is the question of what actually constitutes cultural competence. Is there an end-point that indicates someone is culturally competent? This assumes that there is a deficit in knowledge and skills, and the end-point is cultural competence itself. In fact, cultural competence can be considered on a continuum, and it is movement on this continuum that we should be measuring, indicating growth of knowledge and skills.
According to Finley, et al., there is a continuum of cultural competence with 6 definable stages:
- Cultural destructiveness
- Cultural incapacity
- Cultural blindness
- Cultural pre-competence
- Cultural competence
- Cultural proficiency
Students and faculty start at varying places on this continuum, and in theory, should move toward cultural competence as a result of the curriculum. This is a “growth model” rather than a “deficit model” in that we are aiming for all students to develop these skills, but knowing that some students start at different starting points, and some will progress along the continuum much more quickly than others. The goal is that all students make incremental progress; some may attain cultural proficiency, while others may only attain cultural blindness, but the goal is for each student to make progress, building on his or her current foundation.
- Betancourt JR. Cross-cultural medical education: conceptual approaches and frameworks for evaluation. Acad. Med. 2003 Jun;78(6):560-9
Review of Learning Objectives
Review the learning objectives in the section on Learning Objectives, and review any of your own objectives. Note the domain into which the objectives fall: Knowledge, Attitude, or Skills. Each of these domains should be evaluated by a strategy that is consistent with the objective and its teaching methodology. For example, of the objective says that the student should be able to “define and describe” a concept, the evaluation should ask the student to “define and describe” that concept, usually through an essay type exam. In many cases, a well-written multiple choice or short answer exam may be constructed that will also allow a student to do this. An oral examination or group discussion may also provide evidence of achieving such an objective. If the learning objectives states that a student should “demonstrate the ability to use an interpreter to obtain a patient’s history” then the evaluation should be constructed to that the student actually has to perform the task as stated; it may take the form of a simulation, oral examination, or role play. A multiple choice question type of examination will not provide the ability to evaluate whether the student has met the objective.
General Evaluation Methods
- Written examinations
- Good for evaluating knowledge
- Self-assessment techniques
- Good for evaluating growth or changes in knowledge and attitude
- Demonstrate knowledge, attitude, skills through
- Problem-based learning or case-based learning sessions, with peer and faculty feedback
- Standardized patients
- Oral examinations
Specific Evaluation Methods for Cultural Competence
Self-reflection and Journals/Portfolios
- Record experiences, reflect on them, discuss with faculty or mentor to reinforce positive behaviors and to identify any deficiencies
- Standard setting for qualitative assessment must be done carefully
- Students need to record information in a structured format and describe “challenging” encounters
- Students can fabricate or “game” the system
- Evaluators may need training to provide consistent and constructive feedback
- Includes both self-generated materials and materials collected from research
- Accumulates evidence of behaviors over time and through variety of means
- Student reflects on data from other sources and/or produces his/her own (“Here is an example of cultural competence and here is why I think so”)
- Consider making portfolios a requirement
- (e.g., 10 assessments per year across community experiences, peers, faculty or preceptors, and patients. Include students' reflections on or reactions to their evaluations).
- Clinical vignettes/Cases/ Standardized patients
- Can be utilized in a variety of ways:
- Provide stimuli for needed research, self-reflective portfolios, and instruction
- Used independently for formative and summative assessment
- Used for group discussion and/or team building exercises
- Questions may require written or oral responses, or utilized in discussion exploring different points of view
- Cases depict challenges to cultural competency
- Useful for instruction and assessment
- Stimulate and evaluate
- the framing of cultural competency issues
- rationales used for justifying behaviors
- reasoning processes described or used
- approaches for negotiating values conflicts in different settings at various points in training
- Students consider cultural competence lapses (in themselves or others) to understand why certain behaviors occurred and to begin to develop an understanding of their behaviors
When an institution of higher learning works in collaboration with other institutions and community agencies, each of them is changed in many ways. What may seem to be a relatively insignificant event may actually be the seed for future results. There is a ripple effect, as students from one group interact with students or faculty from another, or with community leaders, or community members. The health professions students develop an appreciation for the anger and hurt that some individuals from the community have experienced, as well as improving communication skills that will serve them well as health care providers in any setting. Health professions students develop an appreciation for the health care team when they meet and interact with each other during special events or at clinical and pre-clinical sites. This ripple effect may be difficult to quantify, but by collecting program data as described above, and following up with community partners, one can get an idea of the magnitude of the effects one simple project can have on a community with a diverse population.
The Le Moyne College Cultural Competence Initiative provided the stimulus for three institutions of higher learning (Le Moyne College, Syracuse University, and Upstate Medical University) to collaborate on student learning experiences, special speakers, faculty development, and community outreach. More than 120 PA students, 200 medical students, 100 undergraduate students, 100 Upward Bound students, 20 faculty members, 500 community members, and 30 community agencies have worked together over the past four years on this project, and those relationships and learning activities that were established as part of the project will continue. We will continue collecting data from surveys of our community partners and our graduates to determine the need for modifications in our curriculum so that our students are better prepared to serve in any community in which they practice and to follow those “ripples” to see where they go and how they affect our larger community.