VISTAS articles are made available for historical reference only and are presented "as is." ACA does not guarantee or represent that the information is current, accurate or indicative of the original or intended quality. These materials are not maintained or updated and may contain outdated or incomplete information. Readers should exercise discretion and verify information independently before relying on it. We assume no responsibility for the use or interpretation of this content.

Article 28

Critically Analyzing and Improving Graduate Student Learning in a Counseling Theories Course

Isaac Burt and Andy Pham

Download PDF

All correspondence regarding this article should be sent to: Isaac Burt, Department of Leadership and Professional Studies, 11200 SW 8th Street, Miami, FL. 33199, Isaac.burt@fiu.edu.

In counselor education programs, assessing student competencies provides faculty with decisive and comprehensive information regarding what graduate students truly know (Burt, Gonzalez, Swank, Asher, & Cunningham, 2011). Program assessment offers counselor educators and graduate students guidelines for intended knowledge, as well as specifying where informational gaps exist (Jenkins, 2011a). However, some educational programs, such as counselor education, lack guidelines indicating where gaps exist in student learning (DiBiasio & Mello, 2004). These gaps can lead to additional problems, such as: (a) absence of program expectations from faculty and students for graduate level coursework, (b) faculty not covering core areas outlined by accreditation standards (i.e., Council for Accreditation of Counseling and Related Educational Programs), and (c) graduate students lacking requisite skills expected of program graduates (Jenkins, 2011b). Despite concerns related to graduate learning, many programs only have basic methods to evaluate students and faculty instruction (Jenkins, 2011a). According to Bailey, Jeong, and Cho (2010), programs cannot continue to rely on outdated and rudimentary methods to evaluate academic functioning. In today’s world of high stakes testing and accountability, programs need to replace outdated methods with outcome-based procedures (Karp, 2011).

Counselor education is a diverse profession in which practitioners serve a multitude of individuals, from the ethnically diverse to persons with disabilities (Lambie & Milsom, 2010). In order to meet the needs of various populations, the Council for Accreditation of Counseling and Related Educational Programs (CACREP) has established core competencies endorsed graduate programs must follow (CACREP, 2009). To qualify for accreditation, programs must clearly state in course syllabi how instructional methods relate to core competencies. Unfortunately, numerous accredited programs struggle with the problem of going from intended outcomes to actual instruction (Jenkins, 2011b). Many curricula appear packed with educational requirements (Karp, 2011), but two essential questions exist: Do counselor education students attain core competencies, and do programs provide effective instruction of these core requirements?

Reasons for Assessing Graduate Student Learning

One way to determine whether students are attaining competencies is through systematic assessment. DiBiasio and Mello (2011) recommended utilizing program assessment in order to gauge educational environment, and determine if a program is attaining academic objectives. In their work, they suggested measuring competencies of graduate students is imperative for two primary goals: (a) program improvement and (b) satisfying accrediting bodies. DiBiasio and Mello are not alone in suggesting that assessment affects program development.

More recently, Newgent, Behrend, Lounsbery, Higgins, and Lo (2010) emphasized the importance of evaluating program improvement in a counselor-education school-based treatment. They indicated more data-driven research needs implementation for a number of reasons. First, partially due to media attention, an increase in academic accountability has occurred. As a result, faculty increasingly remains responsible for student learning. Program assessment can corroborate that faculty are teaching essential topics necessary for effective graduate student learning. Second, critics of counselor education traditionally perceive it as a “soft” profession, and one that is difficult to assess. Transparent, outcome-driven data could possibly alleviate this problem while increasing credibility and academic integrity (Jenkins, 2011b). Third, effective program results expand the body of literature pertaining to counselor education. Research makes the profession more robust and beneficial to students, faculty, and the public. Additionally, in limited economic times, programs need to show why they should exist. With countless educational programs cut due to budget constraints, indicating program effectiveness remains paramount in contemporary education (Bailey et al., 2010).

CACREP and Counselor Education

According to CACREP (2009), students must demonstrate knowledge in eight core areas. These areas are: (1) professional orientation, (2) social and cultural diversity, (3) human growth and development, (4) career, (5) the counseling relationship, (6) group work, (7) assessment, and (8) research. Gladding (2012) states counselor education programs must ensure graduate students receive the experiences needed to integrate these core areas in their professional work. Additionally, by incorporating CACREP standards into teaching methods, Gladding (2012), affirmed that programs assist graduate students in identifying biases and utilizing strengths. Research supports Gladding, as one recent study notes poor instruction not aligned with graduate needs as one of the primary concerns facing students (Leon, Villares, Brigman, Webb, & Peluso, 2011). Burt and Butler (2011) substantiated this notion and asserted that some programs do not integrate CACREP competencies. When programs do not incorporate core competencies into instruction, students may lack the ability to understand complex relationships, multiple perspectives, and cultural diversity (Jenkins 2011b). Ultimately, the question remains: What outcome measures do programs use to assess student learning and ensure faculty teach essential competencies?

Offenstein and Shulock (2010) stated that programs normally request students to complete standard evaluation forms assessing faculty instruction. Likewise, faculty may have an assortment of test bank questions, papers, or experiential activities to assess student knowledge. Karp (2011), however, suggested that traditional methods of assessing both graduate students and faculty need updating. They recommended going beyond superficial modes and critically analyze if students are attaining core competencies. Along with critically analyzing students, Karp recommended faculty instruction needs assessing as well. In order to go beyond traditional methods, innovative assessment practices must take place. Jenkins (2011a) suggested utilizing program assessment as this advanced medium to ensure requirements of competencies.

Jenkins (2011b) indicated program assessment is crucial in identifying not only accountability in graduate settings, but also for evaluation purposes and program integrity. Implementation of program assessment may serve as a novel method to enhance the efficacy of graduate programs, as well as being a tool to gauge student learning (Jenkins, 2011b). Clearly, some form of assessment in graduate programs is needed to ensure students’ acquisition of competencies and that faculty stays abreast of current trends. This study hoped to accomplish these goals by initiating a three-year program assessment project. The goals entailed two central concerns: (a) if graduate students attained core knowledge and skills, and competencies, and (b) if faculty instruction corresponded with CACREP objectives. For the purposes of this study, a counseling theories course was chosen to be the course critiqued and analyzed. Several reasons exist for choosing a theories class. First, counseling theories offer beginning students a firm guide to conduct counseling (Day, 2008). As such, a large number of counseling programs have students take the theories course early in their schedule (Schmidt, Homeyer, & Walker, 2009). Second, theories offer counseling students a strong foundation from which to build, and begin their professional development (Gladding, 2012). Third, the theories course offers an introduction into the field, combining historical significance and pragmatic techniques and skill acquisition into a cohesive class (Day, 2008).

Purpose of the Study

As stated previously, this study had two primary goals. First, the study intended to identify and clarify the knowledge, skills, and competencies from graduate students of a CACREP-accredited counselor education program. The rationale was to investigate if graduate students had opportunities for growth in order to achieve more on an academic, professional, and social level similar to what Gladding (2012) espoused. Second, the study sought to ensure that faculty members covered required material so students attain the requisite skills expected of program graduates. Through program assessment, the hope was to identify where improvements needed to take place, in both graduate student competency attainment and instructional methods.

Coinciding with objectives, this study included two primary research questions. First, do students demonstrate a basic understanding of counseling theories? Second, do students demonstrate an ability to apply counseling theories to practical (real-life) application? In order to explain the study, as well as the choice of research questions, this article includes three sections. In the first section, the authors present the methods which contain demographics, instrumentation, and procedures. The second section provides a description of the statistical analyses, including results from the two research questions. The last section concludes with limitations and implications for counselor educators who may want to assess graduate student learning.

Method

Participants

Participants were clinical mental health and school counseling students in a CACREP-accredited master’s level program. The study included 68 participants over the three-year period with 80% identified as Caucasian (n=54), 10% Latino/Hispanic (n=7), and 10% Black (n=7). Eighty percent of the participants identified as female (n=54). Participants ranged from 23 to 58 years of age. Out of the 68 participants, 23 were in the school counseling track, and 45 were in clinical mental health counseling.

Instrumentation

Counselor Preparation Comprehensive Examination (CPCE). According to Jenkins (2011a), program assessment must occur over a number of years to accurately assess student learning and program integrity. In addition, ongoing assessment allows faculty to detect patterns, develop solutions to problems, implement resolutions, and then evaluate if implementation of resolutions was successful. This study had a number of characteristics that corresponded with Jenkins’ recommendations. For example, the study took place over a three-year period. The first year focused primarily on gathering basic information about student’s knowledge, skills, and competencies. The goal was to create a foundation of information in the first year to draw upon and go further in later years. In order to create a strong foundation, the Counselor Preparation Comprehensive Examination (CPCE) was the primary instrument utilized.

Created by the Center for Credentialing and Education (CCE), the purpose of the CPCE is to assess students' knowledge of counseling (CCE, 2011). Furthermore, the CPCE is a critical assessment tool used by counselor preparation programs, as currently more than 300 universities and colleges use the instrument. According to CCE, the development and administration of the CPCE allows for an objective assessment that includes two primary objectives: (a) comparisons for longitudinal purposes (i.e., compare present students' scores with preceding years' students) and (b) evaluation of students’ scores to national scores. Counselor education programs in 45 states and abroad have approved use of the CPCE as a way to standardize appraisal of students.

Developed in 1997, CCE projects more than 8,500 students in 2012 will take the CPCE as part of program requirements. The CPCE consists of 160 multiple-choice items with 20 items per each of the eight (8) CACREP competency areas. Administration of the CPCE extends up to a 4-hour period. Out of the total number of items, 136 count towards test-taker scores (Schmidt et al., 2009). The remaining 24 serve as questions for future tests CCE evaluates to assess item difficulty. The CPCE Total Score represents the calculation of all subtests. However, according to Schmidt et al. (2009), subtests represent the eight CACREP areas germane to counseling. Thus, in order to assess strengths and weaknesses in students’ learning, subtests are crucial, as they align with CACREP core areas. Lastly, the reliability of the CPCE is .87 (Cronbach’s Alpha) with a standard error of measurement of 4.63 (Schmidt et al., 2009). Hence, the CPCE is a reliable instrument to assess graduate students in counselor education programs (CCE, 2011). Due to its high reliability, the committee decided to implement the CPCE all three years of the study.

Instructor developed midterm and final. In order to supplement the CPCE and gain more information on where gaps and strengths existed in student learning, this study also utilized instructor-developed examinations. In the second year, development of an instructor-created midterm began in the theory course. The rationale for choosing this course is that through information gathered in the first year, students had the lowest CPCE subtests scores corresponding with this class. Consequently, it was determined an instructor-created exam should begin in the second year assessing student knowledge in this course students scored the lowest. The goal was that instructors could create an examination that could produce individual student information and identify specific gaps in student learning. Unfortunately, midterms lacked consistency among instructors, as exams varied in comprehensiveness, difficulty, and format.

Due to the dearth of uniformity, the third year implemented a progressively more standardized midterm and final, in the hope to increase reliability between instructors. The rationale behind standardization was to determine if significant differences existed between the two examinations. Each instructor (two in total in the third year) created a midterm and final; however, it had to correspond with new protocols. First, the midterm and final had to be objective (e.g., multiple choice, matching). Second, each examination had to contain 50 questions. Third, there had to be application, as well as knowledge- based questions. Application questions were pragmatic, and asked students to choose “the best” answer for a counseling situation. Thus, application questions allowed instructors to assess how well students could learn material and apply it to a novel situation. Additionally, each instructor sent their midterm and final for review and approval by an evaluation committee comprised of counseling faculty members. A member of the committee checked for consistency of content items, relevance of material covered, and adherence to examination protocols.

Graduate student surveys and faculty interviews. In the third year, surveys served as a qualitative measure to assess graduate student perceived strengths and weaknesses in learning, as well as addressing graduate concerns. Students went online at the end of the semester and completed a survey that gauged the utility of the theory class in potentially assisting future clients. Participants responded to the following four questions: How effective was the instruction in the theory class in understanding theoretical concepts and knowledge? How effective were materials (i.e., book, class activities) used in the theory class in understanding theoretical concepts and knowledge? How effective was the instruction in the theory class in applying theoretical concepts to real-world applications? How effective were materials (i.e., book, class activities) used in the theory class in applying theoretical concepts to real-world applications?

In addition to graduate student surveys, the evaluation committee (researchers) conducted open interviews with instructors in order to identify any problematic issues with students or curricula. These interviews, conducted in the second and third year, asked questions that were similar to those completed by graduate students. The questions were as follows: How effective was the book in the theory class in helping students to apply theoretical concepts to real-world applications? How effective were students in the theory class in applying theoretical concepts to real-world applications? How effective do you think your instruction was in teaching the theory course?

Procedures

First year. This study used a mixed method design over a three-year period. The impetus for initiating this study stemmed from employer feedback about graduates in the workplace. Due to the university being in a small community, employers had connections to faculty members. A number of different employers voiced concerns to faculty about graduates’ performance in the workplace (e.g., knowledge, skills, professional identity). Additionally, faculty began to notice similarities between employers concerns and students’ performance in class. In order to quantify program/student strengths and weaknesses, a program evaluation committee (the researchers) formed and implemented a three-year assessment.

Each year would build from the preceding, collecting more data and gathering information on how to improve student learning. Thus, the first year was an introductory period in which to build a foundation for subsequent years. The second year (in addition to the CPCE) added faculty interviews and a midterm. The third year (in addition to the CPCE, midterm, and faculty interviews) added a final and student online surveys. Through evaluation committee discussion, the CPCE was determined acceptable to answer the first research question. The committee determined instructor-developed examinations appropriate to answer the second research question. The rationale behind this determination is that the CPCE is a valid indicator of overall student competencies (Schmidt et al., 2009). However, a more hands-on approach (i.e., instructor-developed examinations) assesses application/skills better (Burt et al., 2011).

CPCE. Participants (i.e., students) completed the CPCE in the last year of their degree programs. Participants took the examination in either Fall or Spring semester. A full-time faculty member proctored the paper-and-pencil format examination, allocating 4 hours for completion per CCE requirements (Schmidt et al., 2009). After completion, faculty sent examinations to CCE for scoring. For each completed examination, the researchers collected the following participant data: (1) Demographics (e.g., gender, ethnicity), and (2) degree track (e.g., clinical mental health, school counseling). The evaluation committee determined the CPCE would be an instrument suitable for gathering information and beginning the process. The rationale was that the CPCE, having sound psychometric properties, could help identify program benchmarks (Schmidt et al., 2009). After identifying target areas, programmatic decisions deciding what to do with information could begin.

Second year. After applying the CPCE to identify deficits in student knowledge, the second year concentrated on connecting these deficiencies to specific courses and skills. According to CPCE results, students had the lowest scores in theories. The following brief description indicates what the theory course entailed. This course provided students with an understanding of major theories of counseling and psychotherapy, as well as learning therapeutic techniques and their applications within a multicultural and diverse society. The evaluation committee decided to focus on the lowest score/course in order to make the most significant changes and improvement.

As a supplement to the CPCE, the evaluation committee initiated an instructor- developed midterm that was mandatory for the two instructors teaching the theory course. The rationale behind creating an instructor-developed midterm was twofold. First, instructors have the most contact with students. Therefore, an instructor-developed exam may be better able to pinpoint needed improvements and indicate specifically where students struggle (e.g., concepts). Second, the hope was to provide more information to guide decisions on how to strengthen the program. By having a more focused examination, feedback and discussion between instructors and students could illuminate problems in learning or method of instruction. Qualitative measures, (open question interviews) began in the second and third year. The committee conducted these open question interviews with instructors. The interviews were to assist with the following: (1) understand instructors’ teaching theory, (2) ensure incorporation of CACREP objectives, and (3) evaluate instructors’ methods of teaching. One member of the committee met with the two instructors separately and asked three open-ended questions (previously listed in this article). Interviews lasted approximately 20-45 minutes.

Third year. Third year emphases focused on strengthening faculty teaching and improving students’ ability to think critically (i.e., application of concepts to real-life scenarios). Administration of the CPCE (and CPCE subtests) occurred, with the addition of a standardized instructor developed midterm and final. Results from the second year administration of the CPCE indicated continual weaknesses in the theory class. Complications also existed with the instructor-developed midterm. As a result, a revamping of the midterm occurred. More information regarding the problems with the instructor midterm begins in the discussion section of this article. Nevertheless, due to complications with the midterm, the evaluation committee decided that instructors needed to create two standardized, 50-question multiple-choice examinations (i.e., midterm and final). Each instructor of the course received instructions to create a midterm and final, given in the middle and end of the semester. As mentioned previously, standardizing exams was to create uniformity between courses and instructors. A one-way repeated measures of analysis (ANOVA) was the statistical procedure to measure differences in scores between the midterm and final. In addition to the CPCE and standardized instructor-developed midterm and final, the third year introduced student surveys assessing practicality of classes to experiential courses (i.e., practicum and internship).

Results

This article had two primary research questions. The first research question was as follows: Do students demonstrate an understanding of theories of counseling? In the first year, CPCE theories subtest score showed the national average was 13 (SD=2.34). The average score of the participants in the study was 11.39, (SD=3.0). Thus, participants performed lower than national average. Second year: CPCE subtest indicated the national average was 13.82 (SD=2.22). The average score of participants was 10.87 (SD=2.09). Third year: CPCE national average score was 13.42(SD=2.28). Participant average score was 11.79 (SD=2.11). Participants in the study performed lower than the national average during all three years (see Table 1 for all scores and standard deviation information).

Table 1

Participants’ means, standard deviations, and Repeated measures ANOVA

Second research question: Do students demonstrate an ability to apply counseling theories to practical (real-life) applications? One way repeated measures analysis of variance (ANOVA) used for midterm and final examinations indicated the following: DF=1, 31; F=1.11; Sig.= .299 (p>.05); Partial Eta Squared = .034 (see Table 1). There was not a statistically significant difference between student’s scores on a standardized midterm and final. Second theory class (Different instructor): One way repeated measures analysis of variance (ANOVA) used for midterm and final examination indicated: DF=1,35; F=9.48; Sig.=.004 (p<.05); Partial Eta Squared = .213 (see Table 1). There was a statistically significant difference between student’s scores on a standardized midterm and final.

Qualitative measures suggested that students were not content with the quality of instruction they received from instructors teaching the theory class. For example, student surveys (which did not identify instructors by name) indicated two major themes. The researchers identified the first theme as inadequate instructor preparation and the second as inadequate classroom etiquette. One quote from a student emphasizes the first theme: “We do the teaching in class. He told us the first day we would give individual chapter presentations for each class. We did the teaching and he just gave us tests on what we presented. I didn’t learn anything.” A second student’s statement conveyed the second theme: “He seemed grumpy. We had to get in groups and talk about projects we had so we would go in the halls. When we were out there too long, he would yell at us when we went over time.” Instructor interviews indicated a different perspective.

One instructor stated students “tend to be lazy” and “don’t come to class regularly.” Another instructor said, “we coddle them too much.” An additional problem noted by instructors was the lack of autonomy in choosing books, as books were “mandated by the program.” However, both instructors replied they felt their instruction “was sufficient” and “met the needs of both program and students.”

Discussion

Results indicated a number of interesting findings. First, participants performed at a lower level in the theory course. It was interesting that from student surveys, respondents labeled faculty instruction as deficient in the same course. Thus, a strong connection seemed to exist between student learning and student perception of faculty instruction. However, another reason may be able to explain this connection. For example, students knew of low student CPCE scores. Although faculty did not post individual results, a number of students still found out about their scores. Furthermore, the same year evaluation begun, faculty decided to implement the CPCE as an exit examination required for graduation. Due to low scores, a number of students had to retake the exam. Student sentiment was negative, and surveys may have been a method for students to attack faculty because of dissatisfaction with the CPCE. However, while some students may have held these views, instructor interviews revealed interesting findings. First, the same instructors taught this class over a number of years, and in many ways, thought they “owned” the courses. Subsequently, these same instructors provided the most resistance and reluctance to engage in program evaluation.

Moreover, program policy did not allow instructors to choose the books for their courses. As such, program mandated books for certain courses were, in some instances, not appropriate to course material. In essence, a faculty member chose the book(s) years ago, and the current instructor has no choice but to use it in the course. Further findings concluded some instructors were not teaching core CACREP competencies. By conducting qualitative measures, some instructors used self-instruction (having students teach themselves), were not up to date with current methods (teaching outdated information), not distributing syllabi regularly, or were teaching topics other than counseling (personal interests).

Limitations

This study had a number of limitations. First, the population was limited to participants from a medium-sized northeast, private intuition. The majority of participants were Caucasian, high socio-economic status (SES), and female. Although the study was over a number of years, it started slowly and attempted to build upon each previous year. This limited generalizability of the study as numerous obstacles transpired, and possibly affected outcomes. For example, in the second year, the committee mistakenly had instructors create their own exams without committee approval. This resulted in a large amount of information, but with an inability to analyze the data because it was not consistent. This setback resulted in the third year requiring a uniform midterm and final. In addition, although the third year introduced a more standardized midterm and final, instructors still personally developed their examinations. As such, there existed a range of level of examination difficulty due two different instructors and varying degrees of expertise. Class differences may have existed as well. For instance, groups of students may have had higher academic ability than other groups. Finally, a control group did not exist, although there was a comparative group (respondents of the CPCE). Limitations notwithstanding, this study was one of the few to scrutinize objectively a program and state specific curriculum improvements.

Implications

Results from this study have a number of useful implications for graduate student learning and program evaluation. For example, the program in this study is considering moving towards further standardizing the instructor developed midterm and final. In order to improve graduate learning and assure coverage of CACREP competencies, this study recommends administering examinations that have a uniform number of questions that are application-based. One such recommendation is ensuring that each examination has equal numbers of knowledge- and application-based questions. Instructors still create their own examination, but protocols for approving examinations would be more rigorous. For instance, the researchers recommend that individuals approving examinations receive training in test development. This training would include being able to understand what distinguishes strong and weak questions. Each individual reviewer receives a rubric and looks at multiple exams. Having multiple reviewers increases not only validity, but reliability as well (Bailey et al., 2010). As a result, inter-rater reliability between raters and examinations occurs in order to increase the integrity of midterms/finals. This study also recommends utilization of student surveys. Graduate student survey information can make classes stronger and more applicable to what graduate students may encounter in the field. Although there is a teacher evaluation form at the end of the semester, many times students do not take it seriously (Offenstein & Shulock, 2010).

Lastly, this study recommends implementing student tracking through a computerized system. The ideal is to give students an identification number at the beginning of their program. Examinations taken throughout their academic career would be via a computer, and students immediately submit their exams upon completion. Test results would connect each student to a class, grade, and instructor. This would enable faculty to be able to track individual progression over the course of any student’s career. By operating in this manner, it would enable programs to determine if some students need better preparation for graduate studies and classes. Programs could create workshops to address shortcomings in student learning or difficult subject matter. Programs such as computer tracking correspond with recent research conducted in counselor education (Burt et al., 2011). In conclusion, the merit of this study lies in identifying where program enhancement can begin, as well as improving graduate learning. As suggested by Jenkins (2011a), programs need to be able to indicate their strengths. This study sought to accomplish this goal by critically analyzing students and instructors. The recommendations suggested by this study may help counselor education programs to implement new methods to increase graduate learning.