VISTAS articles are made available for historical reference only and are presented "as is." ACA does not guarantee or represent that the information is current, accurate or indicative of the original or intended quality. These materials are not maintained or updated and may contain outdated or incomplete information. Readers should exercise discretion and verify information independently before relying on it. We assume no responsibility for the use or interpretation of this content.

Article 45

InternetLoggingSystem.Com

An Analytical Tool for Student and Program Evaluation

Teddi J. Cunningham and Kerry T. Hinkle

Download PDF

Professional programs in community and school counseling have benefited through educational reform initiatives which have resulted in discernible changes in guidelines for training and practice. One of the most salient changes has been the formalization of practicum and internship standards as delineated by the Counsel for Accreditation of Counseling and Related Educational Programs (CACREP; 2001). Consequently, university trainers engaged in preparing future professionals must adhere to enhanced levels of accountability, standards, and outcomes (Cobb, 1995). This restructuring of professional preparation programs focuses on the need for performance-based assessment which emphasizes what a student must know and be able to do (Wise & Leibrand, 1996). With this change has come greater attention to the identification and measurement of critical professional competencies acquired and outcomes achieved throughout a student’s training program. To this end, clinical field experiences, including practica and internships, have long been a part of practitioner training in community and school counseling. Thus, a focus on applied experience has become paramount in the practical training in school and clinical psychology.

Practica and internships are designed to be developmental and graduated in order for students to transfer classroom learning to actual clients/students while still under varying degrees of faculty supervision. As such, practica and internships provide students with the opportunity to show what they know, as well as what they can do. It is also a venue to introduce new skills and requirements. For their part, university trainers are responsible for assessing how well the students, as well as the program, meet goals established by state and national standards, and accreditation requirements.

The assessment of student competencies and performance is best conducted through the use of multiple sources as well as multiple methods of gathering data (Waldon, Prus, & Curtis, 2001). One method frequently used by training programs to assess the competencies and performance of students is field experience logs. As a performance appraisal, practicum and internship logs reflect an overt demonstration of skills in an authentic setting. Performance appraisals are used to assess skill development and attainment of knowledge and skills in specific settings and situations. Although practicum and internship logs have historically been used as examples of practice demonstration, traditional pencil-and-paper methods lack the operational precision necessary to be used as process or outcome measures of individual students or programs.

Practice Makes Perfect, But How Can We Be Sure?

One reason why traditional field experience logs are not dependable as performance appraisals rests with their failure to adequately address psychometric elements of the assessment process. As noted by Waldon et al. (2001, p. 4), assessment measures used to examine student competencies and performance should be selected in terms of their relationship with the following factors:

• reliability, validity, and utility;

• the need for multiple methods that repre- sent multiple data sources, multiple environments, and the assessment of multiple domains; and

• the need for methods that allow assessment across time, and provide continuous monitoring of progress toward desired goals and outcomes.

The Internet Logging System as an Analytical Rubric

The Internet Logging System (ILS) (Hinkle, 2002) is a Web-based, password secure program designed for students, field supervisors, and university trainers involved in school or clinic-based practicum or internship experiences. Data are entered into the ILS according to CACREP standard, date, length of activity, service category, type of activity, consumer age, race/ ethnicity, and gender. These data are chronologically recorded on the student Activity Log according to date, length of activity, type of activity, total number of hours, and number of data entries. Quantitative data are captured on the student Detail Log, which records and automatically compiles total time and percent of time spent in each service category as well as specific subcategories. Consumer data describing the person or group receiving student directed services are also recorded on the student’s Detail Log. The program enables access at any time, by any authorized individual, and allows formative and summative data analysis and profiling of student activities.

As an authentic measure of student skill development and attainment, the ILS is specifically designed to address many of the technical inadequacies associated with traditional paper-and-pencil methods. Specifically, the ILS was proposed to more adequately meet the professional criteria set for the appropriate assessment of student and program outcomes.

Reliability

A basic question about traditional practicum and internship logging methods is whether different raters/ observers can agree on which activities a student may be performing or referencing. Without a common rubric, it is impossible for field and university supervisors to establish interrater agreement. The ILS records and reports both qualitative and quantitative data in each of the eight operationally defined service categories, each further divided into specific service subcategories (e.g., consultation with teacher, parent, administrator, or other agency personnel). Through the use of specific categorical criteria, site and program supervisors can more consistently evaluate field- experience data and create more reliable performance appraisals.

Validity

The most important characteristic of any instrument is validity. The ILS provides evidence of several forms of validity. First and foremost, the ILS asks students for information directly related to their field activities. By recording and reporting their activities into operationally defined service categories (e.g., consultation, intervention, assessment, conferences/staffing, in-service training, research/ program evaluation, supervision) students quickly recognize the face validity of the process. Further, content validity of the ILS is reflected through its relationship to national standards set by organizations such as the Council for Accreditation of Counseling and Related Educational Programs (2001), National Association of School Psychologists (2000), American Psychological Association (2002), and American School Counselors Association (2003). That is, the content of the ILS reflects areas of professional and service experience described by the professional literature as best practice for community and school counseling students.

Utility

Traditional paper-and-pencil logs lack consistency by failing to provide common or operationally defined practicum or internship services or activities. As a result, such field-experience data are difficult to qualitatively or quantitatively assimilate. This concern with data aggregation is problematic at both the individual and program level. In contrast, the ILS allows students to record and report a range of service activities and automatically aggregates their experiences into operationally defined categories. The ILS system also allows students to (a) provide site and university supervisors with hard or e-mailed copies of their logs; (b) download their logs into their electronic portfolios, linking course content with field application; (c) use the ILS to distinguish themselves during job interviews by quickly accessing and searching experiences by service area or consumer group; and (d) provide a breakdown of experience for certification and licensure.

The Need for Multiple Methods/Data Sources/ Environments, and Domains

Since most professional training programs utilize a number of practica and internships aimed at addressing an array of professional skills and experiences, the ILS is ideal for monitoring and measuring program standards associated with specific curriculum or area of training (except consultation or intervention). Because the ILS provides trainers with a concise but comprehensive record of each student’s field activities, data aggregation affords multiple measures of student performance in multiple settings, by multiple supervisors within multiple domains.

Methods Assessing Across Time/Goals

The ILS is continuously available, and provides site and university supervisors with a real-time review of student field experiences. A student’s practicum or internship log is accessible from any PC that has Web access. In addition, the ILS provides the supervisor with a clear delineation of services offered at various field placements affording a more exact student-site match. Because of its ease of access, the ILS makes monitoring student activities quick and simple. Monitoring student progress and experiences can be made from any PC location, so instructors can quickly review and adjust student field activities to comply with course, program, or accreditation requirements.

The ILS as a Program Evaluation Tool

The ILS is specifically designed for professional training programs in community and school counseling, school psychology, and clinical/counseling psychology. Student data compiled by the ILS can easily be aggregated to serve a number of important student and program evaluation functions.

The ILS uses clear criteria or rubrics for evaluating student performance. Thus, data can be collected to help trainers’ select appropriate settings through comparison of student experiences in different schools and with different site supervisors. Logs can also be used to evaluate the competencies of students prior to as well as during their practicum or internship experience. ILS data can also be used to review program compliance with state and national credentialing and accreditation standards. Finally, the ILS serves as an off-site repository for student and program data. The Internet Logging System not only makes recording and reporting student and professional experiences simple, secure, user friendly, and informative, but it also serves as a valuable analytical tool for student and program evaluation.