VISTAS articles are made available for historical reference only and are presented "as is." ACA does not guarantee or represent that the information is current, accurate or indicative of the original or intended quality. These materials are not maintained or updated and may contain outdated or incomplete information. Readers should exercise discretion and verify information independently before relying on it. We assume no responsibility for the use or interpretation of this content.

Article 41

An Empirical Method for Spot-Checking Supervision Videotapes

Nicole A. Stargell

Download PDF

Correspondence concerning this article should be addressed to Nicole A. Stargell, Department of Educational Leadership and Counseling, The University of North Carolina at Pembroke, PO Box 1510, Pembroke, NC 28372. E-mail: [email protected]

Videotaped counseling sessions are an invaluable tool for counseling supervision (Borders & Brown, 2005; Huhra, Yamokoski-Maynhart, & Prieto, 2008; Romans, Boswell, Carlozzi, & Ferguson, 1995). Supervision is a distinct relationship in which one counselor responds to another counselor’s professional needs; the supervisor is often more experienced than the supervisee, and the supervisor must consider the supervisee’s developmental level when providing services (Russell-Chapin & Ivey, 2004). The Association for Counselor Education and Supervision (ACES; 2011) encourages counseling supervisors to view supervisees’ videotaped counseling sessions as a form of direct observation, and the Council for Accreditation of Counseling and Related Educational Programs (CACREP; 2009) requires that counselors in training receive supervision that incorporates audio or video recordings during practicum and internship. Myers and Smith (1995) found that almost two thirds of counselor education programs had rooms designated for review of videotaped counseling sessions, and this number has likely increased as technology and CACREP requirements have advanced. Overall, videotape review is a routine component of counseling supervision.

Borders and Brown (2005) suggested that supervisors should watch the entire videotaped counseling session, especially when working with neophyte counselors. Borders and Brown reported that supervisors are better able to assess counselor skills and session flow when the entire session is viewed. However, Borders and Brown also noted that certain situations warranted a “change of pace” (p. 42) when ethically appropriate. Accordingly, segments of videotape might be watched in order to illustrate a specific skill, when an ample amount of videotapes are available for review, or as the supervisee progresses in skill level. Spot-checking might be especially relevant for supervisors in the field who have limited amounts of time.

Definition and Ethical Considerations for Spot-Checking

In preparation for a supervision session, supervisors might choose to watch choice segments of supervisees’ videotapes, and this is called spot-checking; a spot-check is defined as “An inspection or investigation that is carried out at random or limited to a few instances.” (The American Heritage Dictionary, 2009). Although it is suggested that supervisors watch entire videotaped sessions, there are exceptions to this. Kepecs (1979) reported that a video clip lasting 10 to 20 minutes sufficiently represented a full-length (e.g., 50-minute counseling session) interaction. Relatedly, Riley-Tillman, Christ, Chafouleas, Boice-Mallach, and Briesch (2010) found that four 5-minute segments that were spread across a videotaped session more accurately produced observer ratings than a 20-minute observation from one section of the videotape. Spot-checking several time points more accurately captured the pacing and process of the session, while still allowing supervisors to assess supervisees’ skill level.

Although spot-checking is a viable option for supervisory practice, it must be used intentionally and with caution. Supervisors are ethically responsible for ensuring the welfare of their supervisees’ clients (American Counseling Association [ACA], 2014). As such, it is the supervisor’s responsibility to ensure that they are accurately able to judge supervisees’ abilities and skill level (Borders & Brown, 2005). Counseling supervisors are also in a unique position to evaluate supervisees and must ensure this is done to the best of their abilities; it is important that supervisors have a clear understanding of the breadth and depth of supervisee abilities (Borders & Brown, 2005; Huhra et al., 2008). With that in mind, supervisors might identify times when spot-checking would assist in upholding the ethical obligations of the supervisory relationship.

Spot-checking can be useful when supervisors have an abundance of videotapes at their disposal. Myers and Smith (1995) explored counseling laboratories used by counselor education programs to train neophyte counselors and found that they included an average of 4.4 rooms for videotaping counseling sessions. If counselors-in-training are able to record all of their sessions and supervisors are able to access such sessions, this would alleviate the difficulty noted by Borders and Brown (2005) in which supervisors had limited access to direct observation of their supervisees’ counseling sessions. In this case, supervisors might wish to watch some videos in their entirety and spot-check others to gain even more information about the supervisee.

Supervisors might ask their supervisees to prepare a segment of tape for a case presentation or for further exploration in supervision (Borders & Brown, 2005; Huhra et al., 2008). However, counselors are likely to choose a segment that aligns with their view of the client or session (Huhra et al., 2008). As such, supervisors might wish to perform a spot-check in supervision in order to highlight any contrasts between the clip chosen by the supervisee and the themes or concepts discovered in other portions of the tape.

Although the use of videotapes in supervision is a highly-supported form of direct observation in the educational setting (ACES, 2011; Borders & Brown, 2005; CACREP, 2009; Romans et al., 1995), it is not a required supervisory activity when supervising practicing counselors (ACA, 2014). In more advanced supervision, the supervisor must act as a teacher, counselor, and consultant (Johnson & Stewart, 2008). Usher and Borders (1993) found that practicing counselors preferred a supervisor who focused on the relationship rather than tasks and who primarily assumed the consultant role over that of the teacher role. As such, the supervisor must work to uphold the ethical responsibilities outlined by ACA (2014; i.e., to protect the welfare of the client), while also meeting the supervisee’s needs. A supervisor might spot-check counseling videos in preparation for supervisory meetings and spend more time in the supervision session on consultation and building the relationship.

Overall, spot-checking can be a helpful supervisory tool that allows supervisors to gain an even more dynamic understanding of their supervisees’ needs and abilities. Spot- checking can be used with neophyte counselors in addition to viewing entire videotapes. This supervision method can also be used with more advanced counselors who still benefit from the use of videotape review, but also have more complex supervisory needs. The use of spot-checking has been around for many years, and this technique is supported in some research studies (e.g., Kepecs, 1979; Riley-Tillman et al., 2010). It is now important to explore the most effective way to utilize this technique in counseling supervision.

Methods

Although spot-checking is an important tool for use in supervision, there is very limited research on how to best utilize this technique. Riley-Tillman et al. (2010) researched this topic in the context of behavioral observations of children and found that four 5-minute observations produced more accurate results than one 20-minute observation. However, there is no current literature about the use of spot-checking specifically in counseling supervision. As such, the purpose of this study was to determine if supervisors can utilize four 5-minute segments to gain an accurate representation of a full-length counseling session.

Sample

The sample was obtained through an existing database from a counselor training clinic at a state university in the southeastern United States. The database contains videotaped sessions and corresponding client/session information. Master’s and doctoral- level counselor trainees in a practicum course served as the counselors; undergraduate students served as the clients and attended three to seven counseling sessions. Each client and counselor signed informed consent to contribute to the research database, and IRB approval was received for this analysis.

The principal researcher reviewed the database to identify counselor-client pairs that met the data requirements. Requirements included: each counselor-client pair must have a minimum of three sessions, and these sessions must include an intake (first session), middle (session halfway through the counseling relationship), and termination session (final session, which ranged from session 3 through 7); and all three sessions must have lasted at least 45 minutes. The existing data was organized by semester, and the principal researcher began with the most recent semester (i.e., summer 2013) and randomly identified the first five eligible participants. Data within each semester is organized alphabetically; as such, the researcher randomly chose a number between 1 and 26 to identify the letter at which data collection started (i.e., a=1) and three letters were skipped between each collection until five eligible counselor-client pairs were identified.

Measure

In order to build upon Riley-Tillman et al.’s (2010) findings, behavioral observations were used to assess the accuracy of spot-checking in counselor supervision. Although supervisors evaluate a wide range of counselor skills and processes, counselor and client nonverbal behaviors were observed in this study because they are important to the therapeutic relationship and highly quantifiable (Andersen & Andersen, 1982; Richmond, McCroskey, & Johnson, 2003). As such, The Nonverbal Immediacy Scale (NIS) was use to measure counselor and client nonverbal behaviors in this study.

The NIS is a 26-item self- or observer-reported instrument used to measure nonverbal immediacy behaviors (NIB) in a variety of settings (Richmond et al., 2003). The NIS is the most recent and comprehensive measure of NIB (Aydin et al., 2013; Richmond et al., 2003) and measures the majority of NIB constructs identified by Andersen and Andersen (1982). Researchers have consistently found high internal consistency with Cronbach alphas of .90 and above (Madlock, 2008; McCroskey, Richmond, & Bennett, 2006; Richmond et al., 2003). Additionally, Richmond et al. (2003) found strong predictive validity of the measure; the NIS was strongly and positively correlated with warmth and approachability (.74 to .95 disattenuated). As such, the NIS is a psychometrically-sound instrument that can be used to quantify nonverbal behaviors.

Procedures

In order to assess if four 5-minute segments were representative of an entire counseling session, time intervals that captured the beginning, middle, and end of each session were identified. Riley-Tillman et al. (2010) chose four 5-minute segments that spanned the duration of an observation period and found that this method produced representative and reliable ratings. As such, the following segments with 5–10 minutes lapsing between each observation were assessed: 0–5 minutes, 15–20 minutes, 30–35 minutes, and 40–45 minutes.

At least 27 behavioral observations obtained by watching the full video and at least 27 observations obtained by watching four five-minute segments of the same video needed to be compared in order to achieve power of .80, a medium effect size of .5 (Cohen’s d), and an alpha of .05. The five counselor and client pairs each met for three sessions which resulted in 15 videos for analysis. The rater watched half of the videos in whole, and the principal researcher watched those videos in the segmented 20 minutes, and vice versa (see Table 1). Each time a video was watched, one counselor NIS score and one client NIS score was produced (see bracketing procedures below). As such, 15 videos were watched by the principal researcher and the same 15 videos were watched by the rater. This resulted in 30 NIS scores from the rater (one for counselor and one for client for 15 videos) and 30 NIS scores from the principal researcher. Of these, 30 were produced by watching the segmented 20 minutes and 30 were produced by watching the whole session. All intake sessions were watched first (videos 1–5), then middle sessions (videos 6–10), then all termination sessions (videos 11–15) were watched. Rating occurred immediately after watching each video.

Table 1

Schedule of Videotape Viewing

Note. PR stands for principal researcher and R stands for rater.

Training and Bracketing. The principal researcher and a rater completed the observer-rated NIS for both the counselor and client in each session, which could have potentially led to researcher bias. In order to control for this, bracketing was used. First, the principal researcher trained the rater on use of the NIS before tape viewing began. This training included an overview of NIB, ways in which the NIS is used to assess NIB, and a discussion of NIB biases that might affect use of the NIS. When discussing these biases, formal bracketing procedures were used. The principal researcher and rater identified key words and phrases that explain the way in which the researchers might be generally biased toward a counselor or client; this included words and phrases such as gender, aggressive, and can’t help it. These words and phrases were used to create a definition of the potential bias and how it might affect the data collection (Fischer, 2009; Holstein & Gubrium, 1994). This definition was as follows: “As counselors, we tend to view NIB in context; when rating, we must process them as individual occurrences with consistency across the sample.” The principal researcher and rater then identified ways in which this definition could be upheld and biases avoided when rating counselor and client NIB. It was determined that a note card including the definition of NIS biases would be helpful. The principal researcher and rater kept a written copy of the definition in eyesight at all times throughout data collection.

After training and initial bracketing procedures, the principal researcher and rater practiced using the NIS on a few counseling videos from a library archive. The principal researcher and rater continued discussing biases as they watched these videos. When the principal researcher and rater were able to reach an agreement on use of the NIS, tape viewing began. Throughout data collection, the principal researcher and rater reviewed the bracketing procedure before watching each video to ensure researcher bias was minimized throughout the data collection process.

Data Analysis

The data was collected using Microsoft Excel. The principal researcher and rater each kept a separate database in Excel that had identical columns and numbering. Upon completion of data collection, the rater placed the completed database on the principal researcher’s jump drive. The principal researcher then created a formula that aligned with the NIS scoring procedures in order to identify one NIS score for the counselor and one NIS score for the client for each of the 15 tapes (resulting in two sets of 30 scores).

The principal researcher then organized the data for analysis in Microsoft Excel. Ratings 1 through 30 for all whole tapes were placed in a column and ratings 1 through 30 for the four 5-minute segments were in an adjacent column in which each video aligned (e.g., counselor rating for whole video 1 was next to counselor rating for partial video 1). The data were then directly imported into IBM Statistical Package for the Social Sciences 20 (SPSS) for analysis.

The representative nature of the four 5-minute segments in relation to the entire video was analyzed using a paired t-test for two dependent means. Counselor and client NIS scores for the entire video were compared to the corresponding NIS score for the specified four 5-minute segments. This analysis provided information about the representative nature of the segments versus the whole video.

Results

For the 30 NIS scores obtained by watching the entire video, the mean was 85.5 (SD=10.63). For the 30 NIS scores obtained by watching four 5-minute clips, the mean was 84.5 (SD=9.58). The paired t-test comparing the scores obtained by watching the whole video and the scores obtained by watching the four 5-minute segments was non- significant (t=.480; p=.635). As such, it can be concluded that there is not a significant difference between NIS scores obtained by watching the whole video and NIS scores obtained by watching the four 5-minute segments.

Discussion

In this study it was found that NIS scores produced by watching minutes 0–5, 15– 20, 30–35, and 40–45 of a counseling video were not significantly different than NIS scores produced by watching an entire counseling video. Counseling supervisors have used spot-checking for many years, and this supervision technique can now be supported with empirical findings. As such, supervisors with limited time can spot-check supervision videos that might not otherwise be viewed. Supervisors should also consider spot-checking additional video recorded counseling sessions in order to supplement their preparation and interventions in supervision sessions. Supervisors should work to uphold all of the ethical standards of supervision as outlined by ACA (2014) and ACES (2011) and utilize spot-checking to enhance their supervisory practices.

It is important to note that the current study was conducted using the NIS as a standard measure across sessions. Supervisors should note that nonverbal behaviors might be more consistent across sessions than other counseling skills and nuances. As such, supervisors should take care to utilize spot-checking practices only when the goal of supervision could likely be addressed through this abbreviated observation procedure.

Future research should be conducted by comparing the results of other counseling measures from full and spot-checked sessions. This would allow supervisors to gain a stronger understanding of the instances in which spot-checking is most appropriate. Additionally, researchers should continue to explore additional time intervals that might produce representative spot-checking results. It is possible that fewer than four 5-minute segments might allow supervisors to gain insight into supervisee abilities and needs.

Overall, there is strong theoretical support for the use of videotape viewing in preparation for supervisory sessions (ACES, 2011; CACREP, 2009; Borders & Brown, 2005). Additionally, spot-checking is a viable way in which supervisors can view a variety of supervisee recordings while still gaining an accurate representation of each session. Spot-checking can also be used to supplement supervision when watching full counseling sessions is not possible. Supervisors should maintain a keen awareness of ethical obligations and work to optimize their preparation time in order to provide the most comprehensive and effective supervision for counselors.