Establishing the Initial Validity of the REDFLAGS Model

A notable increase in the prevalence of mental health disorders (MHDs) has been documented among the college student population in recent years (Auerbach et al., 2016). Depression, anxiety, self‐injurious behaviors, and suicidality are growing concerns at higher educational institutions across the United States (Lipson, Zhou, Wagner, Beck, & Eisenberg, 2016). In a recent study, 20.3% (n = 1,572) of college students indicated experiencing an MHD with symptomatology that was present for 12 months or longer (Auerbach et al., 2016). More specifically, suicide is the second leading cause of death among college students (Albright & Schwartz, 2017). Data from the 2016 American College Health Association survey indicated that 10.4% of more than 36,385 students reported that they had “seriously considered suicide” in the past 12 months (American College Health Association, 2017, p. 14). Although many colleges and universities offer services aimed at ameliorating some of the mental health concerns experienced by students, only a small number of college students are receiving any treatment for their mental health needs (Auerbach et al., 2016).

Consequently, there is a need for cost‐effective methods for increasing university community members’ familiarity with recognizing warning signs of mental distress in college students. The REDFLAGS model, an acronym of observable warning signs of a student who might be struggling with an MHD, for example, is a cost‐effective tool for increasing university community members’ awareness of warning signs for mental distress (Kalkbrenner, 2016). The primary aim of the present study was to empirically test the REDFLAGS model with 4‐year university students. If validated, the REDFLAGS model has the potential to aid college counselors with training university community members to recognize and refer students to mental health support resources.

College Student Help‐Seeking Behavior

Even though scholars have established that the use of campus mental health resources could contribute positively to students’ mental health (Goodwin, Behan, Kelly, McCarthy, & Horgan, 2016), the underutilization of mental health services on college campuses is well documented in the literature (Auerbach et al., 2016; Rosenthal & Wilson, 2016). Rosenthal and Wilson (2016) estimated that 87% of college students in their sample (n = 847) had not utilized mental health services in the past 6 months. Several factors contribute to the underutilization of mental health services among college students, including stigma (Rosenthal & Wilson, 2016; Talebi, Matheson, & Anisman, 2016), logistics related to securing help (Czyz, Horwitz, Eisenberg, Kramer, & King, 2013; Rosenthal & Wilson, 2016), lack of time to seek help (Czyz et al., 2013), limited understanding of mental health services (Rosenthal & Wilson, 2016), inability to recognize the need for services (Rosenthal & Wilson, 2016), cost of services (Marsh & Wilcoxon, 2015), and lack of social support (Talebi et al., 2016). In addition to these elements, demographic characteristics contribute to the underutilization of mental health services among college students (Eisenberg, Hunt, Speer, & Zivin, 2011). In particular, differences in students’ gender, ethnicity, and knowledge of warning signs for MHDs are associated with a reduced likelihood of utilizing mental health support (Eisenberg et al., 2011).

College students who identify as female attend counseling at higher rates and tend to report a greater willingness to refer classmates to resources for MHDs when compared with their male counterparts (Eisenberg et al., 2011; Kalkbrenner & Hernández, 2017). According to Brownson, Becker, Shadick, Jaggars, and Nitkin‐Kaner (2014), students of minority ethnic backgrounds also face more significant challenges in attaining support for mental illness. In a survey of more than 14,000 students of racial and ethnic backgrounds across the United States, they found students who identify as White reported predictably lower rates of mental health problems such as suicidal ideation when compared with ethnic minority students.

Although there appear to be some ethnic and gender differences in regard to the utilization of mental health services among college students, these variances also are attributable to students’ knowledge of warning signs for MHDs and awareness of university support services (Dobmeier, Kalkbrenner, Hill, & Hernández, 2013). Students keenly aware of warning signs for MHDs reported a significantly higher willingness to refer a classmate suffering from mental distress to resources when compared with students who were unaware of warning signs for MHDs (Kalkbrenner & Hernández, 2017). Unfortunately, only a small proportion of college students reported a high awareness of warning signs for MHDs. In recognizing college students’ mental health needs and the typical pattern related to the underutilization of mental health services, knowledge, and awareness of resources across campuses, can help ameliorate the increase in college students’ mental health needs (Kalkbrenner & Hernández, 2017). However, findings from the literature suggest that many college campuses are not sufficiently equipped to support the growing mental health needs of students (Albright & Schwartz, 2017). The paucity of resources many college counseling centers are facing has underscored the need for referral networks on campus to connect college students to resources for MHDs (Brunner, Wallace, Reymann, Sellers, & McCabe, 2014).

Barriers to Referrals

College students and faculty members are viable referral agents for recognizing and referring students to the counseling center and other resources for MHDs (Kalkbrenner, 2016; White, Park, Israel, & Cordero, 2009). However, the results of a recent national survey of college students (n = 51,294) revealed that more than half of the respondents did not feel prepared to recognize warning signs of mental distress in college students (Albright & Schwartz, 2017). Similarly, Cerel, Bolin, and Moore (2013) found that college students were at risk of missing potential warning signs for suicide in their peers if they had not obtained adequate training. They also found that 65% (n = 76) of students reported knowing at least one person who had attempted or died by suicide (Cerel et al., 2013). However, only 53% (n = 62) reported having any knowledge of the National Suicide Prevention Lifeline (NSPL). These findings suggest a need to direct mental health services not only to those students with self‐reported mental health needs but also to those who have frequent contact with students who are struggling with mental health issues.

Olson, Koscak, Foroudi, Mitalas, and Noble (2016) demonstrated the positive impact peer health education initiatives can have on increasing peer‐to‐peer referrals to resources for MHDs. However, these education initiatives can be costly and impractical in the context of the restricted budgets many university counseling centers are facing (Kraft, 2009; White et al., 2009). The literature is lacking cost‐effective methods for increasing university community members’ familiarity with recognizing warning signs of mental distress in college students. The REDFLAGS model is a cost‐effective tool that was designed to increase university community members’ awareness of indicators of mental distress among college students (Kalkbrenner, 2016).

The REDFLAGS Model

The REDFLAGS model (see Figure 1) comprises eight warning signs of mental distress in college students (Kalkbrenner, 2016). The first letters of the warning signs form the acronym REDFLAGS. For example, the R at the beginning of REDFLAGS stands for “Recurrent class absences that are sudden or uncharacteristic of the student.”

Figure 1

The REDFLAGS Model

Recurrent class absences that are sudden or uncharacteristic of the student

Extreme and unusual emotional reactions

Difficulty concentrating

Frequent display of anxiety or worry about class assignments

Late or incomplete assignments turned in abruptly and with increasing frequency

Apathy toward personal appearance and hygiene

Gut feeling that something doesn’t seem right

Sudden deterioration in quality of work or content of work becomes negative or dark

Note. “The REDFLAGS Model is an acronym that highlights eight warning signs that suggest a student might be struggling with an MHD [mental health disorder]. The REDFLAGS Model should not be used to diagnose a student with an MHD. Rather, the model is intended to be used by faculty and other college administrators as a tool for identifying students who might be struggling with mental health concerns” (Kalkbrenner, 2016, p. 5). From “Recognizing and Supporting Students With Mental Disorders: The REDFLAGS Model” by M. T. Kalkbrenner, 2016, Journal of Education and Training, 3, p. 5. Copyright 2016 by M. T. Kalkbrenner. All rights reserved. Reprinted with permission.

The REDFLAGS model is both similar to and different from existing resources for recognizing mental distress. For example, the NSPL wallet cards include warning signs for mental distress accessible electronically or in hard copy form (NSPL, 2007). Similarly, the REDFLAGS model comprises warning signs for mental distress and can be distributed in electronic or hard copy form. However, the warning signs for mental distress and suicide on the NSPL wallet cards are primarily composed of internal symptomatology or warning signs of mental distress that might be difficult for a student to recognize in a peer (e.g., feeling numb or like nothing matters). Also, the wallet cards have utility for spreading awareness about warning signs of suicidal ideation specifically. The REDFLAGS model is composed of primarily external warning signs of mental distress that are observable and specific to the college student population (e.g., frequent display of anxiety or worry about class assignments). Additionally, the warning signs composing the model are not limited to one particular mental health condition. Furthermore, because the model's cautionary signs compose an acronym (REDFLAGS), the model's warning signs may be easier for counseling referral agents to recognize and remember (Kalkbrenner, 2016).

The REDFLAGS model was designed to be a cost‐effective tool for increasing university community members’ awareness of warning signs for mental health distress in college students (Kalkbrenner, 2016). The model also was intended to encourage university community members to refer students who may be showing signs of mental distress to the college counseling center. The REDFLAGS model shows the potential for campus‐wide implementation because the model can be distributed at little or no cost (Kalkbrenner, 2016).

It should be noted that the items composing the model are not an all‐inclusive list of warning signs for mental distress in college students and should be used as a starting point for identifying students in psychological distress. Kalkbrenner (2016) made several recommendations for action steps students can take once they recognize a peer who might be in mental distress, for example, consulting with a college counselor about one's concern or talking to the student. See Kalkbrenner (2016) for a description and examples of some supportive statements students can use to start a conversation with a peer in mental distress.

Purpose Statement and Research Questions

The REDFLAGS model has the potential to be a valuable tool for assisting college counselors with their outreach and education work through promoting peer‐to‐peer referrals to the counseling center. The primary purpose of the present study was to investigate the reliability and validity of the REDFLAGS model with a sample of 4‐year university students. If validated, the REDFLAGS model could make a substantial contribution to the college counseling knowledge base considering the growing frequency and complexity of MHDs on college campuses (Auerbach et al., 2016), students’ underutilization of counseling services (Rosenthal & Wilson, 2016), and financial restrictions that many college counseling centers are facing (Kraft, 2009). Consequently, we sought to answer the following research questions:

Research Question 1: To what extent do the REDFLAGS model’s warning signs display internal consistency reliability (Cronbach’s coefficient alpha)?

Research Question 2: What is the construct validity of the REDFLAGS model?

Research Question 3: To what extent does students’ recognition of the items on the REDFLAGS model as warning signs of mental distress predict whether they have referred another student to the counseling center?

Research Question 4: Are there demographic differences in students’ recognition of the REDFLAGS model?

Research Question 4a: Are there significant differences by gender in students’ recognition of the items on the REDFLAGS model as warning signs for mental distress?

Method

Participants and Procedure

We first obtained approval from the institutional review board to begin data collection. Data were collected from 328 undergraduate college students who were enrolled in a large public university at the time of data collection. The only inclusion criteria for participation in the present study were active enrollment in at least one college course and an age of at least 18 years at the time of data collection. A nonprobability sampling procedure was used: Participants were recruited as they walked by a table in the student union. Paper copies of the questionnaire were administered. A small bag of candy was offered to incentivize participation. We entered the data into an SPSS (Version 25) spreadsheet for analysis.

Concerning participant demographics, by gender, 63.4% (n = 208) of participants identified as female, 35.4% (n = 116) identified as male, 0.9% (n = 3) identified as nonbinary or third gender, and 0.3% (n = 1) did not specify a gender. By ethnicity, 48.5% (n = 159) identified as African American, 30.5% (n = 100) identified as White, 9.5% (n = 31) identified as multiethnic, 6.1% (n = 20) identified as Hispanic or Latina/o, 3.7% (n = 12) identified as Asian, 0.9% (n = 3) identified as Native Hawaiian or Pacific Islander, 0.3% (n = 1) identified as American Indian/Alaska Native, and 0.6% (n = 2) did not specify their ethnic identity. Participants ranged in age from 18 to 47 (M = 22 years, SD = 5) with the majority of participants, 91.1% (n = 298), between the ages of 18 and 26 at the time of data collection. (Percentages may not add up to 100 because of rounding.)

Instrumentation

Participants responded to demographic questions about their age, gender, ethnicity, referrals to the counseling center, the number of credit hours they were enrolled in, and grade point average. We developed the REDFLAGS questionnaire based on the steps provided by DeVellis (2016) to measure the extent to which respondents recognized each of the items on the REDFLAGS model as cautionary signs of mental distress. Participants responded on a 5‐point Likert‐type scale to the following prompt: “Below are examples of behaviors that may or may not be warning signs that a student is struggling with a mental health issue. Please read each statement carefully and select the response that most accurately reflects your view.” Response options were 1 = I strongly disagree that this behavior is a sign of a mental health issue, 2 = I disagree that this behavior is a sign of a mental health issue, 3 = I'm not sure if this behavior is a sign of a mental health issue, 4 = I agree that this behavior is a sign of a mental health issue, or 5 = I strongly agree that this behavior is a sign of a mental health issue. Likert‐type scaling was used on the basis of the recommendations from DeVellis (2016), and the anchor definitions were adapted from Vagias (2006). Participants’ mean composite scores were computed, with higher scores denoting a greater ability to recognize the items on the REDFLAGS model as warning signs of mental distress in college students.

Data Cleaning and Assumption Checking

We followed the guidelines for assumption checking provided by Field (2013) to ensure the data were appropriate for factor analysis, hierarchical logistic regression, and analysis of variance (ANOVA). A missing values analysis revealed that less than 1% of data were missing for all REDFLAGS model items. Missing values were replaced with the series mean, outliers were Winsorized, and skewness and kurtosis values were largely within the acceptable range (< + 1) of a normal distribution (Field, 2013). The only skewness value that was greater than 1.0 was Item 2, “Extreme and unusual emotional reactions,” skew = 1.10; however, this item was included in the factor analysis because the principal‐axis factor extraction method is robust to moderate violations of normality (Mvududu & Sink, 2013). Pearson product‐moment correlations between the independent variables revealed the data did not show multicollinearity. The results of the Box‐Tidwell procedure (Box & Tidwell, 1962) revealed that the assumption of linearity was met (all continuous independent variables were linearly related to the logit of the dependent variable). The results of a Levene's test demonstrated that the assumption of homogeneity of error variances had not been violated, F(5, 318) = 2.10, p = .07. G∗Power, Version 3.1, statistical power analysis was used to conduct a prior power analysis to ensure the sample size was sufficient for inferential statistical analyses (Faul, Erdfelder, Lang, & Buchner, 2007). Results revealed that a minimum sample size of 192 would provide an 80% power estimate, α = .05, with a moderate effect size (f = 0.25). The guidelines provided by Sink and Mvududu (2010) for a “moderate” (p. 15) effect size, η2p = .06, were entered into G∗Power. In terms of sample size, psychometric researchers have recommended sample sizes include at least 10 to 20 participants for each estimated parameter for factor analysis (Kahn, 2006; Mvududu & Sink, 2013). The sample size in the present study (N = 328) was sufficient for factor analysis, providing 29 participants for each estimated parameter.

Results

Results are presented in two phases as they correspond to the research questions. In Phase 1, the results of tests related to the construct validity of the REDFLAGS model are presented. In particular, tests of internal consistency reliability, interitem correlations, and factor analysis are discussed. In Phase 2, the findings related to the predictive and discriminant validity of the REDFLAGS model are presented, including hierarchical logistic regression and ANOVA.

Phase 1: Reliability and Factorial Validity

Interitem correlations and a test of internal consistency (Cronbach's coefficient alpha) were computed to investigate the reliability of the REDFLAGS model (Research Question 1). Internal consistency reliability analyses revealed a strong reliability coefficient (α = .89) for the REDFLAGS questionnaire. The eight items composing the REDFLAGS model were then entered into an interitem correlation matrix (see Table 1). An interitem correlation matrix in which the majority of correlations between items are at least .20 and not higher than .80 indicates that items are favorable for factor analysis (Kahn, 2006; Mvududu & Sink, 2013). Interitem correlations were promising (see Table 1) and ranged from .41 to .60. Furthermore, Bartlett's test of sphericity, B(28) = 1,187.92, p < .001, and the Kaiser‐Meyer‐Olkin (KMO) test of sampling adequacy (KMO = .89) provided further support that the correlation matrix was favorable for factor analysis. A principal factor analysis with a principal‐axis factor extraction method was computed on the basis of the recommendations of Mvududu and Sink (2013), revealing a single factor solution (see Table 2). The combined results of the factor extraction criteria provided by leading psychometric researchers (Mvududu & Sink, 2013) supported the retention of a single factor, including the Kaiser criterion, percentage of variance accounted for by a derived factor (≥ 5%), Cattell's scree test, and the results of parallel analysis. The guidelines for factor retention criteria provided by Mvududu and Sink were used: factor loading > .40, communality (h2) > .30, and cross‐loading < 0.30. A coherent factor structure emerged (see Table 2) and provided preliminary support for the construct validity of the model (Research Question 2).

Table 1

The REDFLAGS Model Interitem Correlation Matrix

Note. Items have been abbreviated for space. See Figure 1 for full description of the REDFLAGS model.

Table 2

Principal Factor Analysis Results for the REDFLAGS Model

Note. N = 328. Used principal‐axis factoring as extraction method.

Phase 2: Predictive and Discriminant Validity

A hierarchical logistic regression analysis was computed to determine the extent to which participants’ awareness of the items on the REDFLAGS model as warning signs for mental distress predicted whether they had referred another student to the counseling center (Research Question 3). College students who are older or female tend to be more aware of the warning signs for MHDs and more likely to make referrals to resources (Eisenberg, Goldrick‐Rab, Lipson, & Broton, 2016; Kalkbrenner & Hernández, 2017). Age and gender were consequently added as predictor variables in the first regression model and revealed statistical significance, χ2(2) = 12.62, p < .001. The model explained 5% of the variance (Nagelkerke R2) in whether participants had referred another student to the counseling center. Respondents’ composite score on the REDFLAGS model was entered into the second regression model as a predictor variable. The second regression model was a significantly superior predictor of students’ referrals to the counseling center, χ2(3) = 71.16, p < .001. The variance accounted for (Nagelkerke R2) in students’ referrals to the counseling center improved from 5% to 31% when the REDFLAGS model was entered into the regression analysis, and correctly classified 77% of cases. The odds ratios, Exp(B), revealed that an increase of one unit in students’ recognition of items on the REDFLAGS model as warning signs for mental distress was associated with an increase in the odds of having referred a peer to the counseling center by a factor of 7.60, 95% CI [4.40, 13.14].

A 2 (gender) × 3 (ethnicity) ANOVA was computed to investigate group demographic differences in students’ recognition of the items on the REDFLAGS model as warning signs for mental distress (Research Question 4). A Bonferroni correction was applied. The first independent variable, gender, comprised two levels (1 = female or 2 = male). The second, ethnicity, comprised three levels (1 = African American, 2 = White, or 3 = other ethnicity). On the basis of the recommendations of Kaneshiro, Geling, Gellert, and Millar (2011), the third level of the ethnicity variable, “other ethnicity,” was aggregated from participants who did not identify as African American or White (21%, n = 69) to ensure that sample sizes were sufficient for making group comparisons. Disparities in the sample size of comparison groups are a common challenge in survey research on ethnicity. This statistical aggregation procedure is frequently used in survey research on ethnicity to provide comparison groups that are sufficient for statistical analysis (Kaneshiro et al., 2011). This procedure is appropriate for use in survey research as long as researchers are transparent about the limitations of the procedure. A significant main effect emerged for gender F(5, 318) = 13.83, p < .001, η2p = .05. Students who identified as female reported a higher awareness of the REDFLAGS model compared with male students (M = 0.16, SD = 0.70 vs. M = –0.20, SD = 0.79). A significant main effect also emerged for ethnicity, F(5, 318) = 6.23, p = .002, η2p = .038. Students who identified as White reported a higher awareness of the REDFLAGS model (M = 0.12, SD = 0.69) compared with students who identified as African American (M = –0.21, SD = 0.77).

Discussion

The results of the present study provided initial support for the reliability and validity of the REDFLAGS model. The single factor solution that emerged in the present study supports Kalkbrenner's (2016) conclusion that the eight items on the REDFLAGS model appear to be tapping into an overarching dimension of warning signs for mental distress. Investigators also found support for the predictive and discriminant validity of the REDFLAGS model. Present findings were consistent with previous investigations; gender emerged as a significant predictor of students’ peer‐to‐peer referral to the counseling center (Kalkbrenner & Hernández, 2017). The findings of the present study also extend previous investigations; the addition of participants’ awareness of the items on the REDFLAGS model as warning signs of mental distress into the second regression block significantly improved the predictability of the model. The substantial rise in odds suggests that increases in students’ recognition of the warning signs of mental distress on the REDFLAGS model might be a valuable strategy for increasing the odds of their making a peer referral to the counseling center.

The results of a two‐way ANOVA provided support for the discriminant validity of the REDFLAGS model. Previous researchers identified discriminant differences by gender and ethnicity on college students’ scores on latent variables related to mental health (Kalkbrenner & Hernández, 2017). In particular, Kalkbrenner and Hernández (2017) found students who identified as male were less likely than female students to support a peer in mental distress. Furthermore, Brownson et al. (2014) found that students who identified as White were more aware of warning signs for mental distress and more likely to seek counseling when compared with students who identified with minority ethnic backgrounds. Consistent with these previous investigations, we found significant differences between students’ scores on a latent variable related to mental health (composite scores on the REDFLAGS model). Specifically, in the present study, significant differences by students’ gender and ethnicity emerged in their ability to recognize the items on the REDFLAGS model as warning signs of mental distress. The effect size of this difference was in the moderate range on the basis of the recommendations of Sink and Mvududu (2010). This moderate effect size suggests that demographic differences accounted for an adequate amount of the variation in respondents’ recognition of the items on the REDFLAGS model as warning signs of mental distress. The findings of the present study have also extended these previous investigations by isolating this gender difference in students’ understanding of MHDs to the eight specific warning signs of mental distress on the REDFLAGS model. Taken together, the results of Phases 1 and 2 provide initial support for the reliability, construct validity, predictive validity, and discriminant validity of the REDFLAGS model. These findings have several implications for college counseling.

Implications for College Counseling

Education and consultation with counseling referral agents are major components in the contemporary practice of college counseling (Brunner et al., 2014). In particular, Brunner and colleagues estimated that approximately 40% of college counselors’ time is devoted to providing indirect mental health services (e.g., education and consultation). There are several existing resources in the college counseling literature to support college counselors’ indirect mental health services work (e.g., wallet cards provided by the NSPL; NSPL, 2007). The REDFLAGS model, however, has the potential to extend the college counseling knowledge base and might offer a novel tool for aiding college counselors with this outreach and consultation work.

The brief nature of the REDLFAGS model (eight items), coupled with the recognizable warning signs (acronym with the first letter of each warning sign composing the REDFLAGS), might make the model a unique contribution to the college counseling literature. In particular, the REDFLAGS model appears to be a potentially useful tool for college counseling practitioners because of its brief and user‐friendly nature. The model was designed to serve as an accessible, cost‐effective, and straightforward tool that can be used by college counselors to promote university community members’ ability to recognize students who might benefit from college counseling or other mental health services. The REDFLAGS model might be particularly unique with the potential to promote peer‐to‐peer referrals to the counseling center, which has become an integral role of college counseling practitioners (Brunner et al., 2014).

Peer‐to‐peer counseling referrals. The REDFLAGS model might be used as a tool for spreading awareness of the warning signs of MHDs on college campuses. It is recommended that college counselors consider the potential utility of distributing the REDFLAGS model on campus. In particular, the model can be provided to students via different delivery methods at little or no cost. These strategies might include sending an electronic version of the model to students via email or campus electronic mailing lists. The model can also be posted on the walls or bulletin boards in academic buildings, restrooms, student unions, and athletic facilities. In addition, college counselors might consider the feasibility of presenting the REDFLAGS model at new student orientations. The model can be printed on flyers or distributed to students electronically and accessed on a smartphone or a tablet. In addition, college counseling practitioners and staff “spend considerable time in classrooms” providing mental health awareness training to students (Brunner et al., 2014, p. 259). The REDFLAGS model might have utility for informing the training materials of college counselors (or peer mentors) for helping students recognize warning signs for mental distress.

The findings of the present study indicated that students who identified as male and African American might be less aware of the warning signs listed on the REDFLAGS model compared with their counterparts. College counselors might consider focusing mental health awareness education initiatives toward reaching populations of college students who might be unaware of warning signs for mental distress (e.g., students who identify as male or African American). Specifically, the REDFLAGS model can be posted in locations on campus that male students tend to frequent, including men's restrooms, the campus recreation center, fraternity homes, the men's locker room, and campus dormitories. It is also recommended that the circulation of the model is focused on reaching African American students. The model, for example, can be distributed in multicultural centers on campus. Also, White et al. (2009) provided evidence for the utility of resident advisors (RAs) in supporting college students’ mental and physical wellness. The REDFLAGS model may have utility both for training RAs to recognize warning signs of mental distress in residents and for spreading mental health awareness in campus residence halls. In particular, the model might be a useful tool that RAs can present at new resident orientations and post on bulletin boards in residence halls.

Peer health education. Peer health education is an effective method for supporting college students’ mental health (White et al., 2009). Peer health education initiatives involve student volunteers providing health education (physical and mental) to other students. Peer health education is a cost‐efficient and effective strategy for reaching a large number of students on campus (White et al., 2009). Specifically, peer health education initiatives are associated with reduced alcohol consumption and the promotion of healthy nutrition habits in college students (White et al., 2009). The process for implementing peer health education on campus is practical and straightforward as college counselors and their constituents train students who volunteer as peer educators in sessions that typically last for a few hours. The peer educators then teach other students about mental and physical wellness. The REDFLAGS model might be a valuable tool that can be added to the curriculum of peer health education initiatives. Peer educators can potentially provide brief explanations of the model and hand out copies to interested students. To maximize the efficiency of outreach efforts, college counselors might consider reaching out to students who identify as male or African American when first establishing peer health education initiatives. These efforts may have a dual benefit: They may both increase the awareness of warning signs for MHDs among more unaware populations of college students and establish peer health education initiatives on campus.

Limitations and Future Research

The results and implications of the current study do not come without limitations. The nonprobability sampling procedure may have limited the generalizability of the findings to other students and other universities. The temporal ordering of phenomena is also a limitation of the present study and of cross‐sectional, correlational predictive designs in general (Shalev, 2007). Specifically, it is possible that a third variable might be a more accurate predictor of students’ referrals to the counseling center (e.g., students’ prior knowledge of a mental illness, in general, might be a stronger predictor of their referrals to the counseling center). Future researchers should continue this line of research using a longitudinal, experimental design to determine the extent to which increasing participants’ awareness of the model over time causes an increase in their frequency of referrals to the counseling center. Furthermore, the odds ratio that emerged from the logistic regression analysis was substantial; however, the odds ratio in any logistic regression analysis should not be confused with increases in probability.

A statistical aggregation procedure was used with the ethnicity variable to conglomerate students who identified with ethnicities other than White or African American into a third “other ethnicity” group. This aggregation procedure is frequently used and a common limitation in survey research on ethnicity; it is used to ensure sample sizes that are sufficient for group comparison (Kaneshiro et al., 2011). However, this procedure may have limited the detection of group differences among students who identified with these other ethnicities. Future researchers should replicate the methodology of the present study with a more ethnically diverse sample. Future researchers should also investigate the validity of the REDFLAGS model with a variety of different populations of university community members. Faculty members, for example, might be valuable resources for recognizing and referring students to the counseling center (Kalkbrenner, 2016). Furthermore, because of their frequent contact with students, RAs are a promising resource for recognizing and referring students to the counseling center (Taub et al., 2013). Future researchers might test the efficacy of the REDFLAGS model as a resource for aiding RAs in recognizing and referring students who might be at risk for MHDs to the counseling center.

Summary and Conclusion

The aim of this study was to begin investigating the efficacy of the REDFLAGS model (with REDFLAGS being an acronym comprising eight cautionary signs of mental health distress in college students; Kalkbrenner, 2016). Results provided initial support for both the reliability and the construct validity of the model. In addition, we found support for the predictive validity of the model; students’ recognition of the REDFLAGS model was significantly associated with increases in the odds of having made a peer‐to‐peer referral to the counseling center. Results also supported the discriminant validity of the model. Consistent with previous investigators, we discovered demographic differences by gender and ethnicity in students’ recognition of the items on the REDFLAGS model as warning signs for mental distress. Implications for college counseling and future research have been provided. Although further research is needed, the REDFLAGS model appears to be a potentially useful resource for promoting college counselors’ outreach and consultation work with counseling referral agents. In particular, the model might be a promising resource for promoting peer‐to‐peer mental health support among college students.