Predicting Suicide Risk among South African University Students: Optimal Cutoff Score for the Center for Epidemiological Studies Depression Scale

Tyrone Pretorius ^†,*, Anita Padmanabhanunni ^†

1. Department of Psychology, University of the Western Cape, Robert Sobukwe Road, Cape Town, South Africa

† These authors contributed equally to this work.

* Correspondence: Tyrone Pretorius

Academic Editor: Luigi De Gennaro

Special Issue: Mental illness During the COVID-19 Pandemic in Vulnerable Population Groups

Received: April 28, 2023 | Accepted: June 28, 2023 | Published: July 03, 2023

OBM Neurobiology 2023, Volume 7, Issue 3, doi:10.21926/obm.neurobiol.2303173

Recommended citation: Pretorius T, Padmanabhanunni A. Predicting Suicide Risk among South African University Students: Optimal Cutoff Score for the Center for Epidemiological Studies Depression Scale. OBM Neurobiology 2023; 7(3): 173; doi:10.21926/obm.neurobiol.2303173.

© 2023 by the authors. This is an open access article distributed under the conditions of the Creative Commons by Attribution License, which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is correctly cited.

1. Introduction

The COVID-19 disease outbreak was a significant psychological stressor. The unpredictable nature of the pandemic, high mortality rates from infection and the implementation of measures globally to curb its spread contributed to significant distress and adverse mental health outcomes, particularly among vulnerable population groups [1]. COVID-19 prevention measures including national lockdowns, restrictions on movement, and closures of non-essential services, schools and higher educational institutions contributed to job and food insecurity and unemployment [2]. These stressors can contribute to significant psychological distress. Those with pre-existing psychiatric disorders were reported to experience increased symptomology. In contrast, others were at heightened risk for developing mental health disorders such as depression, anxiety and post-traumatic stress disorder (PTSD) [3]. Individuals who contracted COVID-19 and their families faced social stigma and isolation, which can aggravate stress [1]. Existing research has confirmed that stress-related psychiatric disorders such as depression, substance use and PTSD are associated with an increased risk of suicidal behavior [4]. There have also been multiple reported [1,5] incidents of COVID-19-related stressors precipitating suicidal behavior and completed suicide. In a meta-analysis investigating COVID-19-related suicide in 54 studies that included data from over 300 000 people, Dubé and Colleagues [4] found elevated rates of suicidal ideation, suicidal behavior and self-harm compared to pre-pandemic studies.

University students have been identified as a particularly vulnerable group for adverse mental health outcomes owing to the distinctive implications of the COVID-19 prevention measures on the educational sector. The closing of institutions of higher learning, migration to online learning environments and lack of access to academic and peer supports has been reported to have markedly elevated stress levels among this population group and contributed to psychological distress [6]. Furthermore, even prior to the pandemic, university students were identified as being at increased risk for adverse mental health outcomes and suicidality [7]. This has been ascribed to the many challenges associated with the transition from adolescence to early adulthood, including diminished parental guidance and support, negotiating a new academic environment and associated pressures, and increased opportunities for risky behavior (e.g., alcohol and drug use) [8]. A cross-sectional study undertaken among Ghanaian students [9] reported prevalence rates for suicidal ideation (15.2%), attempted suicide (6.3%), death wishes (24.3%) and suicidal intention (6.8%). Psychological distress was identified as a salient predictor of suicidality. Using data from a national health survey for higher education in Norway, Sivertsen and Colleagues [8] reported prevalence rates for suicidal thoughts (21.0%, and 7.2% within the past year) and suicide attempts (4.2%). A population-based prevalence survey in Scotland [10] demonstrated that 11% of adults reported a suicide attempt at some stage in their lives and 16% indicated that they had engaged in non-suicidal self-injury.

Although various factors can enhance the risk of suicide, one of the strongest predictors of suicidality is the presence of depressive symptomology [11]. Furthermore, the severity and variability of depressive symptoms have been reported as the only salient indicator of suicide attempts [12,13] among vulnerable individuals. Hence, identifying symptom characteristics that are risk factors for suicidality can facilitate a more accurate risk assessment and allow for the targeting of interventions.

The Center for Epidemiologic Studies Depression Scale (CES-D) is one of the most frequently used self-report instruments for depression [13,14]. The CES-D is a 20-item instrument developed to screen for depression, to assess for the presence of depressive symptoms and to detect vulnerability of having a depressive disorder [15]. It assesses four components namely depressed affect, positive affect, somatic complaints, and interpersonal relations [15]. Total scores can range from 0 to 60. Radloff [15] proposed that people who score 16 and above on the CES-D are likely to be clinically depressed but other studies [14,16] have suggested that the cut-point needs to be above 20 if the instrument is to be an effective diagnostic tool. Using a meta-analytic approach involving a review of 28 studies using the CES-D, Vilagut and Colleagues [17] proposed an optimal cutoff score of 20 with a specificity of 0.78, sensitivity of 0.83, and diagnostic odds ratio of 16.64.

The CES-D has demonstrated good reliability and validity for assessing depression in various population groups including university students [14], older adults [18] as well as in children and adolescents [19]. It has also been reported to predict the risk of suicide. For example, a Chinese study of suicide attempters [20] found that CES-D scores were significantly higher in suicide attempters and that depression was associated with attempted suicide. A study involving a community sample of adults [21] reported that higher scores on the CES-D predicted greater suicidal thoughts and suicidal behavior. Lu and Colleagues [13] found that depressed affect, as measured by the CES-D, was a significant predictor of suicidal behavior among a cohort of adolescents in the United States. These authors [13] also reported that two CES-D items namely “I felt sad” and “I felt lonely” independently predicted suicidal ideation. In a study of community-dwelling adults, St John and Colleagues [22] found that a single item of the CES-D (i.e., “I felt depressed”) predicted suicide and this association persisted even after controlling for age, gender and educational status.

The current study aims to extend this research by using receiver operating characteristic (ROC) analysis to determine the optimal cutoff score on the CES-D when screening for suicide risk in university students during COVID-19 in South Africa. The South African government implemented one of the strictest lockdowns following the disease outbreak in 2020. Universities were required to migrate to online learning and teaching and students were compelled to leave campus residences and return home [6]. This placed considerable pressure on students particularly those who did not have access to digital technology or resided in underprivileged settings with minimal resources. Uncertainty about their academic future, feelings of hopelessness, and concerns about their family's well-being may have aggravated student distress [6]. Although the lockdown eased in 2021 and 2022, Universities continued to use hybrid modes of engagement owing to concerns about contagion.

ROC provides the most comprehensive description of diagnostic accuracy. Diagnostic precision was measured and reported for decades based on the number of correct diagnostic decisions [23]. This percentage-correct measure is disadvantageous as it does not reveal the relative frequencies of false-positive and false-negative errors. These errors have different diagnostic and clinical consequences [24]. ROC has the benefit of being able to estimate and report all of the combinations of specificity and sensitivity that a diagnostic scale can provide [23]. Sensitivity refers to the fraction of individuals with the condition (e.g., clinical depression) in question and who are correctly diagnosed as positive. In contrast, specificity refers to the fraction of individuals without the condition who are correctly diagnosed as negative [24]. ROC indices account for false-positive and false-negative diagnoses implicitly, and a change in the condition's prevalence does not affect the numerical values. ROC reports all of the combinations of sensitivity and specificity that a diagnostic test can provide [24].

2. Materials and Methods

2.1 Participants and Procedure

A cross-sectional research design was adopted in this study. Participants (N = 337) were randomly sampled (5% margin of error, 95% confidence interval) young adults enrolled in undergraduate studies at a Western Cape, South Africa university. We used Google Forms to create an electronic version of the questionnaires used in the study. The electronic link together with the description of the study, as well as an invitation to participate were sent to 1500 randomly selected students. The response rate therefore was 22.5%. The study was conducted during the first wave of COVID-19 in 2020. Most of the sample were women (77.2%) and resided in an urban area (75.4%). The mean age of the sample was 21.95 years (SD = 4.68).

Ethical approval for this study was obtained from the Humanities and Social Sciences Research Committee of the university (Ethics reference number: HS20/5/1). The survey was completed anonymously, and all the participants provided informed consent prior to accessing the survey. The participants were also provided with the contact details of the South African Depression and Anxiety Group and the Centre for Student Support Services at the university if they experienced any psychological distress after completing the questionnaire.

2.2 Instruments

Participants completed the CES-D and the Beck Hopelessness Scale (BHS) [25]. This study was part of a bigger study focusing on the mental health of students; therefore, only limited demographic information was collected.

The CES-D is a 20-item measure of depressive symptoms that is scored on a four-point scale ranging from “rarely or none of the time” (0) to “most or all of the time” (3). An example item of the CES-D is “I had trouble keeping my mind on what I was doing." Radloff [15] provided evidence of reliability across different population groups (Cronbach’s alpha = 0.84 to 0.90), and of validity e.g., the correlation between scores on the CES-D and clinical ratings of depression. Pretorius [26] examined the applicability of the CES-D in a sample of South African students. They reported a reliability coefficient of 0.90 and concerning validity, found that scores on the CES-D were significantly related to a measure of life change. The CES-D was also used with a sample of schoolteachers in South Africa and satisfactory reliability coefficients were reported in that study (alpha = 0.92, omega = 0.93) [27].

The BHS is a 20-item measure of hopelessness and pessimism about the future. Participants respond to the 20 items on a dichotomous scale of true/false. An example of an item of the BHS is “the future seems vague and uncertain to me." Beck reported a reliability coefficient (K-R) of 0.93 for the BHS when used in a sample of hospitalized patients who had attempted suicide [25]. In addition, correlations between the BHS and clinical ratings of hopelessness and other self-report measures of hopelessness provided evidence for the scale's validity. In South Africa, the BHS has been used with young adults [2] and schoolteachers [28], and the reliability coefficients in these two samples were 0.88 and 0.89, respectively.

2.3 Data Analysis

IBM SPSS for Windows version 28 was used for the data analysis. This included descriptive statistics (means and standard deviations), reliabilities (alpha and omega) and correlation between depression and hopelessness (Pearson r). ROC analysis was used to determine the optimal cutoff point on the CES-D for identifying suicide risk. The literature has generally identified a cutoff score of 9 on the BHS as predictive of suicide risk. A meta-analytic review identified 9 as a standard cutoff point in several studies [29]. In the ROC analysis the area under the curve (AUC) provides an estimate of the diagnostic efficiency of the CES-D. Generally, the following guidelines can be used to evaluate AUC: <0.70 poor, 0.70-0.79 fair; 0.80-0.89 good, and 0.90-1.00 excellent [30]. The ROC analysis also provides sensitivity and specificity indices for every cutoff score. Sensitivity, in this instance, would refer to the probability of correctly identifying those who are at a high risk of suicide (true positives), while not classifying those at risk as not being at risk (false negatives). Specificity, on the other hand, is the probability of correctly identifying all those who are not at risk of suicide (true negatives), while not classifying those that are not at risk as being at risk (false positives) [31]. In this study, we decided to optimize sensitivity rather than specificity based on Youngstrom’s [32] view that high sensitivity is more important than specificity with screening tests to avoid missing cases at risk. For diagnostic tests specificity would be more important. Before the ROC analysis of the whole sample, we compared the AUC of men and women to determine if a single AUC can represent the whole sample. SPSS also indicates overall model quality when groups are being compared. If the overall model quality is above 0.5 it indicates good predictive ability. We also used Cohen’s Kappa to measure accuracy, obtained using an Excel worksheet [32]. In general, a Kappa above 0.2 can be regarded as fair. Lastly, we used Excel to obtain positive and negative predictive values for every cutoff score. Positive predictive values (PPV) refer to the probability that someone who is identified as at risk of suicide is actually at risk, while negative predictive values (NPV) refer to the probability that someone who is identified as not at risk is not at risk.

3. Results

The mean score for the CES-D was 27.5 (SD = 13.36, range = 0-57) and for the BHS it was 4.7 (SD = 4.4, range = 0-20). The two instruments demonstrated satisfactory reliability (CES-D = 0.92; BHS = 0.88). There was a substantial positive association between depression and hopelessness (r = 0.56, p < 0.001), indicating that high levels of depression are associated with high levels of hopelessness.

The ROC curve for comparing men and women is shown in Figure 1.

Figure 1 ROC curves for men and women. The blue curve is men; the green curve is women.

The AUC for men was 0.80 (SE = 0.06, 95% CI: 0.68, 0.91) and for women was 0.81 (SE = 0.04, 95% CI: 0.74, 0.88) reflecting a good diagnostic efficiency for the CES-D for both men and women. There was no statistical difference between AUC’s of men and women (z = −0.28, p = 0.78). The overall model quality for men and women is shown in Figure 2.

Figure 2 Overall model quality for men and women (men: gender = 1; women: gender = 2).

Figure 2 indicates that the diagnostic efficiency of the CES-D for women was marginally better than for men. However, the model quality was above 0.5 for both.

The ROC curve for the whole sample is presented in Figure 3.

Figure 3 ROC curve predicting suicide risk based on the CES-D.

The AUC for the total sample when the CES-D was used to screen for suicide risk was 0.81 (SE = 0.03, 95% CI: 0.75, 0.87), which in terms of the guidelines reflects a good diagnostic efficiency for the CES-D in terms of screening for suicide risk. The results of the ROC analysis, together with the indices of accuracy (Cohen’s Kappa) and the predictive values (PPV and NPV) are reported in Table 1.

Table 1 Results of ROC analysis, a measure of accuracy and predictive values.

Table 1 reflects that a cutoff score of 30 with a sensitivity of 89.30 and a specificity of 62.60 and an acceptable Kappa of 0.304 represents an acceptable cutoff point. Thus, a cutoff score of 30 would have an 89.3% probability of correctly identifying true positives and false negatives, and a 62.6% probability of correctly identifying true negatives and false positives. The contingency table in BOX 1 reflects the number of true and false positives and negatives based on a cutoff of 30.

Box 1 Contingency table based on CES-D cutoff score of 30.

4. Discussion

The COVID-19 pandemic significantly impacted the mental health of university students [2]. Existing studies documented elevated levels of psychological distress arising from pandemic-related stressors including the closing of universities, the transition to remote digital learning and disruptions in access to academic and peer support networks [33]. Students also had to contend with the economic uncertainty produced by the pandemic and issues related to job and food security [33]. The current study was conducted at a historically disadvantaged institution (HDI) in South Africa. HDIs were created during the apartheid era as part of segregationist policies of differential education for different race groups and were significantly under-resourced [34]. Although much has changed since the transition to democracy in the country, students at HDIs are predominantly from previously marginalized groups and reside in disadvantaged community contexts characterized by poverty, unemployment, gang violence and increased rates of substance use [34]. These contextual factors can amplify pandemic-related stress and precipitate negative psychological outcomes [34,35]. Elevated levels of depressive symptomology among university students have been consistently identified as a significant mental health consequence of the pandemic and its prevention measures [8,9,36]. Depression confers a significantly greater risk for suicidal ideation and suicide attempts and for completed suicide [11,21]. For this reason, measures of depression must be able to accurately diagnose symptomology and thereby identify risk. The CES-D is a frequently used measure of depression severity and has been utilized in both research settings and by clinicians to screen for depressive symptomology [37]. The instrument has demonstrated sound psychometric properties and has been extensively used among different populations [14,18]. However, the classic cutoff score proposed by the developer in 1977 [15] has yielded many false positives [37]. There is also limited evidence to support the original cutoff score of 16 and outdated cutoff scores can enhance the risk of false positives and false negatives, which can significantly limit the instrument's utility as a screening tool [37,38].

The current study aimed to extend research on the diagnostic accuracy of the CES-D when screening for suicide risk by using ROC analysis to determine the optimal cutoff score on the instrument among a sample of South African university students. The study found that a cutoff score of 30 with a sensitivity of 89.30 and a specificity of 62.60 represented an acceptable cutoff point. This finding contrasts with existing studies that have used ROC analysis. For example, Campo-Arias and Colleagues [38] assessed the psychometric properties of the CES-D among Columbian adults using ROC analysis. They reported that a score of 20 represented this cut-off point, with a sensitivity of 0.96, specificity of 0.73. A study [39] investigating the utility of the CES-D as a depression screening tool among low-income women attending primary care clinics reported a cutoff score 16, which yielded a sensitivity of 0.95 and specificity of 0.70. Quiñones and Colleagues [40] reported that higher cut points for the CES-D performed better in correctly identifying true negatives and true positives for major depressive disorder among veterans. These authors [40] reported a cutoff of 18 with a sensitivity of 92% specificity and 72% among a sample of United States veterans. A study [37] aimed at determining the optimal cutoff on the CES-D for depression in a community corrections sample reported an optimal cutoff of 23 with a sensitivity of 78.2% and specificity of 79.8%. These differences in optimal cutoff points reinforce arguments [37,40] that the cutoff for measures of mental health outcomes must be determined for specific populations rather than relying on a universal cutoff point for screening a particular population. This is due to the possibility that measurement scales assess different constructs among diverse population groups. While previous studies have investigated cutoff points for clinical depression, this study differs from previous research as it focuses on the utility of CES-D cutoff points in screening for suicide risk. Although the CES-D has demonstrated excellent discrimination between individuals who experience depression and are at greater risk of suicidality, screening scales are essentially rough measures of psychiatric disorders. Therefore, further clinical evaluation and assessment is needed to confirm the accuracy of scores.

The present study is limited by a cross-sectional design, which impacts the generalizability of our findings and conclusions about causality. Although we conjecture that increased psychological distress may be due to COVID-19-related factors, other variables could probably have contributed to this. The study was also undertaken in one geographic area and among a student sample from one institution. The gender distribution of participants was not proportional and the sample consisted predominantly of women but women are generally over-represented in the higher education sector. Existing studies [41] suggest that women are more vulnerable to depression, which can impact the study findings. Future research using a longitudinal design and a more diverse sample is needed to confirm our results. In addition, the study used a self-report method, and measurement and social bias are possible, which can impact the results. Using a triangulation design involving clinician-administered standardized interviews and self-report measures may yield greater accuracy in diagnosis. Despite these limitations, the study has important implications. It highlights the potential need for cutoff points on measures of mental health disorders to be population specific to enhance the accuracy of diagnostic decision-making, the early identification of risk and the timely implementation of interventions.

5. Conclusions

In conclusion, the CES-D was demonstrated to be a valid screening tool for use among a student population at a historically disadvantaged South African university. A cutoff score of 30 was the most fitting for predicting suicide risk. It is recommended that when the instrument is used in clinical settings and among student populations, those with scores of 30 and above need to be followed up with a clinician-administered interview to confirm risk and facilitate timely intervention. Overall, the CES-D has the potential to identify those at risk of suicide and can provide an important first step in intervention efforts. It can also potentially be used as an outcome measure to assess whether interventions that seek to reduce the risk of suicide are effective.

Author Contributions

Conceptualization and methodology: Anita Padmanabhanunni & Tyrone Pretorius; Analysis and data curation: Tyrone Pretorius. All authors discussed and interpreted obtained results. All authors contributed to the writing and editing of the manuscript over several iterations.

Competing Interests

The authors have declared that no competing interests exist.