A Pilot Study Testing the Clinical Congruency of Healthcare Providers to Use a Clinical Decision Support Tool to Assess Falls in Older Adults

Deanna Gray-Miceli ^1,*, Sarah J. Ratcliffe ²

1. Jefferson College of Nursing, Thomas Jefferson University, 901 Walnut Street, Suite 707, Philadelphia, PA 19104, USA

2. University of Virginia, School of Medicine, Senior Vice Chair for Research, and Director of the Division of Biostatistics, in the Department of Public Health Sciences, Charlottesville, VA, USA

* Correspondence: Deanna Gray-Miceli

Academic Editor: Giuseppe Ferdinando Colloca

Special Issue: Advances in Geropsychiatric Nursing

Received: January 30, 2024 | Accepted: February 28, 2025 | Published: March 10, 2025

OBM Geriatrics 2025, Volume 9, Issue 1, doi:10.21926/obm.geriatr.2501303

Recommended citation: Gray-Miceli D, Ratcliffe SJ. A Pilot Study Testing the Clinical Congruency of Healthcare Providers to Use a Clinical Decision Support Tool to Assess Falls in Older Adults. OBM Geriatrics 2025; 9(1): 303; doi:10.21926/obm.geriatr.2501303.

© 2025 by the authors. This is an open access article distributed under the conditions of the Creative Commons by Attribution License, which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is correctly cited.

1. Introduction

Falls are a common clinical problem managed by practitioners caring for older adults residing in long-term care facilities, especially in nursing homes, which have the highest rate of falls [1]. Injuries from falls account for a large percentage of potentially 48 preventable emergency department visits by NH residents [2]. As a public health problem of increasing magnitude, authorities note fall-related deaths and serious injury are likely to be averted when: a) recognized as an “adverse preventable event” with identifiable etiology; b) health care professionals critically evaluate fall events for likely causes and design appropriate interventions; and c) when healthcare facilities utilize existing clinical guidelines for fall prevention [3].

An integral component of any successful fall prevention program in NHs is healthcare professionals and clinical staff’s acceptance and utilization of evidence-based practices, i.e. Clinical Practice Guidelines [4] and national recommendations for fall prevention [5,6,7]. These guidelines require healthcare professionals to identify fall circumstance and recognize underlying cause(s) whenever possible [8]. Yet, the tools available for use by clinicians to identify underlying fall causes, i.e. fall risk scales and post-fall assessment tools, are either not comprehensive or haven’t been empirically tested, thus failing to provide the clinician with a consistent set of reliable and valid types of questions and examination parameters to substantially guide inquiry about a fall [9]. A best practice approach to fall prevention utilizes a comprehensive tool containing a standard set of validated questions deemed of value in determining potential causes of falls. Further, such a comprehensive post fall assessment tool is critical to carry out the necessary assessments recommended in the evidenced-based guidelines for the secondary prevention of falls among institutional-dwelling older adults and to guide decisions made about fall prevention care [10].

The absence of a comprehensive and empirically tested post-fall assessment tool for clinicians, specifically RNs, to use in practice spurred the development and validation of the PFI™, a valid and reliable comprehensive post fall assessment tool, reflective of evidenced-based guidelines for fall prevention [11]. The PFI™ adds value to the assessment by addressing the older adult’s perception of the event along with a fall-focused history and physical assessment thereby providing the RN with a rich set of comprehensive data for which clinical determination of the underlying cause of the fall is possible.

The purpose of this research study was to determine the clinical usefulness, and concurrent validity of the PFI™ when used by RNs, and other clinical staff such as advanced practice nurses (APNs) and physicians. Consideration of whether or not the PFI™ could be utilized and ultimately incorporated into the clinical protocol for fall prevention in a healthcare facility requires further inquiry and response from staff about its clinical usefulness (i.e. content is appropriate, tool is practical and feasible to use). Even if a tool is clinically useful to RNs, questions remain about the tools content and applicability to all healthcare professionals who assess older adult’s falls in NHs. When answers to the questions of efficacy have been empirically determined, clinicians can be confident that the assessment tool may be of value to their practice.

Three specific research questions guiding this study were to determine: 1) is the PFI™ feasible for use for collecting history and physical assessment data by the RN (feasibility defined as ‘can RNs use the PFI™’)?; 2) does the data collected within the PFI™ allow the RN to determine potential underlying causal event factors of the fall? and 3) does concurrent validity or congruency exist between various levels of clinical practitioners in determining potential underlying causal event factors of the fall? Human Subjects Protection was provided through Institutional Review Board approval.

2. Methods

2.1 Design

A quasi-experimental study using mixed qualitative and quantitative methodologies was selected to describe the clinical usefulness of the PFI™ by RN staff and to establish concurrent validity of the PFI™ among three levels of healthcare professionals.

2.2 The PFI™ Clinical Decision Support Tool

The PFI™ is a 30-item comprehensive post fall assessment, and clinical decision support tool developed by the PI according to national professional guidelines for falls prevention in older adults [5,6,10] for use by RNs in long-term care. Although lengthier than incident reports, the PFI™’s comprehensiveness was deemed of higher value to staff nurses. Large absolute agreement of items (70-100%) indicated good interrater reliability among RNs. Content validity was determined through two rounds of testing with national and clinical experts in falls prevention where a refined 30-item PFI™ emerged having at least 75% endorsement across all judges. Feasibility of using the PFI™ by RNs in practice in the nursing home was also established. Additional details of the psychometric properties of the PFI™ when used by RNs in practice is available [11]. The PFI™ was further studied as an evidence-based intervention, in a 3-year cohort study, prospectively, facility-wide for 1 year by registered nurses using a pretest-posttest design. A 29.4% reduction in the fall rate (z = 3.89, p < 0.001), 27.6% decline in total falls experienced by all fallers (p < 0.001), and a 34.0% decline for recurrent fallers (p = 0.025) from pre-intervention to intervention year was observed when trained nurses used the PFI™ to categorize falls according to perceived causes [12]. Data from this cohort study were further analyzed in an effort to determine the effect of the PFI™ on device and equipment costs for falls prevention [13]. Findings suggest potential cost savings to facilities which invest in the infrastructure of improving nursing care delivery [13].

2.3 Setting

The present study was conducted in two 120-bed skilled nursing units within two long-term care facilities located in the northeastern United States.

2.4 Sample

A total of three samples of subjects were included in this study for analysis; the primary participants, and two samples of secondary participants. The primary participants were older adult subjects who had fallen while residing on the skilled nursing unit. Data collected from these subjects resulted in narrative descriptions and clinical vignettes whereby concurrent validity determinations were sought by the secondary participants, i.e. expert judges.

2.5 Methodological Procedures

2.5.1 Recruitment and Enrollment of Primary Participants

Older adult residents, over age 65 years who had a recent fall (within 24 hours) were solicited for participation by the unit head nurse. Inclusion criteria included: being over the age of 65, not receiving hospice care or being deemed “terminally ill” and being determined to be stable medically following their fall. The older adult’s cognitive status post-fall, as determined by the head nurse was utilized to determine if the resident was capable of informed consent to participate in the research. For cognitively impaired residents, family caregivers were solicited for participation via the telephone and informed consent was obtained verbally and again in writing. Within twenty-four hours of the fall, informed consent was administered to the older adult resident by the principal investigator and the older adult was enrolled in the study. Study participants were assigned a random number and a sealed envelope-file was created containing a copy of the signed informed consent, a blank PFI™ and brief information about the date, time and location of the fall. Nursing staff employed by the facility carried out their usual post fall assessment using facility forms such as completing incident reports, writing progress notes and developing plans of care which were collected, but not seen by the RA’s.

A trained registered nurse research assistant (RA) completed the PFI™ within 48 hours of the enrolled subjects fall, placed it in the sealed envelope and dropped it in a mail slot of a locked administrative office. The RA and staff RNs did not communicate, unless the RA discovered a potential medical emergency requiring staff notification. The next morning the sealed envelope containing the completed PFI™ was collected by the PI. While the PFI™ was being administered to the primary participant, baseline demographic data from the resident’s medical record noting chronic and acute medical problems, functional status as determined by the Minimum Data Set (MDS), admission cognitive score and fall risk determination, medications, the most recent nursing plan of care and the incident report were collected on each subject enrolled by a second trained nurse RA. Data were collected from progress/interdisciplinary notes, up to 3 weeks prior to the fall, as well as medications administered up to 3 weeks prior to the fall, allowing for a narrative description of the fall event and the older adult who fell.

2.5.2 Narrative Descriptions Generated for Primary Participant’s Fall

Constructing a case vignette/narrative about the primary participant and their fall using the PFI™ was chosen as a suitable method for further data analysis as it most closely approximated the clinical case report that clinicians rely on to formulate their assessment of the person’s fall and a presumptive diagnosis. Data about the older adult’s fall inclusive of all information obtained from the PFI™ was compiled into a one to one-and-half page, typed clinical vignette by one of two experts, master’s prepared advanced practice nurse (APN) with specialty certification in geriatric nursing. The APNs compiled case vignettes following a standard medical report describing the falling individual, and summarized verbatim information extracted from the incident report completed by the staff RN. Maintenance of confidentiality and anonymity were followed by using fictitious surnames and assignment of a study number.

2.5.3 Recruitment and Enrollment of a Secondary Sample of Experts

A convenience sample of three independent expert clinical judges (a geriatrician, APN, and a staff RN practicing in long-term care) served as the secondary sample. Expert judges were solicited through an advertisement posted in an academic geriatric practice newsletter. Judges were recruited to independently review clinical vignettes and the accompanying completed PFI™ and then to independently, without training or aide, formulate possible clinical diagnosis about potential underlying causal event factors and cause(s) of a fall, prompted by an open ended question: “What is your diagnosis of the potential cause(s) of the fall”? Based on their diagnoses, judges were then asked to respond to an eight-item forced-choice selection of possible fall categories that best described the underlying classification/type of fall which occurred. We utilized, a priori, our non-hierarchical classification scheme for fall etiologies to further classify fall types [11]. Using this schematic, falls arose from one of eight potential underlying causes. Next, judges listed appropriate interventions linked to each of their diagnosis. Judges were supplied with de-identified clinical vignettes from each fall event that included a summary of the fall circumstance, a copy of the completed PFI™, along with a PFI™ rating sheet, and a sealed envelope. Judges were asked to complete 3 sets of 10 clinical narratives within 2-3 weeks and to place their responses in a sealed envelope for retrieval by the RA. After the first set of 10 were received, another set were reviewed following the same procedure, until all three sets were completed.

2.5.4 Recruitment and Enrollment of Secondary Sample of RNs

RNs who participated were identified by nursing administration as the primary care nurses who performed all of the post fall evaluations on the long-term care units. Prior to actually using the PFI™, the RNs were given the 30-item PFI™ and received some basic training. After review, they were given an opportunity to ask questions/discuss areas of concern. RNs were asked to complete the PFI™ within 24 hours of an older adults fall along with their customary fall assessment. RNs were asked to use the PFI™ for 1 month and then to complete an anonymous demographic questionnaire and reply to some open-ended questions related to their thoughts about the clinical usefulness of the PFI™. Specifically, RNs were asked the number of PFI’s™ completed, the length of time to complete, the overall rating of comprehensiveness, ease to use, comparison of the PFI™ to standard tools used at the facility, and the degree to which the PFI™ helped, if at all, to determine possible causes of falls and/or planning appropriate interventions. These questions speak to the clinical usefulness of the PFI™, and address research questions 1 and 2 guiding this study.

2.6 Statistical Analysis

Standard descriptive statistics were used to describe the sample. To test agreement among various levels of practitioners, and to answer research question 3, percent agreement was calculated among three independent judges who evaluated the same clinical case narratives/vignettes and then independently determined what they believed to be the underlying cause of the fall. Stata MP version 14 was used to perform logistic regression analysis.

3. Results

3.1 Primary Participants

The unit of analysis was an individual fall, and the study concluded when a sample of 30 falls was achieved. During the 104-day study period, a total of 103 falls occurred in the facility by 72 persons (for the quarter, the average daily census ranged between 102 and 107 patients). In the skilled nursing unit where the study took place, 23 primary participants experienced a total of fifty-three falls within the 104-day study period. Of these fifty-three falls, were eligible for inclusion, but 7 were discarded because of limited availability for the study RN to assess the older adult within 48 hours’ post-fall. The demographic characteristics of primary participants included 7 males and 16 females with an average age of 83.5 years (n = 23; 71-96 years of age). The mean cognitive impairment score, as determined by the Folstein Mini-Mental Examination was 17.3 (range 2-30) [14]. Participants had on average 9.8 chronic illnesses and 2.4 acute medical events prior to the fall, and took on average 9.1 medications. Most were continent of urine (n = 16; 53%) and walked unassisted (n = 20; 67%). Forty percent of the sample were independent in transferring or required the assistance of 1 person (n = 12; 40%), 3 percent (n = 1) were bedfast.

3.2 Secondary Sample of Expert Judges and RNs

The three clinical judges were all expert practitioners with many years’ experience working with older adults, specifically those who had fallen in NHs. They rated their ability to care for older adults as above average. The RNs who utilized the tool in practice as part of the determination of clinical usefulness, were experienced with educational preparation varying from diploma to associate degree preparation averaging seventeen years of experience in long-term care. They rated their ability to care for older adults who had fallen also as above average. Four RNs from two NHs used the PFI™ to evaluate 12 older adult patients.

Findings from this research answered the three specific research aims, these included:

Research Question 1: Is the PFI™ feasible to use for collecting history and physical assessment data by the RN? All RNs (n = 4) were able to utilize the PFI™ in their practice and rated it as superior to their current post-fall assessment tool used by their facility. RNs reported the PFI™ to be appropriate, practical and acceptable to use and it provided more useful and precise information in determining possible reasons for the fall. About three-fourths of the RNs perceived the information provided in assessing and planning care for the older adult useful, outweighing the time to administer. Use of the PFI™ averaged 45-60 minutes for first time users and 20-20 minutes the second time by the same rater.

Research Question 2: Does the data collected within the PFI™ allow the RN rater to determine a reason for the underlying causal event factors of the fall? In all cases reviewed by the RN rater (n = 30), RNs derived possible underlying causes for the fall based on the clinical vignette and review of the completed PFI™. Compared to other raters, the RN rater generated the most total number of diagnosis with 118 possible diagnoses, averaging 4.0 diagnosis per case. (refer to Table 1). The geriatrician generated the least number of diagnosis, totaling 63, with an average of 2 diagnoses per case, compared to the APN who generated a total of 90 diagnoses, with an average of 3 diagnoses per case. Table 1 further illustrates nine broad diagnostic categories that represent the types of diagnosis independently generated by RNs and other raters. The RN judged 70 percent (n = 21) of the fall cases reviewed to be due to chronic medical problems; 40 percent (n = 12) due to acute medical problems, 23 percent (n = 7) due to either safety reasons or medications; 13 percent due to environmental conditions, (n = 4), 10 percent due to behavioral issues and 7 percent due to equipment/device hazards (refer to Table 1). None of the 30 fall cases were judged by the RN rater to be due to happenstance.

Table 1 Descriptive summary of 30 falls as determined by three raters.

Research Question 3: Can three independent judges (a geriatrician, an advanced practice nurse and a registered nurse) agree on the underlying causal event factor and potential etiology of the fall? Table 2 presents the analysis used to determine diagnostic agreement between the geriatrician and the other judges. For each fall sub-category, agreement between the geriatrician and nurse judges was measured by the number and percentage of cases rated the same. Overall, agreement between the advanced practice nurse (APN) and the geriatrician judges ranged from 57-87 percent agreement (refer to Table 2). Agreement was high (>80%) in three categories; chronic medical conditions, equipment, and behavior.

Table 2 Diagnostic agreement between the physician and other raters for fall-related causal event factors/categories. Number (percentage) of the 30 cases where raters agreed is shown.

The range of diagnostic agreement between the geriatrician and the RN for all fall categories considered varied from as low as 33% agreement for safety reasons to 87% for behavior-related causes.

4. Discussion

This study presents the first attempt to take an empirically tested, comprehensive post-fall assessment tool derived from evidenced based guidelines for fall prevention in older adults and to test its clinical usefulness and feasibility by RNs to use in practice, along with its concurrent validity with various levels of expert judges. We found that when RNs used the PFI™ for one month in practice, all found it to be feasible, appropriate, practical and acceptable. Even though RNs reported it lengthier than their post-fall tools customarily used, the benefit of its content outweighed the length of time to use. However, the small number of RNs sampled precludes broader generalizability. Our findings confirm that the RN [as well as other raters] could independently, without guidance or support, determine fall-related diagnoses for each fall case reviewed, averaging 4.0 diagnoses per case, with a total of 118 diagnoses for 30 cases. The RNs ability to generate fall related diagnosis further speaks to the clinical usefulness of the content contained within the PFI™ (a main assumption guiding the first research question).

Overall, it was not surprising the RN generated the most number of diagnoses, a finding that is likely attributable to the amount of formidable education and training in clinical diagnosis determination. Because the RN possessed less formal training in clinical diagnosis formulation relative to fall diagnosis, it was expected that at least some of the time, they would have chosen happenstance or idiopathic reasons [no known reasons] as a causative explanation. However, the RN was consistent with other raters, who always generated or identified underlying causes for falling in each case. We attribute this finding to the RNs prior clinical expertise in fall care. Questions requiring further inquiry concern whether or not novice RNs with little formal training or experience in fall care could independently generate fall diagnoses and/or identify fall sub-categories, and would they be congruent to other novice raters?

To answer the third research question, we compared the diagnostic agreement between the geriatrician, to the RN judge and then to the APN judge. As evidenced in Table 2, the raw scores for percent agreement ranged from 33 percent to 86.6 percent with six out of the seven categories having above 50 percent agreement between the physician and the RN rater. Similar findings of congruency existed when the geriatrician rating for fall causes was compared to the APN rating, again with over 50 percent agreement in all categories. Of all causes identified by both the geriatrician and the APN, it was the diagnosis of chronic etiology that had the highest agreement and was statistically significant (p < 0.001).

Overall, analysis of data presented in Table 2 provides evidence that all three judges were congruent for many, but not all, sub-sets analyzed for falls, i.e., those due to: acute medical causes (p = 0.70); chronic medical causes (p = 0.29); medication causes (p = 0.50); and equipment/device failure (p = 0.012). For one sub-category, i.e., safety, there was a statistically significant difference observed between the RN and the geriatrician and advanced practice nurse [APN] judges (p < 0.01). It is conceivable that the RNs low response to the fall-safety category was influenced by a preconceived assumption that the NH environment is “safe and secure”.

Table 1 provides evidence in support of the third research question as there was 100 percent congruency among all raters for falls believed to be due to behavioral issues, as well there was total congruency that falls were not due to happenstance. In all of the 90 possible interpretations [each fall case vignette (n = 30) was reviewed three distinct times by three different levels of providers, generating 90 interpretations], no rater ever identified the cause of the fall to be due to “happenstance”. This finding validates the prevailing evidence and standard of practice approach that fall etiologies, in the elderly, occur from identifiable causes and tend to be multifactorial [7]. Collectively, the value of these findings supporting the third research question indicate that additional focused fall education programs with a menu-driven learning formats for specified fall causes may improve clinical diagnosis determination by RNs. The PFI™ may be also useful for broader application/utilization to both advance practice nurses and physician providers. In summary, the PFI™ is capable of helping health care provers distinguish among the actual antecedents that took place immediately prior to the fall event (a dynamic feature) in addition to known events retrieved through a medical record review of previously assessed risk factors (a static feature). While identification of fall risk is an essential standard of care for fall prevention, both before and after a fall, we point out the limitation of sole reliance on risk factor determination especially as comprehensive post-fall assessment tools are developed, tested and made available to support health care provider’s clinical decision making. The widespread availability of fall risk tools in NH settings, as opposed to standardized post-fall assessment tools further add to the confusion about these two distinct, but inter-related approaches to fall prevention. In previous research we have found when fall risk tools are substituted in place of a comprehensive post fall assessments, a common practice in long-term care, missing information and data occur [9]. Plans of care are ultimately affected which can influence the secondary prevention of additional falls. The research findings presented here add to the body of knowledge on fall prevention by underscoring the unique and distinct contribution of post-fall assessment in the clinical practice approach to fall prevention. As part of an effective communication strategy between clinical staff and healthcare providers, healthcare facilities should utilize a comprehensive post-fall assessment tool which is clinically useful to staff, (feasible), accurate (i.e., valid and reliable) and ideally, non-duplicative whenever possible. The finding that three independent raters could generate similar fall related categories reinforces a working assumption that clinical decision making (CDM) for identifying specific fall related causal event factors maybe obtainable by multiple level providers when the correct tools are utilized.

5. Study Limitations

One limitation of this study is that we do not have evidence that the identified fall causes by the judges were actually the real root cause of the fall. Further study, designed to test intra-rater reliability between clinical staff and fall experts who utilize the PFI™ following an actual fall, could help to determine this aspect of the PFI’s™ effectiveness. Another limitation of this study is that we do not know the exact mechanism by which CDM occurred by the RN judge or other raters, as CDM was not formally tested. In this study, CDM is considered the intuitive process that resulted in the outcome variable, the formulation of fall-specified diagnosis and identification of fall sub-categories. Previous research and literature report the influence of nursing experience [15], past knowledge of the patient, i.e., experiential knowledge [16,17], pattern recognition [18], and critical analysis of a comprehensive knowledge base as integral components of decision making [19], among other factors. In our research reported here, many of the culminating diagnosis i.e., product of the rater’s intuitive CDM, were congruent even when the factors of experience and experiential knowledge were constant among raters. We can assert this because we took many steps to de-identified and sanitize the data and to hire independent experts to develop case vignettes so that none of the three raters had previous knowledge of the patients described in the clinical vignettes [i.e. experiential knowledge]. All judges had extensive prior experience in fall care [i.e. they were not novices] and held administrative or clinical positions requiring autonomous and independent CDM. In hypothesizing about the CDM process that did take place, it is plausible that the content within the PFI™ and the presentation of materials was sufficiently comprehensive so as to foster critical thinking and/or clinical pattern recognition.

Because the case vignettes presented narrative clinical information similarly to a case report, it is possible that our experienced rater(s) could identify with the case vignettes because it resembled a priori accounts of other patients brought to conscious recall. For this reason, the raters may have actively been utilizing clinical pattern recognition to formulate their CDM. Researchers have identified clinical pattern recognition to involve a comparison and analysis between prior patients’ and current patient histories [15,20]. Our research findings appear to elucidate the linkage described by O’Neill, Dluhy & Chin (2005) in their conceptual model, “Clinical Decision-Making and Novice Clinical Reasoning Model” [18]. In their model, hypothesis generation for novices occur from clinical experiences, development of working knowledge with organized clinical patterns and recognition of salient cues. Further study looking at the mechanisms by which novice RNs without an advantage of prior experience actually formulate their clinical decisions about fall-care when using a comprehensive tool such as the PFI™ is warranted.

6. Conclusion

In response to the three study objectives, the PFI™ was found feasible for use for collecting history and physical assessment data by the RN, and the data collected within the PFI™ allowed the RN to determine potential underlying causal event factors of the fall. Lastly, concurrent validity or congruency was evident on some fall sub-categories between various levels of clinical practitioners in determining potential underlying causal event factors of the fall.

Additionally, knowledge generated from this study emphasizes that categories of falls can be identified by those with prior clinical expertise in fall care. Specifically demonstrated by the clinical judges is the finding that falls can be conceptualized according to various sub-types, when they (fall) are considered not solely as a monolithic term, but rather as a categorical event. From an epidemiological position, surveillance data for fall and injury prevention can be better understood and health policy ultimately streamlined when falls are identified categorically according to underlying causal event factors. Clinically, description of falls using such a schematic or framework of categories paves the way for further individualized tailored interventions to prevent falls.

Author Contributions

Drs. Gray-Miceli and Ratcliffe were responsible for the study design, data collection methods, analysis, and discussion.

Funding

Research was funded in part from a grant to the PI by Sigma Theta Tau, Xi Chapter, University of Pennsylvania School of Nursing and a Post-Doctoral Scholarship from The John A. Hartford Building Academic Geriatric Nursing Capacity Program (2002-2004), American Academy of Nursing.

Competing Interests

The authors have declared that no competing interests exist.