Vignette

On a beautiful early September morning in southeastern Minnesota a 56 year old female arrives to the emergency department with vague complaints of chest discomfort and fatigue. She was diagnosed with hypertension 5-years ago, but has no history of diabetes, cardiovascular or cerebrovascular disease, venous thromboembolism, malignancy, pneumothorax, anemia, or renal disease.  She walks her dog every day and denies any recent exertional dyspnea or chest pain, though when queried she does note decreased exercise tolerance as she can only walk a mile now compared with 3 miles when she is feeling well.  Her electrocardiogram demonstrates non-specific T-wave flattening in the inferior and lateral leads compared with a tracing 15 years ago and her initial high sensitivity troponin is 30.  You decide to use the HEART Pathway (https://www.mdcalc.com/calc/3975/heart-pathway-early-discharge-acute-chest-pain) and await a second troponin while reviewing the accuracy of history and physical exam for potential acute coronary syndrome (ACS, see http://pmid.us/9786377) and the role of coronary computed tomography angiography (cCTA – see https://emergencymedicine.wustl.edu/items/ct-coronary-angiography-test-characteristics-cost-effectiveness-to-exclude-cad/). Her second hs-Trop is 38 so you use PubMed and the Translating Research into Practice (TRIP) database with the following PICO question to better understand and communicate the tiers of evidence (see the Diagnostic Hierarchy of Evidence image below) for cCTA for risk-stratifying ACS and patient-centered outcomes (https://onlinelibrary.wiley.com/toc/15532712/2015/22/12).

Patient – adults in the emergency department with suspected acute coronary syndrome

Intervention – coronary CT angiography during ED evaluation

Control – clinical gestalt, HEART score/pathway, or other non-imaging disposition strategy

Outcomes – 30-day major adverse cardiac event rate, cardiology consultation rates, emergency department returns, rates of cardiac catheterization and revascularization.

You use PubMed Clinical Queries diagnosis and broad search with the term “acute coronary syndrome” and then combine those 20,000+ documents with the PubMed search terms“emergency*”, and “coronary computed tomography angiography” to narrow the search to 432 documents (https://tinyurl.com/y5mzb39u).

Based upon these findings, you identify multiple systematic reviews over the last 15 years along with consensus recommendations from the Society of Cardiovascular Computed Tomography, American College of Radiology, and North American Society for Cardiovascular Imaging.  You also find multiple published study protocols that are apparently not yet completed, so you select which publications to review guided by the Hierarchy of Diagnostic Evidence (below).

PGY-1’s:  Impact of HEART Score Decision Aid on Coronary Computed Tomography Angiography Utilization and Diagnostic Yield in the Emergency Department

PGY-2’s:  Early Computed Tomography Coronary Angiography in Adults Presenting with Suspected Acute Coronary Syndrome:  The RAPID-CTCA RCT

PGY-3’s:  Comparative Effectiveness of Coronary CT Angiography and Standard of Care for Evaluating Acute Chest Pain:  A Living Systematic Review and Meta-Analysis

Evidence Based Medicine Hierarchy of Evidence for Diagnostics

PGY-1: Critical Review Form - Clinical Prediction or Decision Rule

Impact of HEART Score on Coronary CTA Use in ED, Crit Pathways Cardiology 2023; 22: 45-49

Objective: “To test the hypothesis that implementation of the HEART score-based decision aid (HSDA) would reduce coronary computed tomography angiography (CCTA) utilization and increase diagnostic yield in emergency department (ED) patients with chest pain.” (p. 46)
Methods: Retrospective before (January 1 thru June 30, 2018) and after (January 1 thru June 30, 2020) study at Stony Brook University (New York) Department of Emergency Medicine to quantify the impact of forcing emergency medicine clinicians to enter the components of the HEART score (below) in the electronic health record (EHR) in the “after” period before ordering CCTA and actively discouraging CCTA for low-risk HEART score patients as defined by HEART score ≤ 3.

Exclusion criteria included poor quality CCTA or coronary anomalies not secondary to atherosclerosis, ST-elevation myocardial infarction (STEMI), initial troponin elevation, or glomerular filtration rate less than 30 mL/minute.

This research followed an earlier effort prior to 2018 to reduce CCTA ordering at the same institution by “educating” clinicians about the HEART score but not incorporating it into the EHR, which had not decreased CCTA ordering rates. In 2019, a multi-disciplinary team from Emergency Medicine, Radiology, and Information Technology collaborated to introduce the HEART score-based decision aid (HSDA) into the EHR that required ED clinicians to enter the components of the HEART score into the EHR and the score was automatically computed before the CCTA could be ordered. Clinicians were discouraged from ordering CCTA for low-risk HEART score patients, but that discouragement could be overridden.
CTs were interpreted by board-certified Radiologists training in cardiac imaging. The presence of obstructive coronary artery disease (CAD) was defined by plaque with ≥ 50% stenosis in one (or more) coronary artery. Data were abstracted “automatically” from the EHR and supplemented by manual chart review methods adherent to established Emergency Medicine methodology.

The primary outcome was the percentage of chest pain patients undergoing CCTA and those with obstructive CAD identified. Secondary outcomes were the sensitivity and specificity of the HEART score in predicting obstructive CAD. The percentages of chest pain patients undergoing CCTA pre- and post- the EHR intervention were compared with the χ2 test. The HEART score was dichotomized as low (0-3) or moderate-to-high risk (≥4) for computing sensitivity and specificity.

Guide		Comments
I.	Is this a newly derived instrument (level IV)?
A.	Was validation restricted to the retrospective use of statistical techniques on the original database? (If so, this is a Level IV rule & is not ready for clinical application).	This is not strictly a validation of the HEART score, but uses a different group of patients than the original HEART score derivation and subsequent validation studies (Backus 2010, Backus 2013, Backus 2013, and numerous others).
II.	Has the instrument been validated? (Level II or III). If so, consider the following:
1a	Were all important predictors included in the derivation process?	Yes, including clinician gestalt as a portion of the History.
1b	Were all important predictors present in significant proportion of the study population?	Uncertain since the prevalence of individual components of the HEART score for this population are not reported in this manuscript.
1c	Does the rule make clinical sense?	Yes. The HEART score includes a combination of known historical risk factors, ECG findings, age, cardiac enzymes, and clinical suspicion

2	Did validation include prospective studies on several different populations from that used to derive it (II) or was it restricted to a single population (III)?	The HEART score is at least a Level II clinical decision instrument based on validation studies across geographic settings as detailed in multiple systematic reviews (Van Den Berg 2018, Fernando 2019, Laureano-Phillips 2019). In addition, the HEART score has been deemed an acceptable risk stratification instrument in ACEP Clinical Policy (Tomaszewski 2018) and SAEM GRACE clinical practice guidelines (Musey 2022).
3	How well did the validation study meet the following criteria?
3a	Did the patients represent a wide spectrum of severity of disease?	The investigators dichotomized the HEART score into low-risk (£ 3) or moderate + high-risk (>3). In the 2018 cohort 69.5% (477/686) who underwent CCTA were low risk compared with 28.4% (93/328) in the 2020 cohort. Of those among the low-risk HEART score in 2020, 70% (65/93) had HEART score 3, 21% (20/93) HEART score 2, 7% (7/93) HEART score 1, and 2% (2/93) HEART score 0. Therefore, the 2018 cohort were skewed towards low-risk and the 2020 cohort towards moderate- to high-risk.

3b	Was there a blinded assessment of the gold standard?	The diagnostic gold standard was anatomic with CCTA evidence of obstructive CAD “interpreted by board-certified radiologists trained in cardiac imaging” (p 46). No clear blinding of radiologists to the HEART score or other clinical information is reported. The predictive gold standard was chart review to ascertain whether a major adverse cardiac event (myocardial infarction, death, or revascularization) occurred in the 30 days following ED presentation. Although authors state adherence to Kaji 2014 chart review methods (p 46), no details regarding blinding of chart abstractors to the study hypothesis or reliability of data abstracted is reported.
3c	Was there an explicit and accurate interpretation of the predictor variables & the actual rule without knowledge of the outcome?	Unknown in the 2018 cohort. Yes in the 2020 cohort (but unknown in the 2018 cohort) because in the 2020 cohort before ordering CCTA “practitioners were required to enter the components of the HEART score with automatic calculation of the score. Practitioners were then required to acknowledge the HEART score risk category (low risk, 0-3; moderate risk 4-6; and high risk ³7) of the patient before confirming the order.” (p 46) Uncertain to accurate because authors “did not independently verify or validate the reliability of HEART scores entered into the computer at the time of order the CCTA. It is possible that providers may have overestimated HEART scores to avoid overriding the computer when order CCTA in low-risk chest pain patients.” (p 48)

3d	Did the results of the assessment of the variables or of the rule influence the decision to perform the gold standard?	Yes, by design the HEART score was linked to CCTA ordering. This is differential verification bias which ­ sensitivity & ­specificity for diseases that resolve spontaneously, but ↓sensitivity & ↓specificity for diseases that only become apparent on follow-up
4	How powerful is the rule (in terms of sensitivity & specificity; likelihood ratios; proportions with alternative outcomes; or relative risks or absolute outcome rates)?	From Table 5 on p 48 we can calculate likelihood ratios using https://statpages.info/ctab2x2.html 2018 Cohort (see chart below) 2020 Cohort (see chart below)

III.	Has an impact analysis demonstrated change in clinical behavior or patient outcomes as a result of using the instrument? (Level I). If so, consider the following:
1	How well did the study guard against bias in terms of differences at the start (concealed randomization, adjustment in analysis) or as the study proceeded (blinding, co-intervention, loss to follow-up)?	This was a before/after design (rather than randomized controlled trial or post-hoc adjusted analysis), so multiple potential confounders at the level of the patient, emergency clinician, and radiologists. Patient level confounders include baseline risk for obstructing CAD or 30-day MACE. As noted in 3a above, the 2018 cohort were skewed towards low-risk and the 2020 cohort towards moderate- to high-risk, so theoretically the 2020 cohort would have higher 3-day MACE event rates. Among the 93 patients in the low-risk HEART score subset that underwent CCTA, 12 had obstructive CAD on CCTA and none suffered a 30-day MACE. “We were unable to estimate the number of chest pain patients who did not under CCTA based on low HEART score.” (p 48)   Emergency clinician level confounders include accuracy & reliability of HEART score after 2018 training and experience employing the score in decision-making, as well as trust in the score as an aid to CCTA test ordering. Radiologist level confounders include awareness of the HEART score EHR decision aid effort or other clinical findings that could skew interpretation of presence/absence of obstructive CAD if incorporated into imaging interpretation (Worster 2008 , Kohn 2013).

2	What was the impact on clinician behavior and patient-important outcomes?	The number of chest pain patients in whom CCTA was ordered decreased from 23.4% in 2018 to 12.6% in 2020, representing an 11.1% (95% CI 0.9 – 13) reduction. The presence of obstructive CAD increased from 15.1% in 2018 to 20.1% in 2020 (mean difference 4.9%; 95% CI -10.3 to 0.04).
3.	How will you communicate the findings of this study with your patients to facilitate shared decision-making?	CCTA is a computed tomography (CT) scan of your heart vessels. When heart vessels become obstructed, some patients experience chest pain, shortness of breath, weakness, or fatigue. Because CT scans expose patients to radiation which can have long-term health effects like cancer, physicians are striving to limit this test for patients most likely to benefit from the exposure. In other words, restrict ordering this test for patients at higher risk for obstructed blood vessels in the heart in whom an intervention like a cardiac catheterization might reduce the risk of future heart attacks. The HEART score is a medical calculator that assists clinicians in establishing a baseline risk for artery disease in the heart based on patient’s past medical history, presenting symptoms, EKG findings and blood tests on the day of the emergency department evaluation. This study indicates that asking emergency department clinicians to consider the HEART score before ordering this type of imaging can decrease CT ordering rates while increasing the likelihood of abnormal findings.

Limitations

1. Non-randomized design increases risk of bias at the level of the patients and emergency clinicians, including spectrum bias and differential verification bias.
2. Uncertain blinding of chart abstractors to study hypothesis (Gilbert 1996, Worster 2004, Kaji 2014).
3. Uncertain blinding of radiologists to HEART score or other clinical parameters, which increases the risk of incorporation bias.
4. Retrospective computation of HEART score in 2018 compared with real-time clinician obtainment of HEART score parameters in 2020 is comparing apples-to-oranges in terms of the HEART score.
5. No effort to externally or secondarily assess the reliability or accuracy of the 2018 retrospective HEART score or the real-time 2020 HEART score.
6. Reporting of positive and negative predictive values (which vary with disease prevalence) rather than likelihood ratios.
7. No qualitative assessment of emergency clinicians’ perspective of incorporating the HEART score into EHR as an obstacle to ordering CCTA because understanding level of acceptance and (perceived or real) unintended consequences of a forcing function of the EHR HEART score is essential to overcoming leaks in the Knowledge Translation Pipeline around acceptance, applicability, and able.
8. Single-center design limits external validation to sites with differing levels of acceptance/incorporation of the HEART score into CCTA ordering decisions.
9. Potential temporal bias as CCTA imaging quality improves with newer generation CT technology.

Bottom Line

Compared with an education-only intervention, mandating computation of the HEART score as a forcing function before ordering CCTA reduced CCTA ordering rates by 11% at one institution while increasing the diagnostic yield for obstructive CAD by 4.9% although the HEART score is not accurate to rule-in (LR+ 1.2) or rule-out (LR- 0.57) obstructive CAD. Multiple potential confounders at the level of the patient, emergency clinician, or radiologist may have skewed observed diagnostic accuracy estimates or impact of the HEART score EHR effectiveness.