This was a retrospective cohort study of routinely collected hospital data using a 5-year extract of Hospital Episode Statistics (HES) data from 1 April 2013 to 31 March 2018. HES is the hospital administrative database for the National Health Service (NHS) in England. It contains information on surgical procedures, diagnoses and administrative data. Diagnoses are coded using the International Classification of Diseases-10. Procedures are coded using the classification from the UK Office of Population Censuses and Surveys (OPCS, version 4). The extract contained summary patient records where neurosurgery was the main specialty, and a primary neurosurgical procedure was performed in one of the 24 NHS neurosurgical units in England. The study participants were patients who underwent a neurosurgical procedure identified using the NNAP Coding Framework, a database of approximately 870 OPCS codes (online supplemental table S1). The study pooled procedures into groups within three tiers (online supplemental table S2). Procedures were allocated to the appropriate group (in tier III) using the primary diagnosis as a filter where necessary. The UK General Medical Council specialist register for neurosurgery was used to identify Consultant Neurosurgeons and calculate surgeon-level procedure volumes.5 The primary outcome measure was postoperative all-cause 30-day mortality. Information on the date of death was obtained from the Civil Registration of Mortality data.6

The analysis proceeded in a series of steps, with elective and non-elective procedures considered separately. First, national 30-day mortality rates were calculated for each group. Second, 3-year and 5-year total procedure volumes, and average annual volumes per neurosurgeon and per unit were estimated for each group.

Third, we calculated the minimum sample size needed to detect if a provider’s (neurosurgeon/neurosurgery unit) mortality rate was double the national average, and the proportion of providers achieving the minimum sample sizes in three year and five year periods was estimated. The study used power and sample size analyses to compute the minimum sample sizes, with calculations powered at 80%. That is, the chance of a type II error (false negative) in detecting a provider with double the baseline mortality rate was 20%. A significance level of 0.05 was used. In practice, a doubling in mortality would represent a significant difference in the number of patient deaths and is a reasonable threshold for poor performance. Quality assurance programmes use statistical methods to evaluate outcomes such as funnel plots or control charts.7 8 Funnel plots use control limits to define the acceptable level of variation from the average outcome (usually two or three SD from the mean), and providers that breach of these limits are considered outliers. A doubling in mortality would result in most providers breaching the control limits, except those with the smallest procedure volumes. This approach is consistent with previous studies in this area.2 3 9

Finally, the study modelled the relative difference in mortality from the national average that could be reliably detected for different sample sizes. In addition, the difference in mortality that could be detected for the providers that perform the lowest volumes in five years was estimated. The 10th percentile of provider volume was used—rather than the absolute lowest—to avoid basing the calculation on an anomaly, such as a hospital that does not routinely perform the procedures.

Data analysis was performed using Stata, V.15 (StataCorp). The study followed guidance suggested by the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.10

None.

The UK government made publication of surgical outcomes mandatory in 2013 and the Society of British Neurological Surgeons (SBNS) subsequently established the NNAP to evaluate surgeon-level and hospital-level 30-day mortality rates for elective neurosurgery.5 The mandatory publication of surgeons’ outcomes in England was justified as a mechanism to identify poor performance, provide transparency and provide information for patients and professionals to support patient choice.15 However, there are reasons to be cautious about the reliability, fairness and unintended consequences of publishing surgeons’ outcomes. Adverse outcomes after neurosurgery are unpreventable in some cases, and where medical errors do occur the underlying causes may be broader system-level problems or a series of events, as well as errors by individual clinicians.16 A narrow focus on this metric does not account for the role of surgeons in the wider healthcare team.17 Surgeons that are incorrectly identified as having poor performance could be stigmatised following inquiries into their outcomes.18 Risk-averse behaviour among surgeons could lead to the avoidance of complex or high-risk cases and could affect the quality and quantity of training opportunities.19 20 Patients may not understand how to interpret mortality data, particularly for operations where outcomes are intrinsically poor.18

In addition to the potential issues highlighted above is the lack of statistical validity of using surgeon-level mortality rates to detect poor performance. Neurosurgeons in England do not perform sufficient volumes of procedures to detect outlier mortality rates for elective neurosurgical procedures. For surgeons with the lowest procedure volumes, increases in mortality of 9.3 times the national average (3.8%) for all procedures and 8.5 times (4.2%) for major procedures would be detectable. With increasing volume, smaller deviations would be detectable, but the magnitude of the differences goes far beyond what could reasonably be considered poor performance.

Postoperative mortality rates are much higher in non-elective neurosurgery and the minimum sample sizes are therefore lower, such that 63% of surgeons reached the threshold in five years in the major and all procedure groups. While comparisons based on non-elective activity would be more statistically robust, it is doubtful that this offers a meaningful way of identifying poorly performing surgeons; the problems pertaining to accuracy of case attribution and the complexities of emergency care are summarised in online supplemental table S3.

Hospital-level outcomes should provide a more robust means of evaluating quality of care. All neurosurgical units in England performed enough elective procedures in 5 years to detect statistical outliers. For the lowest volume units, an increase in mortality of 1.8 times the national average could be detected.

Assessing the performance of hospitals based on emergency neurosurgical care can be a useful way of demonstrating variation in quality and eliciting examples of both good and poor services.21–23 All neurosurgical units performed sufficient volumes—in the all, major, cranial and spinal procedure groups—to detect statistical outliers. For most neurosurgical subspecialties, procedure volumes were low and the minimum sample sizes were not achieved in many areas. However, in areas of practice with higher mortality rates—particularly in non-elective practice—many units achieved or exceeded the minimum sample size. This presents the opportunity to use mortality as a metric for neurosurgical subspecialties. Several examples are discussed here.

In elective neurosurgery, neuro-oncology was a subspecialty where 19 of 24 units crossed the sample size threshold. One study has previously explored 30-day mortality rates after resection of brain tumours in England, reporting a median volume of 273 cases per unit (range 29–667).24 It did not observe any instances of higher-than-expected mortality. However, given the minimum sample sizes in our study (410 for elective and 160 for non-elective) it is probable the sample size in some units was too small to detect outliers. Close attention should be paid to the length of the evaluation period and unit-level sample sizes when mortality rates are used as a quality indicator for neuro-oncology.

Trauma neurosurgery had the highest mortality rate at 11%. Mortality is an important outcome measure for traumatic brain injury and data from the national Trauma Audit and Research Network in England and Wales has been used to assess outcomes in over 15 000 patients with traumatic brain injury.25 The study reported several statistical outliers, with both better and worse than expected 30-day mortality rates. In our study, all units exceeded the minimum caseload threshold (based on 13 638 patients overall), which supports the use of mortality as a quality metric for trauma.

A high mortality rate of 10.7% was observed in non-elective cerebrospinal fluid (CSF) diversion procedures but this falls to 3.0% when only CSF shunt surgery was analysed, with a difference in average annual hospital procedure volume of 42% between the two categories (109 vs 63). Additional procedures in the CSF diversion group include external ventricular drain (EVD) insertion and endoscopic third ventriculostomy; the difference in mortality rates suggests a very high mortality rate in these additional procedures. There are a wide variety of indications for EVD insertion and differing practices in the management of acute hydrocephalus.26 Variation in mortality rates for CSF diversion surgery could be related to differences in procedural casemix, patient selection and local management practices—as well as poor quality care. Comparative outcomes would need to be interpreted in the context of these differences.

Several strategies can be employed to improve statistical power, but each introduces its own problems; these are summarised in online supplemental table S4. A key strategy is the pooling of procedures to increase sample size; this increases heterogeneity, which can complicate interpretation of the results and it may introduce problems in relation to risk-adjustment and fairness. Neurosurgical outcomes derived from national data can be used to support quality improvement, but care should be taken to avoid institutional stigma due to unfair or inaccurate comparisons.27 28 Random variation in mortality rates between providers is to be expected; appropriate reporting methods, such as funnel plots, should be used to determine if that variation is excessive.7 Mortality is undoubtedly a serious adverse outcome but there are a range of alternative quality metrics that are relevant to neurosurgeons and patients alike.12 The evaluation of mortality rates may have utility in assuring a minimum level of surgical safety, rather than measuring quality, particularly where rates are intrinsically low and variation is difficult to interpret.

The data in this study are from the period prior to the COVID-19 pandemic; the reduction in surgical activity will exacerbate the challenges relating to statistical power.29 Compromises may be necessary to overcome this, such as having extended audit periods until surgical throughput normalises.

This study used routinely collected administrative data which is subject to errors in the coding of clinical information. Errors could have affected both the classification of neurosurgical procedures and whether all neurosurgical procedures are captured in HES. This study has focused on statistical power, but other aspects to consider when evaluating quality metrics include risk adjustment, relevance of the metric to patients and clinicians, and the extent to which the risk of mortality is modifiable. Outcomes must be risk adjusted to improve the reliability of comparative evaluations, but risk-adjustment models for neurosurgery using hospital administrative data alone may not be adequate.30 Procedures were classified by type in this study but the complexity of cases (either clinically or surgically) has not been explored. Factors relating to case complexity could vary between units and ought to be considered in any evaluations of quality, possibly by risk-adjustment or stratification of cases. Our study did not explore if this was possible using these data.