Research Article| Volume 60, ISSUE 10, P1353-1361, December 2022

Machine learning methods applied to risk adjustment of cumulative sum chart methodology to audit free flap outcomes after head and neck surgery

Published:September 29, 2022DOI:


      We describe a risk adjustment algorithm to benchmark and report free flap failure rates after immediate reconstruction of head and neck defects. A dataset of surgical care episodes for curative surgery for head and neck cancer and immediate reconstruction (n = 1593) was compiled from multiple NHS hospitals (n = 8). The outcome variable was complete flap failure. Classification models using preoperative patient demographic data, operation data, functional status data and tumour stage data, were built. Machine learning processes are described to model free flap failure. Overall complete flap failure was uncommon (4.7%) with a non-statistical difference seen between hospitals. The champion predictive model had acceptable discrimination (AUROC 0.66). This model was used to risk-adjust cumulative sum (CuSUM) charts. The use of CuSUM charts is a viable way to monitor in a ‘Live Dashboard’ this quality metric as part of the quality outcomes in oral and maxillofacial surgery audit.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to British Journal of Oral and Maxillofacial Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Learning from Bristol. The Report of the Public Inquiry into children’s heart surgery at the Bristol Royal Infirmary 1984-1995. Presented to Parliament by Ian Kennedy QC. Available from URL: “The New NHS”, Government White Paper, 1997 (last accessed 20 October 2022).

        • Birkmeyer J.D.
        • Dimick J.B.
        • Birkmeyer N.J.
        Measuring the quality of surgical care: structure, process, or outcomes?.
        J Am Coll Surg. 2004; 198: 626-632
        • Graboyes E.M.
        • Gross J.
        • Kallogjeri D.
        • et al.
        Association of compliance with process-related quality metrics and improved survival in oral cavity squamous cell carcinoma.
        JAMA Otolaryngol Head Neck Surg. 2016; 142: 430-437
        • Kluyver T.
        • Ragan-Kelley B.
        • Pérez F.
        • et al.
        Jupyter Notebooks – a publishing format for reproducible computational workflows.
        in: Loizides F. Schmidt B. Positioning and Power in Academic Publishing: Players, Agents and Agendas. ISO Press, 2016: 87-90
        • Pedregosa F.
        • Varoquaux G.
        • Gramfort A.
        • et al.
        Scikit-learn: machine learning in Python.
        J Mach Learn Res. 2011; 12: 2825-2830
        • Lemaître G.
        • Nogueira F.
        • Aridas C.K.
        Imbalanced-learn: a Python toolbox to tackle the curse of imbalanced datasets in machine learning.
        J Mach Learn Res. 2016; 7: 1-5
        • Chen T.
        • Guestrin C.
        XGBoost: a scalable tree boosting system.
        in: In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Association for Computing Machinery. 2016: 785-794
        • Zhou Z.H.
        • Feng J.
        Deep forest.
        Nat Sci Rev. 2019; 6: 74-86
        • Seiffert C.
        • Khoshgoftaar T.M.
        • Hulse J.V.
        • et al.
        RUSBoost: a hybrid approach to alleviating class imbalance.
        IEEE Trans Syst Man Cybernet – Part A: Syst Hum. 2010; 40: 185-197
        • Kaur H.
        • Pannu H.S.
        • Malhi A.K.
        A systematic review on imbalanced data challenges in machine learning: applications and solutions.
        ACM Computing Surveys. 2020; 52: 1-36
        • Rasmussen T.B.
        • Ulrichsen S.P.
        • Nørgaard M.
        Use of risk-adjusted CUSUM charts to monitor 30-day mortality in Danish hospitals.
        Clin Epidemiol. 2018; 10: 445-456
        • Japkowicz N.
        • Shah M.
        Evaluating Learning Algorithms: a classification perspective.
        Cambridge University Press, 2011
        • Sweeny L.
        • Rosenthal E.L.
        • Light T.
        • et al.
        Outcomes and cost implications of microvascular reconstructions of the head and neck.
        Head Neck. 2019; 41: 930-939
      2. National Emergency Laparotomy Audit Online Reports (Dashboard) Overview. Available from URL: (last accessed 20 October 2022).

        • Ho M.W.
        • Nugent M.
        • Puglia F.
        • et al.
        Results of flap reconstruction: categorisation to reflect outcomes and process in the management of head and neck defects.
        Br J Oral Maxillofac Surg. 2019; 57: 935-937