A direct comparison of patient-based real-time quality control techniques - The importance of the analyte distribution.pdf

Research Article Annals of Clinical Biochemistry 2020, Vol. 57(3) 206–214 ! The Author(s) 2020 Article reuse guidelines: sagepub.com/journals-permissions DOI: 10.1177/0004563220902174 journals.sagepub.com/home/acb A direct comparison of patient-based real-time quality control techniques: The import...

Research Article Annals of Clinical Biochemistry 2020, Vol. 57(3) 206–214 ! The Author(s) 2020 Article reuse guidelines: sagepub.com/journals-permissions DOI: 10.1177/0004563220902174 journals.sagepub.com/home/acb A direct comparison of patient-based real-time quality control techniques: The importance of the analyte distribution Joel D Smith1 , Tony Badrick2 and Francis Bowling1 Abstract Background: Patient-based real-time quality control (PBRTQC) techniques have been described in clinical chemistry for over 50 years. PBRTQC has a number of advantages over traditional quality control including commutability, cost and the opportunity for real-time monitoring. However, there are few systematic investigations assessing how different PBRTQC techniques perform head-to-head. Methods: In this study, we compare moving averages with and without truncation and moving medians. For analytes with skewed distributions such as alanine aminotransferase and creatinine, we also investigate the effect of Box–Cox transformation of the data. We assess the ability of each technique to detect simulated analytical bias in real patient data for multiple analytes and to retrospectively detect a real analytical shift in a creatinine and urea assay. Results: For analytes with symmetrical distributions, we show that error detection is similar for a moving average with and without four standard deviation truncation limits and for a moving median. In contrast to analytes with symmetrically distributed results, moving averages perform poorly for right skewed distributions such as alanine aminotransferase and creatinine and function only with a tight upper truncation limit. Box–Cox transformation of the data both improves the performance of moving averages and allows all data points to be used. This was also confirmed for retrospective detection of a real analytical shift in creatinine and urea. Conclusions: Our study highlights the importance of careful assessment of the distribution of patient results for each analyte in a PBRTQC program with the optimal approaches dependent on whether the patient result distribution is symmetrical or skewed. Keywords Quality assurance and control, statistics, patient-based quality control Accepted: 2nd January 2020 Introduction Patient-based real-time quality control (PBRTQC) techniques have been described in the clinical laboratory for more than 50 years.1 PBRTQC has a number of purported advantages over traditional quality control (QC) including commutability, cost and the opportunity for real-time monitoring.1,2 All PBRTQC techniques suffer from what has been termed the 1 Chemical Pathology, Royal Melbourne Hospital, Melbourne, VIC, Australia 2 RCPA Quality Assurance Programs Pty Ltd, St Leonards, NSW, Australia Corresponding author: Joel D Smith, Chemical Pathology, Royal Melbourne Hospital, Grattan St, Melbourne, VIC 3050, Australia. Email: [email protected] Smith et al. ‘population problem,’1 in which variations in the patient population by time of day, week or month due to phlebotomy schedules or outpatient clinics cannot be distinguished from shifts in analytical performance.1 Censoring data from certain patient populations (e.g. hospital inpatients, intensive care, dialysis unit) can be used but may lead to a reduction in the number of data points in small hospital laboratories with a high number of inpatient samples, delaying the detection of bias.2 There have been various statistical attempts to reduce the impact of population variation in PBRTQC techniques. In the earliest use of the average of (patient) normals (AoN) PBRTQC technique in a clinical chemistry setting, Hoffman and Waid used the daily average of patient results truncated for results falling within the prespecified reference interval in order to reduce the effect of data points from diseased patients biasing the AoN metric.3 Reed modified the approach of Hoffman and Waid to specify that averages are calculated from the same number of tests (uniform block size) rather than daily and to specify that truncation limits were based on the distribution of the laboratories’ own results rather than the prespecified reference interval.4 Similarly, van Rossum and Kemperman5 and Ng et al.6 used truncated means in their moving average (MA) PBRTQC approaches, with van Rossum and Kemperman optimizing truncation limits based on a bias detection curve and Ng et al. optimizing truncation limits and other parameters (e.g. block size) iteratively using a simulated annealing algorithm. In addition to the use of truncation (i.e. trimmed means), moving medians have been used in PBRTQC techniques.7,8 It is also recognized that the distribution of patient results for some analytes is not normally distributed but skewed (for example transaminases). Transformations such as log or square root have been proposed to normalize skewed distributions to improve AoN or MA approaches.9 The median is a more robust indicator of central location less affected by the skewedness of the distribution of patient results and also does not require the censoring of any patient data. This allows a moving median to be calculated more frequently than an MA from truncated data, theoretically allowing shifts to be detected sooner. For the derivation of reference intervals from patient data, the Box–Cox family of transformations has been used to transform skewed and non-Gaussian patient data with success.10 This approach has not yet been used to transform data in a PBRTQC program. Overall, there are few systematic investigations assessing how different PBRTQC techniques perform head-to-head. In this study, we compare MAs with and without truncation and moving medians both with and 207 without and Box–Cox transformation for analytes with skewed distributions. We assess the ability of each technique to detect analytical bias using the technique of Ng et al., simulating bias on historical patient data and calculating the average number of patient samples until the induced bias was detected.6 Methods Patient data and setting Raw de-identified patient results for a number of clinical chemistry analytes (full list presented in Table 1) were downloaded from an Abbott Architect (c1600) using the AMS middleware for the period of 26 December 2017 to 25 June 2018. The laboratory services a
570-bed tertiary hospital in Melbourne, Australia. Samples are mainly received from hospital inpatients, including emergency and intensive care but are also received from subacute and outpatient populations. There were no exclusions based on age or location (e.g. ward, ICU, outpatient), but QC and external quality assurance samples were excluded. The final data-set included 945,498 results and was imported into the R statistical computing environment for further analysis.11 The assays were in control during this time period. Graphical assessment of patient data distribution The distribution of patient data was graphically visually using normal quantile plots in R. Normal Quantile Plots (Q-Q plot) for each of the analytes in Table 1 were constructed for one month of raw patient data using the qqnorm and qqline functions in R. Each Q-Q plot was inspected for symmetry (example in Figure 1(a)) and right skew (example Figure 1(b)). Only analytes with skewed distributions were subject to protocols using Box–Cox transformation as described below. A summary of the properties of each analyte distribution appears in Table 1. PBRTQC protocols Block size. For all protocols, the block size (or filter length) used to calculate the mean was set at 50 samples, which was the optimal block size identified by Fleming and Katayev in their study.9 Block size can also be calculated using the method of Cembrowski et al.,12 based on the ratio of patient population standard deviation to the analytical standard deviation. It should be noted that the study of Fleming and Katayev was performed in a large community laboratory and a block size of 50 may not be ‘optimal’ for our setting; however, a standard block size was chosen for 208 Annals of Clinical Biochemistry 57(3) Table 1. List of analyte and PBRTQC parameters used in this study. Analyte Distribution RCPAQAP APS (%) Low TL (wide) High TL (wide) Low TL (narrow) High TL (narrow) Lambda 6 4 5 1.4 57 3 35 2.1 50 2.6 NA NA Symmetrical distributions Albumin Calcium Symmetrical Symmetrical Cholesterol Symmetrical 6 0 9.4 2.5 6 NA Chloride Bicarbonate Symmetrical Symmetrical 3 10 86 9.9 123 39 100 20 110 30 NA NA Iron Symmetrical 12 0 49 5 20 NA HDL Potassium Symmetrical Symmetrical 12 5 0 1.6 2.8 6.5 0.7 3.1 1.5 4.3 NA NA Magnesium Symmetrical 8 0 1.7 0.6 0.9 NA Sodium Phosphate Symmetrical Symmetrical 2 8 123 0 155 2.6 135 0.3 144 1.3 NA NA Total Protein Symmetrical 5 30 108 60 90 NA Transferrin Symmetrical 0 Low TL (untransformed) 4.7 High TL (untransformed) 1 Low TL (transformed) 3 High TL (transformed) NA Distribution 8 RCPAQAP APS (%) Right skewed Right skewed 12 12 0 0 1.52 2 0.626 0.380 Analyte Lambda Skewed distributions ALP ALT 250 300 1.4 1.6 AST Right skewed 12 0 300 1.25 1.35 0.670 CK Creatinine Right skewed Right skewed 12 8 0 0 1000 150 2.5 0.86 3.5 0.865 0.220 1.150 GGT Right skewed 12 0 250 1.9 2.5 0.343 Glucose LDH Right skewed Right skewed 8 8 0 0 10 350 0.9 1.7 1.2 1.77 0.626 0.545 1.75 0.383 1 1.7 0.140 0.260 Total Bilirubin Right skewed 12 0 25 Triglycerides Urea Right skewed Right skewed 12 12 0 0 2.3 10 1.25 0.6 1 Note: Included are distribution properties, truncation limits applied in the comparison and Box–Cox lambda values where transformation is applied. TL: truncation limits applied to moving averages where applicable; RCPA QAP APS: Royal College of Pathologists of Australasia Quality Assurance Program Analytical Performance specifications; ALT: alanine aminotransferase; ALP: alkaline phosphatase; AST: aspartate aminotransferase; CK: creatine kinase; GGT: gamma-glutamyl transferase; LDH: lactate dehydrogenase; HDL: high-density lipoprotein. simplicity for this comparative study. For a particular analyte and population, the block size will vary, but the validity of the observed outcomes will not be affected by choosing a constant block size. Control limits. Control limits were set at 2.58 standard deviations of the calculated mean of the MA/median. A standard deviation of 2.58 was chosen to allow for a 1% false rejection rate. Individual protocols. The protocols that were assessed differed for symmetrical (e.g. calcium, sodium) and skewed distributions (e.g. alanine aminotransferase [ALT], alkaline phosphatase [ALP]) with Box–Cox transformation only being used for skewed distributions. All MAs and medians were calculated in R using the Zoo package.13 Symmetrical distributions Moving average. A simple MA (block size 50) was calculated using all results without truncation or exclusion. MA with wide truncation limits. For this approach, patient data for each analyte were censored using truncation limits corresponding to the method of Ng et al.6 and suggested as initial limits in recent recommendations.2 Briefly, truncation limits are placed at 4 standard deviations from the patient mean for roughly symmetrical result distributions in order to reduce the influence of outliers on the MA. Truncation limits appear in Table 1. MA with narrow truncation limits. For this protocol, truncation limits were set by visual inspection of the normal quantile plots (Q-Q plot) for each analyte. In theory, Smith et al. 209 Transformed MA and moving median. Patient data for skewed distributions were transformed by the Box–Cox transformation yi ¼ xki  1 if k > 0; k yi ¼ lnxi if k ¼ 0 The Box–Cox transformation is a type of power transformation that can be can be used to transform non-Gaussian data to approximate a normal distribution. Lambda was determined in R using the ‘boxcox’ function in the MASS package.16 Simple MAs and moving medians were then performed on the transformed data. A normal Q-Q plot for ALT before and after Box–Cox transformation appears in Supplemental Figures S2(a) and (b). Transformed MA with narrow truncation limits. For this protocol, truncation limits on the Box–Cox transformed data were set by visual inspection of the normal Q-Q plot for each analyte as described above. Bias simulation and detection Moving median. A simple moving median was calculated using all results without truncation or exclusion. We assessed the performance of each PBRTQC protocol using the method of Ng et al.6 In this method, a bias is introduced into the patient data and the number of patient results falling between the point of bias introduction and the first PBRTQC value exceeding a control limit is calculated as the number of patient samples to error detection (NPed). For each analyte and PBRTQC method, this process was repeated at different points in the data with the order of obtained results being maintained. We performed 10 trials for each analyte and PBRTQC method, introducing bias spaced at 200 sample intervals in the patient data. From these trials, the average number of patient samples to error detection (ANPed) is calculated. Bias was introduced as a percentage based on multiples of the Royal College of Pathologists of Australasia Quality Assurance Program’s Analytical Performance specifications (APS) for each analyte.17 These are shown in Table 1. Skewed distributions Results MA and moving median. Simple MAs and moving medians were performed for skewed distributions as described for symmetrical distributions. Symmetrical distributions Figure 1. (a) Normal QQ Plot for sodium showing symmetrical distribution (but not strictly Gaussian – note deviation in the tails). (b) Normal QQ plot for ALT showing significantly right skewed distribution. the distribution of patient results represents a mixture of diseased and non-diseased patients, with the values from diseased patients concentrated in the upper and lower parts of the distribution with an interval containing values predominantly from non-diseased patients.14,15 Therefore, in an attempt to minimize the influence of diseased population on the MA, limits were set based on deviation from the linear portion of the Q-Q plot (see Supplemental Figure S1). MA with truncation. For this approach, patient data for each analyte with a skewed distribution were censored using truncation limits described by Ng et al.6 at a userdefined inflection point for which the frequency distribution curve of the patient data begins to ‘level off’. For the patient data with roughly symmetrical distributions, bias detection curves for the different PBRTQC approaches appear in Figure 2. Calculated ANPed and standard deviations for each approach can be found in supplemental Table S1. From Figure 2, it can be seen that the detection of bias varies depending on the analyte, the PBRTQC 210 Annals of Clinical Biochemistry 57(3) Figure 2. Bias detection curves for symmetrical distributions. For all analytes see supplemental Figure 3. ANPed: average number of patient samples until error detected; APS: analytical performance specifications; MA: moving average. protocol used and whether the bias is positive or negative. For all PBRTQC protocols, a negative bias in albumin was difficult to detect with an ANPed for a 1 APS shift (6%) greater than 300 samples for each protocol. Similarly, a positive bias in bicarbonate was difficult to detect with a þ1 APS shift (þ10%) also requiring greater than 300 samples to detect. Error detection was better for negative shifts in bicarbonate with an ANPed less than 50 samples for a –1 APS shift (10%). The error detection was most readily achieved by most methods for sodium, with ANPed less than 55 samples for both positive and negative single APS (2%) shifts in sodium. With regard to the protocol used, the addition of 4SD truncation limits (MA wide truncation) adds little benefit over a simple MA with no truncation. One notable exception is magnesium, where ANPed is reduced from 460 and 576 samples to 40 and 56 samples for a positive and negative single APS (8%) shift, respectively. Interestingly, it can be seen that attempting to exclude abnormal results from the protocol with narrow truncation limits paradoxically increases ANPed for larger shifts. This is due to an increasing number of results that fall outside the truncation limits as the introduced bias increases, thereby increasing ANPed.6,18 Overall, both the MA with wide or no truncation and the median perform similarly for these analytes. Skewed distributions For the patient data with right skewed distributions, bias detection curves for the different PBRTQC approaches appear in Figure 3. Again, the calculated ANPed and standard deviations for each approach can be found in supplemental Table S1. From Figure 3, it can be seen that for these right skewed analytes, the overall error detection is poorer than that for the symmetrically distributed analytes with the few protocols being able to detect a single APS shift in less than 100 samples. Additionally, differences in error detection between each PBRTQC protocol are more marked. The protocols without transformation perform poorly with the untransformed MA failing to detect negative shifts at any multiple of APS in ALT, aspartate aminotransferase (AST), bilirubin, creatine kinase (CK), creatinine, gamma-glutamyl transferase (GGT), glucose, lactate dehydrogenase (LDH) and triglycerides. The untransformed median and the untransformed MA with truncation performed Smith et al. 211 Figure 3. Bias detection curves for skewed distributions. For all analytes see supplemental Figure 4. ANPed: average number of patient samples until error detected; APS: analytical performance specifications; MA: moving average. better but still failed to detect negative shifts in CK and had the poorest performance in detecting negative shifts in AST and ALT. Box–Cox transformation of the data markedly improves the performance of a simple MA for a number analytes allowing the MA to have the best overall performance when detecting negative shifts in ALT, AST, bilirubin, LDH, creatinine and urea. Despite being a more robust measure of central tendency, the performance of the moving median is also improved by the Box–Cox transformation of the data. For example, a 1 APS negative shift in creatinine (8%) is detected in 423 samples for the median of the transformed data versus 1653 samples for the untransformed data. Retrospective detection of a real analytical shift We assessed if we could replicate the improved detection of analytical shifts for skewed distributions in detecting a real instrument failure affecting creatinine and urea. Each algorithm was applied to patient results from the analyser retrospectively, for the day the analyser was noted to have a drainage failure, resulting in wash fluid back flowing into the cuvettes. This shift was originally detected by a delta check from patient results, around 1.5 h since starting the analyser, resulting in 85 samples that required re-run and amending. Urea and creatinine were most affected by this failure. Figure 4 shows the selected algorithms against time for urea and creatinine for the day of the failure. Table 2 summarizes the error detection time of the different algorithms. From Table 2, it can be seen that the MA for the Box–Cox transformed data detected the error in the shortest time (20 min for creatinine and 52 min for urea). MA for untransformed data could only detect the shift with tight upper truncation, but did so in a similar time to the MA on Box–Cox transformed data. A simple MA did not detect the shift in creatinine and detected the urea shift in 64 min. Discussion In this study, we directly compare the performance of a number of PBRTQC protocols, assessing truncation limits, data transformation and type of PBRTQC algorithm (MA vs. moving median). We used the performance metric ANPed in our comparison, a metric that has been used extensively in investigations of PBRTQC approaches.6,18,19 Our study highlights the importance of careful assessment of the distribution of patient results for each analyte of interest in a PBRTQC program with the optimal approaches dependent on whether the patient result distribution is symmetrical or skewed. For analytes with symmetrical distributions, we show that ANPed is similar for the MA with and without four SD truncation limits and for the moving median. Narrow truncation limits selected to isolate predominantly non-pathological from pathological specimens based on the normal QQ-plot had the undesired effect of increasing ANPed for greater biases. 212 Annals of Clinical Biochemistry 57(3) Figure 4. Retrospective detection of analytical shift due to analyser drainage failure. (a) MA of Box–Cox transformed data for creatinine, (b) MA of Box–Cox transformed data for creatinine and (c) MA of untransformed data for creatinine. This effect is well known and was shown in the excellent study of MA truncation limits by Ye et al., where ANPed was also used as a performance metric18 and by Ng et al.6 Overall, for our laboratory data, we question the importance of truncation limits for symmetrically distributed analytes. In a PBRTQC program, the use of a median or a mean truncated for extreme outliers (4 SD) is both reasonable, and narrow limits aimed at Smith et al. 213 Table 2. Retrospective analysis of patient data from the day of an analyser error. Method Urea (min) Creatinine (min) Delta check Moving average Moving average (truncated for Cr <150, urea <10) Moving average (Box–Cox transformed) Moving average (Box–Cox transformed and truncated) Moving median Moving median (Box–Cox transformed) Internal QC (scheduled 8 hourly) Not detected 64 57 52 Not detected 62 57 480 64 Not detected 21 21 Not detected 37 36 480 Note: Time (in min) until detection of analytical shift is shown for different algorithms. excluding pathological samples may have the paradoxical effect of increasing the number of samples until bias is detected. For analytes with symmetrically distributed results, the overall variance of the distribution is likely low enough, even when pathological results are present, to allow the mean or median to be a robust measure in PBRTQC. This adds weight to the comment of Ng et al., that the goal of PBRTQC is to ‘monitor the process not the patients’.6 In contrast to analytes with symmetrically distributed result (e.g. sodium, calcium), MA performs poorly for right skewed distributions such as ALT and creatinine and functions only with a tight upper truncation limit. The requirement for tight truncation for analytes with skewed distribution was shown by Ye et al. with ALT with ‘optimal’ truncation limits having an upper limit of 42 U/L which excluded
20% of the patient data.18 For skewed distributions, the median outperforms the simple MA without truncation limits. As the median is a more robust indicator of central tendency, this is expected, but to our knowledge, this has not been shown directly for skewed distributions of patient results. Box–Cox transformation of the data both improves the performance of MA and allows all data points to be used. Box–Cox transformation also largely improves the performance of the moving median for skewed data. This was also confirmed for retrospective detection of a real analytical shift in creatinine and urea due to a drainage failure. To our knowledge, this is the first study to quantitatively assess the importance of transformation for skewed data in a PBRTQC program. In a study on PBRTQC, Fleming and Katayev applied transformations (natural log or square root) to visually skewed data, but did not compare the performance of PBRTQC approach with transformed data to that with untransformed data. Here, data were first assessed for skewness based on a graphical method (normal QQ-plot). For skewed distributions, we used the Box–Cox transformation, as it may be assumed that clinical chemistry analytes from normal subjects follow a Power Normal Distribution which is related to a Gaussian distribution by the Box–Cox transformation.14 Box–Cox transformation has been used previously in reference interval studies,10 but the use of the transformation in a PBRTQC is to our knowledge novel. The logarithmic transformation is more commonly available in middleware software and is a special case of the Box–Cox family of transformations with the exponent k ¼ 0. A direct comparison of logarithmic transformation with Box–Cox transformation is beyond the scope of this paper, but where Box–Cox transformation is unavailable, logarithmic transformation may offer similar benefits for skewed distributions. Due to the utility of the Box–Cox transformation shown here, we endorse the recommendation of Loh et al. that middleware and laboratory information system software vendors support real-time Box–Cox transformation in their software.20 This study has some important limitations that may influence the wider applicability of the findings. Control limits of 2.58 SD of the calculated mean for each algorithm were used to achieve a 4 1% false rejection rate. This was deemed appropriate for this comparative study, and a false rejection rate of 4 1% in a PBRTQC program has also been targeted by other groups.9 However, given the large number of results being produced, these control limits may still result in excessive false alarms. In a clinical setting, initial control limits are often adjusted empirically to achieve the tolerable false rejection rate and wider control limits may be more appropriate. Although widening control limits will increase ANPed and the relative performance of each algorithm, our conclusions are unlikely to be affected. Control limits based on SD were used in this study. These are commonly used in PBRTQC programs allowing plotting on interpretation of algorithms on a Levey-Jennings like plot. However, using symmetrical SD-based control limits for the untransformed or untruncated, MA and median for non-symmetrical distributions will have contributed to the poor performance of these algorithms, particularly on the low side. For untransformed MAs or medians, ideally, 214 asymmetric control limits should be used, but these are difficult to specify and also plot on Levey-Jennings like plot. One of the benefits of data transformation is the ability to use symmetrical SD-based control limits. Alternatively, percentile-based control limits could be considered for skewed distributions. Another limitation is that the various protocols we are assessed on their ability to detect simulated bias in real patient data, introduced as percentage shift at a particular sample. This method has been used in simulation studies previously and is simple to implement computationally.6,18 We also confirmed our findings for skewed distributions on a true analytical shift due to a drainage failure. The findings may, however, not extend to the detection of gradually introduced bias or increased imprecision in an assay. The patient data in our study represent mainly hospital inpatients in a major tertiary referral centre in Melbourne, Australia and the conclusion here may not apply to laboratories servicing a different population. However, given our patient population will contain a high proportion of samples with disease present and be influenced by outpatient clinics and phlebotomy schedules, our findings are likely to be robust for settings with less variation. There is growing interest in PBRTQC with a number of approaches that can be applied.2 There are currently a number of barriers to widespread implementation of PBRTQC with time, statistical skills and software support required to optimize and implement a PBRTQC program.2 We show that managing the ‘population problem’ is not so much about excluding portions of the population but identifying where a population distribution is skewed and variance can be reduced by appropriate transformation. We hope our findings will assist those both selecting PBRTQC approaches for implementation and those designing software for PBRTQC, allowing the use of the most optimal methods and wider adoption. Declaration of conflicting interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding The author(s) received no financial support for the research, authorship, and/ or publication of this article. Ethical approval This retrospective quality assurance (QA) project was approved by the Melbourne Health Office for Research (Project Number QA2019134). Guarantor JDS. Annals of Clinical Biochemistry 57(3) Contributorship JDS, FB and TB conceived the study. JDS performed the simulation and data analysis. JDS wrote the first draft of the article. All authors reviewed and edited the article and approved the final version of the article. ORCID iD Joel D Smith https://orcid.org/0000-0002-2098-3716 Supplemental material Supplemental material is available for this article online. References 1. Badrick T, Cervinski M and Loh TP. A primer on patient-based quality control techniques. Clin Biochem 2019; 64: 1–5. 2. Badrick T, Bietenbeck A, Cervinski MA, et al. Patient-based real-time quality control: review and recommendations. Clin Chem 2019; 65: 962–971. 3. Hoffmann RG and Waid ME. The “Average of Normals” method of quality control. Am J Clin Pathol 1965; 43: 134–141. 4. Reed AH. Use of patient data for quality control of clinical laboratory tests. Clin Chem 1970; 16: 129–134. 5. Rossum HHV and Kemperman H. Implementation and application of moving average as continuous analytical quality control instrument demonstrated for 24 routine chemistry assays. Clin Chem Lab Med 2017; 55: 1142–1151. 6. Ng D, Polito FA and Cervinski MA. Optimization of a moving averages program using a simulated annealing algorithm: the goal is to monitor the process not the patients. Clin Chem 2016; 62: 1361–1371. 7. Bietenbeck A, Thaler MA, Luppa PB, et al. Stronger together: aggregated Z-values of traditional quality control measurements and patient medians improve detection of biases. Clin Chem 2017; 63: 1377–1387. 8. Wilson A, Roberts WL, Pavlov I, et al. Patient result median monitoring for clinical laboratory quality control. Clin Chim Acta 2011; 412: 1441–1446. 9. Fleming JK and Katayev A. Changing the paradigm of laboratory quality control through implementation of real-time test results monitoring: for patients by patients. Clin Biochem 2015; 48: 508–513. 10. Arzideh F, Wosniok W and Haeckel R. Indirect reference intervals of plasma and serum thyrotropin (TSH) concentrations from intralaboratory data bases from several German and Italian medical centres. Clin Chem Lab Med 2011; 49: 659–664. 11. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2018. 12. Cembrowski GS, Chandler EP and Westgard JO. Assessment of “Average of Normals” quality control procedures and guidelines for implementation. Am J Clin Pathol 1984; 81: 492–499. 13. Zeileis A and Grothendieck G. Zoo: S3 infrastructure for regular and irregular time series. J Stat Softw 2005; 14: 1–27. 14. Wosniok W and Haeckel R. A new indirect estimation of reference intervals: truncated minimum chi-square (TMC) approach. Clin Chem Lab Med 2019; 57: 1933–1947. 15. Hoffmann G, Lichtinghagen R and Wosniok W. Simple estimation of reference intervals from routine laboratory data. J Lab Med 2015; 39: 389–402. 16. Venables WN, Ripley BD and Venables WN. Modern applied statistics with S. 4th ed. New York: Springer, 2002, pp.xi–495. 17. Jones GR, Sikaris K and Gill J. ’Allowable Limits of Performance’ for external quality assurance programs – an approach to application of the Stockholm criteria by the RCPA quality assurance programs. Clin Biochem Rev 2012; 33: 133–139. 18. Ye JJ, Ingels SC and Parvin CA. Performance evaluation and planning for patient-based quality control procedures. Am J Clin Pathol 2000; 113: 240–248. 19. Liu J, Tan CH, Badrick T, et al. Moving standard deviation and moving sum of outliers as quality tools for monitoring analytical precision. Clin Biochem 2018; 52: 112–116. 20. Loh TP, Cervinski MA, Katayev A, et al. Recommendations for laboratory informatics specifications needed for the application of patient-based real time quality control. Clin Chim Acta 2019; 495: 625–629.