Method validation

Analytical performance according to CLSI guidelines

Method validation requires demonstrating the intended use. In the biomedical field, this primarily includes the measurement properties, depending on the intended use; this section will focus on these properties.

 

  • Accuracy,
  • Intra- and interserial precision
  • Imprecision profile
  • Limit of detection, limit of quantification (LoD, LoQ)
  • Measuring range
  • Calibration curve
  • Proof of linearity
  • Specificity/Selectivity
  • robustness
  • Stability.

 

ACOMED statistik supports your method validation according to CLSI guidelines (Clinical and Laboratory Standards Institute, formerly NCCLS) through services related to the planning and analysis of experiments. Furthermore, corresponding Statistics training is offered.

 

A common task is adapting the experiments suggested in the guideline to the manufacturer's specific needs, for example, when sample volumes are small or samples cannot be stored. In such cases, the experiments and statistical analysis methods suggested in the guideline can be adapted. ACOMED statistik can assist you with this.

 

  • CLSI Guideline EP5 - Precision
  • CLSI Guideline EP6 - Linearity
  • CLSI Guideline EP7 - Interference
  • CLSI Guideline EP9 - Method Comparison, Bias.

 

Dr Keller was a member of the CLSI subcommittee for the creation of the new version (EP9-A3) of this guideline (observer status).

 

  • CLSI Guideline EP10 - Preliminary Evaluation
  • CLSI Guideline EP12 - Performance Qualitative Tests
  • CLSI Guideline EP14 - Matrix Effects/Commutability
  • CLSI Guideline EP17 - Detection Limit (LoD),
    Limit of Quantification (LoQ)
  • CLSI Guideline GP10 - Diagnostic accuracy, ROC analysis
  • CLSI Guideline C28 - Reference range

 

Dr Keller was a member of the CLSI subcommittee for the creation of the new version (EP25-A3) of this guideline (observer status).


Furthermore, Dr Keller is a member of the IFCC's "Commutability in Metrological Traceability (WG-CMT)" working group, as well as a member of the DGKL's "Decision Limits / Guideline Values" section ( link ).

Clinical performance

If the method is to be used not only for scientific purposes but also for clinical diagnostics on patients, the diagnostic properties depend on the intended application situation (diagnostics, screening, etc.).

 

  • Sensitivity, Specificity
  • Predicted PPV/NPV values
  • ROC curve
  • Alternatively, if applicable: positive and negative percentage agreement using a comparison method
  • Cut-off value
  • possibly up to therapeutic relevance (efficacy studies)

 

One has to decide, whether the study is undertaken to estimate the diagnostic accuracy measures or to prove them vs. predefined acceptance criteria.


The nature of the method used to describe the clinical reality is decisive for the design. A distinction is made (according to CLSI guideline EP12 or FDA guidance) between a diagnostic quality criterion or Reference standard on the one hand (allows estimation of sensitivity, specificity etc.) or a Comparison method on the other hand (allows determination of positive and negative percent agreement PPA and NPA). [This is simply a matter of terminology; the calculation formulas are identical.] 


A final major point is, whether a case control design (known reference standard result is used as inclusion criterion) or a cohort design (inclusion refers to the intented diagnosis) is chosen. Whereas the case control design is easier to implement, it typically leads to a spectrum bias connected with an overestimation of the diagnostic accuracy measures: This is the bias with the largest impact in diagnostic studies.

Decision and reference limits

Knowledge of cut-off values is a prerequisite for determining diagnostic accuracy and for clinical application in general. Determining clinically used cut-off values is a complex issue, as their calculation is time-consuming and also depends on the specific clinical application. A fundamental distinction is made between decision limits (defined based on clinical aspects, e.g., therapeutic consequences) and reference intervals consisting of one or two reference limits. Reference limits are determined based on the "outer limit(s)" (usually percentiles) of a well-defined and carefully selected healthy control group. This determination of reference intervals is described in the CLSI Guideline C28. Alternative concepts also exist and are being investigated by the Reference Limits Working Group of the German Society for Clinical Chemistry and Laboratory Medicine (DGKL), in which Dr. Keller participates. References:
  • Haeckel R, Wosniok W, Arzideh F. A plea for intra-laboratory reference limits. Part 1. General considerations and concepts of determination. Clin Chem Lab Med 2007; 45:1033-42,
  • Arzideh F, Wosniok W, Gurr E, Hinsch W, Schumann G Weinstock N, Haeckel R. A plea for intra-laboratory reference limits. Part 2. A bimodal retrospective concept for determining reference limits from intra-laboratory databases demonstrated by catalytic activity concentrations of enzymes. Clin Chem Lab Med 2007; 45:1043-57

Examples of statistical assistance for method validations

Method comparison according to CLSI-EP9 for different data scales, especially count data (hematology, Eli-Spot, ...)

When comparing different analyzers in hematology, a large number of analytes typically need to be examined. These analytes also exhibit varying properties; in particular, a number of them are count data. Other methods, such as tumor cell count analysis, flow cytometry, or ELISPOT data, also produce count data. These data are characterized by their Poisson distribution. For Poisson-distributed data, homogeneity of variance, as seen, for example, in a difference plot, cannot be achieved by simply displaying either absolute or relative differences. This can also lead to problems in precision analyses, as neither the standard deviation nor—alternatively—the coefficient of variation is independent of the concentration (more accurately, the number of cells).

ACOMED statistik has developed methods to deal with these issues.

Experimental setup of precision experiments according to CLSI-EP05 with small sample material

Precision experiments according to the CLSI EP05 guideline, in which two factors are examined, e.g., lot and tag, require approximately 75–80 measurements, meaning that a corresponding number of aliquots must be prepared from a single sample. However, often insufficient sample material is available.

The solution is to conduct the experiments on multiple samples and statistically pool the results.

ACOMED statistik plans and evaluates your precision experiments under these conditions.

Application example: Varga Z, Lebeau A, Bu H, Hartmann A, Penault-Llorca F, Guerini-Rocco E, Schraml P, Symmans F, Stoehr R, Teng X, Turzynski A, von Wasielewski R, Gürtler C, Laible M, Schlombs K, Joensuu H, Keller T, Sinn P, Sahin U, Bartlett J, Viale G (2017). An international reproducibility study validating quantitative determination of ERBB2, ESR1, PGR, and MKI67 mRNA in breast cancer using MammaTyper®.Breast Cancer Research. Breast Cancer Res. 2017 May 11;19(1):55. doi: 10.1186/s13058-017-0848, link

Qualitative detection limit according to CLSI-EP12 for measurements in difficult matrices

The detection limit of such assays is usually evaluated using the probit method. However, estimation using the probit method is difficult with challenging matrices or small sample sizes. Statisticians at Roche Molecular Diagnostics (USA) have developed an alternative evaluation method. [Vaks JE (2017). New Method of Evaluation of Limit of Detection in Molecular Diagnostics, link ; Canchola JA, Vaks JE, Tang S (2016): Limit of Detection (LoD) Estimation Using Maximum Likelihood from (Hit) Rate Data: The LoD_MLE SAS Macro, link ] .

We programmed this method by JE Vaks in SAS, but without IML, and would be happy to analyze your data with this innovative method.