Why is it important to precisely calculate sample sizes before carrying out studies?

Whether in clinical or industrial studies, numerous standards are emerging to justify sample sizes.

This is the case, for example, of the ISO 13485:2016 standard, which provides a framework for the implementation of quality management systems by medical device manufacturers. One of the new features of this standard is that it requires sample sizes to be justified for verification and validation plans. So why is it so important to calculate a sample size before conducting a study? How should you go about it? And what might a wrong calculation imply?

Our teams are here to answer your questions.

Calculating sample sizes

According to statisticians, it is necessary to calculate sample sizes in advance in order to obtain sufficient statistical power and achieve significant differences between the groups in question. Power is the ability of a study to detect a difference, if one exists.  For example, if the power of a study is calculated after the study has been carried out and is only 50%, this means that the study only has a 50% chance of detecting significant differences, when these differences actually exist. In this case, the study cannot conclude with certainty that there is a difference between the two groups, and resources have been wasted (time, money, etc.).

How can the power of a study be maximised? With large samples?

Power does not only depend on a large number of observations collected; it also depends on a risk of error, and above all on the expected value of the primary endpoint (proportion, mean, odds ratio, etc.) or the expected difference between the groups and its variability. What is ultimately most important when calculating the sample size is to have knowledge of the order of magnitude of the indicator defining the primary endpoint.

How can you have an idea of this order of magnitude, when that is what you are wanting to estimate?

When a study is being conducted, it is generally because an effect is suspected or a significant result has already been found in the literature. A hypothesis can therefore be put forward concerning the expected value of the primary endpoint or the expected difference between the groups. A risk of error of 5% is then set with power of 80% for example, enabling the number of subjects required to be calculated. Depending on budgetary constraints, the risk of error and power may be adjusted. If the study you wish to carry out is a first, then you will need to conduct a pilot study to get an idea of the order of magnitude of the primary endpoint.

What software is used to calculate sample sizes?

PASS is a software program for calculating sample sizes. It also provides a list of bibliographical references to help you understand the calculated estimates. Statistical processing software such as SAS and R can also be used to calculate the number of subjects required.

What happens if the hypothesis for the primary endpoint is poorly formulated?

If you overestimate the expected value of the primary endpoint, you will have fewer subjects to include but you will minimise the statistical power and your chances of reaching a conclusion. If you underestimate the expected value of the primary endpoint, you will have to include many more patients, which will compromise the feasibility of your study.

What experience does Soladis have in calculating sample sizes?

Soladis has relevant expertise in several fields: clinical, genomic, industrial and marketing.

In the clinical field, the regulations are stricter and the justification of sample sizes is required by the health authorities (Légifrance, 2019). The sample size must therefore be specified in the study protocol, which must be drawn up before the study is carried out.  Sometimes, there are several primary endpoints for a single study. In this case, it is necessary to clearly define the acceptance criterion for the clinical study. Based on this definition, the risk of error must be modulated and/or sample sizes calculated for each endpoint, and lastly the maximum size obtained must be selected (SPRIET, Alain and DUPIN-SPRIET, Thérèse, 2004).

In the genomic field, issues of multiple testing correction are combined with sample size calculation issues. Soladis is able to assist its customers in using and improving recently developed packages for calculating the sample sizes required for RNA-seq studies. Note that for these studies, it is necessary to have an idea of the sequencing depth (technical variability) in addition to the biological variability of the primary endpoint (in this case, gene expression).

In the industrial field, sample size calculation issues are commonly encountered. Some of the projects managed by Soladis have focused on detecting defective units within production batches, using the ISO 2859-1 standard among other things. This standard, which seems very practical, does not necessarily apply to all problems and we ensure that it is used appropriately.

Still in the industrial field, the ICH Q2 (R1) standard provides a framework for the validation of analytical procedures. Based on this reference, Soladis has been able to assist its customers in defining their analytical plan at each stage of the validation process: intermediate precision, linearity, trueness studies, etc. Soladis can also help its customers set up equivalence tests, when they want to demonstrate the reproducibility, for example, of several validation batches or when they want to ensure the acceptability of results following a change. As with comparison tests, calculating sample sizes for equivalence tests requires having an idea of the variability of the method, the average difference between samples, and the desired power; it also requires determining the equivalence criterion best suited to the situation.

Lastly, in the field of marketing, the quality of studies depends largely on the quality of the studied sample. Prior to the study, and depending on the objectives, it is therefore necessary to define the methodology that will reduce the error or the gap between reality and the measurement taken. Setting up a survey design helps precisely identify the population studied, define its size, and determine the characteristics required for the study sample. The size of a sample does not depend on the size of the population to be studied but rather on two endpoints: the desired precision of the results and the number of sub-populations to be analysed in this sample. The challenge of marketing research is therefore above all to ensure that the study sample is representative. The most rigorous methods for ensuring representativeness are probabilistic methods (simple random sampling, stratified sampling, cluster sampling, systematic sampling, etc.), although other less expensive techniques are also available (non-probabilistic methods: quota methods, convenience sampling, etc.). Soladis’s work in the marketing field tends to focus on this issue of sample representativeness.

To conclude, what advice would you give your customers on this topic?

The priorities are to clearly define the objective of the study and the target populations and to have an idea of the result you are seeking to highlight. This can be done by referring to the literature, or by conducting a pilot study before the main study. Once the hypothesis for the expected value of the primary endpoint has been formulated, our statisticians are able to calculate appropriate sample sizes.

For more information: onedt@efor-group.fr

References

Légifrance (2019, April 17). Ministerial Order of 21 December 2018 setting the protocol summary format for research involving human beings, mentioned in 3° of Article L. 1121-1 of the French Public Health Code, including only questionnaires or interviews. Retrieved from Légifrance.gouv.fr: https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000037852311

SPRIET, Alain and DUPIN-SPRIET, Thérèse. (2004). Bonne pratique des essais cliniques des médicaments. Karger Publishers.