In this case study, I describe how Dr.DataScience applied optimal statistical methods within the constraints of a limited sample size and complex data characteristics to extract reliable insights, in response to a client’s wish to evaluate the effectiveness of a complex intervention program (including rehabilitation, learning therapy, music therapy, animal-assisted therapy, etc.) from multiple aspects—cognitive function, activities of daily living, QOL (quality of life), and pain.

In particular, I provided practical solutions to statistical challenges frequently encountered in clinical research, such as the evaluation of pre/post-intervention data, adjustment for multiple comparisons, and the analysis of repeated-measures data. In accordance with our confidentiality agreement, no specific figures or detailed clinical background are disclosed.

Background and objective

The client aimed to objectively evaluate the effectiveness of a complex intervention administered to subjects with declining cognitive function and difficulties in daily life. Specifically, they wished to clarify what changes the intervention brought about in the subjects’ cognitive function, activities of daily living, QOL, and pain, and to statistically test the presence and degree of those effects. However, the data had the constraint of a small sample size of around 10 cases, and the choice of appropriate statistical methods and the interpretation of the results required specialized expertise.

Data and variables

This analysis used evaluation data of subjects before and after the intervention program.

1. Outcomes
   • Cognitive function: scores measured on multiple assessment scales (e.g., HDS-R, MMSE).
   • Activities of daily living: scores measured on multiple assessment scales (e.g., BI, FIM).
   • QOL (quality of life): scores measured on a dedicated assessment scale.
   • Pain: scores measured on an assessment scale.
2. Data structure: repeated-measures data from the same subjects before and after the intervention.

Analytical methods

1. Testing the pre/post-intervention difference
   • Consideration of test methods: I considered the paired t-test, the Wilcoxon signed-rank test, and permutation tests (two-sided and one-sided).
   • Reasons for consideration: Because the sample size was small, estimating the distribution was difficult, so careful judgment was needed regarding the application of parametric tests, which have distributional assumptions, and nonparametric tests, which do not require them.
   • Most appropriate method: The permutation test, considered especially effective when the sample size is small, was evaluated as the most appropriate test method in this study.
   • Multiple-comparison correction: Treating the overall hypothesis as a single package, I estimated p-values with Holm correction based on the Parallel Gatekeeping Procedure, a multiple-comparison correction procedure that assigns priority to hypotheses.
   • Results: In the exploratory t-tests and Wilcoxon signed-rank tests, significant pre/post-intervention differences were found for several outcomes including QOL, but in the permutation tests a significant difference was found only for QOL. After multiple-comparison correction, the corrected p-value for QOL was 0.06, and under the condition of a significance level of 0.05 no significant difference was found.
   • Interpretation: Since the research aim was to evaluate the intervention’s effectiveness for each variable, I recommended, in consideration of the sample size, that emphasis be placed on the permutation-test results and that it be claimed that an intervention effect was found only for QOL.
2. Analysis of the magnitude of the intervention’s influence
   • Method: Because the data contained repeated measurements before and after the intervention, I applied generalized estimating equations (GEE).
   • Assumption on distributional structure: Since each variable was an assessment scale, I selected a GEE model assuming a normal distribution.
   • Evaluation items: I evaluated the magnitude of the intervention’s influence on each outcome variable.
   • Univariate analysis: For QOL, a result was obtained in which the score increased significantly with the intervention.
   • Multivariate analysis: Because some variables had missing values, I evaluated two patterns—a “model excluding that variable” and a “full-variable model”—and a strong influence of the intervention on QOL was detected.
3. Comprehensive consideration: The intervention is evaluated as having an effect on QOL.

Overview of the main results and clinical considerations

As a result of this analysis, it was suggested that the complex intervention may have a consistent improvement effect on QOL in particular, among the multiple measured outcomes.

• ・A clear effect on QOL: A significant improvement in the QOL score was confirmed both in the test of the pre/post-intervention difference (permutation test) and in the influence analysis by GEE. This strongly suggests that the intervention may have contributed to improving the subjects’ quality of life.
• ・Influence on other outcomes: For other outcomes such as cognitive function, activities of daily living, and pain, statistically significant improvement was difficult to find, or the results were less consistent. This may be influenced by factors such as the nature and duration of the intervention, the sensitivity of the assessment scales, or the sample size.
• ・The importance of statistical methods with small data: This case shows the importance of selecting and applying appropriate statistical methods such as permutation tests and GEE, in addition to the paired t-test and the Wilcoxon test, especially in clinical research with a small sample size. It also highlighted the complexity of applying multiple-comparison correction and interpreting it.

These findings provide important implications for the client to objectively evaluate the effectiveness of the complex intervention they conducted and to consider the direction of future intervention programs and study design.

Dr.DataScience’s contribution

In this case, Dr.DataScience made the following multifaceted contributions to the complex situation the client faced—a small sample size, multiple outcomes, and repeated-measures data.

• Selecting and applying the optimal test method: Considering the characteristics of parametric and nonparametric tests, I provided a reliable evaluation of the pre/post-intervention difference by placing the permutation test—particularly useful with small data—at the core.
• ・Appropriate handling of multiple comparisons: By using the Parallel Gatekeeping Procedure and Holm correction, I managed the risk of detecting false significance through multiple comparisons and ensured rigor in interpreting the results.
• ・Specialized handling of repeated-measures data: By applying an advanced statistical model, generalized estimating equations (GEE), to the repeated measurements before and after the intervention, I achieved an evaluation of the intervention effect that made the most of the data’s characteristics.
• ・Clear interpretation of complex results: By analyzing in detail the differences among the results obtained from multiple statistical methods (e.g., the change in significance before and after correction) and presenting the statistical and clinical implications behind them in an understandable way, I strongly supported the client in making solid, data-based decisions.

Dr.DataScience demonstrates expertise not only in performing analyses but also in presenting the results in a practical form, powerfully advancing the client’s clinical problem-solving and the use of data in the medical setting.