Permutation Test | 6 Measures Before and After a Parkinson’s Dance Program

In a small before-and-after comparison without a control group, the shape of the distribution is hard to judge from the data, and if several tests are tried, one of them will be significant. This case study uses a 12-week dance program for people with Parkinson’s disease. I fixed the permutation test as the primary test in advance, compared 6 measures before and after the program, and judged how far the conclusions could go.

The study setting, disease, variables, and numbers in this article were newly created by me to explain how the analysis was carried out. The numbers were calculated from a randomly generated dataset and do not reflect the content or circumstances of any actual client project.

34 participants, 3 time points, and 6 outcomes

The study included 34 people with Parkinson’s disease attending the outpatient clinic of 1 center who took part in a group dance program twice a week for 12 weeks. There was no control group. Measurements were made at 3 time points, before the start, at 12 weeks immediately after the end, and 3 months after the end, and 6 participants had no measurement at 3 months (Table 1).

There were 6 outcomes. A single primary outcome, comfortable gait speed, was fixed in advance, and the balance score, timed up-and-go time, depression score, quality-of-life score, and fatigue score were secondary outcomes.

Table 1. Characteristics of the participants (n=34)
Item Value
Age (years)* 69.5±6.0
Female, no. (%) 14 (41.2)
Severity stage, no. (%)
1 4 (11.8)
2 20 (58.8)
3 10 (29.4)
Disease duration (years)† 5.2 (4.2 to 6.7)
Participants with a measurement at 3 months, no. (%) 28 (82.4)
* Mean±SD.
† Median (interquartile range).

Fixing the permutation test as primary, along with the correction and the handling of standard errors

With 34 participants, the data do not have the power to show whether the distribution of the differences is close to normal. I therefore took as the primary test the permutation test (10,000 sign-flipping permutations, two-sided), which makes no assumption about the shape of the distribution and uses only the fact that, if there is no change, the sign of each participant’s difference is equally likely to go either way.

Running the t-test, the Wilcoxon signed-rank test, and one-sided and two-sided versions of each, and then reporting the convenient one, raises the probability of a chance significant result. The Wilcoxon signed-rank test was shown only as a check, and the size of the difference was presented as the Hodges–Lehmann estimate with its 95% confidence interval.

The 5 secondary outcomes were corrected with both the Holm method and the Benjamini–Hochberg procedure. The Holm method holds to 5% the probability of wrongly declaring a difference for even 1 of the 5. The Benjamini–Hochberg procedure holds to 5% the proportion of errors among those declared different; it picks up differences more readily but accepts that some errors are mixed in.

Furthermore, the course across the 3 time points was examined with generalized estimating equations (GEE), with time point as a factor and adjustment for age and severity stage. Robust standard errors come out too small when the number of participants is small, so I used a small-sample correction (the Mancl–DeRouen method).

Gait speed and quality of life changed at 12 weeks, and quality of life had returned at 3 months

Courses of comfortable gait speed across the 3 time points for each participant shown as thin lines, with the mean shown as a thick line
Figure 1. Individual courses and mean of comfortable gait speed, compared before and after with the permutation test. Thin lines are individual participants, and the thick line is the mean among those with a measurement.
Table 2. Values before the start and at 12 weeks for the 6 outcomes, with estimated differences and tests
P value‡ After correction across the 5 secondary outcomes§
Outcome Before* 12 weeks* Estimated difference (95% CI)† Permutation test Wilcoxon signed-rank test Holm method Benjamini–Hochberg procedure
Comfortable gait speed (m/s) (primary) 1.11±0.19 1.19±0.17 0.07 (0.03 to 0.10) <0.001 <0.001 — —
Balance score 46.1±6.6 46.7±6.5 1.0 (-0.5 to 2.0) 0.27 0.25 1.00 0.48
Timed up-and-go time (s) 10.2±2.9 10.0±3.2 -0.2 (-0.8 to 0.4) 0.38 0.56 1.00 0.48
Depression score 9.7±4.1 10.2±4.9 0.5 (-0.5 to 1.5) 0.36 0.23 1.00 0.48
Quality-of-life score 59.5±14.7 65.1±15.7 5.5 (2.5 to 8.5) <0.001 <0.001 0.001 0.001
Fatigue score 32.5±9.9 32.0±10.0 -0.5 (-3.0 to 2.0) 0.61 0.63 1.00 0.61
* Mean±SD.
† Hodges–Lehmann estimate of the difference, the value at 12 weeks minus the value before the start.
‡ The primary test is the permutation test that flips the signs of the paired differences (10,000 permutations, two-sided). The Wilcoxon signed-rank test is shown as a check.
§ Corrected P values from the permutation test.

The primary outcome, comfortable gait speed, was faster by 0.07 m/s (95% CI, 0.03 to 0.10) at 12 weeks, with P<0.001 by the permutation test and the same conclusion from the Wilcoxon signed-rank test (Table 2). Among the secondary outcomes, the quality-of-life score rose by 5.5 points (2.5 to 8.5), with a corrected P=0.001 by both the Holm method and the Benjamini–Hochberg procedure.

Forest plot of the standardized changes and 95% confidence intervals for the 6 outcomes, arranged so that improvement is to the right
Figure 2. Standardized change at 12 weeks: the mean change divided by the standard deviation of the change, with the direction of improvement set to positive.

For the other 4 outcomes, in contrast, I could not conclude that there was a difference with either correction (Figure 2). The confidence intervals for the standardized changes all included changes of about 0.5, so with 34 participants, the absence of change was not shown either.

Table 3. Analysis of the 3 time points with GEE, and linear regression of the change in gait speed
Item Estimate (95% CI) P value
Comfortable gait speed: change from before the start to 12 weeks 0.07 (0.04 to 0.11) <0.001
Comfortable gait speed: change from before the start to 3 months 0.05 (0.02 to 0.08) 0.001
Quality-of-life score: change from before the start to 12 weeks 5.6 (2.6 to 8.6) <0.001
Quality-of-life score: change from before the start to 3 months 0.6 (-2.6 to 3.8) 0.71
Change in gait speed: gait speed at the start (per 0.1 m/s) -0.029 (-0.048 to -0.009) 0.006
Change in gait speed: age (per 10 years) -0.050 (-0.112 to 0.011) 0.11
Change in gait speed: disease duration (per 1 year) 0.001 (-0.011 to 0.013) 0.86
The top 4 rows are results from generalized estimating equations (GEE), with time point as a factor and adjustment for age and severity stage. A small-sample correction (the Mancl–DeRouen method) was used for the standard errors. Gait speed is in m/s and quality of life in points.
The bottom 3 rows are coefficients from a linear regression explaining the change in gait speed at 12 weeks (m/s) by 3 factors.

In the GEE analysis of the 3 time points, gait speed was still faster than before the start by 0.05 m/s (0.02 to 0.08) at 3 months, but quality of life was at 0.6 points (-2.6 to 3.8; P=0.71), back to the level before the start (Table 3). The small-sample correction increased the standard errors by a factor of 1.03 to 1.04, and the conclusions of this analysis did not change.

In the linear regression of the change in gait speed, a gait speed at the start that was slower by 0.1 m/s was associated with a change larger by 0.029 m/s (P=0.006). However, the same association arises from regression to the mean, in which people who happen to be measured low tend to be measured higher the next time, so this cannot be read as slower walkers improving more.

Taken together, the statistical conclusion is that gait speed and the quality-of-life score were higher at 12 weeks than before the start, but that quality of life had returned by 3 months. Because there was no control group, these data cannot separate whether the changes were due to the program or to the passage of time and familiarity with the measurements.

Work performed and estimated fee

Item Details
Analysis method Permutation test of the before-and-after differences (10,000 permutations, two-sided) and the Wilcoxon signed-rank test, Hodges–Lehmann estimates with 95% confidence intervals, analysis of the 3 time points with GEE (small-sample correction), and linear regression
Additional analyses Correction with the Holm method and the Benjamini–Hochberg procedure (FDR 5%)
Tables prepared 1 table of characteristics, 1 table of the 6 outcomes, 1 table of GEE and linear regression; 3 tables in total
Figures prepared 1 figure of individual courses of gait speed, 1 figure of standardized changes; 2 figures in total
Deliverables Analysis report (reasons for the choice of methods and discussion of the results), tables of results (Excel), figures (PNG and TIFF), and a draft of the statistical analysis section for the paper. Analysis code is available on request for an additional fee (¥22,000, incl. tax)
Estimated time About 2 to 3 weeks from receipt of the data
Estimated fee ¥253,000 (incl. tax)

The table lists the main analyses only. I also carry out the finer analyses needed to follow a sound statistical process, such as checking distributions and cleaning the data; these, together with after-care with no time limit (answers to your questions after delivery), are included in the fee.

Data Scientist Kenichiro Suzuki

Data ScientistKenichiro Suzuki

Free consultationSee fee examples

Recent Posts

Linear Mixed Model | 6-Minute Walk Distance in COPD Rehabilitation

A linear mixed model comparing 6-minute walk distance at four time points with missing data. How to read a non-significant group-by-time interaction.

Cox Regression and Multiple Imputation | Mortality in Hemodialysis

Cox regression of 8 factors and all-cause mortality in new hemodialysis patients, with multiple imputation for missing grip strength and other values.

ROC Analysis | Comparing the AUCs of 4 Blood Markers for Bacteremia

ROC analysis of the AUCs of 4 markers for bacteremia: all pairs tested with the paired DeLong test and corrected with the Benjamini–Hochberg procedure.

Kruskal–Wallis Test | Fatigue and Pain in 5 Rheumatoid Arthritis Groups

Kruskal–Wallis test with corrected pairwise comparisons of fatigue and pain scores among 5 rheumatoid arthritis treatment groups, plus factor analysis.

RMST | Pancreatic Cancer Chemotherapy When Survival Curves Cross

RMST (restricted mean survival time) comparing 2 first-line chemotherapies for advanced pancreatic cancer when survival curves cross, with adjustment.

Multiple Regression | Factors Associated With Dementia Caregiver Burden

Multiple regression with 8 prespecified factors for burden and quality of life in family caregivers of people with dementia, with principal components.

Multiple Regression | Monthly Antimicrobial Use and Resistance Over Time

Multiple regression of monthly antimicrobial use and the proportion of resistant Pseudomonas aeruginosa, after unit root tests and differencing.

Cox Regression | Recurrence After 2 Methods of Atrial Fibrillation Ablation

Cox regression comparing recurrence after 2 methods of catheter ablation for atrial fibrillation, with 6 subgroups read from tests of interaction.