In a small before-and-after comparison without a control group, the shape of the distribution is hard to judge from the data, and if several tests are tried, one of them will be significant. This case study uses a 12-week dance program for people with Parkinson’s disease. I fixed the permutation test as the primary test in advance, compared 6 measures before and after the program, and judged how far the conclusions could go.
The study setting, disease, variables, and numbers in this article were newly created by me to explain how the analysis was carried out. The numbers were calculated from a randomly generated dataset and do not reflect the content or circumstances of any actual client project.
34 participants, 3 time points, and 6 outcomes
The study included 34 people with Parkinson’s disease attending the outpatient clinic of 1 center who took part in a group dance program twice a week for 12 weeks. There was no control group. Measurements were made at 3 time points, before the start, at 12 weeks immediately after the end, and 3 months after the end, and 6 participants had no measurement at 3 months (Table 1).
There were 6 outcomes. A single primary outcome, comfortable gait speed, was fixed in advance, and the balance score, timed up-and-go time, depression score, quality-of-life score, and fatigue score were secondary outcomes.
| Item | Value |
|---|---|
| Age (years)* | 69.5±6.0 |
| Female, no. (%) | 14 (41.2) |
| Severity stage, no. (%) | |
| 1 | 4 (11.8) |
| 2 | 20 (58.8) |
| 3 | 10 (29.4) |
| Disease duration (years)† | 5.2 (4.2 to 6.7) |
| Participants with a measurement at 3 months, no. (%) | 28 (82.4) |
† Median (interquartile range).
Fixing the permutation test as primary, along with the correction and the handling of standard errors
With 34 participants, the data do not have the power to show whether the distribution of the differences is close to normal. I therefore took as the primary test the permutation test (10,000 sign-flipping permutations, two-sided), which makes no assumption about the shape of the distribution and uses only the fact that, if there is no change, the sign of each participant’s difference is equally likely to go either way.
Running the t-test, the Wilcoxon signed-rank test, and one-sided and two-sided versions of each, and then reporting the convenient one, raises the probability of a chance significant result. The Wilcoxon signed-rank test was shown only as a check, and the size of the difference was presented as the Hodges–Lehmann estimate with its 95% confidence interval.
The 5 secondary outcomes were corrected with both the Holm method and the Benjamini–Hochberg procedure. The Holm method holds to 5% the probability of wrongly declaring a difference for even 1 of the 5. The Benjamini–Hochberg procedure holds to 5% the proportion of errors among those declared different; it picks up differences more readily but accepts that some errors are mixed in.
Furthermore, the course across the 3 time points was examined with generalized estimating equations (GEE), with time point as a factor and adjustment for age and severity stage. Robust standard errors come out too small when the number of participants is small, so I used a small-sample correction (the Mancl–DeRouen method).
Gait speed and quality of life changed at 12 weeks, and quality of life had returned at 3 months

| P value‡ | After correction across the 5 secondary outcomes§ | ||||||
|---|---|---|---|---|---|---|---|
| Outcome | Before* | 12 weeks* | Estimated difference (95% CI)† | Permutation test | Wilcoxon signed-rank test | Holm method | Benjamini–Hochberg procedure |
| Comfortable gait speed (m/s) (primary) | 1.11±0.19 | 1.19±0.17 | 0.07 (0.03 to 0.10) | <0.001 | <0.001 | — | — |
| Balance score | 46.1±6.6 | 46.7±6.5 | 1.0 (-0.5 to 2.0) | 0.27 | 0.25 | 1.00 | 0.48 |
| Timed up-and-go time (s) | 10.2±2.9 | 10.0±3.2 | -0.2 (-0.8 to 0.4) | 0.38 | 0.56 | 1.00 | 0.48 |
| Depression score | 9.7±4.1 | 10.2±4.9 | 0.5 (-0.5 to 1.5) | 0.36 | 0.23 | 1.00 | 0.48 |
| Quality-of-life score | 59.5±14.7 | 65.1±15.7 | 5.5 (2.5 to 8.5) | <0.001 | <0.001 | 0.001 | 0.001 |
| Fatigue score | 32.5±9.9 | 32.0±10.0 | -0.5 (-3.0 to 2.0) | 0.61 | 0.63 | 1.00 | 0.61 |
† Hodges–Lehmann estimate of the difference, the value at 12 weeks minus the value before the start.
‡ The primary test is the permutation test that flips the signs of the paired differences (10,000 permutations, two-sided). The Wilcoxon signed-rank test is shown as a check.
§ Corrected P values from the permutation test.
The primary outcome, comfortable gait speed, was faster by 0.07 m/s (95% CI, 0.03 to 0.10) at 12 weeks, with P<0.001 by the permutation test and the same conclusion from the Wilcoxon signed-rank test (Table 2). Among the secondary outcomes, the quality-of-life score rose by 5.5 points (2.5 to 8.5), with a corrected P=0.001 by both the Holm method and the Benjamini–Hochberg procedure.

For the other 4 outcomes, in contrast, I could not conclude that there was a difference with either correction (Figure 2). The confidence intervals for the standardized changes all included changes of about 0.5, so with 34 participants, the absence of change was not shown either.
| Item | Estimate (95% CI) | P value |
|---|---|---|
| Comfortable gait speed: change from before the start to 12 weeks | 0.07 (0.04 to 0.11) | <0.001 |
| Comfortable gait speed: change from before the start to 3 months | 0.05 (0.02 to 0.08) | 0.001 |
| Quality-of-life score: change from before the start to 12 weeks | 5.6 (2.6 to 8.6) | <0.001 |
| Quality-of-life score: change from before the start to 3 months | 0.6 (-2.6 to 3.8) | 0.71 |
| Change in gait speed: gait speed at the start (per 0.1 m/s) | -0.029 (-0.048 to -0.009) | 0.006 |
| Change in gait speed: age (per 10 years) | -0.050 (-0.112 to 0.011) | 0.11 |
| Change in gait speed: disease duration (per 1 year) | 0.001 (-0.011 to 0.013) | 0.86 |
The bottom 3 rows are coefficients from a linear regression explaining the change in gait speed at 12 weeks (m/s) by 3 factors.
In the GEE analysis of the 3 time points, gait speed was still faster than before the start by 0.05 m/s (0.02 to 0.08) at 3 months, but quality of life was at 0.6 points (-2.6 to 3.8; P=0.71), back to the level before the start (Table 3). The small-sample correction increased the standard errors by a factor of 1.03 to 1.04, and the conclusions of this analysis did not change.
In the linear regression of the change in gait speed, a gait speed at the start that was slower by 0.1 m/s was associated with a change larger by 0.029 m/s (P=0.006). However, the same association arises from regression to the mean, in which people who happen to be measured low tend to be measured higher the next time, so this cannot be read as slower walkers improving more.
Taken together, the statistical conclusion is that gait speed and the quality-of-life score were higher at 12 weeks than before the start, but that quality of life had returned by 3 months. Because there was no control group, these data cannot separate whether the changes were due to the program or to the passage of time and familiarity with the measurements.
Work performed and estimated fee
| Item | Details |
|---|---|
| Analysis method | Permutation test of the before-and-after differences (10,000 permutations, two-sided) and the Wilcoxon signed-rank test, Hodges–Lehmann estimates with 95% confidence intervals, analysis of the 3 time points with GEE (small-sample correction), and linear regression |
| Additional analyses | Correction with the Holm method and the Benjamini–Hochberg procedure (FDR 5%) |
| Tables prepared | 1 table of characteristics, 1 table of the 6 outcomes, 1 table of GEE and linear regression; 3 tables in total |
| Figures prepared | 1 figure of individual courses of gait speed, 1 figure of standardized changes; 2 figures in total |
| Deliverables | Analysis report (reasons for the choice of methods and discussion of the results), tables of results (Excel), figures (PNG and TIFF), and a draft of the statistical analysis section for the paper. Analysis code is available on request for an additional fee (¥22,000, incl. tax) |
| Estimated time | About 2 to 3 weeks from receipt of the data |
| Estimated fee | ¥253,000 (incl. tax) |
The table lists the main analyses only. I also carry out the finer analyses needed to follow a sound statistical process, such as checking distributions and cleaning the data; these, together with after-care with no time limit (answers to your questions after delivery), are included in the fee.

