Two Sample Hypothesis Tests
Introduction
The two-sample hypothesis testing strategies are based on Algorithm 1 by Gandy & Scott (2021)[gandyandscott2021].
TwoSampleTest
The first basic strategy generates the treatment group
\[\begin{aligned} \theta,\; y_{\text{trtm}} &\sim p\left(\theta, y\right) \\ \theta_{\text{trtm}} &\sim K\left(\theta, \cdot\right), \end{aligned}\]
and the control group as
\[\begin{aligned} \theta_{\text{ctrl}},\; y_{\text{ctrl}} &\sim p\left(\theta, y\right), \\ \end{aligned}\]
where the test compares
\[(\theta_{\text{trtm}}, \, y_{\text{trtm}}) \quad\text{versus}\quad (\theta_{\text{ctrl}}, \, y_{\text{ctrl}}).\]
MCMCTesting.TwoSampleTest
— TypeTwoSampleTest(n_samples, n_mcmc_steps; n_control, n_treatment, n_mcmc_thin)
Two-sample hypothesis testing strategy. Algorithm 1 in Gandy & Scott (2021).
Arguments
n_samples::Int
: Number of samples from the jointp(θ, y)
used for the computing the p-values.n_mcmc_steps::Int
: Number of times the MCMC kernel is applied to initial sample from the joint. (Increasing this value improves the power of the test.)
Keyword Arguments
n_control::Int
: Number of pure samples from the joint (control group). (Default:n_samples
)n_treatment::Int
: Number of samples from the MCMC kernel (treatment group). (Default:n_samples
)n_mcmc_thin::Int
: Number of thinning applied to the MCMC chain. The effect of this argument is the same asn_mcmc_steps
.
Returns
pvalues
: P-value computed for each dimension of the statistic returned fromstatistics
.
Requirements
This test requires the following functions for model
and kernel
to be implemented:
markovchain_transition
sample_joint
Keyword Arguments for Tests
When calling mcmctest
or seqmcmctest
, this tests has an additional keyword argument:
two_sample_test_pvalue
: The p-value calculation strategy.
The default strategy is an approximate two-sample Kolmogorov-Smirnov test. Any function returning a single p-value from a two-sample hypothesis test will work. The format is as follows:
two_sample_test_pvalue(x::AbstractVector, y::AbstractVector)::Real
TwoSampleGibbsTest
The second strategy performs applies an additional Gibbs sampling step when generating the treatment group as
\[\begin{aligned} \theta,\; y &\sim p\left(\theta, y\right) \\ \theta_{\text{trtm}} &\sim K\left(\theta, \cdot\right), \\ y_{\text{trtm}} &\sim p\left(y \mid \theta_{\text{trtm}}\right) \end{aligned}\]
resulting in the treatment group
\[(\theta_{\text{trtm}}, \, y_{\text{trtm}}).\]
The control group is generated the same as TwoSampleTest
.
To use this test, the user needs to implement the following interface for simulating from the conditional likelihood.
MCMCTesting.sample_predictive
— Functionsample_predictive(rng, model, θ)
Sample from the predictive distribution of model
conditionally on θ
Arguments
rng::Random.AbstractRNG
: Random number generator.model
: Model subject to test.θ
: Model parameters to condition on.
Returns
y
: Data generated from conditionally onθ
fromp(y|θ)
MCMCTesting.TwoSampleGibbsTest
— TypeTwoSampleTest(n_samples, n_mcmc_steps; n_control, n_treatment, n_mcmc_thin)
Two-sample hypothesis testing strategy with an additional Gibbs sampling step. Modified version of Algorithm 1 in Gandy & Scott 2021 for increased power.
Arguments
n_samples::Int
: Number of samples from the jointp(θ, y)
used for the computing the p-values.n_mcmc_steps::Int
: Number of times the MCMC kernel is applied to initial sample from the joint. (Increasing this value improves the power of the test.)
Keyword Arguments
n_control::Int
: Number of pure samples from the joint (control group). (Default:n_samples
)n_treatment::Int
: Number of samples from the MCMC kernel (treatment group). (Default:n_samples
)n_mcmc_thin::Int
: Number of thinning applied to the MCMC chain. The effect of this argument is the same asn_mcmc_steps
.
Returns
pvalues
: P-value computed for each dimension of the statistic returned fromstatistics
.
Requirements
This test requires the following functions for model
and kernel
to be implemented:
markovchain_transition
sample_joint
sample_predictive
Keyword Arguments for Tests
When calling mcmctest
or seqmcmctest
, this tests has an additional keyword argument:
two_sample_test_pvalue
: The p-value calculation strategy.
The default strategy is an approximate two-sample Kolmogorov-Smirnov test. Any function returning a p-value to two sample groups will work. The format is as follows:
two_sample_test_pvalue(x::AbstractVector, y::AbstractVector)::Real
References
References
- gandyandscott2021Gandy, A., & Scott, J. (2020). Unit testing for MCMC and other Monte Carlo methods. arXiv preprint arXiv:2001.06465.