Visualizing Ranks
Introduction
This section documents how to visualize ranks sampled using Algorithm 2 by Gandy & Scott (2021)[GS2021]. For more information, refer to the documentation for the exact rank test.
Simulating Ranks
MCMCTesting.simulate_ranks
— Functionsimulate_ranks([rng,] test, subject; kwargs...)
Simulate ranks according to the exact rank test strategy.
Arguments
rng::Random.AbstractRNG
: Random number generator. (Default:Random.default_rng()
.)test::AbstractMCMCTest
: Test strategy.subject::TestSubject
: MCMC algorithm and model subject to test.
Keyword Arguments
tie_epsilon::Real
: The tolerance for declaring a difference in statistic as a tie. (Default:eps(Float32)
.)statistics
: Function for computing test statistics from samples generated from the tests. (See section below for additional description.)show_progress::Bool
: Whether to show progress.
Returns
- ranks::Matrix: The simualted ranks. Each row are the rank samples of each statistic.
Visualizing Ranks with Plots
We provide a Plots
recipe for visualizing the ranks:
MCMCTesting.rankplot
— Functionrankplot(test, ranks; kwargs...)
Plot the simulated ranks using simulate_ranks
. If the test subjects are correct, the ranks should visually resemble samples from a uniform distribution. The black horizontal line shows the density of a uniform distribution, while the colored bands are the 1σ, 2σ, 3σ confidence intervals.
The confidence intervals are derived from the normal approximation of binomials observations and assume uniformly-sized bins.
Plots
must be imported to use this plot recipe.
Arguments
test::ExactRankTest
: The exact rank test object used to simulate the ranks.ranks
: The output ofsimulate_rank
.
Keyword Arguments
stats_names
: The name for the statistics used in the rank simulation. The default argument automatically assign default names. (Default: :auto).- Keyword arguments corresponding
Plots
attributes, such asbins
,layout
,size
, may apply.
This can be used as follows:
using Plots
using MCMCTesting
# Set the simulation here
ranks = simulate_ranks(test, subject)
rankplot(test, ranks; param_names)
Also refer to the tutorial for a working example.
References
- GS2021Gandy, A., & Scott, J. (2020). Unit testing for MCMC and other Monte Carlo methods. arXiv preprint arXiv:2001.06465.