Visualizing Ranks

Introduction

This section documents how to visualize ranks sampled using Algorithm 2 by Gandy & Scott (2021)[GS2021]. For more information, refer to the documentation for the exact rank test.

Simulating Ranks

MCMCTesting.simulate_ranks — Function
simulate_ranks([rng,] test, subject; kwargs...)

Simulate ranks according to the exact rank test strategy.

Arguments

  • rng::Random.AbstractRNG: Random number generator. (Default: Random.default_rng().)
  • test::AbstractMCMCTest: Test strategy.
  • subject::TestSubject: MCMC algorithm and model subject to test.

Keyword Arguments

  • tie_epsilon::Real: The tolerance for declaring a difference in statistic as a tie. (Default: eps(Float32).)
  • statistics: Function for computing test statistics from samples generated from the tests. (See section below for additional description.)
  • show_progress::Bool: Whether to show progress.

Returns

  • ranks::Matrix: The simualted ranks. Each row are the rank samples of each statistic.
source

Visualizing Ranks with Plots

We provide a Plots recipe for visualizing the ranks:

MCMCTesting.rankplot — Function
rankplot(test, ranks; kwargs...)

Plot the simulated ranks using simulate_ranks. If the test subjects are correct, the ranks should visually resemble samples from a uniform distribution. The black horizontal line shows the density of a uniform distribution, while the colored bands are the 1σ, 2σ, 3σ confidence intervals.

Info

The confidence intervals are derived from the normal approximation of binomials observations and assume uniformly-sized bins.

Info

Plots must be imported to use this plot recipe.

Arguments

  • test::ExactRankTest: The exact rank test object used to simulate the ranks.
  • ranks: The output of simulate_rank.

Keyword Arguments

  • stats_names: The name for the statistics used in the rank simulation. The default argument automatically assign default names. (Default: :auto).
  • Keyword arguments corresponding Plots attributes, such as bins, layout, size, may apply.
source

This can be used as follows:

using Plots
using MCMCTesting

# Set the simulation here

ranks = simulate_ranks(test, subject)
rankplot(test, ranks; param_names)

Also refer to the tutorial for a working example.

References

  • GS2021Gandy, A., & Scott, J. (2020). Unit testing for MCMC and other Monte Carlo methods. arXiv preprint arXiv:2001.06465.