Package 'pcal'

Title: Calibration of P-Values for Point Null Hypotheses
Description: Calibrate p-values under a robust perspective using the methods developed by Sellke, Bayarri, and Berger (2001) <doi:10.1198/000313001300339950> and obtain measures of the evidence provided by the data in favor of point null hypotheses which are safer and more straightforward to interpret.
Authors: Pedro Fonseca [aut, cre] , Rui Paulo [ctb, ths] , FCT [fnd]
Maintainer: Pedro Fonseca <[email protected]>
License: MIT + file LICENSE
Version: 1.1.0
Built: 2024-10-18 04:00:17 UTC
Source: https://github.com/pedro-teles-fonseca/pcal

Help Index

Lower Bounds on Bayes Factors for Point Null Hypotheses

Description

Calibrate p-values under a robust Bayesian perspective so that they can be directly interpreted as lower bounds on Bayes factors in favor of point null hypotheses.

Usage

bcal(p)

Arguments

p

A numeric vector with values in the [0,1] interval.

Details

bcal is a vectorized implementation of the calibration of p-values into lower bounds for Bayes factors developed by Sellke et al. (2001). The calibration is:

B(p)=−e p log(p)B(p) = -e \, p \, log(p)

for p<1/ep < 1/e, where pp is a p-value on a classical test statistic, and B(p)=1B(p) = 1 otherwise. B(p)B(p) should be interpreted as an approximation to the lower bound of the Bayes factor (in favor of the null hypothesis) that is found by changing the prior distribution of the parameter of interest (under the alternative hypothesis) over wide classes of distributions.

Sellke et al. (2001) noted that a scenario in which they definitely recommend this calibration is when investigating fit to the null model/hypothesis with no explicit alternative in mind. Pericchi and Torres (2011) warn that despite the usefulness and appropriateness of this p-value calibration it does not depend on sample size and hence the lower bounds obtained with large samples may be conservative.

Value

bcal returns a numeric vector with the same length as p. A warning message is thrown if there are NA or NaN values in p.

References

Pericchi L, Torres D (2011). “Quick anomaly detection by the Newcomb—Benford law, with applications to electoral processes data from the USA, Puerto Rico and Venezuela.” Statistical Science, 26(4), 502–516.

Sellke T, Bayarri MJ, Berger JO (2001). “Calibration of p values for testing precise null hypotheses.” The American Statistician, 55(1), 62–71.

See Also

  • bfactor_interpret for the interpretation of Bayes factors.

  • bfactor_to_prob to turn Bayes factors into posterior probabilities.

  • pcal for a p-value calibration that returns lower bounds for the posterior probabilities of point null hypotheses.

Examples

# Calibration of a typical "threshold" p-value:
# ----------------------------------------------------------------
bcal(.05)

# Calibration of typical "threshold" p-values:
# ----------------------------------------------------------------
bcal(c(.1, .05, .01, .005, .001))

# Application: chi-squared goodness-of-fit test,
# lower bound on the Bayes factor in favor of the null hypothesis:
# ----------------------------------------------------------------
data <- matrix(c(18, 12, 10, 12, 10, 23), ncol = 2)
bcal(chisq.test(data)[["p.value"]])

Interpretation of Bayes factors

Description

Quantify the strength of the evidence provided by the data to a model/hypothesis according to Bayes factor interpretation scales suggested by Jeffreys (1961) and Kass and Raftery (1995).

Usage

bfactor_interpret(bf, scale = "jeffreys")

Arguments

bf

A numeric vector of non-negative values.
scale

A string of characters specifying either "jeffreys" or "kass-raftery". Not case sensitive.

Details

Bayes factors are a summary of the evidence provided by the data to a model/hypothesis. Jeffreys (1961) suggested the interpretation of Bayes factors in half-units on the base 10 logarithmic scale, as indicated in the following table:

log10(Bayes factor) Bayes factor Evidence
[-Inf, 0[ [0, 1[ Negative
[0, 0.5[ [1, 3.2[ Weak
[0.5, 1[ [3.2, 10[ Substantial
[1, 1.5[ [10, 32[ Strong
[1.5, 2[ [32, 100[ Very Strong
[2, +Inf[ [100, +Inf[ Decisive

By default, bfactor_interpret takes Bayes factors as input and returns the strength of the evidence in favor of the model/hypothesis in the numerator of the Bayes factors (usually the null hypothesis) according to the aforementioned table.

Alternatively, and because it can be useful to consider twice the natural logarithm of the Bayes factor (which is in the same scale as the familiar deviance and likelihood ratio test statistics), Kass and Raftery (1995) suggested the following scale:

2*log(Bayes factor) Bayes factor Evidence
[-Inf, 0[ [0, 1[ Negative
[0, 2[ [1, 3[ Weak
[2, 6[ [3, 20[ Positive
[6, 10[ [20, 150[ Strong
[10, +Inf[ [150, +Inf[ Very Strong

To interpret Bayes factors according to the latter table use scale = "kass-raftery".

When comparing Bayes factors with results from standard likelihood ratio tests it is convenient to put the null hypothesis in the denominator of the Bayes factor so that bfactor_interpret returns the strength of the evidence against the null hypothesis. If bf was obtained with the null hypothesis on the numerator, one can use bfactor_interpret(1/bf) or bfactor_interpret(1/bf, scale = "kass-raftery") to obtain the strength of the evidence against the null hypothesis.

Value

Returns a character vector with the same length as bf.

References

Jeffreys H (1961). Theory of probability, Oxford Classic Texts In The Physical Sciences, 3 edition. Oxford University Press.

Kass RE, Raftery AE (1995). “Bayes factors.” Journal of the American Statistical Association, 90(430), 773–795.

See Also

  • bfactor_log_interpret for the interpretation of the logarithms of Bayes factors.

  • bfactor_to_prob to turn Bayes factors into posterior probabilities.

  • bcal for a p-value calibration that returns lower bounds on Bayes factors in favor of point null hypotheses.

Examples

# Interpretation of one Bayes factor
bfactor_interpret(4)
bfactor_interpret(4, scale = "jeffreys")
bfactor_interpret(4, scale = "kass-raftery")

# Interpretation of more than one Bayes factors
bfactor_interpret(c(0.1, 1.2, 3.5, 13.9, 150))
bfactor_interpret(c(0.1, 1.2, 3.5, 13.9, 150), scale = "jeffreys")
bfactor_interpret(c(0.1, 1.2, 3.5, 13.9, 150), scale = "kass-raftery")

# Application: chi-squared goodness-of-fit test.
# Strength of the evidence provided by the lower
# bound of the Bayes factor in favor of the null hypothesis:
data <- matrix(c(18, 12, 10, 12, 10, 23), ncol = 2)
bfactor_interpret(bcal(chisq.test(data)[["p.value"]]))

Interpretation of Bayes factors

Description

Quantify the strength of the evidence provided by the data to a model/hypothesis according a Bayes factor interpretation scale suggested by Kass and Raftery (1995).

Usage

bfactor_interpret_kr(bf)

Arguments

bf

A numeric vector of non-negative values.

Details

Bayes factors are a summary of the evidence provided by the data to a model/hypothesis. Because it can be useful to consider twice the natural logarithm of the Bayes factor, which is in the same scale as the familiar deviance and likelihood ratio test statistics, Kass and Raftery (1995) suggested the following Bayes factor interpretation scale:

2*log(Bayes factor) Bayes factor Evidence
[-Inf, 0[ [0, 1[ Negative
[0, 2[ [1, 3[ Weak
[2, 6[ [3, 20[ Positive
[6, 10[ [20, 150[ Strong
[10, +Inf[ [150, +Inf[ Very strong

bfactor_interpret_kr takes Bayes factors as input and returns the strength of the evidence in favor of the model/hypothesis in the numerator of the Bayes factors (usually the null hypothesis) according to the aforementioned table.

When comparing results with those from standard likelihood ratio tests, it is convenient to put the null hypothesis in the denominator of the Bayes factor so that bfactor_interpret_kr returns the strength of the evidence against the null hypothesis. If bf was obtained with the null hypothesis on the numerator, one can use bfactor_interpret_kr(1/bf) to obtain the strength of the evidence against the null hypothesis.

Value

Returns a character vector with the same length as bf.

References

Kass RE, Raftery AE (1995). “Bayes factors.” Journal of the American Statistical Association, 90(430), 773–795.

See Also

Examples

# Interpretation of one Bayes factor
bfactor_interpret_kr(1.5)

# Interpretation of many Bayes factors
bfactor_interpret_kr(c(0.1, 1.2, 3.5, 13.9, 150))

# Application: chi-squared goodness-of-fit test.
# Strength of the evidence provided by the lower
# bound on the Bayes factor in favor of the null hypothesis:
x <- matrix(c(12, 15, 14, 15), ncol = 2)
bfactor_interpret_kr(bcal(chisq.test(x)[["p.value"]]))

Interpretation of the logarithms of Bayes factors

Description

Quantify the strength of the evidence provided by the data to a model/hypothesis according to Bayes factor interpretation scales suggested by Jeffreys (1961) and Kass and Raftery (1995).

Usage

bfactor_log_interpret(bf, scale = "jeffreys", base = exp(1))

Arguments

bf

A numeric vector.
scale

A string of characters specifying either "jeffreys" or "kass-raftery". Not case sensitive.
base

A numeric vector of length one. Must be a positive number.

Details

Bayes factors are a summary of the evidence provided by the data to a model/hypothesis, and are often reported on a logarithmic scale. Jeffreys (1961) suggested the interpretation of Bayes factors in half-units on the base 10 logarithmic scale, as indicated in the following table:

log10(Bayes factor) Bayes factor Evidence
[-Inf, 0[ [0, 1[ Negative
[0, 0.5[ [1, 3.2[ Weak
[0.5, 1[ [3.2, 10[ Substantial
[1, 1.5[ [10, 32[ Strong
[1.5, 2[ [32, 100[ Very Strong
[2, +Inf[ [100, +Inf[ Decisive

By default, bfactor_log_interpret takes (base base) logarithms of Bayes factors as input and returns the strength of the evidence in favor of the model/hypothesis in the numerator of the Bayes factors (usually the null hypothesis) according to the aforementioned table.

Alternatively, and because it can be useful to consider twice the natural logarithm of the Bayes factor (which is in the same scale as the familiar deviance and likelihood ratio test statistics), Kass and Raftery (1995) suggested the following scale:

2*log(Bayes factor) Bayes factor Evidence
[-Inf, 0[ [0, 1[ Negative
[0, 2[ [1, 3[ Weak
[2, 6[ [3, 20[ Positive
[6, 10[ [20, 150[ Strong
[10, +Inf[ [150, +Inf[ Very Strong

To interpret base base logarithms of Bayes factors according to the latter table use scale = "kass-raftery".

When comparing Bayes factors with results from standard likelihood ratio tests it is convenient to put the null hypothesis in the denominator of the Bayes factor so that bfactor_log_interpret returns the strength of the evidence against the null hypothesis. If bf was obtained with the null hypothesis on the numerator, one can use bfactor_log_interpret(1/bf) or bfactor_log_interpret(1/bf, scale = "kass-raftery") to obtain the strength of the evidence against the null hypothesis.

Value

Returns a character vector with the same length as bf.

References

Jeffreys H (1961). Theory of probability, Oxford Classic Texts In The Physical Sciences, 3 edition. Oxford University Press.

Kass RE, Raftery AE (1995). “Bayes factors.” Journal of the American Statistical Association, 90(430), 773–795.

See Also

  • bfactor_interpret for the interpretation of Bayes factors in levels.

  • bfactor_to_prob to turn Bayes factors into posterior probabilities.

  • bcal for a p-value calibration that returns lower bounds on Bayes factors in favor of point null hypotheses.

Examples

# Interpretation of the natural log of a Bayes factors
bfactor_log_interpret(log(1.5))
bfactor_log_interpret(log(1.5), scale = "jeffreys")
bfactor_log_interpret(log(1.5), scale = "kass-raftery")

# Interpretation of the natural log of many Bayes factors
bfactor_log_interpret(log(c(0.1, 1.2, 3.5, 13.9, 150)))
bfactor_log_interpret(log(c(0.1, 1.2, 3.5, 13.9, 150)), scale = "kass-raftery")

# Interpretation of the base-10 log of many Bayes factors
bfactor_log_interpret(log10(c(0.1, 1.2, 3.5, 13.9, 150)), base = 10)
bfactor_log_interpret(log10(c(0.1, 1.2, 3.5, 13.9, 150)), base = 10, scale = "kass-raftery")

Interpretation of the logarithms of Bayes factors

Description

Quantify the strength of the evidence provided by the data to a model/hypothesis according a Bayes factor interpretation scale suggested by Kass and Raftery (1995).

Usage

bfactor_log_interpret_kr(bf, base = exp(1))

Arguments

bf

A numeric vector.
base

A numeric vector of length one. Must be a positive number.

Details

Bayes factors are a summary of the evidence provided by the data to a model/hypothesis, and are often reported on a logarithmic scale. Because it can be useful to consider twice the natural logarithm of the Bayes factor, which is in the same scale as the familiar deviance and likelihood ratio test statistics, Kass and Raftery (1995) suggested the following Bayes factor interpretation scale:

2*log(Bayes factor) Bayes factor Evidence
[-Inf, 0[ [0, 1[ Negative
[0, 2[ [1, 3[ Weak
[2, 6[ [3, 20[ Positive
[6, 10[ [20, 150[ Strong
[10, +Inf[ [150, +Inf[ Very strong

bfactor_log_interpret_kr takes (base base) logarithms of Bayes factors as input and returns the strength of the evidence provided by the data in favor of the model/hypothesis in the numerator of the Bayes factors (usually the null hypothesis) according to the aforementioned table.

When comparing results with those from standard likelihood ratio tests, it is convenient to put the null hypothesis in the denominator of the Bayes factor so that bfactor_log_interpret_kr returns the strength of the evidence against the null hypothesis. If bf was obtained with the null hypothesis on the numerator, one can use bfactor_log_interpret_kr(1/bf) to obtain the strength of the evidence against the null hypothesis.

Value

Returns a character vector with the same length as bf.

References

Kass RE, Raftery AE (1995). “Bayes factors.” Journal of the American Statistical Association, 90(430), 773–795.

See Also

Examples

# Interpretation of one Bayes factor, natural log
bfactor_log_interpret_kr(1.5)

# Interpretation of many Bayes factors, natural log
bfactor_log_interpret_kr(c(0.1, 1.2, 3.5, 13.9, 150))

# Interpretation of many Bayes factors, base 10 log
bfactor_log_interpret_kr(log10(c(0.1, 1.2, 3.5, 13.9, 150)), base = 10)

Turn Bayes Factors Into Posterior Probabilities

Description

Update prior probabilities of models/hypotheses to posterior probabilities using Bayes factors.

Usage

bfactor_to_prob(bf, prior_prob = 0.5)

Arguments

bf

A numeric vector of non-negative values.
prior_prob

A numeric vector with values in the [0,1] interval. If length(bf) == 1 then prior_prob can be of any positive length, but if length(bf) > 1 then the length of prior_prob can only be 1 or equal to the length of bf.

Details

bfactor_to_prob turns Bayes factors into posterior probabilities using a vectorized version of the following equation from Berger and Delampady (1987):

P(H0∣x)=(1+1−π0π0 1B01(x))−1P(H_0 | x) = \left( 1 + \frac{1 - \pi_0}{\pi_0} \, \frac{1}{B_{01}(x)} \right)^{-1}

where B01(x)B_{01}(x) is a Bayes factor in favor of the null hypothesis (given the data xx), π0\pi_0 is the prior probability of the null hypothesis and 1−π01 - \pi_0 is the prior probability of the alternative hypothesis.

If bf is a vector of Bayes factors (in favor of the null hypothesis) and prior_prob is a vector with the prior probabilities of those hypotheses then bfactor_to_prob(bf, prior_prob) updates prior_prob to posterior probabilities. The posterior probabilities of the alternative hypotheses can be obtained with 1 - bfactor_to_prob(bf, prior_prob).

The prior_prob argument is optional and is set to 0.5 by default, implying prior equiprobability of hypotheses. prior_prob can only be of length equal to length(bf), in which case each prior probability in prior_prob will be updated using the corresponding element of bf, or of length 1, in which case it will be recycled (if length(bf) > 1) and each element of bf will update the same prior_prob value.

Value

If length(bf) > 1 then bfactor_to_prob returns a numeric vector with the same length as bf, otherwise it returns a numeric vector with the same length as prior_prob. Warning messages are thrown if there are NA or NaN values in bf or in prior_prob.

References

Berger JO, Delampady M (1987). “Testing precise hypotheses.” Statistical Science, 2(3), 317–335.

See Also

Examples

# With a Bayes factor that is indifferent between the null
# and the alternative hypotheses:
# --------------------------------------------------------
bfactor_to_prob(1)

# Same as above but the null hypothesis has high prior probability:
# -----------------------------------------------------------------
bfactor_to_prob(1, .99)

# Posterior probability of the null hypothesis as a function
# of the prior probability:
# -----------------------------------------------------------------
bfactor_to_prob(1, seq(.5, 1, .1))

# With Bayes factors that favor the null hypothesis:
# -----------------------------------------------------------------
round(bfactor_to_prob(seq(2, 50, 2.5)), 3)

# Same as above but the null hypothesis has low prior probability:
# -----------------------------------------------------------------
round(bfactor_to_prob(seq(2, 50, 2.5), prior_prob = .01), 3)

# Posterior probabilities obtained with Bayes factors that
# favor the alternative hypothesis:
# -----------------------------------------------------------------
round(bfactor_to_prob(seq(0, 1, .05)), 3)

# Same as above but the null hypothesis has high prior probability:
# -----------------------------------------------------------------
round(bfactor_to_prob(seq(0, 1, .05), prior_prob = .99), 3)

# Application: chi-squared goodness-of-fit test,
# lower bound on the posterior probability of the null hypothesis:
# -----------------------------------------------------------------
x <- matrix(c(12, 41, 25, 33), ncol = 2)
bfactor_to_prob(bcal(chisq.test(x)[["p.value"]]), prior_prob = .9)

Lower Bounds on Posterior Probabilities for Point Null Hypotheses

Description

Calibrate p-values under a robust perspective so that they can be directly interpreted as either lower bounds on the posterior probabilities of point null hypotheses or lower bounds on the probabilities of type I errors.

Usage

pcal(p, prior_prob = 0.5)

Arguments

p

A numeric vector with values in the [0,1] interval.
prior_prob

A numeric vector with values in the [0,1] interval. If length(p) == 1 then prior_prob can be of any positive length, but if length(p) > 1 then the length of prior_prob can only be 1 or equal to the length of p.

Details

pcal is a vectorized implementation of the calibration of p-values into lower bounds for the posterior probabilities of point null hypotheses (or lower bounds for the probabilities of type I errors) developed by Sellke et al. (2001). The calibration is:

α(p)=(1+[−e p log(p)]−1)−1\alpha(p) = (1 + \lbrack -e \, p \, log(p) \rbrack^{-1})^{-1}

where pp is a p-value on a classical test statistic. This calibration assumes that both the null and the alternative hypotheses have 0.5 prior probability. We generalized the aforementioned calibration for prior probabilities other than 0.5:

α(p)=(1+1−π0π0 [−e p log(p)]−1)−1\alpha(p) = \left( 1 + \frac{1 - \pi_0}{\pi_0} \, \lbrack -e \, p \, log(p) \rbrack^{-1} \right)^{-1}

where π0\pi_0 is the prior probability of the null hypothesis and 1−π01 - \pi_0 is the prior probability of the alternative hypothesis.

For each element of p, pcal returns an approximation of the smallest posterior probability of the null hypothesis that is found by changing the prior distribution of the parameter of interest (under the alternative hypothesis) over wide classes of distributions. Alternatively, the output of pcal can also be interpreted as lower bounds on the probabilities of type I errors, which means that this calibration has both Bayesian and Frequentist interpretations. Sellke et al. (2001) noted that a scenario in which they definitely recommend this calibration is when investigating fit to the null model with no explicit alternative in mind. Pericchi and Torres (2011) warn that despite the usefulness and appropriateness of this p-value calibration it does not depend on sample size, and hence the lower bounds obtained with large samples may be conservative.

The prior_prob argument is optional and is set to 0.5 by default, implying prior equiprobability of hypotheses. prior_prob can only be of length equal to the length of p, in which case each prior probability in prior_prob is used in the calibration of the corresponding element of p, or of length 1, in which case it will be recycled (if length(p) > 1) and the same prior_prob value is used in the calibration of all the elements of p.

Note that pcal(p, prior_prob) is equivalent to bfactor_to_prob(bcal(p), prior_prob).

Value

If length(p) > 1 then pcal returns a numeric vector with the same length as p, otherwise it returns a numeric vector with the same length as prior_prob. Warning messages are thrown if there are NA or NaN values in p or in prior_prob.

References

Pericchi L, Torres D (2011). “Quick anomaly detection by the Newcomb—Benford law, with applications to electoral processes data from the USA, Puerto Rico and Venezuela.” Statistical Science, 26(4), 502–516.

Sellke T, Bayarri MJ, Berger JO (2001). “Calibration of p values for testing precise null hypotheses.” The American Statistician, 55(1), 62–71.

See Also

  • bcal for a p-value calibration that returns lower bounds on Bayes factors in favor of point null hypotheses.

Examples

# Calibration of a typical "threshold" p-value:
# ----------------------------------------------------------------
pcal(.05)

# Calibration of typical "threshold" p-values:
# ----------------------------------------------------------------
pcal(c(.1, .05, .01, .005, .001))

# Application: chi-squared goodness-of-fit test,
# lower bound on the posterior probability of the null hypothesis:
# ----------------------------------------------------------------
data <- matrix(c(18, 12, 10, 12, 10, 23), ncol = 2)
pcal(chisq.test(data)[["p.value"]])