Simulation and Estimation Diagnostics
Model diagnostics are used to check if the model fits the data well. Diagnostics for continuous data are different from for discrete data.
The inspect
Function
The main function to call when looking at diagnostics from a fit is inspect
with the signature:
Pumas.inspect
— Functioninspect(fpm::AbstractFittedPumasModel; wres_approx::LikelihoodApproximation, nsim::Int, rng)
Output a summary of the model predictions, residuals, Empirical Bayes estimates, and NPDEs (when requested).
Called on a fit
output and allows the keyword argument wres_approx
for approximation method to be used in residual calculation. The default value is the approximation method used for the marginal likelihood calculation in the fit
that produced fpm
. The keyword nsim
controls the number of times each subject is simulated for npde
computation. A FittedPumasModelInspection
object with pred
, wres
, ebes
, ewres
, and npdes
is output.
The function outputs a summary of the model predictions, residuals, Empirical Bayes estimates, and Normalised Prediction Distribution Errors (NPDEs) and Expected Weighted RESiduals (EWRES) (when requested). It is called on a fit output and allows the keyword argument wres_approx
for approximation method to be used in residual calculation. The default value is the approximation method used for the marginal likelihood calculation in the fit that produced fpm
. The keyword nsim
controls the number of times each subject is simulated for the npde
and ewres
computations. rng
is a random number generator that can be used to seed any random elements such as those in npde
and ewres
computations.
inspect
returns a FittedPumasModelInspection
object with the
- model predictions
- residuals
- empirical Bayes estimates
- individual coefficients
- dose control parameters
- NPDEs and EWRES with
nsim
set
Example syntax
my_inspect = inspect(my_fit)
my_inspect_df = DataFrame(my_inspect)
The result of inspect(my_fit)
is a FittedPumasModelInspection
object, which can then be converted into tabular form using the DataFrame
call. This is useful if one is interested to export the results into a spreadsheet, or use the resulting DataFrame
for plotting.
Weighted Residuals
In Pumas, you can calculate individual and population weighted residuals using wresiduals
as well as expected individual weighted residuals with the signature
Pumas.wresiduals
— Functionwresiduals(fpm::AbstractFittedPumasModel, approx::LikelihoodApproximation; nsim=nothing)
Calculate the individual and population weighted residual.
Takes a fit
result, an approximation method for the marginal likelihood calculation which defaults to the method used in the fit
and the number of simulations with the keyword argument nsim
. If nsim
is specified only the Expected Simulation based Individual Weighted Residuals (EIWRES) is included in the output as individual residual and population residual is not computed. Using the FO
approximation method corresponds to the WRES and while FOCE(I)
corresponds to CWRES. The output is a SubjectResidual
object that stores the population (wres
) and individual (iwres
) residuals along with the subject
and approximation method (approx
).
Takes a fit result, an approximation method for the marginal likelihood calculation which defaults to the method used in the fit and the number of simulations with the keyword argument nsim
. If nsim
is specified only the Expected Simulation based Individual Weighted Residuals (EIWRES) is included in the output as individual residual and population residual is not computed. Using the FO approximation method corresponds to the WRES and while FOCE corresponds to CWRES. The output is a SubjectResidual
object that stores the population (wres) and individual (iwres) residuals along with the subject and approximation method (approx).
Predictions
To calculate predictions from a fit use the predict
function with the signature:
StatsAPI.predict
— Functionpredict(
model::AbstractPumasModel,
subject::Subject,
param::NamedTuple,
[randeffs,];
[obstimes::AbstractVector,
diffeq_options::NamedTuple]
)::Union{SubjectPrediction,Vector{SubjectPrediction}}
Compute population and individual predictions for the single subject
based on model
and the population parameters param
. A NamedTuple
of random effects, randeffs
, can be omitted or provided by the user. If they are omitted, they will be estimated from the data in the subject
.
If the optional obstimes
argument is passed then the time points in obstimes
are used for the predictions. Otherwise, the time points of the observations of the subject
are used for the predictions.
The function allows for extra keyword arguments to be passed on to the differential equations solver through the diffeq_options
keyword. See the online documentation for more details.
predict(
fpm::AbstractFittedPumasModel,
[population::Union{Subject,Population}];
[obstimes::AbstractVector]
)::Union{SubjectPrediction,Vector{SubjectPrediction}}
Compute population and individual predictions for the fitted model fpm
. By default, the predictions are computed for the estimation data but the predictions can also be computed for user supplied data by passing either a single subject or a vector of subjects (Population
) as the population
argument.
If the optional obstimes
argument is passed then the time points in obstimes
are used for the predictions. Otherwise, the time points of the observations for each subject in the population
are used for the predictions.
Any optional keyword arguments used when fitting fpm
are reused when computing the predictions.
Model Metrics
Akaike information criterion:
StatsAPI.aic
— Functionaic(fpm::AbstractFittedPumasModel)
Calculate the Akaike information criterion (AIC) of the fitted Pumas model fpm
.
Bayesian information criterion:
StatsAPI.bic
— Functionbic(fpm::AbstractFittedPumasModel)
Calculate the Bayesian information criterion (BIC) of the fitted Pumas model fpm
.
Shrinkage
Pumas.ηshrinkage
— Functionηshrinkage(fpm::AbstractFittedPumasModel)
Calculate the η-shrinkage.
Takes the result of a fit
as the only input argument. A named tuple of the random effects and corresponding η-shrinkage values is output.
Pumas.ϵshrinkage
— Functionϵshrinkage(fpm::AbstractFittedPumasModel)
Calculate the ϵ-shrinkage.
Takes the result of a fit
as the only input argument. A named tuple of derived variables and corresponding ϵ-shrinkage values is output.
If the observations for an individual is informative, η-shrinkage will be low. If, on the other hand, the observations are non-informative, the individual estimates (EBEs) will shrink towards the population mean. Hence, the variance of the EBEs will also shrink and η-shrinkage will be high.
When data gets sparse the IWRES distribution also shrink towards it’s mean (zero). This is often called overfitting or ϵ-shrinkage.