Title: | PEcAn Functions Used for Ecological Forecasts and Reanalysis |
---|---|
Description: | The Predictive Ecosystem Carbon Analyzer (PEcAn) is a scientific workflow management tool that is designed to simplify the management of model parameterization, execution, and analysis. The goal of PECAn is to streamline the interaction between data and models, and to improve the efficacy of scientific investigation. |
Authors: | Mike Dietze [aut], Istem Fer [aut, cre], Ryan Kelly [aut], University of Illinois, NCSA [cph] |
Maintainer: | Istem Fer <istem.fer@fmi.fi> |
License: | BSD_3_clause + file LICENSE |
Version: | 1.8.0.9000 |
Built: | 2024-08-15 13:34:12 UTC |
Source: | https://github.com/PecanProject/pecan |
Run Batch PDA
assim.batch(settings)
assim.batch(settings)
settings |
a PEcAn settings list |
Updated settings list
Ryan Kelly
Automatically calculate and apply burnin value
autoburnin(jags_out, return.burnin = FALSE, ...)
autoburnin(jags_out, return.burnin = FALSE, ...)
jags_out |
JAGS output |
return.burnin |
Logical. If |
... |
Additional arguments for |
Michael Dietze, Alexey Shiklomanov
z1 <- coda::mcmc(c(rnorm(2500, 5), rnorm(2500, 0))) z2 <- coda::mcmc(c(rnorm(2500, -5), rnorm(2500, 0))) z <- coda::mcmc.list(z1, z2) z_burned <- autoburnin(z)
z1 <- coda::mcmc(c(rnorm(2500, 5), rnorm(2500, 0))) z2 <- coda::mcmc(c(rnorm(2500, -5), rnorm(2500, 0))) z <- coda::mcmc.list(z1, z2) z_burned <- autoburnin(z)
bounded
bounded(xnew, rng)
bounded(xnew, rng)
xnew |
new x coordinate |
rng |
range |
calculate.prior
calculate.prior(samples, priors)
calculate.prior(samples, priors)
samples |
Matrix of MCMC samples |
priors |
prior list |
numeric matrix or data.frame
correlationPlot( mat, density = "smooth", thin = "auto", method = "pearson", whichParameters = NULL )
correlationPlot( mat, density = "smooth", thin = "auto", method = "pearson", whichParameters = NULL )
mat |
matrix or data frame of variables |
density |
type of plot to do |
thin |
thinning of the matrix to make things faster. Default is to thin to 5000 |
method |
method for calculating correlations |
whichParameters |
all params or some |
Florian Hartig
The code for the correlation density plot originates from Hartig, F.; Dislich, C.; Wiegand, T. & Huth, A. (2014) Technical Note: Approximate Bayesian parameterization of a process-based tropical forest model. Biogeosciences, 11, 1261-1272.
ddist
ddist(x, prior)
ddist(x, prior)
x |
vector of values (e.g. observations) to be evaluated by the specified probability density function |
prior |
data.frame specifying a prior probability distribution in terms of the distribution name (distn) and first and second parameters (parama, paramb) |
Calculates Gelman diagnostic cumulatively. This is a much more conservative approach than the moving-window method.
gelman_diag_gelmanPlot(x, ...)
gelman_diag_gelmanPlot(x, ...)
x |
MCMC samples |
... |
additional arguments |
Alexey Shiklomanov
Calculate Gelman diagnostic on moving window
gelman_diag_mw( x, width_fraction = 0.1, width = ceiling(coda::niter(x) * width_fraction), njump = 50, include.mpsrf = TRUE, ... )
gelman_diag_mw( x, width_fraction = 0.1, width = ceiling(coda::niter(x) * width_fraction), njump = 50, include.mpsrf = TRUE, ... )
x |
MCMC samples, of class |
width_fraction |
Fractional width of moving window. Default=0.1. |
width |
Width of moving window. Default is niter(x)*width_fraction |
njump |
Number of windows to calculate over |
include.mpsrf |
Whether to calculate multivariate PSRF and include in output (default = FALSE). |
... |
additional arguments |
Gelman Diagnostic 3D array. First dim – mean (1) and 95% confidence (2). Second dim – iteration
Alexey Shiklomanov
Helper function that generates the hierarchical posteriors
generate_hierpost(mcmc.out, prior.fn.all, prior.ind.all)
generate_hierpost(mcmc.out, prior.fn.all, prior.ind.all)
mcmc.out |
hierarchical MCMC outputs |
prior.fn.all |
list of all prior functions |
prior.ind.all |
indices of the targeted params |
hierarchical MCMC outputs in original parameter space
Istem Fer
get_ss
get_ss(gp, xnew, pos.check)
get_ss(gp, xnew, pos.check)
gp |
Gaussian Process |
xnew |
new x coordinate |
pos.check |
check if value needs to be positive (if TRUE, returns -Inf when GP is negative) |
get_y
get_y(SSnew, xnew, llik.fn, priors, llik.par)
get_y(SSnew, xnew, llik.fn, priors, llik.par)
SSnew |
new summary statistic |
xnew |
new x coordinate |
llik.fn |
list that contains likelihood functions |
priors |
prior list |
llik.par |
parameters to be passed llik functions |
Automatically detect burnin based on one of several methods.
getBurnin( jags_out, threshold = 1.1, use.confidence = TRUE, method = "moving.window", plotfile = "/dev/null", ... )
getBurnin( jags_out, threshold = 1.1, use.confidence = TRUE, method = "moving.window", plotfile = "/dev/null", ... )
jags_out |
List of MCMC sample matrices or |
threshold |
Maximum value of Gelman diagnostic |
use.confidence |
Logical. If TRUE (default), use 95% confidence interval for Gelman Diagnostic. If FALSE, use the point estimate. |
method |
Character string indicating method. Options are "moving.window" (default) or "gelman.plot". |
plotfile |
path |
... |
Other parameters to methods |
See "gelman_diag_mw" and "gelman_diag_gelmanPlot"
Alexey Shiklomanov, Michael Dietze
z1 <- coda::mcmc(c(rnorm(2500, 5), rnorm(2500, 0))) z2 <- coda::mcmc(c(rnorm(2500, -5), rnorm(2500, 0))) z <- coda::mcmc.list(z1, z2) burnin <- getBurnin(z, threshold = 1.05)
z1 <- coda::mcmc(c(rnorm(2500, 5), rnorm(2500, 0))) z2 <- coda::mcmc(c(rnorm(2500, -5), rnorm(2500, 0))) z <- coda::mcmc.list(z1, z2) burnin <- getBurnin(z, threshold = 1.05)
Calculates the probability of a set of parameter values, given by xnew
gpeval(xnew, k, mu, tau, psi, x, rng, splinefuns)
gpeval(xnew, k, mu, tau, psi, x, rng, splinefuns)
xnew |
new x coordinate |
k |
Specific absorption coefficient (400 - 2500nm) |
mu |
The mean parameter of the distribution; NOTE this is not equal to the mean |
tau |
spatial var |
psi |
spatial corr |
x |
Name of variable to plot on X axis |
rng |
range |
splinefuns |
spline functions |
Michael Dietze
Hierarchical MCMC
hier.mcmc( settings, gp.stack, nstack = NULL, nmcmc, rng_orig, jmp0, mu_site_init, nparam, nsites, llik.fn, prior.fn.all, prior.ind.all )
hier.mcmc( settings, gp.stack, nstack = NULL, nmcmc, rng_orig, jmp0, mu_site_init, nparam, nsites, llik.fn, prior.fn.all, prior.ind.all )
settings |
a pecan settings list |
gp.stack |
list of GPs |
nstack |
list of number of observations, currently not used |
nmcmc |
number of MCMC iterations |
rng_orig |
range of knots |
jmp0 |
initial jump vars |
mu_site_init |
initial parameter values per site |
nparam |
number of parameters |
nsites |
number of sites |
llik.fn |
list of likelihood functions |
prior.fn.all |
list of prior functions |
prior.ind.all |
indices of targeted parameters |
Istem Fer
is.accepted
is.accepted(ycurr, ynew, format = "lin")
is.accepted(ycurr, ynew, format = "lin")
ycurr |
current value on y axis |
ynew |
new y coordinate |
format |
lin = lnlike fcn, log = log(lnlike) |
Helper function that loads history from previous PDA run, but returns only requested objects
load_pda_history(workdir, ensemble.id, objects)
load_pda_history(workdir, ensemble.id, objects)
workdir |
path of working dir e.g. '/fs/data2/output/PEcAn_***' |
ensemble.id |
ensemble id of a previous PDA run, from which the objects will be retrieved |
objects |
object names that are common to all multi PDA runs, e.g. llik.fn, prior.list etc. |
a list of objects that will be used in joint and hierarchical PDA
Istem Fer
Load Ameriflux L2 Data From NetCDF
Load Dataset for Paramater Data Assimilation
load.L2Ameriflux.cf(file.in) load.pda.data(settings, bety, external.formats = NULL)
load.L2Ameriflux.cf(file.in) load.pda.data(settings, bety, external.formats = NULL)
file.in |
= the netcdf file of L2 data |
settings |
= PEcAn settings list |
bety |
database connection object |
external.formats |
formats list |
A data frame of all variables in the netcdf
A list containg the loaded input data, plus metadata
Ryan Kelly
Ryan Kelly, Istem Fer
Make MCMC list from samples list
makeMCMCList(samps)
makeMCMCList(samps)
samps |
samples list (output from invert.custom) |
Function to sample from a GP model that is assumed to be a -lnLikelihood surface with flat priors and bounded region
mcmc.GP( gp, x0, nmcmc, rng, format = "lin", mix = "joint", splinefuns = NULL, jmp0 = 0.35 * (rng[, 2] - rng[, 1]), ar.target = 0.5, priors = NA, settings, run.block = TRUE, n.of.obs, llik.fn, hyper.pars, resume.list = NULL )
mcmc.GP( gp, x0, nmcmc, rng, format = "lin", mix = "joint", splinefuns = NULL, jmp0 = 0.35 * (rng[, 2] - rng[, 1]), ar.target = 0.5, priors = NA, settings, run.block = TRUE, n.of.obs, llik.fn, hyper.pars, resume.list = NULL )
gp |
Gaussian Process |
x0 |
initial values |
nmcmc |
number of iterations |
rng |
range of knots |
format |
lin = lnlike fcn, log = log(lnlike) |
mix |
each = jump each dim. independently, joint = jump all at once |
splinefuns |
spline functions, not used |
jmp0 |
initial jump variances |
ar.target |
acceptance rate target |
priors |
prior list |
settings |
PEcAn settings list |
run.block |
is this a new run or making the previous chain longer |
n.of.obs |
number of observations |
llik.fn |
list that contains likelihood functions |
hyper.pars |
hyper parameters |
resume.list |
list of needed info if we are running the chain longer |
Michael Dietze
minimize.GP
minimize.GP(gp, rng, x0, splinefuns = NULL)
minimize.GP(gp, rng, x0, splinefuns = NULL)
gp |
Gaussian Process |
rng |
range |
x0 |
initial values |
splinefuns |
spline functions |
Michael Dietze
Adjust PDA MCMC jump size
pda.adjust.jumps(settings, jmp.list, accept.rate, pnames = NULL)
pda.adjust.jumps(settings, jmp.list, accept.rate, pnames = NULL)
settings |
a PEcAn settings list |
jmp.list |
list of jump variances |
accept.rate |
acceptance rate |
pnames |
parameter names |
A PEcAn settings list updated to reflect adjusted jump distributions
Ryan Kelly
Adjust PDA block MCMC jump size
pda.adjust.jumps.bs(settings, jcov, accept.count, params.recent)
pda.adjust.jumps.bs(settings, jcov, accept.count, params.recent)
settings |
a PEcAn settings list |
jcov |
jump covariance matrix |
accept.count |
aceeptance count |
params.recent |
parameters accepted since previous adjustment |
A PEcAn settings list updated to reflect adjusted jump distributions
Ryan Kelly
autocorrelation correction
pda.autocorr.calc(input, model = "heteroskedastic.laplacian")
pda.autocorr.calc(input, model = "heteroskedastic.laplacian")
input |
list that contains time-series data vector and parameters for heteroskedastic.laplacian |
model |
data model type, for flux data heteroskedastic laplacian, normal is an example |
rho AR(1)
Istem Fer
Paramater Data Assimilation using BayesianTools R Package
pda.bayesian.tools( settings, external.data = NULL, external.priors = NULL, external.formats = NULL, ensemble.id = NULL, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, remote = FALSE, ... )
pda.bayesian.tools( settings, external.data = NULL, external.priors = NULL, external.formats = NULL, ensemble.id = NULL, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, remote = FALSE, ... )
settings |
= a pecan settings list |
external.data |
list of external inputs |
external.priors |
list of external priors |
external.formats |
bety formats used when function is used without a DB connection, e.g. remote |
ensemble.id |
ensemble IDs |
params.id |
id of pars |
param.names |
names of pars |
prior.id |
ids of priors |
chain |
how many chains |
iter |
how many iterations |
adapt |
adaptation intervals |
adj.min |
to be used in adjustment |
ar.target |
acceptance rate target |
jvar |
jump variance |
remote |
logical, if TRUE no DB connection is established |
... |
additional arguments |
settings
Istem Fer
Calculate sufficient statistics
pda.calc.error(settings, con, model_out, run.id, inputs, bias.terms)
pda.calc.error(settings, con, model_out, run.id, inputs, bias.terms)
settings |
list |
con |
DB connection |
model_out |
list |
run.id |
run ID |
inputs |
list |
bias.terms |
matrix |
pda.errors
Istem Fer
Calculate Likelihoods for PDA
pda.calc.llik(pda.errors, llik.fn, llik.par)
pda.calc.llik(pda.errors, llik.fn, llik.par)
pda.errors |
calculated errors |
llik.fn |
list of likelihood fcns |
llik.par |
parameters to be passed llik functions |
Total log likelihood (i.e., sum of log likelihoods for each dataset)
Ryan Kelly, Istem Fer
Calculate likelihood parameters
pda.calc.llik.par(settings, n, error.stats, hyper.pars)
pda.calc.llik.par(settings, n, error.stats, hyper.pars)
settings |
list |
n |
named vector, sample sizes of inputs |
error.stats |
list, Sufficient Statistics |
hyper.pars |
list, hyperparameters |
Istem Fer
Helper function for creating log-priors compatible with BayesianTools package
pda.create.btprior(prior.sel)
pda.create.btprior(prior.sel)
prior.sel |
|
prior.sel
must contain the following columns:
distn
– String describing a distribution; e.g. norm
for dnorm
, rnorm
, etc.
parama
, paramb
– First and second parameters, respectively, of the corresponding distribution
Optionally, prior.sel
may also contain the following columns:
param_name
– Parameter name, which will be carried through to the prior object and sampler
lower
, upper
– Lower and upper bounds, respectively. These can be leveraged by the BayesianTools samplers.
best
– Best guess for a parameter estimate. BayesianTools can also use this, though I'm not sure how...
out Prior class object for BayesianTools package
Istem Fer, Alexey Shiklomanov
Create PDA Ensemble
pda.create.ensemble(settings, con, workflow.id)
pda.create.ensemble(settings, con, workflow.id)
settings |
a PEcAn settings list |
con |
DB connection |
workflow.id |
workflow ID |
Ensemble ID of the created ensemble
Ryan Kelly
Define PDA Likelihood Functions
pda.define.llik.fn(settings)
pda.define.llik.fn(settings)
settings |
PEcAn settings list |
List of likelihood functions, one for each dataset to be assimilated against.
Ryan Kelly, Istem Fer
Define PDA Prior Functions
pda.define.prior.fn(prior)
pda.define.prior.fn(prior)
prior |
prior dataframe |
List of prior functions containing dprior, rprior, qprior, dmvprior, rmvprior. Each of these is a list with one distribution function per parameter.
Ryan Kelly
Paramater Data Assimilation using emulator
pda.emulator( settings, external.data = NULL, external.priors = NULL, external.knots = NULL, external.formats = NULL, ensemble.id = NULL, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL, individual = TRUE, remote = FALSE )
pda.emulator( settings, external.data = NULL, external.priors = NULL, external.knots = NULL, external.formats = NULL, ensemble.id = NULL, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL, individual = TRUE, remote = FALSE )
settings |
a pecan settings list |
external.data |
list of external inputs |
external.priors |
list of external priors |
external.knots |
list of external knots |
external.formats |
bety formats used when function is used without a DB connection, e.g. remote |
ensemble.id |
ensemble IDs |
params.id |
id of pars |
param.names |
names of pars |
prior.id |
ids of priors |
chain |
how many chains |
iter |
how many iterations |
adapt |
adaptation intervals |
adj.min |
to be used in adjustment |
ar.target |
acceptance rate target |
jvar |
jump variance |
n.knot |
number of knots requested |
individual |
logical, if TRUE it becomes a site-level PDA |
remote |
logical, if TRUE runs are submitted to remote and objects prepared accordingly |
nothing. Diagnostic plots, MCMC samples, and posterior distributions are saved as files and db records.
Mike Dietze
Ryan Kelly, Istem Fer
Paramater Data Assimilation using emulator on multiple sites in three modes: local, global, hierarchical First draft, not complete yet
pda.emulator.ms(multi.settings)
pda.emulator.ms(multi.settings)
multi.settings |
= a pecan multi-settings list |
settings
Istem Fer
This is a helper function for preparing PDA external objects, but it doesn't cover all the cases yet, use it with care You can use this function just to generate either one of the external.* PDA objects, but note that some args cannot be blank depending on what you aim to generate
pda.generate.externals( external.data = FALSE, obs = NULL, varn = NULL, varid = NULL, n_eff = NULL, align_method = "match_timestep", par = NULL, model_data_diag = FALSE, model.out = NULL, start_date = NULL, end_date = NULL, external.formats = FALSE, external.priors = FALSE, prior.list = NULL, external.knots = FALSE, knots.list = NULL, ind.list = NULL, nknots = NULL )
pda.generate.externals( external.data = FALSE, obs = NULL, varn = NULL, varid = NULL, n_eff = NULL, align_method = "match_timestep", par = NULL, model_data_diag = FALSE, model.out = NULL, start_date = NULL, end_date = NULL, external.formats = FALSE, external.priors = FALSE, prior.list = NULL, external.knots = FALSE, knots.list = NULL, ind.list = NULL, nknots = NULL )
external.data |
boolean, if TRUE function will generate external.data for PDA, then you need to pass varn and obs too, as well as align_method if different than "match_timestep" |
obs |
your data as a(n ordered) list where each sublist corresponds to a data frame of your constraining variable with two columns, variable name - posix IMPORTANT: your obs must be in the same units as PEcAn standards already, this function doesn't do unit conversions! IMPORTANT: your obs must be ready to compare with model outputs in general, e.g. if you're passing flux data it should already be ustar filtered e.g. obs[[1]] NEE posix 4.590273e-09 2017-01-01 00:00:00 NA 2017-01-01 00:30:00 NA 2017-01-01 01:00:00 NA 2017-01-01 01:30:00 NA 2017-01-01 02:00:00 4.575248e-09 2017-01-01 02:30:00 if you have more than variable make sure the order you pass the data is the same as varn. E.g. for varn=c("NEE", "Qle"), external.data should be obs[[1]] NEE posix NA 2018-05-09 NA 2018-05-10 NA 2018-05-11 NA 2018-05-12 ... ... ... obs[[2]] Qle posix NA 2018-05-09 NA 2018-05-10 NA 2018-05-11 NA 2018-05-12 ... ... ... |
varn |
a vector of PEcAn standard variable name(s) to read from model outputs, e.g. c("NEE", "Qle") |
varid |
a vector of BETY variable id(s) of your constraints, e.g. for varn = c("NEE", "Qle"), varid = c(297, 298) |
n_eff |
effective sample size of constraints, PDA functions estimates it for NEE and LE, and uses it in the heteroskedastic Laplacian only, if you already know it passing it now will save you some time |
align_method |
one of the benchmark::align_data align_method options "match_timestep" or "mean_over_larger_timestep", defaults to "match_timestep" |
par |
list with vector sublists of likelihood parameters of heteroskedastic laplacian for flux data, function calculates it if NULL for NEE, FC, and Qle. Leave empty for other variables e.g. AMF.params <- PEcAn.uncertainty::flux.uncertainty(...fill in...) par <- list(c(AMF.params$intercept, AMF.params$slopeP, AMF.params$slopeN)) |
model_data_diag |
optional for diagnostics, if you want to check whether your model and data will be aligned in PDA properly you can return a dataframe as well as plot a quick & dirty timerseries graph |
model.out |
an example model output folder to align your data with model, e.g. "/data/workflows/PEcAn_15000000111/out/15000186876" |
start_date |
the start date of the model.out run, e.g. "2017-01-01" |
end_date |
the end date of the model.out run, e.g. "2018-12-31" |
external.formats |
boolean, if TRUE make sure to pass the varn argument |
external.priors |
boolean, if TRUE pass prior.list argument too |
prior.list |
a list of prior dataframes (one per pft, make sure the order is the same as it is in your <assim.batch> block), if you're using this make sure the targeted parameters are on the list e.g. prior.list <- list(data.frame(distn = c("norm", "beta"), parama = c(4, 1), paramb = c(7,2), n = rep(NA, 2), row.names = c("growth_resp_factor", "leaf_turnover_rate")), data.frame(distn = c("unif", "unif"), parama = c(10, 4), paramb = c(40,27), n = rep(NA, 2), row.names = c("psnTOpt", "half_saturation_PAR"))) |
external.knots |
boolean, if TRUE pass prior.list, ind.list, nknots OR knots.list arguments too |
knots.list |
a list of dataframes (one per pft) where each row is a parameter vector, i.e. training points for the emulator. If not NULL these are used, otherwise knots will be generated using prior.list, ind.list and nknots. |
ind.list |
a named list of vectors (one per pft), where each vector indicates the indices of the parameters on the prior.list targeted in the PDA e.g. ind.list <- list(temperate.deciduous = c(2), temperate.conifer = c(1,2)) |
nknots |
number of knots you want to train the emulator on |
## Not run: pda.externals <- pda.generate.externals(external.data = TRUE, obs = obs, varn = "NEE", varid = 297, n_eff = 106.9386, external.formats = TRUE, model_data_diag = TRUE, model.out = "/tmp/out/outdir", start_date = "2017-01-01", end_date = "2018-12-31") ## End(Not run)
## Not run: pda.externals <- pda.generate.externals(external.data = TRUE, obs = obs, varn = "NEE", varid = 297, n_eff = 106.9386, external.formats = TRUE, model_data_diag = TRUE, model.out = "/tmp/out/outdir", start_date = "2017-01-01", end_date = "2018-12-31") ## End(Not run)
Generate Parameter Knots for PDA Emulator
pda.generate.knots( n.knot, sf, probs.sf, n.param.all, prior.ind, prior.fn, pname )
pda.generate.knots( n.knot, sf, probs.sf, n.param.all, prior.ind, prior.fn, pname )
n.knot |
number of knots |
sf |
scaling factor |
probs.sf |
values for sf |
n.param.all |
number of all params |
prior.ind |
indices of targeted parameters in the prior dataframe |
prior.fn |
list of prior functions |
pname |
name of parameters |
A list of probabilities and parameter values, with one row for each knot in the emulator.
Ryan Kelly, Istem Fer
Generate scaling factor knots for PDA Emulator
pda.generate.sf(n.knot, sf, prior.list)
pda.generate.sf(n.knot, sf, prior.list)
n.knot |
number of knots |
sf |
scaling factor |
prior.list |
list of prior dataframes |
Istem Fer
Get Model Output for PDA
pda.get.model.output(settings, run.id, bety, inputs, external.formats = NULL)
pda.get.model.output(settings, run.id, bety, inputs, external.formats = NULL)
settings |
PEcAn settings list |
run.id |
run ID |
bety |
database connection |
inputs |
inputs list |
external.formats |
format list |
A list containing model outputs extracted to correspond to each observational dataset being used for PDA.
Ryan Kelly, Istem Fer
Initialise Parameter Matrix for PDA
pda.init.params(settings, chain, pname, n.param.all)
pda.init.params(settings, chain, pname, n.param.all)
settings |
a PEcAn settings list |
chain |
number of chain |
pname |
parameter name |
n.param.all |
number of all params |
A list containing 'start' and 'finish' counters for MCMC, as well as the params table, which is an empty matrix concatenated to any param samples from a previous PDA run, if provided.
Ryan Kelly
Initialise Model Runs for PDA
pda.init.run( settings, con, my.write.config, workflow.id, params, n = ifelse(is.null(dim(params)), 1, nrow(params)), run.names = paste("run", 1:n, sep = ".") )
pda.init.run( settings, con, my.write.config, workflow.id, params, n = ifelse(is.null(dim(params)), 1, nrow(params)), run.names = paste("run", 1:n, sep = ".") )
settings |
a PEcAn settings list |
con |
DB connection |
my.write.config |
model write config fcn name |
workflow.id |
workflow ID |
params |
parameters of the run |
n |
number of runs |
run.names |
names of runs |
Vector of run IDs for all model runs that were set up (including write.configs)
Ryan Kelly
Load Priors for Paramater Data Assimilation
pda.load.priors(settings, con, extension.check = FALSE)
pda.load.priors(settings, con, extension.check = FALSE)
settings |
a PEcAn settings list |
con |
database connection |
extension.check |
check if this is another round or longer run |
A previously-generated posterior distribution, to be used as the prior for PDA.
Ryan Kelly, Istem Fer
Paramater Data Assimilation using MCMC
pda.mcmc( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL )
pda.mcmc( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL )
settings |
= a pecan settings list |
params.id |
id of pars |
param.names |
names of pars |
prior.id |
ids of priors |
chain |
how many chains |
iter |
how many iterations |
adapt |
adaptation intervals |
adj.min |
to be used in adjustment |
ar.target |
acceptance rate target |
jvar |
jump variance |
n.knot |
number of knots requested |
Brute-force, only to be used on simple models
nothing. Diagnostic plots, MCMC samples, and posterior distributions are saved as files and db records.
Mike Dietze
Ryan Kelly
Parameter Data Assimilation using MCMC with block sampling
pda.mcmc.bs( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL )
pda.mcmc.bs( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL )
settings |
= a pecan settings list |
params.id |
id of pars |
param.names |
names of pars |
prior.id |
ids of priors |
chain |
how many chains |
iter |
how many iterations |
adapt |
adaptation intervals |
adj.min |
to be used in adjustment |
ar.target |
acceptance rate target |
jvar |
jump variance |
n.knot |
number of knots requested |
Brute-force, only to be used on simple models
nothing. Diagnostic plots, MCMC samples, and posterior distributions are saved as files and db records.
Mike Dietze
Ryan Kelly
Clean up a failed PDA run
pda.mcmc.recover( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL, burnin = NULL )
pda.mcmc.recover( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL, burnin = NULL )
settings |
PEcAn param list |
params.id |
id of pars |
param.names |
names of pars |
prior.id |
ids of priors |
chain |
how many chains |
iter |
how many iterations |
adapt |
adaptation intervals |
adj.min |
to be used in adjustment |
ar.target |
acceptance rate target |
jvar |
jump variance |
n.knot |
number of knots requested |
burnin |
burnin |
An updated settings list
Ryan Kelly
Autocorelation correction and efficient sample size calculation on latent process
pda.neff.calc(inputs, recalculate = FALSE)
pda.neff.calc(inputs, recalculate = FALSE)
inputs |
list |
recalculate |
repeat neff calculation or not |
What we're trying to do is to calculate the autocorrelation of the latent state, after attempting to "remove" the observation error. The first step is thus to estimate the latent state using a simple 'process free' state-space model (e.g. random walk).
inputs list, updated inputs with n_eff
Istem Fer
Plot PDA Parameter Diagnostics
pda.plot.params( settings, mcmc.param.list, prior.ind, par.file.name = NULL, sffx )
pda.plot.params( settings, mcmc.param.list, prior.ind, par.file.name = NULL, sffx )
settings |
PEcAn settings list |
mcmc.param.list |
MCMC param list to be sorted |
prior.ind |
indices of the targeted parameters |
par.file.name |
output file name |
sffx |
suffix to the output file names |
Nothing. Plot is generated and saved to PDF.
Ryan Kelly, Istem Fer
Postprocessing for PDA Results
pda.postprocess( settings, con, mcmc.param.list, pname, prior, prior.ind, sffx = NULL )
pda.postprocess( settings, con, mcmc.param.list, pname, prior, prior.ind, sffx = NULL )
settings |
PEcAn settings list |
con |
DB connection |
mcmc.param.list |
output of PDA MCMC |
pname |
parameter names |
prior |
prior list |
prior.ind |
indices of targeted parameters |
sffx |
suffix to the output files, e.g. "hierarchical" |
PEcAn settings list, updated with <params.id> pointing to the new params file.
Ryan Kelly, Istem Fer
Set PDA Settings
pda.settings( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL, run.round = FALSE )
pda.settings( settings, params.id = NULL, param.names = NULL, prior.id = NULL, chain = NULL, iter = NULL, adapt = NULL, adj.min = NULL, ar.target = NULL, jvar = NULL, n.knot = NULL, run.round = FALSE )
settings |
a PEcAn settings list |
params.id |
id of pars |
param.names |
names of pars |
prior.id |
ids of priors |
chain |
how many chains |
iter |
how many iterations |
adapt |
adaptation intervals |
adj.min |
to be used in adjustment |
ar.target |
acceptance rate target |
jvar |
jump variance |
n.knot |
number of knots requested |
run.round |
another round or not |
An updated settings list
Ryan Kelly, Istem Fer
Helper function for applying BayesianTools specific settings from PEcAn general settings
pda.settings.bt(settings)
pda.settings.bt(settings)
settings |
PEcAn settings |
bt.settings list of BayesianTools::runMCMC settings
Istem Fer
Function to sort Hierarchical MCMC samples
pda.sort.params( mcmc.out, sub.sample = "mu_global_samp", ns = NULL, prior.all, prior.ind.all.ns, sf = NULL, n.param.orig, prior.list, prior.fn.all )
pda.sort.params( mcmc.out, sub.sample = "mu_global_samp", ns = NULL, prior.all, prior.ind.all.ns, sf = NULL, n.param.orig, prior.list, prior.fn.all )
mcmc.out |
MCMC samples |
sub.sample |
which subsample to return |
ns |
site number |
prior.all |
prior dataframe |
prior.ind.all.ns |
indices of targeted parameters on the prior.all dataframe |
sf |
scaling factor if used |
n.param.orig |
original indices of parameters on the prior.list |
prior.list |
list of prior dataframes |
prior.fn.all |
prior functions |
helper function for submitting remote pda runs
prepare_pda_remote(settings, site = 1, multi_site_objects)
prepare_pda_remote(settings, site = 1, multi_site_objects)
settings |
PEcAn settings list |
site |
site number (which site) |
multi_site_objects |
information needed for remote runs |
return_hyperpars
return_hyperpars(assim.settings, inputs)
return_hyperpars(assim.settings, inputs)
assim.settings |
PEcAn settings list |
inputs |
inputs list |
Istem Fer
This is a helper function partly uses pda.emulator code
return_multi_site_objects(multi.settings)
return_multi_site_objects(multi.settings)
multi.settings |
PEcAn multi settings object |
return.bias
return.bias( settings, isbias, model.out, inputs, prior.list.bias, run.round = FALSE, pass2bias = NULL )
return.bias( settings, isbias, model.out, inputs, prior.list.bias, run.round = FALSE, pass2bias = NULL )
settings |
settings list |
isbias |
bias variable index |
model.out |
model output list |
inputs |
inputs list |
prior.list.bias |
prior list, bias prior to be added |
run.round |
extension flag |
pass2bias |
if this is another round, this is re-sampled MCMC samples, will go with the rest of model params |
Istem Fer
Run Batch module
runModule.assim.batch(settings)
runModule.assim.batch(settings)
settings |
a PEcAn settings list |
Helper function to sample from previous MCMC chain while proposing new knots
sample_MCMC( mcmc_path, n.param.orig, prior.ind.orig, n.post.knots, knots.params.temp, prior.list, prior.fn, sf, sf.samp )
sample_MCMC( mcmc_path, n.param.orig, prior.ind.orig, n.post.knots, knots.params.temp, prior.list, prior.fn, sf, sf.samp )
mcmc_path |
path to previous emulator mcmc samples object |
n.param.orig |
vector, number of parameters targeted in each (pft) sublist |
prior.ind.orig |
list, actual indices of parameters targeted in each (pft) sublist |
n.post.knots |
number of new samples requested |
knots.params.temp |
list of parameter samples proposed from the original PDA-prior |
prior.list |
PDA-prior list |
prior.fn |
list for parameter d/r/q/p functions |
sf |
SF parameter names |
sf.samp |
SF parameters MCMC samples |
Istem Fer
helper function for syncing remote pda runs this function resembles remote.copy.from but we don't want to sync everything back
sync_pda_remote(multi.settings, ensembleidlist, register = FALSE)
sync_pda_remote(multi.settings, ensembleidlist, register = FALSE)
multi.settings |
PEcAn multi settings |
ensembleidlist |
ensemble id list for remote runs |
register |
if register==TRUE, the last files returned will be registered to the DB, TO BE DONE |
Function to write posterior distributions of the scaling factors
write_sf_posterior(sf.samp.list, sf.prior, sf.samp.filename)
write_sf_posterior(sf.samp.list, sf.prior, sf.samp.filename)
sf.samp.list |
scaling factor MCMC samples |
sf.prior |
scaling factor prior |
sf.samp.filename |
scaling factor posterior output file name |