From: "Rajanikanth Madabushi" <rajanim@ufl.edu>
Subject:[NMusers] FO vs FOCE vs LAPLACIAN
Date: Tue, 15 Jul 2003 19:28:12 -0400

Hi all,
I have sparse PK data( two points per individual) following i.v
bolus admnistration of a drug known to follow two compartment
model. I am using ADVAN3 TRANS4 subroutine. 
 
When i use FO method, the minimization is terminated and
the following error message appears
 
MINIMIZATION TERMINATED
DUE TO PROXIMITY OF NEXT ITERATION EST. TO A VALUE
AT WHICH THE OBJ. FUNC. IS INFINITE
AT THE LAST COMPUTED INFINITE VALUE OF THE OBJ. FUNCT.:
PRED EXIT CODE = 1
 
When FOCE method is used, minimization is successful.
 
When the LAPLACIAN method is used,minimization is successful but
 
MINIMIZATION SUCCESSFUL
R MATRIX ALGORITHMICALLY SINGULAR
AND ALGORITHMICALLY NON-POSITIVE-SEMIDEFINITE
COVARIANCE STEP ABORTED
 
Is it not usually considered that FO method is suitable for a sparse data
situation while FOCE for a rich data situation?  My question is why did
FO method not converge successfully when FOCE and LAPLACIAN methods
gives successful minimization?
 
The other question was in the event of successful minimization obtained
for all the three of the above methods but if the parameter estimates
for the parameters are quite different, which method of estimation
be adopted??
 
Thanking in advance.
 
Raj
 
Rajanikanth Madabushi
Post Doc Assoc
Department of Pharmaceutics
University of Florida
Gainesville, FL 32610
_______________________________________________________

From: Nick Holford <n.holford@auckland.ac.nz>
Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN
Date: Wed, 16 Jul 2003 13:50:48 +1200
S
nmusers@globomaxnm.com

Raj,

IMHO FO should usually be avoided. Its approximations are known to
be worse than FOCE. FOCE also has its problems but they may not be
as bad. I think the myth about FO being suitable for sparse data has
arisen because when you sparse data you can learn very little and it
therefore hard to be mislead too far by FO. 

The LAPLACIAN option is probably preferable over simple FOCE because
it works harder at getting what should be the right answer. 

Don't sweat about $COV. The output is hardly worth the electrons
used to generate it. There is no evidence that I know of that getting
$COV to run is a reliable sign of a better model. My experience has
often been the opposite. Often only crummy and naively simple models
run with $COV and more sensible models which clearly fit the data better
(based on eyeball tests of predictions matching observations) will fall
over with the kind of error you report below.

Nick
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

_______________________________________________________

From: "Mike Davenport" <mike.davenport@richmond.ppdi.com>
Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN
Date: Wed, 16 Jul 2003 08:03:39 -0400

My basic understanding of NONMEM is that both the FO and
FOCE methods are using a first order Taylor series expansion procedure.

If this is true, then I my recollection is that the function needs
to be 1st and 2nd order differentiable. The 2nd derivatives form the
basis of Fisher Information matrix, which is your variance-covariance
matrix. If this matrix is NON-POSITIVE-SEMIDEFINITE then the matrix is
not invertable, the variance and covariances are not estimable and the
parameter estimates would not be any good either. To me, in a naive
sense, this seems like reporting a mean without
any measure of dispersion.

What am I missing on this??

thanks
mike
_______________________________________________________

From: Nick Holford <n.holford@auckland.ac.nz>
Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN
Date: Fri, 18 Jul 2003 19:16:28 +1200

Mike,

You are correct that NONMEM relies on approximations to get
its estimates. However, the measure of imprecision that might
be provided by the covariance step is also affected by these
same approximations so getting NONMEM's asymptotic SE does not
help to correct for any bias arising from the approximations. I do
not agree that because the FIM is not invertable that the parameter
estimates are not any good. It is only the estimates of the uncertainty
of the parameters under the assumptions used to derive asymptotic
standard errors that is a problem.

That said, if you want a better estimate of parameter imprecision
(under the assumption that the approximation is not too harmful) then
I would recommend using the bootstrap method (http://wfn.sourceforge.net/wfnbs.htm).
This at least does not require one to make the assumption that the parameter
uncertainty is normally distributed when one wants to define a confidence
interval and is more or less guaranteed to give you some idea of the confidence
interval. A second method is to use the likelihood profile but this depends on
the assumption that changes in objective function are chi-squared distributed
under the null -- an assumption known to be false (especially for FO).

Nick
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/
_______________________________________________________

From: "Kowalski, Ken" <Ken.Kowalski@pfizer.com>
Subject: RE: [NMusers] Re: FO vs FOCE vs LAPLACIAN
Date: Fri, 18 Jul 2003 08:38:12 -0400

Nick,

I know we're rehashing old ground...but I can't resist.  Too often the value
of $COV is equated solely with getting SEs and Wald-based CIs which we know
often don't have the right coverage probabilites.  This to me is not the
real value of the $COV.  When the $COV fails, the hessian (R-matrix) may be
singular and/or non-positive semi-definite indicating that the estimates may
not have converged to a global optimum and thus they may not be
(approximate) maximum likelihood estimates.  That being said, I agree with
you that this does not mean the estimates aren't any good.  However, I do
think it means we should proceed cautiously because they might not be good
(particularly for extrapolation).  In this situation you have previously
advocated bootstrapping to assess the stability and predictive performance
of the model.  I think this a good course of action and a cautious way to
proceed.  Many who do not place any value in the $COV argue that they often
obtain reasonable estimates even when the $COV fails.  I argue that they are
making use of prior information to make that assessment.  If so, I would
much rather be more explicit in using that prior information via fixing
parameters, fitting a penalized likelihood, or using a fully Bayesian
estimation procedure to remove the instability in the model fit.  The real
value of the $COV is as a diagnostic to assess instability.  Of course, a
successful $COV does not guarantee that the model is stable nor does it
guarantee that the estimates have converged to a global optimum.  It is
merely a diagnostic to assess stablity and provides some indication as to
when we should proceed cautiously.  You may think I am jumping through a lot
more hoops needlessly to obtain stable models but to me it is no more work
than bootstrapping every model with a failed $COV in the development
process.  I realize that we are going to have to agree to disagree...just
don't take $COV away from NONMEM VI as I'm willing to pay for those extra
electrons.  

Ken
_______________________________________________________

From: Nick Holford <n.holford@auckland.ac.nz>
Subject: [NMusers] Re: FO vs FOCE vs LAPLACIAN
Date: Tue, 22 Jul 2003 15:45:40 +1200

Ken,

I am primarily interested in avoiding local minima (so that I can test model building
hypotheses) and obtaining minimally biased and imprecise parameter estimates. I agree
with you that success or failure of $COV probably does not help diagnose a local minimum
problem. I have no evidence to support this. But what about bias and imprecision?

I have a somewhat anecdotal but nevertheless evidence based comment on this.
I recently completed a PK model analysis using WT, AGE, SCR and SEX as
covariates (697 subjects, 2567 concs). The model did not run $COV, in fact
it didn't even minimize successfully. Other evidence convinced me it was not
far away from an appropriate minimum and because it had a more biologically sound
basis than its more successful neighbours I preferred this model. I bootstrapped
the original data set using the preferred model and found 28% of 1055 bootstrap
runs minimized successfully and 7.1% ran the $COV step. 

The mean of the parameters obtained from all bootstrap runs and the mean from those which
ran the $COV step were all within 2%. I conclude that $COV does not indicate lower bias
compared with runs that do not minimize. 

To assess imprecision I computed the ratio of the mean standard error from the $COV
successful runs to the bootstrap standard error obtained from all runs. For THETA:se
estimates the $COV SE was on average 3% smaller but for OMEGA:se the $COV SE was 58%
larger than the overall bootstrap SE. I conclude from this that the imprecision of
THETA:se was negligibly different when the $COV step was successful. The difference
in the OMEGA :se may reflect the intrinsic difficulty in obtaining estimates of OMEGA
and OMEGA:se. Perhaps the asymptotic assumptions involved in $COV produce an upward bias.

95% confidence intervals obtained from all the bootstrap runs were very similar to
those obtained from minimization successful and $COV successful runs. The 95% CI
predicted from the asymptotic SE was on average 21% larger (range 15-35%) than the
bootstrap CI.
 
In order to explore the issue a bit further I simulated a data set using the mean
bootstrap parameter estimates from all runs. I then bootstrapped this simulated
data set (1772 runs). The minimization success rate was double (56%) that of the
original real data bootstrap runs and 12.5% ran $COV.

Because the true parameter values for the simulation are known the absolute bias
can be computed. Only 3 out of 29 parameters had an absolute bias larger than 10%. There
were negligible differences between the absolute bias using estimates from all runs, 
minimization successful runs or $COV successful runs. This means the $COV step is not
a guide to reduced bias.

The imprecision pattern was similar with the simulated data but the magnitude of
differences between the mean $COV SE and the mean bootstrap SE were larger than
those seen with the original real dataset. For $COV SE the THETA:se estimates were
about 50% smaller while OMEGA:se were 400% larger than the bootstrap SE. There were
no real differences depending on whether all runs, minimization successful or $COV
successful runs were used ($COV successful runs tended to be a bit larger).

95% confidence intervals obtained from all the bootstrap runs on the simulated dataset
were very similar to those obtained from minimization successful and $COV successful
runs. The 95% CI predicted from the asymptotic SE was on average 22% larger (range
14-46%) than the bootstrap CI.

My conclusion from this empirical exploration of one data set and model suggests that
a successful $COV is of no value for selection of models with improved bias or
imprecision. It is a quicker way of obtaining some idea of the parameter 95% confidence
interval but it is upwardly biased compared with the bootstrap estimate. I am not
typically interested in parameter CIs for every model I run. I am happy to leave that
until I have finished model building and prefer to rely on bootstrap CIs.

I think we are in agreement on almost all issues that you raise except for the diagnostic
value of the $COV in relation to the thing you call "stability". I dont know what
stability means so perhaps you would like to offer a definition and some evidence
for your assertion. 

Nick

PS Just in case anyone else it tempted to try this kind of experiment it took just about
2 months continuous operation on a 1.7 GHz Athlon MP2000 to do 2827 bootstraps. I'm still
waiting for a response from the journal editor about the MS describing the preferred model
so I had the time to do the computation while visiting PAGE etc.
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

_______________________________________________________

From: "Kowalski, Ken" <Ken.Kowalski@pfizer.com>
Subject: RE: [NMusers] Re: FO vs FOCE vs LAPLACIAN
Date: Tue, 22 Jul 2003 12:50:46 -0400

Nick, 

Wow!...Talk about proceeding cautiously.  See below for my response to your
comments.

Ken

-----Original Message-----
From: Nick Holford [mailto:n.holford@auckland.ac.nz]
Sent: Monday, July 21, 2003 11:46 PM
To: nmusers@globomaxnm.com
Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN


Ken,

I am primarily interested in avoiding local minima (so that I can test model
building hypotheses) and obtaining minimally biased and imprecise parameter
estimates. I agree with you that success or failure of $COV probably does
not help diagnose a local minimum problem. I have no evidence to support
this. But what about bias and imprecision?

[Ken Kowalski]  Actually, what I said was that a successful $COV does not
guarantee that one has converged to a global minimum.  However, a failure of
$COV should raise a concern that we might not have converged to a global
minimum.  That is, conditions may be ripe for converging to a local minimum.
I have encountered many instances (as I'm sure you have as well) where for
two hierarchical models, the bigger model converges to a higher OFV than the
smaller model (ie., delta-OFV is negative).  In these instances the bigger
model has clearly converged to a local minimum and often the $COV will fail
for the bigger model.  However, I have also seen instances where the $COV
runs successfully for the bigger model but inspection of the output
indicates large SEs for one or more parameters (as well as high pairwise
correlations of the estimates) and this is the diagnostic information that
leads one to be concerned that the bigger model is over-parameterized,
leading to an ill-conditioning of the R-matrix (e.g., nearly singular) and
instability in the estimation.  The bigger model is unstable in the sense
that if I change the starting values, I may converge to a different set of
final estimates.  When the likelihood surface for the bigger model is very
flat, there may be many sets of solutions leading to very similar values of
the minimum OFV.  In this setting we may get a minimimization failure, or
the $COV may fail due to a non-singular R-matrix, or if $COV does run some
of the parameters will have very large SEs.  In this sense the $COV does
provide information on the precision or lack thereof (imprecision) in the
estimation.  One can't use the $COV to assess bias if the true model is
unknown, however, if the model is unstable we should be concerned that we
might have biased estimates.

I have a somewhat anecdotal but nevertheless evidence based comment on this.
I recently completed a PK model analysis using WT, AGE, SCR and SEX as
covariates (697 subjects, 2567 concs). The model did not run $COV, in fact
it didn't even minimize successfully. Other evidence convinced me it was not
far away from an appropriate minimum and because it had a more biologically
sound basis than its more successful neighbours I preferred this model. I
bootstrapped the original data set using the preferred model and found 28%
of 1055 bootstrap runs minimized successfully and 7.1% ran the $COV step. 

The mean of the parameters obtained from all bootstrap runs and the mean
from those which ran the $COV step were all within 2%. I conclude that $COV
does not indicate lower bias compared with runs that do not minimize. 

[Ken Kowalski] It concerns me that such a low proportion minimized
successfully.  What happens if you substantially change your starting
values?  You might find than one or more of the bootstrap parameter means
are substantially different.  We can't assess bias from bootstrapping a real
data set where the true model for the data is unknown.  Just because the
$COV step ran that does not mean your model is not over-parameterized.
Those runs where the $COV was successful may be numerically nonsingular but
still nearly singular (output from $COV can help assess this).  Thus, it
doesn't surprise me that the mean estimates aren't different between those
where the $COV was successful and all the runs.  The value of the $COV goes
well beyond a simple success/failure flag.  When the $COV is successful we
still need to inspect the output from the $COV to assess the stability.
When the $COV fails we don't have this luxury but we get warning messages
regarding invertability problems with the hessian that indicate we have a
stability problem.  Again, if we have a stability problem we may continue to
proceed with the over-parameterized model but we should do so cautiously.

To assess imprecision I computed the ratio of the mean standard error from
the $COV successful runs to the bootstrap standard error obtained from all
runs. For THETA:se estimates the $COV SE was on average 3% smaller but for
OMEGA:se the $COV SE was 58% larger than the overall bootstrap SE. I
conclude from this that the imprecision of THETA:se was negligibly different
when the $COV step was successful. The difference in the OMEGA :se may
reflect the intrinsic difficulty in obtaining estimates of OMEGA and
OMEGA:se. Perhaps the asymptotic assumptions involved in $COV produce an
upward bias.

[Ken Kowalski]  I believe Mats Karlsson has shown that asymptotic SEs for
elements of Omega are not very good. 

95% confidence intervals obtained from all the bootstrap runs were very
similar to those obtained from minimization successful and $COV successful
runs. The 95% CI predicted from the asymptotic SE was on average 21% larger
(range 15-35%) than the bootstrap CI.
 
In order to explore the issue a bit further I simulated a data set using the
mean bootstrap parameter estimates from all runs. I then bootstrapped this
simulated data set (1772 runs). The minimization success rate was double
(56%) that of the original real data bootstrap runs and 12.5% ran $COV.

Because the true parameter values for the simulation are known the absolute
bias can be computed. Only 3 out of 29 parameters had an absolute bias
larger than 10%. There were negligible differences between the absolute bias
using estimates from all runs, minimization successful runs or $COV
successful runs. 

[Ken Kowalski] The fact that you get 44% minimization failures and 87.5%
$COV failures when you bootstrapped your simulated data set, based on the
model developed from your original data set, provides evidence that your
model is unstable. This is what I claim the $COV step failure from your
original model fit was diagnosing.  You indicate that 3 out of 29 parameters
had an absolute bias greater than 10%.  Perhaps the instability in your
model is related to the estimates of these parameters.  If they are not
important parameters then perhaps I wouldn't be concerned.  To illustrate my
point consider the simple example where we have very little sampling in the
absorption phase.  Perhaps in each individual the observed Tmax corresponds
to the first observation.  In this setting we can have convergence and/or
$COV failures and wildly biased estimates for ka (e.g., an estimate of ka

>>100 hr^-1).  Fortunately, we may find that even though ka is not well

estimated we can still get relatively accurate estimates of CL/F which we
may be more interested in.  Thus, the instability in the model is related to
the estimation of ka and the limitations of the design at early time points.
I would be inclined to fix ka at some prior estimate (if I had one) to
remove the instability and obtain successful minimization and $COV and
acknowledge the limitations of the design/model to estimate ka.
Alternatively, we could use simulation/bootstrapping to verify that poor
(biased) estimation of ka is not likely to unduly bias CL/F.  Further
evaluation perhaps using simulation and bootstrapping is a cautious way to
proceed when considering over-parameterized models.  For me, I like to fit
alternative models (often reduced hierarchical models) as a set of
diagnostic runs and inspect the $COV output so that I can understand where
the limitations are with the design/data/model.  Both approaches can help us
stay out of trouble.

This means the $COV step is not a guide to reduced bias.

[Ken Kowalski]  The $COV should not be used as a guide to reduce bias.  One
can fit a mispecified reduced (smaller) model that is quite stable with a
successful $COV and get biased estimates just as one can fit the true model
and get biased estimates if the design/data doesn't support fitting all the
parameters of the true model (ie., the true model may be over-parameterized
resulting in a failed $COV).  If the reduced model is not very plausible we
may discard it or recognize its limitations particularly for extrapolation.
However, it is naive to simply trust an over-parameterized model fit simply
because the model is more plausible and proceed without caution.  A true but
over-parameterized model fit may have difficulty in estimation due to the
limitations of the design to support the model.  If the fit converges to a
local minimum we still need to be concerned about bias even though we are
fitting the true model.  If you have a strong belief that the
over-parameterized model is more plausible this is where explicity
incorporating prior information on parameters that may be difficult to
estimate from the existing design/data may be helpful.   

The imprecision pattern was similar with the simulated data but the
magnitude of differences between the mean $COV SE and the mean bootstrap SE
were larger than those seen with the original real dataset. For $COV SE the
THETA:se estimates were about 50% smaller while OMEGA:se were 400% larger
than the bootstrap SE. There were no real differences depending on whether
all runs, minimization successful or $COV successful runs were used ($COV
successful runs tended to be a bit larger).

95% confidence intervals obtained from all the bootstrap runs on the
simulated dataset were very similar to those obtained from minimization
successful and $COV successful runs. The 95% CI predicted from the
asymptotic SE was on average 22% larger (range 14-46%) than the bootstrap
CI.

My conclusion from this empirical exploration of one data set and model
suggests that a successful $COV is of no value for selection of models with
improved bias or imprecision. It is a quicker way of obtaining some idea of
the parameter 95% confidence interval but it is upwardly biased compared
with the bootstrap estimate. I am not typically interested in parameter CIs
for every model I run. I am happy to leave that until I have finished model
building and prefer to rely on bootstrap CIs.

I think we are in agreement on almost all issues that you raise except for
the diagnostic value of the $COV in relation to the thing you call
"stability". I dont know what stability means so perhaps you would like to
offer a definition and some evidence for your assertion. 

[Ken Kowalski]  Hopefully the above responses give you some sense of what I
mean by stability.  Over-parameterization of the model (too many parameters
estimated relative to the information content of the data/design),
ill-conditioning of the hessian (R-matrix) which can lead to numerically
unstable inversion of the R-matrix ($COV failures) and instability of the
paramater estimation (overly sensitive to the starting values) are all
symptoms of a stability problem with the model.

_______________________________________________________