From: "Rajanikanth Madabushi" <rajanim@ufl.edu> Subject:[NMusers] FO vs FOCE vs LAPLACIAN Date: Tue, 15 Jul 2003 19:28:12 -0400 Hi all, I have sparse PK data( two points per individual) following i.v bolus admnistration of a drug known to follow two compartment model. I am using ADVAN3 TRANS4 subroutine. When i use FO method, the minimization is terminated and the following error message appears MINIMIZATION TERMINATED DUE TO PROXIMITY OF NEXT ITERATION EST. TO A VALUE AT WHICH THE OBJ. FUNC. IS INFINITE AT THE LAST COMPUTED INFINITE VALUE OF THE OBJ. FUNCT.: PRED EXIT CODE = 1 When FOCE method is used, minimization is successful. When the LAPLACIAN method is used,minimization is successful but MINIMIZATION SUCCESSFUL R MATRIX ALGORITHMICALLY SINGULAR AND ALGORITHMICALLY NON-POSITIVE-SEMIDEFINITE COVARIANCE STEP ABORTED Is it not usually considered that FO method is suitable for a sparse data situation while FOCE for a rich data situation? My question is why did FO method not converge successfully when FOCE and LAPLACIAN methods gives successful minimization? The other question was in the event of successful minimization obtained for all the three of the above methods but if the parameter estimates for the parameters are quite different, which method of estimation be adopted?? Thanking in advance. Raj Rajanikanth Madabushi Post Doc Assoc Department of Pharmaceutics University of Florida Gainesville, FL 32610 _______________________________________________________ From: Nick Holford <n.holford@auckland.ac.nz> Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN Date: Wed, 16 Jul 2003 13:50:48 +1200 S nmusers@globomaxnm.com Raj, IMHO FO should usually be avoided. Its approximations are known to be worse than FOCE. FOCE also has its problems but they may not be as bad. I think the myth about FO being suitable for sparse data has arisen because when you sparse data you can learn very little and it therefore hard to be mislead too far by FO. The LAPLACIAN option is probably preferable over simple FOCE because it works harder at getting what should be the right answer. Don't sweat about $COV. The output is hardly worth the electrons used to generate it. There is no evidence that I know of that getting $COV to run is a reliable sign of a better model. My experience has often been the opposite. Often only crummy and naively simple models run with $COV and more sensible models which clearly fit the data better (based on eyeball tests of predictions matching observations) will fall over with the kind of error you report below. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/ _______________________________________________________ From: "Mike Davenport" <mike.davenport@richmond.ppdi.com> Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN Date: Wed, 16 Jul 2003 08:03:39 -0400 My basic understanding of NONMEM is that both the FO and FOCE methods are using a first order Taylor series expansion procedure. If this is true, then I my recollection is that the function needs to be 1st and 2nd order differentiable. The 2nd derivatives form the basis of Fisher Information matrix, which is your variance-covariance matrix. If this matrix is NON-POSITIVE-SEMIDEFINITE then the matrix is not invertable, the variance and covariances are not estimable and the parameter estimates would not be any good either. To me, in a naive sense, this seems like reporting a mean without any measure of dispersion. What am I missing on this?? thanks mike _______________________________________________________ From: Nick Holford <n.holford@auckland.ac.nz> Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN Date: Fri, 18 Jul 2003 19:16:28 +1200 Mike, You are correct that NONMEM relies on approximations to get its estimates. However, the measure of imprecision that might be provided by the covariance step is also affected by these same approximations so getting NONMEM's asymptotic SE does not help to correct for any bias arising from the approximations. I do not agree that because the FIM is not invertable that the parameter estimates are not any good. It is only the estimates of the uncertainty of the parameters under the assumptions used to derive asymptotic standard errors that is a problem. That said, if you want a better estimate of parameter imprecision (under the assumption that the approximation is not too harmful) then I would recommend using the bootstrap method (http://wfn.sourceforge.net/wfnbs.htm). This at least does not require one to make the assumption that the parameter uncertainty is normally distributed when one wants to define a confidence interval and is more or less guaranteed to give you some idea of the confidence interval. A second method is to use the likelihood profile but this depends on the assumption that changes in objective function are chi-squared distributed under the null -- an assumption known to be false (especially for FO). Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/ _______________________________________________________ From: "Kowalski, Ken" <Ken.Kowalski@pfizer.com> Subject: RE: [NMusers] Re: FO vs FOCE vs LAPLACIAN Date: Fri, 18 Jul 2003 08:38:12 -0400 Nick, I know we're rehashing old ground...but I can't resist. Too often the value of $COV is equated solely with getting SEs and Wald-based CIs which we know often don't have the right coverage probabilites. This to me is not the real value of the $COV. When the $COV fails, the hessian (R-matrix) may be singular and/or non-positive semi-definite indicating that the estimates may not have converged to a global optimum and thus they may not be (approximate) maximum likelihood estimates. That being said, I agree with you that this does not mean the estimates aren't any good. However, I do think it means we should proceed cautiously because they might not be good (particularly for extrapolation). In this situation you have previously advocated bootstrapping to assess the stability and predictive performance of the model. I think this a good course of action and a cautious way to proceed. Many who do not place any value in the $COV argue that they often obtain reasonable estimates even when the $COV fails. I argue that they are making use of prior information to make that assessment. If so, I would much rather be more explicit in using that prior information via fixing parameters, fitting a penalized likelihood, or using a fully Bayesian estimation procedure to remove the instability in the model fit. The real value of the $COV is as a diagnostic to assess instability. Of course, a successful $COV does not guarantee that the model is stable nor does it guarantee that the estimates have converged to a global optimum. It is merely a diagnostic to assess stablity and provides some indication as to when we should proceed cautiously. You may think I am jumping through a lot more hoops needlessly to obtain stable models but to me it is no more work than bootstrapping every model with a failed $COV in the development process. I realize that we are going to have to agree to disagree...just don't take $COV away from NONMEM VI as I'm willing to pay for those extra electrons. Ken _______________________________________________________ From: Nick Holford <n.holford@auckland.ac.nz> Subject: [NMusers] Re: FO vs FOCE vs LAPLACIAN Date: Tue, 22 Jul 2003 15:45:40 +1200 Ken, I am primarily interested in avoiding local minima (so that I can test model building hypotheses) and obtaining minimally biased and imprecise parameter estimates. I agree with you that success or failure of $COV probably does not help diagnose a local minimum problem. I have no evidence to support this. But what about bias and imprecision? I have a somewhat anecdotal but nevertheless evidence based comment on this. I recently completed a PK model analysis using WT, AGE, SCR and SEX as covariates (697 subjects, 2567 concs). The model did not run $COV, in fact it didn't even minimize successfully. Other evidence convinced me it was not far away from an appropriate minimum and because it had a more biologically sound basis than its more successful neighbours I preferred this model. I bootstrapped the original data set using the preferred model and found 28% of 1055 bootstrap runs minimized successfully and 7.1% ran the $COV step. The mean of the parameters obtained from all bootstrap runs and the mean from those which ran the $COV step were all within 2%. I conclude that $COV does not indicate lower bias compared with runs that do not minimize. To assess imprecision I computed the ratio of the mean standard error from the $COV successful runs to the bootstrap standard error obtained from all runs. For THETA:se estimates the $COV SE was on average 3% smaller but for OMEGA:se the $COV SE was 58% larger than the overall bootstrap SE. I conclude from this that the imprecision of THETA:se was negligibly different when the $COV step was successful. The difference in the OMEGA :se may reflect the intrinsic difficulty in obtaining estimates of OMEGA and OMEGA:se. Perhaps the asymptotic assumptions involved in $COV produce an upward bias. 95% confidence intervals obtained from all the bootstrap runs were very similar to those obtained from minimization successful and $COV successful runs. The 95% CI predicted from the asymptotic SE was on average 21% larger (range 15-35%) than the bootstrap CI. In order to explore the issue a bit further I simulated a data set using the mean bootstrap parameter estimates from all runs. I then bootstrapped this simulated data set (1772 runs). The minimization success rate was double (56%) that of the original real data bootstrap runs and 12.5% ran $COV. Because the true parameter values for the simulation are known the absolute bias can be computed. Only 3 out of 29 parameters had an absolute bias larger than 10%. There were negligible differences between the absolute bias using estimates from all runs, minimization successful runs or $COV successful runs. This means the $COV step is not a guide to reduced bias. The imprecision pattern was similar with the simulated data but the magnitude of differences between the mean $COV SE and the mean bootstrap SE were larger than those seen with the original real dataset. For $COV SE the THETA:se estimates were about 50% smaller while OMEGA:se were 400% larger than the bootstrap SE. There were no real differences depending on whether all runs, minimization successful or $COV successful runs were used ($COV successful runs tended to be a bit larger). 95% confidence intervals obtained from all the bootstrap runs on the simulated dataset were very similar to those obtained from minimization successful and $COV successful runs. The 95% CI predicted from the asymptotic SE was on average 22% larger (range 14-46%) than the bootstrap CI. My conclusion from this empirical exploration of one data set and model suggests that a successful $COV is of no value for selection of models with improved bias or imprecision. It is a quicker way of obtaining some idea of the parameter 95% confidence interval but it is upwardly biased compared with the bootstrap estimate. I am not typically interested in parameter CIs for every model I run. I am happy to leave that until I have finished model building and prefer to rely on bootstrap CIs. I think we are in agreement on almost all issues that you raise except for the diagnostic value of the $COV in relation to the thing you call "stability". I dont know what stability means so perhaps you would like to offer a definition and some evidence for your assertion. Nick PS Just in case anyone else it tempted to try this kind of experiment it took just about 2 months continuous operation on a 1.7 GHz Athlon MP2000 to do 2827 bootstraps. I'm still waiting for a response from the journal editor about the MS describing the preferred model so I had the time to do the computation while visiting PAGE etc. -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/ _______________________________________________________ From: "Kowalski, Ken" <Ken.Kowalski@pfizer.com> Subject: RE: [NMusers] Re: FO vs FOCE vs LAPLACIAN Date: Tue, 22 Jul 2003 12:50:46 -0400 Nick, Wow!...Talk about proceeding cautiously. See below for my response to your comments. Ken -----Original Message----- From: Nick Holford [mailto:n.holford@auckland.ac.nz] Sent: Monday, July 21, 2003 11:46 PM To: nmusers@globomaxnm.com Subject: Re: [NMusers] Re: FO vs FOCE vs LAPLACIAN Ken, I am primarily interested in avoiding local minima (so that I can test model building hypotheses) and obtaining minimally biased and imprecise parameter estimates. I agree with you that success or failure of $COV probably does not help diagnose a local minimum problem. I have no evidence to support this. But what about bias and imprecision? [Ken Kowalski] Actually, what I said was that a successful $COV does not guarantee that one has converged to a global minimum. However, a failure of $COV should raise a concern that we might not have converged to a global minimum. That is, conditions may be ripe for converging to a local minimum. I have encountered many instances (as I'm sure you have as well) where for two hierarchical models, the bigger model converges to a higher OFV than the smaller model (ie., delta-OFV is negative). In these instances the bigger model has clearly converged to a local minimum and often the $COV will fail for the bigger model. However, I have also seen instances where the $COV runs successfully for the bigger model but inspection of the output indicates large SEs for one or more parameters (as well as high pairwise correlations of the estimates) and this is the diagnostic information that leads one to be concerned that the bigger model is over-parameterized, leading to an ill-conditioning of the R-matrix (e.g., nearly singular) and instability in the estimation. The bigger model is unstable in the sense that if I change the starting values, I may converge to a different set of final estimates. When the likelihood surface for the bigger model is very flat, there may be many sets of solutions leading to very similar values of the minimum OFV. In this setting we may get a minimimization failure, or the $COV may fail due to a non-singular R-matrix, or if $COV does run some of the parameters will have very large SEs. In this sense the $COV does provide information on the precision or lack thereof (imprecision) in the estimation. One can't use the $COV to assess bias if the true model is unknown, however, if the model is unstable we should be concerned that we might have biased estimates. I have a somewhat anecdotal but nevertheless evidence based comment on this. I recently completed a PK model analysis using WT, AGE, SCR and SEX as covariates (697 subjects, 2567 concs). The model did not run $COV, in fact it didn't even minimize successfully. Other evidence convinced me it was not far away from an appropriate minimum and because it had a more biologically sound basis than its more successful neighbours I preferred this model. I bootstrapped the original data set using the preferred model and found 28% of 1055 bootstrap runs minimized successfully and 7.1% ran the $COV step. The mean of the parameters obtained from all bootstrap runs and the mean from those which ran the $COV step were all within 2%. I conclude that $COV does not indicate lower bias compared with runs that do not minimize. [Ken Kowalski] It concerns me that such a low proportion minimized successfully. What happens if you substantially change your starting values? You might find than one or more of the bootstrap parameter means are substantially different. We can't assess bias from bootstrapping a real data set where the true model for the data is unknown. Just because the $COV step ran that does not mean your model is not over-parameterized. Those runs where the $COV was successful may be numerically nonsingular but still nearly singular (output from $COV can help assess this). Thus, it doesn't surprise me that the mean estimates aren't different between those where the $COV was successful and all the runs. The value of the $COV goes well beyond a simple success/failure flag. When the $COV is successful we still need to inspect the output from the $COV to assess the stability. When the $COV fails we don't have this luxury but we get warning messages regarding invertability problems with the hessian that indicate we have a stability problem. Again, if we have a stability problem we may continue to proceed with the over-parameterized model but we should do so cautiously. To assess imprecision I computed the ratio of the mean standard error from the $COV successful runs to the bootstrap standard error obtained from all runs. For THETA:se estimates the $COV SE was on average 3% smaller but for OMEGA:se the $COV SE was 58% larger than the overall bootstrap SE. I conclude from this that the imprecision of THETA:se was negligibly different when the $COV step was successful. The difference in the OMEGA :se may reflect the intrinsic difficulty in obtaining estimates of OMEGA and OMEGA:se. Perhaps the asymptotic assumptions involved in $COV produce an upward bias. [Ken Kowalski] I believe Mats Karlsson has shown that asymptotic SEs for elements of Omega are not very good. 95% confidence intervals obtained from all the bootstrap runs were very similar to those obtained from minimization successful and $COV successful runs. The 95% CI predicted from the asymptotic SE was on average 21% larger (range 15-35%) than the bootstrap CI. In order to explore the issue a bit further I simulated a data set using the mean bootstrap parameter estimates from all runs. I then bootstrapped this simulated data set (1772 runs). The minimization success rate was double (56%) that of the original real data bootstrap runs and 12.5% ran $COV. Because the true parameter values for the simulation are known the absolute bias can be computed. Only 3 out of 29 parameters had an absolute bias larger than 10%. There were negligible differences between the absolute bias using estimates from all runs, minimization successful runs or $COV successful runs. [Ken Kowalski] The fact that you get 44% minimization failures and 87.5% $COV failures when you bootstrapped your simulated data set, based on the model developed from your original data set, provides evidence that your model is unstable. This is what I claim the $COV step failure from your original model fit was diagnosing. You indicate that 3 out of 29 parameters had an absolute bias greater than 10%. Perhaps the instability in your model is related to the estimates of these parameters. If they are not important parameters then perhaps I wouldn't be concerned. To illustrate my point consider the simple example where we have very little sampling in the absorption phase. Perhaps in each individual the observed Tmax corresponds to the first observation. In this setting we can have convergence and/or $COV failures and wildly biased estimates for ka (e.g., an estimate of ka >>100 hr^-1). Fortunately, we may find that even though ka is not well estimated we can still get relatively accurate estimates of CL/F which we may be more interested in. Thus, the instability in the model is related to the estimation of ka and the limitations of the design at early time points. I would be inclined to fix ka at some prior estimate (if I had one) to remove the instability and obtain successful minimization and $COV and acknowledge the limitations of the design/model to estimate ka. Alternatively, we could use simulation/bootstrapping to verify that poor (biased) estimation of ka is not likely to unduly bias CL/F. Further evaluation perhaps using simulation and bootstrapping is a cautious way to proceed when considering over-parameterized models. For me, I like to fit alternative models (often reduced hierarchical models) as a set of diagnostic runs and inspect the $COV output so that I can understand where the limitations are with the design/data/model. Both approaches can help us stay out of trouble. This means the $COV step is not a guide to reduced bias. [Ken Kowalski] The $COV should not be used as a guide to reduce bias. One can fit a mispecified reduced (smaller) model that is quite stable with a successful $COV and get biased estimates just as one can fit the true model and get biased estimates if the design/data doesn't support fitting all the parameters of the true model (ie., the true model may be over-parameterized resulting in a failed $COV). If the reduced model is not very plausible we may discard it or recognize its limitations particularly for extrapolation. However, it is naive to simply trust an over-parameterized model fit simply because the model is more plausible and proceed without caution. A true but over-parameterized model fit may have difficulty in estimation due to the limitations of the design to support the model. If the fit converges to a local minimum we still need to be concerned about bias even though we are fitting the true model. If you have a strong belief that the over-parameterized model is more plausible this is where explicity incorporating prior information on parameters that may be difficult to estimate from the existing design/data may be helpful. The imprecision pattern was similar with the simulated data but the magnitude of differences between the mean $COV SE and the mean bootstrap SE were larger than those seen with the original real dataset. For $COV SE the THETA:se estimates were about 50% smaller while OMEGA:se were 400% larger than the bootstrap SE. There were no real differences depending on whether all runs, minimization successful or $COV successful runs were used ($COV successful runs tended to be a bit larger). 95% confidence intervals obtained from all the bootstrap runs on the simulated dataset were very similar to those obtained from minimization successful and $COV successful runs. The 95% CI predicted from the asymptotic SE was on average 22% larger (range 14-46%) than the bootstrap CI. My conclusion from this empirical exploration of one data set and model suggests that a successful $COV is of no value for selection of models with improved bias or imprecision. It is a quicker way of obtaining some idea of the parameter 95% confidence interval but it is upwardly biased compared with the bootstrap estimate. I am not typically interested in parameter CIs for every model I run. I am happy to leave that until I have finished model building and prefer to rely on bootstrap CIs. I think we are in agreement on almost all issues that you raise except for the diagnostic value of the $COV in relation to the thing you call "stability". I dont know what stability means so perhaps you would like to offer a definition and some evidence for your assertion. [Ken Kowalski] Hopefully the above responses give you some sense of what I mean by stability. Over-parameterization of the model (too many parameters estimated relative to the information content of the data/design), ill-conditioning of the hessian (R-matrix) which can lead to numerically unstable inversion of the R-matrix ($COV failures) and instability of the paramater estimation (overly sensitive to the starting values) are all symptoms of a stability problem with the model. _______________________________________________________