From: Leonid Gibiansky    
Subject:Re: [NMusers] T matrix  
Date:Fri, March 8, 2002 5:21 pm  

Just to pick up on the issue:

>You should understand that the SEs provided by NONMEM are not worth much.

If you want to use them as measure of the confidence interval for your
parameters then they rely on at least two untenable asssumptions 1) you
have an infinite amount of data 2) the confidence intervals are symmetrical
around the parameter estimate. Alternative empirical methods e.g. bootstrap
or log likelihood profile are better way of understanding the confidence
you have in your parameters.
>

In my experience, SEs provided by NONMEM worth a lot. We compared them with
the bootstrap or log likelihood profile SEs and found out that in most
cases the results are similar with the exception of the parameters with
intrinsically non-symmetric distribution (e.g., bounded by zero, defined
with large error and with the estimate close to zero). Moreover,  log
likelihood profile SEs with FO are less reliable (too narrow) compared to
NONMEM estimates. Bootstrap is reasonable, but with fitting of 1000 or so
problems you cannot be sure that your tails of the bootstrap distributions
are not the local minimums (not to mention extra time and efforts that you
need to invest in bootstrap). Summarizing, we found that NONMEM SEs provide
quick and in most cases reliable information which is rarely corrected by
more computer-intensive techniques.

Leonid

*******


From:Nick Holford   
Subject:Re: [NMusers] T matrix  
Date:Sat, March 9, 2002 4:09 pm  

Leonid,

Thanks for these helpful comments.

Leonid Gibiansky wrote:
> In my experience, SEs provided by NONMEM worth a lot. We compared them with
> the bootstrap or log likelihood profile SEs and found out that

Is this work published yet? In Jul 2000 you said you were planning to publish.
http://www.boomer.org/pkin/PK00/PK2000103.html
I would be keen on seeing the details of your results.

> in most
> cases the results are similar with the exception of the parameters with
> intrinsically non-symmetric distribution (e.g., bounded by zero, defined
> with large error and with the estimate close to zero).

It is exactly these parameters with an asymmetric estimation confidence that are the
main reason I am personally interested in obtaining a confidence interval e.g. if I
estimate Emax as part of a model to test if a drug is effective or not then I want
to know how reliable the Emax value is. Typically Emax will be bounded by zero and
defined with large error and may well have an estimate close to zero for many drugs
in early drug development.

> Moreover,  log
> likelihood profile SEs with FO are less reliable (too narrow) compared to
> NONMEM estimates. Bootstrap is reasonable, but with fitting of 1000 or so
> problems you cannot be sure that your tails of the bootstrap distributions
> are not the local minimums (not to mention extra time and efforts that you
> need to invest in bootstrap). Summarizing, we found that NONMEM SEs provide
> quick and in most cases reliable information which is rarely corrected by
> more computer-intensive techniques.
I accept your point that FO may be misleading unless one has taken the time to
establish the log-likelihood difference required to define the desired confidence
interval e.g. Assessment of Actual Significance Levels for Covariate Effects in
NONMEM Wählby U.; Jonsson E.N.; Karlsson M.O.
Journal of Pharmacokinetics and Pharmacodynamics, June 2001, vol. 28, no. 3, pp.
231-252(22).
Note that I would not use the log-likelihood profile to estimate the parameter SE
(which IMHO is of no intrinsic merit) but would use it to define a confidence
interval for the parameter.

I also accept that bootstrap methods may be impractical for routine use but if the
parameter is really critical e.g. Emax during drug development, then resources spent
on bootstrap may be more cost-effective than relying on asymptotic SEs with the
unlikely assumption that the confidence interval is symmetrical.

My comments about SEs not being worth much should be interpreted in the context of
the application of using the SE and what value arises from that. For most modelling
projects I have been involved in it is very unusual for the the project owner to
have demonstrated any value from knowing the SEs.



*******

From:Leonid Gibiansky   
Subject:Re: [NMusers] T matrix  
Date:Mon, March 11, 2002 9:11 am  

Nick,
We published these observations (concerning NONMEM, bootstrap, profiling
and jackknife SE and CI) in three posters on the recent AAPS meeting:

· L. Gibiansky, E. Gibiansky, Parameter estimates and confidence intervals
for population PK model, American Association of Pharmaceutical Scientists,
Annual Meeting, Denver, CO, 2001.

· E. Gibiansky, L. Gibiansky, S. Bramer, Comparison of NONMEM, bootstrap,
jackknife and profiling parameter estimates and confidence intervals for
the aripiprazole population PK model, American Association of
Pharmaceutical Scientists, Annual Meeting, Denver, CO, 2001.

· L. Gibiansky, E. Gibiansky, R.Z. Yu, and R.S. Geary, ISIS 2302:
Validation of the population PK model and PK/PD analysis, American
Association of Pharmaceutical Scientists, Annual Meeting, Denver, CO, 2001. 


Leonid


*******


From:Nick Holford   
Subject:Re: [NMusers] T matrix  
Date:Mon, March 11, 2002 5:56 pm  

Leonid,

Thanks for posting these references and thank you also for sending me separately PPT
copies of your poster material. One of the posters (aripiprazole) has a table that
allows numerical comparison of the symmetry and coverage produced by 95% confidence
intervals derived from log likelihood profiling (LLP) and NONMEM standard errors
(SE). Using your own numbers I find that the LLP method in several cases has
substantially different coverage and is often asymmetrical.

Parameter       Sym     Coverage
q1   CL          84%     92%
q2   V          107%     97%
q3   Ka         107%    200%
q4   CAT1 CL    102%    103%
q5   LBW CL     103%    106%
q6   AGE_V       92%    138%
q7   WT_V        81%    158%
q8   omega1     100%     86%
q9   omega2     126%     77%
q10  omega3      88%    103%
q11  sigma      100%     40%
Sym=(abs((Hi-Est)/(Lo-Est)) LLP)/(abs((Hi-Est)/(Lo-Est)) SE)
Coverage=((Hi-Lo) LLP)/((Hi-Lo) SE)
Est=parameter estimate; Lo=95% CI lower; Hi=95% CI upper bound

This poster concludes "NONMEM standard errors and CI of the parameter estimates were
similar to the ones obtained by more computer-intensive methods.". I would not draw
the same conclusion. It depends on what you want to call similar but I don't
consider the coverage on KA, AGE_V, WT_V to be similar. The asymmetry of CL and WT_V
is also more than trivial.

Nick

TSR Matrices thread
From:Leonid Gibiansky   
Subject:[NMusers] NONMEM SE and CI  
Date:Tue, March 12, 2002 9:06 am  


Nick,

Let's put this discussion into the prospective:

1. We wrote "similar", but not identical, so I would not expect a
one-to-one agreement.
2. Asymmetry of CL: if you look on the plots, asymmetry of profiling CI is
in the different direction compared to the asymmetry of bootstrap CI. If
so, it would be hard to argue which of them to trust.
3. Asymmetry of WT_V: yeas, I agree, there is some asymmetry that is not
recognizable by NONMEM. But is it really important difference:
 0.489-0.953 (profiling), 0.599-0.895 (NONMEM) with the estimate equal to
0.746 ? Is it really "not similar" when you see 20% difference in the
symmetry  of the confidence interval for the parameter if the covariate
model ? 
4. Coverage on KA: yes, difference is large here, but bootstrap CI are 1/3
way between the NONMEM CI and profiling CI.  So NONMEM is not so bad. 
5. Coverage on AGE_V, WT_V: again, there is a difference, but bootstrap CI
are between NONMEM and profiling.  So again NONMEM is reasonable. 

Let me also add that for the parameters that describe random effect
variances, omega1-omega3, NONMEM CI are approximately half-way between the
profiling and bootstrap CI, with a rather significant gap between these
"better" methods. For  the variance of the error term, sigma, the NONMEM CI
coincides with the bootstrap CI whereas the profiling CI are twice more
narrow. So if you would be forced to trust the results of just one method,
I would trust NONMEM for this particular problem, with the other methods
giving similar although not identical results. 

Another peace of information: this is FOCEI, so bootstrap runs took about 2
weeks of computer time if I remember correctly (500 out of 1000 runs
converged). With profiling, there were many-many runs that were interrupted
by numerical errors, and we started them again and again with the new
initial values (I would estimate, 250 more runs were made). It took a lot
of time even with our automated routines. I am not sure that this is an
adequate price for the 20% improvement in the CI interval for WT_V, or even
40% improvement that we would get for KA CI. 

From CI, I can get the following qualitative information: 
- parameter is well-defined; 
- parameter is defined but not too well; 
- the NONMEM converged but the parameter value cannot be trusted.

This qualitative information and also quite reliable quantitative
information can be readily extracted from the NONMEM SE, and none of the
more elaborate techniques will change it. It is surely sufficient during
the model development and for most of the final models as well.

If one need specific, up to 20-30% precise CI, then bootstrap, profiling,
something else, can help sometimes. This may be needed if you plan to use
the model for simulations. Even  then, posterior predictive checks methods
would be a reasonable alternative to the bootstrap or profiling.

I am not trying to diminish the significance of the profiling or the
bootstrap methods: I routinely did them on my projects, and they bring new
more detailed information about the model. My main objection was the
comment that NONMEM SE does not worth much. Actually, they allow you
immediately and  correctly capture "the big picture" and even most of the
fine print, with the few details that can be studied, if really needed, by
some more elaborate methods.


Leonid


*******

From:Nick Holford 
Subject:Re: [NMusers] NONMEM SE and CI  
Date:Tue, March 12, 2002 4:15 pm  

Leonid,

Thanks for your remarks about perspective. My perspective is that the NONMEM SEs are
not reliable for making CI predictions. When you really want a CI then you do not
know if you can trust the CI predicted from SE (e.g. in your aripiprazole example,
you would not have known that you could predict CL but not KA). Because of this
uncertainty I maintain they are of little worth for making CI predictions *when you
really need* this prediction. 

Remember also that your models were quite straightforward with essentially linear
parameters. I would be reluctant to generalize your observations comparing the
performance of SE, profiling and bootstrap to other models e.g. when estimating
Emax.

There is another use of SE that I sometimes hear about. It is claimed that if the
$COV step runs then this somehow means that the model is alright. But I have had
runs finish with a successful $COV but then re-running with slightly different
initial estimates the run terminates or minimizes successfully at a lower obj
function value but unable to run $COV. In this case I dont think the initial $COV
success is any confirmation of the model. 

Nick
-- 
Nick Holford, Divn Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x6730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/