From: "Leonid Gibiansky" leonidg@metrumrg.com
Subject:  [NMusers] Probabilistic model 
Date: Fri, May 13, 2005 12:54 pm 

Dear All,
I recently worked with the PK/PD model with the categorical 6 level score PD data.
Example of the 
code is given below, just to avoid description of the model. The run time was about
an hour, so I 
was able to experiment with the model. I used the following procedure:

1. Run the model with some initial conditions
2. Looked on the final estimate and updated the initial conditions according to the
following rule:
   new initial parameter value = final estimate * exp(mean eta value). Mean eta
value varied between 
  0 and 0.3 (exponential eta model) during the first 2-3 iterations and between 0
and 0.15 later on.
3. Run the model with new initial conditions, etc.

  After 10 or so iterations, objective function decreased by about 100 points (with
some visible 
improvement of the fit), variances of the random  effects decreased by 2-3 times. It
looks like the 
OF surface is very shallow, with a lot of local minimums that attract the solution.

My question is whether you have seen the same behavior in similar problems, and if
yes, how can we 
improve convergence (modify the model?), how to make sure that the final result is
valid? Should 
such fine-tuning after convergence be a necessary step of any similar model? Have
you developed any 
rules/scripts to automate the process?

Thanks
Leonid

;Model Desc: PK/PD model
$PROB RUN# 005M
$INPUT C = DROP ID TIME AMT DV EVID MDV WT
$DATA data.csv IGNORE=C

$SUBROUTINES ADVAN7 TRANS1
$MODEL
   NCOMPS=3
   COMP=COMP1;   CENTRAL
   COMP=COMP2;   PERIPH
   COMP=COMP3;   EFFECT

$PK
;PK
    K12=0.0526
    K21=0.0241
    V1 = 0.73*WT
    K10 = 0.0151
; EFFECT COMPARTMENT
      KE0=THETA(1)*EXP(ETA(1))
      K13=0.001*K10
      K31=KE0
      V3= K13*V1/K31
; PD
     SLOP= THETA(2)*EXP(ETA(2))
     EC50= THETA(8)*EXP(ETA(4))
; Baseline odds
      B0=THETA(3)*EXP(ETA(3))
      B1=B0+THETA(4)
      B2=B1+THETA(5)
      B3=B2+THETA(6)
      B4=B3+THETA(7)
$ERROR
; Drug effect
      CE=A(3)/V3
      EFF=SLOP*CE/(EC50+CE)

;LOGITS FOR Y<=5,Y<=4, Y<=3, Y<=2, Y<=1, Y<=0
         C0=EXP(B0 + EFF)
         C1=EXP(B1 + EFF)
         C2=EXP(B2 + EFF)
         C3=EXP(B3 + EFF)
         C4=EXP(B4 + EFF)

;CUMULATIVE PROBABILITIES
         P0=C0/(1+C0)
         P1=C1/(1+C1)
         P2=C2/(1+C2)
         P3=C3/(1+C3)
         P4=C4/(1+C4)
; P(Y=M)
         PR5 = (1 -P4)
         PR4 = (P4-P3)
         PR3 = (P3-P2)
         PR2 = (P2-P1)
         PR1 = (P1-P0)
         PR0 =  P0

   IF (DV.LT.0.5)  Y=PR0
   IF (DV.GE.0.5.AND.DV.LT.1.5)  Y=PR1
   IF (DV.GE.1.5.AND.DV.LT.2.5)  Y=PR2
   IF (DV.GE.2.5.AND.DV.LT.3.5)  Y=PR3
   IF (DV.GE.3.5.AND.DV.LT.4.5)  Y=PR4
   IF (DV.GE.4.5)  Y=PR5
$THETA
...
$OMEGA
.....

$EST MAXEVAL=9999 SIGDIG = 4 METHOD=1 LIKE LAPLACE NUMERICAL NOABORT
_______________________________________________________

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] Probabilistic model 
Date:  Fri, May 13, 2005 1:44 pm

Leonid,

I've never had much success in fitting ordered categorical models with more
than a single eta on the baseline of the logit domain i.e., logit(p) = base
+ eff + eta.  Also, since the logit model transforms the (0,1) probability
domain to the real number domain (-inf, inf) the values of theta3 through
theta7, could be positive or negative depending on whether the baseline
proportions are < or > p=0.5.  Thus, I'm not sure why you want to use an
exp(eta) on the baseline response.

Ken 
_______________________________________________________

From: "Leonid Gibiansky" leonidg@metrumrg.com
Subject: Re: [NMusers] Probabilistic model 
Date: Fri, May 13, 2005 2:05 pm

Ken,
I have a very detailed (once a minute) PD measurements, so I thought that I can
determine individual 
KE0, EMAX, EC50 (parameters with random effects). Probabilistic part has only one
(baseline logit) 
eta. B0 estimate is around -15, so it should not be positive for any subject
(probability of the 
event without a drug is nearly zero, much smaller than 0.5). If so, additive or
proportional eta 
models should be similar; this is not the source of the problem, but I agree that
additive eta may 
be adequate in this case.
Thanks for the reply
Leonid
_______________________________________________________

From: "Piotrovskij, Vladimir [PRDBE]" VPIOTROV@PRDBE.jnj.com
Subject: RE: [NMusers] Probabilistic model 
Date: Tue, May 17, 2005 9:01 am 

Leonid,

Each time I deal with ordered categorical responses with more than 3 - 4
categories I observe the same behavior as you describe. What I recommend is
to combine some categories to make the objective fuction surface less flat.

Best regards,
Vladimir

--------------------------------------------------------------------
Vladimir Piotrovsky, PhD
Research Fellow
Global Clinical Pharmacokinetics & Clinical Pharmacology
J&J Pharmaceutical Research and Development
Beerse, Belgium
_______________________________________________________

From:  "Nick Holford" n.holford@auckland.ac.nz
Subject:  Re: [NMusers] Probabilistic model
Date: Tue, May 17, 2005 9:25 am 

Vladimir, 

If the only way to get a solution is to throw away information by combining
categories then what relevance does the solution have? An alternative is to treat
the categories as continuous variables which NONMEM is perhaps able to handle more
robustly and certainly the resulting parameters are more readily interpretable.

Nick
_______________________________________________________

From: Chuanpu.Hu@sanofi-aventis.com
Subject: Re: [NMusers] Probabilistic model 
Date: Tue, May 17, 2005 10:29 am 

While treating categorical variables as continuous allows better
estimation, often it is improper because the categories are not
homogenious. For example, difference between category 1 and 2 is usually
not comparable to that
 between category 2 and 3. Combining caegories will make sense if one does
not care whether the category is 1 or 2, and the main objective is to
assess the probability of, say, <=2 vs >2. However, if that is not clear a
priori, then combining caegories would lose information indeed.

I suspect that the problem still lies in over-parameterization, e.g., the
number of ETAs, based on the phenominom Leonid describes.

Chuanpu
-------------------------------------------------------------------
Chuanpu Hu, Ph.D.
Biostatistics
sanofi-aventis
9 Great Valley Parkway
Malvern, PA 19355-1304
Tel: (610) 889-6774
Fax: (610) 889-6932
-------------------------------------------------------------------
_______________________________________________________

From:   "Piotrovskij, Vladimir [PRDBE]"  
Subject:   RE: [NMusers] Probabilistic model 
Date:   Tue, May 17, 2005 10:46 am 
Nick,

I think it's a bit risky to treat 6 categories as a continuous variable. I
usually do this with 10 categories or so. 

You are right saying that combining categories is not an ideal solution
since some information is lost. However, modeling is always a way to
compress information: instead of, say, 1000 observations you get just a few
parameters. 

Best regards,
Vladimir
_______________________________________________________
From:   "Nick Holford"  
Subject:   Re: [NMusers] Probabilistic model 
Date:   Tue, May 17, 2005 7:41 pm 
Chuanpu et al,

Thanks for taking the bait :-)

I knew you and your statistical colleagues would take me to task for even thinking
of treating categorical variables as continuous. While I have heard this caution
many times before I have never seen a numerical example which illustrates the
practical consequences given some realistic example.

There are other examples of statistical 'knowledge' that have not been borne out
when examined by experiment (e.g. distribution of delta OBJ under the null,
meaningfulness of getting covariance step to run when assessing parameter
reliability) so I wonder if anyone has done any work with NONMEM in this area?

Many uses of categorical variables in drug development reflect naive attempts by
investigators to capture what is really a continuous scale variable e.g. pain,
neutropenia. IMHO such categorical scales are intepreted by those who look at the
results as if they were indeed a continuous scale variable. It seems quite
reasonable if you have even a 5 point categorical scale to consider this as
continuous. Depending on the a priori knowledge of the system you might choose to
fix the residual error on each category to some reasonable value e.g. a pain score
on a 5 point scale might have a residual SD of 0.5 units (i.e. 3 is usually clearly
different from 2 or 4).

Nick
_______________________________________________________

From:   "Leonid Gibiansky"  
Subject:   Re: [NMusers] Probabilistic model 
Date:   Tue, May 17, 2005 8:40 pm 

Let me add an example in support of Nick's suggestion:

In the project (real data, consecutive PK, then PK/PD) that motivated my small
example we noticed 
that the expected score
     ESC= SUM(SCORE_i*P_i)
defined as a sum of (level * probability of the score at that level) described the
observed data 
with a very good accuracy. That motivated two continuous models. In one, we fitted
ESC as defined 
above to the observed DV (score). The second model was a model for ESC as an EMAX
function of 
concentration. Individual predictions of these two continuous models were as good as
individual 
predictions of the probabilistic model. We tried predictive check simulations and
found out that all 
three models over-estimated the frequency of the highest scores (with the strongest
effect). The 
probabilistic model was slightly better than continuous in this regard. Continuous
models took much 
less time (many hours instead of many days) and efforts to converge (e.g., initial
values of the 
parameters were obtained by FO; then FOCEI converged starting from the FO final
estimates): this was 
much simpler than guessing initial conditions for the probabilistic model. Both
types of models 
predicted a very similar covariate PD effect (requiring about 25-30% dose adjustment
for a subgroup 
of patients). Continuous models were more stable and they actually converged (i.e.,
start from 
different initial conditions led to similar solutions) while the probabilistic model
exhibited 
behavior described in the original example that started this discussion.
  Based on this example, it would be hard to recommend any of the approaches over
the other: each 
has own advantages and problems.

Leonid
_______________________________________________________

From:   "Mats Karlsson"  
Subject:   RE: [NMusers] Probabilistic model 
Date:   Wed, May 18, 2005 3:03 am 

Hi Nick,

The primary endpoints of sildenafil were six-category scales. The
statistical analysis plan said that these were to be treated as continuous
endpoints. Therefore in the PKPD analysis we (Peter Milligan, Scott Marshall
and I) analysed them as such but also as categorical endpoints. I have to
say that model development as categorical data was considerably simpler than
that as continuous. However, in the end there were no differences in
conclusions between the two approaches when based on simulations from the
two models. This was presented at PAGE in 1999. I can probably find you the
presentation if you're interested.

Best regards,
Mats

--
Mats Karlsson, PhD
Professor of Pharmacometrics
Div. of Pharmacokinetics and Drug Therapy
Dept. of Pharmaceutical Biosciences
Faculty of Pharmacy
Uppsala University
Box 591
SE-751 24 Uppsala
Sweden
phone +46 18 471 4105
fax   +46 18 471 4003
mats.karlsson@farmbio.uu.se
_______________________________________________________

From:   "Nick Holford"  
Subject:   Re: [NMusers] Probabilistic model 
Date:   Wed, May 18, 2005 6:07 am 

Mats,

Leonid's experience comparing continuous and categorical models was in line with my
own intuition -- continuous models ran more quickly and were easier to develop (in
part because parameters were more meaningful). Yet you seem to have the opposite
experience. Can you explain in more detail why you say "model development as
categorical data was considerably simpler than that as continuous"?

Nick
_______________________________________________________

From:   jeffrey.a.wald@gsk.com 
Subject:   Re: [NMusers] Probabilistic model 
Date:   Wed, May 18, 2005 8:33 am 

You cannot throw away information you do not possess.  If you have a 6
category scale but a few of the categories are not populated with a
sufficient number of observations, then combing them is perfectly valid
and will add stability to the final solution. 

A bigger danger in my mind is to assume that you can extrapolate, on the
basis of arbitrarily converting categories to continuous responses, to
nonobserved responses.  This is not necessarily a function of the number
of categories.  Take an 11-point pain scale. You might have very robust (and
apparently continuous data) in the high to middle range of the scale.  Now
treat patients with an mildly effective drug.  Absent a large placebo response,
you are just not going to see enough of the 0's, 1's and 2's to resolve
individual probabilities for these scores.   

As a friend and erstwhile mentor would say, "there is no substitute for
no data". (I am still trying to figure that one out :-) 

Jeff 

Jeff Wald, PhD
jeffrey.a.wald@gsk.com
Clinical Pharmacokinetics/Modeling and Simulation 
Neurology and GI
RTP, NC 
_______________________________________________________


From:   Chuanpu.Hu@sanofi-aventis.com 
Subject:   Re: [NMusers] Probabilistic model 
Date:   Wed, May 18, 2005 12:00 pm 

Nick,

I knew you were up to something. :-)
Still, I think your comments came from situations where the interest was in
some kind of "averaged" response. I agree that in many instances you can
get reasonable conclusions. I just prefer not to deal with the potential
objections/difficulties. However, in an instance of my work with Steve
Shafer (COST B1 1997 - details never published), categorical analysis did
show insights/knowledge that we would not get if data were treated as
categorical. If that would be of value, maybe I should try to convince
Steve that it is worthwhile to finish the paper. ;-)

Chuanpu
_______________________________________________________


From:   "Mats Karlsson"  
Subject:   RE: [NMusers] Probabilistic model 
Date:   Thu, May 19, 2005 2:46 am 

Hi Nick,

For a 6-grade scale (0-5), predictions outside the range has even less
meaning than those in between the categories. Therefore, we needed to put
quite some effort into appropriate constraining such that the model could
simulate well. Also, maybe not surprising, it is difficult to create a
residual error model that appropriately describes the residual error when
the observed data are constrained to 6 levels and a considerable amount lies
at the edges of the prediction range.

Best regards,
Mats 

--
Mats Karlsson, PhD
Professor of Pharmacometrics
Div. of Pharmacokinetics and Drug Therapy
Dept. of Pharmaceutical Biosciences
Faculty of Pharmacy
Uppsala University
Box 591
SE-751 24 Uppsala
Sweden
phone +46 18 471 4105
fax   +46 18 471 4003
mats.karlsson@farmbio.uu.se
_______________________________________________________


From:   "Nick Holford"  
Subject:   Re: [NMusers] Probabilistic model 
Date:   Thu, May 19, 2005 5:44 am 

Mats,

I guess it must be hard with sildenafil whichever model you choose? ;-)

Nick
_______________________________________________________

From: "Nick Holford" n.holford@auckland.ac.nz
Subject: Re: [NMusers] Probabilistic model 
Date: Thu, May 19, 2005 10:47 pm 

Jeff,

jeffrey.a.wald@gsk.com wrote:
> 
> You cannot throw away information you do not possess.  

But if you have information and merge it with other information without keeping
track of the original state then information must be lost. You CAN throw away
information that you do possess --- but it isn't a good idea. This is why I do not
like the idea of combining categories.

> If you have a 6 category scale but a few of the categories are not populated with
a sufficient number of observations, then combing them is perfectly valid and will
add stability to the final solution.
> 
> A bigger danger in my mind is to assume that you can extrapolate, on the basis of
arbitrarily converting categories to continuous responses, to nonobserved
responses.  This is not necessarily a function of the number of categories.  Take
an 11-point pain scale. You might have very robust (and apparently continuous
data) in the high to middle range of the scale.  Now treat patients with an mildly
effective drug.  Absent a large placebo response, you are just not going to see
enough of the 0's, 1's and 2's to resolve individual probabilities for these
scores.
> 

This is a different issue. The design may indeed make it hard to identify certain
levels of response but this is a problem for continuous as well as categorical
analysis too.

Nick
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/
_______________________________________________________