From: pbonate@ilexonc.com
Subject: [NMusers] order of covariate inclusion
Date: 9/25/2003 12:06 PM

Is anyone aware of any studies showing whether the order of covariate inclusion
has any impact on the final model using a forward addition model development strategy?
For example, suppose you have weight, sex, and age as covariates and then test weight,
then age, and then sex in the model using the likelihood ratio test.  Does this or
can this result in a different model than if you tested sex then age then weight?  

Thanks,

pete bonate


Peter L. Bonate, PhD, FCP
Director, Pharmacokinetics
ILEX Oncology, Inc
4545 Horizon Hill Blvd
San Antonio, TX  78229
phone: 210-949-8662
fax: 210-949-8219
email: pbonate@ilexonc.com
_______________________________________________________

From: lgibiansky@emmes.com
Subject: Re: [NMusers] order of covariate inclusion
Date: 9/25/2003 12:39 PM

I do not know specific references, but I am sure this has a great impact. That is
why forward addition procedure assumes testing all covariates, and then adding the
most significant one. This is not the perfect way to build the model, but at least
it gives unique model. As an extreme example, suppose that you have two identical
covariates. Then the one that you test first will be significant, and the second
one will add nothing. For your example, if sex and weight are confounded, you may
end up including into the model the first that you test even if the second one
gives you slightly better model.
Leonid 
_______________________________________________________

From: pbonate@ilexonc.com
Subject: Re: [NMusers] order of covariate inclusion
Date: 9/25/2003 1:02 PM

Leonid,
Thanks for your thoughts.  I am not sure that order has an impact provided the degree
of correlation between covariates is small to moderate.  In the example you cite, when
you have two covariates that are identical (the most extreme case) or highly correlated
what will happen is that when both are entered into the model simultaneously most likely
neither will show significance because of ill conditioning.  I did a paper on this a
few years ago.  As the correlation between two variables increases the standard errors
of the parameters increase eventually to the point where neither becomes significant.

It seems surprising that no one has looked at this.  Because if it has "great impact"
as you suggest then we need to re-examine all the published pop pk models to date as
most published reports fail to include alternative forward addition strategies in
their development process.

If order is important what guidelines are available or how do modelers choose their
order of testing in the model?

Any other comments?  Anyone?

pete
_______________________________________________________

From: lgibiansky@emmes.com
Subject: Re: [NMusers] order of covariate inclusion
Date: 9/25/2003 1:18 PM

Pete,
Forward addiction procedure is implemented as follows:
If you have covariates A, B, and C, you test all of the models
Base + A
Base + B
Base + C
and then choose the best model (say, Base +A, if Base +A is significantly better
than Base and better than Base+B)


Then you test
(Base + A) +B
(Base +A) +C
and then choose the best one (say, Base +A+B)

Then you continue until you exhorted significant covariates.

In this way, the procedure is unique. Therefore, all the published models
that site "forward addition procedure" uniquely define the model, as above.

It seems that you have in mind the following procedure:

Let's add A to Base. OK, it is better, then lets add B to (Base + A), etc.

This is incorrect for the reasons that you mentioned: the final model
will depend on the order.

Best,
Leonid 
_______________________________________________________

From: harry.mager.hm@bayer-ag.de
Subject: Re: [NMusers] order of covariate inclusion
Date: 9/25/2003 1:23 PM

Pete,

It seems to be easy to provide a counter example, i.e., if two variables are
highly negatively correlated. Of course standard errors will increase with
increasing correlations, but this does not necessarily result in insignificance
of both variables, what really increases dramatically is model uncertainty,
meaning in this case the decision which variable to select is more dictated by
random variation / fluctuation than by the underlying relationships.

Apart from that, what really counts is the multiple interdependence.  The latter
may be high even if the pairwise correlations are moderate. If at all,
re-examination of Pk/Pd's published has to take into account the entire
covariate pool (all those that were originally considered), the approximation
methods used  [e.g., U. Whälby, N. Jonsson, M. Karlsson published very nice
papers on the latter aspects], etc.

Harry

_______________________________________________________

From: bachmanw@globomax.com
Subject: RE: [NMusers] Re: order of covariate inclusion
Date: 9/25/2003 1:38 PM

Leonid,

are you implying that this can be habit-forming? (...Forward addiction)


Bill
_______________________________________________________

From: lgibiansky@emmes.com
Subject: RE: [NMusers] Re: order of covariate inclusion
Date: 9/25/2003 1:50 PM

sorry, I am becoming famous for English misprints and mistakes... Usually
Nick picking them up (but now seems to be a night time on that side of the globe)
_______________________________________________________

From: pbonate@ilexonc.com
Subject: RE: [NMusers] Re: order of covariate inclusion
Date: 9/25/2003 2:01 PM

Leonid,

I guess I think of forward stepwise modeling buidling more in terms of how I
was taught to do it with linear regression.  Like with SAS the most significant
covariate (based on an F-test) is entered into the model first.  If it is
significant (based on SLENTRY) it stays in the model.  Then the next most
significant covariate is entered into the model.  This process continues until
all covariates are exhausted.  Once a covariate is in the model it stays in the model.

Again, I guess it depends on the correlations between covariates for whether
something stays or goes into the model.  Covariates that are uncorrelated should
be unaffected by this method.

So going back to your example.  Suppose two covariates are highly correlated
(age and weight).  You test
   base + age
   base + weight.
Both are significant.  Then you test
   base + age + weight
and one or both of them turn up as insignificant.  What would you do?

Thanks,

pete

_______________________________________________________

From: Ken.Kowalski@pfizer.com
Subject:  RE: [NMusers] order of covariate inclusion
Date: 9/25/2003 2:04 PM

Pete,

I agree with Leonid.  Stepwise procedures and in particular forward
selection can be very sensitive to collinearity (correlation) among the
covariates.  I'm out of the office the rest of the week so I can't provide
you with specific references right now.  You should be able to find
discussions on the effects of multicollinearity in multiple linear
regression in standard statistical texts on linear regression.  The effects
of multicollinearity on estimation extend to covariate model building with
simple and mixed effects nonlinear models even though such effects may not
have been discussed in the population modeling literature. 

If you have a copy of my JPP 2001 paper on the WAM algorithm, I believe I
reference a paper by Burk (Technometrics 1977?) that discusses some of the
pitfalls with stepwise procedures when there is collinearity among the
regressors.

Ken
_______________________________________________________

From: alan_xiao@merck.com
Subject: RE: [NMusers] Re: order of covariate inclusion
Date: 9/25/2003 2:23 PM

If you follow the standard approach to let the machine pick the covariates,
there should be no problem to duplicate the model by different people. If
you pre-select covariates first and then test the model for those
non-selected covariates, you could get the problem.

Alan.

_______________________________________________________

From: lgibiansky@emmes.com
Subject: [NMusers] Re: order of covariate inclusion
Date: 9/25/2003 2:25 PM

Pete,
Let's first define what is "significant". In NONMEM discussions, significant
means that the difference in objective function between the nested models is
less then significance threshold delta.

If you test
   base (with objective function OFbase)
   base + age (OFage)
   base + weight (OFweight)
and see that OFbase - OFage  > delta, OFbase- OFweight   > delta
and OFage > OFweight, you include weight into the model (because weight
provides you with the lower objective function).

Then you test
base + weight + age (OFboth)

If OFweight - OFboth < delta, you do not include age into the model.
If OFweight - OFboth > delta, you include age into the model in
addition to weight.
End of the forward addition procedure.

Note that the end of forward addition procedure may or may not
coincide with the end of modeling process. You may found out
that although the model with age and weight gives you better
OF that weight alone, the standard errors of the estimates (for
coefficients that describe covariates) increase dramatically, leading
to the model instability due to collaterality of covariates. Then
you need to decide (and as far as I know, there are no formal rules
how to do it) whether to include extra covariate into the model
and accept  large uncertainty of the parameter estimates or use
fewer covariates with larger confidence in the parameters.
Leonid 
_______________________________________________________

From: Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] order of covariate inclusion
Date: 9/25/2003 2:42 PM

Pete,

As I indicated in my previous message, stepwise procedures have their
pitfalls in the presence of collinearity among the covariates.  You might
try a backward elimination approach starting from a full model (all
covariate parameters included simultaneously) and see if you arrive at the
same model as the forward selection approach.  Of course, if you have a very
high degree of collinearity among the covariates you may not be able to fit
the full model (rounding errors) but this is useful to know as the modeler
may be faced with having to make difficult choices on reducing the set of
covariates effects to investigate based on his/her understanding of which
covariate effects are the most plausible.  One of the problems with stepwise
procedures are that they result in selecting a single model and don't give a
sense of competing models that might also provide a good fit.  Ignoring the
effects of collinearity just because a forward selection can find a good
fitting model doesn't mean we should feel complacent about the model chosen
by forward selection.  Because of the order of testing there may be other
models that will never be evaluated by the procedure that are as good and
maybe better than the one obtained by forward selection.  This is one of the
reasons that motivated me to develop the WAM algorithm.  It provides a
ranking of all 2^k possible models (where k is the number of covariate
parameters) and the top 10-15 models are then run in NONMEM to give some
sense of the competing good-fitting models.  Try it...you might like it. 

Ken
_______________________________________________________

From: sduffull@pharmacy.uq.edu.au
Subject: RE: [NMusers] order of covariate inclusion
Date: 9/25/2003 6:14 PM

Hi

Implementation of covariate model building in this manner:


>> Forward addiction procedure is implemented as follows:
>> If you have covariates A, B, and C, you test all of the 
>> models Base + A Base + B Base + C and then choose the best 
>> model (say, Base +A, if Base +A is significantly 
>> better than Base and better than Base+B)


... does provide initial models that are not nested.  The 'sequential'
stepwise approach generally avoids this problem (i.e. adding a
additional covariates sequentially [in theory] will produce a sequence
of nested models).

You can of course also consider biological plausibility as an additional
guide to choice of initial covariates - for example the effects of
adding a covariate describing male/female can often be described
completely by consideration of patient weight.

Regards

Steve
=========================================
Stephen Duffull
School of Pharmacy
University of Queensland
Brisbane 4072
Australia
Tel  +61 7 3365 8808
Fax +61 7 3365 1688
University Provider Number: 00025B
Email: sduffull@pharmacy.uq.edu.au
www: http://www.uq.edu.au/pharmacy/sduffull/duffull.htm
PFIM: http://www.uq.edu.au/pharmacy/sduffull/pfim.htm
MCMC PK example: http://www.uq.edu.au/pharmacy/sduffull/MCMC_eg.htm
=========================================

_______________________________________________________

From: marc.gastonguay@snet.net
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches
Date: 9/25/2003 9:10 PM

Dear Pete, Leonid, Ken & others,
At the risk of complicating this already controversial topic, I'll throw my
two cents in...

For all the reasons that Ken metioned and more, I think we should be very
careful with stepwise approaches (see this website for more reasons "Why
stepwise regression is dumb":
http://cvu.strath.ac.uk/HyperNews/get/guss-fprt/9.html). Anyway, I've been
thinking about this for a while and I'm not sure that we should even bother
with stepwise covariate model building.

We typically go through this exercise to identify "statistically
significant" covariates and the most parsimonious model. When interpreting
the modeling results, we usually examine the "significant" covariate effects
and make some judgement about the clinical relevance of those covariates,
usually explaining away those statistically significant effects that do not
produce a clinically relevant change in the parameter(s). We also often
state that covariate effects that were not statistically significant have no
effect on the parameter in question (which is not entirely accurate).

Shouldn't we just focus on clinical relevance and forget about significance
of covariate effects? After all, the methods we use to assess significance
with the Likelihood ratio test are usually wrong (1 - 3) and inappropriate
(due to the multiple comparisons and retrospective nature of the analysis).

Let's consider an approach where one builds a full covariate model based on
prior scientific knowledge, or particular interest in a set of covariates.
This full model must be carefully & thoughtfully constructed to avoid highly
correlated/colinear covariates, but it is quite possible to create such a
model that will still converge.

Inference based on this full model is conducted not via stepwise regression
and the likelihood ratio test, but by estimating model parameters and a
measure of their uncertainty (bootstrap 95% confidence intervals, for
example). The expected clinical impact of covariate effects are then
evaluated given the parameter estimates and the uncertainties around these
estimates. In addition, conclusions about covariates that had relatively
little impact on model parameters can be made with some understanding of how
precisely these small ("insignificant") effects were estimated. So instead
of saying that a covariate has no effect on a model parameter, one can
assess if the lack of effect is actually due to the lack of a relationship,
or if the finding is due to insufficient data. I've also read that this full
model approach leads to standard errors that are more accurate than a
stepwise regression approach, which results in overly optimisitic standard
errors (4).

The other benefit of this approach is that once the full model has been
developed, computations are spent on getting estimates of parameter
precision (bootstrap) rather than a lengthy stepwise regression process.

Of course there are some practical challenges with this idea, and I have to
admit that I still routinely use stepwise backward elimination from a full
model as the primary covariate model building tool. I'm working on building
a set of case studies to convince myself that the full model/bootstrap
approach is sound.

Thanks in advance for your thoughts.

Marc Gastonguay


References:
	(1) 	Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance
levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn 2001;
28(3):231-252.
	(2) 	Wahlby U, Bouw MR, Jonsson EN, Karlsson MO. Assessment of type I error
rates for the statistical sub-model in NONMEM. J Pharmacokinet Pharmacodyn
2002; 29(3):251-269.
	(3) 	Gobburu JV, Lawrence J. Application of resampling techniques to
estimate exact significance levels for covariate selection during nonlinear
mixed effects model building: some inferences. Pharm Res 2002; 19(1):92-98.
     (4)    Altman, D. G. and P. K. Andersen. 1989. Bootstrap investigation
of the stability of a Cox regression model. Statistics in Medicine 8:
771-783
_______________________________________________________

From: gary.maier@sepracor.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches
Date: Friday, September 26, 2003 7:54 AM

Hello Marc

I think that some of your criticisms of the stepwise approach are right,
however as an industrial scientist I believe that we need some sort of
acceptable and "easy to implement strategy" in selecting model covariates.
If we rely exclusively upon what the modeler believes to be "the relevant
set of variables" to examine we introduce too much "art" into the process.
The issue then becomes when we send our pop pk reports to FDA there will not
be any guideposts for them  to follow and the sponsor could end up in
endless discussions with regulatory agencies.

I think that if stepwise regression is not the answer it is up to the more
statistically oriented users to propose an alternative that relies upon some
type of statistical guidelines not just art/judgment alone

Gary Maier
Sepracor
gmaier@sepracor.com

_______________________________________________________

From: marc.gastonguay@snet.net
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches
Date: 9/26/2003 8:19 AM

Gary,
Thanks for your comments. As you indicated, we are industrial scientists,
and I think that we should allow science to drive our model building - not
some formula that is pre-defined by "more statistically oriented users". In
fact, some statisticians feel the same way:

Here's a quote from Henderson and Velleman's paper "Building multiple
regression models interactively" (1981, Biometrics 37: 391-411): "The data
analyst knows more than the computer," ..."failure to use that knowledge
produces inadequate data analysis".

Whatever approach you choose to take, please remeber Box's point of view
that the resulting model is still the "wrong" model. If we use science (not
art and not statistics) to guide the covariate model I think we'd all agree
that the resulting model is a more "useful" one.

That's not to say that we should not provide evidence of goodness of fit,
predictive performance, and lack of bias with respect to remaining
covariates, etc. I just think we should spend more time on evaluating the
model and parameter estimates and less time on a prescribed stepwise model
building approach that is known to be problematic.

Best regards,
Marc
_______________________________________________________

From: bachmanw@globomax.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 8:59 AM

Marc,

I don't see any provision for exploratory analysis in your proposal.  Do we
always know the model?  Have we never serendipitously discovered
relationships that were contrary to our apriori conceptions or known
science?  

How do you know if something is clinically relevant if you aren't aware of
it?    The building approach, severely flawed as it is, at least has a shot
at uncovering new covariates.  I read your approach correctly, you are
getting your full model from prior knowledge and covariates of interest.
Did I miss something here?  Have you abandoned exploratory analysis?

Bill
_______________________________________________________

From: Jakob.Ribbing@farmbio.uu.se
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches
Date: 9/26/2003 9:33 AM

Dear all,

A few comments on the recent discussion on stepwise covariate modelling.

We have just submitted a paper (Jakob Ribbing and E. Niclas Jonsson, “Power, Selection Bias
and Predictive Performance of the Population Pharmacokinetic Covariate Model”) on a simulation
study that investigates the effects of stepwise covariate modelling and in short the conclusions
relevant to what has been discussed on NMusers are:

1. Stepwise comparison should NOT be performed on a SMALL DATASET (? 50 subjects)
if the purpose is predictive modelling:
   1. Weak covariates are heavily biased when selected based on a statistical criterion.
   Selection bias is caused by the selection procedure used and is not due to
   the estimation method used.
   2. Because of the heavy selection bias a weak covariate could be expected to
   worsen the predictive performance if selected
   3. A weak and clinically insignificant covariate cannot be separated from a clinically
   significant covariate because of this selection bias. Thus, the covariates which are
   statistically significant will also most often appear clinically significant even if they aren’t!
   4. Bias correction or other selection criteria than the p-value may allow stepwise
   regression even on small datasets.
2. Testing correlated covariates for inclusion in the model is not harming the predictive performance
of the final model. However, a large dataset is required in order to select, with enough certainty,
the better of two highly correlated covariates.

To connect to what was said by Marc on this topic, I do agree to that requiring statistical significance
of covariates SOMETIMES can be harmful and contra productive if the purpose is predictive modelling. However,
even in these cases stepwise regression could be useful for hypothesis generation. Marc suggested selecting
the covariate model based purely on prior knowledge, regardless of statistical significance in the dataset
analyzed, to estimate the covariate-model parameters. On the other hand, this prior knowledge can be partly
elicited from stepwise covariate modelling on a prior dataset. This is an appealing strategy that we will
compare to others in a current simulation study, but no results from this are available yet.

Best regards,

Jakob

Jakob Ribbing, MSc
Division of Pharmacokinetics and Drug Therapy
Department of Pharmaceutical Biosciences
Uppsala University
Box 591
SE-751 24 Uppsala
SWEDEN
Phone: +46 18 471 44 37
Mobile phone: +46 70 450 33 77
Fax: +46 18 471 40 03
Email: jakob.ribbing@farmbio.uu.se
_______________________________________________________

From: mark.e.sale@gsk.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches
Date: 9/26/2003 9:33 AM


My own perspective, which many of you have heard already.

My concern is usually more with getting the model right than with any statistical test.  As
I understand it the primary problems with step wise regression are:

1.  Inflated type 1 error, and the related inflated Rsquare values, downward bias standard
errors for parameters etc.  You basically are data dredging, and if you look at enough random
effects you'll find one.  This concerns me only somewhat, since post-hoc models should always be
regarded as hypothesis generation, in a statistical sense. 

From Frank Harrell - Regression Modeling Strategies: "Step wise variable selection ... if this procedure
had just been proposed as a statistical method, it would most likely be rejected because it violates
every principle of statistical estimation and hypothesis testing"

Also, I typically want the "best" model (whatever that means), so I worry more about type 2 errors than type 1 errors.  

2. (More important in my view) - confounding between variables, nicely demonstrated by this paper:

Interaction between structural, statistical and covariate models in population pharmacokinetic analysis
(Wade JR, Beal SL, Sambol NC J Pharmacokinetics and Biopharmaceutics, 1994 Vol 22 (2) 165-177)

Basically, this means that the answer you get depends on how you get there.  IMHO, the only way to
get the "best" answer is with a formal search of the models that are considered plausible or of
interest (either based on previous data or biology).  A formal search of the plausible models does not
address the issues of inflated type 1 error (in fact it may make it worse).  But again, I'm usually not
concerned (much) about that, I want the best model.  Penalties can introduced to addresses issues of
parsimony and Bayesian priors. 
This method is:
Objective
Can be predefined (although that only partly helps with inflated type 1)
Includes all effects (e.g., compartments, omega terms, residual errors, lag times etc), not just covariates.
Robust - it apparently will invariably find the "best" model among those considered.
Fast - we are currently running this using distributed computing on 1000 computers,
Can be put in a Bayesian framework with prior knowledge (although this isn't currently implemented).  This may
represent a compromise between Marc Gs comment that we could just build a full covariate model (assumes that we
know the model - completely informative prior), and a more traditional (hypothesis testing - uninformative prior) view.

Mark Sale M.D.
Global Director, Research Modeling and Simulation
GlaxoSmithKline
5 Moore Drive
RTP NC, 27709
919-483-1808

_______________________________________________________

From: chuanpu.2.hu@gsk.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis?
Date: 9/26/2003 9:36 AM

Marc,

I enjoyed reading your posts. Stepwise procedure, as a specific case of model exploration,
has awful properties in statistical inference. For this reason, I advocated restricting model
explorations, in a scenario where the inference properties are important, at the last PAGE meeting
(http://www.page-meeting.org/page/page2003/Chuanpu.pdf). The essence is that model exploration
increases the chance of finding the "right" model (by that I mean "better", I should say), but also
making the inference properties worse, hence there is a balance that needs to be maintained. The
optimal balance may depend on your goal of the modeling exercise.

One particular type of practical challenge comes from estimating a large model that the data could
not quite support. As you suggested, one frequently has to make compromises in actual applications.

A note: the term you used "not statistics" could be terribly misunderstood. (I think you mean
something like "not stepwise regression.") To produce credible results, our procedures must
have sound statistical properties, which includes correct standard errors, type I errors, etc.

Best regards,
Chuanpu
--------------------------------------------------------------------------
Chuanpu Hu, Ph.D.
Research Modeling and Simulation
Clinical Pharmacology Discovery Medicine
GlaxoSmithKline
P.O. Box 13398
Five Moore Drive
Research Triangle Park, NC 27709
Tel: 919-483-8205  
Fax: 919-483-6380
--------------------------------------------------------------------------
_______________________________________________________

From: Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches
Date: 9/26/2003 10:25 AM

Marc,

I agree with you that the full model is a better basis for making inference
such as in constructing confidence intervals.  However, I'm not ready to
throw away development of parsimonious models either.  When the parsimonious
model has considerably fewer parameters than the full model, it may provide
better predictions by smoothing out some of the noise in the full model
predictions since many of the parameter estimates from the full model are
just estimating noise. Many view development of a full model as a means to
an end...I don't.  I like to report out results for the base (no
covariates), full, and final parsimonious models and use each of these
models for different purposes.  The latter, parsimonious models, I typically
like to use for predictions.  That being said, I think we need to be
cautious when using stepwise procedures or any model building procedure
(including WAM) particularly when we are dealing with a high degree of
collinearity among the covariates.  Forward selection procedures can lull
one into a false sense of security if the modeler is not cognizant of the
collinearity.  Whereas building a full model will often require the modeler
to deal with it. 

Ken
_______________________________________________________

From:Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 10:43 AM

Bill,

You make a good point.  There will often be a need to consider ad hoc
covariates that weren't specified during the initial building of a full
model.  But that is also true of any systematic model building procedure
including forward selection.  Before beginning a forward selection procedure
we still need to list a priori what covariate effects we wish to consider
for inclusion in the first step of the procedure.  I maintain that
incorporating an ad hoc covariate into a final model after using a stepwise
procedure is not as informative as incorporating the ad hoc covariate into a
previously developed full model. 

Ken
_______________________________________________________

From: bachmanw@globomax.com
Subject:  RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 10:48 AM

Ken,

I was not necessarily advocating a stepwise procedure but pointing out the
need for an exploratory mechanism in whatever covariate selection/model
selection process we use.  Finding out things we don't know is the fun part.

Bill
_______________________________________________________

From: marc.gastonguay@snet.net
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 10:52 AM

Ken, Bill, Jakob, Chuanpu and Mark,

Thanks for your feedback. As I indicated, the full model/bootstrap approach
is still an idea and there are issues to be worked-out. Perhaps we can find
a compromise that addresses all of the issues you've raised. Let me try to
address the main issues.

First of all, I did not mean to denegrate statistics as a discipline and I
should have said stepwise regression. Thanks for pointing this out, Chuanpu.

On exploratory analysis and "Knowing" the model:
Of course we never really know the model and I do think that we should use
the usual goodness of fit diagnostics to compare possible alternatives and
guide the development of the structural model, while keeping prior
information in mind. In building the full covariate model, I suggested that
covariates should be included based on prior scientific knowledge AND your
interest in exploring a particular covariate effect. This does not assume
that you know the model ahead of time. If you are interested enough to do an
exploratory analysis on a particular covariate, you should include it in the
full model. I don't think we should proceed with the "kitchen sink"
approach, though. As has been mentioned before, you've got to be careful
about how you construct the full model so that you avoid problems with
correlated/colinear covariates (especially when the data set is small). You
may even need a few alternative full models to assess the form of the
covariate-parameter relationships (perhaps comparing linear and nonlinear
covariate relationships) in order to arrive at a stable full model. This is
where graphical exploration of the form of the covariate-parameter
relationship can be useful. We don't need stepwise regression to do any of
this.

On parsimony:
I agree that there are certainly advantages to arriving at a parsimonious
model. One of the things that is overlooked in a parsimonious model,
however, is why a particular covariate was excluded. Was it because the
covariate truly has no effect on the parameter of interest or was it
excluded because the data are not informative about this potential covariate
effect? A full model with point and interval estimates does address this
issue.

You could envision an approach where the full model is developed and
confidence intervals for all parameters are obtained. Then, decisions about
moving to a more parsimonious model are made based on the clinical relevance
of estimated covariate effects where those covariates having little or no
impact are dropped from the model. This preserves the assesment of why a
covariate is "insignificant", while allowing a more parsimonious model.

I would also suggest that you investigate any remaining trends in covariates
that were not included in the full model as part of the model evaluation
step. If the model performs poorly with respect to a particular covariate,
you may need to go back and pose a new full model.

Marc
_______________________________________________________

From: Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 11:07 AM

Marc,

Very well said.

Ken
_______________________________________________________

From: bachmanw@globomax.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 11:21 AM

Marc,

I know I'm just being a devil's advocate in this discussion, but, there is
an argument for (and precedent in other fields) for the "kitchen sink"
approach.  In the past the argument against it was time consumption. In
favor, was the "throwing away data" argument if you didn't use information
that was collected at great expense.

Now, with the automation of many of these methods (WAM, GAM42, etc) and
distributed computing via clusters and grids, time is not as large a factor.
Marc, nice segway into my blatant plug - come see our poster at AAPS:

Use of a Linux Cluster with PDx-Pop and NONMEM V to Streamline Population
Pharmacokinetic Analysis

[sorry, the devil made me do it!]

Bill

ps - It's nice to see a discussion revival on nmuser's, I was beginning to
think our listserver was down!
_______________________________________________________

From: mark.e.sale@gsk.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 11:48 AM

Bill,  
  I have to agree.  At the risk of being accused of being a frequentist, the Bayesian
  view can be taken too far. (collective gasp!!!).  But, the reality is that we are
  often (usually??) wrong about our models, that why we do studies.  We shouldn't (yet)
  complete discard the old "scientific method" of hypothesis generation - data -  hypothesis
  test/new hypothesis generation.  The learn/confirm view is frequently more like

GenerateHypothesis
Convince oneself that hypothesis is true
CompletePhD
age = 30  
Do while not (hypothesis = true)
    GetData
    age = age + 1
    Data suggests hypothesis isn't true
    if age > 99 then exit
end do
end

Those in academics should add a call to WriteGrant inside the do loop

Mark
_______________________________________________________

From: marc.gastonguay@snet.net
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 12:05 PM

Bill,
I think the problems with a "kitchen sink" approach go further than time
consumption. One of these is the increased chance of obtaining false
positive covariates and overfitting along with uninterpretable results. As
Mark Sale indicated, even with a broad search approach like the Genetic
Algorithm, you need to make rational choices to obtain models that are
"considered plausible or of interest (either based on previous data or
biology)".

Why not use your LINUX cluster to run 1000 bootstrap replicates on the full
model?

Marc

[Thanks for the shameless sales plug.]
_______________________________________________________

From: lgibiansky@emmes.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding  stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 12:59 PM

Dear All,

With all the respect to the other methods that may be more attractive, we cannot
rule our forward-addition approach. If you have a model with 5-6 random effects and
20-30 covariates (say, demographics, lab data and concomitant medications, this can
easily give you 30), it is unrealistic to fit the full model (30 parameters for each of
the random effects).  We need to  screen the covariates, via diagnostic plots of random effects
versus covariates, GAM, significance relative to the base model, etc. If the list of covariates
shortens so that you can fit the full model, that is great. But if not, what would you do ?

 I would add the most significant covariate and continue from that point (forward selection
 procedure), may be in chunks, adding this covariate to all the parameters at once, and
 then removing not-significant ones.

Mark, is there any alternative to this process if the full model is not converging ?

Thanks,
Leonid 
_______________________________________________________

From: mark.e.sale@gsk.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 1:30 PM

We (the royal we) think that formal search algorithms work very well in the setting
of many possible combinations of covariates, structural and error models.

Mark

_______________________________________________________

From: david_john.garbutt@pharma.novartis.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: Friday, September 26, 2003 12:55 PM


Hi,
Marc said:
66
You could envision an approach where the full model is developed and
confidence intervals for all parameters are obtained. Then, decisions about
moving to a more parsimonious model are made based on the clinical relevance
of estimated covariate effects where those covariates having little or no
impact are dropped from the model. This preserves the assessment of why a
covariate is "insignificant", while allowing a more parsimonious model.
99

But there is a big problem here - all those decisions about clinical relevance
are as dependent on the data gathered as the results of the automated selection.
(eg patient population, concomitant medications, the discovery of a new gene
related to metabolism, etc...)
So this method cannot be inherently better than the other.

To put it another way what make the process (ie selection procedure) more scientific isn't
the evidence we find supporting the model it is the fact we can find evidence against our
model that can lead us to reject it - paying too much attention to prior knowledge leads
us in the opposite direction.

Clinical relevance is too shaky to pin everything on - after all before vitamins were
discovered no one believed such small quantities of substances in food were important to dietetics.

regards,

Dave Garbutt

DIT, Basel
NEW:-> WSJ-310 2.09.22 <-> +41 61 32 49 521
_______________________________________________________

From: marc.gastonguay@snet.net
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/26/2003 1:55 PM

Hello Dave,
You raise an important point. I can see why you'd say that clinical relevance is dependent
upon the data gathered in as much that extrapolation of any conclusions from a clinical trial
are dependent upon the characteristics of the population studied, etc.
 
Let's assume we're in the case where you can extrapolate results of a trial to the general patient
population. Isn't clinical relevance of a model parameter something that must be assessed given
some understanding of what is a meaningful change in some endpoint (concentration, response, toxicity,
etc.). For example - the clinical relevance of a covariate-induced change in clearance could be
assessed by understanding how much exposure can vary before the pharmacodynamic response yeilds
unacceptible toxicity or lack of efficacy.
 
If we cannot determine the clinical relevance of a covariate effect, it is very difficult to make
use of the model in a "learning" mode (it may still be useful for prediction). When we start with
a full model, we can objectively evaluate whether or not a covariate effect is supported by the data
at hand by examining point and interval estimates of the covariate parameters. If the original hypothesis
is that all of the covariates in the full model are clinically relevant, the data may very well
contradict this by resulting in a covariate parameter estimate that is precise and near zero (or the
null value for that covariate).
 
The approach I described for covariate modeling does utilize some Bayesian notions in that inference
is based on parameter estimates and uncertainties, rather than p-values, but it does not completely
rely on informative priors. Not every parameter in the full model has to be based on prior knowledge.
The full model could also include covariates that you know nothing about, but are interested in exploring.
 
Marc
_______________________________________________________

From: sduffull@pharmacy.uq.edu.au
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis?
Date: 9/26/2003 5:57 PM

Mark
 
I don't think that building a model from prior information is necessarily particularly Bayesian.
Indeed the use of prior information does not necessarily imply a Bayesian analysis (as shown in
the recent JPKPD article using prior in a frequentist sense [I'm at home and don't have the
particular details at hand]).  You are right though that hypothesis testing is essentially a
frequentist concept... nevertheless there are Bayesian alternatives that also do the "same" thing.
 
Consider an hypothesis test (reject/fail to reject) as just a special case of the consideration of
the posterior probability that the data arose from a particular model, but with quantal rather than
quantitative criterion.  Although this is a simplification (and I stand to get many emails about this!)
the process of how you go about building models is not necessarily a function of the statistical
framework of your analysis ... the method of selection of the best model (however) is.
 
Incidentally I would suggest setting age_max to be << 99 :)
 
Steve

_______________________________________________________

From: marc.gastonguay@snet.net
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/27/2003 9:07 AM

Would it be fair to summarize this lengthy discussion as follows?:

1. Stepwise regression has its problems, but we still use this for covariate
model development on a routine basis (in addition to other less commonly
employed solutions, such as Wald Approximation Method and Genetic
Algorithm).

2. There are different opinions on whether to build a covariate model using
stepwise forward addition (or forward/backward) or to start at a full model
and proceed with a stepwise backward elimination method.

3. We also have different opinions on how to select the covariate-parameter
relationships to be investigated. This ranges from one extreme, where every
measured variable should be investigated as a potential effect on every
model parameter to the other, where the "best" model is known a priori. The
optimal solution is undoubtedly somewhere in the middle.

4. There are different views on how to select the best model, where one
approach relies on p-values obtained using the Likelihood Ratio Test and
another possibility is based on point and interval estimates (or even full
posterior distributions) of model parameters.

5. We need more comparisons and examples to help us work the balance between
theoretically appealing approaches and practical implementation.

I think the take-home point is that there is NOT just one way to build a
model and that whatever you end up with is still the wrong model. Maybe we
should spend more time on model evaluation and assessing sensitivity to
assumptions once we do arrive at the "best" model.

Marc
_______________________________________________________

From: mark.e.sale@gsk.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/29/2003 8:18 AM

Nice try at diplomacy Marc, but I'm going to be refractory.  We have preliminary data, and
Rob Bies has presented some data that show that forward addition/backward elimination rarely
(if ever) gives the best answer.  Again, my interest is in finding the best answer.  The
assumption behind any step-wise method is that the search space is monotonically down hill,
that there are no interactions between the different effects being considered. That is probably
never true.  The search spaces we examine are very complicated, not only among covariates, but
between structural effects, random effects and covariates.  I think that the reason we all use
forward addition/backward elimination is related to the metaphor, "If the only tool you have is
a hammer, then everything looks like a nail"  Lets list a few of the nail-like properties of this problem:

1.  If you don't think about it too hard, this algorithm can be used for hypothesis testing.  Think about
it too hard would include all those pesky statistical assumptions, none of hich are true.

2.  If you don't think about it too hard, it can be put in a Bayesian framework.

3.  There is no reason to believe that it ever gives the right answer, and we have data to support this.

Fortunately, other disciplines have thought about efficient ways to search discrete spaces, and the
assumptions required for the different methods. we probably should learn from them.

For those interested in learning about modern methods of model selection, we have openings
in US (North Carolina and Phil), UK and Italy.


Mark Sale M.D.
Global Director, Research Modeling and Simulation
GlaxoSmithKline
5 Moore Drive
RTP NC  27709
919-483-1808 
_______________________________________________________

From: harry.mager.hm@bayer-ag.de
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/29/2003 8:59 AM

Dear all,

it seems that the decisive problem is that covariate selection and hypothesis
testing have to be different processes. If not, the statistical properties of
the various estimators will not be known, regardless which selection criterion /
criteria or "modern" method is used. Since this prerequisite is too restrictive
in most cases in practice, we have to accept that we are relying on
approximations to the truth at the very best. Actually, there is no special
problem with stepwise procedures that is not inherent in other procedures
(genetic algorithms etc., etc.), too. At its very end, stepwise procedures may
result into "all regression", examining all possible subsets. With increasing
number of covariates to be explored, all regression may result in a
computational burden not easy to handle. Most of the modern methods essentially
only try to reduce computation time, so the actual problem will remain. If we
want to select the "best" model, the very first task would be to define what is
"best", i.e., the criterion / criteria to be met have to be defined.

It has be shown using a vast amount of simulations, and the results are also
supported by theoretical considerations, that whatever criterion is used, the
results of a selection process will be overly optimistics with regard to the
selection criterion.

Harry

_______________________________________________________

From: Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/29/2003 9:58 AM

Marc, Bill, et al.,

Nice summary but to me the key point to this discussion based on Pete
Bonate's original message in this thread is that we need to be cognizant of
the effects of collinearity when we build covariate models regardless of the
model building procedure we use.  The problem with the 'kitchen sink'
approach is that we are putting too much trust in the statistical algorithm
to sort out the true covariates  from the nuisiance covariates that may be
highly correlated with them.  The more things we toss into the 'kitchen
sink' the more likely we will pick up some of these nuisance covariates in
our model.  Why is it that we are willing to make very strong mechanistic
assumptions regarding the structural model but when it comes to a list of
covariates to be investigated we are unwilling to prune the list based on
this same mechanistic reasoning?  Of course we do try to make the list
somewhat plausible (e.g., we don't typically include shoe size as a
covariate but its likely to be correlated with weight and may actually
explain some of the interindividual variation), we just need to be a little
more discriminating (rather than just pruning out the obvious such as shoe
size).

With regards to building a full model, it is a bit of a 'straw man' argument
to say full models are difficult to develop when there are 20-30 covariates.
The difficulty is not the number of covariates but the amount of independent
information contained in these 20-30 covariates.  The full model approach
requires the data analyst to deal head-on with the collinearity issue.  The
issue is not of computational speed.  Just because we now have procedures
that may be computationally faster than what we had say 10 years ago,
doesn't mean that we should blindly (i.e., ignoring collinearity)
investigate more covariates just because we can.

Ken
_______________________________________________________

From: tgordi@buffalo.edu
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?
Date: 9/29/2003 11:00 AM

Hi!

I am not as much experienced with NONMEM (or statistics) as many of the contributors
but I guess there are a couple of simple rules one can follow with regard to inclusion
of covariates in any model. The first is, of course, just consider those covariates that
are of interest. What is of interest might differ from case to case. If it is a exploratory
study, one might want to look at many parameters. If one want a more "practical" model, weight
and gender might be enough (just examples). A lot of information can be collected in a clinical
 study but not all of it will be or should be tested. The second simple rule is that the more
 tests one does, the higher is the risk of finding something that is not there (as Ken mentions
 below). Although there probably are several methods to deal with the situation, one simple
 approach would be to decrease the level of significance, let's say use 0.001 instead of 0.05,
 for a covariate to be deemed influential. What value to choose will be dependent on the
 number of tests you do, although there are some variations of the rule, which are less
 conservative. The negative side is, of course, that there will be more difficult to show
 any significance. However, this might function as a driving force to choose the best
 candidates only. I understand that this is not the answer to all of the problems but I would
 say that following these rules helps one to be less "wrong" in the analysis.

This discussion also brings me to another issue I have been thinking about. If we agree
that conducting several tests lead to inflated TYPE I error, what are the consequences
when several different structural models are tried in our routine PK/PD modeling? It is
a common practice to accept a drop of 3.8 in the OFV (which is related to a significance
level of 0.05) as a "proof" of the superiority of a model over another. My point is not
whether using OFV is optimal. The question is whether one should be more restrictive
and use 0.01 or even less.

Toufigh Gordi
_______________________________________________________

From: lgibiansky@emmes.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding  stepwise a pproaches -> abandoning exploratory analysis?
Date: 9/29/2003 11:16 AM

Ken,
Your message seems to imply that
(i) Large number (e.g., 30) of covariates can be found ONLY in problems where we
have a lot of collinear covariates,

(ii) Thinking through the list of covariates, we ALWAYS can reduce it to the
manageable number (and sort them out between different random effects).

I would disagree with both statements.

If you cannot create manageable full model starting from the base model, you need to
screen covariates, that leads us to the forward-addition algorithm (unless you work
for Mark Sale with the access to the genetic algorithm and a cluster of 1000+ computers
waiting for your input). Unspoken assumption (sufficient condition of convergency)
behind the forward addition algorithm is the convexity of the -log(likelihood) as a
function of covariates. One can easily create an artificial example where this assumption
is violated but it would be interesting to see any real example where this is not true.

Sure, I agree that collaterality is the issue that needs to be thought ahead of time, but
this is not the only source of the large covariate list. As to the pruning the list, I think
it is dangerous to do it too aggressively: this can be very subjective; can prevent us from
uncovering new totally unexpected dependencies. I would prefer to do more or less formal
search of the explanatory variables, and then interpret them in clinically relevant terms
rather than look for clinically relevant dependencies only. This may reduce subjectivity
of the analysis, restricting it to the subjective explanation of uncovered dependencies
instead of the subjective choice of alternatives to investigate.

Leonid 
_______________________________________________________

From: GUZY@xoma.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding  stepwise a pproaches -> abandoning exploratory analysis?
Date: 9/29/2003 12:22 PM

I like the "shoe size" example for covariate. I would like to add my
part in this interesting discussion. I am a statistician and therefore
aware of how statistics can erroneously lead us to no sense results if
we do not pay attention. Suppose we have 100 covariate and let say that
there is a 5% chance to get by chance a covariate which exhibit high
correlation (we would therefore take it as a part of our model). In
average we would have 5 covariate with high correlation that are the
result of pure chance. Suppose now that we have 5 other covariate that
really exhibit a real correlation and they show up when performing the
analysis. If we don't do any statistical adjustment, we will take the 10
covariate and one of them would be for example shoe size. How can we
know that shoe size was the result of chance while age for example was
the result of a real trend when they both statistically exhibit the same
strength of association? 
Now if you have 100 covariate, you must make a statistical adjustment to
prevent inflation of the alfa type error and you can easily reach the
situation where you will not be able to accept any of the covariate(you
lost completely statistical power). Using purely a machine that would do
everything for you without using any prior information (does not mean
Bayesian) is not reasonable. I believe that experts should discuss the
problem in advance and come up with up to 5 covariate maximum and then
performing the analysis. 
The other issues related to the use of high number of covariate has been
addressed before and I will not reiterate (collinearity, other numerical
difficulty etc..)
Serge
Head of Preclinical Statistics and Pharmacometrics, XOMA Corporation 
President POP-PHARM 
_______________________________________________________

From: mark.e.sale@gsk.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis?
Date: 9/29/2003 12:41 PM

Leonid
  My last email on this, I promise.  We have several examples of cases where
  your convexity assumption (which I called monotonically downhill) is not the
  case, formally reported by Janet Wade et al.  What this means is that the answer
  you get depends on how you get there, and different users will get different
  answers for the same data set, with the same prior - not a good result for a
  scientific analysis.  This is not uncommon, in fact it is essentially invariably the case. 
  For my last commercial, all will become clear at Paolo Vicini's symposium at
  AAPS Wed Oct 29, "What is the right model? Issues surrounding the evaluation of
  Pharmacostatisical Models in Drug Development" (Well, maybe not completely clear).



Mark

_______________________________________________________

From: Ken.Kowalski@pfizer.com
Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? 
Date: 9/29/2003 2:37 PM

Leonid,

I think you are missing my point.  I have nothing against investigating a
large number of covariates (e.g., >=30) if they truly provide scientifically
relevant and independent information.  However, typically what happens when
we have a long list of covariates is that many covariates may be collinear
and hence are redundant.  Such redundancy based on these nuisance covariates
that are correlated with the important mechanistic covariates can cause
havoc with any model building procedure, masking our ability to discern the
true covariate effects.  Forward selection procedures are particularly
vulnerable because they can be blind to the collinearity issue as they can
often find a good fitting model without running into
stability/over-parameterization issues that a full model would be confronted
with.  However, as I've said previously, just because forward selection can
find a good fitting model doesn't mean it found the right one.  For example,
by chance a nuisance covariate that is highly correlated with the true
covariate may be selected first by a forward selection procedure and because
of the order of testing the true covariate may never get further evaluated
in the forward selection procedure and hence will be excluded in favor of
the nuisance covariate.  I can't tell you how many times that a modeler will
say they have a difficult time interpreting covariate effects in the final
model selected by a stepwise procedure because they felt certain excluded
covariates were more scientifically plausible.  I'm merely suggesting that
we be a bit more discriminatory with developing our list of plausible
covariates to investigate so that we have a set that are the most
scientifically plausible and independent.   

The main reason we use a combination of forward selection/backward
elimination with a higher alpha level for inclusion (e.g., alpha = 0.05) is
precisely to help mitigate the problem with forward selection alone.  By
increasing the alpha level for inclusion we allow for "bigger" models to be
tested before pruning to a parsimonious model using backward elimination
with a smaller alpha level for exclusion (e.g., alpha =0.01 or 0.001).  Of
course, if we increase the alpha level for inclusion towards 1.0 the
combination forward selection/backward elimination procedure will collapse
to a purely backward elimination procedure.  Moreover, it may only take
setting the alpha level for inclusion to 0.20 to begin to develop bigger
models using forward selection that will encounter the ill-conditioning
problems due to collinearity that we observe with the full model (if we
don't become a bit more discriminating in our choice of covariates).

I agree that we could err on the other side as well and eliminate important
explanatory covariates if we are too discriminatory.  Nevertheless, we need
to use our best scientific judgement as well as good statistical principles
to really uncover the important covariates.  Blindly ignoring the
limitations of  forward selection and just turning the crank to allow the
algorithm to identify covariate effects is risky.  That is not to say that
I'm against forward selection, just that we need to know when it is
appropriate to use it and when its not.  I still maintain it is better to
identify where the redundancies are and eliminate the least plausible
covariates when redundancies exist.  Certainly if a less plausible covariate
is fairly independent of the other covariates then we don't need to
eliminate it.  I have no problem using forward selection to identify a
parsimonious model once we've streamlined the list to those covariates that
provide the most independent information where redundancies are removed
based on the covariates that are the least plausible.  The problem is
knowing  when we are in a situation where there is a lot of redundancy.
Building a full model and looking at the diagnostics from the full model fit
(e.g., COV step) will help us to know when we are in a situation where we
should deal with the collinearity.

Ken
_______________________________________________________