From: chyihung.hsu@pharma.novartis.com Subject: [NMusers] Significance? Date: Tue, 2 Apr 2002 16:56:12 -0400 Dear NONMEM users, I am fitting a 3-comp. population model to a data set. Based on the base model (model without any covariate), significant differences in post-hoc estimates of CL and their corresponding etas were suggested between 2 groups of subjects (Groups A and B). But adding the grouping covariate to the TVCL part of the base model did not seem to help, and the coefficient and the reduction in OBJ were far from statistical significance. On the contrary, this agreed with previous diagnostic and non-compartmental results. Should significant coefficient be expected? any suggestions? Thanks in advance. Best regards, Chyi-Hung Additional data and model information: Data: intensive samples. Model: random effects: CL,V1,Q2, and V2 were assumed to be log-normally distributed, with var-cov matrix=block(block(CL,V1), block(V2,Q2)), error = additive + multiplicative, and V2 > V3. Estimation: CONDITIONAL INTERACTION. ******* From: LSheinerSubject: Re: [NMusers] Significance? Date: Tue, 02 Apr 2002 13:24:46 -0800 Sounds like,a contradiction -- The post-hoc clearances are different between the groups, but including the group indcator in the clearance model does not help. I'd bet that if both findings are clear (ie,., the post-hoc clearances really are quite different, while inclusion of the group effect in the model really yields no appreciable change in OBJ), then you've made some interesting mistake. The question clearly is not which to believe, but why the contradiction? LBS. chyihung.hsu@pharma.novartis.com wrote: -- _/ _/ _/_/ _/_/_/ _/_/_/ Professor Lewis B Sheiner, MD _/ _/ _/ _/_ _/_/ mail: Box 0626, UCSF, SF, CA,94143 _/ _/ _/ _/ _/ courier: Rm C255, UCSF, SF, CA,94122 _/_/ _/_/ _/_/_/ _/ 415-476-1965 (v), 415-476-2796 (fax) ******* From: "Piotrovskij, Vladimir [PRDBE]" Subject: RE: [NMusers] Significance? Date: Wed, 3 Apr 2002 12:38:47 +0200 Chyi-Hung, In general, posthoc estimates give you only some hints about potential fixed effects. If you cannot prove these by the likelihood ratio test or other statistical tools it is not a problem. Do not consider this as a contradiction. Best regards, Vladimir ******* From: LSheiner Subject: Re: [NMusers] Significance? Date: Wed, 03 Apr 2002 11:23:18 -0800 Quite the contrary. In my experience, the obj function is far more sensitive to differences in goodness of fit than the graphics. I will often find a marginal increment in goodness of fit (say <=10 points) with some modeling change, but be unable to find any picture that looks "better." I do not accept such changes. In contrast, I cannot recall any case in which I could see a clear signal in a picture, which I could not then confirm by making the indicated model change and finding that the obj function changed markedly -- Except where I had made some kind of modeling or coding error! I would definitely not let siuch a thing pass without a very thorough investigation. LBS -- _/ _/ _/_/ _/_/_/ _/_/_/ Professor Lewis B Sheiner, MD _/ _/ _/ _/_ _/_/ mail: Box 0626, UCSF, SF, CA,94143 _/ _/ _/ _/ _/ courier: Rm C255, UCSF, SF, CA,94122 _/_/ _/_/ _/_/_/ _/ 415-476-1965 (v), 415-476-2796 (fax) ******* From: "Piotrovskij, Vladimir [PRDBE]" Subject: RE: [NMusers] Significance? Date: Thu, 4 Apr 2002 08:42:32 +0200 I agree: OBJ is more sensitive than the graphics. And it also tells you something about the significance of an effect whereas the graphics doesn't. As to whether accept an effect or not, I think the drop in OBJ by 10 is a good signal to accept (P<0.005) irrespective to what the pictures show. You may eventually not generate the plots. Of course, you have to be sure that there is no mistakes in the control streem... Best regards, Vladimir ******* From: "Bachman, William" Subject: RE: [NMusers] Significance? Date: Thu, 4 Apr 2002 09:42:04 -0500 Perhaps I am a bit naive, but, I would not recommend using just one metric (obj fcn versus plots) for determining goodness of fit (or model discrimination) in any case. I consider the change in objective function, change in plots AND change in variance estimates. Any one alone could be misleading. Maybe I missed the point here. ******* From: "Piotrovskij, Vladimir [PRDBE]" Subject: RE: [NMusers] Significance? Date: Fri, 5 Apr 2002 12:24:31 +0200 Bill, Let me be more concrete. Suppose we are exploring the sex differences in CL. Box plots show this and also OMEGA(CL) is reduced when we run the model with the fixed effect of SEX factor, however, the drop in OBJ is insufficient to accept the effect at the preselected level of significance (say, 0.005). This may happen if we have too few females in the study group (not unfrequent case). What would you do? ******* From: "Bachman, William" Subject: RE: [NMusers] Significance? Date: Mon, 8 Apr 2002 09:21:00 -0400 Vladimir, In your specific example of too few females [the story of my life ;)], I would be reluctant to accept the sex difference. (Unless there is supporting evidence such as a known difference in metabolic activity for males versus females from prior biochemical studies). Bill ******* From: "Wang, Bing" Subject: RE: [NMusers] Significance? Date: Mon, 15 Apr 2002 11:59:21 -0700 Chyi-Hung, Some questions/recommendations: 1. When you separate the two CL, (CL1 for Group A and CL2 for Group B), are these two CL estimates greatly different? (Although you mentioned OBJ did not change much, what about actual CL estimates?) 2. Try to separate the residual errors for Groups A and B in the $ERROR block. If you do so, how different are those ERR estimates? Is it possible that one group could be better described by a model other than the 3-c model...? Do you expect the PK in Groups A and B to be the same? 3. If PK is linear, the modeling shall give results fairly close to the NC analysis. Could the model be over parameterized somehow? 4. It may help to re-check the coding, and to confirm the comparison of OBJs is fair: FOCE vs FOCE, but not FO vs FOCE... 5. If overall goodness-of-fit graphs look OK and the OBJ change says nothing significant, and noncompartmental analysis (AUCs) did not show a trend between Groups A and B, continue your modeling without considering such group difference. Once you reach the final model, in your model validation step, separate the CLs to see if it makes any difference there... Regards. Bing Wang Amgen *******