From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca Subject: [NMusers] Permutation test with small number of possible permutations Date: Thu, November 25, 2004 10:37 am Dear NONMEM users, I am a new NONMEM user. I am working on modeling the Pharmacokinetics of two drugs. I have only eight patients per drug. I was screening for any potential significant covariates, but I know that the results would be suspect and the p values are bigger than the nominal when we have a small number of individuals 1. I am using FOCE INTER method. So I decided to do a permutation randomization test 1 for the significant covariates at the theoretical level of 0.05 to have the actual p values. (Although this approach was subsequently criticised 2 and I am aware of the results and conclusions of Ribbing et al.) To perform a permutation test, ideally we must do all the possible permutations but in practice this number is astronomical so we sample say 1000 from the possible permutations and do the test. My question is if we have a small number of permutations must we only do all possible permutations? In my case I have a dichotomous covariate taking values 0 and 1, 4 patients (0) and 4 with (1) so the number of possible permutations is: 8! / (4! * 4!) = 70. I am using S-plus to do the test, is there a function that do the permutation that guarantee that the permutation at each time is different from all the previous ones? sample(…,replace=F) do random permutations only. Samer MOUKSASSI 1: Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn. 2001 Jun;28(3):231-52. 2: Beal SL. Commentary on significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn. 2002 Aug;29(4):403-10; discussion 411-2. No abstract available. 2: Ribbing J, Jonsson EN Power, selection bias and predictive performance of the Population Pharmacokinetic Covariate Model. J Pharmacokinet Pharmacodyn. 2004 Apr;31(2):109-34. _______________________________________________________ From: "Nick Holford" n.holford@auckland.ac.nz Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Fri, November 26, 2004 5:05 pm Mouksassi, This a good question. I am afraid I cannot help you with an algorithm to generate all possible permutations but I believe that in principle this is the correct approach. It is the underlying method of Fisher's Exact Test i.e. to compute P from all possible permutations of the data. The empirical methods of sampling at random from randomly generated permuations are easy to implement and usually the only practical method for anything more than small samples. I don't think that Stu Beal's commentary contained any substantive criticism of the basic method of employing a randomization test procedure to define a null distribution and estimate the P value. Indeed he has been co-author of a paper which have used exactly this method e.g. Zhang L, Beal SL, Sheiner LB. Simultaneous vs. sequential analysis for population PK/PD data I: best-case performance. J Pharmacokinet Pharmacodyn 2003;30(6):387-404. Nick -- Nick Holford, Dept Pharmacology &anp; Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/ _______________________________________________________ From: Leonid Gibiansky"Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Fri, November 26, 2004 8:47 pm Samer, Actually, there will be only 35 different permutations (permutation 1234 versus 5678 is the same as 5678 versus 1234). The code below gives them all: N <- 0 i <-1 for(j in (i+1):6) for(k in (j+1):7) for(n in (k+1):8) { if(i < j & j < k & k < n) { N <- N+1 print(paste("Permutation", N,", class A subjects:", i,j,k,n)) } } P-value based on many (say, 1000) random permutations should be approximately the same as a p-value based on all (35 in this case) permutations. Leonid _______________________________________________________ From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Fri, November 26, 2004 11:05 pm Leonid, First thanks for the script. Is there a general formula for the number of permutations? If class A is assigned to 5678 or 1234 it would be different because we would have then : A A A A B B B B B B B B A A A A these are not equivalent, am I wrong? ID 1 2 3 4 5 6 7 8 The p value should be the same but the variance of it would be much bigger as it is (p*(1-p))/n for a computed 0.05 sd(p)= 0.00689 (1000) sd(p)= 0.036 (35) Too large ..…..to be useful sd(p)= 0.026 (70) Nick, maybe I used the wrong word “criticized”, Stu Beal said that this may not hold in all the settings and he added “ it is instructive to consider another idea for calibrating the critical value…” as the posterior predictive check. Samer _______________________________________________________ From: "Leonid Gibiansky" leonidg@metrumrg.com Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Sat, November 27, 2004 1:48 am Samer A A A A B B B B B B B B A A A A are equivalent, solutions will differ by the sign of the effect or, equivalently, by the class label only (for example, CL_A > CL_B versus CL_B > CL_A). A general formula for the number of permutations (for two classes A and B) is N! /( K!*(N-K)!), where N is the number of subjects, K is the number of elements in class A. But in this particular case, the number of elements in classes A and B are the same. This creates an additional symmetry. This symmetry can be used to reduce the number of permutations. You can use 70 permutations, if you like, but the result will be the same. If you like to use 70, just replace the first line of the script (i <- 1) by for(i in 1:5) A more general script (for more than two classes) is more difficult to write, I did it at some point in FORTRAN but I do not have it handy now. I am not sure that your interpretation of the variance is correct. These 35 (or 70) permutations are not random: they are carefully chosen to represent all possible permutations. Thus, they give you an exact distribution, equivalent to the infinite number of random permutations. You cannot use binomial sd(p) formula in this situation. Leonid _______________________________________________________ From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Sat, November 27, 2004 1:10 pm Leonid, Thanks for the comments. I do have actually a covariate with 3 classes and I see that we must consider the symmetry. ( As we usually there is no interaction between the covariates in the model ) I agree with you that this is an exact distribution so the quantiles describe it and the sd is not applicable here as it is not a random sample… Samer _______________________________________________________ From: andreas.krause@pharma.novartis.com Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Thu, December 2, 2004 5:12 am The approach looks reasonable. Using all possible permutations or sampling with a sample size similar to the number of possible permutations does not make much difference in practice going by my experience. If you want to go down that road and as you say you use S-Plus anyway, you can generate all permutations of a given set of values there. I don't think there is a canned function for it, but this one will do: # Create all possible subsets of a given vector for a given sample size. # This function generates a matrix with each row containing one of the possible combinations. # Andreas Krause, May 2002 all.combinations <- function(values=c("one", "two", "three"), samples=2) { nvalues <- length(values) out <- matrix(NA, nvalues^samples, samples) for(i in 1:ncol(out)) out[, i] <- rep(values, rep(nvalues^(i-1), nvalues)) return(out) } You could eliminate from the result the redundant combinations as has been suggested. If computation time is small, you might as well ignore it though. If you want to go for it, develop a code to characterize each combination of values. That code must be identical for redundant combinations and non-identical otherwise. Derive that code for each row of the matrix obtained from the above and eliminate the rows corresponding to non-unique codes. S-Plus functions to use are apply and duplicated. Hope this helps. Andreas Dr. Andreas Krause Novartis Pharma AG Biostatistics Modeling and Simulation WSJ-27.1.073 4002 Basel Switzerland _______________________________________________________ From: "Kevin Dykstra" kdykstra@pharsight.com Subject: RE: [NMusers] Permutation test with small number of possible permutations Date: Thu, December 2, 2004 9:52 am Andreas and Mouksassi--There is, in fact, a canned function in SPlus to create all possible combinations of several vectors. This would be a fairly natural way to do the exersize you outline. It is expand.grid: > a = c("A", "B", "C") > b = c(1, 2) > expand.grid(a, b) Var1 Var2 1 A 1 2 B 1 3 C 1 4 A 2 5 B 2 6 C 2 Kind regards, Kevin Dykstra Pharsight _______________________________________________________