Wsrtjtyk

I am running a Latent Dirichlet topic model in R using the follwing code:

for(k in 2:30) {

    ldaOut <-LDA(dtm,k, method="Gibbs", 

                 control=list(nstart=nstart, seed = seed, best=best, 

                              burnin = burnin, iter = iter, thin=thin))

    assign(paste("ldaOut", k, sep = "_"), ldaOut)

}

The dtm has 12 million elements, and each loop takes up to two hours on average. Meanwhile, R uses only 1 of my 8 logical processors ( i have i7-2700K CPU @ 3.50GHz wtih 4 cores). How can I make R use all the computational power available when I run one LDA topic model or when using a loop (as in this code)?

Thank you

EDIT: follwing gc_'s advice, I used the following code:

library(doParallel)



    n.cores <- detectCores(all.tests = T, logical = T) 

    cl <- makePSOCKcluster(n.cores) 



doParallel::registerDoParallel(cl)



burnin <- 4000 

iter <- 2000

thin <- 500 

seed <-list(2003,10,100,10005,765)

nstart <- 5 

best <- TRUE 



var.shared <- c("ldaOut", "dtm", "nstart", "seed", "best", "burnin", "iter", "thin", "n.cores")

library.shared <- "topicmodels" # Same for library or functions.





ldaOut <- c()



    foreach (k = 2:(30 / n.cores - 1), .export = var.shared, .packages = library.shared) %dopar% {

        ret <- LDA(dtm, k*n.cores , method="Gibbs", 

                   control=list(nstart=nstart, seed = seed, best=best, 

                                burnin = burnin, iter = iter, thin=thin))

        assign(paste("ldaOut", k*n.cores, sep = "_"), ret)

    }

The code ran without errors, but now there are 16 "R for Windows front-end" processes, 15 of which use 0% of the CPU and 1 is using 16-17%...And when the process was over i got this message:

A LDA_Gibbs topic model with 16 topics.



    Warning messages:

    1: In e$fun(obj, substitute(ex), parent.frame(), e$data) :

      already exporting variable(s): dtm, nstart, seed, best, burnin, iter, thin, n.cores

    2: closing unused connection 10 (<-MyPC:11888) 

    3: closing unused connection 9 (<-MyPC:11888) 

    4: closing unused connection 8 (<-MyPC:11888) 

    5: closing unused connection 7 (<-MyPC:11888) 

    6: closing unused connection 6 (<-MyPC:11888) 

    7: closing unused connection 5 (<-MyPC:11888) 

    8: closing unused connection 4 (<-MyPC:11888) 

    9: closing unused connection 3 (<-MyPC:11888) 

You can use the library doParallel

library(doParallel)

To get the number of cores of your computer:

n.cores <- detectCores(all.tests = T, logical = T) 

You can see the distinction between logical and physical cores.

Now you need to assign the core and set up all the process:

cl <- makePSOCKcluster(n.cores) 

doParallel::registerDoParallel(cl)

You can create more processes than you have cores on your computer.
As R is creating new processes you need to define the library and variables you need to share with the workers.

var.shared <- c("ldaOut", "dtm", "nstart", "seed", "best", "burnin", "iter", "thin", "n.cores")

library.shared <- c() # Same for library or functions.

Then the loop will change to:

 ldaOut <- #Init the output#



 foreach (k = 2:(30 / n.cores - 1), .export = var.shared, .packages = library.shared)) %dopar% {

      ret <- LDA(dtm, k*n.cores , method="Gibbs", 

                     control=list(nstart=nstart, seed = seed, best=best, 

                                  burnin = burnin, iter = iter, thin=thin))

      assign(paste("ldaOut", k*n.cores, sep = "_"), ret)

}

I have never used LDA before so you might need to modify a bit the code above in order to make it works.