Using the caret::train package for calculating prediction error (MdAE) of glmms with beta-binomial errors












0















The question is more or less as the title indicates. I would like to use the caret::train function with beta-binomial models made with glmmTMB package (although I am not opposed to other functions capable of fitting beta-binomial models) to calculate median absolute error (MdAE) estimates through jack-knife (leave-one-out) cross-validation. The glmmTMBControl function is already capable of estimating the optimal dispersion parameter but I was hoping to retain this information somehow as well... or having caret do the calculation possibly?



The dataset I am working with looks like this:



 df <- data.frame(Effect = rep(seq(from = 0.05, to = 1, by = 0.05), each = 5), Time = rep(seq(1:20), each = 5))


Ideally I would be able to pass the glmmTMB function to trainControl like so:



BB.glmm1 <- train(Time ~ Effect, 
data = df, method = "glmmTMB",
method = "", metric = "MAD")


The output would be as per the examples contained in train, although possibly with estimates for the dispersion parameter.



Although I am in no way opposed to work arounds - Thank you in advance!










share|improve this question























  • is MdAE just median of the residuals?

    – missuse
    Nov 22 '18 at 7:07











  • Yes, as I understand it: en.wikipedia.org/wiki/Median_absolute_deviation

    – André.B
    Nov 22 '18 at 16:13
















0















The question is more or less as the title indicates. I would like to use the caret::train function with beta-binomial models made with glmmTMB package (although I am not opposed to other functions capable of fitting beta-binomial models) to calculate median absolute error (MdAE) estimates through jack-knife (leave-one-out) cross-validation. The glmmTMBControl function is already capable of estimating the optimal dispersion parameter but I was hoping to retain this information somehow as well... or having caret do the calculation possibly?



The dataset I am working with looks like this:



 df <- data.frame(Effect = rep(seq(from = 0.05, to = 1, by = 0.05), each = 5), Time = rep(seq(1:20), each = 5))


Ideally I would be able to pass the glmmTMB function to trainControl like so:



BB.glmm1 <- train(Time ~ Effect, 
data = df, method = "glmmTMB",
method = "", metric = "MAD")


The output would be as per the examples contained in train, although possibly with estimates for the dispersion parameter.



Although I am in no way opposed to work arounds - Thank you in advance!










share|improve this question























  • is MdAE just median of the residuals?

    – missuse
    Nov 22 '18 at 7:07











  • Yes, as I understand it: en.wikipedia.org/wiki/Median_absolute_deviation

    – André.B
    Nov 22 '18 at 16:13














0












0








0








The question is more or less as the title indicates. I would like to use the caret::train function with beta-binomial models made with glmmTMB package (although I am not opposed to other functions capable of fitting beta-binomial models) to calculate median absolute error (MdAE) estimates through jack-knife (leave-one-out) cross-validation. The glmmTMBControl function is already capable of estimating the optimal dispersion parameter but I was hoping to retain this information somehow as well... or having caret do the calculation possibly?



The dataset I am working with looks like this:



 df <- data.frame(Effect = rep(seq(from = 0.05, to = 1, by = 0.05), each = 5), Time = rep(seq(1:20), each = 5))


Ideally I would be able to pass the glmmTMB function to trainControl like so:



BB.glmm1 <- train(Time ~ Effect, 
data = df, method = "glmmTMB",
method = "", metric = "MAD")


The output would be as per the examples contained in train, although possibly with estimates for the dispersion parameter.



Although I am in no way opposed to work arounds - Thank you in advance!










share|improve this question














The question is more or less as the title indicates. I would like to use the caret::train function with beta-binomial models made with glmmTMB package (although I am not opposed to other functions capable of fitting beta-binomial models) to calculate median absolute error (MdAE) estimates through jack-knife (leave-one-out) cross-validation. The glmmTMBControl function is already capable of estimating the optimal dispersion parameter but I was hoping to retain this information somehow as well... or having caret do the calculation possibly?



The dataset I am working with looks like this:



 df <- data.frame(Effect = rep(seq(from = 0.05, to = 1, by = 0.05), each = 5), Time = rep(seq(1:20), each = 5))


Ideally I would be able to pass the glmmTMB function to trainControl like so:



BB.glmm1 <- train(Time ~ Effect, 
data = df, method = "glmmTMB",
method = "", metric = "MAD")


The output would be as per the examples contained in train, although possibly with estimates for the dispersion parameter.



Although I am in no way opposed to work arounds - Thank you in advance!







r r-caret glm






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 20 '18 at 22:08









André.BAndré.B

1519




1519













  • is MdAE just median of the residuals?

    – missuse
    Nov 22 '18 at 7:07











  • Yes, as I understand it: en.wikipedia.org/wiki/Median_absolute_deviation

    – André.B
    Nov 22 '18 at 16:13



















  • is MdAE just median of the residuals?

    – missuse
    Nov 22 '18 at 7:07











  • Yes, as I understand it: en.wikipedia.org/wiki/Median_absolute_deviation

    – André.B
    Nov 22 '18 at 16:13

















is MdAE just median of the residuals?

– missuse
Nov 22 '18 at 7:07





is MdAE just median of the residuals?

– missuse
Nov 22 '18 at 7:07













Yes, as I understand it: en.wikipedia.org/wiki/Median_absolute_deviation

– André.B
Nov 22 '18 at 16:13





Yes, as I understand it: en.wikipedia.org/wiki/Median_absolute_deviation

– André.B
Nov 22 '18 at 16:13












1 Answer
1






active

oldest

votes


















1














I am unsure how to perform the required operation with caret without creating a custom method but I trust it is fairly easy to implement it with a for (lapply) loop.
In the example I will use the sleepstudy data set since your example data throws a bunch of warnings.



library(glmmTMB)


to perform LOOCV - for every row, create a model without that row and predict on that row:



data(sleepstudy,package="lme4")

LOOCV <- lapply(1:nrow(sleepstudy), function(x){
m1 <- glmmTMB(Reaction ~ Days + (Days|Subject),
data = sleepstudy[-x,])
return(predict(m1, sleepstudy[x,], type = "response"))
})


get the median of the residuals (I think this is MdAE? if not post a comment on how its calculated):



median(abs(unlist(LOOCV) - sleepstudy$Reaction))





share|improve this answer

























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53402340%2fusing-the-carettrain-package-for-calculating-prediction-error-mdae-of-glmms%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    I am unsure how to perform the required operation with caret without creating a custom method but I trust it is fairly easy to implement it with a for (lapply) loop.
    In the example I will use the sleepstudy data set since your example data throws a bunch of warnings.



    library(glmmTMB)


    to perform LOOCV - for every row, create a model without that row and predict on that row:



    data(sleepstudy,package="lme4")

    LOOCV <- lapply(1:nrow(sleepstudy), function(x){
    m1 <- glmmTMB(Reaction ~ Days + (Days|Subject),
    data = sleepstudy[-x,])
    return(predict(m1, sleepstudy[x,], type = "response"))
    })


    get the median of the residuals (I think this is MdAE? if not post a comment on how its calculated):



    median(abs(unlist(LOOCV) - sleepstudy$Reaction))





    share|improve this answer






























      1














      I am unsure how to perform the required operation with caret without creating a custom method but I trust it is fairly easy to implement it with a for (lapply) loop.
      In the example I will use the sleepstudy data set since your example data throws a bunch of warnings.



      library(glmmTMB)


      to perform LOOCV - for every row, create a model without that row and predict on that row:



      data(sleepstudy,package="lme4")

      LOOCV <- lapply(1:nrow(sleepstudy), function(x){
      m1 <- glmmTMB(Reaction ~ Days + (Days|Subject),
      data = sleepstudy[-x,])
      return(predict(m1, sleepstudy[x,], type = "response"))
      })


      get the median of the residuals (I think this is MdAE? if not post a comment on how its calculated):



      median(abs(unlist(LOOCV) - sleepstudy$Reaction))





      share|improve this answer




























        1












        1








        1







        I am unsure how to perform the required operation with caret without creating a custom method but I trust it is fairly easy to implement it with a for (lapply) loop.
        In the example I will use the sleepstudy data set since your example data throws a bunch of warnings.



        library(glmmTMB)


        to perform LOOCV - for every row, create a model without that row and predict on that row:



        data(sleepstudy,package="lme4")

        LOOCV <- lapply(1:nrow(sleepstudy), function(x){
        m1 <- glmmTMB(Reaction ~ Days + (Days|Subject),
        data = sleepstudy[-x,])
        return(predict(m1, sleepstudy[x,], type = "response"))
        })


        get the median of the residuals (I think this is MdAE? if not post a comment on how its calculated):



        median(abs(unlist(LOOCV) - sleepstudy$Reaction))





        share|improve this answer















        I am unsure how to perform the required operation with caret without creating a custom method but I trust it is fairly easy to implement it with a for (lapply) loop.
        In the example I will use the sleepstudy data set since your example data throws a bunch of warnings.



        library(glmmTMB)


        to perform LOOCV - for every row, create a model without that row and predict on that row:



        data(sleepstudy,package="lme4")

        LOOCV <- lapply(1:nrow(sleepstudy), function(x){
        m1 <- glmmTMB(Reaction ~ Days + (Days|Subject),
        data = sleepstudy[-x,])
        return(predict(m1, sleepstudy[x,], type = "response"))
        })


        get the median of the residuals (I think this is MdAE? if not post a comment on how its calculated):



        median(abs(unlist(LOOCV) - sleepstudy$Reaction))






        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 22 '18 at 12:10

























        answered Nov 22 '18 at 11:33









        missusemissuse

        11.9k2723




        11.9k2723
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53402340%2fusing-the-carettrain-package-for-calculating-prediction-error-mdae-of-glmms%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            這個網誌中的熱門文章

            Xamarin.form Move up view when keyboard appear

            Post-Redirect-Get with Spring WebFlux and Thymeleaf

            Anylogic : not able to use stopDelay()