tf.data.Dataset with constant size batches












1















I have a dataset with 19 elements and a batch size of 10. I set my dataset to continuously iterate over the same elements but I noticed that the last batch has only 4 elements instead of 5, and then it starts over with 5, 5, 5, 4, and so on.



How is is possible to force the iterator to fill up shorter batches with elements coming from the next iteration so that all the batches have the same size?



P.S. just to understand, isn't this the obvious behavior when training a model?






tensorflow tensorflow-datasets







share|improve this question




















  • 1





    If you look at the documentation of tf.data.Dataset.batch, there is an optional drop_remainder parameter to discard the last incomplete batch. I don't know if there is a way to complete the last batch with the beginning of the next iteration. If you're shuffling your data it shouldn't matter discarding the last batch, but anyway it would be interesting to have a way to do that.

    – jdehesa
    Nov 19 '18 at 21:35


















1















I have a dataset with 19 elements and a batch size of 10. I set my dataset to continuously iterate over the same elements but I noticed that the last batch has only 4 elements instead of 5, and then it starts over with 5, 5, 5, 4, and so on.



How is is possible to force the iterator to fill up shorter batches with elements coming from the next iteration so that all the batches have the same size?



P.S. just to understand, isn't this the obvious behavior when training a model?






tensorflow tensorflow-datasets







share|improve this question




















  • 1





    If you look at the documentation of tf.data.Dataset.batch, there is an optional drop_remainder parameter to discard the last incomplete batch. I don't know if there is a way to complete the last batch with the beginning of the next iteration. If you're shuffling your data it shouldn't matter discarding the last batch, but anyway it would be interesting to have a way to do that.

    – jdehesa
    Nov 19 '18 at 21:35
















1












1








1








I have a dataset with 19 elements and a batch size of 10. I set my dataset to continuously iterate over the same elements but I noticed that the last batch has only 4 elements instead of 5, and then it starts over with 5, 5, 5, 4, and so on.



How is is possible to force the iterator to fill up shorter batches with elements coming from the next iteration so that all the batches have the same size?



P.S. just to understand, isn't this the obvious behavior when training a model?






tensorflow tensorflow-datasets







share|improve this question
















I have a dataset with 19 elements and a batch size of 10. I set my dataset to continuously iterate over the same elements but I noticed that the last batch has only 4 elements instead of 5, and then it starts over with 5, 5, 5, 4, and so on.



How is is possible to force the iterator to fill up shorter batches with elements coming from the next iteration so that all the batches have the same size?



P.S. just to understand, isn't this the obvious behavior when training a model?






tensorflow tensorflow-datasets




tensorflow tensorflow-datasets






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 19 '18 at 21:26







petrux

















asked Nov 19 '18 at 21:17









petruxpetrux

688822




688822








  • 1





    If you look at the documentation of tf.data.Dataset.batch, there is an optional drop_remainder parameter to discard the last incomplete batch. I don't know if there is a way to complete the last batch with the beginning of the next iteration. If you're shuffling your data it shouldn't matter discarding the last batch, but anyway it would be interesting to have a way to do that.

    – jdehesa
    Nov 19 '18 at 21:35
















  • 1





    If you look at the documentation of tf.data.Dataset.batch, there is an optional drop_remainder parameter to discard the last incomplete batch. I don't know if there is a way to complete the last batch with the beginning of the next iteration. If you're shuffling your data it shouldn't matter discarding the last batch, but anyway it would be interesting to have a way to do that.

    – jdehesa
    Nov 19 '18 at 21:35










1




1





If you look at the documentation of tf.data.Dataset.batch, there is an optional drop_remainder parameter to discard the last incomplete batch. I don't know if there is a way to complete the last batch with the beginning of the next iteration. If you're shuffling your data it shouldn't matter discarding the last batch, but anyway it would be interesting to have a way to do that.

– jdehesa
Nov 19 '18 at 21:35







If you look at the documentation of tf.data.Dataset.batch, there is an optional drop_remainder parameter to discard the last incomplete batch. I don't know if there is a way to complete the last batch with the beginning of the next iteration. If you're shuffling your data it shouldn't matter discarding the last batch, but anyway it would be interesting to have a way to do that.

– jdehesa
Nov 19 '18 at 21:35














1 Answer
1






active

oldest

votes


















1














To have this behavior, the .repeat() method should be invoked before the batch() or padded_batch() one. So:



file_names = [...]

def my_map_func(record):

    ....

dataset = tf.data.TFRecordDataset(file_names)

    .map(map_func=my_map_func)

    .repeat()  # here!

    .batch(5)





share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382769%2ftf-data-dataset-with-constant-size-batches%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    To have this behavior, the .repeat() method should be invoked before the batch() or padded_batch() one. So:



    file_names = [...]
    
def my_map_func(record):
    
    ....
    
dataset = tf.data.TFRecordDataset(file_names)
    
    .map(map_func=my_map_func)
    
    .repeat()  # here!
    
    .batch(5)





    share|improve this answer




























      1














      To have this behavior, the .repeat() method should be invoked before the batch() or padded_batch() one. So:



      file_names = [...]
      
def my_map_func(record):
      
    ....
      
dataset = tf.data.TFRecordDataset(file_names)
      
    .map(map_func=my_map_func)
      
    .repeat()  # here!
      
    .batch(5)





      share|improve this answer


























        1












        1








        1







        To have this behavior, the .repeat() method should be invoked before the batch() or padded_batch() one. So:



        file_names = [...]
        
def my_map_func(record):
        
    ....
        
dataset = tf.data.TFRecordDataset(file_names)
        
    .map(map_func=my_map_func)
        
    .repeat()  # here!
        
    .batch(5)





        share|improve this answer













        To have this behavior, the .repeat() method should be invoked before the batch() or padded_batch() one. So:



        file_names = [...]
        
def my_map_func(record):
        
    ....
        
dataset = tf.data.TFRecordDataset(file_names)
        
    .map(map_func=my_map_func)
        
    .repeat()  # here!
        
    .batch(5)






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 20 '18 at 8:36









        petruxpetrux

        688822




        688822
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382769%2ftf-data-dataset-with-constant-size-batches%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            這個網誌中的熱門文章

            Xamarin.form Move up view when keyboard appear

            圖片
            .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I'm trying to build a Chat app UI, the idea of the Layout was pretty simple: When the input bar is focused, keyboard show up and "push" up the chat bar, as it's a grid, the ListView will resize to fit the screen: I update the input bar's margin to "push" it up: NSValue result = (NSValue)args.Notification.UserInfo.ObjectForKey(new NSString(UIKeyboard.FrameEndUserInfoKey)); CGSize keyboardSize = result.RectangleFValue.Size; if (Element != null){ Element.Margin = new Thickness(0, 0, 0,keyboardSize.Height); //push the entry up to keyboard height when keyboard is activated } And this is ...
            閱讀完整內容

            Post-Redirect-Get with Spring WebFlux and Thymeleaf

            圖片
            .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 2 Can someone explain to me how to implement the post-redirect-get pattern in Spring WebFlux and Thymeleaf? What subscribes on the database save method? @GetMapping("/register") public String showRegisterForm(Model model) { model.addAttribute("user", new User()); return "register"; } @PostMapping public String processRegisterForm(@Valid User user, BindingResult bindingResult) { if (bindingResult.hasErrors()) { return "register"; } else { userRepository.save(user); //what subscribes on this? //how to redirect on e.g. "/login"? ...
            閱讀完整內容

            Anylogic : not able to use stopDelay()

            圖片
            -2 I am building a railway station model. I want a train to depart when I click a button. I want to call stopDelay() for that but I am not sure how to do that and from which place. anylogic share | improve this question asked Nov 23 '18 at 10:12 Anupam Singh Anupam Singh 1 Hi Anupam, welcome to SOF. Please check how to ask good questions, see link below. Your question is...
            閱讀完整內容