Appending numpy arrays into a text file












2














I have a function which returns a 2 Dimensional numpy array and this function will be inside a loop. At each iteration, I want to append this numpy array into a file.



filename = "xyz"
for i in range(10):
np_array = function_to_get_numpy()
now append this `np_array` into filename


I can continue to append the numpy array in the script and dump once, but I want to avoid that.



Also I would prefer to store this in non-binary format.










share|improve this question






















  • np.savetxt writes a csv style output. It can take a file name, or an opened file. If you open the file in append mode, you can write to the same file multiple times. Alternatively you could format each 'row' of the array how ever you want, and use a normal file write. For text files, regular Python file writes are just fine (even print with a file parameter).
    – hpaulj
    Nov 11 at 21:23


















2














I have a function which returns a 2 Dimensional numpy array and this function will be inside a loop. At each iteration, I want to append this numpy array into a file.



filename = "xyz"
for i in range(10):
np_array = function_to_get_numpy()
now append this `np_array` into filename


I can continue to append the numpy array in the script and dump once, but I want to avoid that.



Also I would prefer to store this in non-binary format.










share|improve this question






















  • np.savetxt writes a csv style output. It can take a file name, or an opened file. If you open the file in append mode, you can write to the same file multiple times. Alternatively you could format each 'row' of the array how ever you want, and use a normal file write. For text files, regular Python file writes are just fine (even print with a file parameter).
    – hpaulj
    Nov 11 at 21:23
















2












2








2







I have a function which returns a 2 Dimensional numpy array and this function will be inside a loop. At each iteration, I want to append this numpy array into a file.



filename = "xyz"
for i in range(10):
np_array = function_to_get_numpy()
now append this `np_array` into filename


I can continue to append the numpy array in the script and dump once, but I want to avoid that.



Also I would prefer to store this in non-binary format.










share|improve this question













I have a function which returns a 2 Dimensional numpy array and this function will be inside a loop. At each iteration, I want to append this numpy array into a file.



filename = "xyz"
for i in range(10):
np_array = function_to_get_numpy()
now append this `np_array` into filename


I can continue to append the numpy array in the script and dump once, but I want to avoid that.



Also I would prefer to store this in non-binary format.







python-3.x numpy






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 11 at 21:07









random_28

728515




728515












  • np.savetxt writes a csv style output. It can take a file name, or an opened file. If you open the file in append mode, you can write to the same file multiple times. Alternatively you could format each 'row' of the array how ever you want, and use a normal file write. For text files, regular Python file writes are just fine (even print with a file parameter).
    – hpaulj
    Nov 11 at 21:23




















  • np.savetxt writes a csv style output. It can take a file name, or an opened file. If you open the file in append mode, you can write to the same file multiple times. Alternatively you could format each 'row' of the array how ever you want, and use a normal file write. For text files, regular Python file writes are just fine (even print with a file parameter).
    – hpaulj
    Nov 11 at 21:23


















np.savetxt writes a csv style output. It can take a file name, or an opened file. If you open the file in append mode, you can write to the same file multiple times. Alternatively you could format each 'row' of the array how ever you want, and use a normal file write. For text files, regular Python file writes are just fine (even print with a file parameter).
– hpaulj
Nov 11 at 21:23






np.savetxt writes a csv style output. It can take a file name, or an opened file. If you open the file in append mode, you can write to the same file multiple times. Alternatively you could format each 'row' of the array how ever you want, and use a normal file write. For text files, regular Python file writes are just fine (even print with a file parameter).
– hpaulj
Nov 11 at 21:23














2 Answers
2






active

oldest

votes


















2














In [64]: with open('xyz','w') as f:
...: for n in range(1,4):
...: arr = np.arange(n*n).reshape(n,n)
...: np.savetxt(f, arr, fmt='%5d', delimiter=',')
...:
In [65]: cat xyz
0
0, 1
2, 3
0, 1, 2
3, 4, 5
6, 7, 8


If the number of columns varies, as it does here, it will be hard(er) to read. csv readers like genfromtxt won't like it.



If the number of columns is consistent, it can be loaded as one big array. Separating the writes and reloading them is possible, but more involved.






share|improve this answer





























    0














    I'm going to take the opportunity to plug nppretty, a pretty printer for numpy that I've been working on. It provides a class ArrayStream that I think will do exactly what you need.



    Install nppretty with:



    pip install nppretty


    You can use ArrayStream much like a file object. For example, this code:



    from nppretty import ArrayStream
    import numpy as np

    arrstr = ArrayStream('arraystream.txt', name='arr2D')
    for i in range(10):
    arr = np.arange(10*i, 10*(i + 1))
    arrstr.write(arr.reshape(2,5))
    arrstr.close()


    will produce a text file called arraystream.txt with the following contents:



    arr2D = [
    [0, 1, 2, 3, 4],
    [5, 6, 7, 8, 9],
    [10, 11, 12, 13, 14],
    [15, 16, 17, 18, 19],
    [20, 21, 22, 23, 24],
    [25, 26, 27, 28, 29],
    [30, 31, 32, 33, 34],
    [35, 36, 37, 38, 39],
    [40, 41, 42, 43, 44],
    [45, 46, 47, 48, 49],
    [50, 51, 52, 53, 54],
    [55, 56, 57, 58, 59],
    [60, 61, 62, 63, 64],
    [65, 66, 67, 68, 69],
    [70, 71, 72, 73, 74],
    [75, 76, 77, 78, 79],
    [80, 81, 82, 83, 84],
    [85, 86, 87, 88, 89],
    [90, 91, 92, 93, 94],
    [95, 96, 97, 98, 99],
    ]


    Notes on ArrayStream



    ArrayStream accepts all of the same arguments as the standard Python open method. The one additional keyword arg is name, which sets the name of the array in the file (name defaults to "array" if left blank). Just like the file object returned by a call to open, an ArrayStream instance can used in a with statement. For example, the following code will produce the same output as above:



    with ArrayStream('arraystream.txt', name='arr2D') as f:
    for i in range(10):
    arr = np.arange(10*i, 10*(i + 1))
    f.write(arr.reshape(2,5))





    share|improve this answer























      Your Answer






      StackExchange.ifUsing("editor", function () {
      StackExchange.using("externalEditor", function () {
      StackExchange.using("snippets", function () {
      StackExchange.snippets.init();
      });
      });
      }, "code-snippets");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "1"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53253244%2fappending-numpy-arrays-into-a-text-file%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      2














      In [64]: with open('xyz','w') as f:
      ...: for n in range(1,4):
      ...: arr = np.arange(n*n).reshape(n,n)
      ...: np.savetxt(f, arr, fmt='%5d', delimiter=',')
      ...:
      In [65]: cat xyz
      0
      0, 1
      2, 3
      0, 1, 2
      3, 4, 5
      6, 7, 8


      If the number of columns varies, as it does here, it will be hard(er) to read. csv readers like genfromtxt won't like it.



      If the number of columns is consistent, it can be loaded as one big array. Separating the writes and reloading them is possible, but more involved.






      share|improve this answer


























        2














        In [64]: with open('xyz','w') as f:
        ...: for n in range(1,4):
        ...: arr = np.arange(n*n).reshape(n,n)
        ...: np.savetxt(f, arr, fmt='%5d', delimiter=',')
        ...:
        In [65]: cat xyz
        0
        0, 1
        2, 3
        0, 1, 2
        3, 4, 5
        6, 7, 8


        If the number of columns varies, as it does here, it will be hard(er) to read. csv readers like genfromtxt won't like it.



        If the number of columns is consistent, it can be loaded as one big array. Separating the writes and reloading them is possible, but more involved.






        share|improve this answer
























          2












          2








          2






          In [64]: with open('xyz','w') as f:
          ...: for n in range(1,4):
          ...: arr = np.arange(n*n).reshape(n,n)
          ...: np.savetxt(f, arr, fmt='%5d', delimiter=',')
          ...:
          In [65]: cat xyz
          0
          0, 1
          2, 3
          0, 1, 2
          3, 4, 5
          6, 7, 8


          If the number of columns varies, as it does here, it will be hard(er) to read. csv readers like genfromtxt won't like it.



          If the number of columns is consistent, it can be loaded as one big array. Separating the writes and reloading them is possible, but more involved.






          share|improve this answer












          In [64]: with open('xyz','w') as f:
          ...: for n in range(1,4):
          ...: arr = np.arange(n*n).reshape(n,n)
          ...: np.savetxt(f, arr, fmt='%5d', delimiter=',')
          ...:
          In [65]: cat xyz
          0
          0, 1
          2, 3
          0, 1, 2
          3, 4, 5
          6, 7, 8


          If the number of columns varies, as it does here, it will be hard(er) to read. csv readers like genfromtxt won't like it.



          If the number of columns is consistent, it can be loaded as one big array. Separating the writes and reloading them is possible, but more involved.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 12 at 3:26









          hpaulj

          110k774140




          110k774140

























              0














              I'm going to take the opportunity to plug nppretty, a pretty printer for numpy that I've been working on. It provides a class ArrayStream that I think will do exactly what you need.



              Install nppretty with:



              pip install nppretty


              You can use ArrayStream much like a file object. For example, this code:



              from nppretty import ArrayStream
              import numpy as np

              arrstr = ArrayStream('arraystream.txt', name='arr2D')
              for i in range(10):
              arr = np.arange(10*i, 10*(i + 1))
              arrstr.write(arr.reshape(2,5))
              arrstr.close()


              will produce a text file called arraystream.txt with the following contents:



              arr2D = [
              [0, 1, 2, 3, 4],
              [5, 6, 7, 8, 9],
              [10, 11, 12, 13, 14],
              [15, 16, 17, 18, 19],
              [20, 21, 22, 23, 24],
              [25, 26, 27, 28, 29],
              [30, 31, 32, 33, 34],
              [35, 36, 37, 38, 39],
              [40, 41, 42, 43, 44],
              [45, 46, 47, 48, 49],
              [50, 51, 52, 53, 54],
              [55, 56, 57, 58, 59],
              [60, 61, 62, 63, 64],
              [65, 66, 67, 68, 69],
              [70, 71, 72, 73, 74],
              [75, 76, 77, 78, 79],
              [80, 81, 82, 83, 84],
              [85, 86, 87, 88, 89],
              [90, 91, 92, 93, 94],
              [95, 96, 97, 98, 99],
              ]


              Notes on ArrayStream



              ArrayStream accepts all of the same arguments as the standard Python open method. The one additional keyword arg is name, which sets the name of the array in the file (name defaults to "array" if left blank). Just like the file object returned by a call to open, an ArrayStream instance can used in a with statement. For example, the following code will produce the same output as above:



              with ArrayStream('arraystream.txt', name='arr2D') as f:
              for i in range(10):
              arr = np.arange(10*i, 10*(i + 1))
              f.write(arr.reshape(2,5))





              share|improve this answer




























                0














                I'm going to take the opportunity to plug nppretty, a pretty printer for numpy that I've been working on. It provides a class ArrayStream that I think will do exactly what you need.



                Install nppretty with:



                pip install nppretty


                You can use ArrayStream much like a file object. For example, this code:



                from nppretty import ArrayStream
                import numpy as np

                arrstr = ArrayStream('arraystream.txt', name='arr2D')
                for i in range(10):
                arr = np.arange(10*i, 10*(i + 1))
                arrstr.write(arr.reshape(2,5))
                arrstr.close()


                will produce a text file called arraystream.txt with the following contents:



                arr2D = [
                [0, 1, 2, 3, 4],
                [5, 6, 7, 8, 9],
                [10, 11, 12, 13, 14],
                [15, 16, 17, 18, 19],
                [20, 21, 22, 23, 24],
                [25, 26, 27, 28, 29],
                [30, 31, 32, 33, 34],
                [35, 36, 37, 38, 39],
                [40, 41, 42, 43, 44],
                [45, 46, 47, 48, 49],
                [50, 51, 52, 53, 54],
                [55, 56, 57, 58, 59],
                [60, 61, 62, 63, 64],
                [65, 66, 67, 68, 69],
                [70, 71, 72, 73, 74],
                [75, 76, 77, 78, 79],
                [80, 81, 82, 83, 84],
                [85, 86, 87, 88, 89],
                [90, 91, 92, 93, 94],
                [95, 96, 97, 98, 99],
                ]


                Notes on ArrayStream



                ArrayStream accepts all of the same arguments as the standard Python open method. The one additional keyword arg is name, which sets the name of the array in the file (name defaults to "array" if left blank). Just like the file object returned by a call to open, an ArrayStream instance can used in a with statement. For example, the following code will produce the same output as above:



                with ArrayStream('arraystream.txt', name='arr2D') as f:
                for i in range(10):
                arr = np.arange(10*i, 10*(i + 1))
                f.write(arr.reshape(2,5))





                share|improve this answer


























                  0












                  0








                  0






                  I'm going to take the opportunity to plug nppretty, a pretty printer for numpy that I've been working on. It provides a class ArrayStream that I think will do exactly what you need.



                  Install nppretty with:



                  pip install nppretty


                  You can use ArrayStream much like a file object. For example, this code:



                  from nppretty import ArrayStream
                  import numpy as np

                  arrstr = ArrayStream('arraystream.txt', name='arr2D')
                  for i in range(10):
                  arr = np.arange(10*i, 10*(i + 1))
                  arrstr.write(arr.reshape(2,5))
                  arrstr.close()


                  will produce a text file called arraystream.txt with the following contents:



                  arr2D = [
                  [0, 1, 2, 3, 4],
                  [5, 6, 7, 8, 9],
                  [10, 11, 12, 13, 14],
                  [15, 16, 17, 18, 19],
                  [20, 21, 22, 23, 24],
                  [25, 26, 27, 28, 29],
                  [30, 31, 32, 33, 34],
                  [35, 36, 37, 38, 39],
                  [40, 41, 42, 43, 44],
                  [45, 46, 47, 48, 49],
                  [50, 51, 52, 53, 54],
                  [55, 56, 57, 58, 59],
                  [60, 61, 62, 63, 64],
                  [65, 66, 67, 68, 69],
                  [70, 71, 72, 73, 74],
                  [75, 76, 77, 78, 79],
                  [80, 81, 82, 83, 84],
                  [85, 86, 87, 88, 89],
                  [90, 91, 92, 93, 94],
                  [95, 96, 97, 98, 99],
                  ]


                  Notes on ArrayStream



                  ArrayStream accepts all of the same arguments as the standard Python open method. The one additional keyword arg is name, which sets the name of the array in the file (name defaults to "array" if left blank). Just like the file object returned by a call to open, an ArrayStream instance can used in a with statement. For example, the following code will produce the same output as above:



                  with ArrayStream('arraystream.txt', name='arr2D') as f:
                  for i in range(10):
                  arr = np.arange(10*i, 10*(i + 1))
                  f.write(arr.reshape(2,5))





                  share|improve this answer














                  I'm going to take the opportunity to plug nppretty, a pretty printer for numpy that I've been working on. It provides a class ArrayStream that I think will do exactly what you need.



                  Install nppretty with:



                  pip install nppretty


                  You can use ArrayStream much like a file object. For example, this code:



                  from nppretty import ArrayStream
                  import numpy as np

                  arrstr = ArrayStream('arraystream.txt', name='arr2D')
                  for i in range(10):
                  arr = np.arange(10*i, 10*(i + 1))
                  arrstr.write(arr.reshape(2,5))
                  arrstr.close()


                  will produce a text file called arraystream.txt with the following contents:



                  arr2D = [
                  [0, 1, 2, 3, 4],
                  [5, 6, 7, 8, 9],
                  [10, 11, 12, 13, 14],
                  [15, 16, 17, 18, 19],
                  [20, 21, 22, 23, 24],
                  [25, 26, 27, 28, 29],
                  [30, 31, 32, 33, 34],
                  [35, 36, 37, 38, 39],
                  [40, 41, 42, 43, 44],
                  [45, 46, 47, 48, 49],
                  [50, 51, 52, 53, 54],
                  [55, 56, 57, 58, 59],
                  [60, 61, 62, 63, 64],
                  [65, 66, 67, 68, 69],
                  [70, 71, 72, 73, 74],
                  [75, 76, 77, 78, 79],
                  [80, 81, 82, 83, 84],
                  [85, 86, 87, 88, 89],
                  [90, 91, 92, 93, 94],
                  [95, 96, 97, 98, 99],
                  ]


                  Notes on ArrayStream



                  ArrayStream accepts all of the same arguments as the standard Python open method. The one additional keyword arg is name, which sets the name of the array in the file (name defaults to "array" if left blank). Just like the file object returned by a call to open, an ArrayStream instance can used in a with statement. For example, the following code will produce the same output as above:



                  with ArrayStream('arraystream.txt', name='arr2D') as f:
                  for i in range(10):
                  arr = np.arange(10*i, 10*(i + 1))
                  f.write(arr.reshape(2,5))






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Nov 12 at 3:11

























                  answered Nov 12 at 2:39









                  tel

                  6,10311430




                  6,10311430






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Stack Overflow!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.





                      Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                      Please pay close attention to the following guidance:


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53253244%2fappending-numpy-arrays-into-a-text-file%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      這個網誌中的熱門文章

                      Xamarin.form Move up view when keyboard appear

                      Post-Redirect-Get with Spring WebFlux and Thymeleaf

                      Anylogic : not able to use stopDelay()