How to capture intermediate error in python subprocess's piped execution











up vote
3
down vote

favorite












Following python doc to replace shell-pipeline, I have a piece of code that looks like this.



p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=PIPE, stdout=PIPE)
p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stdout=outfile)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]


outfile is where I want to redirect the output of head command. The log file is very large and hence I am doing a 'head' on it



The chaining is like p1 | p2 | p3 | ..... | Pn > outfile



If there's an error in execution of p1 e.g. the user does not have read permissions on the /var/log/some_process_log.output file, error message in p1.stderr is not piped through when I do Pn.communicate()



If I do p1.stderr.readline() at every stage, then it takes long time to process. This is mentioned in pydocs:




Note The data read is buffered in memory, so do not use this method if
the data size is large or unlimited.




I am avoiding subprocess.check_output since it does not handle piping and plus it needs the unsafe shell=True



Any help would be appreciated. Thanks










share|improve this question


























    up vote
    3
    down vote

    favorite












    Following python doc to replace shell-pipeline, I have a piece of code that looks like this.



    p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=PIPE, stdout=PIPE)
    p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stdout=outfile)
    p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
    output = p2.communicate()[0]


    outfile is where I want to redirect the output of head command. The log file is very large and hence I am doing a 'head' on it



    The chaining is like p1 | p2 | p3 | ..... | Pn > outfile



    If there's an error in execution of p1 e.g. the user does not have read permissions on the /var/log/some_process_log.output file, error message in p1.stderr is not piped through when I do Pn.communicate()



    If I do p1.stderr.readline() at every stage, then it takes long time to process. This is mentioned in pydocs:




    Note The data read is buffered in memory, so do not use this method if
    the data size is large or unlimited.




    I am avoiding subprocess.check_output since it does not handle piping and plus it needs the unsafe shell=True



    Any help would be appreciated. Thanks










    share|improve this question
























      up vote
      3
      down vote

      favorite









      up vote
      3
      down vote

      favorite











      Following python doc to replace shell-pipeline, I have a piece of code that looks like this.



      p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=PIPE, stdout=PIPE)
      p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stdout=outfile)
      p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
      output = p2.communicate()[0]


      outfile is where I want to redirect the output of head command. The log file is very large and hence I am doing a 'head' on it



      The chaining is like p1 | p2 | p3 | ..... | Pn > outfile



      If there's an error in execution of p1 e.g. the user does not have read permissions on the /var/log/some_process_log.output file, error message in p1.stderr is not piped through when I do Pn.communicate()



      If I do p1.stderr.readline() at every stage, then it takes long time to process. This is mentioned in pydocs:




      Note The data read is buffered in memory, so do not use this method if
      the data size is large or unlimited.




      I am avoiding subprocess.check_output since it does not handle piping and plus it needs the unsafe shell=True



      Any help would be appreciated. Thanks










      share|improve this question













      Following python doc to replace shell-pipeline, I have a piece of code that looks like this.



      p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=PIPE, stdout=PIPE)
      p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stdout=outfile)
      p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
      output = p2.communicate()[0]


      outfile is where I want to redirect the output of head command. The log file is very large and hence I am doing a 'head' on it



      The chaining is like p1 | p2 | p3 | ..... | Pn > outfile



      If there's an error in execution of p1 e.g. the user does not have read permissions on the /var/log/some_process_log.output file, error message in p1.stderr is not piped through when I do Pn.communicate()



      If I do p1.stderr.readline() at every stage, then it takes long time to process. This is mentioned in pydocs:




      Note The data read is buffered in memory, so do not use this method if
      the data size is large or unlimited.




      I am avoiding subprocess.check_output since it does not handle piping and plus it needs the unsafe shell=True



      Any help would be appreciated. Thanks







      python subprocess pipe






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 8 at 22:09









      Yogesh lele

      167111




      167111
























          1 Answer
          1






          active

          oldest

          votes

















          up vote
          0
          down vote













          You could create a separate pipe with:



          import os
          errread, errwrite = os.pipe()


          And set the write-end as the stderr for all your Popen instances:



          p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=errwrite, stdout=PIPE)
          p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stderr=errwrite, stdout=outfile)


          Remember to close the write-end when it's done:



          os.close(errwrite)


          And get your error messages with either:



          data_group = os.read(errread, buf_size)


          or:



          import io
          data = io.open(errread, 'rb', buf_size).read()





          share|improve this answer





















          • Please note that, all error data is still buffered in memory.
            – Fantix King
            Nov 9 at 2:50











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53216903%2fhow-to-capture-intermediate-error-in-python-subprocesss-piped-execution%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          0
          down vote













          You could create a separate pipe with:



          import os
          errread, errwrite = os.pipe()


          And set the write-end as the stderr for all your Popen instances:



          p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=errwrite, stdout=PIPE)
          p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stderr=errwrite, stdout=outfile)


          Remember to close the write-end when it's done:



          os.close(errwrite)


          And get your error messages with either:



          data_group = os.read(errread, buf_size)


          or:



          import io
          data = io.open(errread, 'rb', buf_size).read()





          share|improve this answer





















          • Please note that, all error data is still buffered in memory.
            – Fantix King
            Nov 9 at 2:50















          up vote
          0
          down vote













          You could create a separate pipe with:



          import os
          errread, errwrite = os.pipe()


          And set the write-end as the stderr for all your Popen instances:



          p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=errwrite, stdout=PIPE)
          p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stderr=errwrite, stdout=outfile)


          Remember to close the write-end when it's done:



          os.close(errwrite)


          And get your error messages with either:



          data_group = os.read(errread, buf_size)


          or:



          import io
          data = io.open(errread, 'rb', buf_size).read()





          share|improve this answer





















          • Please note that, all error data is still buffered in memory.
            – Fantix King
            Nov 9 at 2:50













          up vote
          0
          down vote










          up vote
          0
          down vote









          You could create a separate pipe with:



          import os
          errread, errwrite = os.pipe()


          And set the write-end as the stderr for all your Popen instances:



          p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=errwrite, stdout=PIPE)
          p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stderr=errwrite, stdout=outfile)


          Remember to close the write-end when it's done:



          os.close(errwrite)


          And get your error messages with either:



          data_group = os.read(errread, buf_size)


          or:



          import io
          data = io.open(errread, 'rb', buf_size).read()





          share|improve this answer












          You could create a separate pipe with:



          import os
          errread, errwrite = os.pipe()


          And set the write-end as the stderr for all your Popen instances:



          p1 = Popen(["tac" , "/var/log/some_process_log.output"], stderr=errwrite, stdout=PIPE)
          p2 = Popen(["head", "-n", "1000"], stdin=p1.stdout, stderr=errwrite, stdout=outfile)


          Remember to close the write-end when it's done:



          os.close(errwrite)


          And get your error messages with either:



          data_group = os.read(errread, buf_size)


          or:



          import io
          data = io.open(errread, 'rb', buf_size).read()






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 9 at 2:47









          Fantix King

          693510




          693510












          • Please note that, all error data is still buffered in memory.
            – Fantix King
            Nov 9 at 2:50


















          • Please note that, all error data is still buffered in memory.
            – Fantix King
            Nov 9 at 2:50
















          Please note that, all error data is still buffered in memory.
          – Fantix King
          Nov 9 at 2:50




          Please note that, all error data is still buffered in memory.
          – Fantix King
          Nov 9 at 2:50


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53216903%2fhow-to-capture-intermediate-error-in-python-subprocesss-piped-execution%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          這個網誌中的熱門文章

          Xamarin.form Move up view when keyboard appear

          Post-Redirect-Get with Spring WebFlux and Thymeleaf

          Anylogic : not able to use stopDelay()