Java Multithread File Read + Operation + Write












1















I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading







share|improve this question























  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22


















1















I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading







share|improve this question























  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22
















1












1








1








I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading







share|improve this question














I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading




java multithreading






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 16 '18 at 22:01









user10665129user10665129

61




61













  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22





















  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22



















Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

– EdH
Nov 16 '18 at 22:22







Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

– EdH
Nov 16 '18 at 22:22














2 Answers
2






active

oldest

votes


















1














Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
files is a list of files you already read in.



files.parallelStream()

    .map(file -> new Pair(file, countWords(file)))

    .forEach((file, count) -> writeOutput(file, count));





share|improve this answer































    0














    I like your enthusiastic in learning multi-threaded programming!

    Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



    This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



    you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






    share|improve this answer























      Your Answer






      StackExchange.ifUsing("editor", function () {
      StackExchange.using("externalEditor", function () {
      StackExchange.using("snippets", function () {
      StackExchange.snippets.init();
      });
      });
      }, "code-snippets");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "1"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346013%2fjava-multithread-file-read-operation-write%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      1














      Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
      files is a list of files you already read in.



      files.parallelStream()
      
    .map(file -> new Pair(file, countWords(file)))
      
    .forEach((file, count) -> writeOutput(file, count));





      share|improve this answer




























        1














        Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
        files is a list of files you already read in.



        files.parallelStream()
        
    .map(file -> new Pair(file, countWords(file)))
        
    .forEach((file, count) -> writeOutput(file, count));





        share|improve this answer


























          1












          1








          1







          Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
          files is a list of files you already read in.



          files.parallelStream()
          
    .map(file -> new Pair(file, countWords(file)))
          
    .forEach((file, count) -> writeOutput(file, count));





          share|improve this answer













          Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
          files is a list of files you already read in.



          files.parallelStream()
          
    .map(file -> new Pair(file, countWords(file)))
          
    .forEach((file, count) -> writeOutput(file, count));






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 16 '18 at 22:24









          K. M. Fazle Azim BabuK. M. Fazle Azim Babu

          1264




          1264

























              0














              I like your enthusiastic in learning multi-threaded programming!

              Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



              This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



              you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






              share|improve this answer




























                0














                I like your enthusiastic in learning multi-threaded programming!

                Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



                This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



                you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






                share|improve this answer


























                  0












                  0








                  0







                  I like your enthusiastic in learning multi-threaded programming!

                  Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



                  This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



                  you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






                  share|improve this answer













                  I like your enthusiastic in learning multi-threaded programming!

                  Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



                  This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



                  you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Nov 16 '18 at 23:13









                  doronydorony

                  351111




                  351111






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Stack Overflow!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346013%2fjava-multithread-file-read-operation-write%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      這個網誌中的熱門文章

                      Tangent Lines Diagram Along Smooth Curve

                      圖片
                      7 1 I am trying to replicate a diagram and have a minimal example. I don't know how to add additional tangents between the specified points and preserve smoothness. documentclass{article} usepackage{tikz} usetikzlibrary{calc,intersections} begin{document} begin{tikzpicture} % Axes draw [->, name path=x] (-1,0) -- (11,0) node [right] {$x$}; draw [->] (0,-1) -- (0,6) node [above] {$y$}; % Origin node at (0,0) [below left] {$0$}; % Points coordinate (start) at (1,-0.8); coordinate (c1) at (3,3); coordinate (c2) at (5.5,1.5); coordinate (c3) at (8,4); coordinate (end) at (10.5,-0.8); % show the points foreach n in {start,c1,c2,c3,end} fill [black] (n) circle (2pt) node [below] {}; % join the coordinates draw [thick,name path=curve] (start) to[out=70,in=180] (c1) to[out=0,in=180]
                      閱讀完整內容

                      Yusuf al-Mu'taman ibn Hud

                      圖片
                      Main article: Taifa of Zaragoza Al-Mu'taman Ruler of Zaragoza Reign 1081–1085 Predecessor Ahmad al-Muqtadir Successor Al-Mustain II Born Zaragoza Died 1085 Full name Yusuf ibn Ahmad al-Mu'taman ibn Hūd House Banu Hud Yusuf ibn Ahmad al-Mu'taman ibn Hūd (Arabic: المؤتمن بالله يوسف إبن أحمد إبن هود ‎, al-Mutaman bi l-Lah , died c. 1085) was the third king of the Banu Hud dynasty who ruled over the Taifa of Zaragoza from 1081 to 1085. Yusuf al-Mutaman reigned during the height of power of Muslim Zaragoza , following the thriving period of his father Ahmad al-Muqtadir. He continued his father's efforts and created around him a court of intellectuals, living in the beautiful palace of Aljafería, nicknamed as "the palace of joy". Al-Mu'taman was also a scholarly king and a patron of science, philosophy and arts. As an example of a wise king, he knew astrology, philosophy, and especially mathematics, a discipline in which
                      閱讀完整內容

                      Zucchini

                      圖片
                      This article is about the vegetable. For other uses, see Zucchini (disambiguation). Zucchini (courgette) A striped and a uniform color zucchini Genus Cucurbita Species Cucurbita pepo Origin 19th century northern Italy The zucchini ( / z uː ˈ k iː n i / , American English) or courgette ( / k ʊər ˈ ʒ ɛ t / , British English) is a summer squash which can reach nearly 1 metre (100 cm; 39 in) in length, but is usually harvested when still immature at about 15 to 25 cm (6 to 10 in). [1] A zucchini is a thin-skinned cultivar of what in Britain and Ireland is referred to as a marrow. [2] [3] In South Africa, a zucchini is known as a baby marrow. Along with certain other squashes and pumpkins, the zucchini belongs to the species Cucurbita pepo . It can be dark or light green. A related hybrid, the golden zucchini, is a deep yellow or orange color. [4] In a culinary context, the zucchini is treated as a vegetable; it is usually cooked and presented as a savory dish o
                      閱讀完整內容