How to replace escaped newline in spark











up vote
0
down vote

favorite












I have a csv, that is not quoted, have added an example below



New lines are escaped with , as shown in the 2nd row, is there a way to replace that with some other character using apache spark..



Input CSV



Banana,23,Male,5,11,2017

Cat,32,Fe

male,2,11,2017

Dragon,28,Male,1,11,2017


Expected Output



Banana,23,Male,5,11,2017

Cat,32,Fe-male,2,11,2017

Dragon,28,Male,1,11,2017


Note: the original file is huge (around 40GB)



Edit 1
I just found an answer to use "sc. wholeTextFiles" instead of "sc.textFile", but given the big size I m not sure if it is memory efficient, please advise






apache-spark pyspark







share|improve this question




























    up vote
    0
    down vote

    favorite












    I have a csv, that is not quoted, have added an example below



    New lines are escaped with , as shown in the 2nd row, is there a way to replace that with some other character using apache spark..



    Input CSV



    Banana,23,Male,5,11,2017
    
Cat,32,Fe
    
male,2,11,2017
    
Dragon,28,Male,1,11,2017


    Expected Output



    Banana,23,Male,5,11,2017
    
Cat,32,Fe-male,2,11,2017
    
Dragon,28,Male,1,11,2017


    Note: the original file is huge (around 40GB)



    Edit 1
    I just found an answer to use "sc. wholeTextFiles" instead of "sc.textFile", but given the big size I m not sure if it is memory efficient, please advise






    apache-spark pyspark







    share|improve this question


























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I have a csv, that is not quoted, have added an example below



      New lines are escaped with , as shown in the 2nd row, is there a way to replace that with some other character using apache spark..



      Input CSV



      Banana,23,Male,5,11,2017
      
Cat,32,Fe
      
male,2,11,2017
      
Dragon,28,Male,1,11,2017


      Expected Output



      Banana,23,Male,5,11,2017
      
Cat,32,Fe-male,2,11,2017
      
Dragon,28,Male,1,11,2017


      Note: the original file is huge (around 40GB)



      Edit 1
      I just found an answer to use "sc. wholeTextFiles" instead of "sc.textFile", but given the big size I m not sure if it is memory efficient, please advise






      apache-spark pyspark







      share|improve this question















      I have a csv, that is not quoted, have added an example below



      New lines are escaped with , as shown in the 2nd row, is there a way to replace that with some other character using apache spark..



      Input CSV



      Banana,23,Male,5,11,2017
      
Cat,32,Fe
      
male,2,11,2017
      
Dragon,28,Male,1,11,2017


      Expected Output



      Banana,23,Male,5,11,2017
      
Cat,32,Fe-male,2,11,2017
      
Dragon,28,Male,1,11,2017


      Note: the original file is huge (around 40GB)



      Edit 1
      I just found an answer to use "sc. wholeTextFiles" instead of "sc.textFile", but given the big size I m not sure if it is memory efficient, please advise






      apache-spark pyspark




      apache-spark pyspark






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 5 at 2:03

























      asked Nov 5 at 1:42









      Geethanadh

      327




      327





























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147280%2fhow-to-replace-escaped-newline-in-spark%23new-answer', 'question_page');
          }
          );

          Post as a guest























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















           

          draft saved


          draft discarded



















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147280%2fhow-to-replace-escaped-newline-in-spark%23new-answer', 'question_page');
          }
          );

          Post as a guest






































































          -

          這個網誌中的熱門文章

          Xamarin.form Move up view when keyboard appear

          圖片
          .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I'm trying to build a Chat app UI, the idea of the Layout was pretty simple: When the input bar is focused, keyboard show up and "push" up the chat bar, as it's a grid, the ListView will resize to fit the screen: I update the input bar's margin to "push" it up: NSValue result = (NSValue)args.Notification.UserInfo.ObjectForKey(new NSString(UIKeyboard.FrameEndUserInfoKey)); CGSize keyboardSize = result.RectangleFValue.Size; if (Element != null){ Element.Margin = new Thickness(0, 0, 0,keyboardSize.Height); //push the entry up to keyboard height when keyboard is activated } And this is ...
          閱讀完整內容

          Post-Redirect-Get with Spring WebFlux and Thymeleaf

          圖片
          .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 2 Can someone explain to me how to implement the post-redirect-get pattern in Spring WebFlux and Thymeleaf? What subscribes on the database save method? @GetMapping("/register") public String showRegisterForm(Model model) { model.addAttribute("user", new User()); return "register"; } @PostMapping public String processRegisterForm(@Valid User user, BindingResult bindingResult) { if (bindingResult.hasErrors()) { return "register"; } else { userRepository.save(user); //what subscribes on this? //how to redirect on e.g. "/login"? ...
          閱讀完整內容

          Anylogic : not able to use stopDelay()

          圖片
          -2 I am building a railway station model. I want a train to depart when I click a button. I want to call stopDelay() for that but I am not sure how to do that and from which place. anylogic share | improve this question asked Nov 23 '18 at 10:12 Anupam Singh Anupam Singh 1 Hi Anupam, welcome to SOF. Please check how to ask good questions, see link below. Your question is...
          閱讀完整內容