Issues in upgrading ML libraries using AWS Sagemaker Notebook's Lifecycle configurations












0














I use the following script to automate upgrading of my libraries.



My script (Start Notebook):



#!/bin/bash

set -e

echo 'Before:'
echo $PATH

export PATH=/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:$PATH

echo 'After:'
echo $PATH

echo `pwd`

source /home/ec2-user/anaconda3/bin/activate tensorflow_p36

pip install pandas --upgrade

pip install tensorflow-gpu --upgrade


Error:



I get the following error, how can I point to the correct location(/home/ec2-user) of keras instead of /root



cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory 


Full Logs:



Before:
/sbin:/bin:/usr/sbin:/usr/bin
After:
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/sbin:/bin:/usr/sbin:/usr/bin
/home/ec2-user
cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory


Without lifecycle configuration:



All the commands in the above script works.



Actual keras.json file is exising, under /home/ec2-user when I remove the lifecycle configuration with the following value.



sh-4.2$ cat .keras/keras.json
{
"backend": "tensorflow"
}









share|improve this question



























    0














    I use the following script to automate upgrading of my libraries.



    My script (Start Notebook):



    #!/bin/bash

    set -e

    echo 'Before:'
    echo $PATH

    export PATH=/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:$PATH

    echo 'After:'
    echo $PATH

    echo `pwd`

    source /home/ec2-user/anaconda3/bin/activate tensorflow_p36

    pip install pandas --upgrade

    pip install tensorflow-gpu --upgrade


    Error:



    I get the following error, how can I point to the correct location(/home/ec2-user) of keras instead of /root



    cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory 


    Full Logs:



    Before:
    /sbin:/bin:/usr/sbin:/usr/bin
    After:
    /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/sbin:/bin:/usr/sbin:/usr/bin
    /home/ec2-user
    cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory


    Without lifecycle configuration:



    All the commands in the above script works.



    Actual keras.json file is exising, under /home/ec2-user when I remove the lifecycle configuration with the following value.



    sh-4.2$ cat .keras/keras.json
    {
    "backend": "tensorflow"
    }









    share|improve this question

























      0












      0








      0







      I use the following script to automate upgrading of my libraries.



      My script (Start Notebook):



      #!/bin/bash

      set -e

      echo 'Before:'
      echo $PATH

      export PATH=/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:$PATH

      echo 'After:'
      echo $PATH

      echo `pwd`

      source /home/ec2-user/anaconda3/bin/activate tensorflow_p36

      pip install pandas --upgrade

      pip install tensorflow-gpu --upgrade


      Error:



      I get the following error, how can I point to the correct location(/home/ec2-user) of keras instead of /root



      cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory 


      Full Logs:



      Before:
      /sbin:/bin:/usr/sbin:/usr/bin
      After:
      /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/sbin:/bin:/usr/sbin:/usr/bin
      /home/ec2-user
      cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory


      Without lifecycle configuration:



      All the commands in the above script works.



      Actual keras.json file is exising, under /home/ec2-user when I remove the lifecycle configuration with the following value.



      sh-4.2$ cat .keras/keras.json
      {
      "backend": "tensorflow"
      }









      share|improve this question













      I use the following script to automate upgrading of my libraries.



      My script (Start Notebook):



      #!/bin/bash

      set -e

      echo 'Before:'
      echo $PATH

      export PATH=/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:$PATH

      echo 'After:'
      echo $PATH

      echo `pwd`

      source /home/ec2-user/anaconda3/bin/activate tensorflow_p36

      pip install pandas --upgrade

      pip install tensorflow-gpu --upgrade


      Error:



      I get the following error, how can I point to the correct location(/home/ec2-user) of keras instead of /root



      cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory 


      Full Logs:



      Before:
      /sbin:/bin:/usr/sbin:/usr/bin
      After:
      /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin:/home/ec2-user/anaconda3/bin/:/usr/libexec/gcc/x86_64-amazon-linux/4.8.5:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/sbin:/bin:/usr/sbin:/usr/bin
      /home/ec2-user
      cp: cannot stat ‘/root/.keras/keras_tensorflow.json’: No such file or directory


      Without lifecycle configuration:



      All the commands in the above script works.



      Actual keras.json file is exising, under /home/ec2-user when I remove the lifecycle configuration with the following value.



      sh-4.2$ cat .keras/keras.json
      {
      "backend": "tensorflow"
      }






      python tensorflow keras amazon-sagemaker amazon-machine-learning






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 12 '18 at 9:55









      aspdeepak

      1,00511529




      1,00511529
























          1 Answer
          1






          active

          oldest

          votes


















          1














          Looks like pip install in your case executed "outside" of virtualenv



          try to change from:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36



          to:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36 && pip install pandas tensorflow-gpu --upgrade



          and delete redundant lines






          share|improve this answer























          • Thanks @EI Ruso it worked!!
            – aspdeepak
            Dec 12 '18 at 10:04













          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53259647%2fissues-in-upgrading-ml-libraries-using-aws-sagemaker-notebooks-lifecycle-config%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          Looks like pip install in your case executed "outside" of virtualenv



          try to change from:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36



          to:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36 && pip install pandas tensorflow-gpu --upgrade



          and delete redundant lines






          share|improve this answer























          • Thanks @EI Ruso it worked!!
            – aspdeepak
            Dec 12 '18 at 10:04


















          1














          Looks like pip install in your case executed "outside" of virtualenv



          try to change from:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36



          to:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36 && pip install pandas tensorflow-gpu --upgrade



          and delete redundant lines






          share|improve this answer























          • Thanks @EI Ruso it worked!!
            – aspdeepak
            Dec 12 '18 at 10:04
















          1












          1








          1






          Looks like pip install in your case executed "outside" of virtualenv



          try to change from:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36



          to:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36 && pip install pandas tensorflow-gpu --upgrade



          and delete redundant lines






          share|improve this answer














          Looks like pip install in your case executed "outside" of virtualenv



          try to change from:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36



          to:



          source /home/ec2-user/anaconda3/bin/activate tensorflow_p36 && pip install pandas tensorflow-gpu --upgrade



          and delete redundant lines







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Dec 12 '18 at 10:25









          aspdeepak

          1,00511529




          1,00511529










          answered Nov 13 '18 at 16:55









          El Ruso

          2,14821433




          2,14821433












          • Thanks @EI Ruso it worked!!
            – aspdeepak
            Dec 12 '18 at 10:04




















          • Thanks @EI Ruso it worked!!
            – aspdeepak
            Dec 12 '18 at 10:04


















          Thanks @EI Ruso it worked!!
          – aspdeepak
          Dec 12 '18 at 10:04






          Thanks @EI Ruso it worked!!
          – aspdeepak
          Dec 12 '18 at 10:04




















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53259647%2fissues-in-upgrading-ml-libraries-using-aws-sagemaker-notebooks-lifecycle-config%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          這個網誌中的熱門文章

          Xamarin.form Move up view when keyboard appear

          Post-Redirect-Get with Spring WebFlux and Thymeleaf

          Anylogic : not able to use stopDelay()