AWS CodeBuild fails for golang source with sub-packages












0















When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:



The application is organized as follows:



/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...


This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>) where <some path> varies according to what triggers the build.



I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN), but since the CodeBuild environment variable for the GOPATH directory inserts /go: (/go:/codebuild/output/srcNNN) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.



My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH conforms to my assumptions.



Any ideas would be appreciated.





Clarification:
When importing packages in the code, external packages are imported as follows:



import (
"context"
...
}


Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app). However, AWS CodeBuild doesn't bring in the code this way.










share|improve this question

























  • Can you please show how do you import your packages in the code?

    – Alex Pliutau
    Nov 20 '18 at 8:59











  • Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.

    – Seva
    Nov 20 '18 at 9:26











  • Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.

    – ajl
    Nov 20 '18 at 16:05
















0















When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:



The application is organized as follows:



/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...


This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>) where <some path> varies according to what triggers the build.



I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN), but since the CodeBuild environment variable for the GOPATH directory inserts /go: (/go:/codebuild/output/srcNNN) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.



My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH conforms to my assumptions.



Any ideas would be appreciated.





Clarification:
When importing packages in the code, external packages are imported as follows:



import (
"context"
...
}


Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app). However, AWS CodeBuild doesn't bring in the code this way.










share|improve this question

























  • Can you please show how do you import your packages in the code?

    – Alex Pliutau
    Nov 20 '18 at 8:59











  • Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.

    – Seva
    Nov 20 '18 at 9:26











  • Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.

    – ajl
    Nov 20 '18 at 16:05














0












0








0








When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:



The application is organized as follows:



/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...


This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>) where <some path> varies according to what triggers the build.



I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN), but since the CodeBuild environment variable for the GOPATH directory inserts /go: (/go:/codebuild/output/srcNNN) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.



My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH conforms to my assumptions.



Any ideas would be appreciated.





Clarification:
When importing packages in the code, external packages are imported as follows:



import (
"context"
...
}


Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app). However, AWS CodeBuild doesn't bring in the code this way.










share|improve this question
















When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:



The application is organized as follows:



/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...


This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>) where <some path> varies according to what triggers the build.



I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN), but since the CodeBuild environment variable for the GOPATH directory inserts /go: (/go:/codebuild/output/srcNNN) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.



My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH conforms to my assumptions.



Any ideas would be appreciated.





Clarification:
When importing packages in the code, external packages are imported as follows:



import (
"context"
...
}


Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app). However, AWS CodeBuild doesn't bring in the code this way.







go aws-codebuild






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 20 '18 at 15:59







ajl

















asked Nov 19 '18 at 20:30









ajlajl

42148




42148













  • Can you please show how do you import your packages in the code?

    – Alex Pliutau
    Nov 20 '18 at 8:59











  • Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.

    – Seva
    Nov 20 '18 at 9:26











  • Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.

    – ajl
    Nov 20 '18 at 16:05



















  • Can you please show how do you import your packages in the code?

    – Alex Pliutau
    Nov 20 '18 at 8:59











  • Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.

    – Seva
    Nov 20 '18 at 9:26











  • Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.

    – ajl
    Nov 20 '18 at 16:05

















Can you please show how do you import your packages in the code?

– Alex Pliutau
Nov 20 '18 at 8:59





Can you please show how do you import your packages in the code?

– Alex Pliutau
Nov 20 '18 at 8:59













Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.

– Seva
Nov 20 '18 at 9:26





Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.

– Seva
Nov 20 '18 at 9:26













Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.

– ajl
Nov 20 '18 at 16:05





Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.

– ajl
Nov 20 '18 at 16:05












1 Answer
1






active

oldest

votes


















0














I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.



In my buildspec.yaml I am able to get the build to work by:




  1. Getting ${THEGOPATH} from ${GOPATH} by removing the "/go:" from the beginning

  2. Copying all the code to the ${THEGOPATH}/src/<app path>

  3. Copying other repositories to ${THEGOPATH}/src/<other app path>

  4. Importing external dependencies as normal (in our case, go get ./... or explicit)

  5. Building and forcing the output name (when launched from CodeBuild it used a different directory name)


The buildspec.yaml looks something like the following:



phases:
install:
commands:
- echo GOPATH - $GOPATH
- export THEGOPATH=`echo $GOPATH | cut -c 5-`
- echo THEGOPATH = $THEGOPATH
- mkdir -p ${THEGOPATH}/src/company/app1
- mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
- echo Copy source files to go root
- cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
- ls ${THEGOPATH}/src/
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo DOING THE BUILD
- go build -ldflags "<some flags>" -o "appname"
- go test ./...
post_build:
commands:
- echo Build completed on `date`
- ls -al
- pwd
artifacts:
files:
- appname





share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382185%2faws-codebuild-fails-for-golang-source-with-sub-packages%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.



    In my buildspec.yaml I am able to get the build to work by:




    1. Getting ${THEGOPATH} from ${GOPATH} by removing the "/go:" from the beginning

    2. Copying all the code to the ${THEGOPATH}/src/<app path>

    3. Copying other repositories to ${THEGOPATH}/src/<other app path>

    4. Importing external dependencies as normal (in our case, go get ./... or explicit)

    5. Building and forcing the output name (when launched from CodeBuild it used a different directory name)


    The buildspec.yaml looks something like the following:



    phases:
    install:
    commands:
    - echo GOPATH - $GOPATH
    - export THEGOPATH=`echo $GOPATH | cut -c 5-`
    - echo THEGOPATH = $THEGOPATH
    - mkdir -p ${THEGOPATH}/src/company/app1
    - mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
    - echo Copy source files to go root
    - cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
    - cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
    - ls ${THEGOPATH}/src/
    build:
    commands:
    - echo Build started on `date`
    - echo Getting packages
    - go get ./...
    - echo DOING THE BUILD
    - go build -ldflags "<some flags>" -o "appname"
    - go test ./...
    post_build:
    commands:
    - echo Build completed on `date`
    - ls -al
    - pwd
    artifacts:
    files:
    - appname





    share|improve this answer




























      0














      I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.



      In my buildspec.yaml I am able to get the build to work by:




      1. Getting ${THEGOPATH} from ${GOPATH} by removing the "/go:" from the beginning

      2. Copying all the code to the ${THEGOPATH}/src/<app path>

      3. Copying other repositories to ${THEGOPATH}/src/<other app path>

      4. Importing external dependencies as normal (in our case, go get ./... or explicit)

      5. Building and forcing the output name (when launched from CodeBuild it used a different directory name)


      The buildspec.yaml looks something like the following:



      phases:
      install:
      commands:
      - echo GOPATH - $GOPATH
      - export THEGOPATH=`echo $GOPATH | cut -c 5-`
      - echo THEGOPATH = $THEGOPATH
      - mkdir -p ${THEGOPATH}/src/company/app1
      - mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
      - echo Copy source files to go root
      - cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
      - cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
      - ls ${THEGOPATH}/src/
      build:
      commands:
      - echo Build started on `date`
      - echo Getting packages
      - go get ./...
      - echo DOING THE BUILD
      - go build -ldflags "<some flags>" -o "appname"
      - go test ./...
      post_build:
      commands:
      - echo Build completed on `date`
      - ls -al
      - pwd
      artifacts:
      files:
      - appname





      share|improve this answer


























        0












        0








        0







        I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.



        In my buildspec.yaml I am able to get the build to work by:




        1. Getting ${THEGOPATH} from ${GOPATH} by removing the "/go:" from the beginning

        2. Copying all the code to the ${THEGOPATH}/src/<app path>

        3. Copying other repositories to ${THEGOPATH}/src/<other app path>

        4. Importing external dependencies as normal (in our case, go get ./... or explicit)

        5. Building and forcing the output name (when launched from CodeBuild it used a different directory name)


        The buildspec.yaml looks something like the following:



        phases:
        install:
        commands:
        - echo GOPATH - $GOPATH
        - export THEGOPATH=`echo $GOPATH | cut -c 5-`
        - echo THEGOPATH = $THEGOPATH
        - mkdir -p ${THEGOPATH}/src/company/app1
        - mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
        - echo Copy source files to go root
        - cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
        - cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
        - ls ${THEGOPATH}/src/
        build:
        commands:
        - echo Build started on `date`
        - echo Getting packages
        - go get ./...
        - echo DOING THE BUILD
        - go build -ldflags "<some flags>" -o "appname"
        - go test ./...
        post_build:
        commands:
        - echo Build completed on `date`
        - ls -al
        - pwd
        artifacts:
        files:
        - appname





        share|improve this answer













        I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.



        In my buildspec.yaml I am able to get the build to work by:




        1. Getting ${THEGOPATH} from ${GOPATH} by removing the "/go:" from the beginning

        2. Copying all the code to the ${THEGOPATH}/src/<app path>

        3. Copying other repositories to ${THEGOPATH}/src/<other app path>

        4. Importing external dependencies as normal (in our case, go get ./... or explicit)

        5. Building and forcing the output name (when launched from CodeBuild it used a different directory name)


        The buildspec.yaml looks something like the following:



        phases:
        install:
        commands:
        - echo GOPATH - $GOPATH
        - export THEGOPATH=`echo $GOPATH | cut -c 5-`
        - echo THEGOPATH = $THEGOPATH
        - mkdir -p ${THEGOPATH}/src/company/app1
        - mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
        - echo Copy source files to go root
        - cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
        - cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
        - ls ${THEGOPATH}/src/
        build:
        commands:
        - echo Build started on `date`
        - echo Getting packages
        - go get ./...
        - echo DOING THE BUILD
        - go build -ldflags "<some flags>" -o "appname"
        - go test ./...
        post_build:
        commands:
        - echo Build completed on `date`
        - ls -al
        - pwd
        artifacts:
        files:
        - appname






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 20 '18 at 17:09









        ajlajl

        42148




        42148
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382185%2faws-codebuild-fails-for-golang-source-with-sub-packages%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            這個網誌中的熱門文章

            Tangent Lines Diagram Along Smooth Curve

            Yusuf al-Mu'taman ibn Hud

            Zucchini