AWS CodeBuild fails for golang source with sub-packages
When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:
The application is organized as follows:
/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...
This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>
) where <some path>
varies according to what triggers the build.
I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN
), but since the CodeBuild environment variable for the GOPATH
directory inserts /go: (/go:/codebuild/output/srcNNN
) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.
My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH
conforms to my assumptions.
Any ideas would be appreciated.
Clarification:
When importing packages in the code, external packages are imported as follows:
import (
"context"
...
}
Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app
). However, AWS CodeBuild doesn't bring in the code this way.
go aws-codebuild
add a comment |
When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:
The application is organized as follows:
/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...
This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>
) where <some path>
varies according to what triggers the build.
I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN
), but since the CodeBuild environment variable for the GOPATH
directory inserts /go: (/go:/codebuild/output/srcNNN
) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.
My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH
conforms to my assumptions.
Any ideas would be appreciated.
Clarification:
When importing packages in the code, external packages are imported as follows:
import (
"context"
...
}
Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app
). However, AWS CodeBuild doesn't bring in the code this way.
go aws-codebuild
Can you please show how do you import your packages in the code?
– Alex Pliutau
Nov 20 '18 at 8:59
Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.
– Seva
Nov 20 '18 at 9:26
Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.
– ajl
Nov 20 '18 at 16:05
add a comment |
When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:
The application is organized as follows:
/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...
This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>
) where <some path>
varies according to what triggers the build.
I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN
), but since the CodeBuild environment variable for the GOPATH
directory inserts /go: (/go:/codebuild/output/srcNNN
) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.
My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH
conforms to my assumptions.
Any ideas would be appreciated.
Clarification:
When importing packages in the code, external packages are imported as follows:
import (
"context"
...
}
Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app
). However, AWS CodeBuild doesn't bring in the code this way.
go aws-codebuild
When I attempt to build my golang project using CodeBuild golang image 1.10, it fails, unable to find the subpackages. Some background:
The application is organized as follows:
/go/src/company/app
/go/src/company/app/sub1
/go/src/company/app/sub2
etc...
This builds fine on my dev machine. However, when pulled by codebuild it is pulled into a different directory (/codebuild/output/srcNNN/src/<some path>
) where <some path>
varies according to what triggers the build.
I originally got it working by copying the code from where it is pulled to the the golang directory (/codebuild/output/srcNNN
), but since the CodeBuild environment variable for the GOPATH
directory inserts /go: (/go:/codebuild/output/srcNNN
) in the front, I used an observed number of ../../... to copy. However, this is ugly and failed as soon as I triggered the build a different way.
My question is whether there is a good way to get this working? My next idea is to apply string manipulation to the observed path and copy there for (hopefully) more reliability. But that will only work as long as the GOPATH
conforms to my assumptions.
Any ideas would be appreciated.
Clarification:
When importing packages in the code, external packages are imported as follows:
import (
"context"
...
}
Subpackages are not explicitly imported, but found when code is deployed as shown above (/go/src/company/app
). However, AWS CodeBuild doesn't bring in the code this way.
go aws-codebuild
go aws-codebuild
edited Nov 20 '18 at 15:59
ajl
asked Nov 19 '18 at 20:30
ajlajl
42148
42148
Can you please show how do you import your packages in the code?
– Alex Pliutau
Nov 20 '18 at 8:59
Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.
– Seva
Nov 20 '18 at 9:26
Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.
– ajl
Nov 20 '18 at 16:05
add a comment |
Can you please show how do you import your packages in the code?
– Alex Pliutau
Nov 20 '18 at 8:59
Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.
– Seva
Nov 20 '18 at 9:26
Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.
– ajl
Nov 20 '18 at 16:05
Can you please show how do you import your packages in the code?
– Alex Pliutau
Nov 20 '18 at 8:59
Can you please show how do you import your packages in the code?
– Alex Pliutau
Nov 20 '18 at 8:59
Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.
– Seva
Nov 20 '18 at 9:26
Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.
– Seva
Nov 20 '18 at 9:26
Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.
– ajl
Nov 20 '18 at 16:05
Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.
– ajl
Nov 20 '18 at 16:05
add a comment |
1 Answer
1
active
oldest
votes
I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.
In my buildspec.yaml I am able to get the build to work by:
- Getting
${THEGOPATH}
from${GOPATH}
by removing the "/go:" from the beginning - Copying all the code to the
${THEGOPATH}/src/<app path>
- Copying other repositories to
${THEGOPATH}/src/<other app path>
- Importing external dependencies as normal (in our case,
go get ./...
or explicit) - Building and forcing the output name (when launched from CodeBuild it used a different directory name)
The buildspec.yaml looks something like the following:
phases:
install:
commands:
- echo GOPATH - $GOPATH
- export THEGOPATH=`echo $GOPATH | cut -c 5-`
- echo THEGOPATH = $THEGOPATH
- mkdir -p ${THEGOPATH}/src/company/app1
- mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
- echo Copy source files to go root
- cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
- ls ${THEGOPATH}/src/
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo DOING THE BUILD
- go build -ldflags "<some flags>" -o "appname"
- go test ./...
post_build:
commands:
- echo Build completed on `date`
- ls -al
- pwd
artifacts:
files:
- appname
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382185%2faws-codebuild-fails-for-golang-source-with-sub-packages%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.
In my buildspec.yaml I am able to get the build to work by:
- Getting
${THEGOPATH}
from${GOPATH}
by removing the "/go:" from the beginning - Copying all the code to the
${THEGOPATH}/src/<app path>
- Copying other repositories to
${THEGOPATH}/src/<other app path>
- Importing external dependencies as normal (in our case,
go get ./...
or explicit) - Building and forcing the output name (when launched from CodeBuild it used a different directory name)
The buildspec.yaml looks something like the following:
phases:
install:
commands:
- echo GOPATH - $GOPATH
- export THEGOPATH=`echo $GOPATH | cut -c 5-`
- echo THEGOPATH = $THEGOPATH
- mkdir -p ${THEGOPATH}/src/company/app1
- mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
- echo Copy source files to go root
- cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
- ls ${THEGOPATH}/src/
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo DOING THE BUILD
- go build -ldflags "<some flags>" -o "appname"
- go test ./...
post_build:
commands:
- echo Build completed on `date`
- ls -al
- pwd
artifacts:
files:
- appname
add a comment |
I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.
In my buildspec.yaml I am able to get the build to work by:
- Getting
${THEGOPATH}
from${GOPATH}
by removing the "/go:" from the beginning - Copying all the code to the
${THEGOPATH}/src/<app path>
- Copying other repositories to
${THEGOPATH}/src/<other app path>
- Importing external dependencies as normal (in our case,
go get ./...
or explicit) - Building and forcing the output name (when launched from CodeBuild it used a different directory name)
The buildspec.yaml looks something like the following:
phases:
install:
commands:
- echo GOPATH - $GOPATH
- export THEGOPATH=`echo $GOPATH | cut -c 5-`
- echo THEGOPATH = $THEGOPATH
- mkdir -p ${THEGOPATH}/src/company/app1
- mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
- echo Copy source files to go root
- cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
- ls ${THEGOPATH}/src/
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo DOING THE BUILD
- go build -ldflags "<some flags>" -o "appname"
- go test ./...
post_build:
commands:
- echo Build completed on `date`
- ls -al
- pwd
artifacts:
files:
- appname
add a comment |
I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.
In my buildspec.yaml I am able to get the build to work by:
- Getting
${THEGOPATH}
from${GOPATH}
by removing the "/go:" from the beginning - Copying all the code to the
${THEGOPATH}/src/<app path>
- Copying other repositories to
${THEGOPATH}/src/<other app path>
- Importing external dependencies as normal (in our case,
go get ./...
or explicit) - Building and forcing the output name (when launched from CodeBuild it used a different directory name)
The buildspec.yaml looks something like the following:
phases:
install:
commands:
- echo GOPATH - $GOPATH
- export THEGOPATH=`echo $GOPATH | cut -c 5-`
- echo THEGOPATH = $THEGOPATH
- mkdir -p ${THEGOPATH}/src/company/app1
- mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
- echo Copy source files to go root
- cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
- ls ${THEGOPATH}/src/
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo DOING THE BUILD
- go build -ldflags "<some flags>" -o "appname"
- go test ./...
post_build:
commands:
- echo Build completed on `date`
- ls -al
- pwd
artifacts:
files:
- appname
I was able to get an answer working. I'll post it here in case it is helpful to others, but it relies on observed behavior in AWS CodeBuild to work, so I don't think it is ideal.
In my buildspec.yaml I am able to get the build to work by:
- Getting
${THEGOPATH}
from${GOPATH}
by removing the "/go:" from the beginning - Copying all the code to the
${THEGOPATH}/src/<app path>
- Copying other repositories to
${THEGOPATH}/src/<other app path>
- Importing external dependencies as normal (in our case,
go get ./...
or explicit) - Building and forcing the output name (when launched from CodeBuild it used a different directory name)
The buildspec.yaml looks something like the following:
phases:
install:
commands:
- echo GOPATH - $GOPATH
- export THEGOPATH=`echo $GOPATH | cut -c 5-`
- echo THEGOPATH = $THEGOPATH
- mkdir -p ${THEGOPATH}/src/company/app1
- mkdir -p ${THEGOPATH}/src/company/other_repository_dependency
- echo Copy source files to go root
- cp -a ${CODEBUILD_SRC_DIR}/. ${THEGOPATH}/src/company/app1/${PACKAGE}
- cp -a ${CODEBUILD_SRC_DIR_other_dep}/. ${THEGOPATH}/src/app/other_repository_dependecy/.
- ls ${THEGOPATH}/src/
build:
commands:
- echo Build started on `date`
- echo Getting packages
- go get ./...
- echo DOING THE BUILD
- go build -ldflags "<some flags>" -o "appname"
- go test ./...
post_build:
commands:
- echo Build completed on `date`
- ls -al
- pwd
artifacts:
files:
- appname
answered Nov 20 '18 at 17:09
ajlajl
42148
42148
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382185%2faws-codebuild-fails-for-golang-source-with-sub-packages%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you please show how do you import your packages in the code?
– Alex Pliutau
Nov 20 '18 at 8:59
Not expert on CodeBuild, but had similar issues with other AWS build system for go. I believe if you just vendor your dependencies into /go/src/company/app/vendor sub-directory, it should solve your issue.
– Seva
Nov 20 '18 at 9:26
Thanks, but if I understand you correctly, our own subpackages are already so vendor'ed, but are not found because the code is actually brought into (to put it in your terms) /go/src/<some path>/company/app/vendor. So, when go looks in /go/src/app/vendor it is not found.
– ajl
Nov 20 '18 at 16:05