How could I override configuration value in Apache Flink?
I'm trying to gather metrics from Apache Flink into Prometheus. Flink documentation says that I need to add following lines to my flink-conf.yaml:
metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: localhost
metrics.reporter.promgateway.port: 9091
metrics.reporter.promgateway.jobName: myJob
I want to mark different jobs with different names inside of Prometheus. How could I override configuration parameter metrics.reporter.promgateway.jobName on per-job basis (each job is running inside of its own Flink cluster session)?
There is a couple of problems:
- I can't override flink-conf.yaml. I've found only FLINK_CONF_DIR parameter to override whole configuration directory. But it doesn't look like a right solution to override configuration directory for every single job.
- I can't override initial configuration of StreamExecutionEnvironment because it is being constructed inside of StreamExecutionEnvironment.getExecutionEnvironment method and can't be modified after environment's initialization.
monitoring apache-flink prometheus
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
add a comment |
I'm trying to gather metrics from Apache Flink into Prometheus. Flink documentation says that I need to add following lines to my flink-conf.yaml:
metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: localhost
metrics.reporter.promgateway.port: 9091
metrics.reporter.promgateway.jobName: myJob
I want to mark different jobs with different names inside of Prometheus. How could I override configuration parameter metrics.reporter.promgateway.jobName on per-job basis (each job is running inside of its own Flink cluster session)?
There is a couple of problems:
- I can't override flink-conf.yaml. I've found only FLINK_CONF_DIR parameter to override whole configuration directory. But it doesn't look like a right solution to override configuration directory for every single job.
- I can't override initial configuration of StreamExecutionEnvironment because it is being constructed inside of StreamExecutionEnvironment.getExecutionEnvironment method and can't be modified after environment's initialization.
monitoring apache-flink prometheus
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
add a comment |
I'm trying to gather metrics from Apache Flink into Prometheus. Flink documentation says that I need to add following lines to my flink-conf.yaml:
metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: localhost
metrics.reporter.promgateway.port: 9091
metrics.reporter.promgateway.jobName: myJob
I want to mark different jobs with different names inside of Prometheus. How could I override configuration parameter metrics.reporter.promgateway.jobName on per-job basis (each job is running inside of its own Flink cluster session)?
There is a couple of problems:
- I can't override flink-conf.yaml. I've found only FLINK_CONF_DIR parameter to override whole configuration directory. But it doesn't look like a right solution to override configuration directory for every single job.
- I can't override initial configuration of StreamExecutionEnvironment because it is being constructed inside of StreamExecutionEnvironment.getExecutionEnvironment method and can't be modified after environment's initialization.
monitoring apache-flink prometheus
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
I'm trying to gather metrics from Apache Flink into Prometheus. Flink documentation says that I need to add following lines to my flink-conf.yaml:
metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: localhost
metrics.reporter.promgateway.port: 9091
metrics.reporter.promgateway.jobName: myJob
I want to mark different jobs with different names inside of Prometheus. How could I override configuration parameter metrics.reporter.promgateway.jobName on per-job basis (each job is running inside of its own Flink cluster session)?
There is a couple of problems:
- I can't override flink-conf.yaml. I've found only FLINK_CONF_DIR parameter to override whole configuration directory. But it doesn't look like a right solution to override configuration directory for every single job.
- I can't override initial configuration of StreamExecutionEnvironment because it is being constructed inside of StreamExecutionEnvironment.getExecutionEnvironment method and can't be modified after environment's initialization.
monitoring apache-flink prometheus
monitoring apache-flink prometheus
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
asked Nov 19 '18 at 14:32
Gorini4Gorini4
153138
153138
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You can modify the effective configuration by specifying a dynamic property when starting a Flink job cluster. Assuming that you are deploying to Yarn the command would look like:
bin/flink run -m yarn-cluster -yD metrics.reporter.promgateway.jobName=myCustomJob <USER_CODE_JAR>
The dynamic properties are sent to the Yarn cluster and overwrite existing configuration key value pairs.
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53376812%2fhow-could-i-override-configuration-value-in-apache-flink%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can modify the effective configuration by specifying a dynamic property when starting a Flink job cluster. Assuming that you are deploying to Yarn the command would look like:
bin/flink run -m yarn-cluster -yD metrics.reporter.promgateway.jobName=myCustomJob <USER_CODE_JAR>
The dynamic properties are sent to the Yarn cluster and overwrite existing configuration key value pairs.
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
add a comment |
You can modify the effective configuration by specifying a dynamic property when starting a Flink job cluster. Assuming that you are deploying to Yarn the command would look like:
bin/flink run -m yarn-cluster -yD metrics.reporter.promgateway.jobName=myCustomJob <USER_CODE_JAR>
The dynamic properties are sent to the Yarn cluster and overwrite existing configuration key value pairs.
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
add a comment |
You can modify the effective configuration by specifying a dynamic property when starting a Flink job cluster. Assuming that you are deploying to Yarn the command would look like:
bin/flink run -m yarn-cluster -yD metrics.reporter.promgateway.jobName=myCustomJob <USER_CODE_JAR>
The dynamic properties are sent to the Yarn cluster and overwrite existing configuration key value pairs.
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
You can modify the effective configuration by specifying a dynamic property when starting a Flink job cluster. Assuming that you are deploying to Yarn the command would look like:
bin/flink run -m yarn-cluster -yD metrics.reporter.promgateway.jobName=myCustomJob <USER_CODE_JAR>
The dynamic properties are sent to the Yarn cluster and overwrite existing configuration key value pairs.
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
answered Nov 20 '18 at 15:40
Till RohrmannTill Rohrmann
9,31111237
9,31111237
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53376812%2fhow-could-i-override-configuration-value-in-apache-flink%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
