Bigquery Sql vs Spark Sql - Memory Management
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have been working on Bigquery standard sql a lot lately and the exact same data is being dealt with in spark at the same time .
I have noticed something strange in terms of query execution in bigquery though .
For Eg: If i fire a standard sql query with window analytic functions such as row_number() or rank() on a bigquery table that holds close to 500 GB data and around 60 million rows , it gives me a "Out of memory" error. And i read since window functions require sorting (partition by column name order by column name) , the bigquery engine forces the entire chunk of data into a single node for sorting purposes , which is understandable and hence the error .
But when i read the data into spark and fire the exact same query in spark sql after registering the dataset as a temptable , i get the output in a matter of minutes . I also notice the partitions and number of executors being active in the spark UI which basically means the data is being partitioned properly in spark .
Can anybody shed some light on why and how the 2 differ and in what way .
apache-spark-sql google-bigquery
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
add a comment |
I have been working on Bigquery standard sql a lot lately and the exact same data is being dealt with in spark at the same time .
I have noticed something strange in terms of query execution in bigquery though .
For Eg: If i fire a standard sql query with window analytic functions such as row_number() or rank() on a bigquery table that holds close to 500 GB data and around 60 million rows , it gives me a "Out of memory" error. And i read since window functions require sorting (partition by column name order by column name) , the bigquery engine forces the entire chunk of data into a single node for sorting purposes , which is understandable and hence the error .
But when i read the data into spark and fire the exact same query in spark sql after registering the dataset as a temptable , i get the output in a matter of minutes . I also notice the partitions and number of executors being active in the spark UI which basically means the data is being partitioned properly in spark .
Can anybody shed some light on why and how the 2 differ and in what way .
apache-spark-sql google-bigquery
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
add a comment |
I have been working on Bigquery standard sql a lot lately and the exact same data is being dealt with in spark at the same time .
I have noticed something strange in terms of query execution in bigquery though .
For Eg: If i fire a standard sql query with window analytic functions such as row_number() or rank() on a bigquery table that holds close to 500 GB data and around 60 million rows , it gives me a "Out of memory" error. And i read since window functions require sorting (partition by column name order by column name) , the bigquery engine forces the entire chunk of data into a single node for sorting purposes , which is understandable and hence the error .
But when i read the data into spark and fire the exact same query in spark sql after registering the dataset as a temptable , i get the output in a matter of minutes . I also notice the partitions and number of executors being active in the spark UI which basically means the data is being partitioned properly in spark .
Can anybody shed some light on why and how the 2 differ and in what way .
apache-spark-sql google-bigquery
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
I have been working on Bigquery standard sql a lot lately and the exact same data is being dealt with in spark at the same time .
I have noticed something strange in terms of query execution in bigquery though .
For Eg: If i fire a standard sql query with window analytic functions such as row_number() or rank() on a bigquery table that holds close to 500 GB data and around 60 million rows , it gives me a "Out of memory" error. And i read since window functions require sorting (partition by column name order by column name) , the bigquery engine forces the entire chunk of data into a single node for sorting purposes , which is understandable and hence the error .
But when i read the data into spark and fire the exact same query in spark sql after registering the dataset as a temptable , i get the output in a matter of minutes . I also notice the partitions and number of executors being active in the spark UI which basically means the data is being partitioned properly in spark .
Can anybody shed some light on why and how the 2 differ and in what way .
apache-spark-sql google-bigquery
apache-spark-sql google-bigquery
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
asked Nov 24 '18 at 12:01
user1411837user1411837
57851230
57851230
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53457934%2fbigquery-sql-vs-spark-sql-memory-management%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53457934%2fbigquery-sql-vs-spark-sql-memory-management%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
