Why using Selenium to scroll down and click the button sometimes fails to load all the page
I am trying to load the page with Selenium and parse it with Beautiful Soup. I have tried different ways to simulate the click on the load more button, only the code here works. [How to scroll down in Python Selenium step by step
read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']')
for read_more in read_mores:
driver.execute_script("arguments[0].scrollIntoView();", read_more)
driver.execute_script("$(arguments[0]).click();", read_more)
soup = BeautifulSoup(driver.page_source, 'html.parser')
However, it fails to load the whole page sometimes. I know probably "clickandwait" will work but have no idea where to put in the code. I would also love to know if there is other way to deal with it. Any help would be greatly appreciated!
python selenium web-scraping beautifulsoup
asked Nov 21 '18 at 7:14
ChloeChloe
32
add a comment |
I am trying to load the page with Selenium and parse it with Beautiful Soup. I have tried different ways to simulate the click on the load more button, only the code here works. [How to scroll down in Python Selenium step by step
read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']')
for read_more in read_mores:
driver.execute_script("arguments[0].scrollIntoView();", read_more)
driver.execute_script("$(arguments[0]).click();", read_more)
soup = BeautifulSoup(driver.page_source, 'html.parser')
However, it fails to load the whole page sometimes. I know probably "clickandwait" will work but have no idea where to put in the code. I would also love to know if there is other way to deal with it. Any help would be greatly appreciated!
python selenium web-scraping beautifulsoup
asked Nov 21 '18 at 7:14
ChloeChloe
32
Is there an URL you can share?
– QHarr
Nov 21 '18 at 7:19
add a comment |
I am trying to load the page with Selenium and parse it with Beautiful Soup. I have tried different ways to simulate the click on the load more button, only the code here works. [How to scroll down in Python Selenium step by step
read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']')
for read_more in read_mores:
driver.execute_script("arguments[0].scrollIntoView();", read_more)
driver.execute_script("$(arguments[0]).click();", read_more)
soup = BeautifulSoup(driver.page_source, 'html.parser')
However, it fails to load the whole page sometimes. I know probably "clickandwait" will work but have no idea where to put in the code. I would also love to know if there is other way to deal with it. Any help would be greatly appreciated!
python selenium web-scraping beautifulsoup
asked Nov 21 '18 at 7:14
ChloeChloe
32
I am trying to load the page with Selenium and parse it with Beautiful Soup. I have tried different ways to simulate the click on the load more button, only the code here works. [How to scroll down in Python Selenium step by step
read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']')
for read_more in read_mores:
driver.execute_script("arguments[0].scrollIntoView();", read_more)
driver.execute_script("$(arguments[0]).click();", read_more)
soup = BeautifulSoup(driver.page_source, 'html.parser')
However, it fails to load the whole page sometimes. I know probably "clickandwait" will work but have no idea where to put in the code. I would also love to know if there is other way to deal with it. Any help would be greatly appreciated!
python selenium web-scraping beautifulsoup
python selenium web-scraping beautifulsoup
asked Nov 21 '18 at 7:14
ChloeChloe
32
asked Nov 21 '18 at 7:14
ChloeChloe
32
asked Nov 21 '18 at 7:14
ChloeChloe
32
asked Nov 21 '18 at 7:14
ChloeChloe
32
asked Nov 21 '18 at 7:14
ChloeChloe
32
32
Is there an URL you can share?
– QHarr
Nov 21 '18 at 7:19
add a comment |
Is there an URL you can share?
– QHarr
Nov 21 '18 at 7:19
Is there an URL you can share?
– QHarr
Nov 21 '18 at 7:19
Is there an URL you can share?
– QHarr
Nov 21 '18 at 7:19
add a comment |
1 Answer
1
active
oldest
votes
I recommend to do:
driver.execute_script("document.querySelector('[data-teach-id="" + tid + ""]').click()")
Rather than select in selenium and pass the element to the browser context, do it all in the browser context. Less possiblity for things to go horribly wrong that way.
Just to get fancy...
Let's abstract the click to a function (I think I'll do this from now on):
def click(css):
global driver
driver.execute_script("document.querySelector('" + css + "').click()")
Now we can do:
click('a[data-teach-id="' + tid + '"]')
Ah, much less painful.
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visiblewhileyou can click it then use that for your conditions.
– Kamikaze_goldfish
Nov 21 '18 at 21:40
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53406956%2fwhy-using-selenium-to-scroll-down-and-click-the-button-sometimes-fails-to-load-a%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I recommend to do:
driver.execute_script("document.querySelector('[data-teach-id="" + tid + ""]').click()")
Rather than select in selenium and pass the element to the browser context, do it all in the browser context. Less possiblity for things to go horribly wrong that way.
Just to get fancy...
Let's abstract the click to a function (I think I'll do this from now on):
def click(css):
global driver
driver.execute_script("document.querySelector('" + css + "').click()")
Now we can do:
click('a[data-teach-id="' + tid + '"]')
Ah, much less painful.
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visiblewhileyou can click it then use that for your conditions.
– Kamikaze_goldfish
Nov 21 '18 at 21:40
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
add a comment |
I recommend to do:
driver.execute_script("document.querySelector('[data-teach-id="" + tid + ""]').click()")
Rather than select in selenium and pass the element to the browser context, do it all in the browser context. Less possiblity for things to go horribly wrong that way.
Just to get fancy...
Let's abstract the click to a function (I think I'll do this from now on):
def click(css):
global driver
driver.execute_script("document.querySelector('" + css + "').click()")
Now we can do:
click('a[data-teach-id="' + tid + '"]')
Ah, much less painful.
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visiblewhileyou can click it then use that for your conditions.
– Kamikaze_goldfish
Nov 21 '18 at 21:40
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
add a comment |
I recommend to do:
driver.execute_script("document.querySelector('[data-teach-id="" + tid + ""]').click()")
Rather than select in selenium and pass the element to the browser context, do it all in the browser context. Less possiblity for things to go horribly wrong that way.
Just to get fancy...
Let's abstract the click to a function (I think I'll do this from now on):
def click(css):
global driver
driver.execute_script("document.querySelector('" + css + "').click()")
Now we can do:
click('a[data-teach-id="' + tid + '"]')
Ah, much less painful.
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
I recommend to do:
driver.execute_script("document.querySelector('[data-teach-id="" + tid + ""]').click()")
Rather than select in selenium and pass the element to the browser context, do it all in the browser context. Less possiblity for things to go horribly wrong that way.
Just to get fancy...
Let's abstract the click to a function (I think I'll do this from now on):
def click(css):
global driver
driver.execute_script("document.querySelector('" + css + "').click()")
Now we can do:
click('a[data-teach-id="' + tid + '"]')
Ah, much less painful.
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
answered Nov 21 '18 at 9:03
pguardiariopguardiario
36.4k980116
36.4k980116
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visiblewhileyou can click it then use that for your conditions.
– Kamikaze_goldfish
Nov 21 '18 at 21:40
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
add a comment |
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visiblewhileyou can click it then use that for your conditions.
– Kamikaze_goldfish
Nov 21 '18 at 21:40
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
Thanks a lot! But it fails to load the whole page...
– Chloe
Nov 21 '18 at 16:21
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visible
while you can click it then use that for your conditions.– Kamikaze_goldfish
Nov 21 '18 at 21:40
@Chloe Is it just loading some of the content thus pushing the page lower and you would have to click it again? If so, then run that in a loop. If the element is visible
while you can click it then use that for your conditions.– Kamikaze_goldfish
Nov 21 '18 at 21:40
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
Sorry, without an url I can't see what's going on there.
– pguardiario
Nov 21 '18 at 23:59
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53406956%2fwhy-using-selenium-to-scroll-down-and-click-the-button-sometimes-fails-to-load-a%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is there an URL you can share?
– QHarr
Nov 21 '18 at 7:19