Python and Soup HTTPS Web Scrape - Open.SSL.Error
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
Still figuring out this web scraping thing. Coming across an error when trying to scrape an HTTPS site. Something to do with SSL certificates and the site side rejecting my connection? This is my code:
from bs4 import BeautifulSoup
import requests
import csv
with open('UrlsList.csv', newline='') as f_urls, open('Output.csv', 'w', newline='') as f_output:
csv_urls = csv.reader(f_urls)
csv_output = csv.writer(f_output)
for line in csv_urls:
page = requests.get(line[0], verify='.Cert.cer').text
soup = BeautifulSoup(page, 'html.parser')
results = soup.findAll('td', {'class' :' alpha'})
for r in range(len(results)):
csv_output.writerow([results[r].text])
...Which gives me a big screen of issues with the following error at the bottom:
raise exception_type(errors)
OpenSSL.SSL.Error:
I have tried just putting the verify=False as well, and that gives me the following error:
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
I've tried to research the answer on my own, but I can't seem to make sense of any solution so far. I've recently just updated my PyOpenSSL to version 18 as well. Just seems the site I'm trying to scrape doesn't accept my connection, but the URL is real and I can view the site no problem from Chrome?
Thanks a lot!
python-3.x beautifulsoup python-requests ssl-certificate
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
|
show 9 more comments
Still figuring out this web scraping thing. Coming across an error when trying to scrape an HTTPS site. Something to do with SSL certificates and the site side rejecting my connection? This is my code:
from bs4 import BeautifulSoup
import requests
import csv
with open('UrlsList.csv', newline='') as f_urls, open('Output.csv', 'w', newline='') as f_output:
csv_urls = csv.reader(f_urls)
csv_output = csv.writer(f_output)
for line in csv_urls:
page = requests.get(line[0], verify='.Cert.cer').text
soup = BeautifulSoup(page, 'html.parser')
results = soup.findAll('td', {'class' :' alpha'})
for r in range(len(results)):
csv_output.writerow([results[r].text])
...Which gives me a big screen of issues with the following error at the bottom:
raise exception_type(errors)
OpenSSL.SSL.Error:
I have tried just putting the verify=False as well, and that gives me the following error:
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
I've tried to research the answer on my own, but I can't seem to make sense of any solution so far. I've recently just updated my PyOpenSSL to version 18 as well. Just seems the site I'm trying to scrape doesn't accept my connection, but the URL is real and I can view the site no problem from Chrome?
Thanks a lot!
python-3.x beautifulsoup python-requests ssl-certificate
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
Try this solution: stackoverflow.com/questions/15445981/…. Basically set theverifyparameter toFalse.
– Andrej Kesely
Aug 11 '18 at 6:40
Are you on Mac?
– jlaur
Aug 11 '18 at 11:39
If so it's a well known Mac issue. Remove the verify-argument in requests and do a pip install certifi. You can read about this Mac-issue here: cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra
– jlaur
Aug 11 '18 at 11:47
Thanks @AndrejKesely but like I said above, I've tried setting verify=False and I just get another error message?
– Matt Wilson
Aug 11 '18 at 18:03
1
Just tried your url and this solution worked for me: stackoverflow.com/questions/43165341/…
– Paula Thomas
Aug 12 '18 at 9:29
|
show 9 more comments
Still figuring out this web scraping thing. Coming across an error when trying to scrape an HTTPS site. Something to do with SSL certificates and the site side rejecting my connection? This is my code:
from bs4 import BeautifulSoup
import requests
import csv
with open('UrlsList.csv', newline='') as f_urls, open('Output.csv', 'w', newline='') as f_output:
csv_urls = csv.reader(f_urls)
csv_output = csv.writer(f_output)
for line in csv_urls:
page = requests.get(line[0], verify='.Cert.cer').text
soup = BeautifulSoup(page, 'html.parser')
results = soup.findAll('td', {'class' :' alpha'})
for r in range(len(results)):
csv_output.writerow([results[r].text])
...Which gives me a big screen of issues with the following error at the bottom:
raise exception_type(errors)
OpenSSL.SSL.Error:
I have tried just putting the verify=False as well, and that gives me the following error:
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
I've tried to research the answer on my own, but I can't seem to make sense of any solution so far. I've recently just updated my PyOpenSSL to version 18 as well. Just seems the site I'm trying to scrape doesn't accept my connection, but the URL is real and I can view the site no problem from Chrome?
Thanks a lot!
python-3.x beautifulsoup python-requests ssl-certificate
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
Still figuring out this web scraping thing. Coming across an error when trying to scrape an HTTPS site. Something to do with SSL certificates and the site side rejecting my connection? This is my code:
from bs4 import BeautifulSoup
import requests
import csv
with open('UrlsList.csv', newline='') as f_urls, open('Output.csv', 'w', newline='') as f_output:
csv_urls = csv.reader(f_urls)
csv_output = csv.writer(f_output)
for line in csv_urls:
page = requests.get(line[0], verify='.Cert.cer').text
soup = BeautifulSoup(page, 'html.parser')
results = soup.findAll('td', {'class' :' alpha'})
for r in range(len(results)):
csv_output.writerow([results[r].text])
...Which gives me a big screen of issues with the following error at the bottom:
raise exception_type(errors)
OpenSSL.SSL.Error:
I have tried just putting the verify=False as well, and that gives me the following error:
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
I've tried to research the answer on my own, but I can't seem to make sense of any solution so far. I've recently just updated my PyOpenSSL to version 18 as well. Just seems the site I'm trying to scrape doesn't accept my connection, but the URL is real and I can view the site no problem from Chrome?
Thanks a lot!
python-3.x beautifulsoup python-requests ssl-certificate
python-3.x beautifulsoup python-requests ssl-certificate
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
asked Aug 11 '18 at 5:22
Matt WilsonMatt Wilson
707
707
Try this solution: stackoverflow.com/questions/15445981/…. Basically set theverifyparameter toFalse.
– Andrej Kesely
Aug 11 '18 at 6:40
Are you on Mac?
– jlaur
Aug 11 '18 at 11:39
If so it's a well known Mac issue. Remove the verify-argument in requests and do a pip install certifi. You can read about this Mac-issue here: cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra
– jlaur
Aug 11 '18 at 11:47
Thanks @AndrejKesely but like I said above, I've tried setting verify=False and I just get another error message?
– Matt Wilson
Aug 11 '18 at 18:03
1
Just tried your url and this solution worked for me: stackoverflow.com/questions/43165341/…
– Paula Thomas
Aug 12 '18 at 9:29
|
show 9 more comments
Try this solution: stackoverflow.com/questions/15445981/…. Basically set theverifyparameter toFalse.
– Andrej Kesely
Aug 11 '18 at 6:40
Are you on Mac?
– jlaur
Aug 11 '18 at 11:39
If so it's a well known Mac issue. Remove the verify-argument in requests and do a pip install certifi. You can read about this Mac-issue here: cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra
– jlaur
Aug 11 '18 at 11:47
Thanks @AndrejKesely but like I said above, I've tried setting verify=False and I just get another error message?
– Matt Wilson
Aug 11 '18 at 18:03
1
Just tried your url and this solution worked for me: stackoverflow.com/questions/43165341/…
– Paula Thomas
Aug 12 '18 at 9:29
Try this solution: stackoverflow.com/questions/15445981/…. Basically set the
verify parameter to False.– Andrej Kesely
Aug 11 '18 at 6:40
Try this solution: stackoverflow.com/questions/15445981/…. Basically set the
verify parameter to False.– Andrej Kesely
Aug 11 '18 at 6:40
Are you on Mac?
– jlaur
Aug 11 '18 at 11:39
Are you on Mac?
– jlaur
Aug 11 '18 at 11:39
If so it's a well known Mac issue. Remove the verify-argument in requests and do a pip install certifi. You can read about this Mac-issue here: cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra
– jlaur
Aug 11 '18 at 11:47
If so it's a well known Mac issue. Remove the verify-argument in requests and do a pip install certifi. You can read about this Mac-issue here: cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra
– jlaur
Aug 11 '18 at 11:47
Thanks @AndrejKesely but like I said above, I've tried setting verify=False and I just get another error message?
– Matt Wilson
Aug 11 '18 at 18:03
Thanks @AndrejKesely but like I said above, I've tried setting verify=False and I just get another error message?
– Matt Wilson
Aug 11 '18 at 18:03
1
1
Just tried your url and this solution worked for me: stackoverflow.com/questions/43165341/…
– Paula Thomas
Aug 12 '18 at 9:29
Just tried your url and this solution worked for me: stackoverflow.com/questions/43165341/…
– Paula Thomas
Aug 12 '18 at 9:29
|
show 9 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51796724%2fpython-and-soup-https-web-scrape-open-ssl-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51796724%2fpython-and-soup-https-web-scrape-open-ssl-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Try this solution: stackoverflow.com/questions/15445981/…. Basically set the
verifyparameter toFalse.– Andrej Kesely
Aug 11 '18 at 6:40
Are you on Mac?
– jlaur
Aug 11 '18 at 11:39
If so it's a well known Mac issue. Remove the verify-argument in requests and do a pip install certifi. You can read about this Mac-issue here: cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra
– jlaur
Aug 11 '18 at 11:47
Thanks @AndrejKesely but like I said above, I've tried setting verify=False and I just get another error message?
– Matt Wilson
Aug 11 '18 at 18:03
1
Just tried your url and this solution worked for me: stackoverflow.com/questions/43165341/…
– Paula Thomas
Aug 12 '18 at 9:29