How to get past the login page with Wget?












227















I am trying to use Wget to download a page, but I cannot get past the login screen.



How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?










share|improve this question




















  • 3





    For curl: stackoverflow.com/questions/12399087/…

    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    May 7 '15 at 9:04
















227















I am trying to use Wget to download a page, but I cannot get past the login screen.



How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?










share|improve this question




















  • 3





    For curl: stackoverflow.com/questions/12399087/…

    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    May 7 '15 at 9:04














227












227








227


136






I am trying to use Wget to download a page, but I cannot get past the login screen.



How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?










share|improve this question
















I am trying to use Wget to download a page, but I cannot get past the login screen.



How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?







wget






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited May 28 '15 at 19:23









Peter Mortensen

13.6k1984111




13.6k1984111










asked Aug 24 '09 at 19:59









Señor Reginold FrancisSeñor Reginold Francis

5,882155071




5,882155071








  • 3





    For curl: stackoverflow.com/questions/12399087/…

    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    May 7 '15 at 9:04














  • 3





    For curl: stackoverflow.com/questions/12399087/…

    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    May 7 '15 at 9:04








3




3





For curl: stackoverflow.com/questions/12399087/…

– Ciro Santilli 新疆改造中心 六四事件 法轮功
May 7 '15 at 9:04





For curl: stackoverflow.com/questions/12399087/…

– Ciro Santilli 新疆改造中心 六四事件 法轮功
May 7 '15 at 9:04












9 Answers
9






active

oldest

votes


















306














Based on the manual page:



# Log in to the server.  This only needs to be done once.
wget --save-cookies cookies.txt
--keep-session-cookies
--post-data 'user=foo&password=bar'
--delete-after
http://server.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt
http://server.com/interesting/article.php


Make sure the --post-data parameter is properly percent-encoded (especially ampersands!) or the request will probably fail. Also make sure that user and password are the correct keys; you can find out the correct keys by sleuthing the HTML of the login page (look into your browser’s “inspect element” feature and find the name attribute on the username and password fields).






share|improve this answer





















  • 10





    add --keep-session-cookies to the first command, or the second?

    – Felipe Alvarez
    Nov 9 '11 at 2:56






  • 4





    You don't need -p (--page-requisites) for this.

    – ændrük
    Jan 6 '12 at 17:24








  • 12





    It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

    – Jim Hunziker
    Jan 2 '13 at 15:41






  • 1





    I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

    – Mowgli
    Mar 28 '13 at 1:23








  • 5





    --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

    – wadim
    May 11 '14 at 17:09



















48














I directly gave cookies of an existing connection to wget with --no-cookies and the Cookie HTTP request header. In my case it was a Moodle university login where logging in looks more complex (using multiple requests with a login ticket). I added --post-data because it was a POST request.
For example, get all Moodle users list:



wget --no-cookies --header "Cookie: <name>=<value>" --post-data 'tab=search&name=+&personsubmit=Rechercher&keywords=&keywordsoption=allmine' https://moodle.unistra.fr/message/index.php






share|improve this answer





















  • 4





    Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

    – Tuxdude
    Jul 27 '16 at 18:29






  • 2





    You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

    – Phil C
    May 25 '18 at 13:28



















33














If you need it for one time use, you can log in via browser and copy needed headers afterwards:



screenshot
Use "Copy as cURL" in the Network tab of Developer Tools (reload the page after opening) and replace curl's header flag -H and --data with wget's --header and --post-data.






share|improve this answer


























  • Great solution!

    – menteith
    Mar 27 '18 at 18:59



















24














I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.



When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.






share|improve this answer





















  • 1





    unfortunately not applicable in automated scripting

    – Znik
    Aug 21 '15 at 13:49



















9














You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2' works just fine. Note: you can also pass a file name to wget with the POST data in the file.






share|improve this answer

































    9














    I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:



    wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'
    (this is one line, though it likely wraps on your browser)



    If you want the output saved to a file, change -O - to -O /some/file/name






    share|improve this answer

































      8














      If they're using basic authentication:



      wget http://username:password@www.domain.com/page.html


      If they're using POSTed form data, you'll need to use something like cURL instead.






      share|improve this answer


























      • I dont have access to change anything on the server, it is read only

        – Señor Reginold Francis
        Aug 24 '09 at 20:13






      • 7





        So? None of this requires you to change anything on the server.

        – ceejayoz
        Aug 24 '09 at 20:15



















      3














      A solution which uses lynx and wget.



      Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work



      When you want to use wget to download some file from a site which requires login, you just need a cookie file.
      In order to generate the cookie file, I choose lynx.
      lynx is a text web browser.
      First you need a configure file for lynx to save cookie.
      Create a file lynx.cfg. Write these configuration into the file.



      SET_COOKIES:TRUE
      ACCEPT_ALL_COOKIES:TRUE
      PERSISTENT_COOKIES:TRUE
      COOKIE_FILE:cookie.file


      Then start lynx with this command:



      lynx -cfg=lynx.cfg http://the.site.com/login


      After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout.
      The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.



      Then wget can download file from the site with this command.



      wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz





      share|improve this answer


























      • what about if the login requires javascript? lynx does not seems to support javascript.

        – Tiberiu
        Dec 13 '18 at 17:41



















      1














      Example to download with wget on server a big file link that can be obtained in your browser.



      In example using Google Chrome.



      Login where you need, and press download. Go to download and copy your link.



      enter image description here



      Then open DevTools on a page where you where login, go to Console and get your cookies, by entering document.cookie



      enter image description here



      Now, go to server and download your file: wget --header "Cookie: <YOUR_COOKIE_OUTPUT_FROM_CONSOLE>" <YOUR_DOWNLOAD_LINK>



      enter image description here






      share|improve this answer























        Your Answer






        StackExchange.ifUsing("editor", function () {
        StackExchange.using("externalEditor", function () {
        StackExchange.using("snippets", function () {
        StackExchange.snippets.init();
        });
        });
        }, "code-snippets");

        StackExchange.ready(function() {
        var channelOptions = {
        tags: "".split(" "),
        id: "1"
        };
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function() {
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled) {
        StackExchange.using("snippets", function() {
        createEditor();
        });
        }
        else {
        createEditor();
        }
        });

        function createEditor() {
        StackExchange.prepareEditor({
        heartbeatType: 'answer',
        autoActivateHeartbeat: false,
        convertImagesToLinks: true,
        noModals: true,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: 10,
        bindNavPrevention: true,
        postfix: "",
        imageUploader: {
        brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
        contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
        allowUrls: true
        },
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        });


        }
        });














        draft saved

        draft discarded


















        StackExchange.ready(
        function () {
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f1324421%2fhow-to-get-past-the-login-page-with-wget%23new-answer', 'question_page');
        }
        );

        Post as a guest















        Required, but never shown

























        9 Answers
        9






        active

        oldest

        votes








        9 Answers
        9






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes









        306














        Based on the manual page:



        # Log in to the server.  This only needs to be done once.
        wget --save-cookies cookies.txt
        --keep-session-cookies
        --post-data 'user=foo&password=bar'
        --delete-after
        http://server.com/auth.php

        # Now grab the page or pages we care about.
        wget --load-cookies cookies.txt
        http://server.com/interesting/article.php


        Make sure the --post-data parameter is properly percent-encoded (especially ampersands!) or the request will probably fail. Also make sure that user and password are the correct keys; you can find out the correct keys by sleuthing the HTML of the login page (look into your browser’s “inspect element” feature and find the name attribute on the username and password fields).






        share|improve this answer





















        • 10





          add --keep-session-cookies to the first command, or the second?

          – Felipe Alvarez
          Nov 9 '11 at 2:56






        • 4





          You don't need -p (--page-requisites) for this.

          – ændrük
          Jan 6 '12 at 17:24








        • 12





          It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

          – Jim Hunziker
          Jan 2 '13 at 15:41






        • 1





          I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

          – Mowgli
          Mar 28 '13 at 1:23








        • 5





          --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

          – wadim
          May 11 '14 at 17:09
















        306














        Based on the manual page:



        # Log in to the server.  This only needs to be done once.
        wget --save-cookies cookies.txt
        --keep-session-cookies
        --post-data 'user=foo&password=bar'
        --delete-after
        http://server.com/auth.php

        # Now grab the page or pages we care about.
        wget --load-cookies cookies.txt
        http://server.com/interesting/article.php


        Make sure the --post-data parameter is properly percent-encoded (especially ampersands!) or the request will probably fail. Also make sure that user and password are the correct keys; you can find out the correct keys by sleuthing the HTML of the login page (look into your browser’s “inspect element” feature and find the name attribute on the username and password fields).






        share|improve this answer





















        • 10





          add --keep-session-cookies to the first command, or the second?

          – Felipe Alvarez
          Nov 9 '11 at 2:56






        • 4





          You don't need -p (--page-requisites) for this.

          – ændrük
          Jan 6 '12 at 17:24








        • 12





          It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

          – Jim Hunziker
          Jan 2 '13 at 15:41






        • 1





          I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

          – Mowgli
          Mar 28 '13 at 1:23








        • 5





          --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

          – wadim
          May 11 '14 at 17:09














        306












        306








        306







        Based on the manual page:



        # Log in to the server.  This only needs to be done once.
        wget --save-cookies cookies.txt
        --keep-session-cookies
        --post-data 'user=foo&password=bar'
        --delete-after
        http://server.com/auth.php

        # Now grab the page or pages we care about.
        wget --load-cookies cookies.txt
        http://server.com/interesting/article.php


        Make sure the --post-data parameter is properly percent-encoded (especially ampersands!) or the request will probably fail. Also make sure that user and password are the correct keys; you can find out the correct keys by sleuthing the HTML of the login page (look into your browser’s “inspect element” feature and find the name attribute on the username and password fields).






        share|improve this answer















        Based on the manual page:



        # Log in to the server.  This only needs to be done once.
        wget --save-cookies cookies.txt
        --keep-session-cookies
        --post-data 'user=foo&password=bar'
        --delete-after
        http://server.com/auth.php

        # Now grab the page or pages we care about.
        wget --load-cookies cookies.txt
        http://server.com/interesting/article.php


        Make sure the --post-data parameter is properly percent-encoded (especially ampersands!) or the request will probably fail. Also make sure that user and password are the correct keys; you can find out the correct keys by sleuthing the HTML of the login page (look into your browser’s “inspect element” feature and find the name attribute on the username and password fields).







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Sep 13 '17 at 12:14









        9999years

        651611




        651611










        answered Sep 16 '09 at 10:25









        jarnoanjarnoan

        3,39811215




        3,39811215








        • 10





          add --keep-session-cookies to the first command, or the second?

          – Felipe Alvarez
          Nov 9 '11 at 2:56






        • 4





          You don't need -p (--page-requisites) for this.

          – ændrük
          Jan 6 '12 at 17:24








        • 12





          It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

          – Jim Hunziker
          Jan 2 '13 at 15:41






        • 1





          I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

          – Mowgli
          Mar 28 '13 at 1:23








        • 5





          --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

          – wadim
          May 11 '14 at 17:09














        • 10





          add --keep-session-cookies to the first command, or the second?

          – Felipe Alvarez
          Nov 9 '11 at 2:56






        • 4





          You don't need -p (--page-requisites) for this.

          – ændrük
          Jan 6 '12 at 17:24








        • 12





          It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

          – Jim Hunziker
          Jan 2 '13 at 15:41






        • 1





          I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

          – Mowgli
          Mar 28 '13 at 1:23








        • 5





          --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

          – wadim
          May 11 '14 at 17:09








        10




        10





        add --keep-session-cookies to the first command, or the second?

        – Felipe Alvarez
        Nov 9 '11 at 2:56





        add --keep-session-cookies to the first command, or the second?

        – Felipe Alvarez
        Nov 9 '11 at 2:56




        4




        4





        You don't need -p (--page-requisites) for this.

        – ændrük
        Jan 6 '12 at 17:24







        You don't need -p (--page-requisites) for this.

        – ændrük
        Jan 6 '12 at 17:24






        12




        12





        It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

        – Jim Hunziker
        Jan 2 '13 at 15:41





        It's also worth adding --delete-after to the first retrieval so you don't end up saving the result page from logging in.

        – Jim Hunziker
        Jan 2 '13 at 15:41




        1




        1





        I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

        – Mowgli
        Mar 28 '13 at 1:23







        I am getting error WGET64: missing URL I put whole wget command in one line and removed ``

        – Mowgli
        Mar 28 '13 at 1:23






        5




        5





        --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

        – wadim
        May 11 '14 at 17:09





        --keep-session-cookies is needed for the first command only. It tells the first command to include session cookies when saving cookies to the file. The second command simply reads all cookies from the provided file.

        – wadim
        May 11 '14 at 17:09













        48














        I directly gave cookies of an existing connection to wget with --no-cookies and the Cookie HTTP request header. In my case it was a Moodle university login where logging in looks more complex (using multiple requests with a login ticket). I added --post-data because it was a POST request.
        For example, get all Moodle users list:



        wget --no-cookies --header "Cookie: <name>=<value>" --post-data 'tab=search&name=+&personsubmit=Rechercher&keywords=&keywordsoption=allmine' https://moodle.unistra.fr/message/index.php






        share|improve this answer





















        • 4





          Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

          – Tuxdude
          Jul 27 '16 at 18:29






        • 2





          You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

          – Phil C
          May 25 '18 at 13:28
















        48














        I directly gave cookies of an existing connection to wget with --no-cookies and the Cookie HTTP request header. In my case it was a Moodle university login where logging in looks more complex (using multiple requests with a login ticket). I added --post-data because it was a POST request.
        For example, get all Moodle users list:



        wget --no-cookies --header "Cookie: <name>=<value>" --post-data 'tab=search&name=+&personsubmit=Rechercher&keywords=&keywordsoption=allmine' https://moodle.unistra.fr/message/index.php






        share|improve this answer





















        • 4





          Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

          – Tuxdude
          Jul 27 '16 at 18:29






        • 2





          You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

          – Phil C
          May 25 '18 at 13:28














        48












        48








        48







        I directly gave cookies of an existing connection to wget with --no-cookies and the Cookie HTTP request header. In my case it was a Moodle university login where logging in looks more complex (using multiple requests with a login ticket). I added --post-data because it was a POST request.
        For example, get all Moodle users list:



        wget --no-cookies --header "Cookie: <name>=<value>" --post-data 'tab=search&name=+&personsubmit=Rechercher&keywords=&keywordsoption=allmine' https://moodle.unistra.fr/message/index.php






        share|improve this answer















        I directly gave cookies of an existing connection to wget with --no-cookies and the Cookie HTTP request header. In my case it was a Moodle university login where logging in looks more complex (using multiple requests with a login ticket). I added --post-data because it was a POST request.
        For example, get all Moodle users list:



        wget --no-cookies --header "Cookie: <name>=<value>" --post-data 'tab=search&name=+&personsubmit=Rechercher&keywords=&keywordsoption=allmine' https://moodle.unistra.fr/message/index.php







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Feb 24 '16 at 13:02

























        answered May 4 '14 at 10:00









        baptxbaptx

        1,23811733




        1,23811733








        • 4





          Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

          – Tuxdude
          Jul 27 '16 at 18:29






        • 2





          You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

          – Phil C
          May 25 '18 at 13:28














        • 4





          Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

          – Tuxdude
          Jul 27 '16 at 18:29






        • 2





          You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

          – Phil C
          May 25 '18 at 13:28








        4




        4





        Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

        – Tuxdude
        Jul 27 '16 at 18:29





        Awesome tip. This is useful when you can access the cookie from your own machine and then use that from another headless machine from the command line. :)

        – Tuxdude
        Jul 27 '16 at 18:29




        2




        2





        You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

        – Phil C
        May 25 '18 at 13:28





        You can set multiple cookies at the same time also, --header "Cookie: access_token=IKVYJ;XSRF-TOKEN=5e10521d"

        – Phil C
        May 25 '18 at 13:28











        33














        If you need it for one time use, you can log in via browser and copy needed headers afterwards:



        screenshot
        Use "Copy as cURL" in the Network tab of Developer Tools (reload the page after opening) and replace curl's header flag -H and --data with wget's --header and --post-data.






        share|improve this answer


























        • Great solution!

          – menteith
          Mar 27 '18 at 18:59
















        33














        If you need it for one time use, you can log in via browser and copy needed headers afterwards:



        screenshot
        Use "Copy as cURL" in the Network tab of Developer Tools (reload the page after opening) and replace curl's header flag -H and --data with wget's --header and --post-data.






        share|improve this answer


























        • Great solution!

          – menteith
          Mar 27 '18 at 18:59














        33












        33








        33







        If you need it for one time use, you can log in via browser and copy needed headers afterwards:



        screenshot
        Use "Copy as cURL" in the Network tab of Developer Tools (reload the page after opening) and replace curl's header flag -H and --data with wget's --header and --post-data.






        share|improve this answer















        If you need it for one time use, you can log in via browser and copy needed headers afterwards:



        screenshot
        Use "Copy as cURL" in the Network tab of Developer Tools (reload the page after opening) and replace curl's header flag -H and --data with wget's --header and --post-data.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Sep 28 '18 at 0:18









        Pablo Bianchi

        302210




        302210










        answered Jun 12 '16 at 23:35









        useruser

        7,89615660




        7,89615660













        • Great solution!

          – menteith
          Mar 27 '18 at 18:59



















        • Great solution!

          – menteith
          Mar 27 '18 at 18:59

















        Great solution!

        – menteith
        Mar 27 '18 at 18:59





        Great solution!

        – menteith
        Mar 27 '18 at 18:59











        24














        I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.



        When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.






        share|improve this answer





















        • 1





          unfortunately not applicable in automated scripting

          – Znik
          Aug 21 '15 at 13:49
















        24














        I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.



        When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.






        share|improve this answer





















        • 1





          unfortunately not applicable in automated scripting

          – Znik
          Aug 21 '15 at 13:49














        24












        24








        24







        I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.



        When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.






        share|improve this answer















        I had the same problem. My solution was to do the login via Chrome and save the cookies data to a textfile. This is easily done with this Chrome extention: Chrome cookie.txt export extension.



        When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited May 26 '15 at 0:48









        Peter Mortensen

        13.6k1984111




        13.6k1984111










        answered Mar 9 '14 at 18:55









        Thor-Erik RødlandThor-Erik Rødland

        24122




        24122








        • 1





          unfortunately not applicable in automated scripting

          – Znik
          Aug 21 '15 at 13:49














        • 1





          unfortunately not applicable in automated scripting

          – Znik
          Aug 21 '15 at 13:49








        1




        1





        unfortunately not applicable in automated scripting

        – Znik
        Aug 21 '15 at 13:49





        unfortunately not applicable in automated scripting

        – Znik
        Aug 21 '15 at 13:49











        9














        You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2' works just fine. Note: you can also pass a file name to wget with the POST data in the file.






        share|improve this answer






























          9














          You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2' works just fine. Note: you can also pass a file name to wget with the POST data in the file.






          share|improve this answer




























            9












            9








            9







            You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2' works just fine. Note: you can also pass a file name to wget with the POST data in the file.






            share|improve this answer















            You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2' works just fine. Note: you can also pass a file name to wget with the POST data in the file.







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited May 26 '15 at 0:47









            Peter Mortensen

            13.6k1984111




            13.6k1984111










            answered Dec 15 '10 at 5:12









            J. PielJ. Piel

            10112




            10112























                9














                I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:



                wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'
                (this is one line, though it likely wraps on your browser)



                If you want the output saved to a file, change -O - to -O /some/file/name






                share|improve this answer






























                  9














                  I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:



                  wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'
                  (this is one line, though it likely wraps on your browser)



                  If you want the output saved to a file, change -O - to -O /some/file/name






                  share|improve this answer




























                    9












                    9








                    9







                    I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:



                    wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'
                    (this is one line, though it likely wraps on your browser)



                    If you want the output saved to a file, change -O - to -O /some/file/name






                    share|improve this answer















                    I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:



                    wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'
                    (this is one line, though it likely wraps on your browser)



                    If you want the output saved to a file, change -O - to -O /some/file/name







                    share|improve this answer














                    share|improve this answer



                    share|improve this answer








                    edited Dec 26 '17 at 3:05









                    Rolazaro Azeveires

                    249414




                    249414










                    answered Oct 11 '16 at 9:07









                    Caleb GrayCaleb Gray

                    2,41521625




                    2,41521625























                        8














                        If they're using basic authentication:



                        wget http://username:password@www.domain.com/page.html


                        If they're using POSTed form data, you'll need to use something like cURL instead.






                        share|improve this answer


























                        • I dont have access to change anything on the server, it is read only

                          – Señor Reginold Francis
                          Aug 24 '09 at 20:13






                        • 7





                          So? None of this requires you to change anything on the server.

                          – ceejayoz
                          Aug 24 '09 at 20:15
















                        8














                        If they're using basic authentication:



                        wget http://username:password@www.domain.com/page.html


                        If they're using POSTed form data, you'll need to use something like cURL instead.






                        share|improve this answer


























                        • I dont have access to change anything on the server, it is read only

                          – Señor Reginold Francis
                          Aug 24 '09 at 20:13






                        • 7





                          So? None of this requires you to change anything on the server.

                          – ceejayoz
                          Aug 24 '09 at 20:15














                        8












                        8








                        8







                        If they're using basic authentication:



                        wget http://username:password@www.domain.com/page.html


                        If they're using POSTed form data, you'll need to use something like cURL instead.






                        share|improve this answer















                        If they're using basic authentication:



                        wget http://username:password@www.domain.com/page.html


                        If they're using POSTed form data, you'll need to use something like cURL instead.







                        share|improve this answer














                        share|improve this answer



                        share|improve this answer








                        edited May 23 '17 at 11:33









                        Community

                        11




                        11










                        answered Aug 24 '09 at 20:08









                        ceejayozceejayoz

                        141k34217300




                        141k34217300













                        • I dont have access to change anything on the server, it is read only

                          – Señor Reginold Francis
                          Aug 24 '09 at 20:13






                        • 7





                          So? None of this requires you to change anything on the server.

                          – ceejayoz
                          Aug 24 '09 at 20:15



















                        • I dont have access to change anything on the server, it is read only

                          – Señor Reginold Francis
                          Aug 24 '09 at 20:13






                        • 7





                          So? None of this requires you to change anything on the server.

                          – ceejayoz
                          Aug 24 '09 at 20:15

















                        I dont have access to change anything on the server, it is read only

                        – Señor Reginold Francis
                        Aug 24 '09 at 20:13





                        I dont have access to change anything on the server, it is read only

                        – Señor Reginold Francis
                        Aug 24 '09 at 20:13




                        7




                        7





                        So? None of this requires you to change anything on the server.

                        – ceejayoz
                        Aug 24 '09 at 20:15





                        So? None of this requires you to change anything on the server.

                        – ceejayoz
                        Aug 24 '09 at 20:15











                        3














                        A solution which uses lynx and wget.



                        Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work



                        When you want to use wget to download some file from a site which requires login, you just need a cookie file.
                        In order to generate the cookie file, I choose lynx.
                        lynx is a text web browser.
                        First you need a configure file for lynx to save cookie.
                        Create a file lynx.cfg. Write these configuration into the file.



                        SET_COOKIES:TRUE
                        ACCEPT_ALL_COOKIES:TRUE
                        PERSISTENT_COOKIES:TRUE
                        COOKIE_FILE:cookie.file


                        Then start lynx with this command:



                        lynx -cfg=lynx.cfg http://the.site.com/login


                        After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout.
                        The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.



                        Then wget can download file from the site with this command.



                        wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz





                        share|improve this answer


























                        • what about if the login requires javascript? lynx does not seems to support javascript.

                          – Tiberiu
                          Dec 13 '18 at 17:41
















                        3














                        A solution which uses lynx and wget.



                        Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work



                        When you want to use wget to download some file from a site which requires login, you just need a cookie file.
                        In order to generate the cookie file, I choose lynx.
                        lynx is a text web browser.
                        First you need a configure file for lynx to save cookie.
                        Create a file lynx.cfg. Write these configuration into the file.



                        SET_COOKIES:TRUE
                        ACCEPT_ALL_COOKIES:TRUE
                        PERSISTENT_COOKIES:TRUE
                        COOKIE_FILE:cookie.file


                        Then start lynx with this command:



                        lynx -cfg=lynx.cfg http://the.site.com/login


                        After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout.
                        The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.



                        Then wget can download file from the site with this command.



                        wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz





                        share|improve this answer


























                        • what about if the login requires javascript? lynx does not seems to support javascript.

                          – Tiberiu
                          Dec 13 '18 at 17:41














                        3












                        3








                        3







                        A solution which uses lynx and wget.



                        Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work



                        When you want to use wget to download some file from a site which requires login, you just need a cookie file.
                        In order to generate the cookie file, I choose lynx.
                        lynx is a text web browser.
                        First you need a configure file for lynx to save cookie.
                        Create a file lynx.cfg. Write these configuration into the file.



                        SET_COOKIES:TRUE
                        ACCEPT_ALL_COOKIES:TRUE
                        PERSISTENT_COOKIES:TRUE
                        COOKIE_FILE:cookie.file


                        Then start lynx with this command:



                        lynx -cfg=lynx.cfg http://the.site.com/login


                        After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout.
                        The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.



                        Then wget can download file from the site with this command.



                        wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz





                        share|improve this answer















                        A solution which uses lynx and wget.



                        Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work



                        When you want to use wget to download some file from a site which requires login, you just need a cookie file.
                        In order to generate the cookie file, I choose lynx.
                        lynx is a text web browser.
                        First you need a configure file for lynx to save cookie.
                        Create a file lynx.cfg. Write these configuration into the file.



                        SET_COOKIES:TRUE
                        ACCEPT_ALL_COOKIES:TRUE
                        PERSISTENT_COOKIES:TRUE
                        COOKIE_FILE:cookie.file


                        Then start lynx with this command:



                        lynx -cfg=lynx.cfg http://the.site.com/login


                        After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout.
                        The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.



                        Then wget can download file from the site with this command.



                        wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz






                        share|improve this answer














                        share|improve this answer



                        share|improve this answer








                        edited Nov 3 '18 at 8:00









                        alls0rts

                        496




                        496










                        answered Oct 19 '17 at 8:32









                        PokerFacePokerFace

                        490310




                        490310













                        • what about if the login requires javascript? lynx does not seems to support javascript.

                          – Tiberiu
                          Dec 13 '18 at 17:41



















                        • what about if the login requires javascript? lynx does not seems to support javascript.

                          – Tiberiu
                          Dec 13 '18 at 17:41

















                        what about if the login requires javascript? lynx does not seems to support javascript.

                        – Tiberiu
                        Dec 13 '18 at 17:41





                        what about if the login requires javascript? lynx does not seems to support javascript.

                        – Tiberiu
                        Dec 13 '18 at 17:41











                        1














                        Example to download with wget on server a big file link that can be obtained in your browser.



                        In example using Google Chrome.



                        Login where you need, and press download. Go to download and copy your link.



                        enter image description here



                        Then open DevTools on a page where you where login, go to Console and get your cookies, by entering document.cookie



                        enter image description here



                        Now, go to server and download your file: wget --header "Cookie: <YOUR_COOKIE_OUTPUT_FROM_CONSOLE>" <YOUR_DOWNLOAD_LINK>



                        enter image description here






                        share|improve this answer




























                          1














                          Example to download with wget on server a big file link that can be obtained in your browser.



                          In example using Google Chrome.



                          Login where you need, and press download. Go to download and copy your link.



                          enter image description here



                          Then open DevTools on a page where you where login, go to Console and get your cookies, by entering document.cookie



                          enter image description here



                          Now, go to server and download your file: wget --header "Cookie: <YOUR_COOKIE_OUTPUT_FROM_CONSOLE>" <YOUR_DOWNLOAD_LINK>



                          enter image description here






                          share|improve this answer


























                            1












                            1








                            1







                            Example to download with wget on server a big file link that can be obtained in your browser.



                            In example using Google Chrome.



                            Login where you need, and press download. Go to download and copy your link.



                            enter image description here



                            Then open DevTools on a page where you where login, go to Console and get your cookies, by entering document.cookie



                            enter image description here



                            Now, go to server and download your file: wget --header "Cookie: <YOUR_COOKIE_OUTPUT_FROM_CONSOLE>" <YOUR_DOWNLOAD_LINK>



                            enter image description here






                            share|improve this answer













                            Example to download with wget on server a big file link that can be obtained in your browser.



                            In example using Google Chrome.



                            Login where you need, and press download. Go to download and copy your link.



                            enter image description here



                            Then open DevTools on a page where you where login, go to Console and get your cookies, by entering document.cookie



                            enter image description here



                            Now, go to server and download your file: wget --header "Cookie: <YOUR_COOKIE_OUTPUT_FROM_CONSOLE>" <YOUR_DOWNLOAD_LINK>



                            enter image description here







                            share|improve this answer












                            share|improve this answer



                            share|improve this answer










                            answered Apr 13 '18 at 9:24









                            Alex IvasyuvAlex Ivasyuv

                            3,787165383




                            3,787165383






























                                draft saved

                                draft discarded




















































                                Thanks for contributing an answer to Stack Overflow!


                                • Please be sure to answer the question. Provide details and share your research!

                                But avoid



                                • Asking for help, clarification, or responding to other answers.

                                • Making statements based on opinion; back them up with references or personal experience.


                                To learn more, see our tips on writing great answers.




                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function () {
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f1324421%2fhow-to-get-past-the-login-page-with-wget%23new-answer', 'question_page');
                                }
                                );

                                Post as a guest















                                Required, but never shown





















































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown

































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown







                                這個網誌中的熱門文章

                                Tangent Lines Diagram Along Smooth Curve

                                Yusuf al-Mu'taman ibn Hud

                                Zucchini