Databases
StackOverflow2013
Postsdbo
POHow to fill and submit form using Python 3 and only Python 3 libraries
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to fill and submit form using Python 3 and only Python 3 libraries
    primarykey
    data
    text
    <p>I am working on this as a free time project.</p> <p>I would like to use Python 3 to login to a site (fill form and submit) and then I plan to scrape the page for some data. The reason I am strictly looking for a solution in Python 3 is because I am trying to learn more about Python and figured I would go straight to Python 3. I have seen several great looking tools such as mechanize but they only seem to support Python 2.</p> <p>Planning to use this for a financial investment site but let's just use starbucks as an example.</p> <pre><code>def loginToStockSite(username, pwd): url = "https://www.starbucks.com/account/signin" values = {"Account.UserName" : username, "Account.PassWord" : pwd} data = urllib.parse.urlencode(values) data = data.encode('utf-8') req = urllib.request.Request(url, data) sock = urllib.request.urlopen(req) htmlSource = sock.read().decode('utf-8') return htmlSource </code></pre> <p>I have gotten very confused from the various encode, decode, url opener, etc. examples. I have not found a solution that has worked for me yet.</p> <p>Thanks in advance for any help with my "for fun" project.</p> <hr> <p>Update</p> <p>Here is the html I get back (with a bunch cut out to fit the character limit for a question)</p> <pre><code>&lt;!DOCTYPE html&gt; &lt;head&gt; &lt;meta charset="utf-8" /&gt; &lt;title&gt;Account Sign In | Starbucks Coffee Company&lt;/title&gt; &lt;meta name="viewport" content="width=device-width, initial-scale=1" /&gt; &lt;/head&gt; &lt;div class="fields"&gt; &lt;div class="region size1of1"&gt; &lt;div class="validation_summary warning validation_medium"&gt;&lt;h2&gt;Please Enable Cookies to Continue&lt;/h2&gt;&lt;p&gt;To sign in to the Starbucks web site, please enable cookies in your web browser.&lt;/p&gt;&lt;/div&gt; &lt;div class="fields"&gt; &lt;form action="/account/signin" class="siren region size1of2" id="accountForm" method="post"&gt; &lt;fieldset class="login_position"&gt; &lt;legend class="hidden_visually"&gt;I have a Starbucks account.&lt;/legend&gt; &lt;h2 class="existing_acc_h3"&gt;I have a Starbucks account.&lt;/h2&gt; &lt;div id="user_info" class="region size1of2 reset"&gt; Hi, &lt;span id="info_user"&gt;&lt;/span&gt; &lt;/div&gt; &lt;div class="size1of2"&gt; &lt;a id="not_me" href="#notme"&gt;Not You?&lt;/a&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;div id="connect_cont" &gt; &lt;div id="text_cont" class="hidden"&gt; &lt;h3&gt;Have a Starbucks account?&lt;/h3&gt; &lt;p&gt;Connect your Facebook account to your Starbucks account by logging in here.&lt;/p&gt; &lt;/div&gt; &lt;/div&gt; &lt;/li&gt; &lt;li&gt; &lt;label for="Account_UserName"&gt;Username &lt;span class="required"&gt;*&lt;/span&gt;&lt;/label&gt; &lt;label for="Account_UserName" class="hidden"&gt;Starbucks Username &lt;span class="required"&gt;*&lt;/span&gt;&lt;/label&gt; &lt;input class="field_xlarge" id="Account_UserName" maxlength="200" name="Account.UserName" type="text" value="MY_USERNAME_WAS_HERE" /&gt; &lt;/li&gt; &lt;li&gt; &lt;label for="Account_PassWord"&gt;Password &lt;span class="required"&gt;*&lt;/span&gt;&lt;/label&gt; &lt;label for="Account_PassWord" class="hidden"&gt;Starbucks Password &lt;span class="required"&gt;*&lt;/span&gt;&lt;/label&gt; &lt;input class="field_xlarge password" id="Account_PassWord" maxlength="200" name="Account.PassWord" type="password" value="MY_PASSWORD_WAS_HERE" /&gt; &lt;/li&gt; &lt;li&gt; Forgot your &lt;a href="/account/forgot-username?AllowGuest=False"&gt;username&lt;/a&gt; or &lt;a href="/account/forgot-password?AllowGuest=False"&gt;password&lt;/a&gt;? &lt;p class="privacy_policy hidden"&gt; &lt;a href="/about-us/company-information/online-policies/privacy-policy"&gt;Concerned about privacy?&lt;/a&gt; &lt;/p&gt; &lt;/li&gt; &lt;li class="inline push"&gt; &lt;input type="checkbox" id="Account.IsRememberMe" name="Account.IsRememberMe" value="True" class="checkbox" /&gt; &lt;label for="Account.IsRememberMe"&gt;Keep me signed in.&lt;/label&gt; &lt;/li&gt; &lt;/ol&gt; &lt;/fieldset&gt; &lt;fieldset&gt; &lt;input id="ReturnUrl" name="ReturnUrl" type="hidden" value="" /&gt; &lt;input id="AllowGuest" name="AllowGuest" type="hidden" value="False" /&gt; &lt;input id="isJavaScriptDisabled" name="isJavaScriptDisabled" type="hidden" value="True" /&gt; &lt;span class="button"&gt;&lt;button type="submit"&gt;Sign In&lt;/button&gt;&lt;/span&gt; &lt;/fieldset&gt; &lt;fieldset class="submit"&gt; &lt;div id="fb_container"&gt; &lt;div id="fb_btn_cont"&gt; Or log in using Facebook. &lt;p&gt;&lt;a class="fb_button fb_button_medium" id="connect" href="#connect"&gt;&lt;span class="fb_button_text"&gt;Login with Facebook&lt;/span&gt;&lt;/a&gt;&lt;/p&gt; &lt;/div&gt; &lt;/div&gt; &lt;/fieldset&gt; &lt;/form&gt; &lt;div class="region size1of2 block_login"&gt; &lt;h2&gt;I need a Starbucks account.&lt;/h2&gt; &lt;p&gt;&lt;span class="button"&gt;&lt;a href="/account/create"&gt;Create An Account&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; &lt;p&gt;With a Starbucks account you can register and manage your Starbucks Cards and participate in &lt;a href="/card/rewards"&gt;My Starbucks Rewards&lt;/a&gt;.&lt;/p&gt; &lt;ul class ="basic"&gt; &lt;li&gt;Enjoy a free drink on your birthday&lt;/li&gt; &lt;li&gt;Protect your balance if your Starbucks Card is missing or stolen.&lt;/li&gt; &lt;li&gt;Transfer money between cards.&lt;/li&gt; &lt;li&gt;Track your earnings in My Starbucks Rewards&lt;/li&gt; &lt;li&gt;Reload your Card balance automatically&lt;/li&gt; &lt;/ul&gt; &lt;/div&gt; &lt;/div&gt; &lt;div class="fields"&gt; &lt;div class="region size1of1"&gt; &lt;ul id="breadcrumb"&gt; &lt;li&gt;&lt;a href="/card"&gt;Card&lt;/a&gt; .&lt;ul&gt; &lt;li&gt;&lt;a href="/card/rewards"&gt;My Starbucks Rewards&lt;/a&gt; .&lt;ul&gt; &lt;li&gt;View Your Stars&lt;/li&gt; &lt;/ul&gt;&lt;/li&gt; &lt;/ul&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;div id="footer"&gt; &lt;div class="container"&gt; &lt;form id="search" method="get" action="/search"&gt; &lt;fieldset&gt; &lt;input id="searchbox" name="keywords" title="Search Keyword" maxlength="100" class="search_input" /&gt; &lt;span class="button button_search"&gt;&lt;button id="submit_search_util" type="submit"&gt;Search&lt;/button&gt;&lt;/span&gt; &lt;/fieldset&gt; &lt;/form&gt; &lt;div class="fields"&gt; &lt;div class="region size5of6 suffix1of6"&gt; &lt;div class="footer_categorical"&gt;&lt;ol class="blocks blocks-five-up"&gt; &lt;li&gt;&lt;h4&gt; &lt;a href="/shop/card"&gt;Buy a Card&lt;/a&gt; &lt;/h4&gt; &lt;/li&gt; &lt;li&gt;&lt;h4&gt; &lt;a href="/card"&gt;Manage Your Card&lt;/a&gt; &lt;/h4&gt; &lt;ol&gt; &lt;li&gt;&lt;a href="/Card#cardBalanceWrapper"&gt;Check Balance&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/card/reload/one-time"&gt;Reload Your Card&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/card/manage/transfer"&gt;Transfer Funds&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/card/manage/history"&gt;View Transactions&lt;/a&gt;&lt;/li&gt; &lt;/ol&gt; &lt;/li&gt; &lt;li&gt;&lt;h4&gt; &lt;a href="/card/rewards"&gt;My Starbucks Rewards&lt;/a&gt; &lt;/h4&gt; &lt;ol&gt; &lt;li&gt;&lt;a href="/account/create/register"&gt;Register Your Card&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/account"&gt;View Your Stars&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/card/rewards/gold"&gt;Keep Your Gold Benefits&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/card/rewards/rewards-program-ts-and-cs"&gt;Rewards Program Terms and Conditions&lt;/a&gt;&lt;/li&gt; &lt;/ol&gt; &lt;/li&gt; &lt;li&gt;&lt;h4&gt; Learn More &lt;/h4&gt; &lt;ol&gt; &lt;li&gt;&lt;a href="/card/card-terms-and-conditions"&gt;Card Terms and Conditions&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/card/egift"&gt;What is a Starbucks Card eGift?&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/customer-service/faqs/card"&gt;Card FAQs&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="/account"&gt;Manage Your Account&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="http://mystarbucksidea.force.com/ideaList?ext=0&amp;amp;lsi=0&amp;amp;category=Starbucks+Card"&gt;My Starbucks Idea&lt;/a&gt;&lt;/li&gt; &lt;/ol&gt; &lt;/li&gt; </code></pre> <p>I expected to get back a page like I get when I login in using a browser and have a div with the current balance on whatever the default gift card is that is registered.</p> <pre><code> &lt;div class="balance-amount numbers"&gt; </code></pre> <p>That is the div that is visible. I found the balance in another place in the page source also but I do not see it on the page.</p> <pre><code>&lt;p class="card_balance numbers"&gt; &lt;span&gt;$27.68&lt;/span&gt; &lt;span class="datestamp"&gt;3/10/2013 2:12 PM&lt;/span&gt; &lt;/p&gt; </code></pre> <p>Anyways, I would like to input the login info and submit the form using Python 3 or post the data somehow to login and then (as a use case but outside what is needed to answer the question) I would pull the account balance from the html.</p> <p>I did find that my username and password are apparently in their corresponding fields and in the validation summary it says</p> <pre><code>&lt;div class="validation_summary warning validation_medium"&gt;&lt;h2&gt;Please Enable Cookies to Continue&lt;/h2&gt;&lt;p&gt;To sign in to the Starbucks web site, please enable cookies in your web browser.&lt;/p&gt;&lt;/div&gt; </code></pre> <p>I did see some examples that dealt with cookies. Is that the problem? I will research that possible solution. In the meantime, hope this helps you help me. Thanks.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USd456
    1. USd456
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POHow to fill and submit form using Python 3 and only Python 3 libraries
      1. This table or related slice is empty.
      1. VTUpMod
    1. COCould you also include in your question what you're seeing when you run this and how that differs from what you expected?
      singulars
      1. POHow to fill and submit form using Python 3 and only Python 3 libraries
      1. USMartin Atkins
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload