Using Python To Submit A Web Form With Liburl Or Requests

I am trying to submit a form on the webpage and get to the html of the next page after the form is submitted. I found two ways of doing this using either requests or liburl. import

Solution 1:

The form on that page is will actually be submitted by JavaScript, so just looking at the <form /> element is not (necessarily) enough. You can use e.g. Firebug's network tab or the Chrome developer tools to inspect the POST request after you submit the form in order to see what's actually submitted.

This seems to work:

import requests

url = 'https://fcq.colorado.edu/scripts/broker.exe'

payload = {
    "_PROGRAM": "fcqlib.fcqdata.sas",
    "_SERVICE": "fcq",
    "camp": "BD",
    "fileFrmt": "HTM",
    "ftrm": "1",
    "fyr": "2007",
    "grp1": "ALL",
    "jjj": "mytst",
    "ltrm": "7",
    "lyr": "2013",
    "sort": "descending YEARTERM SUBJECT COURSE SECTION",
}

payload.update({
    'subj': 'CSCI',
    'crse': '1300',
})


def main():
    r = requests.post(url, payload)
    withopen("requests_results.html", "w") as f:
        f.write(r.content)
    return 0

if __name__ == '__main__':
    main()

Solution 2:

The target URL of the form is https://fcq.colorado.edu/scripts/broker.exe (see the action attribute of the <form> tag). So you need to replace:

url = 'https://fcq.colorado.edu/UCBdata.htm'

with

url = 'https://fcq.colorado.edu/scripts/broker.exe'

Solution 3:

I don't know if you're still interested, but I actually did something similar with the exact same site, and had to deal with the exact same problem. I used the Mechanize and Requests library to build a Python scraper API of sorts.

You can see my code for it on github, and I welcome any pull requests if you feel like you can do something better.