Databases
StackOverflow2013
Postsdbo
POPython and BS4 - stop reading after certain amount of times
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython and BS4 - stop reading after certain amount of times
    primarykey
    data
    text
    <p>First time Python 3 user and starting to get the hang of it. As an exercise I'm trying to read the table (with BeautifulSoup4) from <a href="http://rateyourmusic.com/customchart" rel="nofollow noreferrer">http://rateyourmusic.com/customchart</a> and convert the Rank, Artist, Album, and Year into a dictionary. I want to then put the dictionary into a MySQL database. I was able to get all the info from the table and put them into variables which I then put into a dictionary, but I have a slight problem. The last entry in the table is an advertisement so it doesn't follow the other table rows above it. I want to only read the first 100 rows of the table. I get an error when trying to read the advertisement row.</p> <p>Here is my code. Please any help would be great. Also if you see any errors in my code or how I could've done things better please let me know. </p> <p>So it is printing the dictionaries and everything looks good, but it gives me an error after printing all of them out.</p> <pre><code>from bs4 import BeautifulSoup from urllib.request import Request, urlopen url = "http://rateyourmusic.com/customchart" req = Request(url, headers={'User-Agent': 'Mozilla/5.0'}) soup = BeautifulSoup(urlopen(req)) table = soup.find("table", {"class" : "mbgen"}) totalList = [] for row in table.findAll("tr"): cells = row.findAll("td") rank = int(cells[0].find(class_="ooookiig").text) artist = cells[2].find(class_="artist").text album = cells[2].find(class_="album").text year = cells[2].find(class_="mediumg").text year = int(year[1:5]) chartData = {"Rank":rank, "Artist":artist, "Album":album, "Year":year} totalList.append(chartData) print(chartData) </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USThe Nomad
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COCan you please provide the full traceback?
      singulars
      1. POPython and BS4 - stop reading after certain amount of times
      1. USaIKid
    2. COTraceback (most recent call last): File "C:\Programming\RateYourMusicCrawler\AlbumInfoCrawler.py", line 21, in <module> rank = int(cells[0].find(class_="ooookiig").text) AttributeError: 'NoneType' object has no attribute 'text'
      singulars
      1. POPython and BS4 - stop reading after certain amount of times
      1. USThe Nomad
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload