Databases
StackOverflow2013
Postsdbo
POHow do I decrease the memory used by a large list in python
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow do I decrease the memory used by a large list in python
    primarykey
    data
    text
    <p>I'm writing a program, it works fine but when it loads the database(a 100MB text file) to a list it's memory usage becomes 700-800MB</p> <p>Code used to load the file to a list:</p> <pre><code>database = [] db = open('database/db.hdb') dbcontent = db.read() for line in dbcontent.split('\n'): line = line.split(':') database.append(line) </code></pre> <p>Snipplet from db.hdb:</p> <pre><code>14200:917cb8a3d1d9eb24af6c5bcf3bf7e401:Trojan.Downloader-1420 7168:a105e2cc8148158cd048360eb847c7d0:Trojan.Downloader-1421 7168:c61ef67b5e7eef19ef732f55116742f6:Trojan.Downloader-1422 7168:851b6320148122104f50445ea2684c9f:Trojan.Downloader-1423 7168:ca128383c79a56d930eb4a7ff5026e31:Trojan.Downloader-1424 355204:4af89f8d219f94462cf2f8cb8eb4c6d7:Trojan.Bancos-2053 356984:2bfb53d76891059b79122e13d1537e4a:Trojan.Bancos-2054 363520:edbbdf497cda1ba79c06ea40673d963e:Trojan.Bancos-2055 367616:d85f719b032dbf39800d90ca881fd225:Trojan.Bancos-2056 370688:6cb572fd2452416dc4ea09e3ad917e66:Trojan.Bancos-2057 370688:ef34885677230061649d30ea66d7b0a1:Trojan.Bancos-2058 399360:8578b664706cfdc2f653680bac1b1b6e:Trojan.Bancos-2059 401408:de62af250b5a3e1ba1e9c517629383dd:Trojan.Bancos-2060 622592:8a236340c0a8c76343f6fb581314fadf:Trojan.Bancos-2061 622592:29f3499488ba1814c62fac3c2f3bda54:Trojan.Bancos-2062 622592:5d023bccf2ff097ccbc0ab0eab4a6ee7:Trojan.Bancos-2063 622592:3d6a25ed1f0e2001e72812ce1adf37d3:Trojan.Bancos-2064 622592:eaff242b601807e5805c189752d39124:Trojan.Bancos-2065 623104:8cd8e788d33cf40412d3346a525e4cce:Trojan.Bancos-2066 625152:25470d6895cb0e5c2e7181cb9a201ae0:Trojan.Bancos-2067 625152:436d574cef37b2e62d9b801b8fc2c4f1:Trojan.Bancos-2068 647168:51eb4e43f24cf511e6715cc8667babcd:Trojan.Bancos-2069 </code></pre> <p>(The full file has ~1800000 lines)</p> <p>How do I decrease the memory usage</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USbummi
    1. USYannick
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTDuplicate
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow do I decrease the memory used by a large list in python
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow do I decrease the memory used by a large list in python
      1. USn611x007
      1. VTFavorite
    3. VO
      singulars
      1. POHow do I decrease the memory used by a large list in python
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIt depends on what you are going to do with that data. You might just simply load piece by piece. Or buy more RAM. :) 800MB is no longer a problem nowadays.
      singulars
      1. POHow do I decrease the memory used by a large list in python
      1. USfreakish
    2. COUnless I've a raspberry pi which has 512MB ram and isn't upgradable :(, it's no problem on my windows pc but on a raspberry pi it is
      singulars
      1. POHow do I decrease the memory used by a large list in python
      1. USYannick
    3. COAlso, I can't see how you can achieve something meaningful with your databse being a list: searching anything in a list is linear in time -- that is, if you have to check if a given signature is in your databse, you have to check all items one by one. You should _at least_ use a dictionary and use the signature field as key
      singulars
      1. POHow do I decrease the memory used by a large list in python
      1. USjsbueno
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload