Databases
StackOverflow2013
Postsdbo
POMore than 4 billion key value pairs in Redis?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMore than 4 billion key value pairs in Redis?
    text
    copied!<p>I am trying to store ip numbers in redis along with associated ISP information. I have Maxmind data and the csv files contain start and end numbers for each ISP.</p> <p>When querying in SQL I can check for an IP(after converting it to a number) to be available in a range and get the associated ISP.</p> <p>I was thinking of converting all the ranges to individual numbers and submit all the key values pairs in Redis for faster lookup. This approximately will result in 4 billion key value pairs in the Redis store. I have done this for a few hundred million key value pairs but I am looking for advice/suggestions when moving to 4 billion pairs in Redis. Any performance issues I must be aware of or are there ways I can do this better ?</p> <p>Thank you for all the suggestions.</p> <p>UPDATE: Thanks to the suggestions below I could get this working. Thought I'd share the Python code (quick and dirty) for this here :</p> <pre> import redis import pymysql conn = pymysql.connect(host='localhost',user='user',passwd='password',db='foo') cur = conn.cursor() cur.execute('select startipnum,endipnum,isp from wiki.ipisp order by endipnum;') result = cur.fetchall() r = redis.StrictRedis(host='localhost', port=6379, db=0) ispctr = 1 for row in result: tempDict = {'ispname':row[2],'fromval':row[0],'toval':row[1]} namefield = ispctr r.hmset(namefield,tempDict) r.zadd('ispmaxindex',row[1],namefield) ispctr = ispctr+1 conn.close() ipstotest = ['23.23.23.23','24.96.185.10','203.59.91.235','188.66.105.50','99.98.163.93'] for ip in ipstotest: ipvalsList = [int(ipoct) for ipoct in ip.split('.')] ipnum = (16777216*ipvalsList[0]) + (65536*ipvalsList[1]) + (256*ipvalsList[2]) + ipvalsList[3] ipnum = long(ipnum) tempVal1 = r.zrangebyscore('ispmaxindex',ipnum,float('Inf'),0,1) tempval2 = r.hgetall(tempval1[0]) print tempval2['ispname'] </pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload