StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
primarykey
Id
19011470
data
AcceptedAnswerId
0
AnswerCount
0
ClosedDate
CommentCount
1
CommunityOwnedDate
CreationDate
2013-09-25T17:30:16.083
FavoriteCount
0
LastActivityDate
2013-09-25T17:30:16.083
LastEditDate
LastEditorUserId
0
OwnerUserId
2744342
ParentId
18993956
PostTypeId
2
Score
1
ViewCount
0
LastEditorDisplayName
text
Body
It looks like your data in both the files are already sorted. If not, first sort them by an external tool or using Python. Then, you can go through the two files in parallel. You read a number from the <code>Numbers.txt</code> file, and see if it is in a range in <code>Ranges.txt</code> file, reading as many lines from that file as needed to answer that question. Then read the next number from <code>Numbers.txt</code>, and repeat. The idea is similar to merging two sorted arrays, and should run in <code>O(n+m)</code> time, <code>n</code> and <code>m</code> are the sizes of the files. If you need to sort the files, the run time is <code>O(n lg(n) + m lg(m))</code>. Here is a quick program I wrote to implement this: <pre><code>import sys from collections import Counter class Gen(object): __slots__ = ('rfp', 'nfp', 'mn', 'mx', 'num', 'd', 'n') def __init__(self, ranges_filename, numbers_filename): self.d = Counter() # sum of elements keyed by range self.n = Counter() # number of elements keyed by range self.rfp = open(ranges_filename) self.nfp = open(numbers_filename) # Read the first number and the first range values self.num = float(self.nfp.next()) # Current number self.mn, self.mx = [int(x) for x in self.rfp.next().split()] # Current range def go(self): while True: if self.mx < self.num: try: self.mn, self.mx = [int(x) for x in self.rfp.next().split()] except StopIteration: break else: if self.mn <= self.num <= self.mx: self.d[(self.mn, self.mx)] += self.num self.n[(self.mn, self.mx)] += 1 try: self.num = float(self.nfp.next()) except StopIteration: break self.nfp.close() self.rfp.close() return self.d, self.n def run(ranges_filename, numbers_filename): r = Gen(ranges_filename, numbers_filename) d, n = r.go() for mn, mx in sorted(d): s, N = d[(mn, mx)], n[(mn, mx)] if s: av = s/N else: av = 0 sys.stdout.write('%d %d %.3f\n' % (mn, mx, av)) </code></pre> On files with 10,000,000 numbers in each of the files, the above runs in about 1.5 minute on my computer without the output part.
Tags
Title
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. POEfficient Python way to process two huge files?
 singulars
 PostTypePostTypeId
 PTQuestion
PostTypePostTypeId
1. PTAnswer
UserLastEditorUserId
1. This table or related slice is empty.
UserOwnerUserId
1. USAlok--
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. This table or related slice is empty.
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId
1. CONot sure why I made a class for this but hopefully you get the idea :-).
 singulars
 PostPostId
 PO
 UserUserId
 USAlok--

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.