Databases
StackOverflow2013
Postsdbo
POHow to handle billions of objects without "Outofmemory" error
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to handle billions of objects without "Outofmemory" error
    primarykey
    data
    text
    <p>I have an application which may needs to process billions of objects.Each object of is of TRange class type. These ranges are created at different parts of an algorithm which depends on certain conditions and other object properties. As a result, if you have 100 items, you can't directly create the 100th object without creating all the prior objects. If I create all the (billions of) objects and add to the collection, the system will throw Outofmemory error. Now I want to iterate through each object mainly for two purposes:</p> <ol> <li>To apply an operation for each TRange object(eg:Output certain properties)</li> <li>To get a cumulative sum of a certain property.(eg: Each range has a weight property and I want to retreive totalweight that is a sum of all the range weights).</li> </ol> <p>How do I effectively create an Iterator for these object without raising Outofmemory?</p> <p>I have handled the first case by passing a function pointer to the algorithm function. For eg:</p> <pre><code>procedure createRanges(aProc: TRangeProc);//aProc is a pointer to function that takes a //TRange var range: TRange; rangerec: TRangeRec; begin range:=TRange.Create; try while canCreateRange do begin//certain conditions needed to create a range rangerec := ReturnRangeRec; range.Update(rangerec);//don't create new, use the same object. if Assigned(aProc) then aProc(range); end; finally range.Free; end; end; </code></pre> <p>But the problem with this approach is that to add a new functionality, say to retrieve the Total weight I have mentioned earlier, either I have to duplicate the algorithm function or pass an optional out parameter. Please suggest some ideas.</p> <p>Thank you all in advance<br/> Pradeep</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAndrzej Doyle
    1. USPradeep
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COIf you need design/algorithm advice, we need much more information...
      singulars
      1. POHow to handle billions of objects without "Outofmemory" error
      1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload