Databases
StackOverflow2013
Postsdbo
POIs there a Java local queue library I can use that keeps memory usage low by dumping to the hard drive?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs there a Java local queue library I can use that keeps memory usage low by dumping to the hard drive?
    primarykey
    data
    text
    <p>This maybe not possible but I thought I might just give it a try. I have some work that process some data, it makes 3 decisions with each data it proceses: keep, discard or modify/reprocess(because its unsure to keep/discard). This generates a very large amount of data because the reprocess may break the data into many different parts.</p> <p>My initial method was to send it to my executionservice that was processing the data but because the number of items to process was large I would run out of memory very quickly. Then I decided to maybe offload the queue off to a messaging server(rabbitmq) which works fine but now I'm bound by network IO. What I like about rabbitmq is it keeps messages in memory up to a certain level and then dumps old messages to the local drive so if I have 8 gigs of memory on my server I can still have a 100 gig message queue.</p> <p>So my question is, is there any library that has a similar feature in Java? Something that I can use as a nonblocking queue that keeps only X items in queue(either by number of items or size) and writes the rest to the local drive. </p> <p>note: Right now I'm only asking for this to be used on one server. In the future I might add more servers but because each server is self-generating data I would try to take messages from one queue and push them to another if one server's queue is empty. The library would not need to have network access but I would need to access the queue from another Java process. I know this is a long shot but thought if anyone knew it would be SO.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USLostsoul
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIs there a Java local queue library I can use that keeps memory usage low by dumping to the hard drive?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POIs there a Java local queue library I can use that keeps memory usage low by dumping to the hard drive?
      1. USMarianP
      1. VTFavorite
    1. COI have never done it, but Apache ActiveMQ is another MQ server, which is open-source and written in Java. I believe it can be used as an embedded service.
      singulars
      1. POIs there a Java local queue library I can use that keeps memory usage low by dumping to the hard drive?
      1. USAmir Pashazadeh
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload