Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>How are your one thousand devices connecting? If each one establishes a TCP/IP connection to your server you'll need to make sure you have enough file descriptors in the machine to which they connect. Look at <code>/proc/sys/fs/file-max</code> to see the maximum. A thousand client connections to a single server machine is considered a large number.</p> <p>How much data is in each record? Will you overwhelm your network hardware? If each record is ten bytes, then you're talking about twenty million bytes coming in per second, or one hundred and sixty million bits. A 100 megabit per second ethernet interface won't be nearly enough. Even a gigabit interface is questionable: it's hard to maintain enormous throughput. Keep in mind that if the DBMS is on a different machine from the server that receives your data, these records will have to both come in and go out, doubling your network throughput.</p> <p>How are you going to handle the possibility that your DBMS or some other part of your system will get behind in its workload? An occasional thirty-second delay by the DBMS in accepting INSERT commands is very possible, but an enormous amount of unhandled data will accumulate during that time.</p> <p>You should consider partitioning this problem into groups of, maybe, 50 or 100 devices, and 20 or ten central server setups collecting data. If you do that you won't have a single point of failure, you won't be pushing your networking hardware extremely hard, and you may be able to work out some kind of failover strategy if you lose some hardware. You'll also be able to use much cheaper and more cost-effective server and networking equipment.</p> <p>On MySQL, use as few indexes as you can to support the queries you need to do. Keep in mind that doing summary queries (like <code>SELECT COUNT(*) FROM raw WHERE timestamp &gt; NOW() - INTERVAL 1 HOUR</code>) can dramatically slow down INSERT operations while they are running.</p> <p>You may want to consider using a queuing system such as ActiveMQ to handle your data flow. </p>
    singulars
    1. This table or related slice is empty.
    1. PODatabase server setup to handle 10K inserts and to frequent select operations
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USO. Jones
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COalso total amount of storage - how many days worth of this information makes sense?
      singulars
      1. PO
      1. USRandy
    2. COThanks for your valuable comment. How about using infobright?
      singulars
      1. PO
      1. USshivashankar
    3. COCould you expand a bit on the ActiveMQ option? Thanks
      singulars
      1. PO
      1. USGevorg
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload