Databases
StackOverflow2013
Postsdbo
POReducing the number of table columns
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POReducing the number of table columns
    primarykey
    data
    text
    <p>I'm trying to turn a huge spreadsheet of data into a database to make data analysis easier, but I'm running into problems with too many columns. I've tried my best to learn about normalization, but I'm having a hard time applying it to this use case.</p> <h2>Scenario</h2> <p>We are performing N <em>independent</em> measurements on rectangular blocks. Measurements include:</p> <ul> <li>Length (or Measurement 0)</li> <li>Width (or Measurement 1)</li> <li>Height (or Measurement 2)</li> <li>Mass (or Measurement 3)</li> <li>Color (or Measurement 4)</li> <li>...</li> <li>Measurement N</li> </ul> <p>There are over 7000 measurements (complicated blocks)! The measurements have limits. If a block fails one or more measurements, all measurements are repeated to verify. If it fails again, the block is deemed a failure.</p> <p>The blocks are serialized and there are thousands of them.</p> <h2>Data Source</h2> <p>A huge spreadsheet (table). The fields are: Block Number, Length, Width, Height, Mass, Color, ..., Measurement N. Each row represents one test run or execution of all measurements. Since we have a retest policy, there may be multiple rows with results from the same block.</p> <h2>Help!</h2> <p>This source table seems like an intuitive format, but doesn't seem like the best format for a database. At first I tried to put it in an SQLite database and ran into the 2000 column limit. Yes I could recompile SQLite with more columns or use another database engine, but this sounds like more of a database design issue. Do you have a better design idea?</p> <p>P.S. Sorry so long, but thanks for reading!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USWes
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POReducing the number of table columns
      1. This table or related slice is empty.
      1. VTUpMod
    1. COAre all of the measured values the same data type? That is, are all the measured values integers, or all floats, or all decimal?
      singulars
      1. POReducing the number of table columns
      1. USMike Sherrill 'Cat Recall'
    2. COMost measurements are floats, but some are text, like color. If there was a significant benefit, I suppose I could split measurements into groups base on their value format. In this post processing application maximum performance isn't as critical as accessibility.
      singulars
      1. POReducing the number of table columns
      1. USWes
    3. CODo you always make 7000 measurements at a time, or do you stop short sometimes?
      singulars
      1. POReducing the number of table columns
      1. USMike Sherrill 'Cat Recall'
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload