StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>You may find an ETL (Extract-Transform-Load) tool will make your life easier than trying to solve your problem in code.:</p> <ul> <li><a href="http://msdn.microsoft.com/en-us/library/ms141026.aspx" rel="nofollow noreferrer">SSIS</a> (SQL Server Integration Services)</li> <li><a href="http://www.oracle.com/webapps/dialogue/dlgpage.jsp?p_ext=Y&p_dlg_id=6853205&src=6652651&Act=5&sckw=NAMK08051462MPP001.GCM.8316.100" rel="nofollow noreferrer">ODI</a> (Oracle Data Integrator)</li> <li><a href="http://vip.informatica.com/?elqPURLPage=1032&gclid=CIzH8oXR2ZgCFQG7GgodwX8tcw" rel="nofollow noreferrer">Informatica PowerCenter</a></li> <li><a href="http://www.google.com/search?hl=en&q=etl+tools&btnG=Search" rel="nofollow noreferrer">Many others</a></li> </ul> <p>These are designed specifically for the type of data loading work that you described.</p> <p><strong>EDIT</strong></p> <p>While I still maintain that an ETL tool will best serve your needs, if you insist on doing it in code, you should think about implementing ETL as a pattern. The reason for this is that ETL is a well established best practice for loading data from various sources. You should take some time to study how ETL is implemented.</p> <p>At a basic level, you should have three layers, an extraction layer, a transformation layer, and a loading layer.</p> <p>The <strong>extraction</strong> layer should be responsible for retrieving the data from the source. It should not worry about the shape of the data at this points. To keep the layer clean, you should only implement code that "gets" the data here. Worry about shaping it in the transformation layer.</p> <p>The <strong>transformation</strong> layer should be responsible for taking data extracted from various sources and transforming it to the destination's shape. ETL tools do this very efficiently by treating the data as pipelines. These can be split and parallelized. You probably won't have the time or resources to do this. An alternative may be to load the data into staging tables (a less normalized representation of the data).</p> <p>The <strong>load</strong> layer takes the transformed data (in the above case from the staging tables) and loads them in to the final destination location.</p> <p>This sufficiently separates your layers so that you can protect yourself from future change. Keep in mind, however, that you're really just doing what an ETL tool will do for you out of the box.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload