Databases
StackOverflow2013
Postsdbo
POOptimizing a denormalization of a csv file
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOptimizing a denormalization of a csv file
    primarykey
    data
    text
    <p>I am reading in a csv file to be stored in an immutable data structure. Each row is an entrance. Each entrance has one station. Each station can have multiple entrances. Is there a way I can do this in a single pass instead of the double pass you see below?</p> <pre><code>object NYCSubwayEntrances { def main(args: Array[String]) = { import com.github.tototoshi.csv.CSVReader //http://www.mta.info/developers/data/nyct/subway/StationEntrances.csv val file = new java.io.File("StationEntrances.csv") val reader = CSVReader.open(file) reader.readNext //consume headers val entranceMap = list2multimap( reader.all map { case fields: List[String] =&gt; // println(fields) ( fields(2), Entrance( fields(14).toBoolean, Option(fields(15)), fields(16).toBoolean, fields(17), fields(18) match {case "YES" =&gt; true case _ =&gt; false}, fields(19) match {case "YES" =&gt; true case _ =&gt; false}, fields(20), fields(21), fields(22), fields(23), fields(24).toInt, fields(25).toInt ) ) } ) reader.close val reader2 = CSVReader.open(file) reader2.readNext //consume headers val stations = reader2.all map { case fields: List[String] =&gt; Station( fields(2), fields(0), fields(1), colate(scala.collection.immutable.ListSet[String]( fields(3), fields(4), fields(5), fields(6), fields(7), fields(8), fields(9), fields(10), fields(11), fields(12), fields(13) )), entranceMap(fields(2)).toList ) } reader2.close import net.liftweb.json._ import net.liftweb.json.Serialization.write implicit val formats = Serialization.formats(NoTypeHints) println(pretty(render(parse(write(stations.toSet))))) } import scala.collection.mutable.{HashMap, Set, MultiMap} def list2multimap[A, B](list: List[(A, B)]) = list.foldLeft(new HashMap[A, Set[B]] with MultiMap[A, B]){(acc, pair) =&gt; acc.addBinding(pair._1, pair._2)} def colate(set: scala.collection.immutable.ListSet[String]): List[String] = ((List[String]() ++ set) diff List("")).reverse } case class Station(name: String, division: String, line: String, routes: List[String], entrances: List[Entrance]) {} case class Entrance(ada: Boolean, adaNotes: Option[String], freeCrossover: Boolean, entranceType: String, entry: Boolean, exitOnly: Boolean, entranceStaffing: String, northSouthStreet: String, eastWestStreet: String, corner: String, latitude: Integer, longitude: Integer) {} </code></pre> <p>An sbt project with all the correct dependencies can be found at <a href="https://github.com/AEtherSurfer/NYCSubwayEntrances" rel="nofollow">https://github.com/AEtherSurfer/NYCSubwayEntrances</a></p> <p>StationEntrances.csv was obtained from <a href="http://www.mta.info/developers/sbwy_entrance.html" rel="nofollow">http://www.mta.info/developers/sbwy_entrance.html</a></p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USGabriel
    1. USGabriel
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POOptimizing a denormalization of a csv file
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThe input file seems to be sorted by `Division, Line, Station_Name`. Is that an assumption you are willing to rely on? It could speed up processing and also lessen memory requirement (although the file doesn't seem very big).
      singulars
      1. POOptimizing a denormalization of a csv file
      1. UShuynhjl
    2. COYes, I am willing to assume that the csv is sorted by `Division, Line, Station_Name`.
      singulars
      1. POOptimizing a denormalization of a csv file
      1. USGabriel
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload