Databases
StackOverflow2013
Postsdbo
POHow would you architect this application
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow would you architect this application
    primarykey
    data
    text
    <p>I come from a background of PHP OOP and LAMP setups. with some experience in server side java. </p> <p>I need to build a system to collects data from our internal sources and store them for querying, data analyzing etc (mysql DB) </p> <p>I'd like to for once create something that's architected after the larger online apps instead of one crawler script with multiple classes (think mulit-threaded taskers and queues here)</p> <p><strong>About the application:</strong> Data needs to be completed about entities that go through processes through out our system. It's a big mess. Some of the data about these entities are in log files on other servers, some in csv and some are querying directories full of XMLs that represent an entity. </p> <p>These need to collected, organized (e.g. entity 123ABC from server1 was processed by X with output message Y on <code>date</code>. Then failed on a different process on serverb with failure message Z</p> <p>Store and organize these in a datastore. </p> <p>Build a front-end app complete separate from the back-end that queries these data and helps operators track the entities (where they are and when they were processed by A,B,C servers)</p> <p><strong>My Thought Process</strong></p> <p>Since I've recently been fiddling around with Ruby and LOVE it, i'd like to build this in Ruby. So for the backend, I'd like to have Inserters or "Collecters" that are objects designed to run via different ways to go get the data. They are specialized for each of the processes. i.e. once is designed to read XMLs. one to read CSV, another to run a linux command on a directory to collect stats. They will be in a OOP hierarchy like so: Collector:XMLCollector:CatalogXMLCollector Collector:CVSCollector:VericusCollector </p> <p>I'd like these Collector objects to be run separately (different processes) so that one doesn't take down the others. Does ruby handle threading well? Quick googling tells me no.</p> <p>I'd imagine these Collectors will need to insert into DB, Should they do it directly? or insert into some kind of JSON Queue where a Reader listens on? </p> <p>Also. The Collectors are not blind crawlers, they do need to find UNIQUE data in some cases. They may need to know what they sent already and not resend the entire 200k line CSV again (useless extra processing of duplicates the DB will reject) </p> <p>The front end: I'll need to create custom views/reports (on a weekly bases) to gather analytic and stats. It would be awesome if the operators can build their own views by selecting a view type(table, text, graph->type), then select which data points they need and any filters/clauses on the data. Then run it once, save it for later on the dashboard to be run by others as well. Or even schedule it to be run automatically. </p> <p>This front-end is a beast by itself, but it's the part I am most confident developing and can do it progressively. </p> <p><strong>Help I Need</strong> critique the above. Am I making this too complicated on myself? I have basically no set deadline. Articles, tips you have for me? </p> <p>Thanks for reading my huge wall of text. </p> <p>edit:minor clarification</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USdlite922
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow would you architect this application
      1. This table or related slice is empty.
      1. VTDownMod
    2. VO
      singulars
      1. POHow would you architect this application
      1. This table or related slice is empty.
      1. VTDownMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload