Databases
StackOverflow2013
Postsdbo
POSharding in mongodb with c# for getting the records as required
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSharding in mongodb with c# for getting the records as required
    primarykey
    data
    text
    <p>I have 2 millions of data in some csv files... I am getting those files and reducing it to get result sumthing like this.,..</p> <pre><code>{ "_id" : "08-08-2012 05:00", "value" : { "CollectionDate" : "08-08-2012 05:00", "count" : 20000 } } { "_id" : "08-08-2012 05:15", "value" : { "CollectionDate" : "08-08-2012 05:15", "count" : 20000 } } { "_id" : "08-08-2012 05:30", "value" : { "CollectionDate" : "08-08-2012 05:30", "count" : 20000 } } { "_id" : "08-08-2012 05:45", "value" : { "CollectionDate" : "08-08-2012 05:45", "count" : 20000 } } { "_id" : "08-08-2012 06:00", "value" : { "CollectionDate" : "08-08-2012 06:00", "count" : 20001 } } { "_id" : "08-08-2012 06:15", "value" : { "CollectionDate" : "08-08-2012 06:15", "count" : 20000 } } { "_id" : "08-08-2012 06:30", "value" : { "CollectionDate" : "08-08-2012 06:30", "count" : 20000 } } .... { "_id" : "08-08-2012 10:30", "value" : { "CollectionDate" : "08-08-2012 06:30", "count" : 20000 } } { "_id" : "08-08-2012 10:45", "value" : { "CollectionDate" : "08-08-2012 06:30", "count" : 20000 } } { "_id" : "08-08-2012 11:00", "value" : { "CollectionDate" : "08-08-2012 06:30", "count" : 20000 } } </code></pre> <p>Source code for getting this result is as:</p> <pre><code>class DirectImport { private static bool forOnce = true; static void Main(string[] args) { Stopwatch stopwatch = new Stopwatch(); bool reset = true; BsonString lastAggregatedDate = ""; String lastCollName = ""; MongoServer mongo = MongoServer.Create(); mongo.Connect(); Console.WriteLine("Connected"); Console.WriteLine(); var db = mongo.GetDatabase("SampleData"); using (mongo.RequestStart(db)) { List&lt;BsonDocument&gt; mDoc = new List&lt;BsonDocument&gt;(); IEnumerable&lt;String&gt; CollectionList = Directory.EnumerateFiles(@"c:\mongoDb\Final"); DateTime oldDt = new DateTime(1, 1, 1); foreach (String coll in CollectionList.ToArray&lt;String&gt;()) { String collName = coll.Substring(coll.LastIndexOf(@"\") + 1); collName = collName.Substring(0, collName.LastIndexOf('.')); //Console.WriteLine(collName); string[] records = File.ReadAllLines(@"C:\mongoDB\Final\" + collName + ".csv"); mDoc = new List&lt;BsonDocument&gt;(); var newCollection = db.GetCollection&lt;BsonDocument&gt;(collName); for (int l = 1; l &lt; records.Length; l++) { string[] abc = records[l].Split(','); //Console.WriteLine(abc[1] + "_" + abc[2] + "_" + abc[5]); BsonDocument book1 = new BsonDocument(); book1.Add("_id", BsonString.Create(abc[1] + "_" + abc[2] + "_" + abc[5])); book1.Add("Base_Name", BsonString.Create(abc[0])); book1.Add("ObjectType", BsonString.Create(abc[1])); book1.Add("C_ID", BsonString.Create(abc[2])); book1.Add("AssociateDimension1", BsonString.Create(abc[3])); book1.Add("AssociateDimension2", BsonInt32.Create(abc[4])); book1.Add("CollectionDateHour", BsonInt32.Create(Convert.ToDateTime(BsonString.Create(abc[5])).Hour)); book1.Add("CollectionDate", BsonString.Create(abc[5])); book1.Add("Granularity", BsonInt32.Create(abc[6])); book1.Add("Counter1_1", BsonDouble.Create(abc[7])); book1.Add("Counter1_2", BsonDouble.Create(abc[8])); book1.Add("Counter1_3", BsonDouble.Create(abc[9])); book1.Add("Counter1_4", BsonDouble.Create(abc[10])); book1.Add("Counter1_5", BsonDouble.Create(abc[11])); book1.Add("Counter1_6", BsonDouble.Create(abc[12])); book1.Add("Counter1_7", BsonDouble.Create(abc[13])); if (forOnce) { forOnce = false; oldDt = Convert.ToDateTime(BsonString.Create(abc[5])); lastCollName = collName; } DateTime dt = Convert.ToDateTime(BsonString.Create(abc[5])); mDoc.Add(book1); newCollection.Insert(book1); String[] strparam = { "CollectionDate" }; if (!newCollection.IndexExists(strparam)) newCollection.CreateIndex(strparam); if (oldDt &gt; dt) Console.WriteLine(dt + ":::" + oldDt); if (((dt - oldDt).Hours &gt;= 1) || oldDt &gt; dt) { stopwatch.Start(); Console.WriteLine("Calling MAPReduce at: " + oldDt); Console.WriteLine(); //var coll1 = db.GetCollection&lt;BsonDocument&gt;("MSS1"); //coll1.InsertBatch&lt;BsonDocument&gt;(mDoc); MapReduce(db, lastCollName, dt, lastAggregatedDate, reset); lastCollName = collName; //coll1.Drop(); lastAggregatedDate = BsonString.Create(abc[5]); reset = false; mDoc = new List&lt;BsonDocument&gt;(); oldDt = dt; TimeSpan ts = stopwatch.Elapsed; Console.WriteLine("RunTime: " + ts.Hours + ":" + ts.Minutes + ":" + ts.Seconds + ":" + ts.Milliseconds / 10); stopwatch.Stop(); } } } } Console.Read(); mongo.Disconnect(); } private static void MapReduce(MongoDatabase db, String collName, BsonValue bsonValue, BsonString lastAggregateDate, bool reset) { var collection = db.GetCollection&lt;BsonDocument&gt;(collName); Console.WriteLine(collName); String map = @"function() { emit(this.CollectionDate, {CollectionDate : this.CollectionDate, count: 1}); }"; String reduce = @"function(key, values) { var result = {CollectionDate:key, count: 0}; values.forEach(function(value){ result.count += value.count; }); return result; }"; var options = new MapReduceOptionsBuilder(); IMongoQuery[] queries = { Query.GTE("CollectionDate", lastAggregateDate) }; //if(reset) // queries = new IMongoQuery[] { Query.LT("CollectionDate", bsonValue) }; options.SetOutput(MapReduceOutput.Reduce("MSS_REDUCE")); //options.SetOutput(MapReduceOutput.Reduce); IMongoQuery query = Query.And(queries); var results = collection.MapReduce(queries[0], map, reduce, options); } </code></pre> <p>Now the problem is that my required output is : </p> <pre><code>{ "_id" : "08-08-2012 05:00", "value" : { "CollectionDate" : "08-08-2012 05:00", "count" : 20000 } } { "_id" : "08-08-2012 06:00", "value" : { "collectionDate" : "08-08-2012 06:00", "count" : 80000 } } { "_id" : "08-08-2012 07:00", "value" : { "collectionDate" : "08-08-2012 07:00", "count" : 80000 } } { "_id" : "08-08-2012 08:00", "value" : { "CollectionDate" : "08-08-2012 07:45", "count" : 80000 } } { "_id" : "08-08-2012 09:00", "value" : { "collectionDate" : "08-08-2012 09:00", "count" : 80000 } } { "_id" : "08-08-2012 10:00", "value" : { "collectionDate" : "08-08-2012 10:00", "count" : 80000 } } { "_id" : "08-08-2012 11:00", "value" : { "CollectionDate" : "08-08-2012 10:45", "count" : 80000 } } </code></pre> <p>Now I think it can be done by sharding on _id key.. but how to do this in c# and how to achieve this result.. Please help me</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USRavi Sharma
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSharding in mongodb with c# for getting the records as required
      1. USRavi Sharma
      1. VTFavorite
    1. COAnd if its not sharding.. then also tell me the way through which I can get the same output as I need.. Thanks in advance..
      singulars
      1. POSharding in mongodb with c# for getting the records as required
      1. USRavi Sharma
    2. COwhat version of mongodb are you running?
      singulars
      1. POSharding in mongodb with c# for getting the records as required
      1. USAsya Kamsky
    3. COI'm not sure I can see what you are asking, but I think it is not to do with sharding. Are you trying to get the mapreduce to give the results given in your final example? If so, what is the distinction between `5:00` and `7:00` such that the `5:00` group does not have a `5:45` while the `7:00` group does?
      singulars
      1. POSharding in mongodb with c# for getting the records as required
      1. USAndre de Frere
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload