StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POElastic Storm Topology / Storm-Hadoop Coexisting
text
Body
copied!<p>We are evaluating pursuing Storm for a deployment, but I am a little concerned. We currently run Hadoop MapReduce, and would want to transition some of our processing from MapReduce to Storm processes. Note that that is some, but not all. We would still have some MapReduce functionality.</p> <p>I had found Mesos, which could (potentially) allow for us to maintain a Storm and Hadoop deployment on the same hardware, but had a few other issues:</p> <ul> <li><p>I envision the ideal situation as being able to "borrow" slots between Storm and Hadoop arbitrarily. ex. both would use the same resources as needed. Unfortunately this is a fixed deployment, and isn't "cloud based" like EC2 or the such.</p></li> <li><p>I want to avoid bottlenecks in our Storm environment. An ideal case would be to "spin up" (or the inverse) more instances of Bolts as demand requires. Is this possible / realistic?</p></li> <li><p>"Restarting" a topology seems like a fairly expensive operation, and I'm not sure is really an option. Ideally, I would want it to be as seamless as possible.</p></li> </ul> <p>Are we approaching this problem correctly? Essentially, a Storm topology would "feed" a MapReduce batch job. Some of our processing can be processed in a streaming fashion, and would be much better as a Storm topology, while some of it requires batch processing.</p> <p>Any general feedback, even if it doesn't address my specific questions, would be welcome. This is more of an exploratory phase at this point, and I might be totally approaching this the wrong way.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload