Databases
StackOverflow2013
Postsdbo
POIs make -j distcc possible to scale over 5 times?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs make -j distcc possible to scale over 5 times?
    text
    copied!<p>Since distcc cannot keep states and just possible to send jobs and headers and let those servers to use only the data just sent and preprocess and compile, I think the lastest distcc has problem in scalability.<br> In my local build environment which has appx. 10,000 c/c++ files to build, I could only make 2 times faster than not using distcc (but using make -j) when having 20 build servers.<br> What do you think is the problem? </p> <p>If anyone has achieved scalability more than 10 - 20 times using make -j and distcc, please let me know.</p> <p>The following product claims that it is impossible to scale make -j and distcc faster than 5 times. <a href="http://www.electric-cloud.com/products/electricaccelerator.php" rel="nofollow noreferrer">http://www.electric-cloud.com/products/electricaccelerator.php</a></p> <p>I think this can be improved by: </p> <ul> <li>Letting the distccd server to maintain sessions</li> <li>Tied to those sessions, they will cache their own header directories</li> <li>Preprocess will be done demand base from the distccd server</li> <li>This will be done through a LD_PRELOADed library libdistcc.so which will replace stat/open syscalls and fetches the header files over network. ...</li> </ul> <p>Has anyone done this kind of thing?</p> <hr> <p>I think Electric Cloud does similar stuff but I think we have more space to optimize: </p> <ol> <li>Servers should share the same source code repository over really fast network filesystem.</li> <li><p>We should make the build file parsing and include header parsing in parallel. </p></li> <li><p>seems to be quite tough without changing the build descriptions drastically.<br> Any ideas, prior art, workaround is welcome.</p></li> </ol>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload