StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>There are many things you can try after optimizing your Python code for speed. If this program doesn't need C extensions, you can run it under <a href="http://pypy.org/download.html" rel="nofollow noreferrer">PyPy</a> to benefit from its JIT compiler. You can try making a <a href="https://stackoverflow.com/questions/4616129/in-python-why-is-a-module-implemented-in-c-faster-than-a-pure-python-module-and/4619520#4619520">C extension</a> for possibly <a href="https://stackoverflow.com/questions/4650243/convert-python-program-to-c-c-code">huge speedups</a>. <a href="http://code.google.com/p/shedskin/" rel="nofollow noreferrer">Shed Skin</a> will even allow you to convert your Python program to a standalone C++ binary.</p> <p>I'm willing to time your program under these different optimization scenarios if you can provide enough code for benchmarking,</p> <p><strong>Edit</strong>: First of all, I have to agree with everyone else: are you sure you're measuring the time correctly? The example code runs 100 times in under 0.1 seconds here, so there is a good chance the either the time is wrong or you have a bottleneck (IO?) that isn't present in the code sample.</p> <p>That said, I made it 300000 people so times were consistent. Here's the adapted code, shared by CPython (2.5), PyPy and Shed Skin:</p> <pre><code>from time import time import random import sys class person(object): def __init__(self, util): self.utility = util self.customer = 0 class population(object): def __init__(self, numpeople, util): self.people = [] self.cus = [] self.noncus = [] for u in util: per = person(u) self.people.append(per) def f_w_append(popn): '''Function with append''' P = 75 cus = [] noncus = [] # Help CPython a bit # cus_append, noncus_append = cus.append, noncus.append for per in popn.people: if per.utility >= P: per.customer = 1 cus.append(per) else: per.customer = 0 noncus.append(per) return len(cus) def f_wo_append(popn): '''Function without append''' P = 75 for per in popn.people: if per.utility >= P: per.customer = 1 else: per.customer = 0 numcustomers = 0 for per in popn.people: if per.customer == 1: numcustomers += 1 return numcustomers def main(): try: numpeople = int(sys.argv[1]) except: numpeople = 300000 print "Running for %s people, 100 times." % numpeople begin = time() random.seed(1) # Help CPython a bit uniform = random.uniform util = [uniform(0.0, 300.0) for _ in xrange(numpeople)] # util = [random.uniform(0.0, 300.0) for _ in xrange(numpeople)] popn1 = population(numpeople, util) start = time() for _ in xrange(100): r = f_wo_append(popn1) print r print "Without append: %s" % (time() - start) popn2 = population(numpeople, util) start = time() for _ in xrange(100): r = f_w_append(popn2) print r print "With append: %s" % (time() - start) print "\n\nTotal time: %s" % (time() - begin) if __name__ == "__main__": main() </code></pre> <p>Running with PyPy is as simple as running with CPython, you just type 'pypy' instead of 'python'. For Shed Skin, you must convert to C++, compile and run:</p> <pre><code>shedskin -e makefaster.py && make # Check that you're using the makefaster.so file and run test python -c "import makefaster; print makefaster.__file__; makefaster.main()" </code></pre> <p>And here is the Cython-ized code:</p> <pre><code>from time import time import random import sys cdef class person: cdef readonly int utility cdef public int customer def __init__(self, util): self.utility = util self.customer = 0 class population(object): def __init__(self, numpeople, util): self.people = [] self.cus = [] self.noncus = [] for u in util: per = person(u) self.people.append(per) cdef int f_w_append(popn): '''Function with append''' cdef int P = 75 cdef person per cus = [] noncus = [] # Help CPython a bit # cus_append, noncus_append = cus.append, noncus.append for per in popn.people: if per.utility >= P: per.customer = 1 cus.append(per) else: per.customer = 0 noncus.append(per) cdef int lcus = len(cus) return lcus cdef int f_wo_append(popn): '''Function without append''' cdef int P = 75 cdef person per for per in popn.people: if per.utility >= P: per.customer = 1 else: per.customer = 0 cdef int numcustomers = 0 for per in popn.people: if per.customer == 1: numcustomers += 1 return numcustomers def main(): cdef int i, r, numpeople cdef double _0, _300 _0 = 0.0 _300 = 300.0 try: numpeople = int(sys.argv[1]) except: numpeople = 300000 print "Running for %s people, 100 times." % numpeople begin = time() random.seed(1) # Help CPython a bit uniform = random.uniform util = [uniform(_0, _300) for i in xrange(numpeople)] # util = [random.uniform(0.0, 300.0) for _ in xrange(numpeople)] popn1 = population(numpeople, util) start = time() for i in xrange(100): r = f_wo_append(popn1) print r print "Without append: %s" % (time() - start) popn2 = population(numpeople, util) start = time() for i in xrange(100): r = f_w_append(popn2) print r print "With append: %s" % (time() - start) print "\n\nTotal time: %s" % (time() - begin) if __name__ == "__main__": main() </code></pre> <p>For building it, it's nice to have a setup.py like this one:</p> <pre><code>from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext ext_modules = [Extension("cymakefaster", ["makefaster.pyx"])] setup( name = 'Python code to speed up', cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules ) </code></pre> <p>You build it with: python setupfaster.py build_ext --inplace</p> <p>Then test: python -c "import cymakefaster; print cymakefaster.<strong>file</strong>; cymakefaster.main()" </p> <p>Timings were run five times for each version, with Cython being the fastest and easiest of the code generators to use (Shed Skin aims to be simpler, but cryptic error messages and implicit static typing made it harder here). As for best value, PyPy gives impressive speedup in the counter version with no code changes. </p> <pre><code>#Results (time in seconds for 30000 people, 100 calls for each function): Mean Min Times CPython 2.5.2 Without append: 35.037 34.518 35.124, 36.363, 34.518, 34.620, 34.559 With append: 29.251 29.126 29.339, 29.257, 29.259, 29.126, 29.272 Total time: 69.288 68.739 69.519, 70.614, 68.746, 68.739, 68.823 PyPy 1.4.1 Without append: 2.672 2.655 2.655, 2.670, 2.676, 2.690, 2.668 With append: 13.030 12.672 12.680, 12.725, 14.319, 12.755, 12.672 Total time: 16.551 16.194 16.196, 16.229, 17.840, 16.295, 16.194 Shed Skin 0.7 (gcc -O2) Without append: 1.601 1.599 1.599, 1.605, 1.600, 1.602, 1.599 With append: 3.811 3.786 3.839, 3.795, 3.798, 3.786, 3.839 Total time: 5.704 5.677 5.715, 5.705, 5.699, 5.677, 5.726 Cython 0.14 (gcc -O2) Without append: 1.692 1.673 1.673, 1.710, 1.678, 1.688, 1.711 With append: 3.087 3.067 3.079, 3.080, 3.119, 3.090, 3.067 Total time: 5.565 5.561 5.562, 5.561, 5.567, 5.562, 5.572 </code></pre> <p><strong>Edit</strong>: Aaaand more meaningful timings, for 80000 calls with 300 people each:</p> <pre><code>Results (time in seconds for 300 people, 80000 calls for each function): Mean Min Times CPython 2.5.2 Without append: 27.790 25.827 25.827, 27.315, 27.985, 28.211, 29.612 With append: 26.449 24.721 24.721, 27.017, 27.653, 25.576, 27.277 Total time: 54.243 50.550 50.550, 54.334, 55.652, 53.789, 56.892 Cython 0.14 (gcc -O2) Without append: 1.819 1.760 1.760, 1.794, 1.843, 1.827, 1.871 With append: 2.089 2.063 2.100, 2.063, 2.098, 2.104, 2.078 Total time: 3.910 3.859 3.865, 3.859, 3.944, 3.934, 3.951 PyPy 1.4.1 Without append: 0.889 0.887 0.894, 0.888, 0.890, 0.888, 0.887 With append: 1.671 1.665 1.665, 1.666, 1.671, 1.673, 1.681 Total time: 2.561 2.555 2.560, 2.555, 2.561, 2.561, 2.569 Shed Skin 0.7 (g++ -O2) Without append: 0.310 0.301 0.301, 0.308, 0.317, 0.320, 0.303 With append: 1.712 1.690 1.733, 1.700, 1.735, 1.690, 1.702 Total time: 2.027 2.008 2.035, 2.008, 2.052, 2.011, 2.029 </code></pre> <p>Shed Skin becomes fastest, PyPy surpasses Cython. All three speed things up a lot compared to CPython.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload