<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>(With apologies to those who just want to search linguistic corpora and aren't interested in the lowlevel stuff)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="color:rgb(0,0,0);font-size:12.8px">Sure, but it's unlikely that we're ever going to solve the "stochastic neoclassical growth model, the workhorse of modern macroeconomics" with CQP.</span></blockquote><div>I'm not sure summing up numbers is a typical task either. Unfortunately there's no R entry on the</div><div>benchmarks game, but for most nontrivial workloads, Python (even Py3) and Perl are roughly equal:</div><div><a href="http://benchmarksgame.alioth.debian.org/u64/perl.php">http://benchmarksgame.alioth.debian.org/u64/perl.php</a><br></div><div><br></div><div>As I said, the most compelling argument for Python wouldn't be that it's faster than the others</div><div>(because Python and R users mostly rely on the speed of underlying C modules that do all the work),</div><div>but that it's (along with Java) taught relatively widely.</div><div><br></div><div>But let's look at what people would do for a simple count look of the form</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> for x in xrange(N+1):<br>
sum = sum + x<br></blockquote><div>if they were using numpy:</div><div>--- 8< ---<br></div><div>from numpy import arange</div><div><br></div><div>a=arange(N+1)</div><div>sum = a.sum()</div><div>--- 8< ---</div><div>This takes 0.08 seconds when run from the command line for 20M numbers, and 0.21 seconds for 100M numbers.</div><div><br></div><div>--- 8< ---</div><div>from future import __print_function__</div><div><div>import cython</div><div><br></div><div>def test_fun(N):</div><div> return cython.inline(</div><div> '''</div><div> cdef int s, x, n</div><div> s=0; n=N+1</div><div> for x from 0<=x<n:</div><div> s+=x</div><div> return s</div><div> ''')</div><div><br></div><div>print(test_fun(20000000))</div></div><div>---8< ---</div><div>takes 0.39 seconds, including running the Cython compiler and a call to gcc to compile the inline code.</div><div>100M iterations (after cleaning out Cython.inline's cache) take 0.45 seconds, 0.235 seconds if the</div><div>compiled code is in cache.</div><div><br></div><div>Numba is even simpler:</div><div>--- 8< ---</div><div>from numba import autojit</div><div><br></div><div>@autojit</div><div>def test_fun(N):</div><div> s=0</div><div> for x in xrange(N+1):</div><div> s+=x</div><div> return s</div><div>--- 8< ---</div><div>which takes 0.23 seconds right away for either 20M or 100M additions. (I.e. compilation is much faster</div><div>because it can use LLVM instead of gcc).</div><div><br></div><div>Leaving out the numba bit and just using Python 2.7., I get 0.62 seconds for 20M and 2.9 seconds</div><div>for 100M, so my notebook may be faster than the computer you tried this on.</div><div>Perl code using a foreach loop uses 1.0 seconds for 20M and 4.7 seconds for 100M, so it looks like the overall speed may be different.</div><div>If I take the code out of the function, Python makes the variables global, which leads to them being slower, and we end up with results analogous to what you had in your list - 1.8 seconds for 20M and 8.98 sec. for 100M.</div><div><br></div><div>This gives, for 100M integers:</div><div><font face="monospace, monospace">bad Python code: 8.98 sec., 11 Mops/sec</font></div><div><font face="monospace, monospace">Perl code: 4.7 sec., 21 Mops/sec</font></div><div><font face="monospace, monospace">normal Python code: 2.9 sec., 34 Mops/sec</font></div><div><font face="monospace, monospace">Cython inline: 0.45 sec., 222 Mops/sec</font></div><div><font face="monospace, monospace">Numba autojit: 0.23 sec., 434 Mops/sec</font></div><div><font face="monospace, monospace">Numpy sum: 0.21 sec., 476 Mops/sec</font></div><div><br></div><div>There's a bit of setup involved with Numba -- unless you install Anaconda (a Python distribution from Continuum.io that works for Windows, Mac and Linux and uses its own binary packages, which makes it newbie-friendly on Windows but slightly awkward on Linux), you'd have to install LLVM and Numba, but the results are definitely nice.</div><div><br></div><div>BTW, a bit of random googling turned up this neat C library for fast (de)compression of</div><div>integer blocks:</div><div><a href="https://github.com/lemire/simdcomp">https://github.com/lemire/simdcomp</a></div><div>Daniel Lemire has also a C++ library for intersecting compressed integer lists</div><div><a href="https://github.com/lemire/SIMDCompressionAndIntersection">https://github.com/lemire/SIMDCompressionAndIntersection</a><br></div><div><br></div><div>Cheers,</div><div>Yannick</div></div></div></div>