What is Reddit's opinion of PyPy?

From 3.5 billion Reddit comments

String concatenation is cheap in CPython, but not in PyPy or possibly other implementations. There it's quadratic because the JIT can't optimize it.

.join() is more portably performant.

Head down to "String concatenation is expensive."

http://pypy.org/performance.html#micro-tuning-tips

Looks cool. Tried it out on Mac OS Sierra. Didn't need to do anything about the 32-bit, so that was easy :)

Have you taken a look at Pypy? It could potentially increase the performance dramatically.

I tried starting it with Pypy, but no window came up. Mouse got captured, and no exceptions. So it's probably some minor detail, as Pyglet should otherwise work http://pypy.org/compat.html

I hate Javascript as much as the next grey haired nerd, but this is a little over the top. To me, as an old schooler, ES6 looks fine. And I'm fairly sure it's the fastest dynamic language. PyPi is really good, btw.

On the other side of the coin, 'modern' C++ (C++14, by general consensus) is really very different to what we had in the 90's. I think you can write without ever using pointer arithmetic, but don't quote me on it.

There's that, and there's webassembly which isn't great but it's better than what we have. If there's moderately efficient interop with OpenGL then it starts to ... well, you could deliver macOS over it. Or a Playstation.

So I think it's actually looking up for front end right now.

> I just never realized that it is compiled at runtime. I guess this is true for all interpreted languages as well.

Actually, it isn't. The byte code is the product of parsing the code and breaking it down to instructions that will be interpreted at run-time. Each byte code will cause the interpreter to call another function in a C library, which itself is actually fully compiled to machine code.

When byte code is "compiled at runtime", as you put it, this is called "JIT" or "Just In Time" compilation. Normally, Python byte code does not undergo JIT. It's interpreted as I described before.

Other languages, like Java and C#, do undergo JIT. They are actually compiled down to machine code for the current machine architecture, and then the raw machine code is executed.

Now, to confuse this just a little - Python CAN execute with JIT. If you use PyPy to execute Python, it will use JIT on the byte code and then execute the machine code. "Normal Python" uses CPython though, which is a bytecode interpreter. This is the Python everyone uses by default as that is the one that is available on python.org.

• Python can be compiled.
• Python can be made into JVM bytecode.
• Python can be compiled Just-in-Time.

A programming language is never inherently interpreted or compiled or whatever. It's rather more accurate to say that the standard way of executing a lanaguage is such-and-such. And indeed, CPython, the reference standard implementation of Python, is interpreted.

Note that PyPy is still looking for donations, for Python3 http://pypy.org/py3donate.html, NumPy and general pypy progress

Note that when you are looking at http://pypy.org/py3donate.html and would like to donate, the "Donate towards NumPy in pypy" link in upper left might still be selected. Chose for what you primarily want to donate there (unused founds can be reassigned on 1 March 2012).

A Python process only makes use of a single CPU core, so in this era when low-end laptops (and even phones) have multiple cores, there are applications for which that's kind of annoying.

Writing correct programs that handle parallel execution is a hard topic in any language, but some approaches just haven't been doable in Python at all.

(You can run multiple python processes that run at the same time and talk to each other, but that's not quite the same thing.)

Pypy is doing some very interesting research in this area, but that branch of pypy is still very much in research mode. Promising but not yet usable.

I'm guessing this might have something to do with it:

> Really short-running scripts: A rule of thumb is if something runs below 0.2s the JIT has no chance, but it depends a lot on the program in question. In general, make sure you warm up your program before running benchmarks, if you're measuring something long-running like a server. The time required to warm up the JIT varies; give it at least a couple of seconds. (PyPy's JIT takes an especially long time to warm up.)

From here.

their main focus is py2 as probably most of the users work in enterprise

on the plus side:

> Good news is that it is not specifically targeted for the NumPy library and the PyPy virtual machine. Any interpreter (written in RPython) is able make use of the vectorization.

so this should be fairly easy to push into the py3 effort (and also into HippyVM!)

http://pypy.org/py3donate.html

unrelated: it would be nice to see this subreddit host a monthly/bimonthly donations drive to several fairly significant projects/modules (PSF, PyPy, Requests, Jython, IronPython if it still exists, etc)

> PyPy is going to be the shit when numpy gets ported to it.

http://pypy.org/download.html#installing-numpy

I have to say: there's quite a bit of fluff to what you said. I believe you that it speeds up your scripts a lot, but I doubt that it's anywhere near V8 performance yet. V8 is pretty magical, and it's been optimized for years by paid programmers. PyPy is very promising, but it still has a long way to go before we can really compare it with V8. That's not to say that it won't reach that point either! The PyPy contributors made a lot of progress, and I don't doubt that they're going to continue.

Great question. STM will actually slow down a serial program quite a bit. The main reason STM is good is that you could do multi threading without using the multithreading library and it's associated GIL, and get speed ups from that. I encourage you to read http://pypy.org/tmdonate.html for the arguments, as it explains it very rigorously.

This is cool, but seems to have the ~~same~~ downside ~~as pypy by~~ of being a limited subset of the language which will prevent most existing legacy modules from working (e.g., scipy, etc). (Granted pypy has been improving on that).

I also find it very weird that as a new project it targets python2.7 versus python3. (Yes, most everyone uses python2 these days, but python3 is the future and has been for years).

But good to see Dropbox is utilizing BDFL for more than just in-house stuff.

EDIT: Struck out a few words above. pypy fully supports code written in python that doesn't need to access the C API. Just only supports the python C API at alpha/beta levels.

That's not entirely correct.

Python also does compile it's code into bytecode, so it's not exactly interpreted, but rather executed on a VM, much like Java.

The main reasons Python is slow are:

• Lazy evaluation, so expensive_function_always_returning_zero() will get executed every single time, never get optimized away as constants

• Dynamic, duck typing - it has to check types of variables every time you compare them, and functions are just another members - they have to be checked for in the object/class/module dictionary, which is also slow as hell

• Another reason is that CPython simply does not do any serious optimizations, if you want fast Python, take a look at PyPy. It's not as fast as C, at least not yet, but it comes close in many situations, including loops like the one in question.

As for your question OP: Content of a loop is evaluated every iteration, this includes memory allocation for variables, and by doing these simple checks with if, you're suddenly reducing amount of numbers to check by nearly half. Allocating f and r before the while loop would probably bring the difference down.

In C++ it doesn't make much difference, because the compiler can and will do some crazy optimizations regarding loops.

And finally: fact that Python is interpreted does not matter - GNU Octave is interpreted and can be many times faster at numerical calculations than C++ simply because that's what it's made to do. Python is not a language aimed at performance, while C++ is supposed be as fast as they can make it

This might not be relevant, but have you tried running your game under PyPy? I don't know if all of the libraries you're using are compatible, but just the first thing that popped into my mind when I saw the title.

CPython is the name of the default implementation of Python. It is the one you get on python.org. There are also other implementations such as PyPy.

Thanks, I went to

• PyPy Status Blog where OP's link goes http://morepypy.blogspot.com/2015/10/pypy-400-released-jit-with-simd.html
• Welcome to PyPy http://pypy.org/
• What is Pypy http://pypy.org/features.html

None of them said that. So I came here, and I thank you for the explanation!

http://pypy.org/tmdonate2.html says:

"OS/X (and likely the various BSDs) seems to handle mmap() better than Linux does, and can remap individual pages of an existing mapping to various pages without hitting a limit of 65536 like Linux."

I am curious what this limit is referring to. Is this vm.max_map_count? While Linux's default is 65536, there is nearly no ill effect in increasing this to, say, a million. Some softwares like DUMA explicitly recommend users to do so.

I'm sure there are a plethora of things that can be done, but one use case that I'll enjoy will be using PyPy on 64-bit mode on Windows. Right now there are only x86 binaries available since it seems like there's a few quirks for the x64 version.

You probably meant PyPI : the Python Package Index (https://pypi.python.org/pypi)

PyPy (http://pypy.org/) is an alternative Python interpreter which (can) contains a JIT and is usually seen as very fast.

Whenever someone talks about Python and scientific computation, the first advice is always "You should check out NumPy and SciPy!", so I'll just say that: you should check out NumPy and SciPy! NumPy especially will help out a lot when it comes to things like matrix operations, which may not be relevant to you now, but is good to keep in mind.

If speed is of paramount important, I'd also check out PyPy, which is a Python interpreter written in Python that includes a pretty great JIT compiler. For CPU-bound applications, the speed-up can be pretty incredible, sometimes magnitudes faster.

There is also Shedskin. It compiles a subset of Python.

The problem with a Python compiler is that if you want full Python parity, you don't get a fraction of the speedups that you might expect from a compiler, because almost everything has to be looked up at runtime in the general case.

For example:

class Foo(object): def str(self): return "Hello"

class Bar(object): pass

f = Foo() print str(f)

f.class = Bar print str(f)

Bar.str = lambda self: "Goodbye" print str(f)

str = id print str(f)

4 identical lines have different output here because everything can change at runtime. For str(f) to work out which code to call, it has to do the __class__ lookup, and then lookup __str__ in the class dictionary, and its superclasses etc. And before any of that, even str itself has to be looked up in the current namespace, as demonstrated by the last one.

This applies to everything - "a + b" has to do lookups at runtime to work out what "+" implementation should be called.

Many other things about Python (e.g. **kwargs etc.) require much heavier runtime implementations of things as simple as a function call, compared to something like C, which was designed to be map onto simple and fast machine code.

So, for the general case, a JIT compiler works better, because it can take advantage of the fact that normally you are not doing the kind of tricks I've done above, and very often some code that does thousands of additions, for example, will end up calling exactly the same "+" implementation. And this is what you have in PyPy.

From the performance page: > Abuse of itertools: The itertools module is often “abused” in the sense that it is used for the wrong purposes. From our point of view, itertools is great if you have iterations over millions of items, but not for most other cases. It gives you 3 lines in functional style that replace 10 lines of Python loops (longer but arguably much easier to read). The pure Python version is generally not slower even on CPython, and on PyPy it allows the JIT to work much better – simple Python code is fast. The same argument also applies to filter(), reduce(), and to some extend map() (although the simple case is JITted), and to all usages of the operator module we can think of.

If you find yourself considering cython to make something python-like but faster, give pypy a shot. It's a full Python implementation, not a c/python pidgin. There are existing python/C wrappers it does not perform terribly well with but it's worth investigating.

In some cases running pypy and using cffi for your C integration will be a good option, as in this recent case study with ijson.

JavaScript has a similar conceptual model, but modern implementations do a lot of optimizations to avoid using hash tables. For example, V8 uses hidden classes to generate struct/object like code whenever possible. Working with a collection of Javascript objects will eventually compile to index math with fixed offsets. A list of Python objects will always be a list of dictionaries and incur a hash for each property access.

As such, CPython's speed depends more heavily on the hash table implementation, whereas JS engines depend more on the JIT's ability to elide property hashing. That's not to say that JS engines have bad hash table implementations (or that Python can't be JIT compiled), just that it's less critical. Plus, Javascript doesn't provide locals(), so local variables don't have to be looked up by name.

> I've never got the impression "Haskell people" are any more averse to newlines than any other programmers - if anything, most of the Haskell code I've seen has shorter lines than that in other languages.

It's not about length. Consider this line:

stroke /= [] ==> validbox box ==> refit box stroke ~~ refit box (refit box stroke)

It's not much longer than a typical Java line, but it's got something like 8 function calls in it. In another language, it would be extremely rare to do so much without splitting it into several pieces. It's not like Haskell doesn't have good ways to let you do that. In your example you have

Just (p, ns') -> S.insert p $ go (S.filter (\n -> n mod p /= 0) ns')

This, again, might not be long, but it's far more complexity than I'll tend to see in another language in a single line. There are something like 9 operations just on that line.

> I could break out the mutable types and possibly catch up that way, but then I'm not really writing idiomatic Haskell any more.

Personally I don't think it's really a big deal; very few things have similar performance problems from disallowing mutation and aren't there safe ways (IO) to do mutation anyway?

> I'm not that familiar with Python - how widespread is use of numpy in practice?

It depends on the field. In "scientific Python" it's extremely common - you'll probably be laughed at if you don't use it. It's also used whenever you deal with data like images or often even when mmaping. It's a pretty standard tool.

Even so, I only really used it so I could do

numbers[::i] = 0

instead of

for j in range(0, n, i): numbers[j] = 0

This would be rather slow in CPython, but that's what PyPy is for.

The main Python (CPython) is an interpreter written in C, and for high volumes of simple operations, like loop/math intensive code, interpreters are a lot slower than compiled code. For many larger operations like string manipulation (where the interpreted code is calling into compiled library code which does most of the work), the difference is much smaller.

For IO intensive apps you might not notice a difference at all. Google's web crawler was mostly Python for example, in the late 90's when speed was a bigger factor than today. And they had a bigger search index than anyone else.

There is a just in time compiled Python called PyPy you could try, if you have a program where interpreter speed is the limiting factor.

My experience:

• Stick with Python. You already know it so you'll be far more productive. A Python version might be fast enough, especially if you write plug-ins for the slow bits in C.

• You may get more speed by using PyPy, or you may not if your platform is not x86.

• To make it go really fast, try multi-threading, using MPI, or using OpenCL (on your GPU). OpenCL is basically a whole new language and will take ages to get working.

• Learn about Python's GIL before trying to write multi-threaded code in Python. You may get more milage by going straight to MPI and skipping the multi-threaded bit.

The languages:

• C - small language spec (so it's easier to learn and fully understand), laborious to write code in, lots of help on Internet.

• C++ - The language spec is thousands of pages long and has features that will make your eyes bleed. Code is still laborious to write despite the massive spec. C++ has features to make larger applications easier to maintain than a C version. Lots of help on the Internet.

• Fortran - Only scientists who write bad code use Fortran, and only because that's the way they've been doing it for the last 50 years. Modern versions of the language have modern features and it compiles into very fast code. However, so does C.

http://pypy.org/ (It's basically just a part of PyPy)

Also, it doesn't have a language spec because it's basically "Python, as long as it doesn't do anything dynamic the RPython toolchain can't deal with, like a = 1; a = "some string"."

Most notably:

• Variables have to have one type. This includes function arguments to functions you write yourself, so you can't easily write generic functions.
• The only way to do generics is by defining a class that has several wrapper subclasses, and using isinstance to suss out which subclass you're dealing with when you need access to wrapped values.
• This also goes for the values in lists and keys and values in dicts.
• Overloading operators is not supported.
• I might be forgetting some things.

Guido van Rossum, inventor of Python, is fond of saying that Python is "fast enough" -- and I think that answer is the real reason that interpreted languages hang around. I'm not familiar enough with the implementation details to be able to tell you why languages are designed interpreted in the first place -- I would welcome some comment on that subject from someone more experienced! -- but most people who are using Python, Ruby, Javascript et. all are using those languages for problems that aren't particularly computationally expensive. Those people have no particular incentive to throw out the languages or their current implementations, since the speed doesn't particularly affect them.

Meanwhile, developers fond of scripting languages who do have computationally expensive problems to solve can just write a little bit of C code to handle the slow parts and leave the rest to their language of choice. There's tons of code for scripting languages written this way: shiny scripting language code wrapping a super-performant C core.

But to answer your last question: there are ways of optimizing programs in scripting languages before runtime, and using those methods to create better implementations of popular scripting languages is an area of active research. A leading project in the Python space is PyPy, which has extensive (but not complete) compatibility with the standard Python implementation.

it really depends on the requirements, if you are planning to send off as many atms as cheaply as possible, then it would probably be C on something embedded (ultra-mini computer effectively), if it was anything more than that, you could probably pick from any language that the company would allow (if they hire mainly java programmers for example, you probably would need use java too).

all things considered, its just another tool you can use to solve the same problems that other languages can, the trade-off is memory usage and speed (the latter is partially fixed with projects like http://pypy.org/)

> Are most programmers expected to know more than one language, and mix them together as necessary?

I'd have to say, it depends once again, if its in a python centric company, which needs lots of speed in their code, C and Python would be used, if it was just normal python stuff, then it would just be python, but the usefulness of knowing multiple languages is important, some languages solve certain problems better than the others

Well, async i/o does help performance in many workloads -- the "one process per request" model used by many django deployments is pretty heavy in comparison -- but the other thing that helps node.js in the speed department is that it runs javascript with a Just-In-Time compiler (JIT).

To get that advantage with Python, you need pypy.

(And pypy is working well with Twisted.)

Note that with Kickstarter, if a project does not reach the specified monetary goal, then all funds will be retracted and the project will not get funded.

Aside from the Kickstarter, consider the way that PyPy accepts monetary contributions from its community.

I don't think they should abandon Python 2 yet, but I would like to see Python 3 on PyPy. The changes in Python 3 are something we want for the long run, and we are now getting a lot of the key libraries ported. I don't want the community to get split over PyPy vs. Python 3.

I have donated a bit to the Python 3 PyPy port - I'd encourage you to do the same.

Using CPython extension modules with PyPy is often possible and has been for years. I don't know exactly how well this works.

You're right that it doesn't currently support NumPy, though. I've seen some work done on it (at least for a subset of NumPy), but the ongoing NumPy refactoring project should help here, eventually at least.

edit: The above linked article is about an old PyPy-to-CPython RPC bridge. This page suggests that some CPython extensions can be compiled and used directly with PyPy. Still not enough for NumPy, but it's something.

> As i gather, it is not possible or necessary to "dive into C" with PyPy. But also, the libraries not following the RPython language are out of the question for use with PyPY.

This is not right. PyPy is working to get full compatibility with Python C extensions. See: http://pypy.org/compat.html

Which says:

> PyPy has alpha-level support for the CPython C API, however, as of 1.4.1 release this feature is not yet complete. Most libraries will require a bit of effort to work, but there are known success stories. Check out PyPy blog for updates.

As I see it, PyPy is the future of Python. CPython has a grave implementation mistake: the Global Interpreter Lock (GIL). It doesn't let it run real multithread programs (instead, all threads run sequentially). PyPy also brings great speed upgrades with it's JIT. Being self-hosting (running Python in Python, no need for other languages) is also often seen as a milestone in a language.

PyPy3.5 is considered beta software. As long as that's the case, it's unlikely that it will get added to the main Debian repository. And it also doesn't fit your description of being "old, popular, stable and well maintained".

> It's a lot faster.

Citation needed.

Python is my preferred language, but I don't know about Python 3 being faster (nor slower) than Python 2.

Also, I just found out that PyPy has beta support for Python 3.

You're trusting Zend on who's the fastest? And their metric of rendering a Mandelbrot fractal? That's a fairly meaningless thing to benchmark for most applications of those languages - nobody does their number crunching in those languages unless they really don't care about the speed. If they did they'd use the FFI to do the heavy lifting in C/C++. (At least in PHP or Ruby with much better FFI support from what I can find.)

The reason PHP manages to win out in doing it is because PHP has many thin C wrappers. That fact is also the source of many of the complaints against PHP in terms of inconsistent error signals and lack of proper unicode support.

Comparable metrics are far more even in who's on top. Metrics like responses per second for trivial get requests, time required to process authentication or traverse the middleware, etc.

PHP is somewhat infamous for the cost of function calls, and it's reflected in metrics including many calls. (Like any featured framework will have in abundance.)

Additionally it's not being entirely fair comparint the JITed PHP7 against the CPython implementation rather than Python's own JIT implementation PyPy.

The commonly used CPython is an interpreter. It converts source code into Python bytecode before it runs through the interpreter, but that's not compilation. There are JIT implementations of Python like PyPy which generally runs significantly faster than CPython and uses less memory, if performance is important to you.

virtualenv is a Python module. pyenv is a Python version manager--written in bash--that allows you to keep track of different versions of Python.

So, assuming your system version of Python is 2.7, you can do pyenv install 3.4.3 to get an installation of Python 3.4.3. Then, to switch to it:

$ pyenv global 3.4.3 #Python version is now 3.4.3 wherever you go

or to only switch to it locally:

$ pyenv local 3.4.3 # Python version is 3.4.3 in this directory and any subdirectory

With the pyenv-virtualenv plugin, you can make virtual environments based off of versions of Python. For example:

$ pyenv install pypy-2.5.0 $ pyenv virtualenv pypy-2.5.0 my-pypy-venv

makes a virtual environment based off of PyPy 2.5.0. Then, you can do pyenv global my-pypy-venv to switch to that virtual environment.

You're welcome! One other thing to also note is that for this kind of high density numbercrunching <code>pypy</code> will blow the doors off the standard python interpreter, finding the correct value in 0.75 seconds instead of 8.3. (!!)

$ time python intsifter.py
232792560
real 0m8.361s, user 0m8.288s, sys 0m0.034s

$ time pypy intsifter.py
232792560 real 0m0.759s, user 0m0.358s, sys 0m0.081s

1) Function calls are expensive because a new stack frame is created (this is the actually expensive part) and pushed onto the stack.

2) No, but python's being interpreted and the nature of its function calls makes it more susceptible than other languages. There is no way to inline functions, for instance - in C the compiler can even do that for you sometimes. edit: Also, CPython (the default implementation) doesn't do any Just-In-Time compiling like a number of other languages/platforms do. Java's JVM, for example, has a very advanced JIT. pypy is a JITted Python implementation.

3) You could look into using generators if you have something that maps to that concept nicely - they are kind of like re-enterable functions and seem to be less expensive than regular function calls, because (I think) the stack frame doesn't need to be re-recreated each time it yields, only when it starts.

The answer to the question, as with any performance question, is really to profile (using cProfile probably) and figure out where you're spending all that extra time.

It's the official Python packages catalog (and for many packages, it has also become their repository). Pip and easy_install will usually look there by default.

Note there is also PyPy which is different implementation of the Python language.

There's already a version of PyPy targeting Python 3 (still a WIP though).

> quite slow compared to native code

Well a goal of PyPy is to approach native speed and the gap is narrowing.

> consider is wrapping C libraries using ctypes instead of writing everything from scratch

The OP did say he doesn't want to use pre-written libraries that do everything for you.

The only plans that are aware of were on the donation page and the occasional PyPy blog. The information on the donation pages was the plan for this release. The best source for information on the roadmap would to ask a question on #pypy on freenode or on their mailing list.

I would expect that now that the first release of PyPy3 is out that future releases containing newer version of Python 3 would occur similar to how they handle newer version of Python 3. Typically its not a huge amount of work to upgrade to a new major release of Python as some times I have seen it done in as little as 1.5 months. This depends on availability of developers so sometimes there are long periods where no one is working on the language updates and instead working on other areas of the PyPy infrastructure. This can be sped up by donating to the PyPy3 effort which would increase the effort that is dedicated towards PyPy3 or to lend a hand with the effort. Of all the areas of PyPy development this is one of the areas which is easiest to be a contributor.

On minor point releases I have seen them get them done on PyPy for Python 2 in a 1-2 weeks for the larger point releases and in a couple of days for the smaller ones. Of course these numbers are when there is a concentrated effort to complete the task and there are times when nobody is working on these updates. Usually what happens is PyPy itself has had enough new improvements that it warrants a new release and just before the release they will make a decision to bump up the Python version and then they just get the work done before the PyPy release. PyPy releases tends to occur every few months.

I don't know too many of the details, I'm a Matlab/C++ guy and recent Julia convert. Python packages never build for me even with CPython on Red Hat or Windows so I've stayed away from it.

Quoting from http://pypy.org/compat.html: > PyPy has alpha/beta-level support for the CPython C API, however, as of 2.3 release this feature is not yet complete. We strongly advise use of CFFI instead. CFFI come builtin with PyPy. Many libraries will require a bit of effort to work, but there are known success stories.

I'm going to guess that "a bit of effort" translates into "if you're the developer of the library you might be able to pull it off, otherwise good luck."

Python's list is implemented in Java, C#, and a restricted subset of Python itself too. It's only the reference implementation that's written in C.

I believe a large chunk of PIL is implemented using CPython. PyPy has alpha/beta-level support for the CPython C API. You will need to recompile PIL for PyPy. See http://pypy.org/compat.html for more information.

From my understanding, even if you manage to use PIL C code with PyPy, you will only get the benefits on the part that is coded with Python. PyPy is good to improve performance of Python code. It is not meant to improve performance on C code.

In the link you provided, PyPy is used to improve the performance of Python code that does the image processing.

If you want to improve the performance of your existing project, I would suggest you look at traditional tips that are still very good for everyone: http://wiki.python.org/moin/PythonSpeed/PerformanceTips . I would pay close attention to the Profiling Code section.

> As of the current release of pypy3, all the tests seem to show that its generally faster.

It has been a little while since I've followed the development of PyPy, I may have been mistaken. It really depends on what you're doing in my experience.

> why is pypy3 coming along so slowly

One must really appreciate the hard work that goes into the PyPy project. It takes a lot of time and effort on the part of the engineers behind it. It is not a trivial thing for them to implement all the features of CPython in PyPy. By contrast CPython has a lot more contributors and a significantly less complex implementation.

There is a call to donate for pypy3 progress. They recently got funding from Mozilla to develop PyPy to support Python 3.5 - You can also see on that page some explanation of the PyPy3 efforts. It's reasonable to believe that PyPy will meet their goals to get PyPy3 out of beta before 2020, hopefully sometime this year.

> so what are they doing? Are they all using pypy with python 2.7?

Keep in mind that CPython2 is still more widely used than Python3 (not that it's a good thing) and further that PyPy3 is considered beta and even though there are stable releases, the fact that it is beta can discourage or prevent a lot of people from using it in a production environment. To the best of my knowledge, PyPy2 is more prevalent, but I'll try to find some stats to support that theory.

You must also consider your use case for PyPy. Do you really need the speed improvements? Usually, I would not use PyPy until I have a case where (1) I need the speed, (2) I have already reasonably optimized the CPython code and (3) PyPy supports the code and actually runs faster, which it does often, but not always.

Put simply, if you need to install scipy, just use CPython3. Hopefully when you really need the speed, it will be fully supported by PyPy.

Thanks!

I'l give PyPy a shot. The performance page says that it does best with long-running things. If I have a fairly small routine that runs a whole bunch of times, does that count as long-running?

Haven't had to myself, but it should be fine.

But I have heard issues regarding C, but that's due to lots of C code that is meant to interface with CPython using the CPython C API: http://pypy.org/compat.html

C code that includes "Python.h" is probably written with the CPython API. From that last link it looks like they're trying to implement the same API but it's alpha/beta. It makes sense though that C code that is written for CPython won't work.

Still, you should be able to use ctypes and import functions from shared libraries all the same. It's not that it can't call C code, but it probably won't work with C libraries that are built using the CPython API specifically.

Not to mention, you can use pypy or cython. Or any number of numerical libraries to accelerate performance, depending on the task. Python is very popular in big data science.

http://pypy.org/ http://cython.org/

If you're interested in how some of those C implementations might look in Python... check out pypy!

Pypy is python written in python.

Random Source:

• pypy / pypy / module / _random /
• pypy / rpython / rlib / rrandom.py

Thanks for that benchmark.

I think I had seen your numbers when I had asked my original question; I was actually more interested in the interpreter you used for Python. There are multiple interpreters:

cPython (reference implementation), Jython, IronPython, etc. If you run Python from your command-line, you're probably running cPython.

The one I was most interested in was PyPy and the version that supports Python 2.7: http://pypy.org

PyPy is a JIT'd interpreter. There is a warm-up time required that is relatively slow, but after the warmup, the execution is generally faster than cPython. Benchmarks: http://speed.pypy.org

I tweaked your benchmark slightly: https://gist.github.com/anonymous/8b387decc6c27e81be82

And then I ran with python (7.9 sec) and pypy (0.69 sec). I didn't expect it to be quite that different, but pypy is constantly improving. It's worth mentioning which interpreter you're using because pypy is a strong interpreter for performance based benchmarks.

> Why use CPython instead of C itself?

What? There are different interpreters for the Python programming language:

CPython is the default and the most widely used implementation of the Python programming language. And it is written in C.

Jython is another implementation of the Python programming language, written in Java. Advantage: You have access to Java libraries.

Pypy is another one, written in Python(more precisely in a dialect of Python called RPython). You can read the advantages in Pypy website

IronPython is an implementation of the Python programming language targeting the .NET Framework and Mono. Written in C#.

This is undoubtedly due to the interpreter's method of working, as you show with the disassembly. I'm sure the logic is that the local variable value can't be changed except through local code in the method body. However the value of an object property could (in principle) be changed during any function call. (A function call could even change the meaning of self.value from int to float or str or even to a function.) So it must always be fetched fresh before each use, and that requires a lookup in the dict of names of members of the class.

There are no function calls in this inner loop, and a smarter interpreter could recognize that and realize that self.value cannot be changed during the loop so the lookup is not required. But to avoid the lookup it would have to cache it during the loop, in effect creating a temporary local variable -- as you did manually.

For "smarter interpeter" see pypy or nuitka.

PyPy is not "usually slower than CPython" -- it averages seven times faster, which makes it comparable to Java. But you make a reasonable point that there are uses where the PyPy JIT compiler doesn't help.

As far as "optimizing the execution away entirely", even assuming that it was actually a no-op, the fact that "do nothing" is faster in Python than in C is still significant. I assuming you're talking about the many comments from "Eric" on the "carefully crafted example" -- as others have pointed out, repeatedly, he's missed the forest for the trees.

> "My hatred for Java"

This line tells me you are either inexperienced or very close minded, both make you a worse programmer (the first one can be fixed tho). Programming is all about knowing the right tool for the job. Would you write an IO-constrained, user-interfacing, dynamic application in C because 'it's faster'? The organization I work for did, and they are loosing 10s of thousands of dollars a day because of that mentality.

Btw, you can totally make language B run faster with a interpreter compiled with language A, than with language A itself. By abstracting yourself from low level details you can work on doing things more efficiently with things like branch prediction and other optimizations that would simply be too hard to do in language A.

Who says that a C++ compiler written in Java can't generate faster code than a C++ compiler written in C++?

For an example of this, see PyPy:
http://pypy.org/
http://speed.pypy.org/

I'm not sure if you're familiar with it, but there is a JIT for python, called PyPy.

As for performance it varies greatly. From slower than CPython, to near C speed, depending on what you're doing (and your language skills, I assume).

google "Why Ironpython", it should come up with the ironpython homepage

there is a page for jython, pypy, and all the rest

http://ironpython.net/ don't even need to scroll

https://wiki.python.org/jython/WhyJython a single click from the homepage

http://pypy.org/ it is literally 90% of the homepage

While the interpreter/VM/JIT in PyPy only directly supports a subset of the language, (nearly) the full language is implemented on top of that and manages to achieve significant speedups because of it. They have a very high level of compatible with the CPython implementation.

> What is the best way to install modules for pypy?

The site recommends you use pip or <code>pypy setup.py install</code>.

> will it take much work to get a module to work?

Depends. Check the compatibility listing. If the module has portions written in C, the chances of it working are much less than if it's a pure Python module.

It's expected that libraries like Twisted, Tornado, etc will have to have a modification made to their event loop to take full advantage of PyPy STM. Some libraries such as these as well as other that are mention in 2nd Call for donations - Transactional Memory in PyPy will be modified as part of the PyPy STM effort.

For the majority of user code using Twisted, Tornado, etc it is not expected that any additional modifications will be required. However, time will tell if all things go as planned.

Agreed that if you used Numpy and PyPy there is some risk at this stage where you'd run into some portion of numpy that was slow or perhaps even unavailable.

However, the sample calculation OP has provided looks like something PyPy would be very good at.

The advantage of PyPy here is you don't have to go off and rewrite parts of your program is a language that isn't quite Python.

The disadvantage is that you do have to use their numpy fork, so you'd have to invest a little time in setting things up. But there's a little set-up time involved with Cython too.

I haven't used Numba at all, so I can't really comment on that option, but at first glance it does also seem pretty well-suited to this application.

the main thing is you can't install numpy, you need to install pypy's numpy

http://pypy.org/download.html#installing-numpy

https://bitbucket.org/pypy/numpy/src/66a970bb8ebbf4737b867d3c9c228061297524a8/INSTALL.txt?at=master

While I agree with you in general, I wanted to point out that Python has PyPy, a JIT interpreter that frequently gets as fast as C code. In the webdev realm, Quora.com has been using it for the last couple of years.

edit: As far as downsides, I'd say:

• Python: the flexibility of it's object model prevents the use of compile-time checks many languages benefit from. That and it's packaging ecosystem needs work (though it's recently improving).

• Ruby: similar to python, but I'd also complain about the widespread use of monkeypatching.

• Javascript: spaghetti of a type system, and object attributes and contents share same namespace.

• PHP: horribly inconsistent design, and the archaic cgi url->filesystem dispatch makes it woefully insecure compared to any other language.

As freak_t pointed out, you need to talk to professors - even if it is by email.

I'd like to point out that plenty of interesting open-source efforts can benefit from research work, and it might suit you as it could include some implementation work as well. Check for example http://pypy.org/, which is doing impressive things and is very much tied to research. Projects like these might have bite-sized problems to work on.

Many projects will definitely not be (perceptively) sped up by PyPy

Things with "lots of loops" would usually have the biggest improvements, for example running this raytracer in CPython takes 1min20 or so, and in PyPy takes about 5 seconds, which is about a 16 times speed up with no code changing

Although PyPy main.. selling point is speed, it does have other benefits. For example, sandboxing and Stackless, and also because it's written in Python, it seems relatively easy to develop compared to CPython

> not a press release

Where would you expect to find "a press release" to programmers on the public PyPy website if not on the public PyPy Status Blog?

If the post had been to pypy-dev you'd have a good point.

:) I've used my intuition NOT my experience (I'm not that good). However, if you want to see code, check TinyPy, pypy and luaJIT's FAQ