A clean, lightweight alternative to Python's twisted? [closed]

Twisted is complex, you're right about that. Twisted is not bloated.

If you take a look here: http://twistedmatrix.com/trac/browser/trunk/twisted you'll find an organized, comprehensive, and very well tested suite of many protocols of the internet, as well as helper code to write and deploy very sophisticated network applications. I wouldn't confuse bloat with comprehensiveness.

It's well known that the Twisted documentation isn't the most user-friendly from first glance, and I believe this turns away an unfortunate number of people. But Twisted is amazing (IMHO) if you put in the time. I did and it proved to be worth it, and I'd recommend to others to try the same.

gevent is eventlet cleaned up.

API-wise it follows the same conventions as the standard library (in particular, threading and multiprocessing modules) where it makes sense. So you have familiar things like Queue and Event to work with.

It only supports libevent (update: libev since 1.0) as reactor implementation but takes full advantage of it, featuring a fast WSGI server based on libevent-http and resolving DNS queries through libevent-dns as opposed to using a thread pool like most other libraries do. (update: since 1.0 c-ares is used to make async DNS queries; threadpool is also an option.)

Like eventlet, it makes the callbacks and Deferreds unnecessary by using greenlets.

Check out the examples: concurrent download of multiple urls, long polling webchat.

I liked the concurrence Python module which relies on either Stackless Python microthreads or Greenlets for light-weight threading. All blocking network I/O is transparently made asynchronous through a single libevent loop, so it should be nearly as efficient as an real asynchronous server.

I suppose it's similar to Eventlet in this way.

The downside is that its API is quite different from Python's sockets/threading modules; you need to rewrite a fair bit of your application (or write a compatibility shim layer)

Edit: It seems that there's also cogen, which is similar, but uses Python 2.5's enhanced generators for its coroutines, instead of Greenlets. This makes it more portable than concurrence and other alternatives. Network I/O is done directly with epoll/kqueue/iocp.

A really interesting comparison of such frameworks was compiled by Nicholas Piël on his blog: it's well worth a read!

None of these solutions will avoid that fact that the GIL prevents CPU parallelism - they are just better ways of getting IO parallelism that you already have with threads. If you think you can do better IO, by all means pursue one of these, but if your bottleneck is in processing the results nothing here will help except for the multiprocessing module.

I wouldn't go as far as to call Twisted bloated, but it is difficult to wrap your head around. I avoided really settling in an learn for quite a while as I always wanted something a little easier for 'small tasks'.

However, now that I have worked with it some more I have to say having all the batteries included is VERY nice.

All the other async libraries I've worked with end being way less mature than they even appear. Twisted's event loop is solid.

I'm not quite sure how to solve the steep Twisted learning curve. It might help if someone would fork it and clean a few things up, like removing all the backwards compatability cruft and the dead projects. But that's the nature of mature software I guess.

Kamaelia hasn't been mentioned yet. Its concurrency model is based on wiring together components with message passing between inboxes and outboxes. Here's a brief overview.

I've started to use twisted for some things. The beauty of it almost is because it's "bloated." There are connectors for just about any of the main protocols out there. You can have a jabber bot that will take commands and post to an irc server, email them to someone, run a command, read from an NNTP server, and monitor a web page for changes. The bad news is it can do all of that and can make things overly complex for simple tasks like the OP explained. The advantage of python though is you only include what you need. So while the download may be 20mb, you may only include 2mb of libraries (which is still a lot). My biggest complaint with twisted is although they include examples, anything beyond a basic tcp server you're on your own.

While not a python solution, I've seen node.js gain a lot more traction as of late. In fact I've considered looking into it for smaller projects but I just cringe when I hear javascript :)

There is a good book on the subject: "Twisted Network Programming Essentials", by Abe Fettig. The examples show how to write very Pythonic code, and to me personally, do not strike me as based on a bloated framework. Look at the solutions in the book, if they aren't clean, then I don't know what clean means.

My only enigma is the same I have with other frameworks, like Ruby. I worry, does it scale up? I would hate to commit a client to a framework that is going to have scalability problems.

Whizzer is a tiny asynchronous socket framework that uses pyev. Its very fast, primarily because of pyev. It attempts to provide a similiar interface as twisted with some slight changes.

Also try Syncless. It's coroutine-based (so it's similar to Concurrence, Eventlet and gevent). It implements drop-in non-blocking replacements for socket.socket, socket.gethostbyname (etc.), ssl.SSLSocket, time.sleep and select.select. It's fast. It needs Stackless Python and libevent. It contains a mandatory Python extension written in C (Pyrex/Cython).

I Confirm the goodness of syncless. It can use libev (the newer, cleaner and better performance version of libevent). A while ago it didn't have as much support as libevent, but now the development process is more advanced and syncless very useful.

If you just want a Simplified, lightweight HTTP Request Library then I find Unirest really good

You are welcome to have a look at PyWorks, which takes a quite different approach. It lets object instances run in their own thread and makes function call's to that object async.

Just let a class inherit from Task instead of object and it is async, all methods calls are Proxies. Return values (if you need them) are Future proxies.

res = obj.method( args )
# code continues here without waiting for method to finish
do_something_else( )
print "Result = %d" % res # Code will block here, if res not calculated yet

PyWorks can be found on http://bitbucket.org/raindog/pyworks