Allegra » Blog Archive » My Twisted Apologies

My Twisted Apologies

Glyph Lefkowitz took offense from some of my comments about the early version of Twisted and its profitability as a web framework. The title of his post ("slaying Medusa") and his reference to me as “some guy” clearly shows that I pissed him off. So, here are my apologies for beeing rude about his code.

Plus a response to his most insightfull comment on Allegra:

Be clear about what you’re trying to do differently

Here is what I’ve done differently from Twisted: instead of reinventing Medusa, I only made a few marginal and practical improvements to a stable, simple and well-understood design.

For instance, let’s compare both implementations of deferred execution and continuation from within an asynchronous loop.

Deferred vs. Scheduled & Finalized

However large and sometimes obscure (see defer.py), Twisted implementation of asynchronous continuations boils down to this:

class Deferred:
    def __init__(self):
        self.callbacks = []

For each deferred, two Python objects are instanciated: a classic instance and an empty list. Using that rather heavy structure as "bread and butter", the implementation goes on to deliver what looks like a small Lisp interpreter. It even provides deferred semaphores and locks, reintroducing much of the complexity associated with threads and which an asynchronous design is supposed to avoid in the first place.

In my not-so-humble-opinion, Allegra provides a much better way.

Allegra Scheduled

First, events scheduled in time should not be confused with more general kind of asynchronous continuations. They deserve an implementation and interface of their own. Here it is, complete, as extracted from async_loop.py:

import time, heapq
	
async_scheduled = []

is a heap queue usually accessed by the function that follows

def async_schedule (when, defered):
    "schedule a call to defered for when"
    heapq.heappush (
        async_scheduled, (when, defered)
        )

and polled from within the main asynchronous loop by:

def async_clock ():
    "call all defered scheduled after now"
    if not async_scheduled:
        return
	
    now = time.time ()
    while async_scheduled:
        # get the next defered ...
        defered = heapq.heappop (async_scheduled)
            if defered[0] > now:
                heapq.heappush (
                    async_scheduled, defered
                    )
                break  # ... nothing to defer now.
	
        try:
            # ... do defer and ...
            defered = defered[1] (defered[0])
        except:
            loginfo.loginfo_traceback ()
        else:
            if defered != None:
                heapq.heappush (
                    async_scheduled, defered
                    )
                #
                # ... maybe continue ...

The practical benefits of Allegra’s scheduler implementation are:

instanciate only one tuple per event
allow scheduled functions to recur or not
avoid the drift in time induced by relative timing

Not only is it a faster, more convenient and probably more precise way to program scheduled events, it is also easier to understand.

Hence, easier to apply.

Allegra Finalization

To program asynchronous workflows, Allegra also provide a simple and practical Python implementation of finalization. Here is its first part as you will find in async_loop.py:

import collections
	
async_finalized = collections.deque ()

is a deque of finalization method and finalized instance tuples, polled from within the main asynchronous loop by:

def async_finalize ():
    "call all finalization queued"
    while True:
        try:
            finalize, finalized = \
                async_finalized.popleft ()
        except IndexError:
            break
	
        try:
            finalize (finalized)
        except:
            loginfo.loginfo_traceback ()

The rest of the implementation is found in finalization.py:

from allegra import async_loop
	
class Finalization (object):
	
    finalization = None
	
    def __del__ (self):
        if self.finalization == None:
            return
	
        async_loop.async_finalized.append ((
            self.finalization, self
            ))
        self.finalization = None

It works as follow:

An instance of Finalization (or derived types) that has a non-null finalization attributes is pushed on the async_finalized deque when collected by CPython’s Garbage Collector (GC), and its finalization is dereferenced.

Then, on the next run of the async_finalize poll loop, this instance is passed as sole argument to the finalization method previously bound (which incidentally allows that method to throw exceptions, something not possible in a __del__).

It may look like a cheap Python hack, yet it delivers an implementation of continuations in pure Python, with minimal overhead and a practical interface. Finalizations take advantage of the CPython GC, so they are certainly faster and probably safer than anything similar written in Python instead of for Python. Also, classes derived from Finalization are unencumbered by a verbose interface and provide a practical place to hold the execution state, with little chance of namespace conflict and other subtle bugs.

For instance, chaining and joining can be implemented as:

from allegra.finalization import Finalization
from allegra.loginfo import log
class Joined (Finalization):
    def __call__ (self, joining):
        log ('%r joined by %r' % (self, joining))

and then applied as:

Finalization ().finalization = Joined ()

Simple and elegant, isn’t?

Conclusion

Twisted and Allegra are two very different things. Twisted is a large set of complex libraries with support for a vast number of protocols and systems. Allegra is a small set of simple modules that supports only a minimal collection of web protocols and focuses on a single application.

Allegra’s core delivers marginal but practical improvements over the original library, in all directions. So, even stripped off its applications, it still fully deserves its own name. Precisely because it is as simple as its predecessor.

And, as-we-all-know, the most difficult path is to make it as simple as possible but not simpler.

Kind regards,

Laurent A.V. Szyster (aka "some guy" ;-)

This entry was posted on Friday, February 3rd, 2006 at 5:06 pm and is filed under Allegra, Documentation, Python. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

20 Responses to “My Twisted Apologies”

Bob Ippolito Says:
February 3rd, 2006 at 6:21 pm
Clearly, you have *no idea* how to use or read the source to Twisted. That’s not at all what Deferred does. The Deferred class has absolutely nothing at all to do with time.

You have described the callLater method of the reactor, which also uses a heapq of tuples.. nothing to see here, move on.
Laurent Szyster Says:
February 3rd, 2006 at 7:17 pm
Indeed, reading Twisted sources is as problematic as using them ;-)

And yes there is a callLater method defined deep into:

twisted.internet.interfaces.IReactorTime.callLater

and implemented in:

twisted.internet.base.ReactorBase

but it does not do what async_schedule does. With callLater, you can’t recur or continue as simply from within the scheduled function and there is no provision made against relative time drift.

Last but not least, the ReactorBase.callLater method does not instanciate a tuple but a DelayedCall instance, with 9 attributes filling its dictionnary.

Not exactly a lightweight structure for an ephemeral!

Read the sources again, and … move on.
Fredrik Says:
February 3rd, 2006 at 7:17 pm
Umm. Did you even get to the part that starts with “To program asynchronous workflows” before you flamed away?
anonymous Says:
February 3rd, 2006 at 11:23 pm
> With callLater, you can’t recur or continue as simply from within the scheduled function and there is no provision made against relative time drift.

I think you missed twisted.internet.task.LoopingCall, which does precisely this.
Valentino Volonghi aka Dialtone Says:
February 4th, 2006 at 12:00 am
Besides the fact that documentation is pretty clear on what deferreds are:
http://twistedmatrix.com/projects/core/documentation/howto/defer.html
http://twistedmatrix.com/projects/core/documentation/howto/gendefer.html
http://twistedmatrix.com/projects/core/documentation/howto/deferredindepth.html
http://twistedmatrix.com/projects/core/documentation/howto/deferrecipes.html

And besides the fact that callLater and scheduling in general is also present in the documentation (and it’s also a hella lot simpler than your approach):
http://twistedmatrix.com/projects/core/documentation/howto/time.html

Besides the fact it’s pretty unclear what:
“”"
With callLater, you can’t recur or continue as simply from within the scheduled function and there is no provision made against relative time drift.
“”"
means, twisted.internet.task.LoopingCall() also corrects relative time drifts and keeps calling a function indefinately every X seconds.

Besides all these things: you clearly don’t know both what a continuation is and what a deferred is let alone the problem it solves.

Oh… And this doesn’t enforce the fact that twisted is ‘complex’ (your code took me 10 minutes to be understood whereas the same thing in twisted took me a far less) instead it tells how little you even tried to know (not even learn) something about it.
Laurent Szyster Says:
February 4th, 2006 at 1:18 am
To Valentino:

First, I’m not sure that the documentation about Twisted Deferred is that clear if it takes four HOW-TOs to explain it.

Second, LoopingCall does not allow the scheduled function to control when or if it will be continued, and how.

Third, the fact that there are two interfaces:

internet.base.reactor.callLater
internet.task.LoopingCall

to achieve *less* than the single:

async_loop.async_schedule

tells volume about their respective quality (I won’t comment more on their implementation, see above).

Finally, if you confess that it took you 10 minutes to understand the implementation of async_schedule, I doubt you have much credential to assert wether or not somebody else understand what continuations are.
Valentino Volonghi aka Dialtone Says:
February 4th, 2006 at 1:34 am
When there is abundance of documentation it’s because that something is hard. When there is no documentation it’s bad because users cannot understand it. I’m sure users won’t ever stop complaning about anything at all but still at least try a bit :).

callLater and LoopingCall are (surprise surprise) different! One keeps calling the function, the other doesn’t (I’ll let the reader match the correspondence). Of course one is implemented using the other.

Then you should feel a bit ‘pressed’ to explain how you can talk about quality while using __del__ for finalization. Let’s talk about cyclic references (where the presence of __del__ holds the gc from working) and the fact the there is _no_ guarantee of _when_ something is garbage collected. There are so many problems with your concept of ‘quality’ that I’m happy you don’t like twisted.
Laurent Szyster Says:
February 4th, 2006 at 2:11 am
To Valentino:

> callLater and LoopingCall are (surprise surprise) different!

I do very well understand that LoopingCall and callLater are different interfaces. What you obviously do not to understand is how it is possible to achieve more with a single interface and a simpler implementation. So, spend another 10 minutes on the 21 lines of code that implement the async_schedule interface. Or even better, try it.

> the presence of __del__ holds the gc from working

The presence of a __del__ method bound to an instance does not prevent the GC “to work”, it simply prevents it to collect that instance when caught in a circular references.

Which is exactly what Finalization requires in order to produce the effect of continuations. As for the timing of garbage collection, it does not matter to the applications of Finalization.

> I’m happy you don’t like twisted.

I may not be found of Twisted sophistication, but it’s community does not deserve the bad name your giving it with such silly comments. So, think twice before you send more flames: you are hurting what you love.
Valentino Volonghi aka Dialtone Says:
February 4th, 2006 at 3:20 am
Now… callLater is 8 lines (counting asserts and split lines for arguments).
LoopingCall functionality (the _reschedule method) is 11 lines. Do you really think you can be so arrogant as to think that they are separated because twisted developers like adding extra complexity just for fun? You can’t be that bad, can you?

The work of the gc is (another surprise surprise) to collect instances (what a surprise, maybe that’s why it’s called collector) and if you prevent it to collect instances then it is not working, thus I fail to understand the quotes around to work, maybe you are not sure of what you are talking about?. If guaranteeing a certain call time is not part of Finalization applications, what is it for? Can you imagine a python where you expect something to be called when you call it, but instead it gets called 5 seconds later? Of course you may have no interest in any guarantee on the calls your application makes. (And I’m not going to talk about the semantics change in those instances of course, let alone the fact that you are requiring inheritance to implement a function call, now talk about quality).

And as a closure:
Don’t worry about me in the twisted community. Think about yourself which is already pretty hard from what I can read.
Glyph Lefkowitz Says:
February 4th, 2006 at 4:00 am
Since the ravening hordes of Twisted minions seem to be doing a good job of tearing apart your misunderstanding of Twisted, I’ll just say “thank you” for the apology. Although there are some mistakes, I appreciated the more technical tone of this entry.

The reason I chose the title “slaying medusa” was not to insult you. I had hoped that the fact that I linked to the unicorn-vs-warewolf battle T-shirt was an indication that it was somewhat tongue-in-cheek. I called it “slaying medusa” because Twisted is supposed to be a full replacement for Medusa, both for simple and complex applications. Both Zope and Bittorrent are moving towards Twisted-derived loops. The whole point of the effort is to concentrate effort in Python towards an environment where you can write asynchronous code that can be run regardless of its “container”, i.e. on Windows, on UNIX, in a PyGTK app, in a Tkinter app, in a PyQT app.

If there is some documentation that would make this choice easier, please feel free to let us know. I note that you’re concerned about efficiency, but opitimizing some Twisted functionality into C would be a lot easier than writing a whole new framework, and would benefit a lot more code - not to mention giving a 10x or 20x speedup instead of the 1.1x or so that you get from using tuples rather than classes. (There is already a C implementation of Deferred, it just needs more testing.)
Laurent Szyster Says:
February 4th, 2006 at 4:50 am
Hi Glyph,

I’m glad this does not turn into a nasty war flame with you, although you should maybe prevent one of you “minions” to make a laughing stock of himself ;-)

Twisted integration with GUI on different plateforms and its large set of packages are undeniable assets for Python, but I believe that there is still room for a more compact and purely network oriented library.

Allegra will probably never outgrow its 50 modules, even if they can still be improved. The application domain it targets are “pure” network peers that do not require what Twisted can supply. As tarlano rightly pointed out:

“When you need everything that twisted offers (reactor, interfaces, demonizer, configurator, adaptors, etc.) then twisted is the best choice for your requirements, but that is not always. I think that both have their merits and that both should be respected for their design decision.”

Regarding Deferred, why don’t you have a long hard look at Allegra’s finalization.py module. After all, its implementation of asynchronous continuations is piggy-backed on the CPython Garbage Collector, which is itself allready made of stable and optimized C code.

I don’t see wether it is possible and how this could be retrofited into Twisted code base, but I assume you will.

Kind Regards,
Damjan Says:
February 4th, 2006 at 6:19 am
> Besides the fact that documentation is pretty clear on what deferreds are:

I wouldn’t say so, I even have the Twisted book (from O’Reiley) and it’s still not clear to me when to use deferred’s and when to use something else.. and what?
Mark Rowe Says:
February 4th, 2006 at 9:01 am
A few random points:

You keep stating that Allegra’s finalizations are “piggy-backed on the CPython Garbage Collector” which a simple examination of the module shows is untrue. It is based on Python’s object finalization. This is orthogonal to the rest of the discussion, but I find it a little unnerving that you’re suggesting people use code that makes heavy use of __del__ while apparently being unclear on the rammifcations. The garbage collector exists in Python primarily to clean up objects involved in reference cycles, which is one point that Pythons reference counting doesn’t cope with. As the gc module documentation states, the presence of __del__ methods and are a part of a reference cycle will cause the entire cycle to be uncollectable. I believe this is one of the points that Valentino was trying to make earlier.

Secondly, Finalization.__del__ method references a global object. Under some circumstances this can cause an exception to be raised. Quoting the Python documentations warning on the __del__ method: “Also, when __del__() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the __del__() method may already have been deleted”. Comments in the source of the allegra.finalization module say that the various classes and methods based on Finalization “make[s] Allegra’s web peer a very interesting pateform for some business network applications development”. Using globals from __del__ strikes me as going against that reliability that business network applications would require.

Finally, Deferred feels a lot better designed than the Finalization class in Allegra. Deferred can be used completely independently of Twisteds reactor, while Finalization is tied to Allegras concept of a main loop. Deferreds design is trivial to port to other languages, while Finalization relies on an object finalization method that can resurrect the object being finalized. Finalization’s reliance on the __del__ method feels very much like a hack as the Python language specification makes no guarantees about when, if at all, __del__ will be called on objects.

Just my 2¢.
Laurent Szyster Says:
February 4th, 2006 at 2:17 pm
Hi Mark,

Yes finalization.py is a “hack”. But it works. For an asynchronous network peer in CPython only, but that’s precisely the plateform targeted and:

practicality beats purity (Zen of Python)

Yes, this design is forcing its applications not to cycle finalized instances.

Is it that hard to code without cycles?

As for the relative quality of Deferred and Finalization design, well, we disagree. I suggest you print and put both sources side by side and ask somebody else wich one is more elegant.

Regards,
Mark Rowe Says:
February 5th, 2006 at 6:33 am
A side-to-side comparison between Deferred and Finalization like you suggest is meaningless, as Finalization implements only a small portion of Deferreds functionality.

That’s the last I’ll say on this as I’m finding your attitude on this whole Allegra v. Twisted thing a touch too self-aggrandizing for my tastes.
Rene Dudfield Says:
February 6th, 2006 at 2:57 am
I like a simpler still event design.

eg.

for event in events.get():
# handle event.
if event.type == types.something:
dosomething()

You can also look at the whole list of events, and decide to act on only some. Allowing you to reason about what information you have available to you now.
eg.

# only act on the last something event in this loop iteration.
# since we don’t care about other ones.
something_events = filter(lambda x:x.type == types.something, events)
if something_events:
one_something = something_events[-1]

For scheduling a task you schedule an event instead. This way you are not throwing around code, but a small amount of data. Data can be more easily modified, and transferred than code in python.

I also like splitting tasks up into using processes rather than reinventing scheduling, memory limits, cpu limits, memory protection, etc etc that the OS already provides. Sometimes. Different tools for different jobs.

Twisted has support for using threads and processes when async just doesn’t cut it.

Allegra looks like it is aiming for simple, and to only work for a limited set of tasks. That’s a goal that I like. If it can be shown that his approach works correctly, as well as shown to be performant then yah! I think it should address the use cases for processes, and threads when needed. So that if you do need them, then you can switch. Twisted took a while to realise that just using async is not a good idea. Hopefully Allegra does too.

# occasional twisted user, simple lover.
Laurent Szyster Says:
February 6th, 2006 at 5:05 am
Hi Rene,

Nice to have a second comment from somebody else than the “raven hordes of Twisted minions”. But, hell, they scared me so much that I looked hard on some code that had been left alone for a few years now.

And the result is a sure sign that peer review produces good things, whatever the malicious intentions of the reviewers.

I should credit Valentino Volonghi for much of the findings ;-)
Allegra » Blog Archive » To the Ravening Hordes: Thank You! Says:
February 6th, 2006 at 9:57 am
[…] My Twisted Apologies drove from their divmod lair a ravening horde of Minions to tear down Allegra’s implementation of continuations in an Python asynchronous loop. They forced me look once more on code that had been left alone for years. Enough to break the “don’t fix something that’s not broken” sceal. And find out that there was an even simpler implementation … with an additionnal functional interface available. […]
Huh? Says:
February 7th, 2006 at 5:48 pm
Are you seriously trying to sell a dependancy on the Python GC as a *feature*? The most arcane, unstable (between versions) and unpredictable system the platform provides? *SERIOUSLY*?

As long as you’re trying to cheese people off with pumped-up claims, you should invite Christian Tismer to comment on your “pure Python continuations”.
Laurent Szyster Says:
February 7th, 2006 at 6:58 pm
Hi “Hu?”,

Who ever you may be.

As far as I know, it works on Python 2.2, 2.3 and now 2.4, on Linux and Windows. The CPython interpreter simply does not release instances themselves until the __del__ is called *and* all its references released.

What else would you expect?

There are no arcanes in Python.

Don’t fear its sources. It is a simple and elegant implementation.

That’s why CPython is faster than Ruby and Perl. Less resource hungry than Java and twice as fast when implemented with Microsoft’s proprietary VM (heck it was written by the author of TurboPascal!).

Deferred vs. Scheduled & Finalized

Allegra Scheduled

Allegra Finalization

Conclusion

20 Responses to “My Twisted Apologies”

microsoft office 2003 oem

how to activate reget deluxe

Oem software bundle oem software bundles 814.

microsoft oem software cd

microsoft office 2003 oem

software oem cd line business

The Logo Creator MEGA pak 3.6

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem cd line business

Mcafee all in 1 2006

inkjet definition oem software

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

microsoft window 2000 professional oem software

The Logo Creator MEGA pak 3.6

microsoft office software oem;

Oem software bundle oem software bundles 814.

The Logo Creator MEGA pak 3.6

The Logo Creator MEGA pak 3.6

oem software cds;

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

inkjet definition oem software

Buy cheap PhotoShop CS 2

microsoft office 2003 oem

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

software oem cd line business

Buy cheap After Effects 5.5 Production Bundle

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

inkjet definition oem software

iCorrect EditLab Pro 4.52

iCorrect EditLab Pro 4.52

locate software oem files

inkjet definition oem software

Oem software bundle oem software bundles 814.

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

locate software oem files