Comments on: Back to the future, in 1996 http://laurentszyster.be/blog/back-to-the-future/ Python on Peers Fri, 18 May 2012 14:12:48 +0000 http://wordpress.org/?v=1.5.1.3 by: Glyph Lefkowitz http://laurentszyster.be/blog/back-to-the-future/#comment-770 Wed, 24 May 2006 05:23:39 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-770 > There are multiple implementations of pauseProducing (), I found one of them which *is* potentially calling stopReading (). What do you want me to do? Look through all that pile of indirections and find each and every way in which, somehow, Twisted sources are still coupled to the event loop? I want you to understand that an interface is not an implementation, and things implemented in terms of IConsumer and ITransport do not necessarily depend on stopReading, a method not present in either IConsumer nor ITransport. As the interfaces are currently specified, properly implemented Twisted code could survive a full re-implementation of the reactor, removing (start¦stop)(Reading¦Writing) and it would work *exactly the same*. These are not interfaces used by application programmers, period, the end. Not only is this the way things are *supposed* to be, this is the way they *are*. I frequently put Twisted code onto different (non-FileDescriptor) transports. Protocols do not ever practically need to call methods outside of their transports' interfaces. >Here’s is the problem: >I/O event loop -> FileDescriptor -> I/O event loop >There’s an unbroken dependency. FileDescriptor calls a method on an IReactorFDSet implementor, present as self.reactor. This is a classic inversion of control, or what you are now calling a "broken dependency". Maybe its methods were called by methods of that very same IReactorFDSet implementor, maybe not. At any rate, FileDescriptor could be totally re-implemented using readable() and writable() (causing a loss of performance) and no application code would have to change. FileDescriptor is an *implementation detail* of a clearly-specified API that does *not* include stopReading and stopWriting. Here is a simpler example: how does an echo protocol work? Input is received from the event loop, which is passed to a handler, which sends data to the event loop. This is your same "unbroken dependency", but it is the only possible way that it could work. Sure, you can obscure the dependency by placing your methods in the protocol implementation, forcing your application author to handle buffering themselves, but what good does that do? What application does that enable? What possible value does it have, beyond your aesthetic preference? To recap, advantages of the Twisted way: - not polling readable() and writable() is measurably faster and more scalable, even when using standard select() - a stateful reactor API provides a way to use even more efficient and scalable multiplexing mechanisms like epoll - a stateful reactor API allows interfacing with existing protocol implementations on top of existing event loops such as libevent, or GUI event loops such as gobject - explicit interfaces for producing and consuming enable applications which use non-socket sources of data - explicit interfaces improve testability by allowing you to provide mock implementations - subprocesses can be treated similarly to network connections by protocol implementations I have yet to hear a recognizable advantage of the Allegra way. > In Twisted, how do you find out in the protocol implementation what is the current state of your FileDescriptor in the event loop? Is it in or is it out, ready for input or stalling output? You don't have a FileDescriptor. Forget about FileDescriptor. It is an implementation detail. You want to know the state of your transport, which probably provides ITransport, IProducer, and IConsumer. Is it ready for input? If you want to know, call registerProducer(thingThatWantsToKnow). This will then be sent notifications (pauseProducing, resumeProducing) when the state changes. You can't *query* the state of the transport from a protocol, and for good reason. It is only reasonable to respond to changes in this state: if the buffer is full, it is time to stop sending data to it, if the buffer is empty, it is time to start sending data again. The state is hidden, or "encapsulated", by the implementor. This is the difference between synchronous polling and event-driven programming. Twisted is event-driven: you are not expected to poll the state of your transport yourself; the framework will *tell* you when it's time to write. You do not ask, "is it time to write? is it time to write? is it time to write?", nor does the framework ask you, "is it okay for me to produce data? can I produce data? huh? is it data-producing time? can I produce data?" That is what readable() and writable() do. Instead, Twisted politely tells you, "Produce some data now, if you would like.", and "I am too busy to handle more data now, please stop producing it" You may respond to those events as you choose. > There are multiple implementations of pauseProducing (), I found one of them which *is* potentially calling stopReading (). What do you want me to do? Look through all that pile of indirections and find each and every way in which, somehow, Twisted sources are still coupled to the event loop?

I want you to understand that an interface is not an implementation, and things implemented in terms of IConsumer and ITransport do not necessarily depend on stopReading, a method not present in either IConsumer nor ITransport. As the interfaces are currently specified, properly implemented Twisted code could survive a full re-implementation of the reactor, removing (start|stop)(Reading|Writing) and it would work *exactly the same*. These are not interfaces used by application programmers, period, the end.

Not only is this the way things are *supposed* to be, this is the way they *are*. I frequently put Twisted code onto different (non-FileDescriptor) transports. Protocols do not ever practically need to call methods outside of their transports’ interfaces.

>Here’s is the problem:
>I/O event loop -> FileDescriptor -> I/O event loop
>There’s an unbroken dependency.

FileDescriptor calls a method on an IReactorFDSet implementor, present as self.reactor. This is a classic inversion of control, or what you are now calling a “broken dependency”. Maybe its methods were called by methods of that very same IReactorFDSet implementor, maybe not. At any rate, FileDescriptor could be totally re-implemented using readable() and writable() (causing a loss of performance) and no application code would have to change. FileDescriptor is an *implementation detail* of a clearly-specified API that does *not* include stopReading and stopWriting.

Here is a simpler example: how does an echo protocol work? Input is received from the event loop, which is passed to a handler, which sends data to the event loop. This is your same “unbroken dependency”, but it is the only possible way that it could work. Sure, you can obscure the dependency by placing your methods in the protocol implementation, forcing your application author to handle buffering themselves, but what good does that do? What application does that enable? What possible value does it have, beyond your aesthetic preference?

To recap, advantages of the Twisted way:

- not polling readable() and writable() is measurably faster and more scalable, even when using standard select()
- a stateful reactor API provides a way to use even more efficient and scalable multiplexing mechanisms like epoll
- a stateful reactor API allows interfacing with existing protocol implementations on top of existing event loops such as libevent, or GUI event loops such as gobject
- explicit interfaces for producing and consuming enable applications which use non-socket sources of data
- explicit interfaces improve testability by allowing you to provide mock implementations
- subprocesses can be treated similarly to network connections by protocol implementations

I have yet to hear a recognizable advantage of the Allegra way.

> In Twisted, how do you find out in the protocol implementation what is the current state of your FileDescriptor in the event loop? Is it in or is it out, ready for input or stalling output?

You don’t have a FileDescriptor. Forget about FileDescriptor. It is an implementation detail. You want to know the state of your transport, which probably provides ITransport, IProducer, and IConsumer.

Is it ready for input? If you want to know, call registerProducer(thingThatWantsToKnow). This will then be sent notifications (pauseProducing, resumeProducing) when the state changes.

You can’t *query* the state of the transport from a protocol, and for good reason. It is only reasonable to respond to changes in this state: if the buffer is full, it is time to stop sending data to it, if the buffer is empty, it is time to start sending data again. The state is hidden, or “encapsulated”, by the implementor.

This is the difference between synchronous polling and event-driven programming. Twisted is event-driven: you are not expected to poll the state of your transport yourself; the framework will *tell* you when it’s time to write. You do not ask, “is it time to write? is it time to write? is it time to write?”, nor does the framework ask you, “is it okay for me to produce data? can I produce data? huh? is it data-producing time? can I produce data?” That is what readable() and writable() do.

Instead, Twisted politely tells you, “Produce some data now, if you would like.”, and “I am too busy to handle more data now, please stop producing it” You may respond to those events as you choose.

]]>
by: Laurent Szyster http://laurentszyster.be/blog/back-to-the-future/#comment-691 Mon, 22 May 2006 04:45:27 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-691 To Glyph: >Therefore there are already multiple potential implementations which your code could be referring to when calling pauseProducing, and some of them do not necessarily depend on pauseProducing. In fact, it would be breaking encapsulation to even determine that. There are multiple implementations of pauseProducing (), I found one of them which *is* potentially calling stopReading (). What do you want me to do? Look through all that pile of indirections and find each and every way in which, somehow, Twisted sources are still coupled to the event loop? I don't have to do that to *prove* that it is the case. >why do you claim it exists in Twisted I demonstrated why in your sources. Maybe not clearly, so I'll try again. The Twisted FileDescriptor's "fab four" are adding and removing file descriptors from the I/O event loop. Right? Now, the same event loop is calling the other FileDescriptor interfaces, the ones that implement transport and protocols. And some of the implementations of those interfaces, not all but many, do call ... the "fab four". Practically, all the ones that must decide wether to include or exclude the sockets from the I/O event pollster. Here's is the problem: I/O event loop -> FileDescriptor -> I/O event loop There's an unbroken dependency. Implementation of what and how to *handle* read and write events must themselves implement inclusion and exclusion from the event loop. You can tolerate that in a single-purpose application. In asyncore.py, the readable () and writable () interface is *called by* the event loop, all dependencies are broken: I/O event loop -> dispatcher The implementation of transport and protocols don't have to set the state of their dispatchers, they can rely on the one evaluated once *for* the event loop before it polls I/O. > By your definition, if pauseProducing depends on stopReading, then readable() in allegra depends on async_net_pull (and indeed, all of Async_net) - because async_net_pull sets attributes which one implementation of readable() uses. That, in turn, means async_loop depends on all of Async_net. But async_loop.async_poll *calls* Async_net.readable () which evaluate a state set by many implementations of the dispatcher's interface. The effect is radically different. As for async_net_pull, it does resume collection of the buffer. Readability of the dispatcher will still be evaluated *by* the event loop, *when* it calls. Doing the contrary is *very* problematic in a framework. In Twisted, how do you find out in the protocol implementation what is the current state of your FileDescriptor in the event loop? Is it in or is it out, ready for input or stalling output? How do you know in the implementation of the transport wether the producer is stalling and that you should not put the socket back in the list to poll for output? You must test everywhere. Or maintain state somewhere. Probably that is why there is a: producerPaused = 0 attribute set *by default* to the FileDescriptor class. To test, everywhere it will be required, wether the producer paused and flagged or not, then remove or not the socket from the I/O event loop. That's a problem Allegra does not suffer from. Because the producer_stalled () interface is called *by* the writable () method. The only places you will ever see producer_stalled () called is in writable (), in a decorator or a boxing producer (like the Composite_producer). The async_core transports implementations and async_chat protocol implementations "stacked" on a channel do not have to worry wether the producer is stalled or not. And the producers have not to worry wether the channel output buffer is full or not. Or if the socket is still connected. Etc ... Can you see the light? >allegra has only weak conventions and the hope that nothing but sockets will ever produce data? Well, if Allegra can be adopted by all the Python network programmers that use sockets *only*, I would be very glad. Regards, To Glyph:

>Therefore there are already multiple potential implementations which your code could be referring to when calling pauseProducing, and some of them do not necessarily depend on pauseProducing. In fact, it would be breaking encapsulation to even determine that.

There are multiple implementations of pauseProducing (), I found one of them which *is* potentially calling stopReading (). What do you want me to do? Look through all that pile of indirections and find each and every way in which, somehow, Twisted sources are still coupled to the event loop?

I don’t have to do that to *prove* that it is the case.

>why do you claim it exists in Twisted

I demonstrated why in your sources. Maybe not clearly, so I’ll try again.

The Twisted FileDescriptor’s “fab four” are adding and removing file descriptors from the I/O event loop. Right? Now, the same event loop is calling the other FileDescriptor interfaces, the ones that implement transport and protocols.

And some of the implementations of those interfaces, not all but many, do call … the “fab four”. Practically, all the ones that must decide wether to include or exclude the sockets from the I/O event pollster.

Here’s is the problem:

I/O event loop -> FileDescriptor -> I/O event loop

There’s an unbroken dependency.

Implementation of what and how to *handle* read and write events must themselves implement inclusion and exclusion from the event loop. You can tolerate that in a single-purpose application.

In asyncore.py, the readable () and writable () interface is *called by* the event loop, all dependencies are broken:

I/O event loop -> dispatcher

The implementation of transport and protocols don’t have to set the state of their dispatchers, they can rely on the one evaluated once *for* the event loop before it polls I/O.

> By your definition, if pauseProducing depends on stopReading, then readable() in allegra depends on async_net_pull (and indeed, all of Async_net) - because async_net_pull sets attributes which one implementation of readable() uses. That, in turn, means async_loop depends on all of Async_net.

But async_loop.async_poll *calls* Async_net.readable () which evaluate a state set by many implementations of the dispatcher’s interface. The effect is radically different.

As for async_net_pull, it does resume collection of the buffer. Readability of the dispatcher will still be evaluated *by* the event loop, *when* it calls.

Doing the contrary is *very* problematic in a framework.

In Twisted, how do you find out in the protocol implementation what is the current state of your FileDescriptor in the event loop? Is it in or is it out, ready for input or stalling output?

How do you know in the implementation of the transport wether the producer is stalling and that you should not put the socket back in the list to poll for output? You must test everywhere. Or maintain state somewhere.

Probably that is why there is a:

producerPaused = 0

attribute set *by default* to the FileDescriptor class. To test, everywhere it will be required, wether the producer paused and flagged or not, then remove or not the socket from the I/O event loop.

That’s a problem Allegra does not suffer from. Because the

producer_stalled ()

interface is called *by* the writable () method. The only places you will ever see producer_stalled () called is in writable (), in a decorator or a boxing producer (like the Composite_producer).

The async_core transports implementations and async_chat protocol implementations “stacked” on a channel do not have to worry wether the producer is stalled or not. And the producers have not to worry wether the channel output buffer is full or not. Or if the socket is still connected. Etc …

Can you see the light?

>allegra has only weak conventions and the hope that nothing but sockets will ever produce data?

Well, if Allegra can be adopted by all the Python network programmers that use sockets *only*, I would be very glad.

Regards,

]]>
by: Chris Siebenmann http://laurentszyster.be/blog/back-to-the-future/#comment-650 Sun, 21 May 2006 06:42:40 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-650 As a semi-aside, it is well known from a lot of research that anything that wants to scale to lots of connections needs to do as little work per connection as possible. This is one large reason that modern high-connection Unix servers prefer poll() over select() (and often prefer even leaner interfaces); the difference actually matters in practice. Of course, many connections is only one version of 'high load', but it is a version that matters for a lot of Internet-facing servers. As a semi-aside, it is well known from a lot of research that anything that wants to scale to lots of connections needs to do as little work per connection as possible. This is one large reason that modern high-connection Unix servers prefer poll() over select() (and often prefer even leaner interfaces); the difference actually matters in practice. Of course, many connections is only one version of ‘high load’, but it is a version that matters for a lot of Internet-facing servers.

]]>
by: Glyph Lefkowitz http://laurentszyster.be/blog/back-to-the-future/#comment-645 Sun, 21 May 2006 06:17:07 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-645 Finally, you keep mentioning dependencies, as if readable() and writable() have none, but Twisted's approach to this has many. You're using a nonstandard definition of "dependency", as I have mentioned, you are using apparently the exact opposite of the definition that the IoC pattern uses to describe dependency: Twisted code depends on interfaces, not implementations, and therefore the dependency is always one-step deep; what happens underneath the interface is *by definition* an implementation detail. By your definition, if pauseProducing depends on stopReading, then readable() in allegra depends on async_net_pull (and indeed, all of Async_net) - because async_net_pull sets attributes which one implementation of readable() uses. That, in turn, means async_loop depends on all of Async_net. I do not believe you think such a dependency exists in Allegra; why do you claim it exists in Twisted, especially given that Twisted uses explicit interfaces to avoid this level of dependency but allegra has only weak conventions and the hope that nothing but sockets will ever produce data? Finally, you keep mentioning dependencies, as if readable() and writable() have none, but Twisted’s approach to this has many.

You’re using a nonstandard definition of “dependency”, as I have mentioned, you are using apparently the exact opposite of the definition that the IoC pattern uses to describe dependency: Twisted code depends on interfaces, not implementations, and therefore the dependency is always one-step deep; what happens underneath the interface is *by definition* an implementation detail.

By your definition, if pauseProducing depends on stopReading, then readable() in allegra depends on async_net_pull (and indeed, all of Async_net) - because async_net_pull sets attributes which one implementation of readable() uses. That, in turn, means async_loop depends on all of Async_net.

I do not believe you think such a dependency exists in Allegra; why do you claim it exists in Twisted, especially given that Twisted uses explicit interfaces to avoid this level of dependency but allegra has only weak conventions and the hope that nothing but sockets will ever produce data?

]]>
by: Glyph Lefkowitz http://laurentszyster.be/blog/back-to-the-future/#comment-642 Sun, 21 May 2006 05:59:11 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-642 Laurent, Since you seem to be missing the point totally here by hyper-focusing on details of code you are not reading properly, let's try a hypothetical example instead. readable() and writable() are methods that are called by an event loop. They are not related to the production of data, but instead one particular way which you can implement an interface to an OSes socket layer. Let's pretend we have another source of data: digits of π. (This is a hypothetical example, and a more realistic one would be too complex to explain in detail in a blog comment, but I assure you there are numerous applications where I have coded something roughly like this.) Our hypothetical PiProducer produces digits of π by running a native function in a thread. After producing each digit it buffers it; after accumulating a buffer of a certain size, it delivers the buffer to the dataReceived() method of its protocol. This data must be delivered to a foreign system, which processes the digits (to what purpose, we might never guess). Sometimes the foreign system becomes overloaded or network congestion interrupts service between the two endpoints. When this happens, the calculation must be paused (the thread must be blocked) until the outgoing buffer can be emptied. One way to accomplish this is to simply perform the calculation, then write each byte to a blocking socket as it is calculated; I think one of the few things we agree on is that this approach has drawbacks. Another is to deliver notifications (pauseProducing, resumeProducing) to the producer of the data asynchronously, so that it does not outrun its buffer by too wide a margin. PiProducer is neither readable nor writable, as it is not associated with a file descriptor or otherwise representative of inter-process communication. Nevertheless, it is somehow producing data, and it needs the facility to be paused and re-started. This is true of numerous resources on non-UNIX platforms which are not sockets. Even linux does not allow you to meaningfully check a regular disk file for readability; you have to implement some alternative system for it. You can call pauseProducing on this hypothetical class; you cannot call stopReading on it. -glyph (P.S.: You also seem incredibly over-focused on the fact that there are four methods: it could easily have been one method: for example, it might have been called "setActivityMask" and taken 2 optional keyword arguments, "reading" and "writing": stopReading, startReading, stopWriting, startWriting would then become setActivityMask(reading=False), setActivityMask(reading=True), setActivityMask(writing=False), setActivityMask(writing=True). Or, more likely, _setActivityMask, so as not to obscure the fact that it is an implementation detail and not designed to be called from any code outside the reactor's implementation. It's four methods because it makes implementing the reactor easier.) Laurent,

Since you seem to be missing the point totally here by hyper-focusing on details of code you are not reading properly, let’s try a hypothetical example instead.

readable() and writable() are methods that are called by an event loop. They are not related to the production of data, but instead one particular way which you can implement an interface to an OSes socket layer.

Let’s pretend we have another source of data: digits of π. (This is a hypothetical example, and a more realistic one would be too complex to explain in detail in a blog comment, but I assure you there are numerous applications where I have coded something roughly like this.)

Our hypothetical PiProducer produces digits of π by running a native function in a thread. After producing each digit it buffers it; after accumulating a buffer of a certain size, it delivers the buffer to the dataReceived() method of its protocol.

This data must be delivered to a foreign system, which processes the digits (to what purpose, we might never guess). Sometimes the foreign system becomes overloaded or network congestion interrupts service between the two endpoints. When this happens, the calculation must be paused (the thread must be blocked) until the outgoing buffer can be emptied.

One way to accomplish this is to simply perform the calculation, then write each byte to a blocking socket as it is calculated; I think one of the few things we agree on is that this approach has drawbacks. Another is to deliver notifications (pauseProducing, resumeProducing) to the producer of the data asynchronously, so that it does not outrun its buffer by too wide a margin.

PiProducer is neither readable nor writable, as it is not associated with a file descriptor or otherwise representative of inter-process communication. Nevertheless, it is somehow producing data, and it needs the facility to be paused and re-started. This is true of numerous resources on non-UNIX platforms which are not sockets. Even linux does not allow you to meaningfully check a regular disk file for readability; you have to implement some alternative system for it. You can call pauseProducing on this hypothetical class; you cannot call stopReading on it.

-glyph

(P.S.: You also seem incredibly over-focused on the fact that there are four methods: it could easily have been one method: for example, it might have been called “setActivityMask” and taken 2 optional keyword arguments, “reading” and “writing”: stopReading, startReading, stopWriting, startWriting would then become setActivityMask(reading=False), setActivityMask(reading=True), setActivityMask(writing=False), setActivityMask(writing=True). Or, more likely, _setActivityMask, so as not to obscure the fact that it is an implementation detail and not designed to be called from any code outside the reactor’s implementation. It’s four methods because it makes implementing the reactor easier.)

]]>
by: Glyph Lefkowitz http://laurentszyster.be/blog/back-to-the-future/#comment-638 Sun, 21 May 2006 05:40:35 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-638 Laurent, >>Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero. >I found at least one. Please read more carefully! What you found is a call to IProducer.stopProducing, an interface method, not FileDescriptor.stopProducing, or potentially ITransport.loseConnection, an interface method, not FileDescriptor.loseConnection. Andrew is correct: there are zero calls. stopProducing() is an implementation detail. pauseProducing() is implemented in various ways in Twisted (and some outside Twisted), not all of them relying on FileDescriptor. For example, twisted/internet/_pollingfile.py:_PollableReadPipe implements it by calling 'deactivate', not 'stopReading'. This is using a completely different API (the Win32 pipes API, and Twisted timers) and is unrelated to select() state. Therefore there are already multiple potential implementations which your code could be referring to when calling pauseProducing, and some of them do not necessarily depend on pauseProducing. In fact, it would be breaking encapsulation to even determine that. This is the whole point of the IoC pattern you're crowing about: to the extent dependency is possible in Python, Twisted protocols depend on ITransport and IConsumer, *not* FileDescriptor. FileDescriptor is a utility class involved in the implementation of most reactors, but not all of them, and is never visible to the Twisted programmer as a dependency. Do you understand the difference between interfaces and concrete implementations yet? Laurent,

>>Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero.

>I found at least one.

Please read more carefully! What you found is a call to IProducer.stopProducing, an interface method, not FileDescriptor.stopProducing, or potentially ITransport.loseConnection, an interface method, not FileDescriptor.loseConnection. Andrew is correct: there are zero calls.

stopProducing() is an implementation detail. pauseProducing() is implemented in various ways in Twisted (and some outside Twisted), not all of them relying on FileDescriptor.

For example, twisted/internet/_pollingfile.py:_PollableReadPipe implements it by calling ‘deactivate’, not ’stopReading’. This is using a completely different API (the Win32 pipes API, and Twisted timers) and is unrelated to select() state.

Therefore there are already multiple potential implementations which your code could be referring to when calling pauseProducing, and some of them do not necessarily depend on pauseProducing. In fact, it would be breaking encapsulation to even determine that.

This is the whole point of the IoC pattern you’re crowing about: to the extent dependency is possible in Python, Twisted protocols depend on ITransport and IConsumer, *not* FileDescriptor. FileDescriptor is a utility class involved in the implementation of most reactors, but not all of them, and is never visible to the Twisted programmer as a dependency.

Do you understand the difference between interfaces and concrete implementations yet?

]]>
by: Laurent Szyster http://laurentszyster.be/blog/back-to-the-future/#comment-614 Sat, 20 May 2006 16:26:29 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-614 To Andrew: >Number of implementations of stopReading, startReading, stopWriting, startWriting outside of twisted.internet: Zero. And the dependencies? When some protocol call stopProducing, this method of FileDescriptor will by default looseConnection () therefore check the state of the transport and maybe lose the connection, calling two of three from those four methods. Do *you* follow dependencies in your count? This is what I'm tracking through a twisted pile of function calls. Also, let's be clear: calling stopReading () is not implementing it. stopReading () is indeed implemented once. As a useless articulation. You could as well, everywhere it is called, directly access the FileDescriptor's reactor and pop the this FileDescriptor from the list to be polled for input. That's why it *has* to be called many times, in many places both inside and outside of twisted.internet, and that' is why it has many obscure dependencies. >Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero. I found at least one. http://twistedmatrix.com/trac/browser/trunk/twisted/web2/channel/http.py?rev=16883 at line 316 in the stopProducing method of the HTTPParser class 314 def stopProducing(self): 315 if not self.finishedReading: 316 self.channel.stopProducing() follow the call to self.channel.stopProducing () it will lead to the FileDescriptor.stopProducing which on its turn depends on stopReading () and stopWriting () through loseConnection (). Do you see the light? Regards, To Andrew:

>Number of implementations of stopReading, startReading, stopWriting, startWriting outside of twisted.internet: Zero.

And the dependencies? When some protocol call stopProducing, this method of FileDescriptor will by default looseConnection () therefore check the state of the transport and maybe lose the connection, calling two of three from those four methods.

Do *you* follow dependencies in your count?

This is what I’m tracking through a twisted pile of function calls.

Also, let’s be clear: calling stopReading () is not implementing it.

stopReading () is indeed implemented once.

As a useless articulation. You could as well, everywhere it is called, directly access the FileDescriptor’s reactor and pop the this FileDescriptor from the list to be polled for input.

That’s why it *has* to be called many times, in many places both inside and outside of twisted.internet, and that’ is why it has many obscure dependencies.

>Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero.

I found at least one.

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/channel/http.py?rev=16883

at line 316 in the stopProducing method of the HTTPParser class

314 def stopProducing(self):
315 if not self.finishedReading:
316 self.channel.stopProducing()

follow the call to self.channel.stopProducing () it will lead to the

FileDescriptor.stopProducing

which on its turn depends on stopReading () and stopWriting () through loseConnection ().

Do you see the light?

Regards,

]]>
by: Andrew http://laurentszyster.be/blog/back-to-the-future/#comment-611 Sat, 20 May 2006 15:25:27 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-611 You seem to be very confused: """Here I disagree. Let’s count. Those four Twisted interfaces stopReading, startReading, stopWriting, startWriting .... Jean Paul counted five modules in Twisted that do implement one of those four calls … outside of twisted.internet. He is missing a lot of the fun. And how many calls are there in each of those five modules. At least twelve. And what about dependencies, all those functions that depend on those calls? """ Please check your numbers, as the ones you state here are totally wrong. You can't have read Jean Paul's comment very closely if this is the conclusion you manage to draw from it. Here are the real counts: Number of implementations of stopReading, startReading, stopWriting, startWriting outside of twisted.internet: Zero. Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero. You seem to be very confused:

“”"Here I disagree. Let’s count.

Those four Twisted interfaces

stopReading, startReading, stopWriting, startWriting

….

Jean Paul counted five modules in Twisted that do implement one of those four calls … outside of twisted.internet. He is missing a lot of the fun. And how many calls are there in each of those five modules. At least twelve. And what about dependencies, all those functions that depend on those calls? “”"

Please check your numbers, as the ones you state here are totally wrong. You can’t have read Jean Paul’s comment very closely if this is the conclusion you manage to draw from it.

Here are the real counts:

Number of implementations of stopReading, startReading, stopWriting, startWriting outside of twisted.internet: Zero.

Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero.

]]>
by: Laurent Szyster http://laurentszyster.be/blog/back-to-the-future/#comment-608 Sat, 20 May 2006 13:53:29 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-608 Errata, replace "none" by "no dependencies": Readable () and Writable () have no dependencies: they are called in async_loop.async_poll directly in the main asynchronous dispatch () loop. Errata, replace “none” by “no dependencies”:

Readable () and Writable () have no dependencies: they are called in

async_loop.async_poll

directly in the main asynchronous dispatch () loop.

]]>
by: Laurent Szyster http://laurentszyster.be/blog/back-to-the-future/#comment-607 Sat, 20 May 2006 13:50:02 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-607 Hi Glyph, Thanks for not flaming. Yes I'm just "some guy who wrote a networking framework" and who "thinks Twisted is no good" ... as it could be. Not good as Medusa, hence not good as the one I forked from it. I'm standing on a giant shoulder, itself inspired by an older USENET news server implemented in C with the same design for asynchronous sockets programming. This design included an IOC defined as the readable () and writable () pair. That interface *was* replaced by four in Twisted and the IOC was unfortunately removed. Nobody can deny that, nobody did: it is written in the sources, everybody can read this critical "implementation detail". So, I'm surprised when you assert that: >He did, the various implementations of readable(), which you discarded as “framework” code. You don’t seem to think of it as a pile, because it is in the “framework”. Twisted prefers to avoid repetition everywhere, not just in “user code”. Here I disagree. Let's count. Those four Twisted interfaces stopReading, startReading, stopWriting, startWriting are all over transports, where Twisted developped a lot. Exactly like pauseProducing and resumeProducing are all over protocols. Jean Paul counted five modules in Twisted that do implement one of those four calls ... outside of twisted.internet. He is missing a lot of the fun. And how many calls are there in each of those five modules. At least twelve. And what about dependencies, all those functions that depend on those calls? In all Allegra's async_loop there are only two places where readable () and writable () are called: the two pollster functions (the one for socket.select and the one for socket.poll). Two, in one module. In all Allegra there are only three documented places where the pair is implemented: async_core, async_net, ansync_chat plus one in udp_peer of which I'm not quite sure about (I'm allways learning ...) and two pairs of decorators for it in async_limits (soon to be documented ...). Readable () and Writable () have none: they are called in async_loop.async_poll directly in the main asynchronous dispatch () loop. And you will not find in Allegra or its applications calls to something like pauseProducing () or resumeProducing (). That too is taken care of by the two implementations of readable () and writable () that implement buffered I/O for stream transport: async_net, async_chat either in the simplest way for netstrings protocols (stall only when buffer are full, stall output only when the queue is empty), or in the most convenient way for encapsulated Internet protocols (async_chat). More with less code. That's the meter for a framework. I'm sorry to disagree on something that hurts the Twisted projects and applications. But it does hurt. You can deny it for a while, but you cannot deny what is written: all those calls, all those dependencies, all that code. Look at the sources of: twisted.web2.http and look for the part that implements chunked-encoding. Now, try to figure out a way to encapsulated further. Think about Base64 encoded MIME part in an HTTP/1.1 file upload. And that's just one possible scenario for encapsulation. Think GZIP transfert, charset decoding, etc. Regards, Hi Glyph,

Thanks for not flaming. Yes I’m just “some guy who wrote a networking framework” and who “thinks Twisted is no good” … as it could be. Not good as Medusa, hence not good as the one I forked from it.

I’m standing on a giant shoulder, itself inspired by an older USENET news server implemented in C with the same design for asynchronous sockets programming.

This design included an IOC defined as the readable () and writable () pair.

That interface *was* replaced by four in Twisted and the IOC was unfortunately removed. Nobody can deny that, nobody did: it is written in the sources, everybody can read this critical “implementation detail”.

So, I’m surprised when you assert that:

>He did, the various implementations of readable(), which you discarded as “framework” code. You don’t seem to think of it as a pile, because it is in the “framework”. Twisted prefers to avoid repetition everywhere, not just in “user code”.

Here I disagree. Let’s count.

Those four Twisted interfaces

stopReading, startReading, stopWriting, startWriting

are all over transports, where Twisted developped a lot. Exactly like pauseProducing and resumeProducing are all over protocols.

Jean Paul counted five modules in Twisted that do implement one of those four calls … outside of twisted.internet. He is missing a lot of the fun. And how many calls are there in each of those five modules. At least twelve. And what about dependencies, all those functions that depend on those calls?

In all Allegra’s async_loop there are only two places where readable () and writable () are called: the two pollster functions (the one for socket.select and the one for socket.poll). Two, in one module.

In all Allegra there are only three documented places where the pair is implemented:

async_core, async_net, ansync_chat

plus one in udp_peer of which I’m not quite sure about (I’m allways learning …) and two pairs of decorators for it in async_limits (soon to be documented …).

Readable () and Writable () have none: they are called in

async_loop.async_poll

directly in the main asynchronous dispatch () loop.

And you will not find in Allegra or its applications calls to something like pauseProducing () or resumeProducing (). That too is taken care of by the two implementations of readable () and writable () that implement buffered I/O for stream transport:

async_net, async_chat

either in the simplest way for netstrings protocols (stall only when buffer are full, stall output only when the queue is empty), or in the most convenient way for encapsulated Internet protocols (async_chat).

More with less code.

That’s the meter for a framework.

I’m sorry to disagree on something that hurts the Twisted projects and applications. But it does hurt. You can deny it for a while, but you cannot deny what is written: all those calls, all those dependencies, all that code.

Look at the sources of:

twisted.web2.http

and look for the part that implements chunked-encoding. Now, try to figure out a way to encapsulated further. Think about Base64 encoded MIME part in an HTTP/1.1 file upload. And that’s just one possible scenario for encapsulation. Think GZIP transfert, charset decoding, etc.

Regards,

]]>
by: Glyph Lefkowitz http://laurentszyster.be/blog/back-to-the-future/#comment-592 Sat, 20 May 2006 06:58:34 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-592 > No. Show me one such pile in Allegra. He did, the various implementations of readable(), which you discarded as "framework" code. You don't seem to think of it as a pile, because it is in the "framework". Twisted prefers to avoid repetition everywhere, not just in "user code". > Twisted broke asyncore and Medusa. If you want this to be a productive discussion, please stop using words like "broke" and "wrong" when you cannot substantiate them. I have yet to see a single compelling argument here about a design flaw in Twisted, and you certainly haven't managed to convince anyone qualified that I respect of your views. By the way, not only Twisted developers have my respect. I think that the guys who wrote ACE, Zymb, Haboob/Sandstorm/SEDA, various MMO networking engines, and Sam Rushing's current asynchronous server (sadly closed-source) are all very, very bright. None of them use Twisted. Allegra is not in the same class as any of those projects (many of which are actually highly superior to Twisted in various ways). You have generally gestured at some areas that need improvement, but you don't clearly understand what the improvements are. > I cannot contribute sources to Twisted. Sure you can. Just port over Allegra to be a Twisted application, rather than a Medusa fork; delet all your mainloop and finalization code, and port it to use Deferreds. :). If you devoted as much energy to understanding Twisted as you did to insulting it, you'd be an expert by now. It seems as though you are having trouble understanding certain basic concepts in Twisted; however, so I can see why you might not have wanted to use it at first. However, clearly various Twisted developers are impressed by your passion, and would be willing to work through those issues with you until you did understand. > But I can contribute peer review. And I do. Eh... don't do us any favors. If you think that this peer-review is helping Twisted, you're wrong. You've attracted Twisted developers here to correct your mistakes, not to improve Twisted in response to your criticism. Your reviews are based on misunderstanding the code and mis-estimating its performance considerations. Believe me, I _LOVE_ peer review. I crave it as much as you seem to -- it is very hard to get people qualified to comment on Twisted to do so, as most of them are too busy writing massively multiplayer games or switching engines for telecom companies. However, I understand all of your criticisms and they are *not valid*. In the strictest sense, "peer" review is when qualified equals give commentary. I don't mean to be insulting when I say this, since you may be very skilled in other areas, and your writing (especially considering English does not appear to be your first language) is quite lucid and interesting: however, you have amply demonstrated that you are not a "peer" of the Twisted team when it comes to network programming and Python application architecture. You could stand to learn a lot by paying closer attention to what has been said here. The reviews JP gave alone are detailed technical analysis that would have cost tens of thousands of consulting dollars to receive. Therefore, if you are doing this to help the Twisted community improve, please stop. If you're not going to pay attention to the responses you've gotten, you're just wasting everyone's time (and perhaps tarnishing Twisted's reputation in the eyes of those who can't understand the finer technical points here and just see that "some guy who wrote a networking framework thinks Twisted is no good... maybe it isn't"). > No. Show me one such pile in Allegra.

He did, the various implementations of readable(), which you discarded as “framework” code. You don’t seem to think of it as a pile, because it is in the “framework”. Twisted prefers to avoid repetition everywhere, not just in “user code”.

> Twisted broke asyncore and Medusa.

If you want this to be a productive discussion, please stop using words like “broke” and “wrong” when you cannot substantiate them. I have yet to see a single compelling argument here about a design flaw in Twisted, and you certainly haven’t managed to convince anyone qualified that I respect of your views.

By the way, not only Twisted developers have my respect. I think that the guys who wrote ACE, Zymb, Haboob/Sandstorm/SEDA, various MMO networking engines, and Sam Rushing’s current asynchronous server (sadly closed-source) are all very, very bright. None of them use Twisted. Allegra is not in the same class as any of those projects (many of which are actually highly superior to Twisted in various ways).

You have generally gestured at some areas that need improvement, but you don’t clearly understand what the improvements are.

> I cannot contribute sources to Twisted.

Sure you can. Just port over Allegra to be a Twisted application, rather than a Medusa fork; delet all your mainloop and finalization code, and port it to use Deferreds. :). If you devoted as much energy to understanding Twisted as you did to insulting it, you’d be an expert by now.

It seems as though you are having trouble understanding certain basic concepts in Twisted; however, so I can see why you might not have wanted to use it at first. However, clearly various Twisted developers are impressed by your passion, and would be willing to work through those issues with you until you did understand.

> But I can contribute peer review. And I do.

Eh… don’t do us any favors.

If you think that this peer-review is helping Twisted, you’re wrong. You’ve attracted Twisted developers here to correct your mistakes, not to improve Twisted in response to your criticism. Your reviews are based on misunderstanding the code and mis-estimating its performance considerations. Believe me, I _LOVE_ peer review. I crave it as much as you seem to — it is very hard to get people qualified to comment on Twisted to do so, as most of them are too busy writing massively multiplayer games or switching engines for telecom companies. However, I understand all of your criticisms and they are *not valid*.

In the strictest sense, “peer” review is when qualified equals give commentary. I don’t mean to be insulting when I say this, since you may be very skilled in other areas, and your writing (especially considering English does not appear to be your first language) is quite lucid and interesting: however, you have amply demonstrated that you are not a “peer” of the Twisted team when it comes to network programming and Python application architecture. You could stand to learn a lot by paying closer attention to what has been said here. The reviews JP gave alone are detailed technical analysis that would have cost tens of thousands of consulting dollars to receive.

Therefore, if you are doing this to help the Twisted community improve, please stop. If you’re not going to pay attention to the responses you’ve gotten, you’re just wasting everyone’s time (and perhaps tarnishing Twisted’s reputation in the eyes of those who can’t understand the finer technical points here and just see that “some guy who wrote a networking framework thinks Twisted is no good… maybe it isn’t”).

]]>
by: Laurent Szyster http://laurentszyster.be/blog/back-to-the-future/#comment-586 Sat, 20 May 2006 04:17:49 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-586 >Anywhere you see pauseProducing in Twisted code, an Asyncore-based program would have a pile of custom logic to make sure it stops returning True from readable(). No. Show me one such pile in Allegra. The readable () method is implemented in a few places. And only in the framework, it will not be implemented in any application. That's the way to write less code. For the Async_dispatcher, Async_net and Async_chat class. The UDP_peer may deserve a better one, but TCP client and server channels don't need anything else. Although async_limit can decorate it (along with the recv and send interface) to meter, timeout on inactivity and throttle channels. With any protocols and transports. >I think your enthusiasm in this problem space is great, Thanks. >and I think you could make a much greater positive impact on the community if you devoted some of your efforts to contributing to Twisted instead of working on Allegra. I cannot contribute sources to Twisted. I allready forked Medusa. Which was allready not as broken as Twisted in a few key aspects for network application programmers. Twisted broke asyncore and Medusa. I'd rather continue to use and improve the original. But I can contribute peer review. And I do. Regards, >Anywhere you see pauseProducing in Twisted code, an Asyncore-based program would have a pile of custom logic to make sure it stops returning True from readable().

No. Show me one such pile in Allegra.

The readable () method is implemented in a few places. And only in the framework, it will not be implemented in any application. That’s the way to write less code.

For the Async_dispatcher, Async_net and Async_chat class. The UDP_peer may deserve a better one, but TCP client and server channels don’t need anything else. Although async_limit can decorate it (along with the recv and send interface) to meter, timeout on inactivity and throttle channels. With any protocols and transports.

>I think your enthusiasm in this problem space is great,

Thanks.

>and I think you could make a much greater positive impact on the community if you devoted some of your efforts to contributing to Twisted instead of working on Allegra.

I cannot contribute sources to Twisted. I allready forked Medusa. Which was allready not as broken as Twisted in a few key aspects for network application programmers. Twisted broke asyncore and Medusa.

I’d rather continue to use and improve the original.

But I can contribute peer review. And I do.

Regards,

]]>
by: Jean-Paul http://laurentszyster.be/blog/back-to-the-future/#comment-585 Sat, 20 May 2006 04:01:44 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-585 >But that readable () is allready called once in the loop as it runs, once in the the async_loop.async_poll function (in Allegra) and *nowhere* else, ever. On the contrary, pauseProducing () is called *everywhere* and *whenever* something, somewhere needs to stall a producer. Anywhere you see pauseProducing in Twisted code, an Asyncore-based program would have a pile of custom logic to make sure it stops returning True from readable(). If your application wants to stop reading, that's just what you have to do. The difference is that in Twisted, there is *one* way to spell this. In an Asyncore program, you have to re-implement it every time you want to say it. Look at every implementation of producer_stalled in Allegra. There are at least five of these. Each has different, unique application logic. In Twisted, each of those would unnecessary: the application would just call pauseProducing. Instead of implementing a check against protocol state to decide if the producer should currently be paused or not, the protocol can simply call a method. Take lib/smtp_client.py:108, for example. This is a terrible, terrible way to implement your SMTP client. You are smearing the logic related to recipient addresses all over the reactor and channel classes, instead of isolating it in the code which actually sends RCPT TO commands. This is a clear violation of encapsulation and tightly couples your producer to the rest of your application logic. In Twisted, there would _be_ no producer during the RCPT TO phase. Once the DATA command had been accepted, a producer would be hooked up to the protocol's transport and started. There would be no need for it to have any idea whatsoever about the rest of the SMTP conversation. > Do it. Show me one. That does not call pauseProducing and that can stall output through encapsulation. I'm not sure what you mean by "stall output through encapsulation". Encapsulation is a thing which you are not supposed to go through. There's no reason to want to avoid using pauseProducing (for the reasons given above and in my previous comments), however many of these pipelining protocol implementations don't actually have to use it anyway: http://twistedmatrix.com/trac/browser/trunk/twisted/mail/pop3.py#L253 http://twistedmatrix.com/trac/browser/trunk/twisted/mail/imap4.py#L457 http://twistedmatrix.com/trac/browser/trunk/twisted/protocols/ftp.py#L1771 http://twistedmatrix.com/trac/browser/trunk/twisted/web/http.py#L945 > I did. Want some, here is some: That code isn't tied to FileDescriptor at all. It will work with any transport that provides IProcessTransport. This is part of the reason protocol/transport separation is so useful. This code will work with any of a number of IProcessTransport implementations: it can even work with implementations which don't actually talk to a process. This is extremely useful when writing unit tests, since a deterministic implementation can be used to remove the possibility of spurious, environment or load related failures from the test suite. This lets application developers unit test their *protocol* implementation separately from the transport implementation, which may have unrelated bugs, be swapped out with various different versions on different platforms, be replaced entirely by new versions of Twisted, or be provided by a third party who doesn't contribute directly to Twisted or that particular application developer's project. In fact, FileDescriptor itself is an implementation detail. One can hook a protocol up to any kind of transport. Several of the reactors included in Twisted share a core set of implementation modules and re-use FileDescriptor for various means because this reduces code-size. However, not all Twisted reactors use FileDescriptor: take a look at the IOCP reactor, for example. > but it does not really look like independant from self.transport That's right. It relies on the transport attribute of the protocol object, because all protocol objects have a transport object, and they are guaranteed that it will behave in a particular way. What exactly this object is may vary, and this is exactly the reason protocol/transport separation is useful. The protocol doesn't need to know if it is talking to a TCP socket, an SSL connection, a reliable layer on top of UDP, a child process, its process's standard output, a unix socket, a FIFO or other pipe, a test fixture, or anything else your heart desires. > and heavily rely on defered to work around something. It's not working around anything. It's using a Deferred to keep track of the bulk data transfer it has initiated. When the transfer completes, the Deferred fires. When the Deferred fires, the code in question closes the standard input of the process it is talking to. This is necessary to correctly implement a CGI gateway for the web2 server. Note how it does these things in four lines of code which aren't duplicated anywhere else. These are the four lines necessary to send a stream to a child CGI process. Not four lines plus half a dozen more that are just wasteful code duplication, as seems common throughout Asyncore-based programs. > Sorry to be mean, but remember I’m mean with code, not with you. Don't worry. I don't think you're being mean. I think you see some problems in Twisted and you're trying to make them known. This can be constructive if done in the right way. And I can certainly take criticism of Twisted without being personally offended. However, I don't think you have actually brought up any valid criticisms. Most of your posts and most of the designs I've seen in Allegra seem based on misunderstandings of Twisted's relatively sophisticated solutions to some important problems in networking. I think your enthusiasm in this problem space is great, and I think you could make a much greater positive impact on the community if you devoted some of your efforts to contributing to Twisted instead of working on Allegra. >But that readable () is allready called once in the loop as it runs, once in the the async_loop.async_poll function (in Allegra) and *nowhere* else, ever. On the contrary, pauseProducing () is called *everywhere* and *whenever* something, somewhere needs to stall a producer.

Anywhere you see pauseProducing in Twisted code, an Asyncore-based program would have a pile of custom logic to make sure it stops returning True from readable(). If your application wants to stop reading, that’s just what you have to do. The difference is that in Twisted, there is *one* way to spell this. In an Asyncore program, you have to re-implement it every time you want to say it.

Look at every implementation of producer_stalled in Allegra. There are at least five of these. Each has different, unique application logic. In Twisted, each of those would unnecessary: the application would just call pauseProducing. Instead of implementing a check against protocol state to decide if the producer should currently be paused or not, the protocol can simply call a method.

Take lib/smtp_client.py:108, for example. This is a terrible, terrible way to implement your SMTP client. You are smearing the logic related to recipient addresses all over the reactor and channel classes, instead of isolating it in the code which actually sends RCPT TO commands. This is a clear violation of encapsulation and tightly couples your producer to the rest of your application logic.

In Twisted, there would _be_ no producer during the RCPT TO phase. Once the DATA command had been accepted, a producer would be hooked up to the protocol’s transport and started. There would be no need for it to have any idea whatsoever about the rest of the SMTP conversation.

> Do it. Show me one. That does not call pauseProducing and that can stall output through encapsulation.

I’m not sure what you mean by “stall output through encapsulation”. Encapsulation is a thing which you are not supposed to go through. There’s no reason to want to avoid using pauseProducing (for the reasons given above and in my previous comments), however many of these pipelining protocol implementations don’t actually have to use it anyway:

http://twistedmatrix.com/trac/browser/trunk/twisted/mail/pop3.py#L253
http://twistedmatrix.com/trac/browser/trunk/twisted/mail/imap4.py#L457
http://twistedmatrix.com/trac/browser/trunk/twisted/protocols/ftp.py#L1771
http://twistedmatrix.com/trac/browser/trunk/twisted/web/http.py#L945

> I did. Want some, here is some:

That code isn’t tied to FileDescriptor at all. It will work with any transport that provides IProcessTransport. This is part of the reason protocol/transport separation is so useful. This code will work with any of a number of IProcessTransport implementations: it can even work with implementations which don’t actually talk to a process. This is extremely useful when writing unit tests, since a deterministic implementation can be used to remove the possibility of spurious, environment or load related failures from the test suite. This lets application developers unit test their *protocol* implementation separately from the transport implementation, which may have unrelated bugs, be swapped out with various different versions on different platforms, be replaced entirely by new versions of Twisted, or be provided by a third party who doesn’t contribute directly to Twisted or that particular application developer’s project.

In fact, FileDescriptor itself is an implementation detail. One can hook a protocol up to any kind of transport. Several of the reactors included in Twisted share a core set of implementation modules and re-use FileDescriptor for various means because this reduces code-size. However, not all Twisted reactors use FileDescriptor: take a look at the IOCP reactor, for example.

> but it does not really look like independant from self.transport

That’s right. It relies on the transport attribute of the protocol object, because all protocol objects have a transport object, and they are guaranteed that it will behave in a particular way. What exactly this object is may vary, and this is exactly the reason protocol/transport separation is useful. The protocol doesn’t need to know if it is talking to a TCP socket, an SSL connection, a reliable layer on top of UDP, a child process, its process’s standard output, a unix socket, a FIFO or other pipe, a test fixture, or anything else your heart desires.

> and heavily rely on defered to work around something.

It’s not working around anything. It’s using a Deferred to keep track of the bulk data transfer it has initiated. When the transfer completes, the Deferred fires. When the Deferred fires, the code in question closes the standard input of the process it is talking to. This is necessary to correctly implement a CGI gateway for the web2 server. Note how it does these things in four lines of code which aren’t duplicated anywhere else. These are the four lines necessary to send a stream to a child CGI process. Not four lines plus half a dozen more that are just wasteful code duplication, as seems common throughout Asyncore-based programs.

> Sorry to be mean, but remember I’m mean with code, not with you.

Don’t worry. I don’t think you’re being mean. I think you see some problems in Twisted and you’re trying to make them known. This can be constructive if done in the right way. And I can certainly take criticism of Twisted without being personally offended. However, I don’t think you have actually brought up any valid criticisms. Most of your posts and most of the designs I’ve seen in Allegra seem based on misunderstandings of Twisted’s relatively sophisticated solutions to some important problems in networking.

I think your enthusiasm in this problem space is great, and I think you could make a much greater positive impact on the community if you devoted some of your efforts to contributing to Twisted instead of working on Allegra.

]]>
by: Laurent Szyster http://laurentszyster.be/blog/back-to-the-future/#comment-563 Fri, 19 May 2006 23:02:39 +0000 http://laurentszyster.be/blog/back-to-the-future/#comment-563 hmm, let's see: > Yep. One uniform interface for one single task. Just like readable() is one uniform interface in Asyncore. But that readable () is allready called once in the loop as it runs, once in the the async_loop.async_poll function (in Allegra) and *nowhere* else, ever. On the contrary, pauseProducing () is called *everywhere* and *whenever* something, somewhere needs to stall a producer. > There are quite a few Twisted implementations of pipelined protocols. Nothing about Twisted’s model precludes this feature. It’s quite easy to implement. Do it. Show me one. That does not call pauseProducing and that can stall output through encapsulation. Something like chunked-encoding for an HTTP/1.1 client and server. Here is one: http://svn.berlios.de/svnroot/repos/allegra/lib/http_reactor.py I would love somebody to trample it ;-) > Take a look at the web2 code. None of it is coupled to FileDescriptor at all. I did. Want some, here is some: http://twistedmatrix.com/trac/browser/trunk/twisted/web2/stream.py?rev=16875 go in: 789 class _ProcessStreamerProtocol(protocol.ProcessProtocol): and more specifically at line 805 of the connectionMade () method 797 def connectionMade(self): 798 p = StreamProducer(self.inputStream) 799 # if the process stopped reading from the input stream, 800 # this is not an error condition, so it oughtn't result 801 # in a ConnectionLost() from the input stream: 802 p.stopProducing = lambda err=None: StreamProducer.stopProducing(p, err) 803 804 d = p.beginProducing(self.transport) 805 d.addCallbacks(lambda _: self.transport.closeStdin(), 806 self._inputError) I'm not sure to understand exactly what this is, but it does not really look like independant from self.transport and heavily rely on defered to work around something. But what? Can you explain exactly what this piece of code is doing? Sorry to be mean, but remember I'm mean with code, not with you. Thanks for replying. hmm, let’s see:

> Yep. One uniform interface for one single task. Just like readable() is one uniform interface in Asyncore.

But that readable () is allready called once in the loop as it runs, once in the the async_loop.async_poll function (in Allegra) and *nowhere* else, ever. On the contrary, pauseProducing () is called *everywhere* and *whenever* something, somewhere needs to stall a producer.

> There are quite a few Twisted implementations of pipelined protocols. Nothing about Twisted’s model precludes this feature. It’s quite easy to implement.

Do it. Show me one. That does not call pauseProducing and that can stall output through encapsulation. Something like chunked-encoding for an HTTP/1.1 client and server. Here is one:

http://svn.berlios.de/svnroot/repos/allegra/lib/http_reactor.py

I would love somebody to trample it ;-)

> Take a look at the web2 code. None of it is coupled to FileDescriptor at all.

I did. Want some, here is some:

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/stream.py?rev=16875

go in:

789 class _ProcessStreamerProtocol(protocol.ProcessProtocol):

and more specifically at line 805 of the connectionMade () method

797 def connectionMade(self):
798 p = StreamProducer(self.inputStream)
799 # if the process stopped reading from the input stream,
800 # this is not an error condition, so it oughtn’t result
801 # in a ConnectionLost() from the input stream:
802 p.stopProducing = lambda err=None:
StreamProducer.stopProducing(p, err)
803
804 d = p.beginProducing(self.transport)
805 d.addCallbacks(lambda _: self.transport.closeStdin(),
806 self._inputError)

I’m not sure to understand exactly what this is, but it does not really look like independant from self.transport and heavily rely on defered to work around something. But what? Can you explain exactly what this piece of code is doing?

Sorry to be mean, but remember I’m mean with code, not with you.

Thanks for replying.

]]>