Allegra » Blog Archive

producer.py

The twelfth module of Allegra is a collection of general purpose producers. It provides a simplistic file producer, a scanning producer for large strings, a stallable producer for generators, a stallable composite producer and a stallable tee producer.

Sources: lib/producer.py

Synopsis

Producers are the basic building blocks used by async_chat channels to produce buffered output. Pushed on a channel’s output_fifo deque, they may produce more data to be buffered or stall the channel’s output.

A stallable producer must implement two interfaces:

more ()
producer_stalled ()

The first is a method that returns a non-empty 8-bit byte string if there is more output data to buffer, or the empty string if there is no more data to produce.

The second is a method that returns False is there is more output data ready or True if there is none, ie if the producer would block.

Produce file

The first class provided by this module is a simplistic file producer:

from allegra import producer
	
p = producer.File (open ('filename', 'rb'))

that takes a file-like object as first argument, or any instance with a read (length) method that returns 8-bit byte strings. By default it will produce chunks of 16KB (the most common maximum buffer size for TCP sockets) but you can specify your own chunk size:

p = producer.File (f, 4096)

Beware that such producer may block the asynchronous loop if used in an application with high concurrency or one serving a slow file system (like a USB memory stick for instance, a congested network share, etc).

Scan strings

The next producer class is intended to produce large strings in smaller chunks without consuming too much memory, scanning the string:

p = producer.Simple (
    'a possibly large 8-bit byte string ...'
    )

Like the File producer it produces 16KB chunks by default, but you can adjust that size to fit your channel’s output buffer size:

p = producer.Simple (s, 4096)

It is a replacement for the original Simple_producer found in asynchat.py and Medusa. Note that it obviously cannot be stalled and will never block.

Stall, generate and react

The third producer class is designed to produce output from a generator:

def produce ():
    yield 'A simple 8-bit bytes string\r\n'
	
    yield 'Yet another string\r\n'
	
p = producer.Stalled_generator ()

stalling as long as no generator has been attributed to the producer instance:

p.generator = produce ()

It is a very practical way to defer output producing when a channel is writable.

Also, it provides a simple interface to stall output as the application waits for another asynchronous event. A common use of Stalled_generator in the Allegra’s applications is to subclass it, overriding the __call__ method:

class React (producer.Stalled_generator):
    def __call__ (self, request):
        self.generator = produce ()

and set an instance of this producer class as finalization of an asynchronous request object:

request.finalization = React ()

In effect, the React producer can be pushed immediately for output but will stall until the request is finalized.

Compose

The fourth producer in the collection is designed to compose output from 8-bit bytes strings and stallable producers:

head = 'Subject: test\r\n\r\n'
def body ():
    yield 'A simple 8-bit bytes string\r\n'
	
    yield producer.File (f = open ('filename', 'rb'))
	
    yield 'Yet another string\r\n'
	
    yield producer.Simple ('A producer\r\n')
	
p = producer.Composite (head, body ())

Note that the composite producer buffers strings produced until a stalled producer is found. Here too, the default size of chunks globbed is set to 16KB, but you can specify your own:

p = producer.Composite (head, body (), 4096)

The composite producer is intended to assemble and serialize appropriately output like MIME messages, XML documents and any other output stream composed of a head string and a body generator.

Tee

The last producer provides an effective way to buffer and tee the output of another producer as it is produced:

p = producer.File (open ('filename', 'rb'))
t1 = producer.Tee (p)
t2 = producer.Tee (p)

Practically the Tee_producer is intended to tee output as it is cached, allowing for instance a web server to serve the same resource simultaneously to many clients.

Applications

This short collection of producers covers the most common use case that emerged from the development of Allegra’s peer applications. The stallable producer interface itself has more applications.

Further in the library you will find specialized implementations, like HTTP/1.1 chunked-encoded or SMTP escaping wrappers.

The great benefit of producers is that they are easy to "box", allowing developers to implement output protocols as independant classes of which instances can be composed at will to produce complex output.

This entry was posted on Monday, May 15th, 2006 at 4:26 pm and is filed under Allegra, Documentation, Python. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “producer.py”

Rene Dudfield Says:
May 16th, 2006 at 3:51 am
I think producer_stalled() has a negative in it. I like to keep negatives out, and left negatives be shown with not. I think this makes it not less clear… I mean more clear ;)

eg. not producer_stalled() means that there is not not some more.

Maybe some_more() instead?
Laurent Szyster Says:
May 17th, 2006 at 3:55 pm
To Rene:

There are at least two reasons I can think of for not changing that interface into a “positive” one. The first is that one should not fix what ain’t broken. I found producer_stalled in the experimental section of Medusa sources and kept it as it was. The second reason is that use case of the interface is indeed a “negative” as found in the Async_chat.writable () method.

Anyway, thanks for the comment and suggestion.
Rene Dudfield Says:
May 18th, 2006 at 1:26 am
Thinking about it more…

more() should raise an exception if it is going to block. This is to avoid a race condition.

It is the old peek, and use pattern.

if producer_stalled():
# in this time between method calls the producer may not be stalled.
# meaning you miss a chance to call more()
more()

In practice it might not be an issue.

Here is the other way…
try:
bla = more()
except ProducerStalled:
print “moving on!”

I’d like to hear why it isn’t a race condition. Or if it is, why it doesn’t matter.
Laurent Szyster Says:
May 18th, 2006 at 10:32 am
To Rene:

Avoiding an error condition on more () when it would block is achieved by excluding the channel from the list of socket polled for output when the first producer in the channel’s output_fifo is stalled.

Developers don’t need to worry about producer stalling in an async_chat channel, because its writable () and handle_write () methods are carefully designed so that such error conditions do not occur. If the first producer is stalled, writable() will be false and more() won’t be called in the event loop.

If you do have to call more () somewhere else, independantly from the event loop, it will most probably be in an accumulator like the composite producer when it globs output. Then, but it is a rare case, you will have to check for the producer beeing stalled.

Regards,

Synopsis

Produce file

Scan strings

Stall, generate and react

Compose

Tee

Applications

4 Responses to “producer.py”

microsoft office 2003 oem

how to activate reget deluxe

Oem software bundle oem software bundles 814.

microsoft oem software cd

microsoft office 2003 oem

software oem cd line business

The Logo Creator MEGA pak 3.6

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem cd line business

Mcafee all in 1 2006

inkjet definition oem software

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

microsoft window 2000 professional oem software

The Logo Creator MEGA pak 3.6

microsoft office software oem;

Oem software bundle oem software bundles 814.

The Logo Creator MEGA pak 3.6

The Logo Creator MEGA pak 3.6

oem software cds;

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

inkjet definition oem software

Buy cheap PhotoShop CS 2

microsoft office 2003 oem

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

software oem cd line business

Buy cheap After Effects 5.5 Production Bundle

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

inkjet definition oem software

iCorrect EditLab Pro 4.52

iCorrect EditLab Pro 4.52

locate software oem files

inkjet definition oem software

Oem software bundle oem software bundles 814.

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

locate software oem files