Allegra » Blog Archive

To Glyph

A provocative Note to Twisted Users at the end of async_limits.py quickly lead comments out of topic, ending in another dispute between Glyph and me. People interested in the argument should follow up here for more.

About Critiques

If you had been seriously interested in criticizing Twisted rather than just maligning it, you would have posted a comparison of code complexity rather than saying "here is how you do it in Allegra, I guess maybe it is worse in Twisted".

My critiques of Twisted sources could not be more serious: they are substantiated with alternative code. And I don’t guess about Twisted, I compare the sources. For instance in the specific case of throttling, Twisted required 4 classes, 48 methods and 238 lines of code (from line 22 to 259):

/twisted/protocols/policies.py

while Allegra only needs 10 functions and 150 lines of code (from line 27 to 77 and from line 105 to 203):

async_limits.py

Do my criticism harm Twisted sources, developers or applications? Rather the contrary. I raised five design issues and provided functional equivalent in a framework built on asyncore and Medusa design.

Twisted developers could use that point of comparison to improve Defered implementation performances, fix FileDescriptor with a new couple, enhance your protocol and transports implementation, move GUI integration to a separate trunk and rework thread support.

If they don’t and instead go back to the future, who is to blame?

My critiques of Twisted or your denial of the problems?

About Malice

Allegra is developed on a design that predates Twisted and was never abandonned by the Python community. Medusa is still popular, asyncore.py and asynchat.py are still the standard asynchronous network programming API for CPython.

Is there any malice in asserting that fact? Is there any malice in trying to improve Medusa for network peer programming? The tone of my review of Twisted sources is provocative and their substance may be harmfull to your reputation, but where is the malice?

In my critiques of Twisted sources compared to Allegra?

Or in your cover-up of a failure to improve on asyncore and Medusa?

About Simplicity

For example, here is an example of why an echo server in Asyncore is 4x as complicated as in Twisted: http://twistedmatrix.com/products/echo_server

Ironically, your example of two echo server applications is the refutation of the claim that goes along the product. It is supposed to promote the simplicity of Twisted API, but behind simple interfaces there is a very complicated implementation.

Count all the lines of codes on which the Twisted application directly depends, all the sources that implement the API applied. Then do the same for the application of asyncore and compare the two figures.

If less code is a metric of simplicity of application and implementation of an API, that example was very unfortunate.

Do you have another metric?

About Proof

Every time you make a claim about it, some basic information (like “here is how you activate a throttler”) seems to prove you wrong.

When did you prove my critics wrong?

Have you benchmarked applications of finalization vs. Defered? Can you infirm my hypothesis and provide sources of such benchmarks to be reviewed by your peers?

Can you provide evidence that once applied to protocols like HTTP/1.1 the asyncore readable and writable are indeed less efficient than the FileDescriptor’s "fab four"?

Can you substantiate your claim that Medusa or Allegra’s implementation of HTTP/1.1 is more complicated that Twisted, with an objective metric like line count?

Do you know how fast Twisted threads and thunks back in the asynchronous loop? Have you benchmarked those figures against Medusa or Allegra implementations of select_trigger and thread_loop?

About Arguing

Please answer with facts, metrics, sources and benchmark figures.

This entry was posted on Monday, May 29th, 2006 at 8:53 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.

15 Responses to “To Glyph”

Sylvain Says:
May 29th, 2006 at 10:04 am
Laurent, the question is “do have you benchmark figures”? AFAUK, you are the one having a go at Twisted therefore you should be the one coming up with real figures. It’s not up to the Twisted guys to waste their time to do your work.
Laurent Szyster Says:
May 29th, 2006 at 10:19 am
To Sylvain,

>AFAUK, you are the one having a go at Twisted therefore you should be the one coming up with real figures.

Not figures: test sources.

Peer reviewer will come up with figures ;-)

>It’s not up to the Twisted guys to waste their time to do your work.

I allready did what Twisted guys should have done long ago: review the code with a critical eye and meter its complexity versus an alternative design developed on asyncore.

Don’t worry, if nobody else produce a benchmark suite for Allegra and Twisted core, I will that do for Twisted people too.

Regards,
Antoine Says:
May 29th, 2006 at 11:09 am
> Don’t worry, if nobody else produce a benchmark suite for Allegra and Twisted core, I will that do for Twisted people too.

Yes, please. (And publish the sources of the benchmark so we can see for ourselves.)

For the moment, the tone of your writing are not to my taste at all, and I find it difficult to take Allegra seriously. If there is some neutral comparison, perhaps this would change.
Glyph Lefkowitz Says:
May 29th, 2006 at 11:52 am
This is the last comment I am going to make here. I had two purposes when I first started commenting on this blog: one was to try to clear up some misconceptions about Twisted for your readers and the other was to convince you to use Twisted and to help us improve Twisted’s HTTP layer, which certainly needs more people to work on it, rather than duplicating a bunch of effort unnecessarily.

While I have succeeded at the first goal, I seem to have failed dismally at the second. I have gone so far as to offer an extensive free tutoring session, which you turned down, apparently because you believe you have nothing substantial to learn about Twisted. If nothing you have seen so far in my comments, and the comments of other Twisted contributors here has convinced you, then there must be some sort of communication barrier that I can’t understand here. Perhaps linguistic, perhaps pride is simply blinding you to the truth, or perhaps Twisted just doesn’t “fit your brain”, but I don’t know what I can say that I haven’t already said to prove this to you: you don’t understand Twisted’s design and you aren’t formulating cogent criticisms of it; without this understanding, Allegra is doomed to be nothing but a poor re-implementation of half of Twisted.

>in the specific case of throttling, Twisted required 4 classes, 48 methods and 238 lines of code (from line 22 to 259)

This comparison is wrong in so many ways it is difficult to decide where to begin.

Let’s start with the raw numbers: even the way you’ve counted is debatable. The following measurement has problems too, since neither example was constructed to be comparable to the other, but just to make the point that the implementations are comparable: if we use a tool intended for source code review (david wheeler’s “sloccount”) rather than newline counting, and compare twisted/protocols/htb.py (which tries to limit itself to throttling, instead of a variety of different tasks) to async_limits.py, we end up with 211 SLOC in allegra vs. 104 SLOC in Twisted. QED: Twisted’s implementation is actually shorter.

Secondly, it is difficult for me to review async_limits at all, because while twisted.protocols.policies has test cases for you to read (in twisted.test.test_policies) there does not appear to be any test coverage for async_limits. Coupled with a comprehensive lack of documentation for allegra — I mean, people complain about Twisted’s docs being bad, but it has hundreds of pages of documentation; allegra’s doc/ directory is literally empty — it is rather hard to determine the intent of the code. The few tests that do exist for allegra appear to be rather brief and haphazard, not using a standardized testing methodology. If you are intending to make an analogy to the scientific community, with peer review and replicatable experiments and soforth, test cases are a baseline requirement, not to mention papers and such. (You can find papers and talks on Twisted in doc/historic in the repository.)

The comparison also doesn’t make sense because neither of these was an example designed to be compared. Twisted has additional code in the twisted.protocols.policies implementation in part because there is a comprehensive wrapper utility class, which is arguably not really necessary for throttling, but is necessary for cooperative application of wrappers. async_limits applies its “decorators” destructively; what happens when two different “decorating” implementations want to modify the same dispatcher? kaboom. Maybe that’s acceptable for your use-cases, but a similar implementation using mutation and dynamic tricks in Twisted might be shorter.

> behind simple interfaces there is a very complicated implementation.

Yes. That’s rather the point. I would not make any claims about Twisted’s _implementation_; my point has always been that Twisted’s way makes _application_ code shorter and simpler. I don’t understand why you think this is a problem. There are lots of good reasons that some parts of Twisted’s implementation is quite hairy, and I imagine that as time goes on, this trend is only going to continue; bits of the reactor will slowly be re-implemented in C, or Pyrex, or Twisted will include its own (optional, of course) socket module for better performance.

For another example, would you rather that Python’s implementation be simpler, so that you have to do your own garbage collection? The GC in Python is quite sophisticated, a combination of numerous techniques blended to get optimal performance and simplicity _from the application layer_. Do you want to have to re-write half of ceval2.c in every Python application? That is also a large, complex file.

>Do my critics harm Twisted sources, developers or applications?

No, but only because they’re being ignored. They’re categorically, systematically incorrect critiques and if they were being believed, yes, it would do harm to Twisted. In particular, Twisted is an integration platform for asynchronous code and it is already an uphill battle to get people to accept the event-driven programming style; discussions like these exhort people to use an incompatible system to write asynchronous code.

The more code is written for legacy systems like Medusa and Allegra the more work has to be done to update them to use Twisted in the future. This does generate unnecessary busy-work for Twisted developers, and it reduces the overall value of the Twisted platform because there would be fewer protocol implementations available to Twisted, and work would be divided between maintaining Twisted protocols and maintaining Allegra protocols, rather than unifying this effort under one banner.

Lastly, of course, you are wasting my time, which could be spent fixing up Twisted. So, as I said at the beginning, this is my last comment here. From now on, when I see a post here which I feel like replying to, I will go fix a bug in Twisted instead.

> Please answer with facts, metrics, sources and benchmark figures.

Establishing facts, metrics (etc etc) is a lot of work. If you started this process; for example, creating benchmarks, writing example code to be compared, function point analysis of both codebases or similar, I might continue to participate in the discussion.

Instead, you toss in a half-understood figure every so often and expect the Twisted community to respond with a slew of well-considered benchmarks and line-by-line code comparisons. Our code has tests - yours does not. Our code has documentation - yours does not. Not only are you asking us to do more work, you are not providing us with the tools that are necessary to make it a reasonable amount of work; your testing Twisted would be substantially easier than us testing Allegra.

If you want such a serious treatment of your ideas, you have to be the first one to invest this effort. Although I have failed in my goal to educate you, I have been putting in the same level of effort on these comments as you have on your entries: a few links here and there, some high-level analysis, but not a comprehensive test suite for allegra and a matching one for Twisted. The burden of proof lies with *you*, and you are not meeting it sufficiently to merit such a time-consuming response.

>Do you know how fast Twisted threads and thunks back in the asynchronous loop? Have you benchmarked those figures against Medusa or Allegra implementations of select_trigger and thread_loop?

I hate to pick on language, but the phrase ‘threads and thunks back’ doesn’t even make sense. Is this some multithreading benchmark? Twisted doesn’t start threads unless the user asks, and all it does in that case is a very simple Queue.put/Queue.get. Perhaps if you had written some code to test this in Twisted I would know what you were talking about.

Although I have benchmarked against Medusa in the past, I sadly don’t have any benchmarks or results on hand; I believed Twisted’s approach to be demonstrated to be superior long ago.

I somehow doubt you would take me at my word, but your readers might: Twisted does scale up better; especially Twisted 2.0+, which has a number of creative performance improvements in the areas of buffering and scheduling. I can say that the results of these benchmarks is one reason that Pythondirector recommends Twisted rather than asyncore.

Since your efforts in this area are considerably more recent: have *you* done any benchmarks? Have you measured allegra’s performance relative to a comparable Twisted application? If you are going to do this, it would be good to have some simple baseline applications to work from and understand the results of before doing the benchmark I know you are dying to, HTTP/1.1 hits per second… an HTTP server is a substantially more complicated piece of code and has a lot more performance-sensitive paths than, for example, a port forwarder.

Intellectually, I know that Twisted can be very high-performance, but that doesn’t mean it _is_. There may be serious performance bugs lurking somewhere in the reactor. Your high-level, mostly-incorrect reading of the code doesn’t help me fix that though. Some benchmarks and test results might have.

>Can you provide evidence that once applied to protocols like HTTP/1.1 the asyncore readable and writable are indeed less efficient than the FileDescriptor’s “fab four”?

HTTP is not a special protocol in this regard. It’s popular, but it just sends bytes over connections like any other protocol would.

First let me repeat again: Twisted’s interface is pauseProducing and resumeProducing. (start/stop/Reading/Writing) is an implementation detail which could quite likely change.

Second, although JP explained this already, I’ll give it one more try. A stateful reactor interface is theoretically more efficient. You can read about a low-level one here that has done extensive testing on that theory: . A stateful multiplexing interface can keep a list of sockets to be polled in an efficient data structure and only invoke user code when it knows data needs to be written. This is O(S*N) per iteration, where S is the factor of the static, direct state test and N is the number of file descriptors. readable() and writable() are less efficient because they must call a method on every iteration, making them O(D*N), where D is the factor of the indirect, dynamic method call to determine readability or writability. S is always smaller than D, because S is one static check and D is, at the very least, an additional method call to perform that same check. With a really efficient multiplexing API, implemented entirely in C and using an efficient, stateful API like epoll or kqueue, S can be as much as 1000x less than D. Finally, it’s actually even *faster* than this comparison might suggest, because we can reduce N and not perform S every mainloop iteration by removing file descriptors which do not need to be polled from the active list depending on their state.

Finally, about malice:

>In my critics of Twisted sources compared to Allegra?
>Or in your cover-up of a failure to improve on asyncore and Medusa?

This is simply insulting. Numerous projects have independently evaluated Twisted and Medusa and found Twisted to be technologically superior. We make no effort to censor discussion, even long, painful, useless threads like the ones relating to certain patches on the mailing list. In fact, simply by participating on this blog we have granted it more legitimacy than it in fact deserves.

The problem with your criticisms is not that they’re wrong, it’s that they’re based on a total misunderstanding of Twisted. My offer of a tutorial session remains open, if you are willing to try to get past your prejudice; obviously you are keen to offer useful criticism, and Twisted can use a lot more of that to improve. In order for criticism to be useful, however, it must be **correct**.

PS:

>About Critics

I think you mean “critiques”, or perhaps “criticisms”

http://www.dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=critic
http://www.dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=critique
http://www.dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=criticism
Tim Parkin Says:
May 29th, 2006 at 12:53 pm
Laurent, I’m not going to go into which framework is better or who should be doing what work. What I will say is that your ongoing baiting of the twisted community is becoming a little tiresome to me (I can only imagine that some other people may find it so too). Perhaps you should do something useful with your time, I’m sure the people using your library would prefer you updating it/writing better documentation/profiling it etc rather than holding trying to stage a pissing contest with a framework that arguably isn’t even targetted at the same problems.

I also think line count isn’t necessarily an indicator of complexity. Only in certain situations should it be considered seriously and even then it’s only a contributing factor.

If you really want to continue the comparison at least produce metrics that are unarguable based on code written to solve a real world application. Even then, your choice of example could be criticised for being partial.

The conclusion should be that the best way to approach these situations is cooperatively, then both frameworks may gain and the python community could be educated about the benefits and disadvantages of each system.

Unfortanately, the bias that has already been demonstrated will nullify any perception of impartiallity. My best advice would be to leave well alone and keep writing code.
Andrew Says:
May 29th, 2006 at 12:58 pm
“Count all the lines of codes on which the Twisted application directly depends, all the sources that implement the API applied. Then do the same for the application of asyncore and compare the two figures.”

That’s a ridiculous metric. Twisted has the capability to do more than Allegra, so it will mostly likely have more code behind the scenes for “the same” application. For example, Twisted supports pipes to subprocesses as well as TCP sockets. It works with GUI event loops. It has all sorts of things. But who cares?

I don’t care precisely which lines, let alone how many, of the Python standard library are needed to execute a Twisted echoserver vs. an Allegra one? Why not libc and the kernel while we’re at it… oh, and clearly any Python program that can still run on Python 1.4 is superior because the code it depends on in Python is so much simpler.

If I’m writing an application with Twisted, the details of how it provides its functionality are irrelevant, unless Twisted screws up. The costs of more lines of code are costs to the developers of Twisted, not the developers of Twisted applications. (Except for rare cases when you really care about how much disk space your libraries use.)

We could say “we bear the burden of writing more complex code for scalable, feature-rich, cross-platform support so that you don’t have to.” That’d be an over-simplification (no framework is magically going to make a random application scalable or portable, for instance), but not entirely wrong…

Mistaking lines of code as reliable measure of speed, simplicity of API, quality, or even functionality is, well, a mistake.

Lines of code it takes to write a given program, however, is often a fair indicator of the expressiveness or richness of a library or language. Far from perfect of course, particularly with toy examples.

It’s much harder, but far more illuminating, to compare real world programs, rather than toy examples and hypotheticals. I would very much like to see “here’s X non-trivial application written in Twisted, and here’s what it looks like converted to use Allegra, and here’s how it’s better/worse/different.”, or vice versa.
Laurent Szyster Says:
May 29th, 2006 at 6:11 pm
To Antoine:

You are right, this dispute will be fruitless until sources of benchmarks have been published.

To Tim:

You are right too, I should rather work on Allegra sources instead of spending time and energy in criticisms of Twisted.

To Glyph:

Thanks for improving my english. As for the rest of your comment, I’m afraid it falls short of a substantiated response to my questions.

You keep evading criticism by comparing apples and oranges. I metered Twisted policies and Allegra async_limits precisely because they implement comparable features: decorating, metering and throttling.

Twisted htb should be compared to async_limits’ five functions dedicated to throttling only.

Also, it is dishonest to brush aside the metrics about the number of classes and methods required to implement comparable features. They do matter as much as SLOC count in the measure of code complexity.

Another pattern in your response is to label every questionable design and implementation choices I pointed in Twisted as “implementation details”. Let me remind you that in software “the devil is in the details”.

As for benchmarks, the question is simple: why in five years none have been produced for Twisted? You have been claiming better performances than asyncore all that long, but why are you waiting for someone else to publish benchmark sources of Twisted versus an application of asyncore like Medusa’s HTTP server?

Even your theoretical discussion of readable/writable vs the fab four is dubious. You claim that:

O(S*N) < O(D*M)

but you don’t take the into account the fact that readability/writability of sockets must be tested in Twisted too, although not in the I/O poll loop. As Sam Rushing explained very well in

http://www.nightmare.com/medusa/async_tweaks.html

checking for the state of all dispatchers will indeed pegg the CPU, but only when there is nothing else to do, when most I/O are stalled.

So, Twisted design may theoretically scale up better than asyncore in only one precise use case: when there is a majority of concurrent stalled connections for simple protocols and applications.

However, in practice this design imply to spend more time testing and setting the readability and writability state of FileDescriptors in the protocol and transport implementations, possibly more than once per I/O loop for buffered transport like streams and pipelined protocols like HTTP/1.1.

My educated guess is that the performance cost of readable/writable in that specific case is at least partialy offset by the additional code required by Twisted “fab four” in applications, protocols and transports sources. Benchmarks of the HTTP/1.1 servers developed on the two competing design should settle that part of the dispute.

What can allready be asserted by reviewing the sources is that the potential gain in scalability requires to pay a high price in term of complications of the implementation. So much that applications developed with Twisted must rely on Defered to decouple protocols implementation from the I/O event loop, for instance when implementing stream producers for web2:

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/stream.py

Regards,
Florian Says:
May 30th, 2006 at 12:49 am
Ahum, yet another framework flamewar.

If you want my opinion, and I know you don’t, ya allegra is missing a *lot* of docs.

On the other hand, you’re trying to argue with Glyph, I mean seriously what do you expect?

Twisted, doesn’t the name say it all?
Laurent Szyster Says:
May 30th, 2006 at 1:16 am
Hi Florian,

Nice to have you back in the comments.

Your opinion does matter. And yes, Allegra is still missing a lot of docs.

The purpose of blogging an article about each module is precisely to document the sources and at the same time give them some exposure to peer review.

Another thing I would tend to agree with you is the fact that Twisted is named appropriately and that Glyph has a very particular way to read and defend his own sources.

Regards,
Antoine Says:
May 30th, 2006 at 10:37 am
Well. That’s enough.

Franchement Laurent, stop talking about Twisted. I agree with Glyph when he said (somewhere above or in an other thread) that you should stop bickering about Twisted and getting work done on Allegra instead.

Allegra is better than Twisted ? Fine. Prove it. Write a bunch of examples and let the truth speak for itself.
Laurent Szyster Says:
May 31st, 2006 at 2:32 pm
Cher Antoine,

>Allegra is better than Twisted ? Fine. Prove it. Write a bunch of examples

Have a look at what is allready documented:

http://laurentszyster.be/blog/allegra/doc/

or read the sources if you need more:

http://svn.berlios.de/svnroot/repos/allegra/lib/

Of course, these are full of “implementation details” …

I only metered finalizations performances and I’m waiting for somebody else to do the same for Twisted defered. With such a difference between the respective number and weight of the instances required by each solution, I will probably have to do it myself.

By the way, you don’t find bizarre that with such proclaimed emphasis on unit testing there has been no benchmarks of Twisted core?

> and let the truth speak for itself.

Truth never speaks for itself.

People speak the truth. Or they don’t. We all are much less than perfect human beeing. That’s why I trust only peer review to find out what is certainly not true and what may probably be.

Regards,
Elmo Says:
May 31st, 2006 at 6:19 pm
Hey I love this thread … and the others on this subject.

Most pythonistas simply don’t understand what Twisted is about and that makes them a uncomfortable, afraid that maybe the problem is with them … therefore refrain from any criticism…

But you know … it might just be that the emperor is actually naked.
Chris Siebenmann Says:
May 31st, 2006 at 8:17 pm
Actually, in many real world Internet scenarios most of the connections are idle; only a few are live for either read or write IO. One good website that talks about the whole problem is Dan Kegel’s ‘The C10K problem’, http://www.kegel.com/c10k.html.

This matters even when the CPU isn’t pegged, because doing more work in the main event loop increases processing latency (and ultimately impacts how many transactions a second you can process). In this scenario, systems that do a fixed amount of work per connection scale much worse than systems that only do work per active connection; the closer you can move the system to only doing work on active connections, the better. (Hence the popularity of kernel level schemes that pay as little attention to inactive connections as possible.)

(I don’t know if this scenario is of interest to Allegra; it may not be. Really heavily used services are still reasonably uncommon on the Internet.)
Steve Says:
June 1st, 2006 at 12:23 pm
Tiresome twaddle.
Laurent Szyster Says:
June 2nd, 2006 at 4:45 am
To Elmo:

Thanks for that comment.

To Chris:

>Actually, in many real world Internet scenarios most of the connections are idle;

Yes. The case libevent was designed for is the one of a server with 10.000 concurrent connections of which only 90% are stalled. That’s a rare case for an Internet peer applications with concurrency in the hundreds.

Twisted “fab four” is a libevent-like interface. Actually, if you were to prototype a very-high-performance application on that design in Python, directly applying Python’s libevent bindings makes a lot more sense for network developers.

See Apricot:

http://cheeseshop.python.org/pypi/apricot/0.2

a web server built directly on libevent. Also have a look at its benchmarks

http://william-os4y.livejournal.com/

>I don’t know if this scenario is of interest to Allegra; it may not be. Really heavily used services are still reasonably uncommon on the Internet

Indeed, Allegra was not designed to be applied in such case. The purpose of the library is to provide a a framework to develop asynchronous network peers, not very-high-performance server. Allegra is meant to scale well wide enough for such application, up and down.

There are however several optimization strategy for the core and its loop. One is to integrate everything async_ into a single C module.

Another is to prioritize dispatchers by writability and readability state in the loop, trunking at a fixed level of concurrent active connections (you can see that the libevent strategy is not very different from its performance graphs: it quickly trunks to fit the test).

But that’s the future and such optimization will be needed only if the rest of the library is successfully applied. And that some day sombedoy requires to run an application of Allegra on a very-high-performance server handling tens of thousand concurrent connections for Internet peers.

Regards,

About Critiques

About Malice

About Simplicity

About Proof

About Arguing

15 Responses to “To Glyph”

microsoft office 2003 oem

how to activate reget deluxe

Oem software bundle oem software bundles 814.

microsoft oem software cd

microsoft office 2003 oem

software oem cd line business

The Logo Creator MEGA pak 3.6

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem cd line business

Mcafee all in 1 2006

inkjet definition oem software

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

microsoft window 2000 professional oem software

The Logo Creator MEGA pak 3.6

microsoft office software oem;

Oem software bundle oem software bundles 814.

The Logo Creator MEGA pak 3.6

The Logo Creator MEGA pak 3.6

oem software cds;

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

inkjet definition oem software

Buy cheap PhotoShop CS 2

microsoft office 2003 oem

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

software oem cd line business

Buy cheap After Effects 5.5 Production Bundle

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

inkjet definition oem software

iCorrect EditLab Pro 4.52

iCorrect EditLab Pro 4.52

locate software oem files

inkjet definition oem software

Oem software bundle oem software bundles 814.

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

locate software oem files