Allegra » Blog Archive » TCP Stack Flaking Out

TCP Stack Flaking Out

That’s how BitTorrent developers described a serial TCP bug in their network peer application. I stumbled on that problem too with Allegra. If you care, here is why it happens, how to fix and reproduce it. If you don’t care, it’s still a fun debugging thriller.

About an old case from Redmond USA. With an elaborated plot, sweet sixteen, real code and a twisted end.

Yet nobody dies …

The Usual Suspect

Like many bugs, this one is a buffer overflow.

To be precise, it’s a TCP stack overflow. On Windows, Linux and probably most Operating System, TCP stacks have a maximum size set to their receive and send window (or buffer).

When that buffer is full, writing more to the socket will block … if you are using a blocking socket. But when using a non-blocking socket, overflowing the TCP stack will raise the following exception on Windows:

WSAECONNRESET - Connection reset by peer

instead of the obviously expected

WSAEWOULDBLOCK - Resource temporarily unavailable.

because Win32 decided to close the abused socket.

Why?

A Windows Feature

Apparently, the Microsoft folks expected non-blocking I/O to be programmed without application buffers but with asynchronous continuations (Windows NT is asynchronous OS!). Rightfully, they made output overflow a showstopper.

Because any other OS behaviour would have either: a) forced the system to grow boundless buffers for every socket and eventually run out of memory when network output stalls; b) prevented asynchronous continuation.

The application can’t callback to its continuation once the data has been buffered, then it will hang forever. Boundless OS buffers on one side, application hanging on the other: it’s a fatal error for the socket at least.

Hence the "Connection reset by peer" error.

The Plot Thinkens

What probably tipped the decision at Redmond was that TCP stack overflow was practicaly a rather rare error condition in its multiplexing and pipelining applications of … the select POSIX interface. There was no need to think further: Win32 came first and it’s a WSAECONNRESET error for all non-blocking socket.

As the size of data exchanged grew, as CPU got faster and network protocol supported pipelining, that possibility soon became a certainty. Because the default system buffers size did not increase. And in ten years, a rare critical error became an common bug.

But only for pipelined protocol applications of Win32 using an asynchronous POSIX interface to socket I/O.

Two Solutions

There are different ways to solve that problem.

The simple but not obvious solution is to buffer output and invert the I/O control in the application so that writing to a socket is allways driven by events, never by the application.

The simplistic solution is to slow down the application to write less at once, at least enough to lower the probability of an error below a reasonable level.

The implementation of that simplism is however less obvious. The simplest way to do that would be to allocate CPU quota to the application at the system level, outside of the application. Not inside.

Out of Twistedland (again)

Twisted apparently fixed BitTorrent 4.1.5 "tcp stack flaking out" problem present since 4.0.1, but I suspect that it did so in its own way: without knowing.

Without much support for pipelining, simplistic delimited text protocol implementations and thick layers of useless indirection everywhere, Twisted is slow enough to solve the problem and never let its application produce more output than 16 kilobytes at once for a socket before it is polled again.

That’s a huge demonstration of how complicated simplistic can be.

The Simple Solution

To avoid writing too much at once to a non-blocking TCP socket, your application must a) buffer output and write it by chunks to the socket b) only when the OS signals it is ready to buffer more output.

That’s why Allegra’s async_chat dispatchers have an ac_out_buffer that is allways refilled by handle_write and don’t have an initiate_send interface anymore.

In an amazingly simple way, the inversion of I/O control in asyncore and the buffering interfaces of asynchat had only to be improved marginaly in order to prevent what was a higly improbable error case in 1996. But which defines high performances on peers now.

Without A Clue

I found out about this problem when the dispatchers I developped on Windows raised the socket "Connection reset by peer" exception for large requests, large responses or under high load when pipelined.

Because ten years later the original design of Medusa is still one that suites high-performance Internet peers. Its simplicity and support for pipelined protocols makes its applications fast enough to quickly overflow a TCP stack.

Today’s asynchronous network peer can have gigabytes files and RAM, gigahertz CPU and gigabit pipes but they still only have kilobytes stacks. On the stable GNU/Linux kernel distribution, the minimal, default and maximum buffer sizes are respectively set to 4KB, 16KB and 128KB. You can easely check your own:

cat /proc/sys/net/ipv4/tcp_wmem

In windoze registery you can set it as

GlobalMaxTcpWindowSize

since 2002 at least and XP’s default is probably set to 16KB today.

Sweet Sixteen Kilobytes

Once I resized async_chat output buffers to that limit and gave exclusive control over socket output to the event loop, that error condition never occured in my applications (although the same socket exception may still be raised, but for very different and better documented reasons).

If you are like I was before, a programmer without a clue about socket programming, I hope you enjoyed the article and can put it to good use. Because if debugging that was a lot of fun, it took my belgian brain an awfull lot of time to put it all together.

If you did not care about TCP stack overflow, sweet sixteen kilobytes is all the teenage romance you will find here.

What else did you expect?

Code at Last!

Checkout Allegra from its SVN repository to a Win32 system in the range of my laptop (1.7 Ghz but RAM is irrelevant) with Python 2.4 installed or better.

Change directory to the project’s root

cd allegra

Install the latest revision

\pytho24\python setup.py install

Then launch a flaking server sending 2048 bytes (1<<11) bytes of data per write event and receiving only 512 bytes (1<<9) per read event:

\python24\python test/flaking_server.py 10
    2> log/flaking.err

and connect 500 flaking clients sending 128KB (1<<17) and receiving 32KB (1<<15) at once per loop run

\python24\python -OO test/flaking_client.py 500 16

When you stop both after a dozen seconds, the flaking.err log should be filled by sequences like:

traceback
  Flaking id="a95610"
    socket.error
    (10054, ‘Connection reset by peer’)
      C:\python24\Lib\site-packages\allegra\async_loop.py | _io_select | 100
      C:\python24\Lib\site-packages\allegra\async_core.py | handle_write_event | 255
      test\flaking_server.py | handle_write | 14
      C:\python24\Lib\site-packages\allegra\async_limits.py | send | 46
      C:\python24\Lib\site-packages\allegra\async_core.py | send | 145
	
debug
  Flaking id=”a95610″
  close
	
debug
  Flaking id="a95610"
  in="18432" out="104448"

The TCP stack is flaking out fast, apparently for no reason.

All TCP traffic happen through a virtual loopback device (127.0.0.1), not actually via the network. But to systematicaly simulate stalling network output for the client, the server I/O are chocked on purpose.

To demonstrate that a pool of async_core clients can move output far too fast for the logging server small input application buffers. That Allegra’s application can get as fast as too fast for the OS.

That’s quite fast enough.

Last Notes from Twisted Users

I’m still out of Twistedland and won’t debate facts with egg-heads.

If you are an offended twisted developer, get real and listen to what users have to say about BitTorrent since it was fu… - hmm - fixed by Twisted: "Sadly, I’ve also had to stop using BitTorrent. Something went wrong about two versions back and it’s never recovered"; "this applications seems CPU hungry more than Alias Maya and Photoshop combined"; "The program constantly throws up a Critical Error: Traceback (most recent call last) which results in stalling any active torrent" etc …

[in http://www.macupdate.com/reviews.php?id=11869].

Who needs benchmark when a Mac user can tell?

The End

Twisted died from design flaws, overweight and permanent instability.

And yet, "That is not dead, which can eternal lie" …

Indeed.

This entry was posted on Friday, August 18th, 2006 at 11:59 am and is filed under Uncategorized, Python, Socket Programming. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

15 Responses to “TCP Stack Flaking Out”

mark Says:
August 18th, 2006 at 1:17 pm
Hey,

while you are cherry picking reviews from that message board (and assuming the problem is twisted) - what about this one:

” If the client sucks I expect it is due to it being written in python and python sucks. ”

You seem to have written Allegra in the wrong language according to your own logic!
Laurent Szyster Says:
August 18th, 2006 at 4:34 pm
>You seem to have written Allegra in the wrong language according to your own logic!

That’s not my logic, it’s the logic of a Mac user who can compare against a Java peer available, and goes out of its way to find out why the 4.0 twisted release could be so slow. I can tell it’s not Python.

According the my result, and you have everything at hand to test that, raw async_core dispatchers can saturate the OS on a loopback device. That’s around what? With buffers between 16KB and 4KB, I reached between 1,2MBps and 0,8MBps on the loopback device under XP.

That’s well above the 512Kbps upload bandwith of a cable-modem peer. With the appropriate buffers, even the added process of async_net or async_chat buffering I/O may not prevent distributed Allegra application peers to saturate their 100Mbps network by all chating together.

I found out about “TCP stack flaking out” while stressing the PNS metabase and the HTTP/1.1, server and clients, as they were simple and “allmost finished but not quite yet”, just working enough. It happened as soon as the MIME body producer filled a buffer much larger than the TCP send window to many clients concurrently. Or with pipelined protocols that fill up the buffer too fast. As soon as you cache content in memory this “bug” must be fixed, because moving bytes in the application address space is much faster than the network.

The trouble with Python is that there *must* be less code in order to run fast enough. Allegra passed that test, moving bytes in and out of memory faster than the OS can handle it over to a starved network.

Python is still as slow. But its C parts are damn fast.

Here less is *practicaly* more.
Stephen Thorne Says:
August 21st, 2006 at 6:16 am
I find it astounding that you’ve not presented the basic solution for this problem. Using select to find out when a socket is writable. You seem to have alluded to this in the text, but haven’t bothered to present it to the reader:

# where readables and writeables are a dict of
# {fd:ConnectionAbstraction()}
rlist, wlist, ignored = select(readables.keys(), writables.keys(), [], timeout)
for r in rlist: readables[r].readData()
for w in wlist: writeables[w].writeData()

where writeData does buffering correctly, limits the .send to 128*1024 (to stop windows from flaking out on a big .send()), and stops when it runs out of data to send or the socket buffer is full. There’s a bunch of error conditions you have to be mindful of, EINTR for signals, EWOULDBLOCK and ENOBUFS for the send, EBADF for the select.

I see allegrya doesn’t handle all the error conditions optimally.

How does allegrya handle EBADF and ENOBUFS? EBADF could be raised from _io_select() if the file descriptor is closed on you, ENOBUFS could be raised from Dispatcher.handle_write(), and if that happens I’m pretty certain from reading the code that you’ll close the socket….
Greg Hazel Says:
August 22nd, 2006 at 12:15 am
So, I’m the developer that switched BitTorrent over to using Twisted.

The problem BitTorrent had is that sometimes select() throws an error if you pass it too many file descriptors. It results in ENOBUFS. This happens particularly with Win9x, and even more so with applications like NOD32 installed. Because python’s select wrapper sucks, the lists you passed it will not be modified, so initially the BitTorrent client would eat 100% cpu (repeating the same call with the same list). They discovered this problem and simply threw the ‘tcp stack flaking out’ message, which is incorrect.

I switched to Twisted so I could get IOCP, which works better and faster than select(), and does not run in to this problem. Twisted is “fast” enough to hit the exact same error with select, which I reported here:

http://twistedmatrix.com/trac/ticket/1228
Laurent Szyster Says:
August 22nd, 2006 at 1:37 am
Hi Stephen,

> I find it astounding that you’ve not presented the basic solution for this problem.

I do. Polling for writability before calling handle_write_event is not enough, your application must also never call the dispatcher’s socket.send outside of that write event handler.

> I see allegrya doesn’t handle all the error conditions optimally.

Thanks for the pointers to various possible TCP errors. Regarding error handling, I’ll point you back to async_loop._io_poll methods and async_core.Dispatcher’s send and recv.

Here is how socket errors are handled.

a) EINTR is tolerated for select or poll call, anything else will be raised this is possibly a good place to exit the loop.

b) EWOULDBLOCK is handled by recv and ECONNRESET, ENOTCONN, ESHUTDOWN in send are translated into handle_close event. Any other exception throwed by the socket in a dependency of handle_read_event and handle_write_event will be catched by the try: except: clause in the io_poll loop.

c) Finally, note that EINPROGRESS, EALREADY and EWOULDBLOCK error condition on a socket.connect are also handled.

Up until now that’s been safe enough for asyncore and asynchat too. I don’t know about ENOBUFS on Win32 and running out of OS buffer is precisely what Allegra is expected *not* to do.

As for the EBADF it’s a fatal error that you can’t handle! Here is en enlightening thread about that exception and select:

http://www.developerweb.net/forum/archive/index.php/t-3247.html

I cite:

“But, really, this shouldn’t be a situation that’s regularly occuring,
either… It indicates some sort of bug in your code, not some
normal, expected condition, which you should try to work around (…)”

and

“Well, it most likely would suggest a coding (or logic) flaw, wherein a failed read() or write() was not detected or handled, but there is a window between the last successful I/O operation and the call to select() during which a socket may go bad due to peer activity.”

and

“Yeah, as Anders says, the situation you describe just shouldn’t ever happen, at all… select() failing with EBADF doesn’t mean there’s some socket error on the FD, or something; it means that the FD is literally bogus, and not usable, at all… Usually, because you’ve probably already closed it, and failed to remove it from your fd_sets, or something… But, no, it doesn’t suggest a simple error with the connection, at all…”

Can I stop now and get out of Twistedland?

Please …
frontera000 Says:
August 24th, 2006 at 8:07 am
Windows TCP/IP nteworking stack code and socket layer code is loosely based on BSD UNIX code. BSD TCP/IP code is one of the original and most widely deployed implementation. Windows networking code is not that different in handling socket level buffering.

The term “stack” is as in protocol stack in previous sentence. Like the OSI 7 layer stack model. There is no “TCP stack” as you use the term, as in function call stack or stack as opposed to heap. That is not a commonly understood terminology.

I guess what you mean is the socket level buffering (as in mbufs). For every socket the kernel socket level code will allocate (or not actually allocate but reserve) a high-water mark which indicates how much data is allowed to be buffered for that socket. There are two socket level buffers for each socket. One is for read side. Another for write side.

A socket programmer who is using the socket API can reserve different amount of these buffers per socket. Using setsockopt(). Not only that, it is possible to get the current size and find out how many data is pending in the socket level buffers. This is done via ioctl() or fcntl() or other means. It is easy to find out how many one can write before overflowing the buffers. Application programmers should write programs so that the code that writes checks the buffering beforehand, especially when the application is I/O intensive.

In case of blocking I/O, the call to write() to socket will block until more buffer space (socket level buffer for write side) is available. This has nothing to do with OS kernel being slower in sending data than what your spiffy application code can do so fast sending out. It has to do with data transfer over network. OS is plenty fast enough, usually faster and more efficiently coded than any application written in python. It has to do with flow control and injecting data at the right rate into the network. TCP depends on many different heuristics to accomplish its throughput and latency issues. Read up on Van Jacobsen slow-start and congestion control algorithms. Sending as fast as you can is not going to give you the best throughput over a network that has dynamic behaviors.

Getting back to the Non-blocking I/O case, as you mentioned “asynchronous” I/O, I suppose you mean that you can write without blocking. In those cases, a proper kernel will notify you with error. Say there is space for 100 bytes in your socket level buffer, and then you call write() repeatedly to that socket via non-blocking I/O (I assume you set a flag or something on that socket like FIONBIO or FIOASYNC). What that does is to attempt to write data into space that is not sufficient to support the request. The request will be rejected and error code will be returned. That is what happens in most kernels, including most Unix style kernels and Windows.

If you are seeing Reset on the connection, it is likely that you are seeing the result of the network event. A TCP packet with RST flag set is probably being received, which resets the connection. In order to see why this happens, you will need to trace the network and see the packets on the wire.

This is not something like what you described in this blog entry. Your understanding is completely wrong.
Laurent Szyster Says:
August 24th, 2006 at 6:46 pm
Hi frontera000,

> This is not something like what you described in this blog entry. Your understanding is completely wrong.

Before publishing this article I took great care to do some research and check my facts. You should do the same before coming to such a definitive conclusion. At least you should have read the sources and run the tests linked in the article:

http://svn.berlios.de/svnroot/repos/allegra/test/flaking_server.py

to

http://svn.berlios.de/svnroot/repos/allegra/test/flaking_client.py

Then you would have noticed that I’m conducting those tests on the loopback device address (127.0.0.1) and that their results are independant from the network state, bandwith, etc.

More puzzling, it looks as if you have not read the article itself, which makes very clear that I’m not talking about a frame stack, POSIX systems or a BSD implementation of the TCP stack.

This article is about a Win32 *API* problem, a reminder about the fact that the implementation of select () must somehow be compatible with the asynchronous nature of Windows NT and its applications.

Read the sources and run the tests on Windows first.

Once you do, I’m sure that you’ll have more relevant comment to make. Because obviously you know a lot more than me about Lisp and TCP socket programming on POSIX systems than I do:

http://sparebandwidth.blogspot.com/2006/08/trivial-p2p-in-newlisp.html

Regards,
frontera000 Says:
August 25th, 2006 at 7:06 pm
Hi, I am aware that you are talking mostly about Windows specific TCP issues. I think that this article is unclear about exactly why the problem is due to Windows TCP. (It’s funny but I guess I am defending Windows TCP socket interface API, shudders.) Because of the lack of clarity in this article, it is hard to see what problem is being described.

I have read your flacking server and client code. They are written to use your library called allegra. I looked into allegra. Allegra does not seem to handle errors in a robust way. For send, it seems to only check for EWOULDBLOCK. Winsock API is sufficiently more complicated than equivalent POSIX socket API. It is also far more capable (I can’t believe myself saying so, but it is true) and versatile. (probably more buggy too.. haha!)

For nonblocking send() in Winsock, the following errors can be returned. ENOBUFS, ENOTCONN, ENOTSOCK, ESHUTDOWN, EINTR, EINPROGRESS, ENETDOWN, EMSGSIZE, EHOSTUNREACH, EINVAL, ECONNABORTED, ECONNRESET, ETIMEDOUT, and of course EWOULDBLOCK.

Similarly there are many different error codes that can result from recv() and select() in Winsock.

Are you sure your code (Allegra) is not at fault? I am not saying so myself, but perhaps it is best to write a simpler test case — not using Allegra but just using socket API to prove your point.

Your test programs use Allegra which makes it look like it could be Allegra problem, not Winsock API implemention on Windoze boxes.

Why not just write a simple test case, in C or python.

I guess I could do that just as easily, but I am not sure exactly what you are trying to do in your tests (due to use of underlying Allegra library). If you are simply trying to use non blocking socket to do lots of writes from the client, while server is reading very slowly, I can write that and test it out. But it would be better if you could write a simple socket test program in C or python, instead of Allegra to prove your point first. Since I cannot understand precisely what your thoughts on Win select/socket API is.
Laurent Szyster Says:
August 25th, 2006 at 7:39 pm
Hi again,

Thanks for that gentler response (although I still can’t see where you found an ENOBUFS constant in Python’s socket bindings on Win32:

c:\python25\python
>>> import socket
>>> [s for s in dir (socket) if s.find (’ENOBUFS’) > -1]
[]

Don’t take my word for granted, try it ;-)

>Your test programs use Allegra which makes it look like it could be Allegra problem, not Winsock API implemention on Windoze boxes.

Indeed it could.

But if you take the patience to follow up what a flaking_client.py imports, namely: read async_loop and async_core, then you will quickly notice that there is very little between this simplistic client and the select () call. Not even application input buffers or output queues (that’s in async_chat and async_net).

>I cannot understand precisely what your thoughts on Win select/socket API is.

To make it brief here is what I think: the implementation of Win32 the POSIX select call must be compatible with the rest of the API … which is tailored for both asynchronous applications and operating system (Windows 95 ran on top of a single process DOS and Microsoft nicely followed the path of least resistance to an asynchronous OS).

Win32 actually “fakes” blocking sockets like it “fakes” threads and processes (hence the blue screen of death). So, when Windows stops faking to block on a socket operation, your application will fall-back on the native asynchronous system call … which do raise a fatal error when the buffer allocated to TCP is overflowed.

So that the application fails instead of the system.

From what I read here and there, Windows can allocate a pretty large elastic buffer shared by all sockets (that’s easy … in an asynchronous OS). That’s why TCP sockets will start to “flake out” when they overflow that buffer. Because as long as its shared output buffer is not full, Windows apparently signal write events on sockets *independantly* from the network or the actual state of the TCP protocol.

If the connected peer is too slow and the volume of data to big, that buffer will overflow. And sockets at fault will be closed. That case is not uncommon for a BitTorrent peer used to distribute very large files to a slow network from peers with limited upload bandwith.

Regards,
frontera000 Says:
August 26th, 2006 at 5:41 am
Hello again. Regarding ENOBUFS not being in Python API, I believe you. That is not something I was suggesting otherwise. We are not talking about Python features. We are talking about whether Windows TCP flakes out.

Since I cannot be certain Allegra sample program is not flaking out due to Allegra itself, I wanted a simpler program. So I wrote one in C. Why? This topic interests me. I wrote a TCP/IP implementation long time ago and it is a pet peeve of mine.

Anyway you can see another posting I made which lists out the C program I used to test. Here is the link: http://sparebandwidth.blogspot.com/2006/08/more-on-tcp-flaking-out.html

In any case, the conclusion, at least on my end, is that Windows socket implementaion and TCP/IP are OK. I think you have some bugs in your code. I think the test I wrote is pretty good. It creates 1000 threads sending to one server that is artificially slowed by a Sleep(). Things work as expected.

If you disagree, I’d like to hear about it.
Laurent Szyster Says:
August 26th, 2006 at 11:58 am
Well, of course I disagree: this is about *opiniated* software ;-)

First, you are using threads to multiplex your clients instead of an asynchronous loop around calls to the select () interface. Second, you are not using two different program instances for the server and the clients, but different threads in the same process. Third you don’t slow down your server enough: reading 34000 bytes from 1000 socket output every 100 milliseconds is still reading more than 300MB per seconds.

So there is no chance that you’ll hit the OS limit on the total size of the buffer space allocated for every sockets. Because everything sent by the client is read at once by the server!

Using threads, you will instead glob all CPU time and bring the OS to a crawling state before you can overflow it’s shared socket buffer.

To reproduce the same error condition, to make client program fast enough to overflow the OS buffer allocated to TCP output, you should remove any contention between client threads *and* between clients and the server thread (that’s why it can “hang” your PC if you run the clients forever, because all CPU is allocated to contention between server and client threads ;-). Then, setting smaller buffers for the server and the client, on input and output, will get your two programs to a state where the client is outputing data so fast that the OS runs out of global buffer.

If you do such experiment in C, be sure that I’m interested in the result!

One thing I agree with you is that ENOBUFS is obviously not available in CPython. So I had a look at it’s socket.c binding to look for a handler of that error condition … and found none:

http://svn.python.org/view/python/trunk/Modules/socketmodule.c

If you look into its sock_send, there is no other error handling but a call to set_error. So, CPython bindings don’t handle the WSAENOBUFS error condition on send, they simply report the OS error condition. Yet what Allegra’s async_core.Dispatcher’s socket.send calls *do* raise is a WSACONNRESET which puzzled me before I could reproduce it systematically and make sense of it.

Anyway, even if we still disagree, this turns out to be a nice way to learn about non-blocking socket programming on Windows for both of us (and our readers).

Regards,
frontera000 Says:
August 26th, 2006 at 11:42 pm
Actually I have done the experiments you proposed already. I have a version of the code that has client threads and server in different processes. It makes no difference. I also have experimented with ridiculously small send and receive buffer sizes at socket level on the server side. No difference. This is just how things should be. There are no problems other than the applications getting ENOBUF errors as they should.

Hope that clarifies your concerns.

I think one of the errors you may not be catching (other than the send not checking ENOBUFS error condition) is that you may be connecting clients fast to the server, while I/O is going on at the same time. Doing that can cause problems in Windows as well as BSD, due to the fact that the backlog on listen is set at 5 maximum.

If you notice, I wait until all clients have connected, before doing the I/O intensive testing. If you don’t do that it can cause RESET conditions as in your case.
frontera000 Says:
August 26th, 2006 at 11:46 pm
More clarifications.

By the way your observation “Using threads, you will instead glob all CPU time and bring the OS to a crawling state before you can overflow it’s shared socket buffer.” is incorrect.

Using threads it is very easy to glob all buffer spaces. Besides the socket buffers are *not* shared. Socket buffers are per socket.

And everything sent by client is not read “at once” by the server. That is incorrect.
Laurent Szyster Says:
August 27th, 2006 at 5:03 am
> If you notice, I wait until all clients have connected, before doing the I/O intensive testing. If you don’t do that it can cause RESET conditions as in your case.

That’s not the case: as you can see from the log entries I get from the server and which clearly shows that:

debug
Flaking id=”a95610″
in=”18432″ out=”104448″

the client managed to send 18432 bytes and then broke the connection.

Also, if you care to read it once more, flaking_client.py do connect all its dispatcher’s socket before polling for I/O.

Finally I could not establish a relation with the number of clients: 500 clients with buffers twice as small than the server would do fine, but when the same 500 clients sent chunks of 128KB at once for a server with a 512 bytes application buffer … many closed after having actually sent only 18432 bytes (in that lapse of time the server looped 51 times around select and managed to send 102KBps in chunks of 2048 bytes). So, presumably the error happened as there was 42MB of data buffered *by the OS* without raising an error, never actually sent.

> And everything sent by client is not read “at once” by the server. That is incorrect.

hmm.

Indeed the server sends and receives twice as less than its clients at once, but it is also presumably a lot faster (there is only one server thread, not a thousand). And even if it sleeps 0.1 seconds between lapses, that’s barely the order of magnitude of the time it takes to loop through a thousand file descriptor in C, then call send and recv once.

FYI a select based TCP server can handle around one connection per millisecond implementing a non-trivial protocol like HTTP … in Python. So, a test-purpose C server doing a lot less *must* be at least ten time faster, maybe a lot more.

Probably well below 0.1 second for 1000 concurrent connections. And maybe twice as fast than 1000 clients threads contending in the same process.

I suggest you time each server loop runs when the clients run in a separate process: you may be surprised …

Regards,
frontera000 Says:
August 27th, 2006 at 7:05 pm
Hi,

I think that you should know I have tried varying sleep times in server (upping to 1 or 2 seconds) as well as using very small buffers on the server. These things don’t make any difference in terms of connections dropping. They don’t. As long as the clients and server do check the error conditions properly (NOBUFS, WOULDBLOCK, etc.) the OS will not send RST to terminate TCP connection. If it did, it would be indeed quite broken. It does not do that. If you don’t believe me, please run the program I have posted, and tweak all the numbers (SND BUF and RCV BUF as well as various other numbers, including number of threads and how much to send, etc.) Do all the tweaking you can, but you won’t see the connections dropping.

That’s about all I can say here. It is true that many software developers are “opinionated”, but I am not. I could be wrong. It would be nice to be corrected with a clear evidence, rather than conjecture. But if you cannot be convinced by the test program I wrote, I have no more to contribute here.

Regards to you and have a good day!

You must be logged in to post a comment.

15 Responses to “TCP Stack Flaking Out”

Leave a Reply

microsoft office 2003 oem

how to activate reget deluxe

Oem software bundle oem software bundles 814.

microsoft oem software cd

microsoft office 2003 oem

software oem cd line business

The Logo Creator MEGA pak 3.6

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem cd line business

Mcafee all in 1 2006

inkjet definition oem software

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

microsoft window 2000 professional oem software

The Logo Creator MEGA pak 3.6

microsoft office software oem;

Oem software bundle oem software bundles 814.

The Logo Creator MEGA pak 3.6

The Logo Creator MEGA pak 3.6

oem software cds;

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

inkjet definition oem software

Buy cheap PhotoShop CS 2

microsoft office 2003 oem

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

software oem cd line business

Buy cheap After Effects 5.5 Production Bundle

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

software oem italiano

microsoft office 2003 oem

iCorrect EditLab Pro 4.52

inkjet definition oem software

iCorrect EditLab Pro 4.52

iCorrect EditLab Pro 4.52

locate software oem files

inkjet definition oem software

Oem software bundle oem software bundles 814.

iCorrect EditLab Pro 4.52

Buy cheap PhotoShop CS 2

Adobe cs2 photo shop total training adobe cs2 photo shop upgrade 459.

locate software oem files