Bringing Together Network Connections and Run Loops

Recommended reading: A Primer on TCP and The Run Loop.

We’ve learned a bit more on how TCP works, by converting incoming segments of data into a stream of todata that can be read linearly, as well as how run loops or event loops help to best utilize a thread of execution. Let’s dive a little deeper on how these can be combined.

Fun fact: everything is a file (descriptor). In Unix, anyway. This is important because, under the hood, most Unix-based run loops (I’ll be referring mostly to my research about CFRunLoop/NSRunLoop, but it also applies to manually-made run loops) act on file descriptors. Opening a file for reading or writing gives you a file descriptor. Opening a socket connection via UDP or TCP gives you a file descriptor. An in the Mach kernel used by macOS systems, you can get mach port file descriptors as well. Want to sent a message from one thread to another, which will wake the thread and execute a code block. And possibly a few more examples I’m forgetting.

What is a file descriptor? A file descriptor is managed by the kernel. When opening a file, as an example, we ask the kernel for it. The kernel associates the file in storage with our process and gives a new file descriptor that is only usable by our process (from any thread within the process). And the process is essentially given back a number: the descriptor.

The beauty of the file descriptor is that it enables us to suspend the thread until there is activity on those descriptors. See select(2), poll(2), and read(2). This is what makes run loops even possible in the first place. Well, efficient ones anyway.

So, a process opens some descriptors at some point. Later, it calls select/poll (hereafter select, though in Mach it might be mach_msg) and passes in a set of the file descriptors. The select function performs a system call. We all remember from CompSci that this triggers a context switch in the CPU, replacing stack registers with that of the kernel to execute a kernel thread. I won’t get into that magic right now, though. The process' thread is now suspended until the kernel executes it again (with that pesky context switch). Usually this would happen quite soon: after whatever normal system call is done and returns the data, the thread will be queued on the kernel’s scheduler to continue. However, with select, the kernel purposefully does not schedule the thread to continue execution. Rather, it will do its own magic (which I don’t care enough to research) and only re-schedule the thread once the read data buffer for the descriptor has some new data, the write data buffer has some new space to write to, or the descriptor was closed (or something similar).

I think that’s the general gist of how select and its variants have.

Blocking network server

Here’s an example of a server that blocks to read its network connection:

int socket_fd = socket(...);
// error handling

bind(socket_fd, ...);
// error handling

listen(socket_fd, ...);
// error handling

int client_fd = accept(socket_fd, ...);
// error handling

while (true) {
	// blocks
	int result = read(client_fd, &buffer, MTU);

	if (result == 0) {
		// handle the closed fd
		break;
	} else (result < 0) {
		// some kind of error happened
		break;
	} else {
		// do something with that buffer's worth of data, which may be 1 byte or the full buffer.
	}
}

The beginning is just setting up a server with a listening and bound socket, getting the first connection (which I believe blocks as well), and then reading all data on the connection until it is disconnected by the client. Obviously don’t ever do the above.

The read call here is blocking. Specifically, we opened the socket and/or the client_fd connection without a non-blocking flag. The read is a system call, which suspends the thread until there is something in the buffer that the kernel manages.

Wow! Wait isn’t that what select does?

Well, yes. But it locked up the entire thread, and we only connected to a single client! What if another client is attempted to connect? What if we have multiple connections already, and want to handle both at the same time. Without a run loop, the solution to this would be: one thread per connection. In Go, this is fine, because goroutines are green threads. Light weight and managed by the Go runtime. But in most other languages, a thread is a kernel thread, which allocates considerably more resources in order to exist, and takes time away from the kernel to manage while scheduling. We have N many threads that are just waiting around and not being utilized effectively. And if read is implemented by some kind of blocking loop, then it’s even worse because each thread is being executed only to be doing nothing (looping until data is available).

Non-blocking network server

A good networking server would be able to do so much more, and do it more effectively. Let’s take a look at another example pseudocode:

let socket_fd = socket(...);
// error handling

bind(socket_fd, ...);
// error handling

listen(socket_fd, ...);
// error handling

var fds = [socket_fd];

while true {

	let affected_fds = select(fds);

	for fd in affected_fds {
		if fd == socket_fd {
			fds.append(accept(socket_fd))
		} else {
			let result = read(fd, &buffer, MTU);

			if result == 0 {
				// close connection
			} else if result == EAGAIN {
				// shouldn't happen: nothing to read on the fd
			} else {
				// send data down the stack to its own handler
			}
		}
	}
}

This is so much better! Why?

and even more importantly: this server can now handle several connections all at once. What we’ve written is a rudimentary run loop using select. Obviously this is all pseudocode and would not hold up for production software, but it gets the point across.

A framework

Something interesting is that the run loop is handling the read for us. Well, we could instead call some kind of handler object that is already associated with the file descriptor and it could perform the read operation, but that is essentially the same.

This is where the framework of a run loop begins to be built.

Hold up a minute. ASP.NET can get the entire HTTP request from the read?

Not quite! It will require several reads, possibly. We’re now getting to the connection between A Primer on TCP and run loops.

A network connection can be (and should be assumed to be) very slow. In an HTTP request example:

HTTP/1.1 PUT /objects/1/documentData
Content-Type: application/pdf
Cookie: (can be very large)
... (many more headers)

Content-Length: (100 MB)
...

A typical TCP maximum segment size (MSS) is up to about 536 bytes. There is no way that every HTTP request can fit in that entirely. And in a lot of cases, not even the entire header can, either. This is why my previous article emphasized buffers so much. Each segment only holds so much data, and fills up buffers as the data comes in.

The client uploading this PDF file in the example above might send the first 5 segments, but then wait for a while. Perhaps the server is full, perhaps the packets are dropping, perhaps the client can’t send the segments out fast enough. There’s a multitude of reasons.

But the effect is that the server may be notified multiple times via the select call that more data is available for reading off of the client connection file descriptor.

This of course imposes some added complexity for the application code and logic. It needs to have its own memory/file buffer for the incoming data, parsing it as it comes until it knows the request has concluded. In HTTP’s case, it parse the entire header (up until the "\n\nContent-Length:\d{1,}"), and then the entire body based off the Content-Length. In SFTP, it’s a binary protocol where the first four bytes define the length of the request, which is admittedly a bit easier.

Combined

Combined with the rest of the duties of a run loop, particularly in GUI/non-server scenarios, it all begins to add up.

In a GUI example, we make a run loop and associate with it:

The select or equivalent method is the core of this. An application not doing anything will not be consuming precious CPU. Whenever user input begins, or data is available on a socket, or a write buffer can be written to on a socket, or an incoming task comes in from another thread (using the message port), or anything else is triggered, the main thread that is listening on those file descriptors can then, and only then, execute them.

Messages are interesting. I wouldn’t be surprised if they’re implemented as just a data buffer that has an established serialization/deserialization method. In my run loop post, I had examples where we scheduled a task on a background thread. Once it ended, it executed some code back on the main thread (you should only affect the UI on the main thread). I think messages are used for this: create a message pointing to a function/selector/closure/block of code, and write it to the message port on the main thread. The kernel alerts the main thread with the run loop of the new data in its message port, it gets deserialized, the closure is found, and then executed now that it is on the main thread.

This of course means that background thread probably has its own run loop too, in order for it to receive that task to execute. This opens the door: an HTTP server could be designed in several ways. Perhaps it has one thread to accept connections, and then a pool of threads to handle each connection’s file descriptor and thus request. Or once new data is available on the main thread’s run loop, it schedules a task to read and handle the data using a random thread of a pool of threads.

Closing thoughts

I think I finally have grasped how event loop models work, now. I’m comfortable enough to continue with my research into SwiftNIO for my SFTP side project. I’ve learned a lot, probably lower-level than I really need to. But it’s fun and writing out my thoughts helps organize everything nicely.

I assume that C# works similarly, but I’m still mystified by it somewhat.