John Cairns bio photo

John Cairns

John is an engineer, architect and mentor who focuses on extremly performance sensitive and ultra high volume applications.

Email Twitter Github

Simplicity has its place in software design. Yet we seem to be in the age of framework dependent design. For obvious reasons engineers are encouraged to reuse existing code and not reimpliment the wheel. We have all heard this hackneyed expression, but should we take it to heart? Short of being completely cynical there is a trade off where writing your own solution may be simpler, faster and easier to maintain.

I recently read an article called “A Web Server in 30 Lines of C”. The 30 line solution uses 0MQ to implement a “raw” socket that can support a streaming protocol such as HTTP. This article describes three camps, “the good”, those who have never built a messaging system, “the bad,” those who’ve built a messaging system and relied on a framework, and “the ugly,” those who’ve built a messaging system but did it the ugly way, using sockets. In the context of that article I am squarely in the “ugly” camp. I don’t simply trust packages and tools I download from the Internet to make my code perform better or improve stability. In fact, I can cite numerous counter examples.

There are certain uglies among us who prefer a straight C solution rather than implementing whatever the framework du jour happens to be. Yet, these cursed souls are frequently vilified for wastefulness and being behind the times. They refuse to yield to the march of progress. For shame, for shame!

The first question that came to mind in reading this “30 line” blog post is how many lines of C does it take to create such a trivial “web server?” It is not hard to create an empty shell that does nothing outside of reading some bytes off of a socket and serving something rudimentary in response. I know a lot of developers that would assume implementing any TCP server is non-trivial but they do so more often out of laziness than personal experience.

My approach to this problem is the same that I would employ in building a much more sophisticated web server. I’m going to write a single threaded event reactor using epoll. This is similar to the model used in nginx and other very high performance network applications.

Worlds Smallest Web Server

First we must bind the server socket, we are going to be using asynchronous I/O because otherwise I/O operations would block our single thread and cause starvation. Here is the function that does this. It sets the O_NONBLOCK flag on the file descriptor and returns. – 10 lines

static int set_fd_nonblock(int fd) {
  int flags = fcntl (fd, F_GETFL, 0);
  if(flags != -1)  {
    flags |= O_NONBLOCK;
    if(fcntl (fd, F_SETFL, flags) != -1) {
      return 0;
    }
  }
  return -1;
}

Next I bind the server socket for the local interface. – 26 lines

int main (int argc, char *argv[]) {
  int serverfd, efd;
  struct epoll_event event;
  struct epoll_event *events;
  struct addrinfo hints;
  struct addrinfo *result;
  struct addrinfo *ritem;

  memset (&hints, 0, sizeof (struct addrinfo));

  hints.ai_family = AF_UNSPEC;     
  hints.ai_socktype = SOCK_STREAM; 
  hints.ai_flags = AI_PASSIVE;     

  if(getaddrinfo (NULL, PORT, &hints, &result) == 0) {
    for (ritem = result; ritem != NULL; ritem = ritem->ai_next)
      {
        serverfd = socket (ritem->ai_family, ritem->ai_socktype, ritem->ai_protocol);
        if (serverfd == -1)
          continue;
        
        if(bind (serverfd, ritem->ai_addr, ritem->ai_addrlen) == 0) break;
        close (serverfd);
      }

    freeaddrinfo (result);

Then I set the server socket to non blocking mode and tell the operating system that I would like to create a new instance of epoll. The epoll_wait system call will notify us when there is a socket state change event. – 5 lines

...
     if(set_fd_nonblock(serverfd) != -1) {
        if(listen (serverfd, SOMAXCONN) == 0) {
          efd = epoll_create1 (0);
          if (efd != -1) {
          ...

Finally, I configure epoll to be edge triggered and tell it that I am interested in read events. In epoll, edge triggered means that you will only be notified once when there is a change in state on the socket. If you didn’t read all the data out of the buffer you won’t get a callback until the next event. After registering epoll, I am free to call next_event, the reactor event handler, forever. – 20 lines

...
            event.data.fd = serverfd;
            event.events = EPOLLIN | EPOLLET;
              
            if(epoll_ctl (efd, EPOLL_CTL_ADD, serverfd, &event) == 0) {
              events = calloc (NEVENTS, sizeof event);

              while(next_event(efd, serverfd, events) == 0);
              
              free (events);
              close (serverfd);
              return EXIT_SUCCESS;
            }
          }
        }
      }
    }
  }
  perror("Failed: ");
  return EXIT_FAILURE;
 }

The next_event function queries epoll and handles the subsequent accept and read events on the socket. – 57 lines

static int next_event(int efd,  int serverfd, struct epoll_event *events) {
  int i, n, quit=0;
  struct epoll_event event;
  n = epoll_wait (efd, events, NEVENTS, -1);
  for (i = 0; i < n; i++)
    {
      if ((events[i].events & EPOLLERR) ||
          (events[i].events & EPOLLHUP)) {
        fprintf (stderr, "epoll error\n");
        close (events[i].data.fd);
        continue;
      } else if (serverfd == events[i].data.fd) {
        // event on serverfd
        struct sockaddr in_addr;
        socklen_t inaddrsize;
        int fd;
        inaddrsize = sizeof in_addr;
        fd = accept (sfd, &in_addr, &inaddrsize);
        if (fd >= 0 && set_fd_nonblock(fd) != -1) {
          event.data.fd = fd;
          event.events = EPOLLIN | EPOLLET;
          if(epoll_ctl (efd, EPOLL_CTL_ADD, fd, &event) == 0) {
            continue;
          } 
        } else {
          perror ("accept failed");
          abort ();
        }
      } else if(events[i].events & EPOLLIN) {
        // handle read
        int done = 0;
        while(done == 0) {
          ssize_t nRead;
          char buffer[1024];
          
          nRead = read (events[i].data.fd, buffer, sizeof buffer);
          if (nRead == -1) {
            if (errno != EAGAIN) 
              done = 1;
            break;
          } else if (nRead  == 0) {
            done = 1;
            break;
          }
          
          quit = parse_http(events[i].data.fd, buffer, nRead);
                          
          if (done)
            {
              close (events[i].data.fd);
            }
        }
      }
    }
  return quit;
}

This code calls the parse_http function with the socket buffer and file descriptor. In the case of this simple example parse_http is trivial, however in a real web server the details could readily be filled in. – 30 lines

static int parse_http(int fd, char* buffer, int nRead) {
  int pathStart;
  int pathEnd;
  int i;
  for(i=0; i<nRead; i++)
    if(buffer[i] == ' ') { pathStart = i+1; break; }

  if(i==nRead)
    if(write(fd, INVALID, strlen(INVALID)) <= 0) close(fd);

  for(i=pathStart+1; i<nRead; i++) 
    if(isspace(buffer[i])) { pathEnd = i; break; }

  if(i==nRead) pathEnd = nRead;
  for(i=pathStart; i<pathEnd; i++) {
    if(buffer[i] == 'q' &&
       i <= pathEnd - 4 &&
       buffer[i+1] == 'u' &&
       buffer[i+2] == 'i' &&
       buffer[i+3] == 't') {
      if(write(fd, OKAY, strlen(OKAY)) <= 0) close(fd);
    return 1;
    }
  }
  
  if(i==pathEnd) {
    if(write(fd, NOTFOUND, strlen(NOTFOUND)) <= 0) close(fd);
    return 0;
  }
}

124 lines

That is my solution, a web server in 124, rather than 30, lines of C. No framework, no catch. This solution has the advantage of being a very high performance single threaded reactor and the compiled binary is only 10k on a Linux box.

It is very hard to accomplish anything useful in 30 or even 100 lines of code. This tells me that simply getting something to work, such as decoding and dispatching an HTTP request does not satisfy any sort of reasonable definition of accomplishment.

Innovation

Essentially the whole reinvent the wheel argument is a straw man. Once you have avoided reinventing the wheel, you still haven’t done any real work. For any real software problem, regardless of your foundation you will still have some very real work in implementing your custom solution.

I’m not going to take time and effort to introduce a dependency on some 3rd party without having it eliminate a substantial amount of code, perhaps 1000 lines or more. Many developers set this threshold at 0 lines: they always go for the framework approach. In many cases this leads to difficult projects with dependencies and interdependencies that simply confound progress.

I frequently think of the implementer of framework solutions and how many of those are implementing their framework because they rejected the previous generation’s framework. Apparently at some point they decided it was a good idea to reinvent the wheel. I’m going to go out on a limb and say that often it is a great idea to reinvent the wheel. In fact there is a word for it: innovation.

Finale

I don’t view the C api for epoll as somehow more opaque than the raw socket extension to 0MQ. But I do view it as straightforward, and already installed in my entire environment. It’s also battle tested and hardened by thousands of implementations, including all the network frameworks that implement it.

Perhaps some of you would argue that framework solutions allow people to solve problems they wouldn’t ordinarily know how to solve on their own. I regard this argument as tantamount to loading a gun and are pointing it at your foot. Don’t pull that trigger.