Safe from the Losing Fight

For my new programming side project I’ve started writing an NNTP server. Like many young children, I once looked up into the night sky and wondered, “what would it be like to write my own news server?” I’ve read the RFC before, but never got around to actually implementing one, mainly because writing my own database never really appealed to me. I’m just crazy like that.

That’s where Core Data comes in. It’s an Apple data modeling technology that wraps SQLite. I’ve been looking for an excuse to learn this very cool technology, and this seemed a good excuse as any. I have to say, Core Data is very easy to use. The only problem I had was I kept wanting to design database tables instead of designing an object model. (Where’s my primary and foreign keys??) Way too much MySQL beforehand.

I’ve decided to code name this ill-advised project “Wombat,” for several reasons. First, Wombat is mentioned a few times in the examples in RFC 977. Second, its a really dumb name, and what other kind of a name would you give to an NNTP server that only runs Mac OS X 10.4 and higher? Plus Wombats look ornery, and if this server is anything, it’s ornery.

I’ve actually got RFC 977 implemented now, with a couple of extensions, which is supposed to be the NNTP standard. After I got most of the commands implemented, I decided to try my server with some Mac OS X native newsreaders, just to see them interact.

Hahahaha!

That’s the sound of all the Mac OS X newsreaders not knowing what the heck to do with a news server that implements the NNTP standard. Or perhaps, more accurately, news servers that only implement the standard.

I tried several NNTP readers: Panic’s Unison, Mozilla’s Thunderbird, the venerable MT-NewsWatcher, and some lesser known ones like Newsflash, OSXnews, MaxNews, and Xnntp. Most of them could get the list of groups and how many articles there were, but that was it.

You see, there’s another RFC, called “Common NNTP Extensions” that describes many ad-hoc extensions that servers started implementing that are not in the standard (RFC 977). Well, it turns out every reader I could find requires at least some of these extensions to be implemented. Namely, most readers want XOVER, LIST OVERVIEW.FMT, and XHDR implemented. Basically, those commands help the reader retrieve header information (subject, from, etc) en masse, and far more efficiently than the standard allows.

I guess what was more shocking to me, was that none of the clients fell back to the standard commands, when The Wombat started kicking back “500″ codes (command not implemented). The readers either treated my response like a bug in the server (that’s what they told the user) or just assumed The Wombat returned an empty data set. I’m guessing that the common extensions have been around for long enough (documented in 2000) and widely implemented enough, that all readers simply assume they’re going to be there.

There were also problems with the couple of extension commands I did implement. Namely I wanted a way to require authorization, so I implemented the AUTHINFO command, as described in the “Common NNTP Extensions” RFC. Apparently I’m the only one reading from that document.

The way AUTHINFO works is the client can issue commands like it normally would, but at any time the server can kick back a response that essentially says “Sorry, you have to authenticate before you do that.” At that point the client gives a username and password, the server authenticates it, and the client goes on its merry way. That’s the way it supposed to work anyway.

Unison, on the other hand, wants to start shoving AUTHINFO commands at the server at random intervals. This is contrary to the RFC that says the client should never initiate authentications, but only provide it when requested. The RFC also says that if the client offers AUTHINFO commands when not requested, the server is supposed to reject them. Foolishly believing the spec, that’s how I implemented The Wombat. Well, if you ever turn Unison down, you hurt its little feelings and it starts talking about you behind your back to the user. Stuff like: “the server doesn’t like you anymore. It rejected your username and password.” Unfortunately, the RFC doesn’t have a response for “you dumb client, you’re already logged in as that user!” Of course, I ended up modifying The Wombat to accommodate Unison’s pushy style of authentication.

So at the end of the day I have a fully implemented NNTP server, with respect to RFC 977, and no reader will work with it. I guess I have some work to do.