I was involved with the effort [to avoid] standardizing instant messaging back in 1999-2000. One of the products the company I was with at the time was offering was an instant messenger (with a twist that is not important for this story), so we tried to support all the prevailing protocols of the time (AIM, ICQ, MSN Messenger, Yahoo Messenger, PowWow, IRC, ... I don't even remember). And I was eagerly following the instant messaging standardisation effort at the IETF.
What became painfully clear to me after a while was that the big guys (AOL=AIM+ICQ, MS, YAHOO) who were running the show were all actively making sure that no standard would be decided upon. It was all dressed up in technical arguments, so it didn't look that way, but it was. Perhaps the engineers participating in the discussion weren't even aware -- it's easy to miss the big picture -- but as an outsider it was painfully obvious that the effective mandate that was given by the respective companies to their representative was to NOT standardize anything.
What ended up being the IETF standard was Jabber -- which was way too complicated and wasteful. And the reason it ended up that way was that while the big guys were avoiding standardisation, someone just put out an open source / open spec product which -- with all its deficiencies -- eventually worked well enough to gain non-trivial adoption, including by the IETF. And all of that happened awhile after the instant messaging working group was disbanded because of "irreconcilable differences" -- those differences being that every participant would only accept a standard that forced others to concede defeat and implement that participant's existing system (always in a way that limited everyone except said participant, of course)
Watching those discussions was an eye opener about how things supposedly work (and even look that way on the surface), and how they really work.
EDIT: REMOVED claim that Jabber/XMPP is still too complicated. I'm sure it still has 90% bandwidth and parsing overhead compared to comparable protocols (unless it's become backward incompatible), but standards have changed, and I guess that's no longer considered wasteful or bad engineering practice.
EDIT: ADDED: Just to be clear, this is an anecdote about the distant past (as in, 12 years ago now!). I have not been following Jabber/XMPP closely since 2004 and not at all since 2006 or so. Back in the day it was ill-and-under specified, ridiculously chatty (as in >90% overhead before you count TCP headers, both in bandwidth and parsing). What is acceptable has changed since then (no one cares about 95% overhead anymore), and for all I know all the kinks have been worked out.
And for the good parts - it is out there, there are multiple server and client implementation, and it works. That's a thousand times better than the product I was working on at the time, which was proprietary closed source, and though it was super efficient in every way, was discontinued and is no longer available.
I haven't been following it since 2006 or so, but unless it got simpler since then, yes it very much is still too complicated. It does get the job done, and is in widespread use, and has multiple server and client implementations, which makes it a gazillion times better than almost anything else out there. Hats off.
It was unbearably chatty - to the point of being unusable as a server-to-server protocol in a high-load environment in 2003. internal protocols (and even external protocols such as OSCAR, ICQ, and MSN Messenger) required 10% of the bandwidth and 10% of the CPU power. Jabber was usable but very expensive as a client-to-server protocol.
Nowadays, CPU is much cheaper and so is bandwidth, so these problems are gone. And perhaps clients are better behaved and standardised now (I have no idea how video, file transfer, etc are handled these days; the answer back in the day was "not at all, or in a complex and non standardized way").
As someone deeply concerned with real-world operation of systems, I find your removal of the "claim" distressing.
It is true that developers routinely create very poorly thought out protocols that have massive overhead, but it is far from good practice, and anyone who tries to foist such a disaster on me for any reason other than business-critical interoperability will be SOL. I simply will not deploy it, they will have to go back and create something that is at least moderately sane.
XMPP does retain massive overhead, in excess of 50% even before considering the XML bloat (I'm just referring to redundant or otherwise unnecessary messages here), and that's the main reason I have refused to use it in my projects.
Premature optimisation is the root of all evil. If you're deeply concerned with real-world operations, you should be painfully aware of this.
XMPP is primarily an instant messaging protocol. An overhead of 50% is nothing if the payload itself is often just a few bytes, and a few kilobytes at most. Of course it makes XMPP unsuitable for a few scenarios, but in almost all cases, XMPP message traffic will be dwarfed by other transfers. I think you have to think pretty hard to come up with a situation where you'd really want to use XMPP if only it wasn't for the overhead. Huge amounts of small messages exchanged between embedded devices, maybe?
Incidently, your message had 722 characters, the whole page had 4.83 kb. Pretty bad overhead, and of course HN is extremely frugal. I'm sure Twitter is much much worse. Apples and oranges, I know.
Argument by cliché, and a badly misunderstood one, at that. It's not a pass to engage in poor design practices. Would you deploy bubblesort on the grounds that quicksort is a premature optimization?
> An overhead of 50% is nothing if the payload itself is often just a few bytes, and a few kilobytes at most
You've just given me an estimated range of three orders of magnitude. As it happens, that is also the range I commonly get from people who have just "designed" a protocol with zero thought.
> I think you have to think pretty hard to come up with a situation where you'd really want to use XMPP if only it wasn't for the overhead. Huge amounts of small messages exchanged between embedded devices, maybe?
When I'm trying to process 10,000 messages per second on a CPU, and most of the overhead is already in ingesting, parsing, and routing each message, a 50% increase in message quantity means I need 50% more CPUs.
> Incidently, your message had 722 characters, the whole page had 4.83 kb.
I don't have to design, implement, or deploy websites, and consider the entire model quite broken for communications, so the comparison changes my opinion not at all.
> Would you deploy bubblesort on the grounds that quicksort is a premature optimization?
Unless it turns out the sorting routine has a significant impact on the overall performance, I wouldn't care a bit if it used Bubblesort, Quicksort or Bogosort.
> You've just given me an estimated range of three orders of magnitude. As it happens, that is also the range I commonly get from people who have just "designed" a protocol with zero thought.
Argument by snark. The assumption is that a chat protocol is developed for certain payload sizes because there is a trade-off involved: you'd do things differently if you had to create a protocol that works for 5 byte messages exactly as well as it does for 5 GB messages. Under that assumption, it's perfectly reasonable to give a range of expected message sizes. And since Jabber was developed in the first place as an IM protocol, 5 to 5000 bytes seems about right, with the distribution heavily skewed towards smaller messages.
> When I'm trying to process 10,000 messages per second on a CPU, and most of the overhead is already in ingesting, parsing, and routing each message, a 50% increase in message quantity means I need 50% more CPUs.
Yes. So? How many people are running large enough Jabber servers that this is a problem for them? And why can't they simply buy another CPU?
> Unless it turns out the sorting routine has a significant impact on the overall performance, I wouldn't care a bit if it used Bubblesort, Quicksort or Bogosort.
By your evident logic, then, Bogosort is the right default choice, because anything else is premature optimization.
> Yes. So? How many people are running large enough Jabber servers that this is a problem for them?
Well, the person I originally replied to appears to have been one of them. As are Google, Microsoft, AOL, Facebook, and Yahoo. I know one much smaller company I can't name who did field trials of XMPP on their embedded consumer device and promptly dropped the project.
Had I made the mistake of trying the same thing, I would now be in such a position.
> And why can't they simply buy another CPU?
And the server to put it in, and the space, and the power, and the cooling. I might be able to get a chip delivered for $200, but it's going to cost me another $400-800 plus people time to put it into service, and another $500+/year to run it.
Far cheaper to make the engineer spend a day or two to come up with something reasonable.
> By your evident logic, then, Bogosort is the right default choice, because anything else is premature optimization.
How does that follow from my logic? What I said was: I don't care about implementation details unless it turns out to be relevant. If I spent 99% of my time waiting for IO, and 1% on sorting, then yeah, the sort routine could be Bogosort for all I care, since the potential gains are tiny.
The right default choice for sorting is your standard library's sort routine. If you use that and sorting still takes a long time, then you can worry about what goes on in there and if it's worth replacing it.
> As are Google, Microsoft, AOL, Facebook, and Yahoo.
> I don't care about implementation details unless it turns out to be relevant.
Then you are reckless. It is both possible and desirable to anticipate obvious problems in a design and avoid them before they present themselves operationally.
> The right default choice for sorting is your standard library's sort routine.
Smarter people than you or I decided what the standard library's sort routine would be, and did so with careful consideration of the advantages and disadvantages of their choice.
Here we are speaking of the design of a protocol from scratch, and/or the explicit choice of a protocol for a given set of requirements. It is your duty to do as the standard library author did, and pay careful consideration to the implications of your design, as it will impact your business for years to come.
> All of which now use XMPP despite the overhead.
They have an interoperability requirement. Did you miss that detail of my earlier comment?
Okay. I agree. But carefully considering your options and paying mind to your requirements is a far fetch from flat out disregaring one option simply because of a detail -- overhead -- that may or may not be relevant in your situation. I still think you could refer to this as a kind of premature optimisation, although I see now that this was unhelpful, because it brought up the whole sorting thing which was kind of a tangent.
> They have an interoperability requirement. Did you miss that detail of my earlier comment?
Some of them do. Some of them explicitly turn off interoperability and still go with XMPP; Whatsapp (no affil) is another one of those. This just stresses the fact that efficiency/overhead is not the only consideration when it comes to protocol choice, and pretty far down the line when it comes to IM protocols.
Considering a single TCP segment will probably be used with or without XML bloat, is that really important nowadays? Not that it isn't bad practice (something in e.g. Protocol Buffers should probably be used), but to refuse to use it seems harsh.
XML bloat would still add to bandwidth costs (or add more CPU/RAM overhead if you compress it), TCP segments are not fixed size.
I know there's this tendency to dismiss bandwidth these days, but it's quite misguided. You literally wouldn't believe the quotes we got in Taiwan -- I thought there was some major misunderstanding until I got there and actually sat down with the network team at the telecom provider.
What became painfully clear to me after a while was that the big guys (AOL=AIM+ICQ, MS, YAHOO) who were running the show were all actively making sure that no standard would be decided upon. It was all dressed up in technical arguments, so it didn't look that way, but it was. Perhaps the engineers participating in the discussion weren't even aware -- it's easy to miss the big picture -- but as an outsider it was painfully obvious that the effective mandate that was given by the respective companies to their representative was to NOT standardize anything.
What ended up being the IETF standard was Jabber -- which was way too complicated and wasteful. And the reason it ended up that way was that while the big guys were avoiding standardisation, someone just put out an open source / open spec product which -- with all its deficiencies -- eventually worked well enough to gain non-trivial adoption, including by the IETF. And all of that happened awhile after the instant messaging working group was disbanded because of "irreconcilable differences" -- those differences being that every participant would only accept a standard that forced others to concede defeat and implement that participant's existing system (always in a way that limited everyone except said participant, of course)
Watching those discussions was an eye opener about how things supposedly work (and even look that way on the surface), and how they really work.
EDIT: REMOVED claim that Jabber/XMPP is still too complicated. I'm sure it still has 90% bandwidth and parsing overhead compared to comparable protocols (unless it's become backward incompatible), but standards have changed, and I guess that's no longer considered wasteful or bad engineering practice.
EDIT: ADDED: Just to be clear, this is an anecdote about the distant past (as in, 12 years ago now!). I have not been following Jabber/XMPP closely since 2004 and not at all since 2006 or so. Back in the day it was ill-and-under specified, ridiculously chatty (as in >90% overhead before you count TCP headers, both in bandwidth and parsing). What is acceptable has changed since then (no one cares about 95% overhead anymore), and for all I know all the kinks have been worked out.
And for the good parts - it is out there, there are multiple server and client implementation, and it works. That's a thousand times better than the product I was working on at the time, which was proprietary closed source, and though it was super efficient in every way, was discontinued and is no longer available.