For example, MAP atom could be followed by a list of fields such as MAP, {1,55}, {2, 71}, {3,4002}, {0,0} to mean that in the current message type field id 1==55, 2==71, 3=4002 in this and later messages of the same type.

This would almost entirely eliminate the need for the developers to pick good values for the field numbers, ‘reserve’ some of the lower ones in case some common field is added later, or to even care much about field id values. They could just be assigned in order as fields are added. This assumes some kind of persistent stream, although it could also be done on a message basis, to optimize the size for arrays of messages for instance and still maintain random access of whole messages.

@tonyg - AMQP and DIY ASN1 PER. Very true and nice’n'compact. Also very safe in that all sizes are known in advance in the parse stream.
But, we need to sort out the extensibility and user-header type system better for AMQP1-0; some kind of TLV encoding would be beneficial (kind of DIY PER + BER, but nicer than BER).
Would like to discuss….

PS: Read up on ASN.1 again tonight and was reminded of why we didn’t use it.

We’ve seen cases where e.g. proxy servers would really prefer not to even scan sub-messages that they don’t care about. • I can see that would be of some benefit, but the proxy server still has to scan through and discard all of the sub-message it doesn’t care about, so all it saves is the effort of parsing it all to know where the end will be. That’s not no effort, but it’s a linear speedup for one rare case, whereas eliminating the effort of measuring the message sizes before sending saves work for every producer on every message sent. It’s neat that you have other uses for that precomputed size given that you have it, but it’s a long way from being a big enough advantage by itself to warrant producers putting that effort in every time!

But, yes, I think in the end, with both sides being heavily-optimized, the difference will probably not be that big. • I’d be curious to know what might be a good benchmark by which to measure things, so I can find out if big speed differences are possible…

Do you cache information about the size of sub-elements when you’re measuring the size of an element, or count it again?

The sizes are computed once and cached.

are there circumstances where it would be much harder to parse the message once and store indexes into it for the bits you need?

We’ve seen cases where e.g. proxy servers would really prefer not to even scan sub-messages that they don’t care about.

It sounds like getting a competitor to compete with what you have in speed would be a big job even if it started from a better wire format.

I think you could go a long way pretty easily by just having an option that replaces varint with some faster integer encoding. Basically all you’d have to do is add the right methods for the new encoding to Coded{Input,Output}Stream and then go through the code generator and make sure that wherever it currently generates calls to the varint methods, it instead uses your new methods. You could also trivially swap in startgroup/endgroup encoding for sub-messages; all the details are already implemented (despite being deprecated).

But, yes, I think in the end, with both sides being heavily-optimized, the difference will probably not be that big.

PB and AMQP are complimentary. AMQP would just provide a carrier for PB encoded messages, so the receiver would unmarshal the AMQP wire frames and then unmarshall the PB message.

HTH,

Ben