I tend to agree :-) This is why I really think I should get my act together and clean up and publish whatever I have. I think the beauty in that scheme is that it doesn’t force you to store and manipulate your strings in one encoding — or even to care about the encoding.

Remember the discussion, for those who were interested and cared to follow it, on the Ruby list where people clamored for Unicode, and Matz promised an all-encompassing scheme [since Japanese tend not to use UTF]? My background is in East-Asian languages, so my focus in mb was on CJK and their encodings [way too many Asian pages NOT in Unicode], although I did add a bunch of Latin* and Windows codepages encodings.

The first few times I spoke of string manipulations on Erlang-related forums — the list and the IRC chat room mostly — I draw mostly the equivalent of blank stares. I guess most users, especially long-time ones, have zero need for anything but ASCII. Erlang shows its roots, and clearly, as far as TEXT is concerned, it sucks. Whether the Erlang team fixes that will depend on the users making them aware of their needs, and probably of our help. Honestly, I don’t want an mb module, I want a new string type and BIFs and code added to the erlang: module.

Check out http://www.erlang.org/ml-archive/erlang-questions/200511/msg00193.html.

(note change to pipermail in October ‘06 :-).

Jim: I can’t see any replies on-list. Care to summarise?

One additional feature to consider is the selective mapping of JSON object member name strings to Erlang atoms. This lets objects look a little more natural as Erlang terms:

    {obj, [{alpha, 0.123}, {renormalized, false}]}

(Ah, if only Erlang had a native dictionary type.)

If you’re worried about arbitrary JSON data exhausting the atom table, you can first try listtoexisting_atom/1 (since the member names your code is prepared to use will be pre-loaded, right?), falling back to the conventional string representation if the conversion fails.

Cons: Codec needs to perform possibly-redundant Unicode encoding/decoding steps to ensure that the binaries hold UTF8 even if, say, UTF32 were the format to be used on the wire

UTF-32 is, to a first approximation, never used on the wire. In principle the argument stands for UTF-16, but UTF-8 is significantly more commonly used in protocols.

Incidentally, there is a thread about this post on the erlang-questions list, subject “strings, json, and what happens now” (not in the archive yet, but it will be here).

The new string type is a tuple, {Encoding::atom(), String::binary()}, and i/o methods fold and unfold the data to and from the tuple. A little tedious, but seems to work so far. This File was produced by mb’s test framework.

The advantage I saw in going this route is that the data stays as it is in reality, and mb strings can accept many encodings transparently. The encoding-safe manipulation functions [eg, getNextChar() retrieves codepoints, not bytes], make sure I don’t botch the original.