Here is something I learned only several weeks ago. While working on Pipe Watch, I strayed into reading the standard.
* The ESC [ command start sequence is actually a compromise for 7 bit systems. The [ character is not chosen by accident. It has an obvious positional relationship to ESC in the ASCII code which is why, informally, Ctrl-[ is the same as ESC.
* If you have an 8-bit-clean channel to the terminal, only a single character is required: the "upper escape" from the C1 control character set (0x80 to 0x9F). This character, 128 + 27 or 0x9B is called CSI: control sequence introducer, which is basically its role in these terminal control sequences. Thus ESC [ is just an alternative way of encoding CSI for 7 bit. E.g instead of CSI 4 A, you use ESC [ 4 A.
> If you have an 8-bit-clean channel to the terminal, only a single character is required: the "upper escape" from the C1 control character set (0x80 to 0x9F).
I'd avoid using this. It conflicts badly with UTF-8's use of 0x80 through 0xBF as continuation characters.
I don't entirely buy the argument because in a regular ASCII control sequence like ESC[5A, you have the same problem. All those characters have a role outside of the signaling, so if either side is in an unexpected state, they get misinterpreted. This is just the risk of in-band signaling.
Of course, if the terminal is ignorant of UTF-8 then this is a nonstarter, because whenever CIS occurs as a continuation byte, it will be misinterpreted. If the terminal is ignorant of UTF-8, why would you send UTF-8 to it, though? If it's going to be interpreting UTF-8 as some branch of ISO Latin, the display will be a mess.
If the terminal and host do handle UTF-8, then this CSI signaling is just an extension of the state machine. It's also nice and simple that, in the absence of any data loss or synchronization error, the CSI code is unambiguously not part of any valid UTF-8 character (except as the second or subsequent byte where the receiver is in the right state to interpret it that way).
In my experience, it's the terminal->host direction where you get mixups, whereby the terminal generates some escape sequence like for an arrow key, but the host is not in the right state, and interprets part of it as data. This is exacerbated by a situation in which the host supports ESC as a UI command. CSI solves the abmbiguity between the control sequence start and ESC just being ESC.
The fundamental problem with mixing C1 controls with UTF-8 is that it forces the terminal emulator to break layering. It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either, because that'll treat a lot of the UTF-8 continuation characters as control sequences. And what you're likely to find if you start using C1 controls is that support for them in terminal emulators is often incomplete and/or buggy. Handling them correctly in conjunction with UTF-8 text is difficult, and many terminals just don't bother.
The ambiguous nature of ESC in the terminal->host direction (as you put it) is unfortunate, but is difficult to fix. Some terminals (like iTerm) can be configured to use C1 controls for function keys, but my experience has been that a lot of software fails to recognize these sequences, making it impractical to use.
The fundamental problem with TCP/IP is that it forces the stack to break layering. In the same frame of bytes, you have a confusing mix of ethernet addressing, IP header, and a payload of application data, all from totally different pieces of software in the system. Even the data itself is fragmented, with some session wrapping around content that are done by different application stacks.
> It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either because that'll treat a lot of the UTF-8 continuation characters as control sequences.
It has to have a state machine which recognizes the combined language of UTF-8 sequences and control sequences. Which is the approach you would take anyway, even with C0 controls.
That combined language is an unambiguous, regular set, so you could code it with your eyes closed.
Starting in an initial state, the legal inputs are: ASCII character, Unicode character, or escape sequence headed by CSI. This is decidable from reading exactly one byte value with no further lookahead.
That's just one way. You can in fact follow a layered approach whereby the terminal decodes everything with UTF-8 before analyzing it for control or data.
For instance, say we decode UTF-8 into integer code points. A valid character decodes into its implied code point. An invalid byte like CSI can decode into some reserved range like U+DCxx. The higher layer of the terminal's firmware then looks for values in that U+DCXX range: that's where it finds the CSI.
I have years of experience with this exact encoding scheme, which I baked into the text I/O streams of a programming language.
For instance, oh, /proc/self/environ is NUL-separated, right? No problem:
The NULs are rendered into \xDC00 codes. This is called the "pseudo-null" character in the terminology of this language, and has a symbolic name: #\pnul:
2> #\xDC00
#\pnul
We can split the data on it to recover the list of environment entries:
Does it? That depends on whether the control sequences are being encoded as UTF-8, or transmitted literally in between UTF-8 characters.
If they are being encoded, this is still useful. Though space isn't saved, any ambiguity between the control bytes and UTF-8 bytes is eliminated. The advantage is still present that CSI is different from ESC, and so in the terminal->host direction, you don't have the ambiguity between ESC as a UI command character versus control sequence signal byte.
* The ESC [ command start sequence is actually a compromise for 7 bit systems. The [ character is not chosen by accident. It has an obvious positional relationship to ESC in the ASCII code which is why, informally, Ctrl-[ is the same as ESC.
* If you have an 8-bit-clean channel to the terminal, only a single character is required: the "upper escape" from the C1 control character set (0x80 to 0x9F). This character, 128 + 27 or 0x9B is called CSI: control sequence introducer, which is basically its role in these terminal control sequences. Thus ESC [ is just an alternative way of encoding CSI for 7 bit. E.g instead of CSI 4 A, you use ESC [ 4 A.