Design Considerations

A conforming shell MUST use UTF-8 as the default encoding for reading text files, writing text files, text passed through pipes between commands, and console input and output. A conforming tool MUST read and write text as UTF-8 unless an encoding is explicitly requested. The default MUST NOT depend on the operating system’s locale, ANSI code page, or OEM console code page.

A conforming shell or tool MUST NOT convert text to or from any other encoding (including the system ANSI code page, the OEM console code page, and UTF-16) unless that conversion was explicitly requested ([te-3]). Input that is not valid UTF-8 MUST NOT be silently reinterpreted in another encoding or silently replaced with substitution characters. It MUST either be passed through as unmodified bytes (binary) or be rejected with an encoding error ([te-6]).

A non-UTF-8 encoding MUST only be used when selected explicitly for that operation, for example --encoding=windows-1252 (long flags follow the SOS grammar). Each use SHOULD emit a diagnostic on the diagnostics channel, and MUST be reported in structured output, including the --dry-run execution tree, as the encoding actually used. A user, profile or environment setting that changes the default away from UTF-8 SHOULD produce a warning when the shell starts, and MUST be discoverable through [te-7].

Readers MUST accept a UTF-8 byte-order mark and MUST NOT expose it as content. Writers MUST NOT add a byte-order mark unless explicitly requested. When a conforming shell or tool rewrites an existing file in place, it SHOULD preserve that file’s existing byte-order mark (present or absent) and line-ending convention. Input that begins with a UTF-16 or UTF-32 byte-order mark MAY be decoded, but MUST be reported as a non-default encoding in structured output and MUST NOT be written back in that encoding unless requested.

Reading a valid UTF-8 text file and writing the result back without modification MUST produce byte-identical output. An edit MUST change only the bytes of the content being edited; characters outside the edited range MUST be preserved exactly.

An encoding failure (invalid byte sequence, unexpected byte-order mark, UTF-16 where UTF-8 was expected, or a conversion that would lose characters) MUST result in a non-zero exit status or a structured error value, never in partially converted output presented as success. The error MUST be available in machine-readable form with at least: the path or stream name, the byte offset of the first failure, the encoding that was assumed, and an error kind.

{
  "error": "encoding",
  "kind": "invalid-sequence",
  "path": "CHANGELOG.adoc",
  "byte_offset": 1842,
  "assumed_encoding": "utf-8"
}

A conforming shell MUST expose, through a documented command or structured variable, the effective encodings it is using for file reads, file writes, pipes and the console, so that an agent can verify them before editing files.

When the host console is configured for a non-UTF-8 code page, a conforming shell SHOULD switch its session to UTF-8. If it cannot, it MUST NOT silently decode native-command output through the legacy code page; it MUST report the mismatch as in [te-6].