ydb/util/charset, branch CLI_2.2.0Mirror of YDB github reposhttps://code.mastervirt.ru/ydb/atom?h=CLI_2.2.02023-02-07T18:14:00ZIntermediate changes2023-02-07T18:14:00Zrobot-pigletrobot-piglet@yandex-team.com2023-02-07T18:14:00Zurn:sha1:9f448c9c675e670d53bba7a8d74a995b2d9a3606Intermediate changes2023-02-07T14:37:14Zrobot-pigletrobot-piglet@yandex-team.com2023-02-07T14:37:14Zurn:sha1:94299c40d78c487307d3b744ecadf539dc13bb1aExternal build system generator release 122023-01-31T03:43:37Zrobot-ya-builderrobot-ya-builder@yandex-team.com2023-01-31T03:43:37Zurn:sha1:84d8b69eb78329102350117cecd3789f6b0a76bc
Update tools: yexportExternal build system generator release 82023-01-17T10:48:00Zrobot-ya-builderrobot-ya-builder@yandex-team.com2023-01-17T10:48:00Zurn:sha1:58a117b001b63f06104246f6a89c60128596b391
Update tools: yexportcharset: do not allow surrogate pairs in UTF-82023-01-16T18:39:14Zdpotapovdpotapov@yandex-team.com2023-01-16T18:39:14Zurn:sha1:328635a6bd949596c49a33c9c2b67d00cc2704db
By [RFC3629 section 3](https://datatracker.ietf.org/doc/html/rfc3629#section-3):
```
The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16
encoding form (as surrogate pairs) and do not directly represent characters.
```
Current implementation of `ReadUTF8CharAndAdvance` allows `UTF-8` encoded surrogate characters
like 0xED 0xA0 0xBD or 0xED 0xB3 0x9A leaving them in the strings
that cannot be processed afterwards by external programs like `iconv`.
This patch provides `strict` template flag that disables this leniency.
This flag is not enabled by default, because Arcadia already has hundreds of
tests with inputs containing such surrogate pairs and these tests breaks in strict mode
and there is a chance that prod might affected too.
SSE4 implementation doesn't perform any validation at all, so it is left unchanged.Sync linux-headers instead of using system ones2022-12-15T16:25:15Zthegeorgthegeorg@yandex-team.com2022-12-15T16:25:15Zurn:sha1:0c31d97afd7858af5f8f858c8addf91655ca9cccEnable linux-arm platform.2022-10-26T13:40:35Zdcherednikdcherednik@ydb.tech2022-10-26T13:40:35Zurn:sha1:b981cca279190176740c3bf5c5b4fd03be51b599Update external build system generation utility2022-09-01T02:47:31Zrobot-ya-builderrobot-ya-builder@yandex-team.com2022-09-01T02:47:31Zurn:sha1:e83b936566d59e48df0aa4c414beaf0c4519c772
* [cmake] Hierarchical add_subdirectroy structure
* [cmake] Support for same file compilation multiple times with different flagsNew cmake semantics for compiling with AVX and other specific flags2022-08-31T05:44:15Zignatignat@yandex-team.com2022-08-31T05:44:15Zurn:sha1:d04c6ee6620c95efeef71dbf0cae8b3fbd8906fafix ya.make2022-07-07T11:41:37Zmonstermonster@ydb.tech2022-07-07T11:41:37Zurn:sha1:06e5c21a835c0e923506c4ff27929f34e00761c2