ydb/library/python/runtime_py3/ya.make, branch meta-1.0.1Mirror of YDB github reposhttps://code.mastervirt.ru/ydb/atom?h=meta-1.0.12025-04-29T03:57:18ZSubinterpretor compatible __res module (2nd attempt)2025-04-29T03:57:18Zsnermolaevsnermolaev@yandex-team.com2025-04-29T03:44:21Zurn:sha1:713adc6a88be8af8a342a9a72a055b7ef6514563
Cython is not yet subinterpreter compatible. There are no ETA when cython is going to support subinterpreters.
This PR removes cython from hermetic python imoprt hooks in order to make them subinterpretr-compatible.
commit_hash:427b6f9db6afa6695659ee147621e1ccb391d3cb
Revert "Subinterpretor compatible __res module" rXXXXXX2025-04-15T05:21:15Zsnermolaevsnermolaev@yandex-team.com2025-04-15T05:08:40Zurn:sha1:0ca7cfb39890921e0fc5a77e2972842717989f96
commit_hash:1d9f5675b9d3ddaa44db4472376a06ad3e811e2d
Subinterpretor compatible __res module2025-04-14T03:28:07Zsnermolaevsnermolaev@yandex-team.com2025-04-14T03:15:42Zurn:sha1:76887f61431dc5999139a8d72882449ca1503660
Cython is not yet subinterpreter compatible. There are no ETA when cython is going to support subinterpreters.
This PR removes cython from hermetic python imoprt hooks in order to make them subinterpretr-compatible.
commit_hash:1b067c37f55a4f1d9a6172df7009c75231cc1e25
Prototype - external python files mode for tests2025-01-16T19:26:51Zprettyboyprettyboy@yandex-team.com2025-01-16T18:01:45Zurn:sha1:2f3fd95aac24e27a3b0aa2badda49db82bf36cc5
Режим позволяет перезапускать python тесты или интеграционные тесты на питон программы при внесении изменений в python файлы без перекомпиляции.
Сценарий:
- прогнали тесты
- внесли изменение в `library/python/func/__init__.py`
- перезапустили, смотрим на времена
На примере интеграционного devtools/ya/handlers/analyze_make/tests в codenv (base конфигурация).
#|
||
ya m -rA
|
ya m -rA --ext-py
||
||
real 10m25,928s
user 0m53,465s
sys 1m14,791s
```
Total time by type:
[935171 ms] [LD] [count: 3, ave time 311723.67 msec]
[ 13657 ms] [prepare:AC] [count: 4, ave time 3414.25 msec]
[ 10436 ms] [TM] [count: 1, ave time 10436.00 msec]
[ 10387 ms] [prepare:get from local cache] [count: 1521, ave time 6.83 msec]
[ 6480 ms] [TS] [count: 1, ave time 6480.00 msec]
[ 3823 ms] [prepare:put into local cache, clean build dir] [count: 7, ave time 546.14 msec]
[ 2279 ms] [CC] [count: 1, ave time 2279.00 msec]
[ 1251 ms] [PY] [count: 1, ave time 1251.00 msec]
[ 372 ms] [prepare:yt-store] [count: 2, ave time 186.00 msec]
[ 154 ms] [AR] [count: 1, ave time 154.00 msec]
...
```
|
real 0m22,094s
user 0m26,372s
sys 0m3,363s
```
Total time by type:
[13140 ms] [TM] [count: 1, ave time 13140.00 msec]
[ 351 ms] [prepare:yt-store] [count: 1, ave time 351.00 msec]
[ 81 ms] [prepare:tools] [count: 8, ave time 10.12 msec]
[ 32 ms] [prepare:AC] [count: 2, ave time 16.00 msec]
[ 31 ms] [prepare:get from local cache] [count: 7, ave time 4.43 msec]
[ 16 ms] [prepare:clean] [count: 3, ave time 5.33 msec]
```
||
|#
В codenv в base конфигуарции (4cpu) невероятно долго идёт линковка. Тот же самый сценарий, но на mous (54cpu)
#|
||
ya m -rA
|
ya m -rA --ext-py
||
||
real 0m25,435s
user 0m40,242s
sys 0m22,434s
```
Total time by type:
[30958 ms] [prepare:get from local cache] [count: 1521, ave time 20.35 msec]
[16293 ms] [prepare:AC] [count: 4, ave time 4073.25 msec]
[11161 ms] [LD] [count: 3, ave time 3720.33 msec]
[ 4256 ms] [TM] [count: 1, ave time 4256.00 msec]
[ 3529 ms] [TS] [count: 1, ave time 3529.00 msec]
[ 1542 ms] [CC] [count: 1, ave time 1542.00 msec]
[ 635 ms] [PY] [count: 1, ave time 635.00 msec]
...
```
|
real 0m18,321s
user 0m21,673s
sys 0m3,204s
```
Total time by type:
[10786 ms] [TM] [count: 1, ave time 10786.00 msec]
[ 1568 ms] [prepare:yt-store] [count: 1, ave time 1568.00 msec]
[ 151 ms] [prepare:tools] [count: 8, ave time 18.88 msec]
[ 61 ms] [prepare:get from local cache] [count: 7, ave time 8.71 msec]
[ 48 ms] [prepare:AC] [count: 2, ave time 24.00 msec]
[ 14 ms] [prepare:clean] [count: 3, ave time 4.67 msec]
```
||
|#
В новом режиме тест ожидаемо замедляется, так как все питон программы при каждом старте вынуждены читать файлы с фс (ещё через арк), а потом ещё и компилировать байткод, т.е. холодный запуск
commit_hash:24c5a46a8385d3c065abfb6fc5b40f7ad24bb1cb
Never compress python build resources to speedup python configuration and build2024-12-13T06:07:25Zspreisspreis@yandex-team.com2024-12-13T05:25:28Zurn:sha1:e90b83df32a421a93c021eaa91710c1193585f9a
There is no runtime perf impact, but
- The change reduces build graph by removing 2 nodes out of 5 for python which accelerates both configuration and build
- surprisingly in all experiments there were positive impact on binary size about 2MiB
commit_hash:909fdeeb91b7f20c749177d638e97d5a469e422d
Sync contrib/tools/python3 layout with upstream2024-02-18T23:50:43Zthegeorgthegeorg@yandex-team.com2024-02-18T23:38:52Zurn:sha1:d96fa07134c06472bfee6718b5cfd1679196fc99
* Move src/ subdir contents to the top of the layout
* Rename self-written lib -> lib2 to avoid CaseFolding warning from the VCS
* Regenerate contrib/libs/python proxy-headers accordingly
4ccc62ac1511abcf0fed14ccade38e984e088f1e
add ydb deps2023-09-29T09:41:34Znkozlovskiynmk@ydb.tech2023-09-29T09:24:06Zurn:sha1:e0e3e1717e3d33762ce61950504f9637a6e669edintermediate changes2022-03-15T10:35:56Zarcadia-devtoolsarcadia-devtools@yandex-team.ru2022-03-15T10:35:56Zurn:sha1:094638589de6a6c9f91fad0005843fc1c1adc957
ref:ca7a95e8c9a9d780f96497136a152091d54e61b5
Restoring authorship annotation for <abodrov@yandex-team.ru>. Commit 2 of 2.2022-02-10T13:47:18Zabodrovabodrov@yandex-team.ru2022-02-10T13:47:18Zurn:sha1:f02a874a7290593efbe4b3aeae69a04b46c1cc86Restoring authorship annotation for <abodrov@yandex-team.ru>. Commit 1 of 2.2022-02-10T13:47:18Zabodrovabodrov@yandex-team.ru2022-02-10T13:47:18Zurn:sha1:1b8c043abfc689b3d27384c4202b20031fc4ae31