ydb/build/plugins/pybuild.py, branch CLI_2.19.0Mirror of YDB github reposhttps://code.mastervirt.ru/ydb/atom?h=CLI_2.19.02025-01-30T07:01:32Z[build/plugins/pybuild] Avoid collision in module names in venv mode2025-01-30T07:01:32Zprettyboyprettyboy@yandex-team.com2025-01-30T06:41:16Zurn:sha1:63389c48bc8b9423279edaea502967abc8126bbc
commit_hash:c46828a2ae03a161b7aa2cc514ee2d83ac5d4c08
Remove outstanding Testenv parts2025-01-17T18:45:41Zmiroslav2miroslav2@yandex-team.com2025-01-17T16:59:52Zurn:sha1:6ddd7726ecd1e96930dd8fd50395e4bb7e7b91fc
commit_hash:5ef4be5db84fab0cc93a9435d7ab4b153c509dac
Prototype - external python files mode for tests2025-01-16T19:26:51Zprettyboyprettyboy@yandex-team.com2025-01-16T18:01:45Zurn:sha1:2f3fd95aac24e27a3b0aa2badda49db82bf36cc5
Режим позволяет перезапускать python тесты или интеграционные тесты на питон программы при внесении изменений в python файлы без перекомпиляции.
Сценарий:
- прогнали тесты
- внесли изменение в `library/python/func/__init__.py`
- перезапустили, смотрим на времена
На примере интеграционного devtools/ya/handlers/analyze_make/tests в codenv (base конфигурация).
#|
||
ya m -rA
|
ya m -rA --ext-py
||
||
real 10m25,928s
user 0m53,465s
sys 1m14,791s
```
Total time by type:
[935171 ms] [LD] [count: 3, ave time 311723.67 msec]
[ 13657 ms] [prepare:AC] [count: 4, ave time 3414.25 msec]
[ 10436 ms] [TM] [count: 1, ave time 10436.00 msec]
[ 10387 ms] [prepare:get from local cache] [count: 1521, ave time 6.83 msec]
[ 6480 ms] [TS] [count: 1, ave time 6480.00 msec]
[ 3823 ms] [prepare:put into local cache, clean build dir] [count: 7, ave time 546.14 msec]
[ 2279 ms] [CC] [count: 1, ave time 2279.00 msec]
[ 1251 ms] [PY] [count: 1, ave time 1251.00 msec]
[ 372 ms] [prepare:yt-store] [count: 2, ave time 186.00 msec]
[ 154 ms] [AR] [count: 1, ave time 154.00 msec]
...
```
|
real 0m22,094s
user 0m26,372s
sys 0m3,363s
```
Total time by type:
[13140 ms] [TM] [count: 1, ave time 13140.00 msec]
[ 351 ms] [prepare:yt-store] [count: 1, ave time 351.00 msec]
[ 81 ms] [prepare:tools] [count: 8, ave time 10.12 msec]
[ 32 ms] [prepare:AC] [count: 2, ave time 16.00 msec]
[ 31 ms] [prepare:get from local cache] [count: 7, ave time 4.43 msec]
[ 16 ms] [prepare:clean] [count: 3, ave time 5.33 msec]
```
||
|#
В codenv в base конфигуарции (4cpu) невероятно долго идёт линковка. Тот же самый сценарий, но на mous (54cpu)
#|
||
ya m -rA
|
ya m -rA --ext-py
||
||
real 0m25,435s
user 0m40,242s
sys 0m22,434s
```
Total time by type:
[30958 ms] [prepare:get from local cache] [count: 1521, ave time 20.35 msec]
[16293 ms] [prepare:AC] [count: 4, ave time 4073.25 msec]
[11161 ms] [LD] [count: 3, ave time 3720.33 msec]
[ 4256 ms] [TM] [count: 1, ave time 4256.00 msec]
[ 3529 ms] [TS] [count: 1, ave time 3529.00 msec]
[ 1542 ms] [CC] [count: 1, ave time 1542.00 msec]
[ 635 ms] [PY] [count: 1, ave time 635.00 msec]
...
```
|
real 0m18,321s
user 0m21,673s
sys 0m3,204s
```
Total time by type:
[10786 ms] [TM] [count: 1, ave time 10786.00 msec]
[ 1568 ms] [prepare:yt-store] [count: 1, ave time 1568.00 msec]
[ 151 ms] [prepare:tools] [count: 8, ave time 18.88 msec]
[ 61 ms] [prepare:get from local cache] [count: 7, ave time 8.71 msec]
[ 48 ms] [prepare:AC] [count: 2, ave time 24.00 msec]
[ 14 ms] [prepare:clean] [count: 3, ave time 4.67 msec]
```
||
|#
В новом режиме тест ожидаемо замедляется, так как все питон программы при каждом старте вынуждены читать файлы с фс (ещё через арк), а потом ещё и компилировать байткод, т.е. холодный запуск
commit_hash:24c5a46a8385d3c065abfb6fc5b40f7ad24bb1cb
Never compress python build resources to speedup python configuration and build2024-12-13T06:07:25Zspreisspreis@yandex-team.com2024-12-13T05:25:28Zurn:sha1:e90b83df32a421a93c021eaa91710c1193585f9a
There is no runtime perf impact, but
- The change reduces build graph by removing 2 nodes out of 5 for python which accelerates both configuration and build
- surprisingly in all experiments there were positive impact on binary size about 2MiB
commit_hash:909fdeeb91b7f20c749177d638e97d5a469e422d
tempo fix for win (pybuild)2024-11-22T17:41:05Zkuzmich321kuzmich321@yandex-team.com2024-11-22T17:28:02Zurn:sha1:770ab0b6bc25adf6a5282f70ac856d222f273ff1
Предлагаю пока так отрубить для винды. Хочется, чтобы покрытие работало
commit_hash:9b3ec4a246f6597795dc0b2d12d6524bb688e129
Python linting macroses2024-09-12T06:53:26Zalevitskiialevitskii@yandex-team.com2024-09-12T06:41:59Zurn:sha1:f3fc9bb5ecb70b4b2a340d935ea06c92307da3b1
Init
93a5052a4192d230f52b4b27c41c9c77bf8d5880
Add another part SDC contrib to allow NO_LINT2024-08-21T09:02:23Zkochanov-s-akochanov-s-a@yandex-team.com2024-08-21T08:47:21Zurn:sha1:84b94f3b97a138085aafa50e6b06a49a8d742256
Еще одна пачка проектных контрибов.
53591f2d7360aa45b12fa2215945c9fc3569df19
Enable windows long manifest for all pyprograms2024-07-30T12:59:42Zv-korovinv-korovin@yandex-team.com2024-07-30T12:40:00Zurn:sha1:f4ea68421d222127ba002ba4721b2c7f991badd3
d773653083f7c4c70a99266c290e3d22ab070ba2
[build] pybuild: Allow NO_LINT in sdg/sdc/contrib2024-06-24T04:19:37Zdeshevoydeshevoy@yandex-team.com2024-06-24T04:07:02Zurn:sha1:42a569433b9070da71d2b5f2f7bb619bd35c5274
dd6af97985a0184ae82d875be6732b1ca6c56962
PR from branch users/denvr/YT-21293_fork_python3_only_source82024-04-17T11:57:20Zdenvrdenvr@yandex-team.com2024-04-17T11:45:46Zurn:sha1:9b1470229ddab1b35b19c0372103f0f44ceafaa8
YT-21293 - Remove unused py2 components (examples)
(cherry picked from commit 44a5892bd46d37a013673c2af37e93a1b222fb6f)
YT-21293 - Fixt test (py2 namespace)
(cherry picked from commit d47f9446d3046823d3a9d1c2fea74a83436db4be)
YT-21293 - Update arcadia policy
(cherry picked from commit e0bbbfbd9b25eceff5c0631dd1bccc0334a10058)
YT-21293 - Recursion for python2, and common checks
(cherry picked from commit f5eaef0c51ac9e7a987a363fc59820026f470c4f)
77f526405863a4923b663a336a3ebeddecbab116