summaryrefslogtreecommitdiffstats
path: root/yql/essentials/providers/common
Commit message (Collapse)AuthorAgeFilesLines
* YQL-21315: Block variant item implementationatarasov54 days1-0/+5
| | | | | BlockVariantItem implementation commit_hash:f35a597dee3dc67ebc5be599ed9d379d7ad57802
* YQL-21404 implemented syscache, table, func callvvvv7 days1-4/+14
| | | | | | | | | | | | | | #### `YQL-21404`: Implemented syscache, table, and function call support for collations ✎ - Added support for PostgreSQL collation handling in function calls, enabling explicit collation specification via named arguments - Implemented syscache and table catalog support for pg_collation system table to provide complete catalog information - Enhanced type annotation and runtime compilation to properly handle collation OIDs and validate collation names - Introduced ICU locale collation support with stable OID assignment based on locale position in generated catalog - Added comprehensive test coverage for collation functionality including standard and ICU locales, collation validation, and error cases - Extended SQL parsing to support COLLATE clauses and proper collation propagation in function calls <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:e9a32cc848bef7b68b4f57f71a1f5f69e0de9653
* YQL-21315: Implement block variant comp nodeatarasov57 days1-0/+13
| | | | | | | | | | | | | | #### Implement Block Variant Computational Node ✎ - Introduces a new computational node for handling variant types in block operations, supporting both tuple and struct variants - Adds type annotation and runtime support for the new variant node with proper type checking and validation - Implements block-level variant operations that can select specific alternatives from variant types based on index or member name - Includes comprehensive unit tests and integration tests covering both tuple and struct variant scenarios - Updates runtime version to 81 and integrates the new node into the compilation and execution pipeline - Provides SQL syntax support for variant operations through the `Variant` function with appropriate type handling <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:a07c1a365815a271cdd34958369156fd542fd5cc
* Fix YQL native YT type flags: Complete Editionziganshinmr11 days1-2/+2
| | | | | | | - Cherry-picked original "Fix YQL native YT type flags" and "Fix YT native types handling for aux columns - 2" with additional fixes (previously reverted) - Enforce mandatory usage of the type v3 schema for output tables - Major refactoring of Skiff encoder/decoder to fix non-complex nativeness support commit_hash:2b4241624a261a690c916a7386b84cb3d483a373
* YQL-21101: Make ToFlow dependent args requiredimunkin11 days1-1/+1
| | | | commit_hash:bc73ed520d5d4ef210b03b983720e6e8a6996db5
* YQL-21370: Support TCoreAttr::Args at SqlCorevitya-smirnov11 days2-20/+49
| | | | | | | | | | | | | #### Support TCoreAttr::Args at SqlCore ✎ - Introduces support for extended SQL flags with arguments in the SqlCore configuration, allowing flags to carry additional parameters for more granular control - Refactors internal flag handling to use a map-based structure instead of a simple set, enabling storage and processing of flag arguments - Updates module resolver and translation settings parsing to work with the new extended flag format, ensuring proper flag propagation and application - Adds new infrastructure for parsing and managing SQL translation flags with arguments, including dedicated parsing logic for specific flag types - Maintains backward compatibility while enabling new functionality for SQL flag configuration through gateways and command-line options <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:8b26974b3bf4ff2bdaf76715cbb210a8a15e1dad
* Use credentials for downloads across UDF index & UDF meta task - second attemptziganshinmr13 days3-7/+46
| | | | | | - Cherry-pick of original "Use credentials for downloads across UDF index & UDF meta task" - Fix wrapping up UDF resolver with QPlayer adapter commit_hash:df724df8c5b821f4c30ebf4359730107a6dbc253
* YQL-21315: Block way implementationatarasov52026-07-071-0/+5
| | | | | | | | | | | | | | | #### Block Way Implementation ✎ - Implemented the `BlockWay` function to extract variant type information from block inputs, returning either tuple indices or struct member names - Added type annotation support for `BlockWay` operations with proper handling of optional variants and different underlying types - Introduced computation node implementation for `BlockWay` that processes both tuple and struct variants efficiently - Enhanced runtime version handling to support the new `BlockWay` functionality with version 80 - Added comprehensive unit tests covering various scenarios including tuple variants, struct variants, optional variants, and edge cases - Integrated `BlockWay` into the SQL parser and program builder with proper type inference and validation - Added SQL format tests and canonical data for tuple and struct variant scenarios with and without optionality <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:70a82f5d13a7c1499a9ad88fc5d9b079e352d7ef
* KIKIMR-25621: Introduce multi-column statisticsazevaykin2026-07-072-0/+23
| | | | | | | | | | | | | | #### Introduce multi-column statistics support ✎ - Adds support for defining multi-column statistics in table schemas, allowing users to specify statistics for combinations of columns rather than individual columns - Introduces new SQL syntax for CREATE TABLE and ALTER TABLE statements to declare statistics with optional type specifications - Implements validation for statistics definitions, ensuring column references are valid and names are unique - Extends the query processing pipeline to handle statistics declarations during table creation and modification operations - Updates syntax highlighting and parsing rules to recognize the new statistics keyword and related constructs - Adds comprehensive unit tests covering various statistics scenarios including multi-column definitions, type specifications, and validation rules <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:5ca3f8721d3e3660f2bb890c1408e420cf7c3e81
* Revert commit rXXXXXX, Use credentials for downloads across UDF index & UDF ↵ziganshinmr2026-07-063-46/+7
| | | | | | | | | | | | | | | meta task #### Revert URL preprocessing and credentials handling in UDF resolution ✎ - Removed URL preprocessing dependency and related token resolution logic from UDF index integration - Simplified UDF resolver interface by eliminating optional URL preprocessing and token resolver parameters - Removed credential parsing and handling from UDF meta tasks, removing the dependency on credential management components - Streamlined file storage operations for UDF downloads by removing URL preprocessing and token handling steps - Updated task creation logic to remove URL preprocessing configuration and credential processing in meta task execution <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:e490d4d4cbb7a4cba7b1fd8dea609ca2d314af0e
* Use credentials for downloads across UDF index & UDF meta taskziganshinmr2026-07-033-7/+46
| | | | | | - Use token resolver for downloading URLs from UDF index - Support credentials passing to yqlworker's UDF_META query commit_hash:bad5c0d991097ece5eeda4a5d81aff0a0a429e36
* YQL-20095: Tune bugprone-argument-commentvitya-smirnov2026-06-3018-70/+70
| | | | | https://clang.llvm.org/extra/clang-tidy/checks/bugprone/argument-comment.html commit_hash:780ff6a63be44998cb2336a05f3dc33e68a676bf
* YQL-16277: Implement ExpandSqlCombine optimizerimunkin2026-06-291-4/+17
| | | | | | | | | | | | | | #### Implement SqlCombine optimizer and enhance ListJoinCore type checking ✎ - Introduce ExpandSqlCombine optimizer to handle SQL COMBINE operations, enabling efficient execution of combine operations with proper key handling and sorting - Refactor ListJoinCore type annotation logic to use tuples instead of structs for key types, improving type safety and consistency in multi-key scenarios - Add auxiliary constants for consistent naming of input prefixes and key members in join operations - Update MKQL compiler and type annotation components to support the new SqlCombine expansion and improved type checking - Enhance test coverage with new test cases for various SqlCombine scenarios including single key, multi-key, optional keys, and type mismatches - Adjust existing tests and canonical data to reflect the changes in type handling and naming conventions <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:376f67bfed851d0e4f7ac354d22220ba4f4737e2
* YQL-21315: Implement block guessatarasov52026-06-261-0/+15
| | | | | | | | | | | | | #### Implement BlockGuess functionality for variant type extraction ✎ - Introduces BlockGuess functionality to extract specific alternatives from variant types, supporting both tuple and struct variants - Adds type annotation and runtime implementation for BlockGuess operations with proper validation and error handling - Implements peephole optimization support for BlockGuess operations to improve query execution plans - Includes comprehensive unit tests and SQL format validation for various BlockGuess scenarios - Updates runtime version to 79 to enable new BlockGuess features and ensures backward compatibility <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:e8f7fc2859cc1986ee0d5fba8b3e25bbf9ba4c41
* SPYT-1033: revert cluster-default network_project from SPYT gatewaymihailagei2026-06-231-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The cluster-level NetworkProject default added in the previous PR was conceptually wrong: spark.ytsaurus.network.project is a property of each user's ABC access, not of the SPYT cluster. There is no universal default that works for all users, so any value baked into gateway config either misleads or breaks queries. The correct path is per-query override via the existing parameters.spark_conf.spark.ytsaurus.network.project hook in SpytOperationsRunner.formatSettings; that path is unaffected by this revert. Reverts the following components added in PR 14020425: - TSpytClusterConfig.NetworkProject proto field - Cluster.CustomFieldType.SPYT_NETWORK_PROJECT enum value - 6th Optional<String> networkProject parameter of Cluster.newSpyt - networkProject reading in ClustersManager.getSpytClusters - SpytOperationsRunner.buildSparkConfFromCluster and the third cluster layer in the spark_conf merge - SPARK_NETWORK_PROJECT_KEY constant - NetworkProject: "test_spyt_project" line in yql-test/gateways.conf - Related tests + the spytCluster(discoveryPath, networkProject) helper overload DiscoveryPath stays — it is a legitimate cluster property (canonical discovery path set by ops once per cluster) and is unrelated to per-user ABC access. commit_hash:9c167a5ebef7c3458f1691fe00c81674bb694789
* SPYT-1033: add network project to SPYT gateway configmihailagei2026-06-221-0/+1
| | | | | | | | | | | | | #### `SPYT-1033`: Add network project support to SPYT gateway configuration ✎ - Introduce network project configuration option for SPYT clusters to resolve connectivity issues in MTN environments - Extend cluster creation logic to propagate network project settings from gateway configuration to driver processes - Implement hierarchical configuration merging that respects cluster defaults, subclass settings, and per-query overrides - Add unit tests to verify network project propagation and override behavior - Update test configuration to include network project for the hahn cluster <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:cc4dd1958ed7ccdd4f8e5680e5d15a19d804eb78
* YQL-21207: Hash runtime settingsatarasov52026-06-193-5/+117
| | | | | | | | | | | | | | #### Hash runtime settings for stable identification ✎ - Introduces a stable hash function for runtime settings to enable consistent identification of configuration states - Implements `SHA-256` based hashing that includes both host settings and UDF settings with deterministic ordering - Ensures hash stability regardless of setting insertion order by using ordered data structures - Adds comprehensive unit tests to verify hash consistency, content sensitivity, and order independence - Enhances configuration serialization to properly count and filter serializable settings - Makes serialization order deterministic by switching from hash map to map for setting storage <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:49aafc63bf011c335d556cd41dd4971473b62f13
* YQL-16277: Implement MKQL provider for ListJoinCoreimunkin2026-06-171-0/+38
| | | | | | | | | | | #### Implement MKQL provider for ListJoinCore callable ✎ - Added support for compiling the `ListJoinCore` callable within the MKQL (MapReduce Query Language) provider, enabling its use in query execution plans - Implemented the necessary type and lambda building logic to correctly handle input structures, key types, and join operations for the new callable - Updated test configurations to remove xfail status and include new test artifacts for the ListJoinCore functionality, indicating successful integration and testing <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:dbddedaedf354d5e9cf66c69ab3c5f9a3b58cc32
* SPYT-1033: YQL-api/QT - SPYT available for usemihailagei2026-05-221-1/+13
| | | | commit_hash:fa96697e13ad7c5b011a3dc789d87d1e65ea8667
* YQL-19813: Remove MergeFrom usageatarasov52026-05-221-1/+1
| | | | commit_hash:655c60d1cc5fa52547ddd15cafa05f97de8f97b7
* YQL-19813: Save and load qplayeratarasov52026-05-205-52/+345
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Режим capture** (qContext.CanWrite()) 1. Проходим по всем элементам source с индексом idx. 2. Для каждого элемента вызываем filter(item) ровно один раз (фильтр может иметь побочные эффекты — например, запись статистики об активации). 3. Элементы, прошедшие фильтр, добавляются в возвращаемый результат. 4. Сохраняем индексы всех прошедших фильтр элементов в QStorage в формате V2: \{"version": "v2", "indexes": \[0, 2, 5, ...\]\} --- **Режим replay** (qContext.CanRead()) 1. Читаем YSON-запись из QStorage по ключу \{Component="Activation", Label=label\}. 2. Определяем версию формата по наличию ключа "version": **Формат V1** (старый, обратная совместимость) — ключ "version" отсутствует: - Структура: \{"имя\_флага": сериализованный\_proto, ...\} - Десериализуем каждый флаг через ParseFromString, возвращаем список напрямую. - Фильтр не вызывается. Список source игнорируется. **Формат V2 (текущий)** — "version": "v2": - Структура: \{"version": "v2", "indexes": \[0, 2, 5, ...\]\} - Читаем сохранённое множество индексов. - Проходим по source с *индексом* idx: - Элемент без условия активации (!HasActivation()) → всегда включается (такие элементы не зависят от активации). - Элемент с условием активации (HasActivation()) → включается только если его idx есть в сохранённом множестве. - Фильтр не вызывается. commit_hash:306113465bcd614cd3c0cd311ab20542d1f8e254
* YQL-21156: add cluster_use to postgres, return 404 for non single clutersvafilonov2026-05-191-0/+1
| | | | commit_hash:8ad57b1923c0f6a3bc1c0699a18d32d44bfe9412
* YQL-20095: Enable modernize-use-emplacevitya-smirnov2026-05-131-2/+2
| | | | commit_hash:4d6e2f6fcc1639f09725c0c96de8d2f7f0f70404
* Fix YQL native YT type flags - revertziganshinmr2026-05-131-2/+2
| | | | commit_hash:fe3d6f4ff2b9f8f5441026ca53f095eae9f6cca3
* YQL-21224 Detach session for subprocesses & survive user-wide SIGUSR1vvvv2026-05-121-2/+7
| | | | | | | | | | | | | | | /\*\* \* @brief set to run command in new session \* @note set this option to off to deliver parent's signals to command as well \* @note currently ignored on windows \* @param detach if command should be run in new session \* @return self \*/ inline TShellCommandOptions& SetDetachSession(bool detach) \{ DetachSession = detach; return \*this; \} commit_hash:cdef06767fe1cc390aec92524a4829aced69c4b9
* YQL-20095: Enable readability-simplify-boolean-exprvitya-smirnov2026-05-072-7/+3
| | | | commit_hash:b3d7609795f9aeaaafe9cb65dbb2c6460582a4ec
* YQL-20095: Enable readability-uppercase-literal-suffixvitya-smirnov2026-05-074-33/+33
| | | | commit_hash:41791bb9a63f1993010fb68562e9a5607927307d
* YQL-19813: Udf implementationatarasov52026-05-074-1/+7
| | | | commit_hash:183e3a952a021c4701c20d6677800f245239d822
* SPYT-1033: YQL-api/QT - SPYT dialect supportmihailagei2026-05-071-0/+1
| | | | commit_hash:ca96a4ac2ac15aea958a8653a91a78760ee82521
* YQL-19813: Pass runtime settings to minikql + YT + minirunatarasov52026-04-291-0/+11
| | | | commit_hash:054395f57e37951159c1184eae2e6d4bc81245f8
* Drop unused yt gateway config settingsaneporada2026-04-281-2/+2
| | | | commit_hash:42306c88433a43cfe349423594406b8a30f9777f
* Secure tmp - refactor configsziganshinmr2026-04-281-45/+0
| | | | | | | | | - Move TVM/YT access provider/Blackbox configuration out of gateway configs in favor of more appropriate locations for static configuration: * yqlworker - add corresponding sections to executor config * ytrun - add --tvm-cfg and --yt-access-provider-cfg CLI args * QT YQL agent/plugin - static configuration handled by ytcfgen - Handle HTTP 429 commit_hash:772196f0803cbd67d3c37360da3e6bd74fa1bc78
* YQL-20095: Enable readability-inconsistent-declaration-parameter-namevitya-smirnov2026-04-281-2/+2
| | | | commit_hash:9589937209af8e742c0c6401bd1c232f073856bf
* Fix YQL native YT type flagsziganshinmr2026-04-231-2/+2
| | | | | | | | | | | | | | Пытаюсь исправить работу native type flags: - Исправления в TYqlRowSpecInfo: * При создании новых таблиц (SetType) - сразу выставляем нужные флаги согласно типу вместо усечения NTCF\_ALL на поздних этапах * Сделал прагму NativeYtTypeCompatibility static per cluster * При чтении \_yql\_row\_spec существующих таблиц - игнорируем записанное значение флагов, вместо этого выводим его из нативной схемы - Пишем правильные флаги \_yql\_row\_spec выходных таблиц (в зависимости от текущих NativeYtTypeCompatibility) - Обновил логику в оптимизаторах насчет выравнивания флагов - Убрал использование NTCF\_VOID и NTCF\_NULL для консистентности чтения/записи флагов (они и так всегда native, при чтении старый формат распознается) - Исправил Skiff схему для этих типов для соответствия поведению кодека (раньше каким-то чудом не стреляло) commit_hash:4a744866861591f9ba4a92c515cf33cfb88fb0a8
* YQL-19813: Make dependencies weakeratarasov52026-04-176-7/+24
| | | | commit_hash:8d58839a0285fd00d1dcc26e4cf2852982de9cba
* YQL-21156: add cluster use into apivafilonov2026-04-161-0/+1
| | | | commit_hash:b0748c2e8d718295be313b7dd241e6de4a667a9b
* YQL-19813: Runtime settingsatarasov52026-04-159-20/+66
| | | | commit_hash:eba49886bcafb9d280526ea3e3af12a9d07844a0
* YQL-21105 capture poly args during discovery of UDFvvvv2026-04-151-0/+2
| | | | commit_hash:17dd1b0e35654280ba5695c037baef93bcbb06a5
* YQL-20095: Enable readability-isolate-declarationvitya-smirnov2026-04-138-18/+46
| | | | commit_hash:c180c2db4897962cb70063ffdc2ddcd21e5e3418
* YQL-19813: Implement serializationatarasov52026-04-103-20/+391
| | | | commit_hash:176ea30d8d1b1a6ff6276ccd6f58697e7bc262a2
* Secure tmp foldersziganshinmr2026-04-081-0/+45
| | | | | Add support for automated allocation of secure TmpFolder on YT clusters for each user commit_hash:2d06fc7616f65d9814c7a5ca33a4da582fb5b016
* YQL-19813: Provide tests for existing behaviour + small fixesatarasov52026-04-075-8/+799
| | | | commit_hash:2fb970a01535af69ff34c3332bafd2547dc48029
* YQL-20095: Enable readability-container-size-emptyvitya-smirnov2026-04-072-2/+2
| | | | commit_hash:464a95fd6de1a395f8ffe20092e837df995e623c
* add csv input formatuzhas2026-04-061-1/+2
| | | | commit_hash:723614931e13d89b92d3c5ab8ca61e02a37e35bb
* YQL-20095: Enable readability-duplicate-includevitya-smirnov2026-04-063-3/+0
| | | | commit_hash:ea7cc749f3373f610d5945dc823ee74493744775
* YQL-21105 partial Udf resolver for Udf/ScriptUdfvvvv2026-04-032-0/+8
| | | | commit_hash:7abbae4a7b1c438affd630ba9dd02622ecbea891
* YQL-20095: Enable readability-avoid-const-params-in-declsvitya-smirnov2026-03-313-4/+4
| | | | | https://clang.llvm.org/extra/clang-tidy/checks/readability/avoid-const-params-in-decls.html commit_hash:17e1ec5c3849a38bcb76cd4927e66979bee6c2ec
* YQL-21101: Preserve ToFlow dependencies in MKQLimunkin2026-03-301-1/+6
| | | | commit_hash:4979fc7ea167afe5b7b478f40d3df13ea6ff0b67
* support cloud auth for CREATE EXTERNAL DATA SOURCE (yql part)yumkam72026-03-233-0/+70
| | | | | | | | | | | | | structured token will require service_account_id and resource_id and AUTH_METHOD=IAM will require SERVICE_ACCOUNT_ID and INITIAL_TOKEN_SECRET[_NAME|_PATH] Initial token is used on EDS creation for resolving resource_id/cloud_id and verifying database access, then service_account_id and resource_id will be used for delegated token issue (about same as used for async replication) commit_hash:3eec95e2dea61654a1939bd92549222f752d9654
* YQL-20095: Enable modernize-use-designated-initializersvitya-smirnov2026-03-172-3/+3
| | | | | https://clang.llvm.org/extra/clang-tidy/checks/modernize/use-designated-initializers.html commit_hash:d73eb463c7bec2bda4c362aab10af49979a7ddd3
generated by cgit v1.3 (git 2.53.0) at 2026-07-21 20:28:28 +0000