summaryrefslogtreecommitdiffstats
path: root/yql/essentials/minikql
Commit message (Collapse)AuthorAgeFilesLines
* YQL-21372, fixed switch data multiplication, fixed hanging of DQ stages with ↵grigoriypisar5 days2-119/+889
| | | | | | | | | | | | | | | | | | | | | multi output, refactor switch state handling Adjusted LLVM code for switch Refactored switch state, and fixed: 1. Stop processing once the active handler is finished. Previously, `Switch` could continue reading from input and forwarding `Yield` even after all handlers had already completed, for example with `Take <stream> <limit>`. 2. Do not finish prematurely when input is finished. If input returned `Finish`, the handler could still produce final `Yield`/data before completing. `Switch` now waits for the handler to fully drain instead of dropping the final output. 3. Preserve buffered input when handler yields early. If a handler returned `Yield` before consuming the whole input buffer, the remaining buffered data could be skipped after switching. This is now handled correctly. 4. Handle repeated `Yield` from input correctly. Previously, after sending `Yield`, the read index could be reset immediately in the handler. If the handler did not produce its own `Yield`, the same input could be reread, producing incorrect results. This case is now covered and fixed. Also fixed previously reverted hanging when switch produce extra yields. Refactoring request: <https://nda.ya.ru/t/8F3L9v3_7isKfN> commit_hash:ef7fd73dbdee60d33c384b6d6b505ad7bee20458
* YQL-21315: Block variant item implementationatarasov55 days11-7/+644
| | | | | BlockVariantItem implementation commit_hash:f35a597dee3dc67ebc5be599ed9d379d7ad57802
* YQL-21404 implemented syscache, table, func callvvvv8 days2-13/+51
| | | | | | | | | | | | | | #### `YQL-21404`: Implemented syscache, table, and function call support for collations ✎ - Added support for PostgreSQL collation handling in function calls, enabling explicit collation specification via named arguments - Implemented syscache and table catalog support for pg_collation system table to provide complete catalog information - Enhanced type annotation and runtime compilation to properly handle collation OIDs and validate collation names - Introduced ICU locale collation support with stable OID assignment based on locale position in generated catalog - Added comprehensive test coverage for collation functionality including standard and ICU locales, collation validation, and error cases - Extended SQL parsing to support COLLATE clauses and proper collation propagation in function calls <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:e9a32cc848bef7b68b4f57f71a1f5f69e0de9653
* YQL-21315: Implement block variant comp nodeatarasov58 days9-1/+246
| | | | | | | | | | | | | | #### Implement Block Variant Computational Node ✎ - Introduces a new computational node for handling variant types in block operations, supporting both tuple and struct variants - Adds type annotation and runtime support for the new variant node with proper type checking and validation - Implements block-level variant operations that can select specific alternatives from variant types based on index or member name - Includes comprehensive unit tests and integration tests covering both tuple and struct variant scenarios - Updates runtime version to 81 and integrates the new node into the compilation and execution pipeline - Provides SQL syntax support for variant operations through the `Variant` function with appropriate type handling <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:a07c1a365815a271cdd34958369156fd542fd5cc
* Intermediate changesrobot-piglet9 days18-285/+336
| | | | commit_hash:c6dfa214bb0ac2d6c28c49e031f19563bcb7039e
* Support Uuid and DyNumber in minikql blocks and Arrowxyligansereja12 days15-36/+607
| | | | | | | | | | | | | | | | Изменения в рамках этого тикета <https://github.com/ydb-platform/ydb/issues/44190> #### Support Uuid and DyNumber in minikql blocks and Arrow ✎ - Added support for Uuid and DyNumber data types within MiniKQL blocks and Arrow computations, enabling their use in block-based operations and serialization - Implemented proper conversion, serialization, and deserialization logic for both data types in the Arrow format - Extended block item handling to support Uuid and DyNumber, including comparator and hasher implementations - Added comprehensive unit tests covering roundtrip operations, serialization, comparison, and conversion for both data types - Integrated Uuid and DyNumber support into built-in functions like Min/Max and their aggregate variants - Updated type system and block builder components to recognize and handle the new data types correctly - Added necessary includes and utility functions to work with Uuid and DyNumber in MiniKQL contexts <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:d1ab82c060cfa90b7cd13a1cb3efbcb9615e6291
* YQL-21101: Make ToFlow dependent args requiredimunkin12 days40-307/+315
| | | | commit_hash:bc73ed520d5d4ef210b03b983720e6e8a6996db5
* fix asan fault by use-after-poison in mkql arrow allocgrigoriypisar12 days2-0/+15
| | | | | | | | | | | | #### Fix use-after-poison issue in MKQL arrow allocator ✎ - Addressed a use-after-poison memory error that occurred when using the default arrow allocator for memory management in MKQL - Added a sanitizer call to make memory region accessible before freeing it, preventing undefined behavior in memory sanitization tools - Introduced a new unit test to verify proper allocation and deallocation when the default arrow allocator is active - Ensured memory safety by properly handling memory access patterns during deallocation in sanitizer environments <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:080ee439f355ca7178f78b02b52302928e60e745
* Fix use-after-dtor in TDirectArrayHolderInplacemakxenov13 days1-4/+6
| | | | | | | | #### Fix use-after-dtor in TDirectArrayHolderInplace ✎ - Replaced the traditional `operator delete` with a destroying delete operator to ensure proper destruction order - The new implementation explicitly calls the destructor before freeing memory, preventing use-after-destructor issues commit_hash:9669691da2c478864eab809124405fc6cc31e57d
* YQL-21315: Implement variant expensive validationatarasov513 days6-83/+171
| | | | | | | | | | | | | #### Implement expensive datum validation for variant types ✎ - Introduce comprehensive validation for variant data types that checks offset invariants to ensure data integrity - Add support for passing validation mode through fuzzer infrastructure to enable both cheap and expensive validation paths - Extend test helpers to support different validation modes and integrate validation into various test cases - Modify datum validation logic to differentiate between cheap and expensive validation modes, with expensive mode performing additional checks on dense union structures - Update fuzzer components to validate data at multiple points during transformation to catch potential issues early <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:eccd98ccb2ffdda6ebb1d19c7256d41abe424757
* Preparing for LLVM18. The first step.Anton Romanov13 days7-92/+98
| | | | | | | | | | Type: refactoring Component: query-tracker --- Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1762 commit_hash:dcf0b6e086145d4d593c0691c38846ac4d87f36f
* YQL-21387: Introduce TStrongAliasatarasov52026-07-073-5/+5
| | | | | | | | | | | | | #### Introduce TStrongAlias type to prevent type confusion ✎ - Introduces `TStrongAlias` template to create distinct types for different conceptual uses of the same underlying data, preventing accidental mixing of semantically different values - Replaces direct `TString` usage with `TStrongAlias<class TRuntimeSettingsStableHashTag, TString>` for runtime settings hash to enforce type safety - Updates build configuration to include new `strong_alias` source files - Fixes a build issue in `mkql_computation_pattern_cache_ut.cpp` by properly accessing the `resize` method on the hash object - Provides comprehensive implementation of `TStrongAlias` including comparison, hashing, and stream output operators for type-safe usage <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:0c856e748ee2198d8545f1a96d24837132cf7d64
* YQL-21315: Block way implementationatarasov52026-07-0715-42/+928
| | | | | | | | | | | | | | | #### Block Way Implementation ✎ - Implemented the `BlockWay` function to extract variant type information from block inputs, returning either tuple indices or struct member names - Added type annotation support for `BlockWay` operations with proper handling of optional variants and different underlying types - Introduced computation node implementation for `BlockWay` that processes both tuple and struct variants efficiently - Enhanced runtime version handling to support the new `BlockWay` functionality with version 80 - Added comprehensive unit tests covering various scenarios including tuple variants, struct variants, optional variants, and edge cases - Integrated `BlockWay` into the SQL parser and program builder with proper type inference and validation - Added SQL format tests and canonical data for tuple and struct variant scenarios with and without optionality <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:70a82f5d13a7c1499a9ad88fc5d9b079e352d7ef
* YQL-21215: Finite values checks ListFromRangeeakozyrev2026-07-072-73/+142
| | | | | | 1. Добавил проверку start/end на равенство бесконечности -\> возвращается пустой список 2. Изменил логику инкремента индекса (перевел на целочисленный тип), чтобы работало валидно при малых значениях float commit_hash:f6b473a1d0227269591f9a683d81614696f0b040
* Revert " fixed Yield generation in switch"imunkin2026-07-012-188/+4
| | | | | | This reverts commit ff476352ebe39ac6796cd501c62bbb1a178bd2fd, reversing changes made to 64437b5c6ae050091a1b137e2feb5c280273aabc. commit_hash:cdef894cb3b7d51b092ca46fc5db7f349a343c8d
* YQL-20875: Pass validation flag to dq transport layeratarasov52026-06-307-48/+23
| | | | | | | | | | | | | | #### Pass validation flag to DQ transport layer ✎ - Introduce and propagate datum validation mode through the DQ transport layer to ensure consistent validation behavior across serialization and deserialization operations - Update transport serializers and deserializers to accept and utilize the validation mode parameter - Pass validation settings from task runners and compute actors to maintain end-to-end validation consistency - Add runtime settings serialization support to enable validation mode propagation in distributed scenarios - Extend channel service components to handle validation mode in both input and output serializers - Update various components including PQ read actors, wide combine operations, and spiller adapters to use the new validation parameter <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:c93ab318c05269f0e8402259c94ea52ab03bc4fa
* YQL-20095: Tune bugprone-argument-commentvitya-smirnov2026-06-3053-763/+763
| | | | | https://clang.llvm.org/extra/clang-tidy/checks/bugprone/argument-comment.html commit_hash:780ff6a63be44998cb2336a05f3dc33e68a676bf
* YQL-16277: Implement ExpandSqlCombine optimizerimunkin2026-06-293-10/+10
| | | | | | | | | | | | | | #### Implement SqlCombine optimizer and enhance ListJoinCore type checking ✎ - Introduce ExpandSqlCombine optimizer to handle SQL COMBINE operations, enabling efficient execution of combine operations with proper key handling and sorting - Refactor ListJoinCore type annotation logic to use tuples instead of structs for key types, improving type safety and consistency in multi-key scenarios - Add auxiliary constants for consistent naming of input prefixes and key members in join operations - Update MKQL compiler and type annotation components to support the new SqlCombine expansion and improved type checking - Enhance test coverage with new test cases for various SqlCombine scenarios including single key, multi-key, optional keys, and type mismatches - Adjust existing tests and canonical data to reflect the changes in type handling and naming conventions <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:376f67bfed851d0e4f7ac354d22220ba4f4737e2
* YQL-21253: Simplify trimmer logicatarasov52026-06-262-26/+14
| | | | | | | | | | | #### Simplify trimmer and transport logic for dense unions ✎ - Refactored dense union trimming logic to use new helper functions that calculate child usage and adjust value offsets in a more efficient and reusable way - Introduced dedicated functions for adjusting dense union value offsets, both as a separate operation and in-place, reducing code duplication and improving maintainability - Updated transport logic to leverage the new helper functions, streamlining the process of handling dense union arrays during data processing <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:637d983c584cc0e0b02a8ac21946d168fe736770
* YQL-21315: Implement block guessatarasov52026-06-269-1/+417
| | | | | | | | | | | | | #### Implement BlockGuess functionality for variant type extraction ✎ - Introduces BlockGuess functionality to extract specific alternatives from variant types, supporting both tuple and struct variants - Adds type annotation and runtime implementation for BlockGuess operations with proper validation and error handling - Implements peephole optimization support for BlockGuess operations to improve query execution plans - Includes comprehensive unit tests and SQL format validation for various BlockGuess scenarios - Updates runtime version to 79 to enable new BlockGuess features and ensures backward compatibility <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:e8f7fc2859cc1986ee0d5fba8b3e25bbf9ba4c41
* YQL-21345: Fix non thread safe usage of readeratarasov52026-06-251-9/+33
| | | | | | | | | | | | #### Fix thread safety issue in block reader usage ✎ - Replaced direct member variables with thread-safe kernel state management to ensure each execution context has its own reader instances - Moved reader initialization into kernel initialization callback to guarantee proper per-thread setup - Updated all reader accesses to use the kernel state's reader instances instead of shared members - Eliminated potential race conditions that could occur when multiple threads accessed the same reader instances concurrently <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:966873558498b7b27970f218598dfab448997512
* YQL-21207: Add langver and settings hash to cacheatarasov52026-06-254-51/+211
| | | | | | | | | | | | #### `YQL-21207`: Enhance cache key with langver and settings hash ✎ - Replaced serialized program string with a composite key consisting of language version, runtime settings hash, and serialized program to uniquely identify computation patterns. - Updated cache lookup, insertion, and notification mechanisms to use the new composite key for improved cache accuracy and consistency. - Modified pattern compilation and subscription logic to utilize the enhanced key for better cache management and pattern reuse. - Added unit tests to verify that the new key correctly distinguishes between different combinations of language version, settings, and program content. <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:db8a7b700579271bf301f1a2d40f662d77da424f
* YQL-20095: Enable performance-move-const-argvitya-smirnov2026-06-2413-40/+42
| | | | | | | | | | | | | https://clang.llvm.org/extra/clang-tidy/checks/performance/move-const-arg.html This check useful, as it shows, when a `std::move` is useless, so to expectations of a programmer are not aligned with reality. The most useful profit is to see `const T&` arguments, that are potentially can be replaced with `T`. Also disabled `CheckTriviallyCopyableMove`, see https://nda.ya.ru/t/Emvgx5Z67fpZY5 why. commit_hash:c80c264a9774721aabbac85b350ccdbb8b9a39a2
* fixed Yield generation in switchgrigoriypisar2026-06-192-4/+188
| | | | | | | | | | | | | | | | | | | Now the Switch can put into result a Yield without receiving a Yield from the input, which causes the DQ compute actor to hang if: * All the input channels have run out of space * There is an empty buffer in all output channels. In this case, terminating the TaskRunner call with the status of PendingInput causes it to hang. #### Fixed Yield generation logic in switch operator ✎ - Replaced deprecated status tracking with explicit finish flag for better control flow - Updated yield and finish detection logic to properly handle stream termination conditions - Simplified state management by removing redundant status assignments and improving conditional checks - Ensured correct yield generation based on stream completion status rather than outdated flags <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:ff476352ebe39ac6796cd501c62bbb1a178bd2fd
* YQL-21207: Hash runtime settingsatarasov52026-06-196-3/+165
| | | | | | | | | | | | | | #### Hash runtime settings for stable identification ✎ - Introduces a stable hash function for runtime settings to enable consistent identification of configuration states - Implements `SHA-256` based hashing that includes both host settings and UDF settings with deterministic ordering - Ensures hash stability regardless of setting insertion order by using ordered data structures - Adds comprehensive unit tests to verify hash consistency, content sensitivity, and order independence - Enhances configuration serialization to properly count and filter serializable settings - Makes serialization order deterministic by switching from hash map to map for setting storage <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:49aafc63bf011c335d556cd41dd4971473b62f13
* Intermediate changesrobot-piglet2026-06-1664-12095/+4835
| | | | commit_hash:2cae56a990383c528801f71d03c3dffae2160ade
* [minikql] optimize BlockMapJoin: use spans instead of row-vectorspavook2026-06-111-24/+35
| | | | commit_hash:911355a37a93170bdfe96fb3ef5634230541782a
* YQL-20875: Enable apache arrow datum light checkatarasov52026-06-0440-159/+254
| | | | commit_hash:ca028259d6fbf7b46aa0cc392187b8b153716d62
* YQL-21018: Implement BlockVariant validatoratarasov52026-06-041-1/+29
| | | | commit_hash:5334d8fbea8ccff97821995e01cbb7229aa822f9
* YQL-21284: Clear Yson2 marks before giving UV to clientatarasov52026-06-021-0/+97
| | | | | actual diff is <https://nda.ya.ru/t/RZq3NT0n7e7Kok> commit_hash:7d05f1fab17a23f9fb8281f67d3034c425f66633
* [minikql] fix a bug with hash collision on multiple-key BlockMapJoinpavook2026-06-011-3/+3
| | | | commit_hash:064dd429ee775897bfe6ac38e2cf3ef0a6580392
* YQL-21018: Implement builder and readers for block variantatarasov52026-06-0113-14/+1508
| | | | commit_hash:6514925c5290f805d279e2db85468c53c3db8f5d
* Change "yql/essentials/minikql/computation/mkql_computation_node_graph.cpp"vokayndzop2026-06-011-1/+2
| | | | commit_hash:594788b63fc30985620719d27c63a136eaab1363
* YQL-16277: Implement ListJoinCore computation nodeimunkin2026-05-299-1/+1137
| | | | commit_hash:4f92a547e4455422b667baf0d6929654cdb66a52
* WideSort better spillingmfilitov2026-05-293-88/+449
| | | | | Improved spilling in `WideSort`. Added a merge stage to avoid having too many states on disk. Also prevented spilling one row at a time. commit_hash:45db10e94dd2308cdd7fda06154cf06a95619fd2
* YQL-21270 allow prefix for custom python udfvvvv2026-05-272-11/+12
| | | | commit_hash:de018319aec8147802e30fb07107ffd2a224d899
* Use flushing mode flag instead finish fetch statusartemmashin2026-05-262-0/+14
| | | | commit_hash:3cfe2e66c9cf5391f2befa804ea429436660b222
* YQL-19813: Send runtime settings statsatarasov52026-05-223-11/+19
| | | | commit_hash:e8ea8f3c4365ed8b87ced31aa1fa34ca0686758c
* Intermediate changesrobot-piglet2026-05-225-410/+275
| | | | commit_hash:3afd7f64d24075150b548238f6f8fae1421cef00
* support max size configuration without cache drop outgvit2026-05-213-43/+139
| | | | | while reconfiguring the cache at i faced with a latency spikes because it's impossible to increase cache without full cache drop out commit_hash:2d090f8dd2348b7dc42d19082aec04d38da8c9df
* YQL-19813: Save and load qplayeratarasov52026-05-206-46/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Режим capture** (qContext.CanWrite()) 1. Проходим по всем элементам source с индексом idx. 2. Для каждого элемента вызываем filter(item) ровно один раз (фильтр может иметь побочные эффекты — например, запись статистики об активации). 3. Элементы, прошедшие фильтр, добавляются в возвращаемый результат. 4. Сохраняем индексы всех прошедших фильтр элементов в QStorage в формате V2: \{"version": "v2", "indexes": \[0, 2, 5, ...\]\} --- **Режим replay** (qContext.CanRead()) 1. Читаем YSON-запись из QStorage по ключу \{Component="Activation", Label=label\}. 2. Определяем версию формата по наличию ключа "version": **Формат V1** (старый, обратная совместимость) — ключ "version" отсутствует: - Структура: \{"имя\_флага": сериализованный\_proto, ...\} - Десериализуем каждый флаг через ParseFromString, возвращаем список напрямую. - Фильтр не вызывается. Список source игнорируется. **Формат V2 (текущий)** — "version": "v2": - Структура: \{"version": "v2", "indexes": \[0, 2, 5, ...\]\} - Читаем сохранённое множество индексов. - Проходим по source с *индексом* idx: - Элемент без условия активации (!HasActivation()) → всегда включается (такие элементы не зависят от активации). - Элемент с условием активации (HasActivation()) → включается только если его idx есть в сохранённом множестве. - Фильтр не вызывается. commit_hash:306113465bcd614cd3c0cd311ab20542d1f8e254
* Intermediate changesrobot-piglet2026-05-1810-605/+385
| | | | commit_hash:fc6492957790acea51dffccfb36a1d198cb9d9bb
* YQL-20162: A tuple's child array must have the same size as the tuple itselfatarasov52026-05-181-1/+3
| | | | commit_hash:a508756f74a720b541b3e8eca19cc54024451006
* YQL-20956: Fix chopper substream leakageatarasov52026-05-182-2/+96
| | | | | <https://nda.ya.ru/t/6Smb2LM87cbJ4m> commit_hash:999f90b84356132dbb8e876c40d6a38d8ac37fc0
* YQL-20095: Enable readability-static-definition-in-anonymous-namespacevitya-smirnov2026-05-157-38/+38
| | | | commit_hash:f7485dac5115e73f75a5edccb3c65c0b657dd8e5
* YQL-19813: Pass owning ptr to computation contextatarasov52026-05-153-10/+20
| | | | commit_hash:62b0f7b453641a0f14878dfdaae79ceda2c2c506
* YQL-20095: Enable performance-trivially-destructiblevitya-smirnov2026-05-142-3/+1
| | | | commit_hash:80e99229c5978038c8f209c23696c2a03b7c103a
* Intermediate changesrobot-piglet2026-05-141-3/+5
| | | | commit_hash:010ce3d7e109d5c664f4b20f7096d8be1d15935e
* YQL-20095: Enable modernize-use-emplacevitya-smirnov2026-05-133-28/+28
| | | | commit_hash:4d6e2f6fcc1639f09725c0c96de8d2f7f0f70404
* Intermediate changesrobot-piglet2026-05-137-419/+410
| | | | commit_hash:ff1b9e92e44a28e12fdc9294efe7a579a943d24f
generated by cgit v1.3 (git 2.53.0) at 2026-07-22 07:56:24 +0000