summaryrefslogtreecommitdiffstats
path: root/yql/essentials/minikql/comp_nodes
Commit message (Collapse)AuthorAgeFilesLines
* YQL-21372, fixed switch data multiplication, fixed hanging of DQ stages with ↵grigoriypisar6 days2-119/+889
| | | | | | | | | | | | | | | | | | | | | multi output, refactor switch state handling Adjusted LLVM code for switch Refactored switch state, and fixed: 1. Stop processing once the active handler is finished. Previously, `Switch` could continue reading from input and forwarding `Yield` even after all handlers had already completed, for example with `Take <stream> <limit>`. 2. Do not finish prematurely when input is finished. If input returned `Finish`, the handler could still produce final `Yield`/data before completing. `Switch` now waits for the handler to fully drain instead of dropping the final output. 3. Preserve buffered input when handler yields early. If a handler returned `Yield` before consuming the whole input buffer, the remaining buffered data could be skipped after switching. This is now handled correctly. 4. Handle repeated `Yield` from input correctly. Previously, after sending `Yield`, the read index could be reset immediately in the handler. If the handler did not produce its own `Yield`, the same input could be reread, producing incorrect results. This case is now covered and fixed. Also fixed previously reverted hanging when switch produce extra yields. Refactoring request: <https://nda.ya.ru/t/8F3L9v3_7isKfN> commit_hash:ef7fd73dbdee60d33c384b6d6b505ad7bee20458
* YQL-21315: Block variant item implementationatarasov56 days8-6/+624
| | | | | BlockVariantItem implementation commit_hash:f35a597dee3dc67ebc5be599ed9d379d7ad57802
* YQL-21315: Implement block variant comp nodeatarasov59 days6-0/+211
| | | | | | | | | | | | | | #### Implement Block Variant Computational Node ✎ - Introduces a new computational node for handling variant types in block operations, supporting both tuple and struct variants - Adds type annotation and runtime support for the new variant node with proper type checking and validation - Implements block-level variant operations that can select specific alternatives from variant types based on index or member name - Includes comprehensive unit tests and integration tests covering both tuple and struct variant scenarios - Updates runtime version to 81 and integrates the new node into the compilation and execution pipeline - Provides SQL syntax support for variant operations through the `Variant` function with appropriate type handling <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:a07c1a365815a271cdd34958369156fd542fd5cc
* Intermediate changesrobot-piglet10 days8-252/+113
| | | | commit_hash:c6dfa214bb0ac2d6c28c49e031f19563bcb7039e
* Support Uuid and DyNumber in minikql blocks and Arrowxyligansereja13 days8-28/+563
| | | | | | | | | | | | | | | | Изменения в рамках этого тикета <https://github.com/ydb-platform/ydb/issues/44190> #### Support Uuid and DyNumber in minikql blocks and Arrow ✎ - Added support for Uuid and DyNumber data types within MiniKQL blocks and Arrow computations, enabling their use in block-based operations and serialization - Implemented proper conversion, serialization, and deserialization logic for both data types in the Arrow format - Extended block item handling to support Uuid and DyNumber, including comparator and hasher implementations - Added comprehensive unit tests covering roundtrip operations, serialization, comparison, and conversion for both data types - Integrated Uuid and DyNumber support into built-in functions like Min/Max and their aggregate variants - Updated type system and block builder components to recognize and handle the new data types correctly - Added necessary includes and utility functions to work with Uuid and DyNumber in MiniKQL contexts <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:d1ab82c060cfa90b7cd13a1cb3efbcb9615e6291
* YQL-21101: Make ToFlow dependent args requiredimunkin13 days37-300/+309
| | | | commit_hash:bc73ed520d5d4ef210b03b983720e6e8a6996db5
* YQL-21315: Implement variant expensive validationatarasov52026-07-095-57/+100
| | | | | | | | | | | | | #### Implement expensive datum validation for variant types ✎ - Introduce comprehensive validation for variant data types that checks offset invariants to ensure data integrity - Add support for passing validation mode through fuzzer infrastructure to enable both cheap and expensive validation paths - Extend test helpers to support different validation modes and integrate validation into various test cases - Modify datum validation logic to differentiate between cheap and expensive validation modes, with expensive mode performing additional checks on dense union structures - Update fuzzer components to validate data at multiple points during transformation to catch potential issues early <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:eccd98ccb2ffdda6ebb1d19c7256d41abe424757
* YQL-21315: Block way implementationatarasov52026-07-079-41/+796
| | | | | | | | | | | | | | | #### Block Way Implementation ✎ - Implemented the `BlockWay` function to extract variant type information from block inputs, returning either tuple indices or struct member names - Added type annotation support for `BlockWay` operations with proper handling of optional variants and different underlying types - Introduced computation node implementation for `BlockWay` that processes both tuple and struct variants efficiently - Enhanced runtime version handling to support the new `BlockWay` functionality with version 80 - Added comprehensive unit tests covering various scenarios including tuple variants, struct variants, optional variants, and edge cases - Integrated `BlockWay` into the SQL parser and program builder with proper type inference and validation - Added SQL format tests and canonical data for tuple and struct variant scenarios with and without optionality <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:70a82f5d13a7c1499a9ad88fc5d9b079e352d7ef
* YQL-21215: Finite values checks ListFromRangeeakozyrev2026-07-072-73/+142
| | | | | | 1. Добавил проверку start/end на равенство бесконечности -\> возвращается пустой список 2. Изменил логику инкремента индекса (перевел на целочисленный тип), чтобы работало валидно при малых значениях float commit_hash:f6b473a1d0227269591f9a683d81614696f0b040
* Revert " fixed Yield generation in switch"imunkin2026-07-012-188/+4
| | | | | | This reverts commit ff476352ebe39ac6796cd501c62bbb1a178bd2fd, reversing changes made to 64437b5c6ae050091a1b137e2feb5c280273aabc. commit_hash:cdef894cb3b7d51b092ca46fc5db7f349a343c8d
* YQL-20875: Pass validation flag to dq transport layeratarasov52026-06-302-6/+6
| | | | | | | | | | | | | | #### Pass validation flag to DQ transport layer ✎ - Introduce and propagate datum validation mode through the DQ transport layer to ensure consistent validation behavior across serialization and deserialization operations - Update transport serializers and deserializers to accept and utilize the validation mode parameter - Pass validation settings from task runners and compute actors to maintain end-to-end validation consistency - Add runtime settings serialization support to enable validation mode propagation in distributed scenarios - Extend channel service components to handle validation mode in both input and output serializers - Update various components including PQ read actors, wide combine operations, and spiller adapters to use the new validation parameter <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:c93ab318c05269f0e8402259c94ea52ab03bc4fa
* YQL-20095: Tune bugprone-argument-commentvitya-smirnov2026-06-302-4/+4
| | | | | https://clang.llvm.org/extra/clang-tidy/checks/bugprone/argument-comment.html commit_hash:780ff6a63be44998cb2336a05f3dc33e68a676bf
* YQL-16277: Implement ExpandSqlCombine optimizerimunkin2026-06-292-7/+7
| | | | | | | | | | | | | | #### Implement SqlCombine optimizer and enhance ListJoinCore type checking ✎ - Introduce ExpandSqlCombine optimizer to handle SQL COMBINE operations, enabling efficient execution of combine operations with proper key handling and sorting - Refactor ListJoinCore type annotation logic to use tuples instead of structs for key types, improving type safety and consistency in multi-key scenarios - Add auxiliary constants for consistent naming of input prefixes and key members in join operations - Update MKQL compiler and type annotation components to support the new SqlCombine expansion and improved type checking - Enhance test coverage with new test cases for various SqlCombine scenarios including single key, multi-key, optional keys, and type mismatches - Adjust existing tests and canonical data to reflect the changes in type handling and naming conventions <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:376f67bfed851d0e4f7ac354d22220ba4f4737e2
* YQL-21315: Implement block guessatarasov52026-06-266-0/+374
| | | | | | | | | | | | | #### Implement BlockGuess functionality for variant type extraction ✎ - Introduces BlockGuess functionality to extract specific alternatives from variant types, supporting both tuple and struct variants - Adds type annotation and runtime implementation for BlockGuess operations with proper validation and error handling - Implements peephole optimization support for BlockGuess operations to improve query execution plans - Includes comprehensive unit tests and SQL format validation for various BlockGuess scenarios - Updates runtime version to 79 to enable new BlockGuess features and ensures backward compatibility <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:e8f7fc2859cc1986ee0d5fba8b3e25bbf9ba4c41
* YQL-21345: Fix non thread safe usage of readeratarasov52026-06-251-9/+33
| | | | | | | | | | | | #### Fix thread safety issue in block reader usage ✎ - Replaced direct member variables with thread-safe kernel state management to ensure each execution context has its own reader instances - Moved reader initialization into kernel initialization callback to guarantee proper per-thread setup - Updated all reader accesses to use the kernel state's reader instances instead of shared members - Eliminated potential race conditions that could occur when multiple threads accessed the same reader instances concurrently <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:966873558498b7b27970f218598dfab448997512
* YQL-20095: Enable performance-move-const-argvitya-smirnov2026-06-241-1/+1
| | | | | | | | | | | | | https://clang.llvm.org/extra/clang-tidy/checks/performance/move-const-arg.html This check useful, as it shows, when a `std::move` is useless, so to expectations of a programmer are not aligned with reality. The most useful profit is to see `const T&` arguments, that are potentially can be replaced with `T`. Also disabled `CheckTriviallyCopyableMove`, see https://nda.ya.ru/t/Emvgx5Z67fpZY5 why. commit_hash:c80c264a9774721aabbac85b350ccdbb8b9a39a2
* fixed Yield generation in switchgrigoriypisar2026-06-192-4/+188
| | | | | | | | | | | | | | | | | | | Now the Switch can put into result a Yield without receiving a Yield from the input, which causes the DQ compute actor to hang if: * All the input channels have run out of space * There is an empty buffer in all output channels. In this case, terminating the TaskRunner call with the status of PendingInput causes it to hang. #### Fixed Yield generation logic in switch operator ✎ - Replaced deprecated status tracking with explicit finish flag for better control flow - Updated yield and finish detection logic to properly handle stream termination conditions - Simplified state management by removing redundant status assignments and improving conditional checks - Ensured correct yield generation based on stream completion status rather than outdated flags <a href="https://nda.ya.ru/t/qa0kX64r7DqvtN"><font size="2">Autodescription by Yandex Code Assistant</font></a> commit_hash:ff476352ebe39ac6796cd501c62bbb1a178bd2fd
* Intermediate changesrobot-piglet2026-06-1656-11990/+4537
| | | | commit_hash:2cae56a990383c528801f71d03c3dffae2160ade
* [minikql] optimize BlockMapJoin: use spans instead of row-vectorspavook2026-06-111-24/+35
| | | | commit_hash:911355a37a93170bdfe96fb3ef5634230541782a
* YQL-20875: Enable apache arrow datum light checkatarasov52026-06-0426-95/+124
| | | | commit_hash:ca028259d6fbf7b46aa0cc392187b8b153716d62
* [minikql] fix a bug with hash collision on multiple-key BlockMapJoinpavook2026-06-011-3/+3
| | | | commit_hash:064dd429ee775897bfe6ac38e2cf3ef0a6580392
* YQL-21018: Implement builder and readers for block variantatarasov52026-06-015-3/+1171
| | | | commit_hash:6514925c5290f805d279e2db85468c53c3db8f5d
* YQL-16277: Implement ListJoinCore computation nodeimunkin2026-05-296-0/+1022
| | | | commit_hash:4f92a547e4455422b667baf0d6929654cdb66a52
* WideSort better spillingmfilitov2026-05-293-88/+449
| | | | | Improved spilling in `WideSort`. Added a merge stage to avoid having too many states on disk. Also prevented spilling one row at a time. commit_hash:45db10e94dd2308cdd7fda06154cf06a95619fd2
* Intermediate changesrobot-piglet2026-05-225-410/+275
| | | | commit_hash:3afd7f64d24075150b548238f6f8fae1421cef00
* Intermediate changesrobot-piglet2026-05-1810-605/+385
| | | | commit_hash:fc6492957790acea51dffccfb36a1d198cb9d9bb
* YQL-20956: Fix chopper substream leakageatarasov52026-05-182-2/+96
| | | | | <https://nda.ya.ru/t/6Smb2LM87cbJ4m> commit_hash:999f90b84356132dbb8e876c40d6a38d8ac37fc0
* YQL-19813: Pass owning ptr to computation contextatarasov52026-05-151-2/+5
| | | | commit_hash:62b0f7b453641a0f14878dfdaae79ceda2c2c506
* Intermediate changesrobot-piglet2026-05-141-3/+5
| | | | commit_hash:010ce3d7e109d5c664f4b20f7096d8be1d15935e
* Intermediate changesrobot-piglet2026-05-137-419/+410
| | | | commit_hash:ff1b9e92e44a28e12fdc9294efe7a579a943d24f
* YQL-21018: Prepare test utils for block variant supportatarasov52026-05-136-193/+193
| | | | commit_hash:d08046f19590c93e827909f0bb9e28cfa829ffb7
* YQL-20095: Enable readability-uppercase-literal-suffixvitya-smirnov2026-05-072-9/+9
| | | | commit_hash:41791bb9a63f1993010fb68562e9a5607927307d
* YQL-19813: Udf implementationatarasov52026-05-073-8/+10
| | | | commit_hash:183e3a952a021c4701c20d6677800f245239d822
* YQL-21204: Make BlockCoalesce immutable frendlyatarasov52026-05-054-23/+93
| | | | | | YQL-21204: Fix wrong immutable calls YQL-21204: Fuzz immutability commit_hash:66fb769418e9c275db6fbfa92d78fc0d57a4b150
* YQL-19813: Pass runtime settings to minikql + YT + minirunatarasov52026-04-298-4/+188
| | | | commit_hash:054395f57e37951159c1184eae2e6d4bc81245f8
* YQL-21164: Fix chain1 map callatarasov52026-04-161-0/+133
| | | | commit_hash:7cef97dc90e03e9f773a44e4bd51a51388a8d954
* YQL-21031: Range decimalatarasov52026-04-141-14/+34
| | | | commit_hash:836406699a568f3d53cbd91256d586d37bf8f4a9
* YQL-20095: Enable readability-isolate-declarationvitya-smirnov2026-04-131-1/+2
| | | | commit_hash:c180c2db4897962cb70063ffdc2ddcd21e5e3418
* YQL-20873: Range PG supportatarasov52026-04-136-203/+378
| | | | commit_hash:9e799ea42a2692eb96ebc792b48e46dd060c182b
* Replace unreachable with *_ENSURE(false, ...)atarasov52026-04-018-14/+14
| | | | commit_hash:7c1445375e838a9327006528cb455c9899e9b32d
* YQL-21101: Preserve ToFlow dependencies in MKQLimunkin2026-03-301-9/+26
| | | | commit_hash:4979fc7ea167afe5b7b478f40d3df13ea6ff0b67
* YQL-19424: Force BlockExpandChunked process stream inputimunkin2026-03-272-5/+5
| | | | commit_hash:183c6a5ff0706ee5c612ff07806a656f19c5bd30
* YQL-20987: Drop MKQL support for runtime versions <67imunkin2026-03-203-1301/+17
| | | | commit_hash:73457e89df05a010148960ee30c67a1481b1b3c2
* Runtime dispatch boolsatarasov52026-03-179-127/+27
| | | | | Introduce runtime dispatcher macros commit_hash:69cd6afc98d510a3f8eb216e52b6f9044245ee98
* YQL-20972: Rewrite methods that has UV as output parameter.atarasov52026-03-0520-107/+166
| | | | commit_hash:6068d33bd6ed2bb8bdfa7836ca5229b841dcd207
* YQL-20095: Apply lost Clang Tidy fixesvitya-smirnov2026-03-051-1/+1
| | | | | | | I do not known why, but these Clang Tidy fixes was lost. Maybe this is because of conditional preprocessing. Maybe Clang Tidy is ran only on changed sources. commit_hash:718e5487699b21b643ed90729ce8347c84cbd6ee
* YQL-20095: Enable modernize-use-usingvitya-smirnov2026-03-041-2/+2
| | | | commit_hash:ea1f0987ec40e3e683c35bffb64b7f5ab35e73ed
* YQL-20987: Drop MKQL support for runtime versions <59imunkin2026-03-025-1409/+105
| | | | commit_hash:dd04095806df285c8fb8a8afa5fe5552e9bcc110
* YQL-20976: Specify invariants on unsuccessfull extraction of some methodsatarasov52026-02-212-6/+15
| | | | commit_hash:ded61446e96c98e6f8366f994fdf704248b4359e
* Refactor codegen for function callsimunkin2026-02-2046-753/+180
| | | | commit_hash:72b10ed4677b5440bf8308bec7a0c1b21ab185c4
generated by cgit v1.3 (git 2.53.0) at 2026-07-23 17:27:52 +0000