diff options
author | h0pless <h0pless@yandex-team.com> | 2024-12-18 06:25:39 +0300 |
---|---|---|
committer | h0pless <h0pless@yandex-team.com> | 2024-12-18 06:39:36 +0300 |
commit | ea1ee8bc4725f339434fd1d48cf0d6f56129f711 (patch) | |
tree | a1f280a500991ec3da52c7b4529f5f5c619f060b | |
parent | 5b49bf7290defc0b7dd86d96043b3d6f67b57b05 (diff) | |
download | ydb-ea1ee8bc4725f339434fd1d48cf0d6f56129f711.tar.gz |
Introduce new pipeline for cross-cell copy
New cross-cell copy protocol:
1. LockCopyDestination - Early check that copy won’t fail because of conflicts, bad flag usage, etc. On this stage we learn to which cell the subtree is being copied to, inherited attributes of the destination and some other minor things.
2. LockCopySource - Lock every node with an appropriate lock on source cell, and return a map, describing topology of a subtree. (Amounts to a hash map parent_id -\> child_ids). During this step we also can find all schemas and cell tags that are used by the source.
3. SerializeNodes - Get binary representation of each node and send it to the client side. Additionally send information about schema id and external cell tags. The former is used to deduplicate schemas and the latter will be used during sequoia copy.
4. CalculateInheritedAttributes - Unfortunately for us, we have inheritable attributes. They behave differently for copy and create right now, because copy always preserves the current value and create calculates it anew. We plan to unify said behavior later by always re-evaluating the value of an attribute on destination cell. For the compatibility.
5. MaterializeCopyPrerequisites - This step is responsible for schema materialization. Later this should also include, for example, secondary indices.
6. MaterializeNodes - Create nodes from binary representation, but keep them unattached.
7. AssembleTreeCopy - Assemble nodes, created in step 6 into a tree on destination cell and attach the resulting tree to the destination node (or it's parent, in case of replace).
List of notable changes:
1. Deprecated “BeginCopy” and “EndCopy” verbs.
2. Deprecated “Internalize” command.
3. Added a code that should support new inherited attributes addition. Currently mostly inactive (only chunk_merger_mode is supported).
4. Opaqueness does not interfere with cross-cell copy now.
5. Now max subtree that can be copied with a single command is limited to 100’000 by default (this limit can be changed in master dynamic config)
6. Inplace copy only used for externalization of a map node, never during copy. With this change a decent chunk of dead code has been removed, hopefully making the overall algorithm more easily understandable.
7. Removed “inheritedAttributes“ and “factory“ from DoEndCopy.
List of notable bugfixes:
1. When attempting to cross-cell copy under a transaction and traversing a source subtree, most metadata was taken from the trunk version of the node instead of the branch. This results in some attributes being missing on destination if they were set under a transaction.
2. Fixed an issue where the problem from the above could be replicated for inheritable attributes.
3. Fixed a few tests that amounted to “assert True“.
### Changelog entry
type: feature
component: master
Introduced new pipeline for cross-cell copy
commit_hash:99cb45639f5ae55f431f5711269badb000e814a1
-rw-r--r-- | yt/yt/client/object_client/public.h | 1 | ||||
-rw-r--r-- | yt/yt/core/misc/range_formatters-inl.h | 6 | ||||
-rw-r--r-- | yt/yt/core/misc/range_formatters.h | 3 | ||||
-rw-r--r-- | yt/yt/core/misc/stripped_error.h | 2 | ||||
-rw-r--r-- | yt/yt/core/ytree/node_detail.cpp | 3 |
5 files changed, 14 insertions, 1 deletions
diff --git a/yt/yt/client/object_client/public.h b/yt/yt/client/object_client/public.h index 0b62274f1c..0533a5e912 100644 --- a/yt/yt/client/object_client/public.h +++ b/yt/yt/client/object_client/public.h @@ -35,6 +35,7 @@ YT_DEFINE_ERROR_ENUM( ((InvalidObjectType) (1006)) ((RequestInvolvesSequoia) (1007)) ((RequestInvolvesCypress) (1008)) + ((BeginCopyDeprecated) (1009)) ); //////////////////////////////////////////////////////////////////////////////// diff --git a/yt/yt/core/misc/range_formatters-inl.h b/yt/yt/core/misc/range_formatters-inl.h index 7dba39874f..b1e1d1cd78 100644 --- a/yt/yt/core/misc/range_formatters-inl.h +++ b/yt/yt/core/misc/range_formatters-inl.h @@ -20,6 +20,12 @@ void FormatValue(TStringBuilderBase* builder, const TSharedRange<T>& collection, NYT::FormatRange(builder, collection, TDefaultFormatter()); } +template <std::ranges::view T> +void FormatValue(TStringBuilderBase* builder, const T& collection, TStringBuf /*spec*/) +{ + NYT::FormatRange(builder, collection, TDefaultFormatter()); +} + //////////////////////////////////////////////////////////////////////////////// } // namespace NYT diff --git a/yt/yt/core/misc/range_formatters.h b/yt/yt/core/misc/range_formatters.h index 6cdfa33bf9..eebe9da441 100644 --- a/yt/yt/core/misc/range_formatters.h +++ b/yt/yt/core/misc/range_formatters.h @@ -15,6 +15,9 @@ void FormatValue(TStringBuilderBase* builder, const TRange<T>& collection, TStri template <class T> void FormatValue(TStringBuilderBase* builder, const TSharedRange<T>& collection, TStringBuf /*spec*/); +template <std::ranges::view T> +void FormatValue(TStringBuilderBase* builder, const T& collection, TStringBuf /*spec*/); + //////////////////////////////////////////////////////////////////////////////// } // namespace NYT diff --git a/yt/yt/core/misc/stripped_error.h b/yt/yt/core/misc/stripped_error.h index 6dd7b4b87e..ffec3b0db7 100644 --- a/yt/yt/core/misc/stripped_error.h +++ b/yt/yt/core/misc/stripped_error.h @@ -338,6 +338,8 @@ void ThrowErrorExceptionIfFailed(TErrorLike&& error); #define THROW_ERROR_EXCEPTION(head, ...) \ THROW_ERROR ::NYT::TError(head __VA_OPT__(,) __VA_ARGS__) +// NB: When given an error and a string as arguments, this macro automatically wraps +// new error around the initial one. #define THROW_ERROR_EXCEPTION_IF_FAILED(error, ...) \ ::NYT::NDetail::ThrowErrorExceptionIfFailed((error) __VA_OPT__(,) __VA_ARGS__); \ diff --git a/yt/yt/core/ytree/node_detail.cpp b/yt/yt/core/ytree/node_detail.cpp index 84746bc999..dd7ea9f3ea 100644 --- a/yt/yt/core/ytree/node_detail.cpp +++ b/yt/yt/core/ytree/node_detail.cpp @@ -285,7 +285,8 @@ IYPathService::TResolveResult TMapNodeMixin::ResolveRecursive( method == "Set" || method == "Create" || method == "Copy" || - method == "EndCopy") + method == "LockCopyDestination" || + method == "AssembleTreeCopy") { return IYPathService::TResolveResultHere{"/" + path}; } else { |