Reimport boost/tokenizer as a separate project

6 files changed, 101 insertions, 0 deletions

diff --git a/contrib/restricted/boost/CMakeLists.txt b/contrib/restricted/boost/CMakeLists.txt
index 9673b98ef9..7477f34a7c 100644
--- a/contrib/restricted/boost/CMakeLists.txt
+++ b/contrib/restricted/boost/CMakeLists.txt
@@ -48,6 +48,7 @@ target_link_libraries(contrib-restricted-boost INTERFACE
   restricted-boost-smart_ptr
   restricted-boost-static_assert
   restricted-boost-throw_exception
+  restricted-boost-tokenizer
   restricted-boost-tti
   restricted-boost-tuple
   restricted-boost-type_index
diff --git a/contrib/restricted/boost/tokenizer/CMakeLists.txt b/contrib/restricted/boost/tokenizer/CMakeLists.txt
new file mode 100644
index 0000000000..6231a3f538
--- /dev/null
+++ b/contrib/restricted/boost/tokenizer/CMakeLists.txt
@@ -0,0 +1,22 @@
+
+# This file was gererated by the build system used internally in the Yandex monorepo.
+# Only simple modifications are allowed (adding source-files to targets, adding simple properties
+# like target_include_directories). These modifications will be ported to original
+# ya.make files by maintainers. Any complex modifications which can't be ported back to the
+# original buildsystem will not be accepted.
+
+
+
+add_library(restricted-boost-tokenizer INTERFACE)
+target_include_directories(restricted-boost-tokenizer INTERFACE
+  ${CMAKE_SOURCE_DIR}/contrib/restricted/boost/tokenizer/include
+)
+target_link_libraries(restricted-boost-tokenizer INTERFACE
+  contrib-libs-cxxsupp
+  yutil
+  restricted-boost-assert
+  restricted-boost-config
+  restricted-boost-iterator
+  restricted-boost-mpl
+  restricted-boost-throw_exception
+)
diff --git a/contrib/restricted/boost/tokenizer/README.md b/contrib/restricted/boost/tokenizer/README.md
new file mode 100644
index 0000000000..a85ce3ed87
--- /dev/null
+++ b/contrib/restricted/boost/tokenizer/README.md
@@ -0,0 +1,78 @@
+
+
+
+# [Boost.Tokenizer](http://boost.org/libs/tokenizer)
+
+
+
+Boost tokenizer is a part of [Boost C++ Libraries](http://github.com/boostorg).  The Boost.Tokenizer package provides a flexible and easy-to-use way to break a string or other character sequence into a series of tokens.
+
+
+## Overview
+
+
+> break up a phrase into words.
+
+ <a target="_blank" href="http://melpon.org/wandbox/permlink/kZeKmQAtqDlpn8if">![Try it online][badge.wandbox]</a>
+
+```c++
+#include <iostream>
+#include <boost/tokenizer.hpp>
+#include <string>
+
+int main(){
+    std::string s = "This is,  a test";
+    typedef boost::tokenizer<> Tok;
+    Tok tok(s);
+    for (Tok::iterator beg = tok.begin(); beg != tok.end(); ++beg){
+        std::cout << *beg << "\n";
+    }
+}
+
+```
+
+> Using Range-based for loop (>c++11)
+
+ <a target="_blank" href="http://melpon.org/wandbox/permlink/z94YLs8PdYSh7rXz">![Try it online][badge.wandbox]</a>
+```c++
+#include <iostream>
+#include <boost/tokenizer.hpp>
+#include <string>
+
+int main(){
+    std::string s = "This is,  a test";
+    boost::tokenizer<> tok(s);
+    for (auto token: tok) {
+        std::cout << token << "\n";
+    }
+}
+```
+
+## Documentation
+
+Documentation can be found at [Boost.Tokenizer](http://boost.org/libs/tokenizer)
+
+## Related Material
+[Boost.Tokenizer](http://theboostcpplibraries.com/boost.tokenizer) Chapter 10 at theboostcpplibraries.com, contains several examples including **escaped_list_separator**.
+
+##Contributing
+
+>This library is being maintained as a part of the [Boost Library Official Maintainer Program](http://beta.boost.org/community/official_library_maintainer_program.html)
+
+
+Open Issues on <a target="_blank" href="https://svn.boost.org/trac/boost/query?status=assigned&status=new&status=reopened&component=tokenizer&col=id&col=summary&col=status&col=owner&col=type&col=milestone&order=priority">![][badge.trac]</a>
+
+
+
+##Acknowledgements
+>From the author:
+>
+I wish to thank the members of the boost mailing list, whose comments, compliments, and criticisms during both the development and formal review helped make the Tokenizer library what it is. I especially wish to thank Aleksey Gurtovoy for the idea of using a pair of iterators to specify the input, instead of a string. I also wish to thank Jeremy Siek for his idea of providing a container interface for the token iterators and for simplifying the template parameters for the TokenizerFunctions. He and Daryle Walker also emphasized the need to separate interface and implementation. Gary Powell sparked the idea of using the isspace and ispunct as the defaults for char_delimiters_separator. Jeff Garland provided ideas on how to change to order of the template parameters in order to make tokenizer easier to declare. Thanks to Douglas Gregor who served as review manager and provided many insights both on the boost list and in e-mail on how to polish up the implementation and presentation of Tokenizer. Finally, thanks to Beman Dawes who integrated the final version into the boost distribution.
+
+##License
+Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
+
+
+[badge.Wandbox]: https://img.shields.io/badge/try%20it-online-blue.svg
+[badge.Trac]:https://svn.boost.org/htdocs/common/trac_logo_mini.png
+
diff --git a/contrib/restricted/boost/boost/token_functions.hpp b/contrib/restricted/boost/tokenizer/include/boost/token_functions.hpp
index 30d6939723..30d6939723 100644
--- a/contrib/restricted/boost/boost/token_functions.hpp
+++ b/contrib/restricted/boost/tokenizer/include/boost/token_functions.hpp
diff --git a/contrib/restricted/boost/boost/token_iterator.hpp b/contrib/restricted/boost/tokenizer/include/boost/token_iterator.hpp
index 42945d7ee9..42945d7ee9 100644
--- a/contrib/restricted/boost/boost/token_iterator.hpp
+++ b/contrib/restricted/boost/tokenizer/include/boost/token_iterator.hpp
diff --git a/contrib/restricted/boost/boost/tokenizer.hpp b/contrib/restricted/boost/tokenizer/include/boost/tokenizer.hpp
index 081e5ba2f7..081e5ba2f7 100644
--- a/contrib/restricted/boost/boost/tokenizer.hpp
+++ b/contrib/restricted/boost/tokenizer/include/boost/tokenizer.hpp