diff options
| author | say <say@yandex-team.com> | 2023-12-08 17:40:48 +0300 |
|---|---|---|
| committer | say <say@yandex-team.com> | 2023-12-08 19:58:59 +0300 |
| commit | 914f57e3243f53dd89dd3adb4d8b6d35c47f46ce (patch) | |
| tree | 98a1f1f1f5e2c38db3a78da10aeb7eb7d4e952e0 /contrib/python | |
| parent | e61293d91ee7c923944f627d8e1138bcb17cacad (diff) | |
| download | ydb-914f57e3243f53dd89dd3adb4d8b6d35c47f46ce.tar.gz | |
Make support_retries() and get_type() methods of object instead of class ones. Reduce get_type_name() usage.
Diffstat (limited to 'contrib/python')
137 files changed, 0 insertions, 20180 deletions
diff --git a/contrib/python/distro/py2/.dist-info/METADATA b/contrib/python/distro/py2/.dist-info/METADATA deleted file mode 100644 index bc42dfcf16..0000000000 --- a/contrib/python/distro/py2/.dist-info/METADATA +++ /dev/null @@ -1,169 +0,0 @@ -Metadata-Version: 2.1 -Name: distro -Version: 1.6.0 -Summary: Distro - an OS platform information API -Home-page: https://github.com/python-distro/distro -Author: Nir Cohen -Author-email: nir36g@gmail.com -License: Apache License, Version 2.0 -Platform: All -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: Intended Audience :: System Administrators -Classifier: License :: OSI Approved :: Apache Software License -Classifier: Operating System :: POSIX :: Linux -Classifier: Operating System :: POSIX :: BSD -Classifier: Operating System :: POSIX :: BSD :: FreeBSD -Classifier: Operating System :: POSIX :: BSD :: NetBSD -Classifier: Operating System :: POSIX :: BSD :: OpenBSD -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Topic :: Software Development :: Libraries :: Python Modules -Classifier: Topic :: System :: Operating System -Description-Content-Type: text/markdown -License-File: LICENSE - -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.linux_distribution(full_distribution_name=False) -('centos', '7.1.1503', 'Core') -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 2.7, 3.4+ and PyPy and on -any distribution that provides one or more of the data sources -covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/sid/python-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python2-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ - - diff --git a/contrib/python/distro/py2/.dist-info/entry_points.txt b/contrib/python/distro/py2/.dist-info/entry_points.txt deleted file mode 100644 index dd4023997f..0000000000 --- a/contrib/python/distro/py2/.dist-info/entry_points.txt +++ /dev/null @@ -1,3 +0,0 @@ -[console_scripts] -distro = distro:main - diff --git a/contrib/python/distro/py2/.dist-info/top_level.txt b/contrib/python/distro/py2/.dist-info/top_level.txt deleted file mode 100644 index 0e0933171d..0000000000 --- a/contrib/python/distro/py2/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -distro diff --git a/contrib/python/distro/py2/LICENSE b/contrib/python/distro/py2/LICENSE deleted file mode 100644 index e06d208186..0000000000 --- a/contrib/python/distro/py2/LICENSE +++ /dev/null @@ -1,202 +0,0 @@ -Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - diff --git a/contrib/python/distro/py2/README.md b/contrib/python/distro/py2/README.md deleted file mode 100644 index 76eaef244d..0000000000 --- a/contrib/python/distro/py2/README.md +++ /dev/null @@ -1,135 +0,0 @@ -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.linux_distribution(full_distribution_name=False) -('centos', '7.1.1503', 'Core') -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 2.7, 3.4+ and PyPy and on -any distribution that provides one or more of the data sources -covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/sid/python-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python2-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ diff --git a/contrib/python/distro/py2/distro.py b/contrib/python/distro/py2/distro.py deleted file mode 100644 index 7892741347..0000000000 --- a/contrib/python/distro/py2/distro.py +++ /dev/null @@ -1,1386 +0,0 @@ -# Copyright 2015,2016,2017 Nir Cohen -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -""" -The ``distro`` package (``distro`` stands for Linux Distribution) provides -information about the Linux distribution it runs on, such as a reliable -machine-readable distro ID, or version information. - -It is the recommended replacement for Python's original -:py:func:`platform.linux_distribution` function, but it provides much more -functionality. An alternative implementation became necessary because Python -3.5 deprecated this function, and Python 3.8 removed it altogether. Its -predecessor function :py:func:`platform.dist` was already deprecated since -Python 2.6 and removed in Python 3.8. Still, there are many cases in which -access to OS distribution information is needed. See `Python issue 1322 -<https://bugs.python.org/issue1322>`_ for more information. -""" - -import argparse -import json -import logging -import os -import re -import shlex -import subprocess -import sys -import warnings - -__version__ = "1.6.0" - -# Use `if False` to avoid an ImportError on Python 2. After dropping Python 2 -# support, can use typing.TYPE_CHECKING instead. See: -# https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING -if False: # pragma: nocover - from typing import ( - Any, - Callable, - Dict, - Iterable, - Optional, - Sequence, - TextIO, - Tuple, - Type, - TypedDict, - Union, - ) - - VersionDict = TypedDict( - "VersionDict", {"major": str, "minor": str, "build_number": str} - ) - InfoDict = TypedDict( - "InfoDict", - { - "id": str, - "version": str, - "version_parts": VersionDict, - "like": str, - "codename": str, - }, - ) - - -_UNIXCONFDIR = os.environ.get("UNIXCONFDIR", "/etc") -_UNIXUSRLIBDIR = os.environ.get("UNIXUSRLIBDIR", "/usr/lib") -_OS_RELEASE_BASENAME = "os-release" - -#: Translation table for normalizing the "ID" attribute defined in os-release -#: files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as defined in the os-release file, translated to lower case, -#: with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_OS_ID = { - "ol": "oracle", # Oracle Linux -} - -#: Translation table for normalizing the "Distributor ID" attribute returned by -#: the lsb_release command, for use by the :func:`distro.id` method. -#: -#: * Key: Value as returned by the lsb_release command, translated to lower -#: case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_LSB_ID = { - "enterpriseenterpriseas": "oracle", # Oracle Enterprise Linux 4 - "enterpriseenterpriseserver": "oracle", # Oracle Linux 5 - "redhatenterpriseworkstation": "rhel", # RHEL 6, 7 Workstation - "redhatenterpriseserver": "rhel", # RHEL 6, 7 Server - "redhatenterprisecomputenode": "rhel", # RHEL 6 ComputeNode -} - -#: Translation table for normalizing the distro ID derived from the file name -#: of distro release files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as derived from the file name of a distro release file, -#: translated to lower case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_DISTRO_ID = { - "redhat": "rhel", # RHEL 6.x, 7.x -} - -# Pattern for content of distro release file (reversed) -_DISTRO_RELEASE_CONTENT_REVERSED_PATTERN = re.compile( - r"(?:[^)]*\)(.*)\()? *(?:STL )?([\d.+\-a-z]*\d) *(?:esaeler *)?(.+)" -) - -# Pattern for base file name of distro release file -_DISTRO_RELEASE_BASENAME_PATTERN = re.compile(r"(\w+)[-_](release|version)$") - -# Base file names to be ignored when searching for distro release file -_DISTRO_RELEASE_IGNORE_BASENAMES = ( - "debian_version", - "lsb-release", - "oem-release", - _OS_RELEASE_BASENAME, - "system-release", - "plesk-release", - "iredmail-release", -) - - -def linux_distribution(full_distribution_name=True): - # type: (bool) -> Tuple[str, str, str] - """ - .. deprecated:: 1.6.0 - - :func:`distro.linux_distribution()` is deprecated. It should only be - used as a compatibility shim with Python's - :py:func:`platform.linux_distribution()`. Please use :func:`distro.id`, - :func:`distro.version` and :func:`distro.name` instead. - - Return information about the current OS distribution as a tuple - ``(id_name, version, codename)`` with items as follows: - - * ``id_name``: If *full_distribution_name* is false, the result of - :func:`distro.id`. Otherwise, the result of :func:`distro.name`. - - * ``version``: The result of :func:`distro.version`. - - * ``codename``: The result of :func:`distro.codename`. - - The interface of this function is compatible with the original - :py:func:`platform.linux_distribution` function, supporting a subset of - its parameters. - - The data it returns may not exactly be the same, because it uses more data - sources than the original function, and that may lead to different data if - the OS distribution is not consistent across multiple data sources it - provides (there are indeed such distributions ...). - - Another reason for differences is the fact that the :func:`distro.id` - method normalizes the distro ID string to a reliable machine-readable value - for a number of popular OS distributions. - """ - warnings.warn( - "distro.linux_distribution() is deprecated. It should only be used as a " - "compatibility shim with Python's platform.linux_distribution(). Please use " - "distro.id(), distro.version() and distro.name() instead.", - DeprecationWarning, - stacklevel=2, - ) - return _distro.linux_distribution(full_distribution_name) - - -def id(): - # type: () -> str - """ - Return the distro ID of the current distribution, as a - machine-readable string. - - For a number of OS distributions, the returned distro ID value is - *reliable*, in the sense that it is documented and that it does not change - across releases of the distribution. - - This package maintains the following reliable distro ID values: - - ============== ========================================= - Distro ID Distribution - ============== ========================================= - "ubuntu" Ubuntu - "debian" Debian - "rhel" RedHat Enterprise Linux - "centos" CentOS - "fedora" Fedora - "sles" SUSE Linux Enterprise Server - "opensuse" openSUSE - "amazon" Amazon Linux - "arch" Arch Linux - "cloudlinux" CloudLinux OS - "exherbo" Exherbo Linux - "gentoo" GenToo Linux - "ibm_powerkvm" IBM PowerKVM - "kvmibm" KVM for IBM z Systems - "linuxmint" Linux Mint - "mageia" Mageia - "mandriva" Mandriva Linux - "parallels" Parallels - "pidora" Pidora - "raspbian" Raspbian - "oracle" Oracle Linux (and Oracle Enterprise Linux) - "scientific" Scientific Linux - "slackware" Slackware - "xenserver" XenServer - "openbsd" OpenBSD - "netbsd" NetBSD - "freebsd" FreeBSD - "midnightbsd" MidnightBSD - ============== ========================================= - - If you have a need to get distros for reliable IDs added into this set, - or if you find that the :func:`distro.id` function returns a different - distro ID for one of the listed distros, please create an issue in the - `distro issue tracker`_. - - **Lookup hierarchy and transformations:** - - First, the ID is obtained from the following sources, in the specified - order. The first available and non-empty value is used: - - * the value of the "ID" attribute of the os-release file, - - * the value of the "Distributor ID" attribute returned by the lsb_release - command, - - * the first part of the file name of the distro release file, - - The so determined ID value then passes the following transformations, - before it is returned by this method: - - * it is translated to lower case, - - * blanks (which should not be there anyway) are translated to underscores, - - * a normalization of the ID is performed, based upon - `normalization tables`_. The purpose of this normalization is to ensure - that the ID is as reliable as possible, even across incompatible changes - in the OS distributions. A common reason for an incompatible change is - the addition of an os-release file, or the addition of the lsb_release - command, with ID values that differ from what was previously determined - from the distro release file name. - """ - return _distro.id() - - -def name(pretty=False): - # type: (bool) -> str - """ - Return the name of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the name is returned without version or codename. - (e.g. "CentOS Linux") - - If *pretty* is true, the version and codename are appended. - (e.g. "CentOS Linux 7.1.1503 (Core)") - - **Lookup hierarchy:** - - The name is obtained from the following sources, in the specified order. - The first available and non-empty value is used: - - * If *pretty* is false: - - - the value of the "NAME" attribute of the os-release file, - - - the value of the "Distributor ID" attribute returned by the lsb_release - command, - - - the value of the "<name>" field of the distro release file. - - * If *pretty* is true: - - - the value of the "PRETTY_NAME" attribute of the os-release file, - - - the value of the "Description" attribute returned by the lsb_release - command, - - - the value of the "<name>" field of the distro release file, appended - with the value of the pretty version ("<version_id>" and "<codename>" - fields) of the distro release file, if available. - """ - return _distro.name(pretty) - - -def version(pretty=False, best=False): - # type: (bool, bool) -> str - """ - Return the version of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the version is returned without codename (e.g. - "7.0"). - - If *pretty* is true, the codename in parenthesis is appended, if the - codename is non-empty (e.g. "7.0 (Maipo)"). - - Some distributions provide version numbers with different precisions in - the different sources of distribution information. Examining the different - sources in a fixed priority order does not always yield the most precise - version (e.g. for Debian 8.2, or CentOS 7.1). - - The *best* parameter can be used to control the approach for the returned - version: - - If *best* is false, the first non-empty version number in priority order of - the examined sources is returned. - - If *best* is true, the most precise version number out of all examined - sources is returned. - - **Lookup hierarchy:** - - In all cases, the version number is obtained from the following sources. - If *best* is false, this order represents the priority order: - - * the value of the "VERSION_ID" attribute of the os-release file, - * the value of the "Release" attribute returned by the lsb_release - command, - * the version number parsed from the "<version_id>" field of the first line - of the distro release file, - * the version number parsed from the "PRETTY_NAME" attribute of the - os-release file, if it follows the format of the distro release files. - * the version number parsed from the "Description" attribute returned by - the lsb_release command, if it follows the format of the distro release - files. - """ - return _distro.version(pretty, best) - - -def version_parts(best=False): - # type: (bool) -> Tuple[str, str, str] - """ - Return the version of the current OS distribution as a tuple - ``(major, minor, build_number)`` with items as follows: - - * ``major``: The result of :func:`distro.major_version`. - - * ``minor``: The result of :func:`distro.minor_version`. - - * ``build_number``: The result of :func:`distro.build_number`. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.version_parts(best) - - -def major_version(best=False): - # type: (bool) -> str - """ - Return the major version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The major version is the first - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.major_version(best) - - -def minor_version(best=False): - # type: (bool) -> str - """ - Return the minor version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The minor version is the second - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.minor_version(best) - - -def build_number(best=False): - # type: (bool) -> str - """ - Return the build number of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The build number is the third part - of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.build_number(best) - - -def like(): - # type: () -> str - """ - Return a space-separated list of distro IDs of distributions that are - closely related to the current OS distribution in regards to packaging - and programming interfaces, for example distributions the current - distribution is a derivative from. - - **Lookup hierarchy:** - - This information item is only provided by the os-release file. - For details, see the description of the "ID_LIKE" attribute in the - `os-release man page - <http://www.freedesktop.org/software/systemd/man/os-release.html>`_. - """ - return _distro.like() - - -def codename(): - # type: () -> str - """ - Return the codename for the release of the current OS distribution, - as a string. - - If the distribution does not have a codename, an empty string is returned. - - Note that the returned codename is not always really a codename. For - example, openSUSE returns "x86_64". This function does not handle such - cases in any special way and just returns the string it finds, if any. - - **Lookup hierarchy:** - - * the codename within the "VERSION" attribute of the os-release file, if - provided, - - * the value of the "Codename" attribute returned by the lsb_release - command, - - * the value of the "<codename>" field of the distro release file. - """ - return _distro.codename() - - -def info(pretty=False, best=False): - # type: (bool, bool) -> InfoDict - """ - Return certain machine-readable information items about the current OS - distribution in a dictionary, as shown in the following example: - - .. sourcecode:: python - - { - 'id': 'rhel', - 'version': '7.0', - 'version_parts': { - 'major': '7', - 'minor': '0', - 'build_number': '' - }, - 'like': 'fedora', - 'codename': 'Maipo' - } - - The dictionary structure and keys are always the same, regardless of which - information items are available in the underlying data sources. The values - for the various keys are as follows: - - * ``id``: The result of :func:`distro.id`. - - * ``version``: The result of :func:`distro.version`. - - * ``version_parts -> major``: The result of :func:`distro.major_version`. - - * ``version_parts -> minor``: The result of :func:`distro.minor_version`. - - * ``version_parts -> build_number``: The result of - :func:`distro.build_number`. - - * ``like``: The result of :func:`distro.like`. - - * ``codename``: The result of :func:`distro.codename`. - - For a description of the *pretty* and *best* parameters, see the - :func:`distro.version` method. - """ - return _distro.info(pretty, best) - - -def os_release_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the os-release file data source of the current OS distribution. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_info() - - -def lsb_release_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the lsb_release command data source of the current OS distribution. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_info() - - -def distro_release_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_info() - - -def uname_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - """ - return _distro.uname_info() - - -def os_release_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the os-release file data source - of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_attr(attribute) - - -def lsb_release_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the lsb_release command output - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_attr(attribute) - - -def distro_release_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_attr(attribute) - - -def uname_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - """ - return _distro.uname_attr(attribute) - - -try: - from functools import cached_property -except ImportError: - # Python < 3.8 - class cached_property(object): # type: ignore - """A version of @property which caches the value. On access, it calls the - underlying function and sets the value in `__dict__` so future accesses - will not re-call the property. - """ - - def __init__(self, f): - # type: (Callable[[Any], Any]) -> None - self._fname = f.__name__ - self._f = f - - def __get__(self, obj, owner): - # type: (Any, Type[Any]) -> Any - assert obj is not None, "call {} on an instance".format(self._fname) - ret = obj.__dict__[self._fname] = self._f(obj) - return ret - - -class LinuxDistribution(object): - """ - Provides information about a OS distribution. - - This package creates a private module-global instance of this class with - default initialization arguments, that is used by the - `consolidated accessor functions`_ and `single source accessor functions`_. - By using default initialization arguments, that module-global instance - returns data about the current OS distribution (i.e. the distro this - package runs on). - - Normally, it is not necessary to create additional instances of this class. - However, in situations where control is needed over the exact data sources - that are used, instances of this class can be created with a specific - distro release file, or a specific os-release file, or without invoking the - lsb_release command. - """ - - def __init__( - self, - include_lsb=True, - os_release_file="", - distro_release_file="", - include_uname=True, - root_dir=None, - ): - # type: (bool, str, str, bool, Optional[str]) -> None - """ - The initialization method of this class gathers information from the - available data sources, and stores that in private instance attributes. - Subsequent access to the information items uses these private instance - attributes, so that the data sources are read only once. - - Parameters: - - * ``include_lsb`` (bool): Controls whether the - `lsb_release command output`_ is included as a data source. - - If the lsb_release command is not available in the program execution - path, the data source for the lsb_release command will be empty. - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is to be used as a data source. - - An empty string (the default) will cause the default path name to - be used (see `os-release file`_ for details). - - If the specified or defaulted os-release file does not exist, the - data source for the os-release file will be empty. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is to be used as a data source. - - An empty string (the default) will cause a default search algorithm - to be used (see `distro release file`_ for details). - - If the specified distro release file does not exist, or if no default - distro release file can be found, the data source for the distro - release file will be empty. - - * ``include_uname`` (bool): Controls whether uname command output is - included as a data source. If the uname command is not available in - the program execution path the data source for the uname command will - be empty. - - * ``root_dir`` (string): The absolute path to the root directory to use - to find distro-related information files. - - Public instance attributes: - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``include_lsb`` (bool): The result of the ``include_lsb`` parameter. - This controls whether the lsb information will be loaded. - - * ``include_uname`` (bool): The result of the ``include_uname`` - parameter. This controls whether the uname information will - be loaded. - - Raises: - - * :py:exc:`IOError`: Some I/O issue with an os-release file or distro - release file. - - * :py:exc:`subprocess.CalledProcessError`: The lsb_release command had - some issue (other than not being available in the program execution - path). - - * :py:exc:`UnicodeError`: A data source has unexpected characters or - uses an unexpected encoding. - """ - self.root_dir = root_dir - self.etc_dir = os.path.join(root_dir, "etc") if root_dir else _UNIXCONFDIR - self.usr_lib_dir = ( - os.path.join(root_dir, "usr/lib") if root_dir else _UNIXUSRLIBDIR - ) - - if os_release_file: - self.os_release_file = os_release_file - else: - etc_dir_os_release_file = os.path.join(self.etc_dir, _OS_RELEASE_BASENAME) - usr_lib_os_release_file = os.path.join( - self.usr_lib_dir, _OS_RELEASE_BASENAME - ) - - # NOTE: The idea is to respect order **and** have it set - # at all times for API backwards compatibility. - if os.path.isfile(etc_dir_os_release_file) or not os.path.isfile( - usr_lib_os_release_file - ): - self.os_release_file = etc_dir_os_release_file - else: - self.os_release_file = usr_lib_os_release_file - - self.distro_release_file = distro_release_file or "" # updated later - self.include_lsb = include_lsb - self.include_uname = include_uname - - def __repr__(self): - # type: () -> str - """Return repr of all info""" - return ( - "LinuxDistribution(" - "os_release_file={self.os_release_file!r}, " - "distro_release_file={self.distro_release_file!r}, " - "include_lsb={self.include_lsb!r}, " - "include_uname={self.include_uname!r}, " - "_os_release_info={self._os_release_info!r}, " - "_lsb_release_info={self._lsb_release_info!r}, " - "_distro_release_info={self._distro_release_info!r}, " - "_uname_info={self._uname_info!r})".format(self=self) - ) - - def linux_distribution(self, full_distribution_name=True): - # type: (bool) -> Tuple[str, str, str] - """ - Return information about the OS distribution that is compatible - with Python's :func:`platform.linux_distribution`, supporting a subset - of its parameters. - - For details, see :func:`distro.linux_distribution`. - """ - return ( - self.name() if full_distribution_name else self.id(), - self.version(), - self.codename(), - ) - - def id(self): - # type: () -> str - """Return the distro ID of the OS distribution, as a string. - - For details, see :func:`distro.id`. - """ - - def normalize(distro_id, table): - # type: (str, Dict[str, str]) -> str - distro_id = distro_id.lower().replace(" ", "_") - return table.get(distro_id, distro_id) - - distro_id = self.os_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_OS_ID) - - distro_id = self.lsb_release_attr("distributor_id") - if distro_id: - return normalize(distro_id, NORMALIZED_LSB_ID) - - distro_id = self.distro_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - distro_id = self.uname_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - return "" - - def name(self, pretty=False): - # type: (bool) -> str - """ - Return the name of the OS distribution, as a string. - - For details, see :func:`distro.name`. - """ - name = ( - self.os_release_attr("name") - or self.lsb_release_attr("distributor_id") - or self.distro_release_attr("name") - or self.uname_attr("name") - ) - if pretty: - name = self.os_release_attr("pretty_name") or self.lsb_release_attr( - "description" - ) - if not name: - name = self.distro_release_attr("name") or self.uname_attr("name") - version = self.version(pretty=True) - if version: - name = name + " " + version - return name or "" - - def version(self, pretty=False, best=False): - # type: (bool, bool) -> str - """ - Return the version of the OS distribution, as a string. - - For details, see :func:`distro.version`. - """ - versions = [ - self.os_release_attr("version_id"), - self.lsb_release_attr("release"), - self.distro_release_attr("version_id"), - self._parse_distro_release_content(self.os_release_attr("pretty_name")).get( - "version_id", "" - ), - self._parse_distro_release_content( - self.lsb_release_attr("description") - ).get("version_id", ""), - self.uname_attr("release"), - ] - version = "" - if best: - # This algorithm uses the last version in priority order that has - # the best precision. If the versions are not in conflict, that - # does not matter; otherwise, using the last one instead of the - # first one might be considered a surprise. - for v in versions: - if v.count(".") > version.count(".") or version == "": - version = v - else: - for v in versions: - if v != "": - version = v - break - if pretty and version and self.codename(): - version = "{0} ({1})".format(version, self.codename()) - return version - - def version_parts(self, best=False): - # type: (bool) -> Tuple[str, str, str] - """ - Return the version of the OS distribution, as a tuple of version - numbers. - - For details, see :func:`distro.version_parts`. - """ - version_str = self.version(best=best) - if version_str: - version_regex = re.compile(r"(\d+)\.?(\d+)?\.?(\d+)?") - matches = version_regex.match(version_str) - if matches: - major, minor, build_number = matches.groups() - return major, minor or "", build_number or "" - return "", "", "" - - def major_version(self, best=False): - # type: (bool) -> str - """ - Return the major version number of the current distribution. - - For details, see :func:`distro.major_version`. - """ - return self.version_parts(best)[0] - - def minor_version(self, best=False): - # type: (bool) -> str - """ - Return the minor version number of the current distribution. - - For details, see :func:`distro.minor_version`. - """ - return self.version_parts(best)[1] - - def build_number(self, best=False): - # type: (bool) -> str - """ - Return the build number of the current distribution. - - For details, see :func:`distro.build_number`. - """ - return self.version_parts(best)[2] - - def like(self): - # type: () -> str - """ - Return the IDs of distributions that are like the OS distribution. - - For details, see :func:`distro.like`. - """ - return self.os_release_attr("id_like") or "" - - def codename(self): - # type: () -> str - """ - Return the codename of the OS distribution. - - For details, see :func:`distro.codename`. - """ - try: - # Handle os_release specially since distros might purposefully set - # this to empty string to have no codename - return self._os_release_info["codename"] - except KeyError: - return ( - self.lsb_release_attr("codename") - or self.distro_release_attr("codename") - or "" - ) - - def info(self, pretty=False, best=False): - # type: (bool, bool) -> InfoDict - """ - Return certain machine-readable information about the OS - distribution. - - For details, see :func:`distro.info`. - """ - return dict( - id=self.id(), - version=self.version(pretty, best), - version_parts=dict( - major=self.major_version(best), - minor=self.minor_version(best), - build_number=self.build_number(best), - ), - like=self.like(), - codename=self.codename(), - ) - - def os_release_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the os-release file data source of the OS distribution. - - For details, see :func:`distro.os_release_info`. - """ - return self._os_release_info - - def lsb_release_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the lsb_release command data source of the OS - distribution. - - For details, see :func:`distro.lsb_release_info`. - """ - return self._lsb_release_info - - def distro_release_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the distro release file data source of the OS - distribution. - - For details, see :func:`distro.distro_release_info`. - """ - return self._distro_release_info - - def uname_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the uname command data source of the OS distribution. - - For details, see :func:`distro.uname_info`. - """ - return self._uname_info - - def os_release_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the os-release file data - source of the OS distribution. - - For details, see :func:`distro.os_release_attr`. - """ - return self._os_release_info.get(attribute, "") - - def lsb_release_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the lsb_release command - output data source of the OS distribution. - - For details, see :func:`distro.lsb_release_attr`. - """ - return self._lsb_release_info.get(attribute, "") - - def distro_release_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the distro release file - data source of the OS distribution. - - For details, see :func:`distro.distro_release_attr`. - """ - return self._distro_release_info.get(attribute, "") - - def uname_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the uname command - output data source of the OS distribution. - - For details, see :func:`distro.uname_attr`. - """ - return self._uname_info.get(attribute, "") - - @cached_property - def _os_release_info(self): - # type: () -> Dict[str, str] - """ - Get the information items from the specified os-release file. - - Returns: - A dictionary containing all information items. - """ - if os.path.isfile(self.os_release_file): - with open(self.os_release_file) as release_file: - return self._parse_os_release_content(release_file) - return {} - - @staticmethod - def _parse_os_release_content(lines): - # type: (TextIO) -> Dict[str, str] - """ - Parse the lines of an os-release file. - - Parameters: - - * lines: Iterable through the lines in the os-release file. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - lexer = shlex.shlex(lines, posix=True) - lexer.whitespace_split = True - - # The shlex module defines its `wordchars` variable using literals, - # making it dependent on the encoding of the Python source file. - # In Python 2.6 and 2.7, the shlex source file is encoded in - # 'iso-8859-1', and the `wordchars` variable is defined as a byte - # string. This causes a UnicodeDecodeError to be raised when the - # parsed content is a unicode object. The following fix resolves that - # (... but it should be fixed in shlex...): - if sys.version_info[0] == 2 and isinstance(lexer.wordchars, bytes): - lexer.wordchars = lexer.wordchars.decode("iso-8859-1") - - tokens = list(lexer) - for token in tokens: - # At this point, all shell-like parsing has been done (i.e. - # comments processed, quotes and backslash escape sequences - # processed, multi-line values assembled, trailing newlines - # stripped, etc.), so the tokens are now either: - # * variable assignments: var=value - # * commands or their arguments (not allowed in os-release) - if "=" in token: - k, v = token.split("=", 1) - props[k.lower()] = v - else: - # Ignore any tokens that are not variable assignments - pass - - if "version_codename" in props: - # os-release added a version_codename field. Use that in - # preference to anything else Note that some distros purposefully - # do not have code names. They should be setting - # version_codename="" - props["codename"] = props["version_codename"] - elif "ubuntu_codename" in props: - # Same as above but a non-standard field name used on older Ubuntus - props["codename"] = props["ubuntu_codename"] - elif "version" in props: - # If there is no version_codename, parse it from the version - match = re.search(r"(\(\D+\))|,(\s+)?\D+", props["version"]) - if match: - codename = match.group() - codename = codename.strip("()") - codename = codename.strip(",") - codename = codename.strip() - # codename appears within paranthese. - props["codename"] = codename - - return props - - @cached_property - def _lsb_release_info(self): - # type: () -> Dict[str, str] - """ - Get the information items from the lsb_release command output. - - Returns: - A dictionary containing all information items. - """ - if not self.include_lsb: - return {} - with open(os.devnull, "wb") as devnull: - try: - cmd = ("lsb_release", "-a") - stdout = subprocess.check_output(cmd, stderr=devnull) - # Command not found or lsb_release returned error - except (OSError, subprocess.CalledProcessError): - return {} - content = self._to_str(stdout).splitlines() - return self._parse_lsb_release_content(content) - - @staticmethod - def _parse_lsb_release_content(lines): - # type: (Iterable[str]) -> Dict[str, str] - """ - Parse the output of the lsb_release command. - - Parameters: - - * lines: Iterable through the lines of the lsb_release output. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - for line in lines: - kv = line.strip("\n").split(":", 1) - if len(kv) != 2: - # Ignore lines without colon. - continue - k, v = kv - props.update({k.replace(" ", "_").lower(): v.strip()}) - return props - - @cached_property - def _uname_info(self): - # type: () -> Dict[str, str] - with open(os.devnull, "wb") as devnull: - try: - cmd = ("uname", "-rs") - stdout = subprocess.check_output(cmd, stderr=devnull) - except OSError: - return {} - content = self._to_str(stdout).splitlines() - return self._parse_uname_content(content) - - @staticmethod - def _parse_uname_content(lines): - # type: (Sequence[str]) -> Dict[str, str] - props = {} - match = re.search(r"^([^\s]+)\s+([\d\.]+)", lines[0].strip()) - if match: - name, version = match.groups() - - # This is to prevent the Linux kernel version from - # appearing as the 'best' version on otherwise - # identifiable distributions. - if name == "Linux": - return {} - props["id"] = name.lower() - props["name"] = name - props["release"] = version - return props - - @staticmethod - def _to_str(text): - # type: (Union[bytes, str]) -> str - encoding = sys.getfilesystemencoding() - encoding = "utf-8" if encoding == "ascii" else encoding - - if sys.version_info[0] >= 3: - if isinstance(text, bytes): - return text.decode(encoding) - else: - if isinstance(text, unicode): # noqa - return text.encode(encoding) - - return text - - @cached_property - def _distro_release_info(self): - # type: () -> Dict[str, str] - """ - Get the information items from the specified distro release file. - - Returns: - A dictionary containing all information items. - """ - if self.distro_release_file: - # If it was specified, we use it and parse what we can, even if - # its file name or content does not match the expected pattern. - distro_info = self._parse_distro_release_file(self.distro_release_file) - basename = os.path.basename(self.distro_release_file) - # The file name pattern for user-specified distro release files - # is somewhat more tolerant (compared to when searching for the - # file), because we want to use what was specified as best as - # possible. - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - if "name" in distro_info and "cloudlinux" in distro_info["name"].lower(): - distro_info["id"] = "cloudlinux" - elif match: - distro_info["id"] = match.group(1) - return distro_info - else: - try: - basenames = os.listdir(self.etc_dir) - # We sort for repeatability in cases where there are multiple - # distro specific files; e.g. CentOS, Oracle, Enterprise all - # containing `redhat-release` on top of their own. - basenames.sort() - except OSError: - # This may occur when /etc is not readable but we can't be - # sure about the *-release files. Check common entries of - # /etc for information. If they turn out to not be there the - # error is handled in `_parse_distro_release_file()`. - basenames = [ - "SuSE-release", - "arch-release", - "base-release", - "centos-release", - "fedora-release", - "gentoo-release", - "mageia-release", - "mandrake-release", - "mandriva-release", - "mandrivalinux-release", - "manjaro-release", - "oracle-release", - "redhat-release", - "sl-release", - "slackware-version", - ] - for basename in basenames: - if basename in _DISTRO_RELEASE_IGNORE_BASENAMES: - continue - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - if match: - filepath = os.path.join(self.etc_dir, basename) - distro_info = self._parse_distro_release_file(filepath) - if "name" in distro_info: - # The name is always present if the pattern matches - self.distro_release_file = filepath - distro_info["id"] = match.group(1) - if "cloudlinux" in distro_info["name"].lower(): - distro_info["id"] = "cloudlinux" - return distro_info - return {} - - def _parse_distro_release_file(self, filepath): - # type: (str) -> Dict[str, str] - """ - Parse a distro release file. - - Parameters: - - * filepath: Path name of the distro release file. - - Returns: - A dictionary containing all information items. - """ - try: - with open(filepath) as fp: - # Only parse the first line. For instance, on SLES there - # are multiple lines. We don't want them... - return self._parse_distro_release_content(fp.readline()) - except (OSError, IOError): - # Ignore not being able to read a specific, seemingly version - # related file. - # See https://github.com/python-distro/distro/issues/162 - return {} - - @staticmethod - def _parse_distro_release_content(line): - # type: (str) -> Dict[str, str] - """ - Parse a line from a distro release file. - - Parameters: - * line: Line from the distro release file. Must be a unicode string - or a UTF-8 encoded byte string. - - Returns: - A dictionary containing all information items. - """ - matches = _DISTRO_RELEASE_CONTENT_REVERSED_PATTERN.match(line.strip()[::-1]) - distro_info = {} - if matches: - # regexp ensures non-None - distro_info["name"] = matches.group(3)[::-1] - if matches.group(2): - distro_info["version_id"] = matches.group(2)[::-1] - if matches.group(1): - distro_info["codename"] = matches.group(1)[::-1] - elif line: - distro_info["name"] = line.strip() - return distro_info - - -_distro = LinuxDistribution() - - -def main(): - # type: () -> None - logger = logging.getLogger(__name__) - logger.setLevel(logging.DEBUG) - logger.addHandler(logging.StreamHandler(sys.stdout)) - - parser = argparse.ArgumentParser(description="OS distro info tool") - parser.add_argument( - "--json", "-j", help="Output in machine readable format", action="store_true" - ) - - parser.add_argument( - "--root-dir", - "-r", - type=str, - dest="root_dir", - help="Path to the root filesystem directory (defaults to /)", - ) - - args = parser.parse_args() - - if args.root_dir: - dist = LinuxDistribution( - include_lsb=False, include_uname=False, root_dir=args.root_dir - ) - else: - dist = _distro - - if args.json: - logger.info(json.dumps(dist.info(), indent=4, sort_keys=True)) - else: - logger.info("Name: %s", dist.name(pretty=True)) - distribution_version = dist.version(pretty=True) - logger.info("Version: %s", distribution_version) - distribution_codename = dist.codename() - logger.info("Codename: %s", distribution_codename) - - -if __name__ == "__main__": - main() diff --git a/contrib/python/distro/py2/ya.make b/contrib/python/distro/py2/ya.make deleted file mode 100644 index e271753611..0000000000 --- a/contrib/python/distro/py2/ya.make +++ /dev/null @@ -1,31 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(1.6.0) - -LICENSE(Apache-2.0) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - distro.py -) - -RESOURCE_FILES( - PREFIX contrib/python/distro/py2/ - .dist-info/METADATA - .dist-info/entry_points.txt - .dist-info/top_level.txt -) - -END() - -RECURSE( - bin -) - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/distro/py3/.dist-info/METADATA b/contrib/python/distro/py3/.dist-info/METADATA deleted file mode 100644 index 4ea28c4980..0000000000 --- a/contrib/python/distro/py3/.dist-info/METADATA +++ /dev/null @@ -1,183 +0,0 @@ -Metadata-Version: 2.1 -Name: distro -Version: 1.8.0 -Summary: Distro - an OS platform information API -Home-page: https://github.com/python-distro/distro -Author: Nir Cohen -Author-email: nir36g@gmail.com -License: Apache License, Version 2.0 -Platform: All -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: Intended Audience :: System Administrators -Classifier: License :: OSI Approved :: Apache Software License -Classifier: Operating System :: POSIX :: Linux -Classifier: Operating System :: POSIX :: BSD -Classifier: Operating System :: POSIX :: BSD :: FreeBSD -Classifier: Operating System :: POSIX :: BSD :: NetBSD -Classifier: Operating System :: POSIX :: BSD :: OpenBSD -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3 :: Only -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Topic :: Software Development :: Libraries :: Python Modules -Classifier: Topic :: System :: Operating System -Requires-Python: >=3.6 -Description-Content-Type: text/markdown -License-File: LICENSE - -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - -To use as a standalone script, download `distro.py` directly: - -```shell -curl -O https://raw.githubusercontent.com/python-distro/distro/master/src/distro/distro.py -python distro.py -``` - -``distro`` is safe to vendor within projects that do not wish to add -dependencies. - -```shell -cd myproject -curl -O https://raw.githubusercontent.com/python-distro/distro/master/src/distro/distro.py -``` - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.name(pretty=True) -'CentOS Linux 8' ->>> distro.id() -'centos' ->>> distro.version(best=True) -'8.4.2105' -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 3.6+ and PyPy and on any -distribution that provides one or more of the data sources covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/stable/python3-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python3-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ diff --git a/contrib/python/distro/py3/.dist-info/entry_points.txt b/contrib/python/distro/py3/.dist-info/entry_points.txt deleted file mode 100644 index 08d29c55c1..0000000000 --- a/contrib/python/distro/py3/.dist-info/entry_points.txt +++ /dev/null @@ -1,2 +0,0 @@ -[console_scripts] -distro = distro.distro:main diff --git a/contrib/python/distro/py3/.dist-info/top_level.txt b/contrib/python/distro/py3/.dist-info/top_level.txt deleted file mode 100644 index 0e0933171d..0000000000 --- a/contrib/python/distro/py3/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -distro diff --git a/contrib/python/distro/py3/LICENSE b/contrib/python/distro/py3/LICENSE deleted file mode 100644 index e06d208186..0000000000 --- a/contrib/python/distro/py3/LICENSE +++ /dev/null @@ -1,202 +0,0 @@ -Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - diff --git a/contrib/python/distro/py3/README.md b/contrib/python/distro/py3/README.md deleted file mode 100644 index de6f3c79d3..0000000000 --- a/contrib/python/distro/py3/README.md +++ /dev/null @@ -1,152 +0,0 @@ -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - -To use as a standalone script, download `distro.py` directly: - -```shell -curl -O https://raw.githubusercontent.com/python-distro/distro/master/src/distro/distro.py -python distro.py -``` - -``distro`` is safe to vendor within projects that do not wish to add -dependencies. - -```shell -cd myproject -curl -O https://raw.githubusercontent.com/python-distro/distro/master/src/distro/distro.py -``` - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.name(pretty=True) -'CentOS Linux 8' ->>> distro.id() -'centos' ->>> distro.version(best=True) -'8.4.2105' -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 3.6+ and PyPy and on any -distribution that provides one or more of the data sources covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/stable/python3-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python3-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ diff --git a/contrib/python/distro/py3/distro/__init__.py b/contrib/python/distro/py3/distro/__init__.py deleted file mode 100644 index 7686fe85a7..0000000000 --- a/contrib/python/distro/py3/distro/__init__.py +++ /dev/null @@ -1,54 +0,0 @@ -from .distro import ( - NORMALIZED_DISTRO_ID, - NORMALIZED_LSB_ID, - NORMALIZED_OS_ID, - LinuxDistribution, - __version__, - build_number, - codename, - distro_release_attr, - distro_release_info, - id, - info, - like, - linux_distribution, - lsb_release_attr, - lsb_release_info, - major_version, - minor_version, - name, - os_release_attr, - os_release_info, - uname_attr, - uname_info, - version, - version_parts, -) - -__all__ = [ - "NORMALIZED_DISTRO_ID", - "NORMALIZED_LSB_ID", - "NORMALIZED_OS_ID", - "LinuxDistribution", - "build_number", - "codename", - "distro_release_attr", - "distro_release_info", - "id", - "info", - "like", - "linux_distribution", - "lsb_release_attr", - "lsb_release_info", - "major_version", - "minor_version", - "name", - "os_release_attr", - "os_release_info", - "uname_attr", - "uname_info", - "version", - "version_parts", -] - -__version__ = __version__ diff --git a/contrib/python/distro/py3/distro/__main__.py b/contrib/python/distro/py3/distro/__main__.py deleted file mode 100644 index 0c01d5b08b..0000000000 --- a/contrib/python/distro/py3/distro/__main__.py +++ /dev/null @@ -1,4 +0,0 @@ -from .distro import main - -if __name__ == "__main__": - main() diff --git a/contrib/python/distro/py3/distro/distro.py b/contrib/python/distro/py3/distro/distro.py deleted file mode 100644 index 89e1868047..0000000000 --- a/contrib/python/distro/py3/distro/distro.py +++ /dev/null @@ -1,1399 +0,0 @@ -#!/usr/bin/env python -# Copyright 2015,2016,2017 Nir Cohen -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -""" -The ``distro`` package (``distro`` stands for Linux Distribution) provides -information about the Linux distribution it runs on, such as a reliable -machine-readable distro ID, or version information. - -It is the recommended replacement for Python's original -:py:func:`platform.linux_distribution` function, but it provides much more -functionality. An alternative implementation became necessary because Python -3.5 deprecated this function, and Python 3.8 removed it altogether. Its -predecessor function :py:func:`platform.dist` was already deprecated since -Python 2.6 and removed in Python 3.8. Still, there are many cases in which -access to OS distribution information is needed. See `Python issue 1322 -<https://bugs.python.org/issue1322>`_ for more information. -""" - -import argparse -import json -import logging -import os -import re -import shlex -import subprocess -import sys -import warnings -from typing import ( - Any, - Callable, - Dict, - Iterable, - Optional, - Sequence, - TextIO, - Tuple, - Type, -) - -try: - from typing import TypedDict -except ImportError: - # Python 3.7 - TypedDict = dict - -__version__ = "1.8.0" - - -class VersionDict(TypedDict): - major: str - minor: str - build_number: str - - -class InfoDict(TypedDict): - id: str - version: str - version_parts: VersionDict - like: str - codename: str - - -_UNIXCONFDIR = os.environ.get("UNIXCONFDIR", "/etc") -_UNIXUSRLIBDIR = os.environ.get("UNIXUSRLIBDIR", "/usr/lib") -_OS_RELEASE_BASENAME = "os-release" - -#: Translation table for normalizing the "ID" attribute defined in os-release -#: files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as defined in the os-release file, translated to lower case, -#: with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_OS_ID = { - "ol": "oracle", # Oracle Linux - "opensuse-leap": "opensuse", # Newer versions of OpenSuSE report as opensuse-leap -} - -#: Translation table for normalizing the "Distributor ID" attribute returned by -#: the lsb_release command, for use by the :func:`distro.id` method. -#: -#: * Key: Value as returned by the lsb_release command, translated to lower -#: case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_LSB_ID = { - "enterpriseenterpriseas": "oracle", # Oracle Enterprise Linux 4 - "enterpriseenterpriseserver": "oracle", # Oracle Linux 5 - "redhatenterpriseworkstation": "rhel", # RHEL 6, 7 Workstation - "redhatenterpriseserver": "rhel", # RHEL 6, 7 Server - "redhatenterprisecomputenode": "rhel", # RHEL 6 ComputeNode -} - -#: Translation table for normalizing the distro ID derived from the file name -#: of distro release files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as derived from the file name of a distro release file, -#: translated to lower case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_DISTRO_ID = { - "redhat": "rhel", # RHEL 6.x, 7.x -} - -# Pattern for content of distro release file (reversed) -_DISTRO_RELEASE_CONTENT_REVERSED_PATTERN = re.compile( - r"(?:[^)]*\)(.*)\()? *(?:STL )?([\d.+\-a-z]*\d) *(?:esaeler *)?(.+)" -) - -# Pattern for base file name of distro release file -_DISTRO_RELEASE_BASENAME_PATTERN = re.compile(r"(\w+)[-_](release|version)$") - -# Base file names to be looked up for if _UNIXCONFDIR is not readable. -_DISTRO_RELEASE_BASENAMES = [ - "SuSE-release", - "arch-release", - "base-release", - "centos-release", - "fedora-release", - "gentoo-release", - "mageia-release", - "mandrake-release", - "mandriva-release", - "mandrivalinux-release", - "manjaro-release", - "oracle-release", - "redhat-release", - "rocky-release", - "sl-release", - "slackware-version", -] - -# Base file names to be ignored when searching for distro release file -_DISTRO_RELEASE_IGNORE_BASENAMES = ( - "debian_version", - "lsb-release", - "oem-release", - _OS_RELEASE_BASENAME, - "system-release", - "plesk-release", - "iredmail-release", -) - - -def linux_distribution(full_distribution_name: bool = True) -> Tuple[str, str, str]: - """ - .. deprecated:: 1.6.0 - - :func:`distro.linux_distribution()` is deprecated. It should only be - used as a compatibility shim with Python's - :py:func:`platform.linux_distribution()`. Please use :func:`distro.id`, - :func:`distro.version` and :func:`distro.name` instead. - - Return information about the current OS distribution as a tuple - ``(id_name, version, codename)`` with items as follows: - - * ``id_name``: If *full_distribution_name* is false, the result of - :func:`distro.id`. Otherwise, the result of :func:`distro.name`. - - * ``version``: The result of :func:`distro.version`. - - * ``codename``: The extra item (usually in parentheses) after the - os-release version number, or the result of :func:`distro.codename`. - - The interface of this function is compatible with the original - :py:func:`platform.linux_distribution` function, supporting a subset of - its parameters. - - The data it returns may not exactly be the same, because it uses more data - sources than the original function, and that may lead to different data if - the OS distribution is not consistent across multiple data sources it - provides (there are indeed such distributions ...). - - Another reason for differences is the fact that the :func:`distro.id` - method normalizes the distro ID string to a reliable machine-readable value - for a number of popular OS distributions. - """ - warnings.warn( - "distro.linux_distribution() is deprecated. It should only be used as a " - "compatibility shim with Python's platform.linux_distribution(). Please use " - "distro.id(), distro.version() and distro.name() instead.", - DeprecationWarning, - stacklevel=2, - ) - return _distro.linux_distribution(full_distribution_name) - - -def id() -> str: - """ - Return the distro ID of the current distribution, as a - machine-readable string. - - For a number of OS distributions, the returned distro ID value is - *reliable*, in the sense that it is documented and that it does not change - across releases of the distribution. - - This package maintains the following reliable distro ID values: - - ============== ========================================= - Distro ID Distribution - ============== ========================================= - "ubuntu" Ubuntu - "debian" Debian - "rhel" RedHat Enterprise Linux - "centos" CentOS - "fedora" Fedora - "sles" SUSE Linux Enterprise Server - "opensuse" openSUSE - "amzn" Amazon Linux - "arch" Arch Linux - "buildroot" Buildroot - "cloudlinux" CloudLinux OS - "exherbo" Exherbo Linux - "gentoo" GenToo Linux - "ibm_powerkvm" IBM PowerKVM - "kvmibm" KVM for IBM z Systems - "linuxmint" Linux Mint - "mageia" Mageia - "mandriva" Mandriva Linux - "parallels" Parallels - "pidora" Pidora - "raspbian" Raspbian - "oracle" Oracle Linux (and Oracle Enterprise Linux) - "scientific" Scientific Linux - "slackware" Slackware - "xenserver" XenServer - "openbsd" OpenBSD - "netbsd" NetBSD - "freebsd" FreeBSD - "midnightbsd" MidnightBSD - "rocky" Rocky Linux - "aix" AIX - "guix" Guix System - ============== ========================================= - - If you have a need to get distros for reliable IDs added into this set, - or if you find that the :func:`distro.id` function returns a different - distro ID for one of the listed distros, please create an issue in the - `distro issue tracker`_. - - **Lookup hierarchy and transformations:** - - First, the ID is obtained from the following sources, in the specified - order. The first available and non-empty value is used: - - * the value of the "ID" attribute of the os-release file, - - * the value of the "Distributor ID" attribute returned by the lsb_release - command, - - * the first part of the file name of the distro release file, - - The so determined ID value then passes the following transformations, - before it is returned by this method: - - * it is translated to lower case, - - * blanks (which should not be there anyway) are translated to underscores, - - * a normalization of the ID is performed, based upon - `normalization tables`_. The purpose of this normalization is to ensure - that the ID is as reliable as possible, even across incompatible changes - in the OS distributions. A common reason for an incompatible change is - the addition of an os-release file, or the addition of the lsb_release - command, with ID values that differ from what was previously determined - from the distro release file name. - """ - return _distro.id() - - -def name(pretty: bool = False) -> str: - """ - Return the name of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the name is returned without version or codename. - (e.g. "CentOS Linux") - - If *pretty* is true, the version and codename are appended. - (e.g. "CentOS Linux 7.1.1503 (Core)") - - **Lookup hierarchy:** - - The name is obtained from the following sources, in the specified order. - The first available and non-empty value is used: - - * If *pretty* is false: - - - the value of the "NAME" attribute of the os-release file, - - - the value of the "Distributor ID" attribute returned by the lsb_release - command, - - - the value of the "<name>" field of the distro release file. - - * If *pretty* is true: - - - the value of the "PRETTY_NAME" attribute of the os-release file, - - - the value of the "Description" attribute returned by the lsb_release - command, - - - the value of the "<name>" field of the distro release file, appended - with the value of the pretty version ("<version_id>" and "<codename>" - fields) of the distro release file, if available. - """ - return _distro.name(pretty) - - -def version(pretty: bool = False, best: bool = False) -> str: - """ - Return the version of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the version is returned without codename (e.g. - "7.0"). - - If *pretty* is true, the codename in parenthesis is appended, if the - codename is non-empty (e.g. "7.0 (Maipo)"). - - Some distributions provide version numbers with different precisions in - the different sources of distribution information. Examining the different - sources in a fixed priority order does not always yield the most precise - version (e.g. for Debian 8.2, or CentOS 7.1). - - Some other distributions may not provide this kind of information. In these - cases, an empty string would be returned. This behavior can be observed - with rolling releases distributions (e.g. Arch Linux). - - The *best* parameter can be used to control the approach for the returned - version: - - If *best* is false, the first non-empty version number in priority order of - the examined sources is returned. - - If *best* is true, the most precise version number out of all examined - sources is returned. - - **Lookup hierarchy:** - - In all cases, the version number is obtained from the following sources. - If *best* is false, this order represents the priority order: - - * the value of the "VERSION_ID" attribute of the os-release file, - * the value of the "Release" attribute returned by the lsb_release - command, - * the version number parsed from the "<version_id>" field of the first line - of the distro release file, - * the version number parsed from the "PRETTY_NAME" attribute of the - os-release file, if it follows the format of the distro release files. - * the version number parsed from the "Description" attribute returned by - the lsb_release command, if it follows the format of the distro release - files. - """ - return _distro.version(pretty, best) - - -def version_parts(best: bool = False) -> Tuple[str, str, str]: - """ - Return the version of the current OS distribution as a tuple - ``(major, minor, build_number)`` with items as follows: - - * ``major``: The result of :func:`distro.major_version`. - - * ``minor``: The result of :func:`distro.minor_version`. - - * ``build_number``: The result of :func:`distro.build_number`. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.version_parts(best) - - -def major_version(best: bool = False) -> str: - """ - Return the major version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The major version is the first - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.major_version(best) - - -def minor_version(best: bool = False) -> str: - """ - Return the minor version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The minor version is the second - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.minor_version(best) - - -def build_number(best: bool = False) -> str: - """ - Return the build number of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The build number is the third part - of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.build_number(best) - - -def like() -> str: - """ - Return a space-separated list of distro IDs of distributions that are - closely related to the current OS distribution in regards to packaging - and programming interfaces, for example distributions the current - distribution is a derivative from. - - **Lookup hierarchy:** - - This information item is only provided by the os-release file. - For details, see the description of the "ID_LIKE" attribute in the - `os-release man page - <http://www.freedesktop.org/software/systemd/man/os-release.html>`_. - """ - return _distro.like() - - -def codename() -> str: - """ - Return the codename for the release of the current OS distribution, - as a string. - - If the distribution does not have a codename, an empty string is returned. - - Note that the returned codename is not always really a codename. For - example, openSUSE returns "x86_64". This function does not handle such - cases in any special way and just returns the string it finds, if any. - - **Lookup hierarchy:** - - * the codename within the "VERSION" attribute of the os-release file, if - provided, - - * the value of the "Codename" attribute returned by the lsb_release - command, - - * the value of the "<codename>" field of the distro release file. - """ - return _distro.codename() - - -def info(pretty: bool = False, best: bool = False) -> InfoDict: - """ - Return certain machine-readable information items about the current OS - distribution in a dictionary, as shown in the following example: - - .. sourcecode:: python - - { - 'id': 'rhel', - 'version': '7.0', - 'version_parts': { - 'major': '7', - 'minor': '0', - 'build_number': '' - }, - 'like': 'fedora', - 'codename': 'Maipo' - } - - The dictionary structure and keys are always the same, regardless of which - information items are available in the underlying data sources. The values - for the various keys are as follows: - - * ``id``: The result of :func:`distro.id`. - - * ``version``: The result of :func:`distro.version`. - - * ``version_parts -> major``: The result of :func:`distro.major_version`. - - * ``version_parts -> minor``: The result of :func:`distro.minor_version`. - - * ``version_parts -> build_number``: The result of - :func:`distro.build_number`. - - * ``like``: The result of :func:`distro.like`. - - * ``codename``: The result of :func:`distro.codename`. - - For a description of the *pretty* and *best* parameters, see the - :func:`distro.version` method. - """ - return _distro.info(pretty, best) - - -def os_release_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the os-release file data source of the current OS distribution. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_info() - - -def lsb_release_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the lsb_release command data source of the current OS distribution. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_info() - - -def distro_release_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_info() - - -def uname_info() -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - """ - return _distro.uname_info() - - -def os_release_attr(attribute: str) -> str: - """ - Return a single named information item from the os-release file data source - of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_attr(attribute) - - -def lsb_release_attr(attribute: str) -> str: - """ - Return a single named information item from the lsb_release command output - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_attr(attribute) - - -def distro_release_attr(attribute: str) -> str: - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_attr(attribute) - - -def uname_attr(attribute: str) -> str: - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - """ - return _distro.uname_attr(attribute) - - -try: - from functools import cached_property -except ImportError: - # Python < 3.8 - class cached_property: # type: ignore - """A version of @property which caches the value. On access, it calls the - underlying function and sets the value in `__dict__` so future accesses - will not re-call the property. - """ - - def __init__(self, f: Callable[[Any], Any]) -> None: - self._fname = f.__name__ - self._f = f - - def __get__(self, obj: Any, owner: Type[Any]) -> Any: - assert obj is not None, f"call {self._fname} on an instance" - ret = obj.__dict__[self._fname] = self._f(obj) - return ret - - -class LinuxDistribution: - """ - Provides information about a OS distribution. - - This package creates a private module-global instance of this class with - default initialization arguments, that is used by the - `consolidated accessor functions`_ and `single source accessor functions`_. - By using default initialization arguments, that module-global instance - returns data about the current OS distribution (i.e. the distro this - package runs on). - - Normally, it is not necessary to create additional instances of this class. - However, in situations where control is needed over the exact data sources - that are used, instances of this class can be created with a specific - distro release file, or a specific os-release file, or without invoking the - lsb_release command. - """ - - def __init__( - self, - include_lsb: Optional[bool] = None, - os_release_file: str = "", - distro_release_file: str = "", - include_uname: Optional[bool] = None, - root_dir: Optional[str] = None, - include_oslevel: Optional[bool] = None, - ) -> None: - """ - The initialization method of this class gathers information from the - available data sources, and stores that in private instance attributes. - Subsequent access to the information items uses these private instance - attributes, so that the data sources are read only once. - - Parameters: - - * ``include_lsb`` (bool): Controls whether the - `lsb_release command output`_ is included as a data source. - - If the lsb_release command is not available in the program execution - path, the data source for the lsb_release command will be empty. - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is to be used as a data source. - - An empty string (the default) will cause the default path name to - be used (see `os-release file`_ for details). - - If the specified or defaulted os-release file does not exist, the - data source for the os-release file will be empty. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is to be used as a data source. - - An empty string (the default) will cause a default search algorithm - to be used (see `distro release file`_ for details). - - If the specified distro release file does not exist, or if no default - distro release file can be found, the data source for the distro - release file will be empty. - - * ``include_uname`` (bool): Controls whether uname command output is - included as a data source. If the uname command is not available in - the program execution path the data source for the uname command will - be empty. - - * ``root_dir`` (string): The absolute path to the root directory to use - to find distro-related information files. Note that ``include_*`` - parameters must not be enabled in combination with ``root_dir``. - - * ``include_oslevel`` (bool): Controls whether (AIX) oslevel command - output is included as a data source. If the oslevel command is not - available in the program execution path the data source will be - empty. - - Public instance attributes: - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``include_lsb`` (bool): The result of the ``include_lsb`` parameter. - This controls whether the lsb information will be loaded. - - * ``include_uname`` (bool): The result of the ``include_uname`` - parameter. This controls whether the uname information will - be loaded. - - * ``include_oslevel`` (bool): The result of the ``include_oslevel`` - parameter. This controls whether (AIX) oslevel information will be - loaded. - - * ``root_dir`` (string): The result of the ``root_dir`` parameter. - The absolute path to the root directory to use to find distro-related - information files. - - Raises: - - * :py:exc:`ValueError`: Initialization parameters combination is not - supported. - - * :py:exc:`OSError`: Some I/O issue with an os-release file or distro - release file. - - * :py:exc:`UnicodeError`: A data source has unexpected characters or - uses an unexpected encoding. - """ - self.root_dir = root_dir - self.etc_dir = os.path.join(root_dir, "etc") if root_dir else _UNIXCONFDIR - self.usr_lib_dir = ( - os.path.join(root_dir, "usr/lib") if root_dir else _UNIXUSRLIBDIR - ) - - if os_release_file: - self.os_release_file = os_release_file - else: - etc_dir_os_release_file = os.path.join(self.etc_dir, _OS_RELEASE_BASENAME) - usr_lib_os_release_file = os.path.join( - self.usr_lib_dir, _OS_RELEASE_BASENAME - ) - - # NOTE: The idea is to respect order **and** have it set - # at all times for API backwards compatibility. - if os.path.isfile(etc_dir_os_release_file) or not os.path.isfile( - usr_lib_os_release_file - ): - self.os_release_file = etc_dir_os_release_file - else: - self.os_release_file = usr_lib_os_release_file - - self.distro_release_file = distro_release_file or "" # updated later - - is_root_dir_defined = root_dir is not None - if is_root_dir_defined and (include_lsb or include_uname or include_oslevel): - raise ValueError( - "Including subprocess data sources from specific root_dir is disallowed" - " to prevent false information" - ) - self.include_lsb = ( - include_lsb if include_lsb is not None else not is_root_dir_defined - ) - self.include_uname = ( - include_uname if include_uname is not None else not is_root_dir_defined - ) - self.include_oslevel = ( - include_oslevel if include_oslevel is not None else not is_root_dir_defined - ) - - def __repr__(self) -> str: - """Return repr of all info""" - return ( - "LinuxDistribution(" - "os_release_file={self.os_release_file!r}, " - "distro_release_file={self.distro_release_file!r}, " - "include_lsb={self.include_lsb!r}, " - "include_uname={self.include_uname!r}, " - "include_oslevel={self.include_oslevel!r}, " - "root_dir={self.root_dir!r}, " - "_os_release_info={self._os_release_info!r}, " - "_lsb_release_info={self._lsb_release_info!r}, " - "_distro_release_info={self._distro_release_info!r}, " - "_uname_info={self._uname_info!r}, " - "_oslevel_info={self._oslevel_info!r})".format(self=self) - ) - - def linux_distribution( - self, full_distribution_name: bool = True - ) -> Tuple[str, str, str]: - """ - Return information about the OS distribution that is compatible - with Python's :func:`platform.linux_distribution`, supporting a subset - of its parameters. - - For details, see :func:`distro.linux_distribution`. - """ - return ( - self.name() if full_distribution_name else self.id(), - self.version(), - self._os_release_info.get("release_codename") or self.codename(), - ) - - def id(self) -> str: - """Return the distro ID of the OS distribution, as a string. - - For details, see :func:`distro.id`. - """ - - def normalize(distro_id: str, table: Dict[str, str]) -> str: - distro_id = distro_id.lower().replace(" ", "_") - return table.get(distro_id, distro_id) - - distro_id = self.os_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_OS_ID) - - distro_id = self.lsb_release_attr("distributor_id") - if distro_id: - return normalize(distro_id, NORMALIZED_LSB_ID) - - distro_id = self.distro_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - distro_id = self.uname_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - return "" - - def name(self, pretty: bool = False) -> str: - """ - Return the name of the OS distribution, as a string. - - For details, see :func:`distro.name`. - """ - name = ( - self.os_release_attr("name") - or self.lsb_release_attr("distributor_id") - or self.distro_release_attr("name") - or self.uname_attr("name") - ) - if pretty: - name = self.os_release_attr("pretty_name") or self.lsb_release_attr( - "description" - ) - if not name: - name = self.distro_release_attr("name") or self.uname_attr("name") - version = self.version(pretty=True) - if version: - name = f"{name} {version}" - return name or "" - - def version(self, pretty: bool = False, best: bool = False) -> str: - """ - Return the version of the OS distribution, as a string. - - For details, see :func:`distro.version`. - """ - versions = [ - self.os_release_attr("version_id"), - self.lsb_release_attr("release"), - self.distro_release_attr("version_id"), - self._parse_distro_release_content(self.os_release_attr("pretty_name")).get( - "version_id", "" - ), - self._parse_distro_release_content( - self.lsb_release_attr("description") - ).get("version_id", ""), - self.uname_attr("release"), - ] - if self.uname_attr("id").startswith("aix"): - # On AIX platforms, prefer oslevel command output. - versions.insert(0, self.oslevel_info()) - elif self.id() == "debian" or "debian" in self.like().split(): - # On Debian-like, add debian_version file content to candidates list. - versions.append(self._debian_version) - version = "" - if best: - # This algorithm uses the last version in priority order that has - # the best precision. If the versions are not in conflict, that - # does not matter; otherwise, using the last one instead of the - # first one might be considered a surprise. - for v in versions: - if v.count(".") > version.count(".") or version == "": - version = v - else: - for v in versions: - if v != "": - version = v - break - if pretty and version and self.codename(): - version = f"{version} ({self.codename()})" - return version - - def version_parts(self, best: bool = False) -> Tuple[str, str, str]: - """ - Return the version of the OS distribution, as a tuple of version - numbers. - - For details, see :func:`distro.version_parts`. - """ - version_str = self.version(best=best) - if version_str: - version_regex = re.compile(r"(\d+)\.?(\d+)?\.?(\d+)?") - matches = version_regex.match(version_str) - if matches: - major, minor, build_number = matches.groups() - return major, minor or "", build_number or "" - return "", "", "" - - def major_version(self, best: bool = False) -> str: - """ - Return the major version number of the current distribution. - - For details, see :func:`distro.major_version`. - """ - return self.version_parts(best)[0] - - def minor_version(self, best: bool = False) -> str: - """ - Return the minor version number of the current distribution. - - For details, see :func:`distro.minor_version`. - """ - return self.version_parts(best)[1] - - def build_number(self, best: bool = False) -> str: - """ - Return the build number of the current distribution. - - For details, see :func:`distro.build_number`. - """ - return self.version_parts(best)[2] - - def like(self) -> str: - """ - Return the IDs of distributions that are like the OS distribution. - - For details, see :func:`distro.like`. - """ - return self.os_release_attr("id_like") or "" - - def codename(self) -> str: - """ - Return the codename of the OS distribution. - - For details, see :func:`distro.codename`. - """ - try: - # Handle os_release specially since distros might purposefully set - # this to empty string to have no codename - return self._os_release_info["codename"] - except KeyError: - return ( - self.lsb_release_attr("codename") - or self.distro_release_attr("codename") - or "" - ) - - def info(self, pretty: bool = False, best: bool = False) -> InfoDict: - """ - Return certain machine-readable information about the OS - distribution. - - For details, see :func:`distro.info`. - """ - return dict( - id=self.id(), - version=self.version(pretty, best), - version_parts=dict( - major=self.major_version(best), - minor=self.minor_version(best), - build_number=self.build_number(best), - ), - like=self.like(), - codename=self.codename(), - ) - - def os_release_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the os-release file data source of the OS distribution. - - For details, see :func:`distro.os_release_info`. - """ - return self._os_release_info - - def lsb_release_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the lsb_release command data source of the OS - distribution. - - For details, see :func:`distro.lsb_release_info`. - """ - return self._lsb_release_info - - def distro_release_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the distro release file data source of the OS - distribution. - - For details, see :func:`distro.distro_release_info`. - """ - return self._distro_release_info - - def uname_info(self) -> Dict[str, str]: - """ - Return a dictionary containing key-value pairs for the information - items from the uname command data source of the OS distribution. - - For details, see :func:`distro.uname_info`. - """ - return self._uname_info - - def oslevel_info(self) -> str: - """ - Return AIX' oslevel command output. - """ - return self._oslevel_info - - def os_release_attr(self, attribute: str) -> str: - """ - Return a single named information item from the os-release file data - source of the OS distribution. - - For details, see :func:`distro.os_release_attr`. - """ - return self._os_release_info.get(attribute, "") - - def lsb_release_attr(self, attribute: str) -> str: - """ - Return a single named information item from the lsb_release command - output data source of the OS distribution. - - For details, see :func:`distro.lsb_release_attr`. - """ - return self._lsb_release_info.get(attribute, "") - - def distro_release_attr(self, attribute: str) -> str: - """ - Return a single named information item from the distro release file - data source of the OS distribution. - - For details, see :func:`distro.distro_release_attr`. - """ - return self._distro_release_info.get(attribute, "") - - def uname_attr(self, attribute: str) -> str: - """ - Return a single named information item from the uname command - output data source of the OS distribution. - - For details, see :func:`distro.uname_attr`. - """ - return self._uname_info.get(attribute, "") - - @cached_property - def _os_release_info(self) -> Dict[str, str]: - """ - Get the information items from the specified os-release file. - - Returns: - A dictionary containing all information items. - """ - if os.path.isfile(self.os_release_file): - with open(self.os_release_file, encoding="utf-8") as release_file: - return self._parse_os_release_content(release_file) - return {} - - @staticmethod - def _parse_os_release_content(lines: TextIO) -> Dict[str, str]: - """ - Parse the lines of an os-release file. - - Parameters: - - * lines: Iterable through the lines in the os-release file. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - lexer = shlex.shlex(lines, posix=True) - lexer.whitespace_split = True - - tokens = list(lexer) - for token in tokens: - # At this point, all shell-like parsing has been done (i.e. - # comments processed, quotes and backslash escape sequences - # processed, multi-line values assembled, trailing newlines - # stripped, etc.), so the tokens are now either: - # * variable assignments: var=value - # * commands or their arguments (not allowed in os-release) - # Ignore any tokens that are not variable assignments - if "=" in token: - k, v = token.split("=", 1) - props[k.lower()] = v - - if "version" in props: - # extract release codename (if any) from version attribute - match = re.search(r"\((\D+)\)|,\s*(\D+)", props["version"]) - if match: - release_codename = match.group(1) or match.group(2) - props["codename"] = props["release_codename"] = release_codename - - if "version_codename" in props: - # os-release added a version_codename field. Use that in - # preference to anything else Note that some distros purposefully - # do not have code names. They should be setting - # version_codename="" - props["codename"] = props["version_codename"] - elif "ubuntu_codename" in props: - # Same as above but a non-standard field name used on older Ubuntus - props["codename"] = props["ubuntu_codename"] - - return props - - @cached_property - def _lsb_release_info(self) -> Dict[str, str]: - """ - Get the information items from the lsb_release command output. - - Returns: - A dictionary containing all information items. - """ - if not self.include_lsb: - return {} - try: - cmd = ("lsb_release", "-a") - stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) - # Command not found or lsb_release returned error - except (OSError, subprocess.CalledProcessError): - return {} - content = self._to_str(stdout).splitlines() - return self._parse_lsb_release_content(content) - - @staticmethod - def _parse_lsb_release_content(lines: Iterable[str]) -> Dict[str, str]: - """ - Parse the output of the lsb_release command. - - Parameters: - - * lines: Iterable through the lines of the lsb_release output. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - for line in lines: - kv = line.strip("\n").split(":", 1) - if len(kv) != 2: - # Ignore lines without colon. - continue - k, v = kv - props.update({k.replace(" ", "_").lower(): v.strip()}) - return props - - @cached_property - def _uname_info(self) -> Dict[str, str]: - if not self.include_uname: - return {} - try: - cmd = ("uname", "-rs") - stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL) - except OSError: - return {} - content = self._to_str(stdout).splitlines() - return self._parse_uname_content(content) - - @cached_property - def _oslevel_info(self) -> str: - if not self.include_oslevel: - return "" - try: - stdout = subprocess.check_output("oslevel", stderr=subprocess.DEVNULL) - except (OSError, subprocess.CalledProcessError): - return "" - return self._to_str(stdout).strip() - - @cached_property - def _debian_version(self) -> str: - try: - with open( - os.path.join(self.etc_dir, "debian_version"), encoding="ascii" - ) as fp: - return fp.readline().rstrip() - except FileNotFoundError: - return "" - - @staticmethod - def _parse_uname_content(lines: Sequence[str]) -> Dict[str, str]: - if not lines: - return {} - props = {} - match = re.search(r"^([^\s]+)\s+([\d\.]+)", lines[0].strip()) - if match: - name, version = match.groups() - - # This is to prevent the Linux kernel version from - # appearing as the 'best' version on otherwise - # identifiable distributions. - if name == "Linux": - return {} - props["id"] = name.lower() - props["name"] = name - props["release"] = version - return props - - @staticmethod - def _to_str(bytestring: bytes) -> str: - encoding = sys.getfilesystemencoding() - return bytestring.decode(encoding) - - @cached_property - def _distro_release_info(self) -> Dict[str, str]: - """ - Get the information items from the specified distro release file. - - Returns: - A dictionary containing all information items. - """ - if self.distro_release_file: - # If it was specified, we use it and parse what we can, even if - # its file name or content does not match the expected pattern. - distro_info = self._parse_distro_release_file(self.distro_release_file) - basename = os.path.basename(self.distro_release_file) - # The file name pattern for user-specified distro release files - # is somewhat more tolerant (compared to when searching for the - # file), because we want to use what was specified as best as - # possible. - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - else: - try: - basenames = [ - basename - for basename in os.listdir(self.etc_dir) - if basename not in _DISTRO_RELEASE_IGNORE_BASENAMES - and os.path.isfile(os.path.join(self.etc_dir, basename)) - ] - # We sort for repeatability in cases where there are multiple - # distro specific files; e.g. CentOS, Oracle, Enterprise all - # containing `redhat-release` on top of their own. - basenames.sort() - except OSError: - # This may occur when /etc is not readable but we can't be - # sure about the *-release files. Check common entries of - # /etc for information. If they turn out to not be there the - # error is handled in `_parse_distro_release_file()`. - basenames = _DISTRO_RELEASE_BASENAMES - for basename in basenames: - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - if match is None: - continue - filepath = os.path.join(self.etc_dir, basename) - distro_info = self._parse_distro_release_file(filepath) - # The name is always present if the pattern matches. - if "name" not in distro_info: - continue - self.distro_release_file = filepath - break - else: # the loop didn't "break": no candidate. - return {} - - if match is not None: - distro_info["id"] = match.group(1) - - # CloudLinux < 7: manually enrich info with proper id. - if "cloudlinux" in distro_info.get("name", "").lower(): - distro_info["id"] = "cloudlinux" - - return distro_info - - def _parse_distro_release_file(self, filepath: str) -> Dict[str, str]: - """ - Parse a distro release file. - - Parameters: - - * filepath: Path name of the distro release file. - - Returns: - A dictionary containing all information items. - """ - try: - with open(filepath, encoding="utf-8") as fp: - # Only parse the first line. For instance, on SLES there - # are multiple lines. We don't want them... - return self._parse_distro_release_content(fp.readline()) - except OSError: - # Ignore not being able to read a specific, seemingly version - # related file. - # See https://github.com/python-distro/distro/issues/162 - return {} - - @staticmethod - def _parse_distro_release_content(line: str) -> Dict[str, str]: - """ - Parse a line from a distro release file. - - Parameters: - * line: Line from the distro release file. Must be a unicode string - or a UTF-8 encoded byte string. - - Returns: - A dictionary containing all information items. - """ - matches = _DISTRO_RELEASE_CONTENT_REVERSED_PATTERN.match(line.strip()[::-1]) - distro_info = {} - if matches: - # regexp ensures non-None - distro_info["name"] = matches.group(3)[::-1] - if matches.group(2): - distro_info["version_id"] = matches.group(2)[::-1] - if matches.group(1): - distro_info["codename"] = matches.group(1)[::-1] - elif line: - distro_info["name"] = line.strip() - return distro_info - - -_distro = LinuxDistribution() - - -def main() -> None: - logger = logging.getLogger(__name__) - logger.setLevel(logging.DEBUG) - logger.addHandler(logging.StreamHandler(sys.stdout)) - - parser = argparse.ArgumentParser(description="OS distro info tool") - parser.add_argument( - "--json", "-j", help="Output in machine readable format", action="store_true" - ) - - parser.add_argument( - "--root-dir", - "-r", - type=str, - dest="root_dir", - help="Path to the root filesystem directory (defaults to /)", - ) - - args = parser.parse_args() - - if args.root_dir: - dist = LinuxDistribution( - include_lsb=False, - include_uname=False, - include_oslevel=False, - root_dir=args.root_dir, - ) - else: - dist = _distro - - if args.json: - logger.info(json.dumps(dist.info(), indent=4, sort_keys=True)) - else: - logger.info("Name: %s", dist.name(pretty=True)) - distribution_version = dist.version(pretty=True) - logger.info("Version: %s", distribution_version) - distribution_codename = dist.codename() - logger.info("Codename: %s", distribution_codename) - - -if __name__ == "__main__": - main() diff --git a/contrib/python/distro/py3/distro/py.typed b/contrib/python/distro/py3/distro/py.typed deleted file mode 100644 index e69de29bb2..0000000000 --- a/contrib/python/distro/py3/distro/py.typed +++ /dev/null diff --git a/contrib/python/distro/py3/ya.make b/contrib/python/distro/py3/ya.make deleted file mode 100644 index 465f72ce57..0000000000 --- a/contrib/python/distro/py3/ya.make +++ /dev/null @@ -1,34 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY3_LIBRARY() - -VERSION(1.8.0) - -LICENSE(Apache-2.0) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - distro/__init__.py - distro/__main__.py - distro/distro.py -) - -RESOURCE_FILES( - PREFIX contrib/python/distro/py3/ - .dist-info/METADATA - .dist-info/entry_points.txt - .dist-info/top_level.txt - distro/py.typed -) - -END() - -RECURSE( - bin -) - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/distro/ya.make b/contrib/python/distro/ya.make deleted file mode 100644 index 82a91a17a2..0000000000 --- a/contrib/python/distro/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/distro/py2) -ELSE() - PEERDIR(contrib/python/distro/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/portalocker/py2/.dist-info/METADATA b/contrib/python/portalocker/py2/.dist-info/METADATA deleted file mode 100644 index d01a6203e0..0000000000 --- a/contrib/python/portalocker/py2/.dist-info/METADATA +++ /dev/null @@ -1,136 +0,0 @@ -Metadata-Version: 2.1 -Name: portalocker -Version: 1.7.1 -Summary: Wraps the portalocker recipe for easy usage -Home-page: https://github.com/WoLpH/portalocker -Author: Rick van Hattem -Author-email: wolph@wol.ph -License: PSF -Keywords: locking,locks,with statement,windows,linux,unix -Platform: any -Classifier: Intended Audience :: Developers -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3.3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Requires-Dist: pywin32 (!=226) ; platform_system == "Windows" -Provides-Extra: docs -Requires-Dist: sphinx (>=1.7.1) ; extra == 'docs' -Provides-Extra: tests -Requires-Dist: pytest (>=4.6.9) ; extra == 'tests' -Requires-Dist: pytest-cov (>=2.8.1) ; extra == 'tests' -Requires-Dist: sphinx (>=1.8.5) ; extra == 'tests' -Requires-Dist: pytest-flake8 (>=1.0.5) ; extra == 'tests' - -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://travis-ci.org/WoLpH/portalocker.svg?branch=master - :alt: Linux Test Status - :target: https://travis-ci.org/WoLpH/portalocker - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: - print >>fh, 'writing some stuff to my cache...' - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LOCK_EX) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking might not be needed in all cases: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -But can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - -Changelog ---------- - -See the `changelog <http://portalocker.readthedocs.io/en/latest/changelog.html>`_ page. - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - - - diff --git a/contrib/python/portalocker/py2/.dist-info/top_level.txt b/contrib/python/portalocker/py2/.dist-info/top_level.txt deleted file mode 100644 index 7bbc14e6fa..0000000000 --- a/contrib/python/portalocker/py2/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -portalocker diff --git a/contrib/python/portalocker/py2/LICENSE b/contrib/python/portalocker/py2/LICENSE deleted file mode 100644 index adb8038169..0000000000 --- a/contrib/python/portalocker/py2/LICENSE +++ /dev/null @@ -1,48 +0,0 @@ -PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 --------------------------------------------- - -1. This LICENSE AGREEMENT is between the Python Software Foundation -("PSF"), and the Individual or Organization ("Licensee") accessing and -otherwise using this software ("Python") in source or binary form and -its associated documentation. - -2. Subject to the terms and conditions of this License Agreement, PSF hereby -grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, -analyze, test, perform and/or display publicly, prepare derivative works, -distribute, and otherwise use Python alone or in any derivative version, -provided, however, that PSF's License Agreement and PSF's notice of copyright, -i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 -Python Software Foundation; All Rights Reserved" are retained in Python alone or -in any derivative version prepared by Licensee. - -3. In the event Licensee prepares a derivative work that is based on -or incorporates Python or any part thereof, and wants to make -the derivative work available to others as provided herein, then -Licensee hereby agrees to include in any such work a brief summary of -the changes made to Python. - -4. PSF is making Python available to Licensee on an "AS IS" -basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR -IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND -DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS -FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT -INFRINGE ANY THIRD PARTY RIGHTS. - -5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON -FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS -A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, -OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. - -6. This License Agreement will automatically terminate upon a material -breach of its terms and conditions. - -7. Nothing in this License Agreement shall be deemed to create any -relationship of agency, partnership, or joint venture between PSF and -Licensee. This License Agreement does not grant permission to use PSF -trademarks or trade name in a trademark sense to endorse or promote -products or services of Licensee, or any third party. - -8. By copying, installing or otherwise using Python, Licensee -agrees to be bound by the terms and conditions of this License -Agreement. - diff --git a/contrib/python/portalocker/py2/README.rst b/contrib/python/portalocker/py2/README.rst deleted file mode 100644 index c013490c76..0000000000 --- a/contrib/python/portalocker/py2/README.rst +++ /dev/null @@ -1,106 +0,0 @@ -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://travis-ci.org/WoLpH/portalocker.svg?branch=master - :alt: Linux Test Status - :target: https://travis-ci.org/WoLpH/portalocker - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: - print >>fh, 'writing some stuff to my cache...' - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LOCK_EX) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking might not be needed in all cases: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -But can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - -Changelog ---------- - -See the `changelog <http://portalocker.readthedocs.io/en/latest/changelog.html>`_ page. - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - diff --git a/contrib/python/portalocker/py2/portalocker/__about__.py b/contrib/python/portalocker/py2/portalocker/__about__.py deleted file mode 100644 index f16fe0cdf7..0000000000 --- a/contrib/python/portalocker/py2/portalocker/__about__.py +++ /dev/null @@ -1,7 +0,0 @@ -__package_name__ = 'portalocker' -__author__ = 'Rick van Hattem' -__email__ = 'wolph@wol.ph' -__version__ = '1.7.1' -__description__ = '''Wraps the portalocker recipe for easy usage''' -__url__ = 'https://github.com/WoLpH/portalocker' - diff --git a/contrib/python/portalocker/py2/portalocker/__init__.py b/contrib/python/portalocker/py2/portalocker/__init__.py deleted file mode 100644 index 9bf27fee0f..0000000000 --- a/contrib/python/portalocker/py2/portalocker/__init__.py +++ /dev/null @@ -1,67 +0,0 @@ -from . import __about__ -from . import constants -from . import exceptions -from . import portalocker -from . import utils - -#: The package name on Pypi -__package_name__ = __about__.__package_name__ -#: Current author and maintainer, view the git history for the previous ones -__author__ = __about__.__author__ -#: Current author's email address -__email__ = __about__.__email__ -#: Version number -__version__ = '1.7.1' -#: Package description for Pypi -__description__ = __about__.__description__ -#: Package homepage -__url__ = __about__.__url__ - - -#: Exception thrown when the file is already locked by someone else -AlreadyLocked = exceptions.AlreadyLocked -#: Exception thrown if an error occurred during locking -LockException = exceptions.LockException - - -#: Lock a file. Note that this is an advisory lock on Linux/Unix systems -lock = portalocker.lock -#: Unlock a file -unlock = portalocker.unlock - -#: Place an exclusive lock. -#: Only one process may hold an exclusive lock for a given file at a given -#: time. -LOCK_EX = constants.LOCK_EX - -#: Place a shared lock. -#: More than one process may hold a shared lock for a given file at a given -#: time. -LOCK_SH = constants.LOCK_SH - -#: Acquire the lock in a non-blocking fashion. -LOCK_NB = constants.LOCK_NB - -#: Remove an existing lock held by this process. -LOCK_UN = constants.LOCK_UN - -#: Locking utility class to automatically handle opening with timeouts and -#: context wrappers -Lock = utils.Lock -RLock = utils.RLock -TemporaryFileLock = utils.TemporaryFileLock -open_atomic = utils.open_atomic - -__all__ = [ - 'lock', - 'unlock', - 'LOCK_EX', - 'LOCK_SH', - 'LOCK_NB', - 'LOCK_UN', - 'LockException', - 'Lock', - 'AlreadyLocked', - 'open_atomic', -] - diff --git a/contrib/python/portalocker/py2/portalocker/constants.py b/contrib/python/portalocker/py2/portalocker/constants.py deleted file mode 100644 index fb0927e2da..0000000000 --- a/contrib/python/portalocker/py2/portalocker/constants.py +++ /dev/null @@ -1,39 +0,0 @@ -''' -Locking constants - -Lock types: - -- `LOCK_EX` exclusive lock -- `LOCK_SH` shared lock - -Lock flags: - -- `LOCK_NB` non-blocking - -Manually unlock, only needed internally - -- `LOCK_UN` unlock -''' -import os - -# The actual tests will execute the code anyhow so the following code can -# safely be ignored from the coverage tests -if os.name == 'nt': # pragma: no cover - import msvcrt - - LOCK_EX = 0x1 #: exclusive lock - LOCK_SH = 0x2 #: shared lock - LOCK_NB = 0x4 #: non-blocking - LOCK_UN = msvcrt.LK_UNLCK #: unlock - -elif os.name == 'posix': # pragma: no cover - import fcntl - - LOCK_EX = fcntl.LOCK_EX #: exclusive lock - LOCK_SH = fcntl.LOCK_SH #: shared lock - LOCK_NB = fcntl.LOCK_NB #: non-blocking - LOCK_UN = fcntl.LOCK_UN #: unlock - -else: # pragma: no cover - raise RuntimeError('PortaLocker only defined for nt and posix platforms') - diff --git a/contrib/python/portalocker/py2/portalocker/exceptions.py b/contrib/python/portalocker/py2/portalocker/exceptions.py deleted file mode 100644 index bb2b35eb7b..0000000000 --- a/contrib/python/portalocker/py2/portalocker/exceptions.py +++ /dev/null @@ -1,19 +0,0 @@ -class BaseLockException(Exception): - # Error codes: - LOCK_FAILED = 1 - - def __init__(self, *args, **kwargs): - self.fh = kwargs.pop('fh', None) - Exception.__init__(self, *args, **kwargs) - - -class LockException(BaseLockException): - pass - - -class AlreadyLocked(BaseLockException): - pass - - -class FileToLarge(BaseLockException): - pass diff --git a/contrib/python/portalocker/py2/portalocker/portalocker.py b/contrib/python/portalocker/py2/portalocker/portalocker.py deleted file mode 100644 index 460cf06e47..0000000000 --- a/contrib/python/portalocker/py2/portalocker/portalocker.py +++ /dev/null @@ -1,148 +0,0 @@ -import os -import sys -from . import exceptions -from . import constants - - -if os.name == 'nt': # pragma: no cover - import win32con - import win32file - import pywintypes - import winerror - import msvcrt - __overlapped = pywintypes.OVERLAPPED() - - if sys.version_info.major == 2: - lock_length = -1 - else: - lock_length = int(2**31 - 1) - - def lock(file_, flags): - if flags & constants.LOCK_SH: - if sys.version_info.major == 2: - if flags & constants.LOCK_NB: - mode = win32con.LOCKFILE_FAIL_IMMEDIATELY - else: - mode = 0 - - else: - if flags & constants.LOCK_NB: - mode = msvcrt.LK_NBRLCK - else: - mode = msvcrt.LK_RLCK - - # is there any reason not to reuse the following structure? - hfile = win32file._get_osfhandle(file_.fileno()) - try: - win32file.LockFileEx(hfile, mode, 0, -0x10000, __overlapped) - except pywintypes.error as exc_value: - # error: (33, 'LockFileEx', 'The process cannot access the file - # because another process has locked a portion of the file.') - if exc_value.winerror == winerror.ERROR_LOCK_VIOLATION: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_) - else: - # Q: Are there exceptions/codes we should be dealing with - # here? - raise - else: - mode = win32con.LOCKFILE_EXCLUSIVE_LOCK - if flags & constants.LOCK_NB: - mode |= win32con.LOCKFILE_FAIL_IMMEDIATELY - - if flags & constants.LOCK_NB: - mode = msvcrt.LK_NBLCK - else: - mode = msvcrt.LK_LOCK - - # windows locks byte ranges, so make sure to lock from file start - try: - savepos = file_.tell() - if savepos: - # [ ] test exclusive lock fails on seek here - # [ ] test if shared lock passes this point - file_.seek(0) - # [x] check if 0 param locks entire file (not documented in - # Python) - # [x] fails with "IOError: [Errno 13] Permission denied", - # but -1 seems to do the trick - - try: - msvcrt.locking(file_.fileno(), mode, lock_length) - except IOError as exc_value: - # [ ] be more specific here - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_) - finally: - if savepos: - file_.seek(savepos) - except IOError as exc_value: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, exc_value.strerror, - fh=file_) - - def unlock(file_): - try: - savepos = file_.tell() - if savepos: - file_.seek(0) - - try: - msvcrt.locking(file_.fileno(), constants.LOCK_UN, lock_length) - except IOError as exc_value: - if exc_value.strerror == 'Permission denied': - hfile = win32file._get_osfhandle(file_.fileno()) - try: - win32file.UnlockFileEx( - hfile, 0, -0x10000, __overlapped) - except pywintypes.error as exc_value: - if exc_value.winerror == winerror.ERROR_NOT_LOCKED: - # error: (158, 'UnlockFileEx', - # 'The segment is already unlocked.') - # To match the 'posix' implementation, silently - # ignore this error - pass - else: - # Q: Are there exceptions/codes we should be - # dealing with here? - raise - else: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_) - finally: - if savepos: - file_.seek(savepos) - except IOError as exc_value: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, exc_value.strerror, - fh=file_) - -elif os.name == 'posix': # pragma: no cover - import fcntl - - def lock(file_, flags): - locking_exceptions = IOError, - try: # pragma: no cover - locking_exceptions += BlockingIOError, - except NameError: # pragma: no cover - pass - - try: - fcntl.flock(file_.fileno(), flags) - except locking_exceptions as exc_value: - # The exception code varies on different systems so we'll catch - # every IO error - raise exceptions.LockException(exc_value, fh=file_) - - def unlock(file_): - fcntl.flock(file_.fileno(), constants.LOCK_UN) - -else: # pragma: no cover - raise RuntimeError('PortaLocker only defined for nt and posix platforms') - diff --git a/contrib/python/portalocker/py2/portalocker/utils.py b/contrib/python/portalocker/py2/portalocker/utils.py deleted file mode 100644 index 8baebc2b20..0000000000 --- a/contrib/python/portalocker/py2/portalocker/utils.py +++ /dev/null @@ -1,256 +0,0 @@ -import os -import time -import atexit -import tempfile -import contextlib -from . import exceptions -from . import constants -from . import portalocker - -current_time = getattr(time, "monotonic", time.time) - -DEFAULT_TIMEOUT = 5 -DEFAULT_CHECK_INTERVAL = 0.25 -LOCK_METHOD = constants.LOCK_EX | constants.LOCK_NB - -__all__ = [ - 'Lock', - 'open_atomic', -] - - -@contextlib.contextmanager -def open_atomic(filename, binary=True): - '''Open a file for atomic writing. Instead of locking this method allows - you to write the entire file and move it to the actual location. Note that - this makes the assumption that a rename is atomic on your platform which - is generally the case but not a guarantee. - - http://docs.python.org/library/os.html#os.rename - - >>> filename = 'test_file.txt' - >>> if os.path.exists(filename): - ... os.remove(filename) - - >>> with open_atomic(filename) as fh: - ... written = fh.write(b'test') - >>> assert os.path.exists(filename) - >>> os.remove(filename) - - ''' - assert not os.path.exists(filename), '%r exists' % filename - path, name = os.path.split(filename) - - # Create the parent directory if it doesn't exist - if path and not os.path.isdir(path): # pragma: no cover - os.makedirs(path) - - temp_fh = tempfile.NamedTemporaryFile( - mode=binary and 'wb' or 'w', - dir=path, - delete=False, - ) - yield temp_fh - temp_fh.flush() - os.fsync(temp_fh.fileno()) - temp_fh.close() - try: - os.rename(temp_fh.name, filename) - finally: - try: - os.remove(temp_fh.name) - except Exception: - pass - - -class Lock(object): - - def __init__( - self, filename, mode='a', timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=False, - flags=LOCK_METHOD, **file_open_kwargs): - '''Lock manager with build-in timeout - - filename -- filename - mode -- the open mode, 'a' or 'ab' should be used for writing - truncate -- use truncate to emulate 'w' mode, None is disabled, 0 is - truncate to 0 bytes - timeout -- timeout when trying to acquire a lock - check_interval -- check interval while waiting - fail_when_locked -- after the initial lock failed, return an error - or lock the file - **file_open_kwargs -- The kwargs for the `open(...)` call - - fail_when_locked is useful when multiple threads/processes can race - when creating a file. If set to true than the system will wait till - the lock was acquired and then return an AlreadyLocked exception. - - Note that the file is opened first and locked later. So using 'w' as - mode will result in truncate _BEFORE_ the lock is checked. - ''' - - if 'w' in mode: - truncate = True - mode = mode.replace('w', 'a') - else: - truncate = False - - self.fh = None - self.filename = filename - self.mode = mode - self.truncate = truncate - self.timeout = timeout - self.check_interval = check_interval - self.fail_when_locked = fail_when_locked - self.flags = flags - self.file_open_kwargs = file_open_kwargs - - def acquire( - self, timeout=None, check_interval=None, fail_when_locked=None): - '''Acquire the locked filehandle''' - if timeout is None: - timeout = self.timeout - if timeout is None: - timeout = 0 - - if check_interval is None: - check_interval = self.check_interval - - if fail_when_locked is None: - fail_when_locked = self.fail_when_locked - - # If we already have a filehandle, return it - fh = self.fh - if fh: - return fh - - # Get a new filehandler - fh = self._get_fh() - try: - # Try to lock - fh = self._get_lock(fh) - except exceptions.LockException as exception: - # Try till the timeout has passed - timeoutend = current_time() + timeout - while timeoutend > current_time(): - # Wait a bit - time.sleep(check_interval) - - # Try again - try: - - # We already tried to the get the lock - # If fail_when_locked is true, then stop trying - if fail_when_locked: - raise exceptions.AlreadyLocked(exception) - - else: # pragma: no cover - # We've got the lock - fh = self._get_lock(fh) - break - - except exceptions.LockException: - pass - - else: - fh.close() - # We got a timeout... reraising - raise exceptions.LockException(exception) - - # Prepare the filehandle (truncate if needed) - fh = self._prepare_fh(fh) - - self.fh = fh - return fh - - def release(self): - '''Releases the currently locked file handle''' - if self.fh: - portalocker.unlock(self.fh) - self.fh.close() - self.fh = None - - def _get_fh(self): - '''Get a new filehandle''' - return open(self.filename, self.mode, **self.file_open_kwargs) - - def _get_lock(self, fh): - ''' - Try to lock the given filehandle - - returns LockException if it fails''' - portalocker.lock(fh, self.flags) - return fh - - def _prepare_fh(self, fh): - ''' - Prepare the filehandle for usage - - If truncate is a number, the file will be truncated to that amount of - bytes - ''' - if self.truncate: - fh.seek(0) - fh.truncate(0) - - return fh - - def __enter__(self): - return self.acquire() - - def __exit__(self, type_, value, tb): - self.release() - - def __delete__(self, instance): # pragma: no cover - instance.release() - - -class RLock(Lock): - """ - A reentrant lock, functions in a similar way to threading.RLock in that it - can be acquired multiple times. When the corresponding number of release() - calls are made the lock will finally release the underlying file lock. - """ - def __init__( - self, filename, mode='a', timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=False, - flags=LOCK_METHOD): - super(RLock, self).__init__(filename, mode, timeout, check_interval, - fail_when_locked, flags) - self._acquire_count = 0 - - def acquire( - self, timeout=None, check_interval=None, fail_when_locked=None): - if self._acquire_count >= 1: - fh = self.fh - else: - fh = super(RLock, self).acquire(timeout, check_interval, - fail_when_locked) - self._acquire_count += 1 - return fh - - def release(self): - if self._acquire_count == 0: - raise exceptions.LockException( - "Cannot release more times than acquired") - - if self._acquire_count == 1: - super(RLock, self).release() - self._acquire_count -= 1 - - -class TemporaryFileLock(Lock): - - def __init__(self, filename='.lock', timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=True, - flags=LOCK_METHOD): - - Lock.__init__(self, filename=filename, mode='w', timeout=timeout, - check_interval=check_interval, - fail_when_locked=fail_when_locked, flags=flags) - atexit.register(self.release) - - def release(self): - Lock.release(self) - if os.path.isfile(self.filename): # pragma: no branch - os.unlink(self.filename) diff --git a/contrib/python/portalocker/py2/ya.make b/contrib/python/portalocker/py2/ya.make deleted file mode 100644 index 0cf6513704..0000000000 --- a/contrib/python/portalocker/py2/ya.make +++ /dev/null @@ -1,31 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(1.7.1) - -LICENSE(PSF-2.0) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - portalocker/__about__.py - portalocker/__init__.py - portalocker/constants.py - portalocker/exceptions.py - portalocker/portalocker.py - portalocker/utils.py -) - -RESOURCE_FILES( - PREFIX contrib/python/portalocker/py2/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() - -RECURSE( - tests -) diff --git a/contrib/python/portalocker/py3/.dist-info/METADATA b/contrib/python/portalocker/py3/.dist-info/METADATA deleted file mode 100644 index fceba37925..0000000000 --- a/contrib/python/portalocker/py3/.dist-info/METADATA +++ /dev/null @@ -1,255 +0,0 @@ -Metadata-Version: 2.1 -Name: portalocker -Version: 2.8.2 -Summary: Wraps the portalocker recipe for easy usage -Author-email: Rick van Hattem <wolph@wol.ph> -License: BSD-3-Clause -Project-URL: bugs, https://github.com/wolph/portalocker/issues -Project-URL: documentation, https://portalocker.readthedocs.io/en/latest/ -Project-URL: repository, https://github.com/wolph/portalocker/ -Keywords: locking,locks,with,statement,windows,linux,unix -Platform: any -Classifier: Development Status :: 5 - Production/Stable -Classifier: Development Status :: 6 - Mature -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: BSD License -Classifier: Natural Language :: English -Classifier: Operating System :: MacOS :: MacOS X -Classifier: Operating System :: MacOS -Classifier: Operating System :: Microsoft :: MS-DOS -Classifier: Operating System :: Microsoft :: Windows -Classifier: Operating System :: Microsoft -Classifier: Operating System :: POSIX :: BSD :: FreeBSD -Classifier: Operating System :: POSIX :: BSD -Classifier: Operating System :: POSIX :: Linux -Classifier: Operating System :: POSIX :: SunOS/Solaris -Classifier: Operating System :: POSIX -Classifier: Operating System :: Unix -Classifier: Programming Language :: Python :: 3 :: Only -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Programming Language :: Python :: 3.11 -Classifier: Programming Language :: Python :: 3.12 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: IronPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Classifier: Programming Language :: Python :: Implementation -Classifier: Programming Language :: Python -Classifier: Topic :: Education :: Testing -Classifier: Topic :: Office/Business -Classifier: Topic :: Other/Nonlisted Topic -Classifier: Topic :: Software Development :: Libraries :: Python Modules -Classifier: Topic :: Software Development :: Libraries -Classifier: Topic :: System :: Monitoring -Requires-Python: >=3.8 -Description-Content-Type: text/x-rst -License-File: LICENSE -Requires-Dist: pywin32 >=226 ; platform_system == "Windows" -Provides-Extra: docs -Requires-Dist: sphinx >=1.7.1 ; extra == 'docs' -Provides-Extra: redis -Requires-Dist: redis ; extra == 'redis' -Provides-Extra: tests -Requires-Dist: pytest >=5.4.1 ; extra == 'tests' -Requires-Dist: pytest-cov >=2.8.1 ; extra == 'tests' -Requires-Dist: pytest-timeout >=2.1.0 ; extra == 'tests' -Requires-Dist: sphinx >=6.0.0 ; extra == 'tests' -Requires-Dist: pytest-mypy >=0.8.0 ; extra == 'tests' -Requires-Dist: types-redis ; extra == 'tests' -Requires-Dist: redis ; extra == 'tests' - -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://github.com/WoLpH/portalocker/actions/workflows/python-package.yml/badge.svg?branch=master - :alt: Linux Test Status - :target: https://github.com/WoLpH/portalocker/actions/ - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Security contact information ------------------------------------------------------------------------------- - -To report a security vulnerability, please use the -`Tidelift security contact <https://tidelift.com/security>`_. -Tidelift will coordinate the fix and disclosure. - -Redis Locks ------------ - -This library now features a lock based on Redis which allows for locks across -multiple threads, processes and even distributed locks across multiple -computers. - -It is an extremely reliable Redis lock that is based on pubsub. - -As opposed to most Redis locking systems based on key/value pairs, -this locking method is based on the pubsub system. The big advantage is -that if the connection gets killed due to network issues, crashing -processes or otherwise, it will still immediately unlock instead of -waiting for a lock timeout. - -First make sure you have everything installed correctly: - -:: - - pip install "portalocker[redis]" - -Usage is really easy: - -:: - - import portalocker - - lock = portalocker.RedisLock('some_lock_channel_name') - - with lock: - print('do something here') - -The API is essentially identical to the other ``Lock`` classes so in addition -to the ``with`` statement you can also use ``lock.acquire(...)``. - -Python 2 --------- - -Python 2 was supported in versions before Portalocker 2.0. If you are still -using -Python 2, -you can run this to install: - -:: - - pip install "portalocker<2" - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: -... print('writing some stuff to my cache...', file=fh) - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LockFlags.EXCLUSIVE) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking is not needed in most cases but omitting it has been known -to cause issues: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -If needed, it can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -To create a cross platform bounded semaphore across multiple processes you can -use the `BoundedSemaphore` class which functions somewhat similar to -`threading.BoundedSemaphore`: - ->>> import portalocker ->>> n = 2 ->>> timeout = 0.1 - ->>> semaphore_a = portalocker.BoundedSemaphore(n, timeout=timeout) ->>> semaphore_b = portalocker.BoundedSemaphore(n, timeout=timeout) ->>> semaphore_c = portalocker.BoundedSemaphore(n, timeout=timeout) - ->>> semaphore_a.acquire() -<portalocker.utils.Lock object at ...> ->>> semaphore_b.acquire() -<portalocker.utils.Lock object at ...> ->>> semaphore_c.acquire() -Traceback (most recent call last): - ... -portalocker.exceptions.AlreadyLocked - - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - - -Versioning ----------- - -This library follows `Semantic Versioning <http://semver.org/>`_. - - -Changelog ---------- - -Every release has a ``git tag`` with a commit message for the tag -explaining what was added and/or changed. The list of tags/releases -including the commit messages can be found here: -https://github.com/WoLpH/portalocker/releases - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - diff --git a/contrib/python/portalocker/py3/.dist-info/top_level.txt b/contrib/python/portalocker/py3/.dist-info/top_level.txt deleted file mode 100644 index 7bbc14e6fa..0000000000 --- a/contrib/python/portalocker/py3/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -portalocker diff --git a/contrib/python/portalocker/py3/LICENSE b/contrib/python/portalocker/py3/LICENSE deleted file mode 100644 index b638bda0d3..0000000000 --- a/contrib/python/portalocker/py3/LICENSE +++ /dev/null @@ -1,11 +0,0 @@ -Copyright 2022 Rick van Hattem - -Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - -1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - -2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - -3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/contrib/python/portalocker/py3/README.rst b/contrib/python/portalocker/py3/README.rst deleted file mode 100644 index c5ef42f614..0000000000 --- a/contrib/python/portalocker/py3/README.rst +++ /dev/null @@ -1,193 +0,0 @@ -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://github.com/WoLpH/portalocker/actions/workflows/python-package.yml/badge.svg?branch=master - :alt: Linux Test Status - :target: https://github.com/WoLpH/portalocker/actions/ - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Security contact information ------------------------------------------------------------------------------- - -To report a security vulnerability, please use the -`Tidelift security contact <https://tidelift.com/security>`_. -Tidelift will coordinate the fix and disclosure. - -Redis Locks ------------ - -This library now features a lock based on Redis which allows for locks across -multiple threads, processes and even distributed locks across multiple -computers. - -It is an extremely reliable Redis lock that is based on pubsub. - -As opposed to most Redis locking systems based on key/value pairs, -this locking method is based on the pubsub system. The big advantage is -that if the connection gets killed due to network issues, crashing -processes or otherwise, it will still immediately unlock instead of -waiting for a lock timeout. - -First make sure you have everything installed correctly: - -:: - - pip install "portalocker[redis]" - -Usage is really easy: - -:: - - import portalocker - - lock = portalocker.RedisLock('some_lock_channel_name') - - with lock: - print('do something here') - -The API is essentially identical to the other ``Lock`` classes so in addition -to the ``with`` statement you can also use ``lock.acquire(...)``. - -Python 2 --------- - -Python 2 was supported in versions before Portalocker 2.0. If you are still -using -Python 2, -you can run this to install: - -:: - - pip install "portalocker<2" - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: -... print('writing some stuff to my cache...', file=fh) - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LockFlags.EXCLUSIVE) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking is not needed in most cases but omitting it has been known -to cause issues: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -If needed, it can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -To create a cross platform bounded semaphore across multiple processes you can -use the `BoundedSemaphore` class which functions somewhat similar to -`threading.BoundedSemaphore`: - ->>> import portalocker ->>> n = 2 ->>> timeout = 0.1 - ->>> semaphore_a = portalocker.BoundedSemaphore(n, timeout=timeout) ->>> semaphore_b = portalocker.BoundedSemaphore(n, timeout=timeout) ->>> semaphore_c = portalocker.BoundedSemaphore(n, timeout=timeout) - ->>> semaphore_a.acquire() -<portalocker.utils.Lock object at ...> ->>> semaphore_b.acquire() -<portalocker.utils.Lock object at ...> ->>> semaphore_c.acquire() -Traceback (most recent call last): - ... -portalocker.exceptions.AlreadyLocked - - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - - -Versioning ----------- - -This library follows `Semantic Versioning <http://semver.org/>`_. - - -Changelog ---------- - -Every release has a ``git tag`` with a commit message for the tag -explaining what was added and/or changed. The list of tags/releases -including the commit messages can be found here: -https://github.com/WoLpH/portalocker/releases - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - diff --git a/contrib/python/portalocker/py3/portalocker/__about__.py b/contrib/python/portalocker/py3/portalocker/__about__.py deleted file mode 100644 index e45c44327d..0000000000 --- a/contrib/python/portalocker/py3/portalocker/__about__.py +++ /dev/null @@ -1,6 +0,0 @@ -__package_name__ = 'portalocker' -__author__ = 'Rick van Hattem' -__email__ = 'wolph@wol.ph' -__version__ = '2.8.2' -__description__ = '''Wraps the portalocker recipe for easy usage''' -__url__ = 'https://github.com/WoLpH/portalocker' diff --git a/contrib/python/portalocker/py3/portalocker/__init__.py b/contrib/python/portalocker/py3/portalocker/__init__.py deleted file mode 100644 index 9170e33e9c..0000000000 --- a/contrib/python/portalocker/py3/portalocker/__init__.py +++ /dev/null @@ -1,76 +0,0 @@ -from . import __about__, constants, exceptions, portalocker, utils - -try: # pragma: no cover - from .redis import RedisLock -except ImportError: # pragma: no cover - RedisLock = None # type: ignore - - -#: The package name on Pypi -__package_name__ = __about__.__package_name__ -#: Current author and maintainer, view the git history for the previous ones -__author__ = __about__.__author__ -#: Current author's email address -__email__ = __about__.__email__ -#: Version number -__version__ = '2.8.2' -#: Package description for Pypi -__description__ = __about__.__description__ -#: Package homepage -__url__ = __about__.__url__ - - -#: Exception thrown when the file is already locked by someone else -AlreadyLocked = exceptions.AlreadyLocked -#: Exception thrown if an error occurred during locking -LockException = exceptions.LockException - - -#: Lock a file. Note that this is an advisory lock on Linux/Unix systems -lock = portalocker.lock -#: Unlock a file -unlock = portalocker.unlock - -#: Place an exclusive lock. -#: Only one process may hold an exclusive lock for a given file at a given -#: time. -LOCK_EX: constants.LockFlags = constants.LockFlags.EXCLUSIVE - -#: Place a shared lock. -#: More than one process may hold a shared lock for a given file at a given -#: time. -LOCK_SH: constants.LockFlags = constants.LockFlags.SHARED - -#: Acquire the lock in a non-blocking fashion. -LOCK_NB: constants.LockFlags = constants.LockFlags.NON_BLOCKING - -#: Remove an existing lock held by this process. -LOCK_UN: constants.LockFlags = constants.LockFlags.UNBLOCK - -#: Locking flags enum -LockFlags = constants.LockFlags - -#: Locking utility class to automatically handle opening with timeouts and -#: context wrappers -Lock = utils.Lock -RLock = utils.RLock -BoundedSemaphore = utils.BoundedSemaphore -TemporaryFileLock = utils.TemporaryFileLock -open_atomic = utils.open_atomic - -__all__ = [ - 'lock', - 'unlock', - 'LOCK_EX', - 'LOCK_SH', - 'LOCK_NB', - 'LOCK_UN', - 'LockFlags', - 'LockException', - 'Lock', - 'RLock', - 'AlreadyLocked', - 'BoundedSemaphore', - 'open_atomic', - 'RedisLock', -] diff --git a/contrib/python/portalocker/py3/portalocker/__main__.py b/contrib/python/portalocker/py3/portalocker/__main__.py deleted file mode 100644 index 658a3ec300..0000000000 --- a/contrib/python/portalocker/py3/portalocker/__main__.py +++ /dev/null @@ -1,98 +0,0 @@ -import argparse -import logging -import os -import pathlib -import re - -base_path = pathlib.Path(__file__).parent.parent -src_path = base_path / 'portalocker' -dist_path = base_path / 'dist' -_default_output_path = base_path / 'dist' / 'portalocker.py' - -_RELATIVE_IMPORT_RE = re.compile(r'^from \. import (?P<names>.+)$') -_USELESS_ASSIGNMENT_RE = re.compile(r'^(?P<name>\w+) = \1\n$') - -_TEXT_TEMPLATE = """''' -{} -''' - -""" - -logger = logging.getLogger(__name__) - - -def main(argv=None): - parser = argparse.ArgumentParser() - - subparsers = parser.add_subparsers(required=True) - combine_parser = subparsers.add_parser( - 'combine', - help='Combine all Python files into a single unified `portalocker.py` ' - 'file for easy distribution', - ) - combine_parser.add_argument( - '--output-file', - '-o', - type=argparse.FileType('w'), - default=str(_default_output_path), - ) - - combine_parser.set_defaults(func=combine) - args = parser.parse_args(argv) - args.func(args) - - -def _read_file(path, seen_files): - if path in seen_files: - return - - names = set() - seen_files.add(path) - for line in path.open(): - if match := _RELATIVE_IMPORT_RE.match(line): - for name in match.group('names').split(','): - name = name.strip() - names.add(name) - yield from _read_file(src_path / f'{name}.py', seen_files) - else: - yield _clean_line(line, names) - - -def _clean_line(line, names): - # Replace `some_import.spam` with `spam` - if names: - joined_names = '|'.join(names) - line = re.sub(fr'\b({joined_names})\.', '', line) - - # Replace useless assignments (e.g. `spam = spam`) - return _USELESS_ASSIGNMENT_RE.sub('', line) - - -def combine(args): - output_file = args.output_file - pathlib.Path(output_file.name).parent.mkdir(parents=True, exist_ok=True) - - output_file.write( - _TEXT_TEMPLATE.format((base_path / 'README.rst').read_text()), - ) - output_file.write( - _TEXT_TEMPLATE.format((base_path / 'LICENSE').read_text()), - ) - - seen_files = set() - for line in _read_file(src_path / '__init__.py', seen_files): - output_file.write(line) - - output_file.flush() - output_file.close() - - logger.info(f'Wrote combined file to {output_file.name}') - # Run black and ruff if available. If not then just run the file. - os.system(f'black {output_file.name}') - os.system(f'ruff --fix {output_file.name}') - os.system(f'python3 {output_file.name}') - - -if __name__ == '__main__': - logging.basicConfig(level=logging.INFO) - main() diff --git a/contrib/python/portalocker/py3/portalocker/constants.py b/contrib/python/portalocker/py3/portalocker/constants.py deleted file mode 100644 index 72733c8546..0000000000 --- a/contrib/python/portalocker/py3/portalocker/constants.py +++ /dev/null @@ -1,58 +0,0 @@ -''' -Locking constants - -Lock types: - -- `EXCLUSIVE` exclusive lock -- `SHARED` shared lock - -Lock flags: - -- `NON_BLOCKING` non-blocking - -Manually unlock, only needed internally - -- `UNBLOCK` unlock -''' -import enum -import os - -# The actual tests will execute the code anyhow so the following code can -# safely be ignored from the coverage tests -if os.name == 'nt': # pragma: no cover - import msvcrt - - #: exclusive lock - LOCK_EX = 0x1 - #: shared lock - LOCK_SH = 0x2 - #: non-blocking - LOCK_NB = 0x4 - #: unlock - LOCK_UN = msvcrt.LK_UNLCK # type: ignore - -elif os.name == 'posix': # pragma: no cover - import fcntl - - #: exclusive lock - LOCK_EX = fcntl.LOCK_EX - #: shared lock - LOCK_SH = fcntl.LOCK_SH - #: non-blocking - LOCK_NB = fcntl.LOCK_NB - #: unlock - LOCK_UN = fcntl.LOCK_UN - -else: # pragma: no cover - raise RuntimeError('PortaLocker only defined for nt and posix platforms') - - -class LockFlags(enum.IntFlag): - #: exclusive lock - EXCLUSIVE = LOCK_EX - #: shared lock - SHARED = LOCK_SH - #: non-blocking - NON_BLOCKING = LOCK_NB - #: unlock - UNBLOCK = LOCK_UN diff --git a/contrib/python/portalocker/py3/portalocker/exceptions.py b/contrib/python/portalocker/py3/portalocker/exceptions.py deleted file mode 100644 index e871d13acb..0000000000 --- a/contrib/python/portalocker/py3/portalocker/exceptions.py +++ /dev/null @@ -1,27 +0,0 @@ -import typing - - -class BaseLockException(Exception): # noqa: N818 - # Error codes: - LOCK_FAILED = 1 - - def __init__( - self, - *args: typing.Any, - fh: typing.Union[typing.IO, None, int] = None, - **kwargs: typing.Any, - ) -> None: - self.fh = fh - Exception.__init__(self, *args) - - -class LockException(BaseLockException): - pass - - -class AlreadyLocked(LockException): - pass - - -class FileToLarge(LockException): - pass diff --git a/contrib/python/portalocker/py3/portalocker/portalocker.py b/contrib/python/portalocker/py3/portalocker/portalocker.py deleted file mode 100644 index 90307b76ea..0000000000 --- a/contrib/python/portalocker/py3/portalocker/portalocker.py +++ /dev/null @@ -1,117 +0,0 @@ -import contextlib -import os -import typing - -from . import constants, exceptions - -# Alias for readability. Due to import recursion issues we cannot do: -# from .constants import LockFlags -LockFlags = constants.LockFlags - - -if os.name == 'nt': # pragma: no cover - import msvcrt - - import pywintypes - import win32con - import win32file - import winerror - - __overlapped = pywintypes.OVERLAPPED() - - def lock(file_: typing.Union[typing.IO, int], flags: LockFlags): - # Windows locking does not support locking through `fh.fileno()` so - # we cast it to make mypy and pyright happy - file_ = typing.cast(typing.IO, file_) - - mode = 0 - if flags & LockFlags.NON_BLOCKING: - mode |= win32con.LOCKFILE_FAIL_IMMEDIATELY - - if flags & LockFlags.EXCLUSIVE: - mode |= win32con.LOCKFILE_EXCLUSIVE_LOCK - - # Save the old position so we can go back to that position but - # still lock from the beginning of the file - savepos = file_.tell() - if savepos: - file_.seek(0) - - os_fh = msvcrt.get_osfhandle(file_.fileno()) # type: ignore - try: - win32file.LockFileEx(os_fh, mode, 0, -0x10000, __overlapped) - except pywintypes.error as exc_value: - # error: (33, 'LockFileEx', 'The process cannot access the file - # because another process has locked a portion of the file.') - if exc_value.winerror == winerror.ERROR_LOCK_VIOLATION: - raise exceptions.AlreadyLocked( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_, - ) from exc_value - else: - # Q: Are there exceptions/codes we should be dealing with - # here? - raise - finally: - if savepos: - file_.seek(savepos) - - def unlock(file_: typing.IO): - try: - savepos = file_.tell() - if savepos: - file_.seek(0) - - os_fh = msvcrt.get_osfhandle(file_.fileno()) # type: ignore - try: - win32file.UnlockFileEx( - os_fh, - 0, - -0x10000, - __overlapped, - ) - except pywintypes.error as exc: - if exc.winerror != winerror.ERROR_NOT_LOCKED: - # Q: Are there exceptions/codes we should be - # dealing with here? - raise - finally: - if savepos: - file_.seek(savepos) - except OSError as exc: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc.strerror, - fh=file_, - ) from exc - -elif os.name == 'posix': # pragma: no cover - import fcntl - - def lock(file_: typing.Union[typing.IO, int], flags: LockFlags): - locking_exceptions = (IOError,) - with contextlib.suppress(NameError): - locking_exceptions += (BlockingIOError,) # type: ignore - # Locking with NON_BLOCKING without EXCLUSIVE or SHARED enabled results - # in an error - if (flags & LockFlags.NON_BLOCKING) and not flags & ( - LockFlags.SHARED | LockFlags.EXCLUSIVE - ): - raise RuntimeError( - 'When locking in non-blocking mode the SHARED ' - 'or EXCLUSIVE flag must be specified as well', - ) - - try: - fcntl.flock(file_, flags) - except locking_exceptions as exc_value: - # The exception code varies on different systems so we'll catch - # every IO error - raise exceptions.LockException(exc_value, fh=file_) from exc_value - - def unlock(file_: typing.IO): - fcntl.flock(file_.fileno(), LockFlags.UNBLOCK) - -else: # pragma: no cover - raise RuntimeError('PortaLocker only defined for nt and posix platforms') diff --git a/contrib/python/portalocker/py3/portalocker/py.typed b/contrib/python/portalocker/py3/portalocker/py.typed deleted file mode 100644 index e69de29bb2..0000000000 --- a/contrib/python/portalocker/py3/portalocker/py.typed +++ /dev/null diff --git a/contrib/python/portalocker/py3/portalocker/redis.py b/contrib/python/portalocker/py3/portalocker/redis.py deleted file mode 100644 index 59ee5ff171..0000000000 --- a/contrib/python/portalocker/py3/portalocker/redis.py +++ /dev/null @@ -1,236 +0,0 @@ -import _thread -import json -import logging -import random -import time -import typing - -from redis import client - -from . import exceptions, utils - -logger = logging.getLogger(__name__) - -DEFAULT_UNAVAILABLE_TIMEOUT = 1 -DEFAULT_THREAD_SLEEP_TIME = 0.1 - - -class PubSubWorkerThread(client.PubSubWorkerThread): # type: ignore - def run(self): - try: - super().run() - except Exception: # pragma: no cover - _thread.interrupt_main() - raise - - -class RedisLock(utils.LockBase): - ''' - An extremely reliable Redis lock based on pubsub with a keep-alive thread - - As opposed to most Redis locking systems based on key/value pairs, - this locking method is based on the pubsub system. The big advantage is - that if the connection gets killed due to network issues, crashing - processes or otherwise, it will still immediately unlock instead of - waiting for a lock timeout. - - To make sure both sides of the lock know about the connection state it is - recommended to set the `health_check_interval` when creating the redis - connection.. - - Args: - channel: the redis channel to use as locking key. - connection: an optional redis connection if you already have one - or if you need to specify the redis connection - timeout: timeout when trying to acquire a lock - check_interval: check interval while waiting - fail_when_locked: after the initial lock failed, return an error - or lock the file. This does not wait for the timeout. - thread_sleep_time: sleep time between fetching messages from redis to - prevent a busy/wait loop. In the case of lock conflicts this - increases the time it takes to resolve the conflict. This should - be smaller than the `check_interval` to be useful. - unavailable_timeout: If the conflicting lock is properly connected - this should never exceed twice your redis latency. Note that this - will increase the wait time possibly beyond your `timeout` and is - always executed if a conflict arises. - redis_kwargs: The redis connection arguments if no connection is - given. The `DEFAULT_REDIS_KWARGS` are used as default, if you want - to override these you need to explicitly specify a value (e.g. - `health_check_interval=0`) - - ''' - - redis_kwargs: typing.Dict[str, typing.Any] - thread: typing.Optional[PubSubWorkerThread] - channel: str - timeout: float - connection: typing.Optional[client.Redis] - pubsub: typing.Optional[client.PubSub] = None - close_connection: bool - - DEFAULT_REDIS_KWARGS: typing.ClassVar[typing.Dict[str, typing.Any]] = dict( - health_check_interval=10, - ) - - def __init__( - self, - channel: str, - connection: typing.Optional[client.Redis] = None, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = False, - thread_sleep_time: float = DEFAULT_THREAD_SLEEP_TIME, - unavailable_timeout: float = DEFAULT_UNAVAILABLE_TIMEOUT, - redis_kwargs: typing.Optional[typing.Dict] = None, - ): - # We don't want to close connections given as an argument - self.close_connection = not connection - - self.thread = None - self.channel = channel - self.connection = connection - self.thread_sleep_time = thread_sleep_time - self.unavailable_timeout = unavailable_timeout - self.redis_kwargs = redis_kwargs or dict() - - for key, value in self.DEFAULT_REDIS_KWARGS.items(): - self.redis_kwargs.setdefault(key, value) - - super().__init__( - timeout=timeout, - check_interval=check_interval, - fail_when_locked=fail_when_locked, - ) - - def get_connection(self) -> client.Redis: - if not self.connection: - self.connection = client.Redis(**self.redis_kwargs) - - return self.connection - - def channel_handler(self, message): - if message.get('type') != 'message': # pragma: no cover - return - - try: - data = json.loads(message.get('data')) - except TypeError: # pragma: no cover - logger.debug('TypeError while parsing: %r', message) - return - - assert self.connection is not None - self.connection.publish(data['response_channel'], str(time.time())) - - @property - def client_name(self): - return f'{self.channel}-lock' - - def acquire( - self, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = None, - ): - timeout = utils.coalesce(timeout, self.timeout, 0.0) - check_interval = utils.coalesce( - check_interval, - self.check_interval, - 0.0, - ) - fail_when_locked = utils.coalesce( - fail_when_locked, - self.fail_when_locked, - ) - - assert not self.pubsub, 'This lock is already active' - connection = self.get_connection() - - timeout_generator = self._timeout_generator(timeout, check_interval) - for _ in timeout_generator: # pragma: no branch - subscribers = connection.pubsub_numsub(self.channel)[0][1] - - if subscribers: - logger.debug( - 'Found %d lock subscribers for %s', - subscribers, - self.channel, - ) - - if self.check_or_kill_lock( - connection, - self.unavailable_timeout, - ): # pragma: no branch - continue - else: # pragma: no cover - subscribers = 0 - - # Note: this should not be changed to an elif because the if - # above can still end up here - if not subscribers: - connection.client_setname(self.client_name) - self.pubsub = connection.pubsub() - self.pubsub.subscribe(**{self.channel: self.channel_handler}) - self.thread = PubSubWorkerThread( - self.pubsub, - sleep_time=self.thread_sleep_time, - ) - self.thread.start() - - subscribers = connection.pubsub_numsub(self.channel)[0][1] - if subscribers == 1: # pragma: no branch - return self - else: # pragma: no cover - # Race condition, let's try again - self.release() - - if fail_when_locked: # pragma: no cover - raise exceptions.AlreadyLocked(exceptions) - - raise exceptions.AlreadyLocked(exceptions) - - def check_or_kill_lock(self, connection, timeout): - # Random channel name to get messages back from the lock - response_channel = f'{self.channel}-{random.random()}' - - pubsub = connection.pubsub() - pubsub.subscribe(response_channel) - connection.publish( - self.channel, - json.dumps( - dict( - response_channel=response_channel, - message='ping', - ), - ), - ) - - check_interval = min(self.thread_sleep_time, timeout / 10) - for _ in self._timeout_generator( - timeout, - check_interval, - ): # pragma: no branch - if pubsub.get_message(timeout=check_interval): - pubsub.close() - return True - - for client_ in connection.client_list('pubsub'): # pragma: no cover - if client_.get('name') == self.client_name: - logger.warning('Killing unavailable redis client: %r', client_) - connection.client_kill_filter(client_.get('id')) - return None - - def release(self): - if self.thread: # pragma: no branch - self.thread.stop() - self.thread.join() - self.thread = None - time.sleep(0.01) - - if self.pubsub: # pragma: no branch - self.pubsub.unsubscribe(self.channel) - self.pubsub.close() - self.pubsub = None - - def __del__(self): - self.release() diff --git a/contrib/python/portalocker/py3/portalocker/utils.py b/contrib/python/portalocker/py3/portalocker/utils.py deleted file mode 100644 index 3b5682e055..0000000000 --- a/contrib/python/portalocker/py3/portalocker/utils.py +++ /dev/null @@ -1,563 +0,0 @@ -import abc -import atexit -import contextlib -import logging -import os -import pathlib -import random -import tempfile -import time -import typing -import warnings - -from . import constants, exceptions, portalocker - -logger = logging.getLogger(__name__) - -DEFAULT_TIMEOUT = 5 -DEFAULT_CHECK_INTERVAL = 0.25 -DEFAULT_FAIL_WHEN_LOCKED = False -LOCK_METHOD = constants.LockFlags.EXCLUSIVE | constants.LockFlags.NON_BLOCKING - -__all__ = [ - 'Lock', - 'open_atomic', -] - -Filename = typing.Union[str, pathlib.Path] - - -def coalesce(*args: typing.Any, test_value: typing.Any = None) -> typing.Any: - '''Simple coalescing function that returns the first value that is not - equal to the `test_value`. Or `None` if no value is valid. Usually this - means that the last given value is the default value. - - Note that the `test_value` is compared using an identity check - (i.e. `value is not test_value`) so changing the `test_value` won't work - for all values. - - >>> coalesce(None, 1) - 1 - >>> coalesce() - - >>> coalesce(0, False, True) - 0 - >>> coalesce(0, False, True, test_value=0) - False - - # This won't work because of the `is not test_value` type testing: - >>> coalesce([], dict(spam='eggs'), test_value=[]) - [] - ''' - return next((arg for arg in args if arg is not test_value), None) - - -@contextlib.contextmanager -def open_atomic( - filename: Filename, - binary: bool = True, -) -> typing.Iterator[typing.IO]: - '''Open a file for atomic writing. Instead of locking this method allows - you to write the entire file and move it to the actual location. Note that - this makes the assumption that a rename is atomic on your platform which - is generally the case but not a guarantee. - - http://docs.python.org/library/os.html#os.rename - - >>> filename = 'test_file.txt' - >>> if os.path.exists(filename): - ... os.remove(filename) - - >>> with open_atomic(filename) as fh: - ... written = fh.write(b'test') - >>> assert os.path.exists(filename) - >>> os.remove(filename) - - >>> import pathlib - >>> path_filename = pathlib.Path('test_file.txt') - - >>> with open_atomic(path_filename) as fh: - ... written = fh.write(b'test') - >>> assert path_filename.exists() - >>> path_filename.unlink() - ''' - # `pathlib.Path` cast in case `path` is a `str` - path: pathlib.Path = pathlib.Path(filename) - - assert not path.exists(), '%r exists' % path - - # Create the parent directory if it doesn't exist - path.parent.mkdir(parents=True, exist_ok=True) - - temp_fh = tempfile.NamedTemporaryFile( - mode=binary and 'wb' or 'w', - dir=str(path.parent), - delete=False, - ) - yield temp_fh - temp_fh.flush() - os.fsync(temp_fh.fileno()) - temp_fh.close() - try: - os.rename(temp_fh.name, path) - finally: - with contextlib.suppress(Exception): - os.remove(temp_fh.name) - - -class LockBase(abc.ABC): # pragma: no cover - #: timeout when trying to acquire a lock - timeout: float - #: check interval while waiting for `timeout` - check_interval: float - #: skip the timeout and immediately fail if the initial lock fails - fail_when_locked: bool - - def __init__( - self, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = None, - ): - self.timeout = coalesce(timeout, DEFAULT_TIMEOUT) - self.check_interval = coalesce(check_interval, DEFAULT_CHECK_INTERVAL) - self.fail_when_locked = coalesce( - fail_when_locked, - DEFAULT_FAIL_WHEN_LOCKED, - ) - - @abc.abstractmethod - def acquire( - self, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = None, - ): - return NotImplemented - - def _timeout_generator( - self, - timeout: typing.Optional[float], - check_interval: typing.Optional[float], - ) -> typing.Iterator[int]: - f_timeout = coalesce(timeout, self.timeout, 0.0) - f_check_interval = coalesce(check_interval, self.check_interval, 0.0) - - yield 0 - i = 0 - - start_time = time.perf_counter() - while start_time + f_timeout > time.perf_counter(): - i += 1 - yield i - - # Take low lock checks into account to stay within the interval - since_start_time = time.perf_counter() - start_time - time.sleep(max(0.001, (i * f_check_interval) - since_start_time)) - - @abc.abstractmethod - def release(self): - return NotImplemented - - def __enter__(self): - return self.acquire() - - def __exit__( - self, - exc_type: typing.Optional[typing.Type[BaseException]], - exc_value: typing.Optional[BaseException], - traceback: typing.Any, # Should be typing.TracebackType - ) -> typing.Optional[bool]: - self.release() - return None - - def __delete__(self, instance): - instance.release() - - -class Lock(LockBase): - '''Lock manager with built-in timeout - - Args: - filename: filename - mode: the open mode, 'a' or 'ab' should be used for writing. When mode - contains `w` the file will be truncated to 0 bytes. - timeout: timeout when trying to acquire a lock - check_interval: check interval while waiting - fail_when_locked: after the initial lock failed, return an error - or lock the file. This does not wait for the timeout. - **file_open_kwargs: The kwargs for the `open(...)` call - - fail_when_locked is useful when multiple threads/processes can race - when creating a file. If set to true than the system will wait till - the lock was acquired and then return an AlreadyLocked exception. - - Note that the file is opened first and locked later. So using 'w' as - mode will result in truncate _BEFORE_ the lock is checked. - ''' - - def __init__( - self, - filename: Filename, - mode: str = 'a', - timeout: typing.Optional[float] = None, - check_interval: float = DEFAULT_CHECK_INTERVAL, - fail_when_locked: bool = DEFAULT_FAIL_WHEN_LOCKED, - flags: constants.LockFlags = LOCK_METHOD, - **file_open_kwargs, - ): - if 'w' in mode: - truncate = True - mode = mode.replace('w', 'a') - else: - truncate = False - - if timeout is None: - timeout = DEFAULT_TIMEOUT - elif not (flags & constants.LockFlags.NON_BLOCKING): - warnings.warn( - 'timeout has no effect in blocking mode', - stacklevel=1, - ) - - self.fh: typing.Optional[typing.IO] = None - self.filename: str = str(filename) - self.mode: str = mode - self.truncate: bool = truncate - self.timeout: float = timeout - self.check_interval: float = check_interval - self.fail_when_locked: bool = fail_when_locked - self.flags: constants.LockFlags = flags - self.file_open_kwargs = file_open_kwargs - - def acquire( - self, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = None, - ) -> typing.IO: - '''Acquire the locked filehandle''' - - fail_when_locked = coalesce(fail_when_locked, self.fail_when_locked) - - if ( - not (self.flags & constants.LockFlags.NON_BLOCKING) - and timeout is not None - ): - warnings.warn( - 'timeout has no effect in blocking mode', - stacklevel=1, - ) - - # If we already have a filehandle, return it - fh: typing.Optional[typing.IO] = self.fh - if fh: - return fh - - # Get a new filehandler - fh = self._get_fh() - - def try_close(): # pragma: no cover - # Silently try to close the handle if possible, ignore all issues - if fh is not None: - with contextlib.suppress(Exception): - fh.close() - - exception = None - # Try till the timeout has passed - for _ in self._timeout_generator(timeout, check_interval): - exception = None - try: - # Try to lock - fh = self._get_lock(fh) - break - except exceptions.LockException as exc: - # Python will automatically remove the variable from memory - # unless you save it in a different location - exception = exc - - # We already tried to the get the lock - # If fail_when_locked is True, stop trying - if fail_when_locked: - try_close() - raise exceptions.AlreadyLocked(exception) from exc - - # Wait a bit - - if exception: - try_close() - # We got a timeout... reraising - raise exceptions.LockException(exception) - - # Prepare the filehandle (truncate if needed) - fh = self._prepare_fh(fh) - - self.fh = fh - return fh - - def release(self): - '''Releases the currently locked file handle''' - if self.fh: - portalocker.unlock(self.fh) - self.fh.close() - self.fh = None - - def _get_fh(self) -> typing.IO: - '''Get a new filehandle''' - return open( # noqa: SIM115 - self.filename, - self.mode, - **self.file_open_kwargs, - ) - - def _get_lock(self, fh: typing.IO) -> typing.IO: - ''' - Try to lock the given filehandle - - returns LockException if it fails''' - portalocker.lock(fh, self.flags) - return fh - - def _prepare_fh(self, fh: typing.IO) -> typing.IO: - ''' - Prepare the filehandle for usage - - If truncate is a number, the file will be truncated to that amount of - bytes - ''' - if self.truncate: - fh.seek(0) - fh.truncate(0) - - return fh - - -class RLock(Lock): - ''' - A reentrant lock, functions in a similar way to threading.RLock in that it - can be acquired multiple times. When the corresponding number of release() - calls are made the lock will finally release the underlying file lock. - ''' - - def __init__( - self, - filename, - mode='a', - timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, - fail_when_locked=False, - flags=LOCK_METHOD, - ): - super().__init__( - filename, - mode, - timeout, - check_interval, - fail_when_locked, - flags, - ) - self._acquire_count = 0 - - def acquire( - self, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = None, - ) -> typing.IO: - if self._acquire_count >= 1: - fh = self.fh - else: - fh = super().acquire(timeout, check_interval, fail_when_locked) - self._acquire_count += 1 - assert fh - return fh - - def release(self): - if self._acquire_count == 0: - raise exceptions.LockException( - 'Cannot release more times than acquired', - ) - - if self._acquire_count == 1: - super().release() - self._acquire_count -= 1 - - -class TemporaryFileLock(Lock): - def __init__( - self, - filename='.lock', - timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, - fail_when_locked=True, - flags=LOCK_METHOD, - ): - Lock.__init__( - self, - filename=filename, - mode='w', - timeout=timeout, - check_interval=check_interval, - fail_when_locked=fail_when_locked, - flags=flags, - ) - atexit.register(self.release) - - def release(self): - Lock.release(self) - if os.path.isfile(self.filename): # pragma: no branch - os.unlink(self.filename) - - -class BoundedSemaphore(LockBase): - ''' - Bounded semaphore to prevent too many parallel processes from running - - This method is deprecated because multiple processes that are completely - unrelated could end up using the same semaphore. To prevent this, - use `NamedBoundedSemaphore` instead. The - `NamedBoundedSemaphore` is a drop-in replacement for this class. - - >>> semaphore = BoundedSemaphore(2, directory='') - >>> str(semaphore.get_filenames()[0]) - 'bounded_semaphore.00.lock' - >>> str(sorted(semaphore.get_random_filenames())[1]) - 'bounded_semaphore.01.lock' - ''' - - lock: typing.Optional[Lock] - - def __init__( - self, - maximum: int, - name: str = 'bounded_semaphore', - filename_pattern: str = '{name}.{number:02d}.lock', - directory: str = tempfile.gettempdir(), - timeout: typing.Optional[float] = DEFAULT_TIMEOUT, - check_interval: typing.Optional[float] = DEFAULT_CHECK_INTERVAL, - fail_when_locked: typing.Optional[bool] = True, - ): - self.maximum = maximum - self.name = name - self.filename_pattern = filename_pattern - self.directory = directory - self.lock: typing.Optional[Lock] = None - super().__init__( - timeout=timeout, - check_interval=check_interval, - fail_when_locked=fail_when_locked, - ) - - if not name or name == 'bounded_semaphore': - warnings.warn( - '`BoundedSemaphore` without an explicit `name` ' - 'argument is deprecated, use NamedBoundedSemaphore', - DeprecationWarning, - stacklevel=1, - ) - - def get_filenames(self) -> typing.Sequence[pathlib.Path]: - return [self.get_filename(n) for n in range(self.maximum)] - - def get_random_filenames(self) -> typing.Sequence[pathlib.Path]: - filenames = list(self.get_filenames()) - random.shuffle(filenames) - return filenames - - def get_filename(self, number) -> pathlib.Path: - return pathlib.Path(self.directory) / self.filename_pattern.format( - name=self.name, - number=number, - ) - - def acquire( - self, - timeout: typing.Optional[float] = None, - check_interval: typing.Optional[float] = None, - fail_when_locked: typing.Optional[bool] = None, - ) -> typing.Optional[Lock]: - assert not self.lock, 'Already locked' - - filenames = self.get_filenames() - - for n in self._timeout_generator(timeout, check_interval): # pragma: - logger.debug('trying lock (attempt %d) %r', n, filenames) - # no branch - if self.try_lock(filenames): # pragma: no branch - return self.lock # pragma: no cover - - if fail_when_locked := coalesce( - fail_when_locked, - self.fail_when_locked, - ): - raise exceptions.AlreadyLocked() - - return None - - def try_lock(self, filenames: typing.Sequence[Filename]) -> bool: - filename: Filename - for filename in filenames: - logger.debug('trying lock for %r', filename) - self.lock = Lock(filename, fail_when_locked=True) - try: - self.lock.acquire() - except exceptions.AlreadyLocked: - self.lock = None - else: - logger.debug('locked %r', filename) - return True - - return False - - def release(self): # pragma: no cover - if self.lock is not None: - self.lock.release() - self.lock = None - - -class NamedBoundedSemaphore(BoundedSemaphore): - ''' - Bounded semaphore to prevent too many parallel processes from running - - It's also possible to specify a timeout when acquiring the lock to wait - for a resource to become available. This is very similar to - `threading.BoundedSemaphore` but works across multiple processes and across - multiple operating systems. - - Because this works across multiple processes it's important to give the - semaphore a name. This name is used to create the lock files. If you - don't specify a name, a random name will be generated. This means that - you can't use the same semaphore in multiple processes unless you pass the - semaphore object to the other processes. - - >>> semaphore = NamedBoundedSemaphore(2, name='test') - >>> str(semaphore.get_filenames()[0]) - '...test.00.lock' - - >>> semaphore = NamedBoundedSemaphore(2) - >>> 'bounded_semaphore' in str(semaphore.get_filenames()[0]) - True - - ''' - - def __init__( - self, - maximum: int, - name: typing.Optional[str] = None, - filename_pattern: str = '{name}.{number:02d}.lock', - directory: str = tempfile.gettempdir(), - timeout: typing.Optional[float] = DEFAULT_TIMEOUT, - check_interval: typing.Optional[float] = DEFAULT_CHECK_INTERVAL, - fail_when_locked: typing.Optional[bool] = True, - ): - if name is None: - name = 'bounded_semaphore.%d' % random.randint(0, 1000000) - super().__init__( - maximum, - name, - filename_pattern, - directory, - timeout, - check_interval, - fail_when_locked, - ) diff --git a/contrib/python/portalocker/py3/ya.make b/contrib/python/portalocker/py3/ya.make deleted file mode 100644 index 8bef0a2613..0000000000 --- a/contrib/python/portalocker/py3/ya.make +++ /dev/null @@ -1,38 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY3_LIBRARY() - -VERSION(2.8.2) - -LICENSE(BSD-3-Clause) - -NO_LINT() - -NO_CHECK_IMPORTS( - portalocker.redis -) - -PY_SRCS( - TOP_LEVEL - portalocker/__about__.py - portalocker/__init__.py - portalocker/__main__.py - portalocker/constants.py - portalocker/exceptions.py - portalocker/portalocker.py - portalocker/redis.py - portalocker/utils.py -) - -RESOURCE_FILES( - PREFIX contrib/python/portalocker/py3/ - .dist-info/METADATA - .dist-info/top_level.txt - portalocker/py.typed -) - -END() - -RECURSE( - tests -) diff --git a/contrib/python/portalocker/ya.make b/contrib/python/portalocker/ya.make deleted file mode 100644 index 4012370b1c..0000000000 --- a/contrib/python/portalocker/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/portalocker/py2) -ELSE() - PEERDIR(contrib/python/portalocker/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/pylev/py2/.dist-info/METADATA b/contrib/python/pylev/py2/.dist-info/METADATA deleted file mode 100644 index 0fe37efb0b..0000000000 --- a/contrib/python/pylev/py2/.dist-info/METADATA +++ /dev/null @@ -1,106 +0,0 @@ -Metadata-Version: 2.1 -Name: pylev -Version: 1.4.0 -Summary: A pure Python Levenshtein implementation that's not freaking GPL'd. -Home-page: http://github.com/toastdriven/pylev -Author: Daniel Lindsley -Author-email: daniel@toastdriven.com -License: UNKNOWN -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: BSD License -Classifier: Operating System :: OS Independent -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 3 -License-File: LICENSE - -pylev -===== - -A pure Python Levenshtein implementation that's not freaking GPL'd. - -Based off the Wikipedia code samples at -http://en.wikipedia.org/wiki/Levenshtein_distance. - - -Requirements ------------- - -* Python 2.7.X, Python 3.3+ or PyPy 1.6.0+ - - -Usage ------ - -Usage is fairly straightforward: - -.. code-block:: python - - import pylev - distance = pylev.levenshtein('kitten', 'sitting') - assert distance == 3 - - -License -------- - -New BSD. - - -Tests ------ - -Setup:: - - $ git clone https://github.com/toastdriven/pylev.git - $ cd pylev - -Running:: - - $ python -m unittest tests - -.. image:: https://travis-ci.com/toastdriven/pylev.svg?branch=main - :target: http://travis-ci.com/toastdriven/pylev - - -Version History ---------------- - -* v1.4.0 - - * Updated for current versions of Python - * Integrated a better Travis matrix. Thanks to @grainert! - * Fixed mistaken docs about the `assert`. Thanks to @adamchainz! - * Reorganized the package. - * Blacked all the source code. - -* v1.3.0 - - * Implemented a considerably faster variants (orders of magnitude). - * Tested & working on Python 2.7.4, Python 3.3.1 & PyPy 1.9.0. - -* v1.2.0 - - * Fixed all incorrect spellings of "Levenshtein" (there's no "c" in it). - * Old methods are aliased for backward-compatibility. - -* v1.1.0 - - * Implemented a much faster variant (several orders of magnitude). - * The older variant was renamed to ``classic_levenschtein``. - * Tested & working on Python 3.3 & PyPy 1.6.0 as well. - -* v1.0.2 - - * Python packaging is **REALLY** hard. Including the README *this time*. - -* v1.0.1 - - * Python packaging is hard. Including the README this time. - -* v1.0.0 - - * Initial release, just the naive implementation of Levenshtein. - - diff --git a/contrib/python/pylev/py2/.dist-info/top_level.txt b/contrib/python/pylev/py2/.dist-info/top_level.txt deleted file mode 100644 index 53ea0c31a0..0000000000 --- a/contrib/python/pylev/py2/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -pylev diff --git a/contrib/python/pylev/py2/LICENSE b/contrib/python/pylev/py2/LICENSE deleted file mode 100644 index a049da3295..0000000000 --- a/contrib/python/pylev/py2/LICENSE +++ /dev/null @@ -1,25 +0,0 @@ -Copyright (c) 2012, Daniel Lindsley -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the pylev nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL pylev BE LIABLE FOR ANY -DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/contrib/python/pylev/py2/README.rst b/contrib/python/pylev/py2/README.rst deleted file mode 100644 index 5af9972a75..0000000000 --- a/contrib/python/pylev/py2/README.rst +++ /dev/null @@ -1,87 +0,0 @@ -pylev -===== - -A pure Python Levenshtein implementation that's not freaking GPL'd. - -Based off the Wikipedia code samples at -http://en.wikipedia.org/wiki/Levenshtein_distance. - - -Requirements ------------- - -* Python 2.7.X, Python 3.3+ or PyPy 1.6.0+ - - -Usage ------ - -Usage is fairly straightforward: - -.. code-block:: python - - import pylev - distance = pylev.levenshtein('kitten', 'sitting') - assert distance == 3 - - -License -------- - -New BSD. - - -Tests ------ - -Setup:: - - $ git clone https://github.com/toastdriven/pylev.git - $ cd pylev - -Running:: - - $ python -m unittest tests - -.. image:: https://travis-ci.com/toastdriven/pylev.svg?branch=main - :target: http://travis-ci.com/toastdriven/pylev - - -Version History ---------------- - -* v1.4.0 - - * Updated for current versions of Python - * Integrated a better Travis matrix. Thanks to @grainert! - * Fixed mistaken docs about the `assert`. Thanks to @adamchainz! - * Reorganized the package. - * Blacked all the source code. - -* v1.3.0 - - * Implemented a considerably faster variants (orders of magnitude). - * Tested & working on Python 2.7.4, Python 3.3.1 & PyPy 1.9.0. - -* v1.2.0 - - * Fixed all incorrect spellings of "Levenshtein" (there's no "c" in it). - * Old methods are aliased for backward-compatibility. - -* v1.1.0 - - * Implemented a much faster variant (several orders of magnitude). - * The older variant was renamed to ``classic_levenschtein``. - * Tested & working on Python 3.3 & PyPy 1.6.0 as well. - -* v1.0.2 - - * Python packaging is **REALLY** hard. Including the README *this time*. - -* v1.0.1 - - * Python packaging is hard. Including the README this time. - -* v1.0.0 - - * Initial release, just the naive implementation of Levenshtein. diff --git a/contrib/python/pylev/py2/pylev/__init__.py b/contrib/python/pylev/py2/pylev/__init__.py deleted file mode 100644 index a4d62dc6ad..0000000000 --- a/contrib/python/pylev/py2/pylev/__init__.py +++ /dev/null @@ -1,44 +0,0 @@ -""" -pylev -===== - -A pure Python Levenshtein implementation that's not freaking GPL'd. - -Based off the Wikipedia code samples at -http://en.wikipedia.org/wiki/Levenshtein_distance. - -Usage ------ - -Usage is fairly straightforward.:: - - import pylev - distance = pylev.levenshtein('kitten', 'sitting') - assert distance == 3 - -""" -from .classic import classic_levenshtein -from .recursive import recursive_levenshtein -from .wf import wf_levenshtein, wfi_levenshtein -from .damerau import damerau_levenshtein - -__author__ = "Daniel Lindsley" -__version__ = (1, 4, 0) -__license__ = "New BSD" - - -levenshtein = wfi_levenshtein - -# Backward-compatibilty because I misspelled. -classic_levenschtein = classic_levenshtein -levenschtein = levenshtein - - -__all__ = [ - "levenshtein", - "classic_levenshtein", - "recursive_levenshtein", - "wf_levenshtein", - "wfi_levenshtein", - "damerau_levenshtein", -] diff --git a/contrib/python/pylev/py2/pylev/classic.py b/contrib/python/pylev/py2/pylev/classic.py deleted file mode 100644 index 98954e6727..0000000000 --- a/contrib/python/pylev/py2/pylev/classic.py +++ /dev/null @@ -1,35 +0,0 @@ -def classic_levenshtein(string_1, string_2): - """ - Calculates the Levenshtein distance between two strings. - - This version is easier to read, but significantly slower than the version - below (up to several orders of magnitude). Useful for learning, less so - otherwise. - - Usage:: - - >>> classic_levenshtein('kitten', 'sitting') - 3 - >>> classic_levenshtein('kitten', 'kitten') - 0 - >>> classic_levenshtein('', '') - 0 - - """ - len_1 = len(string_1) - len_2 = len(string_2) - cost = 0 - - if len_1 and len_2 and string_1[0] != string_2[0]: - cost = 1 - - if len_1 == 0: - return len_2 - elif len_2 == 0: - return len_1 - else: - return min( - classic_levenshtein(string_1[1:], string_2) + 1, - classic_levenshtein(string_1, string_2[1:]) + 1, - classic_levenshtein(string_1[1:], string_2[1:]) + cost, - ) diff --git a/contrib/python/pylev/py2/pylev/damerau.py b/contrib/python/pylev/py2/pylev/damerau.py deleted file mode 100644 index 290212b094..0000000000 --- a/contrib/python/pylev/py2/pylev/damerau.py +++ /dev/null @@ -1,80 +0,0 @@ -import sys - - -PY2 = sys.version_info[0] == 2 - -if PY2: - range = xrange - - -def damerau_levenshtein(string_1, string_2): - """ - Calculates the Damerau-Levenshtein distance between two strings. - - In addition to insertions, deletions and substitutions, - Damerau-Levenshtein considers adjacent transpositions. - - This version is based on an iterative version of the Wagner-Fischer algorithm. - - Usage:: - - >>> damerau_levenshtein('kitten', 'sitting') - 3 - >>> damerau_levenshtein('kitten', 'kittne') - 1 - >>> damerau_levenshtein('', '') - 0 - - """ - if string_1 == string_2: - return 0 - - len_1 = len(string_1) - len_2 = len(string_2) - - if len_1 == 0: - return len_2 - if len_2 == 0: - return len_1 - - if len_1 > len_2: - string_2, string_1 = string_1, string_2 - len_2, len_1 = len_1, len_2 - - prev_cost = 0 - d0 = [i for i in range(len_2 + 1)] - d1 = [j for j in range(len_2 + 1)] - dprev = d0[:] - - s1 = string_1 - s2 = string_2 - - for i in range(len_1): - d1[0] = i + 1 - for j in range(len_2): - cost = d0[j] - - if s1[i] != s2[j]: - # substitution - cost += 1 - - # insertion - x_cost = d1[j] + 1 - if x_cost < cost: - cost = x_cost - - # deletion - y_cost = d0[j + 1] + 1 - if y_cost < cost: - cost = y_cost - - # transposition - if i > 0 and j > 0 and s1[i] == s2[j - 1] and s1[i - 1] == s2[j]: - transp_cost = dprev[j - 1] + 1 - if transp_cost < cost: - cost = transp_cost - d1[j + 1] = cost - - dprev, d0, d1 = d0, d1, dprev - - return d0[-1] diff --git a/contrib/python/pylev/py2/pylev/recursive.py b/contrib/python/pylev/py2/pylev/recursive.py deleted file mode 100644 index 6ec34f29da..0000000000 --- a/contrib/python/pylev/py2/pylev/recursive.py +++ /dev/null @@ -1,56 +0,0 @@ -def recursive_levenshtein( - string_1, string_2, len_1=None, len_2=None, offset_1=0, offset_2=0, memo=None -): - """ - Calculates the Levenshtein distance between two strings. - - Usage:: - - >>> recursive_levenshtein('kitten', 'sitting') - 3 - >>> recursive_levenshtein('kitten', 'kitten') - 0 - >>> recursive_levenshtein('', '') - 0 - - """ - if len_1 is None: - len_1 = len(string_1) - - if len_2 is None: - len_2 = len(string_2) - - if memo is None: - memo = {} - - key = ",".join([str(offset_1), str(len_1), str(offset_2), str(len_2)]) - - if memo.get(key) is not None: - return memo[key] - - if len_1 == 0: - return len_2 - elif len_2 == 0: - return len_1 - - cost = 0 - - if string_1[offset_1] != string_2[offset_2]: - cost = 1 - - dist = min( - recursive_levenshtein( - string_1, string_2, len_1 - 1, len_2, offset_1 + 1, offset_2, memo - ) - + 1, - recursive_levenshtein( - string_1, string_2, len_1, len_2 - 1, offset_1, offset_2 + 1, memo - ) - + 1, - recursive_levenshtein( - string_1, string_2, len_1 - 1, len_2 - 1, offset_1 + 1, offset_2 + 1, memo - ) - + cost, - ) - memo[key] = dist - return dist diff --git a/contrib/python/pylev/py2/pylev/wf.py b/contrib/python/pylev/py2/pylev/wf.py deleted file mode 100644 index b6b89249e9..0000000000 --- a/contrib/python/pylev/py2/pylev/wf.py +++ /dev/null @@ -1,107 +0,0 @@ -import sys - - -PY2 = sys.version_info[0] == 2 - -if PY2: - range = xrange - - -def wf_levenshtein(string_1, string_2): - """ - Calculates the Levenshtein distance between two strings. - - This version uses the Wagner-Fischer algorithm. - - Usage:: - - >>> wf_levenshtein('kitten', 'sitting') - 3 - >>> wf_levenshtein('kitten', 'kitten') - 0 - >>> wf_levenshtein('', '') - 0 - - """ - len_1 = len(string_1) + 1 - len_2 = len(string_2) + 1 - - d = [0] * (len_1 * len_2) - - for i in range(len_1): - d[i] = i - for j in range(len_2): - d[j * len_1] = j - - for j in range(1, len_2): - for i in range(1, len_1): - if string_1[i - 1] == string_2[j - 1]: - d[i + j * len_1] = d[i - 1 + (j - 1) * len_1] - else: - d[i + j * len_1] = min( - d[i - 1 + j * len_1] + 1, # deletion - d[i + (j - 1) * len_1] + 1, # insertion - d[i - 1 + (j - 1) * len_1] + 1, # substitution - ) - - return d[-1] - - -def wfi_levenshtein(string_1, string_2): - """ - Calculates the Levenshtein distance between two strings. - - This version uses an iterative version of the Wagner-Fischer algorithm. - - Usage:: - - >>> wfi_levenshtein('kitten', 'sitting') - 3 - >>> wfi_levenshtein('kitten', 'kitten') - 0 - >>> wfi_levenshtein('', '') - 0 - - """ - if string_1 == string_2: - return 0 - - len_1 = len(string_1) - len_2 = len(string_2) - - if len_1 == 0: - return len_2 - if len_2 == 0: - return len_1 - - if len_1 > len_2: - string_2, string_1 = string_1, string_2 - len_2, len_1 = len_1, len_2 - - d0 = [i for i in range(len_2 + 1)] - d1 = [j for j in range(len_2 + 1)] - - for i in range(len_1): - d1[0] = i + 1 - for j in range(len_2): - cost = d0[j] - - if string_1[i] != string_2[j]: - # substitution - cost += 1 - - # insertion - x_cost = d1[j] + 1 - if x_cost < cost: - cost = x_cost - - # deletion - y_cost = d0[j + 1] + 1 - if y_cost < cost: - cost = y_cost - - d1[j + 1] = cost - - d0, d1 = d1, d0 - - return d0[-1] diff --git a/contrib/python/pylev/py2/ya.make b/contrib/python/pylev/py2/ya.make deleted file mode 100644 index 40ec0bb21e..0000000000 --- a/contrib/python/pylev/py2/ya.make +++ /dev/null @@ -1,26 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(1.4.0) - -LICENSE(BSD-3-Clause) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - pylev/__init__.py - pylev/classic.py - pylev/damerau.py - pylev/recursive.py - pylev/wf.py -) - -RESOURCE_FILES( - PREFIX contrib/python/pylev/py2/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() diff --git a/contrib/python/pylev/py3/.dist-info/METADATA b/contrib/python/pylev/py3/.dist-info/METADATA deleted file mode 100644 index 0fe37efb0b..0000000000 --- a/contrib/python/pylev/py3/.dist-info/METADATA +++ /dev/null @@ -1,106 +0,0 @@ -Metadata-Version: 2.1 -Name: pylev -Version: 1.4.0 -Summary: A pure Python Levenshtein implementation that's not freaking GPL'd. -Home-page: http://github.com/toastdriven/pylev -Author: Daniel Lindsley -Author-email: daniel@toastdriven.com -License: UNKNOWN -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: BSD License -Classifier: Operating System :: OS Independent -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 3 -License-File: LICENSE - -pylev -===== - -A pure Python Levenshtein implementation that's not freaking GPL'd. - -Based off the Wikipedia code samples at -http://en.wikipedia.org/wiki/Levenshtein_distance. - - -Requirements ------------- - -* Python 2.7.X, Python 3.3+ or PyPy 1.6.0+ - - -Usage ------ - -Usage is fairly straightforward: - -.. code-block:: python - - import pylev - distance = pylev.levenshtein('kitten', 'sitting') - assert distance == 3 - - -License -------- - -New BSD. - - -Tests ------ - -Setup:: - - $ git clone https://github.com/toastdriven/pylev.git - $ cd pylev - -Running:: - - $ python -m unittest tests - -.. image:: https://travis-ci.com/toastdriven/pylev.svg?branch=main - :target: http://travis-ci.com/toastdriven/pylev - - -Version History ---------------- - -* v1.4.0 - - * Updated for current versions of Python - * Integrated a better Travis matrix. Thanks to @grainert! - * Fixed mistaken docs about the `assert`. Thanks to @adamchainz! - * Reorganized the package. - * Blacked all the source code. - -* v1.3.0 - - * Implemented a considerably faster variants (orders of magnitude). - * Tested & working on Python 2.7.4, Python 3.3.1 & PyPy 1.9.0. - -* v1.2.0 - - * Fixed all incorrect spellings of "Levenshtein" (there's no "c" in it). - * Old methods are aliased for backward-compatibility. - -* v1.1.0 - - * Implemented a much faster variant (several orders of magnitude). - * The older variant was renamed to ``classic_levenschtein``. - * Tested & working on Python 3.3 & PyPy 1.6.0 as well. - -* v1.0.2 - - * Python packaging is **REALLY** hard. Including the README *this time*. - -* v1.0.1 - - * Python packaging is hard. Including the README this time. - -* v1.0.0 - - * Initial release, just the naive implementation of Levenshtein. - - diff --git a/contrib/python/pylev/py3/.dist-info/top_level.txt b/contrib/python/pylev/py3/.dist-info/top_level.txt deleted file mode 100644 index 53ea0c31a0..0000000000 --- a/contrib/python/pylev/py3/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -pylev diff --git a/contrib/python/pylev/py3/LICENSE b/contrib/python/pylev/py3/LICENSE deleted file mode 100644 index a049da3295..0000000000 --- a/contrib/python/pylev/py3/LICENSE +++ /dev/null @@ -1,25 +0,0 @@ -Copyright (c) 2012, Daniel Lindsley -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the pylev nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL pylev BE LIABLE FOR ANY -DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/contrib/python/pylev/py3/README.rst b/contrib/python/pylev/py3/README.rst deleted file mode 100644 index 5af9972a75..0000000000 --- a/contrib/python/pylev/py3/README.rst +++ /dev/null @@ -1,87 +0,0 @@ -pylev -===== - -A pure Python Levenshtein implementation that's not freaking GPL'd. - -Based off the Wikipedia code samples at -http://en.wikipedia.org/wiki/Levenshtein_distance. - - -Requirements ------------- - -* Python 2.7.X, Python 3.3+ or PyPy 1.6.0+ - - -Usage ------ - -Usage is fairly straightforward: - -.. code-block:: python - - import pylev - distance = pylev.levenshtein('kitten', 'sitting') - assert distance == 3 - - -License -------- - -New BSD. - - -Tests ------ - -Setup:: - - $ git clone https://github.com/toastdriven/pylev.git - $ cd pylev - -Running:: - - $ python -m unittest tests - -.. image:: https://travis-ci.com/toastdriven/pylev.svg?branch=main - :target: http://travis-ci.com/toastdriven/pylev - - -Version History ---------------- - -* v1.4.0 - - * Updated for current versions of Python - * Integrated a better Travis matrix. Thanks to @grainert! - * Fixed mistaken docs about the `assert`. Thanks to @adamchainz! - * Reorganized the package. - * Blacked all the source code. - -* v1.3.0 - - * Implemented a considerably faster variants (orders of magnitude). - * Tested & working on Python 2.7.4, Python 3.3.1 & PyPy 1.9.0. - -* v1.2.0 - - * Fixed all incorrect spellings of "Levenshtein" (there's no "c" in it). - * Old methods are aliased for backward-compatibility. - -* v1.1.0 - - * Implemented a much faster variant (several orders of magnitude). - * The older variant was renamed to ``classic_levenschtein``. - * Tested & working on Python 3.3 & PyPy 1.6.0 as well. - -* v1.0.2 - - * Python packaging is **REALLY** hard. Including the README *this time*. - -* v1.0.1 - - * Python packaging is hard. Including the README this time. - -* v1.0.0 - - * Initial release, just the naive implementation of Levenshtein. diff --git a/contrib/python/pylev/py3/pylev/__init__.py b/contrib/python/pylev/py3/pylev/__init__.py deleted file mode 100644 index a4d62dc6ad..0000000000 --- a/contrib/python/pylev/py3/pylev/__init__.py +++ /dev/null @@ -1,44 +0,0 @@ -""" -pylev -===== - -A pure Python Levenshtein implementation that's not freaking GPL'd. - -Based off the Wikipedia code samples at -http://en.wikipedia.org/wiki/Levenshtein_distance. - -Usage ------ - -Usage is fairly straightforward.:: - - import pylev - distance = pylev.levenshtein('kitten', 'sitting') - assert distance == 3 - -""" -from .classic import classic_levenshtein -from .recursive import recursive_levenshtein -from .wf import wf_levenshtein, wfi_levenshtein -from .damerau import damerau_levenshtein - -__author__ = "Daniel Lindsley" -__version__ = (1, 4, 0) -__license__ = "New BSD" - - -levenshtein = wfi_levenshtein - -# Backward-compatibilty because I misspelled. -classic_levenschtein = classic_levenshtein -levenschtein = levenshtein - - -__all__ = [ - "levenshtein", - "classic_levenshtein", - "recursive_levenshtein", - "wf_levenshtein", - "wfi_levenshtein", - "damerau_levenshtein", -] diff --git a/contrib/python/pylev/py3/pylev/classic.py b/contrib/python/pylev/py3/pylev/classic.py deleted file mode 100644 index 98954e6727..0000000000 --- a/contrib/python/pylev/py3/pylev/classic.py +++ /dev/null @@ -1,35 +0,0 @@ -def classic_levenshtein(string_1, string_2): - """ - Calculates the Levenshtein distance between two strings. - - This version is easier to read, but significantly slower than the version - below (up to several orders of magnitude). Useful for learning, less so - otherwise. - - Usage:: - - >>> classic_levenshtein('kitten', 'sitting') - 3 - >>> classic_levenshtein('kitten', 'kitten') - 0 - >>> classic_levenshtein('', '') - 0 - - """ - len_1 = len(string_1) - len_2 = len(string_2) - cost = 0 - - if len_1 and len_2 and string_1[0] != string_2[0]: - cost = 1 - - if len_1 == 0: - return len_2 - elif len_2 == 0: - return len_1 - else: - return min( - classic_levenshtein(string_1[1:], string_2) + 1, - classic_levenshtein(string_1, string_2[1:]) + 1, - classic_levenshtein(string_1[1:], string_2[1:]) + cost, - ) diff --git a/contrib/python/pylev/py3/pylev/damerau.py b/contrib/python/pylev/py3/pylev/damerau.py deleted file mode 100644 index 290212b094..0000000000 --- a/contrib/python/pylev/py3/pylev/damerau.py +++ /dev/null @@ -1,80 +0,0 @@ -import sys - - -PY2 = sys.version_info[0] == 2 - -if PY2: - range = xrange - - -def damerau_levenshtein(string_1, string_2): - """ - Calculates the Damerau-Levenshtein distance between two strings. - - In addition to insertions, deletions and substitutions, - Damerau-Levenshtein considers adjacent transpositions. - - This version is based on an iterative version of the Wagner-Fischer algorithm. - - Usage:: - - >>> damerau_levenshtein('kitten', 'sitting') - 3 - >>> damerau_levenshtein('kitten', 'kittne') - 1 - >>> damerau_levenshtein('', '') - 0 - - """ - if string_1 == string_2: - return 0 - - len_1 = len(string_1) - len_2 = len(string_2) - - if len_1 == 0: - return len_2 - if len_2 == 0: - return len_1 - - if len_1 > len_2: - string_2, string_1 = string_1, string_2 - len_2, len_1 = len_1, len_2 - - prev_cost = 0 - d0 = [i for i in range(len_2 + 1)] - d1 = [j for j in range(len_2 + 1)] - dprev = d0[:] - - s1 = string_1 - s2 = string_2 - - for i in range(len_1): - d1[0] = i + 1 - for j in range(len_2): - cost = d0[j] - - if s1[i] != s2[j]: - # substitution - cost += 1 - - # insertion - x_cost = d1[j] + 1 - if x_cost < cost: - cost = x_cost - - # deletion - y_cost = d0[j + 1] + 1 - if y_cost < cost: - cost = y_cost - - # transposition - if i > 0 and j > 0 and s1[i] == s2[j - 1] and s1[i - 1] == s2[j]: - transp_cost = dprev[j - 1] + 1 - if transp_cost < cost: - cost = transp_cost - d1[j + 1] = cost - - dprev, d0, d1 = d0, d1, dprev - - return d0[-1] diff --git a/contrib/python/pylev/py3/pylev/recursive.py b/contrib/python/pylev/py3/pylev/recursive.py deleted file mode 100644 index 6ec34f29da..0000000000 --- a/contrib/python/pylev/py3/pylev/recursive.py +++ /dev/null @@ -1,56 +0,0 @@ -def recursive_levenshtein( - string_1, string_2, len_1=None, len_2=None, offset_1=0, offset_2=0, memo=None -): - """ - Calculates the Levenshtein distance between two strings. - - Usage:: - - >>> recursive_levenshtein('kitten', 'sitting') - 3 - >>> recursive_levenshtein('kitten', 'kitten') - 0 - >>> recursive_levenshtein('', '') - 0 - - """ - if len_1 is None: - len_1 = len(string_1) - - if len_2 is None: - len_2 = len(string_2) - - if memo is None: - memo = {} - - key = ",".join([str(offset_1), str(len_1), str(offset_2), str(len_2)]) - - if memo.get(key) is not None: - return memo[key] - - if len_1 == 0: - return len_2 - elif len_2 == 0: - return len_1 - - cost = 0 - - if string_1[offset_1] != string_2[offset_2]: - cost = 1 - - dist = min( - recursive_levenshtein( - string_1, string_2, len_1 - 1, len_2, offset_1 + 1, offset_2, memo - ) - + 1, - recursive_levenshtein( - string_1, string_2, len_1, len_2 - 1, offset_1, offset_2 + 1, memo - ) - + 1, - recursive_levenshtein( - string_1, string_2, len_1 - 1, len_2 - 1, offset_1 + 1, offset_2 + 1, memo - ) - + cost, - ) - memo[key] = dist - return dist diff --git a/contrib/python/pylev/py3/pylev/wf.py b/contrib/python/pylev/py3/pylev/wf.py deleted file mode 100644 index b6b89249e9..0000000000 --- a/contrib/python/pylev/py3/pylev/wf.py +++ /dev/null @@ -1,107 +0,0 @@ -import sys - - -PY2 = sys.version_info[0] == 2 - -if PY2: - range = xrange - - -def wf_levenshtein(string_1, string_2): - """ - Calculates the Levenshtein distance between two strings. - - This version uses the Wagner-Fischer algorithm. - - Usage:: - - >>> wf_levenshtein('kitten', 'sitting') - 3 - >>> wf_levenshtein('kitten', 'kitten') - 0 - >>> wf_levenshtein('', '') - 0 - - """ - len_1 = len(string_1) + 1 - len_2 = len(string_2) + 1 - - d = [0] * (len_1 * len_2) - - for i in range(len_1): - d[i] = i - for j in range(len_2): - d[j * len_1] = j - - for j in range(1, len_2): - for i in range(1, len_1): - if string_1[i - 1] == string_2[j - 1]: - d[i + j * len_1] = d[i - 1 + (j - 1) * len_1] - else: - d[i + j * len_1] = min( - d[i - 1 + j * len_1] + 1, # deletion - d[i + (j - 1) * len_1] + 1, # insertion - d[i - 1 + (j - 1) * len_1] + 1, # substitution - ) - - return d[-1] - - -def wfi_levenshtein(string_1, string_2): - """ - Calculates the Levenshtein distance between two strings. - - This version uses an iterative version of the Wagner-Fischer algorithm. - - Usage:: - - >>> wfi_levenshtein('kitten', 'sitting') - 3 - >>> wfi_levenshtein('kitten', 'kitten') - 0 - >>> wfi_levenshtein('', '') - 0 - - """ - if string_1 == string_2: - return 0 - - len_1 = len(string_1) - len_2 = len(string_2) - - if len_1 == 0: - return len_2 - if len_2 == 0: - return len_1 - - if len_1 > len_2: - string_2, string_1 = string_1, string_2 - len_2, len_1 = len_1, len_2 - - d0 = [i for i in range(len_2 + 1)] - d1 = [j for j in range(len_2 + 1)] - - for i in range(len_1): - d1[0] = i + 1 - for j in range(len_2): - cost = d0[j] - - if string_1[i] != string_2[j]: - # substitution - cost += 1 - - # insertion - x_cost = d1[j] + 1 - if x_cost < cost: - cost = x_cost - - # deletion - y_cost = d0[j + 1] + 1 - if y_cost < cost: - cost = y_cost - - d1[j + 1] = cost - - d0, d1 = d1, d0 - - return d0[-1] diff --git a/contrib/python/pylev/py3/ya.make b/contrib/python/pylev/py3/ya.make deleted file mode 100644 index 97cbb58969..0000000000 --- a/contrib/python/pylev/py3/ya.make +++ /dev/null @@ -1,26 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY3_LIBRARY() - -VERSION(1.4.0) - -LICENSE(BSD-3-Clause) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - pylev/__init__.py - pylev/classic.py - pylev/damerau.py - pylev/recursive.py - pylev/wf.py -) - -RESOURCE_FILES( - PREFIX contrib/python/pylev/py3/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() diff --git a/contrib/python/pylev/ya.make b/contrib/python/pylev/ya.make deleted file mode 100644 index ab0c0f77c6..0000000000 --- a/contrib/python/pylev/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/pylev/py2) -ELSE() - PEERDIR(contrib/python/pylev/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/python-libarchive/py2/libarchive/__init__.py b/contrib/python/python-libarchive/py2/libarchive/__init__.py deleted file mode 100644 index 0c0c63359a..0000000000 --- a/contrib/python/python-libarchive/py2/libarchive/__init__.py +++ /dev/null @@ -1,800 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import os -import stat -import sys -import math -import time -import logging -import warnings - -import contextlib2 - -from libarchive import _libarchive -import six - -logger = logging.getLogger(__name__) - -# Suggested block size for libarchive. Libarchive may adjust it. -BLOCK_SIZE = 10240 - -MTIME_FORMAT = '' - -# Default encoding scheme. -ENCODING = 'utf-8' - -if six.PY2: - def encode(value, encoding): - if type(value) == str: - value = value.decode(encoding, errors='ignore') - return value.encode(encoding) -else: - def encode(value, encoding): - return value.encode(encoding) - - -# Functions to initialize read/write for various libarchive supported formats and filters. -FORMATS = { - None: (_libarchive.archive_read_support_format_all, None), - 'tar': (_libarchive.archive_read_support_format_tar, _libarchive.archive_write_set_format_ustar), - 'pax': (_libarchive.archive_read_support_format_tar, _libarchive.archive_write_set_format_pax), - 'gnu': (_libarchive.archive_read_support_format_gnutar, _libarchive.archive_write_set_format_gnutar), - 'zip': (_libarchive.archive_read_support_format_zip, _libarchive.archive_write_set_format_zip), - 'rar': (_libarchive.archive_read_support_format_rar, None), - '7zip': (_libarchive.archive_read_support_format_7zip, None), - 'ar': (_libarchive.archive_read_support_format_ar, None), - 'cab': (_libarchive.archive_read_support_format_cab, None), - 'cpio': (_libarchive.archive_read_support_format_cpio, _libarchive.archive_write_set_format_cpio_newc), - 'iso': (_libarchive.archive_read_support_format_iso9660, _libarchive.archive_write_set_format_iso9660), - 'lha': (_libarchive.archive_read_support_format_lha, None), - 'xar': (_libarchive.archive_read_support_format_xar, _libarchive.archive_write_set_format_xar), -} - -FILTERS = { - None: (_libarchive.archive_read_support_filter_all, _libarchive.archive_write_add_filter_none), - 'bzip2': (_libarchive.archive_read_support_filter_bzip2, _libarchive.archive_write_add_filter_bzip2), - 'gzip': (_libarchive.archive_read_support_filter_gzip, _libarchive.archive_write_add_filter_gzip), - 'zstd': (_libarchive.archive_read_support_filter_zstd, _libarchive.archive_write_add_filter_zstd), -} - -# Map file extensions to formats and filters. To support quick detection. -FORMAT_EXTENSIONS = { - '.tar': 'tar', - '.zip': 'zip', - '.rar': 'rar', - '.7z': '7zip', - '.ar': 'ar', - '.cab': 'cab', - '.rpm': 'cpio', - '.cpio': 'cpio', - '.iso': 'iso', - '.lha': 'lha', - '.xar': 'xar', -} -FILTER_EXTENSIONS = { - '.bz2': 'bzip2', - '.gz': 'gzip', - '.zst': 'zstd', -} - - -class EOF(Exception): - '''Raised by ArchiveInfo.from_archive() when unable to read the next - archive header.''' - pass - - -def get_error(archive): - '''Retrieves the last error description for the given archive instance.''' - return _libarchive.archive_error_string(archive) - - -def call_and_check(func, archive, *args): - '''Executes a libarchive function and raises an exception when appropriate.''' - ret = func(*args) - if ret == _libarchive.ARCHIVE_OK: - return - elif ret == _libarchive.ARCHIVE_WARN: - warnings.warn('Warning executing function: %s.' % get_error(archive), RuntimeWarning) - elif ret == _libarchive.ARCHIVE_EOF: - raise EOF() - else: - raise Exception('Fatal error executing function, message is: %s.' % get_error(archive)) - - -def get_func(name, items, index): - item = items.get(name, None) - if item is None: - return None - return item[index] - - -def guess_format(filename): - filename, ext = os.path.splitext(filename) - filter = FILTER_EXTENSIONS.get(ext) - if filter: - filename, ext = os.path.splitext(filename) - format = FORMAT_EXTENSIONS.get(ext) - return format, filter - - -def is_archive_name(filename, formats=None): - '''Quick check to see if the given file has an extension indiciating that it is - an archive. The format parameter can be used to limit what archive format is acceptable. - If omitted, all supported archive formats will be checked. - - This function will return the name of the most likely archive format, None if the file is - unlikely to be an archive.''' - if formats is None: - formats = FORMAT_EXTENSIONS.values() - format, filter = guess_format(filename) - if format in formats: - return format - - -def is_archive(f, formats=(None, ), filters=(None, )): - '''Check to see if the given file is actually an archive. The format parameter - can be used to specify which archive format is acceptable. If ommitted, all supported - archive formats will be checked. It opens the file using libarchive. If no error is - received, the file was successfully detected by the libarchive bidding process. - - This procedure is quite costly, so you should avoid calling it unless you are reasonably - sure that the given file is an archive. In other words, you may wish to filter large - numbers of file names using is_archive_name() before double-checking the positives with - this function. - - This function will return True if the file can be opened as an archive using the given - format(s)/filter(s).''' - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - f = exit_stack.enter_context(open(f, 'rb')) - a = _libarchive.archive_read_new() - for format in formats: - format = get_func(format, FORMATS, 0) - if format is None: - return False - format(a) - for filter in filters: - filter = get_func(filter, FILTERS, 0) - if filter is None: - return False - filter(a) - try: - try: - call_and_check(_libarchive.archive_read_open_fd, a, a, f.fileno(), BLOCK_SIZE) - return True - except: - return False - finally: - _libarchive.archive_read_close(a) - _libarchive.archive_read_free(a) - - -def get_archive_filter_names(filename): - with open(filename, 'rb') as afile: - a = _libarchive.archive_read_new() - try: - format_func = get_func(None, FORMATS, 0) - format_func(a) - filter_func = get_func(None, FILTERS, 0) - filter_func(a) - if _libarchive.archive_read_open_fd(a, afile.fileno(), BLOCK_SIZE) == _libarchive.ARCHIVE_OK: - try: - nfilter = _libarchive.archive_filter_count(a) - return [_libarchive.archive_filter_name(a, i).decode(ENCODING) for i in range(nfilter)] - finally: - _libarchive.archive_read_close(a) - finally: - _libarchive.archive_read_free(a) - return [] - - -class EntryReadStream(object): - '''A file-like object for reading an entry from the archive.''' - def __init__(self, archive, size): - self.archive = archive - self.closed = False - self.size = size - self.bytes = 0 - - def __enter__(self): - return self - - def __exit__(self, *args): - return - - def __iter__(self): - if self.closed: - return - while True: - data = self.read(BLOCK_SIZE) - if not data: - break - yield data - - def __len__(self): - return self.size - - def tell(self): - return self.bytes - - def read(self, bytes=-1): - if self.closed: - return - if self.bytes == self.size: - # EOF already reached. - return - if bytes < 0: - bytes = self.size - self.bytes - elif self.bytes + bytes > self.size: - # Limit read to remaining bytes - bytes = self.size - self.bytes - # Read requested bytes - data = _libarchive.archive_read_data_into_str(self.archive._a, bytes) - self.bytes += len(data) - return data - - def close(self): - if self.closed: - return - # Call archive.close() with _defer True to let it know we have been - # closed and it is now safe to actually close. - self.archive.close(_defer=True) - self.archive = None - self.closed = True - - -class EntryWriteStream(object): - '''A file-like object for writing an entry to an archive. - - If the size is known ahead of time and provided, then the file contents - are not buffered but flushed directly to the archive. If size is omitted, - then the file contents are buffered and flushed in the close() method.''' - def __init__(self, archive, pathname, size=None): - self.archive = archive - self.entry = Entry(pathname=pathname, mtime=time.time(), mode=stat.S_IFREG) - if size is None: - self.buffer = six.StringIO() - else: - self.buffer = None - self.entry.size = size - self.entry.to_archive(self.archive) - self.bytes = 0 - self.closed = False - - def __enter__(self): - return self - - def __exit__(self, *args): - self.close() - - def __del__(self): - self.close() - - def __len__(self): - return self.bytes - - def tell(self): - return self.bytes - - def write(self, data): - if self.closed: - raise Exception('Cannot write to closed stream.') - if self.buffer: - self.buffer.write(data) - else: - _libarchive.archive_write_data_from_str(self.archive._a, data) - self.bytes += len(data) - - def close(self): - if self.closed: - return - if self.buffer: - self.entry.size = self.buffer.tell() - self.entry.to_archive(self.archive) - _libarchive.archive_write_data_from_str(self.archive._a, self.buffer.getvalue()) - _libarchive.archive_write_finish_entry(self.archive._a) - - # Call archive.close() with _defer True to let it know we have been - # closed and it is now safe to actually close. - self.archive.close(_defer=True) - self.archive = None - self.closed = True - - -class Entry(object): - '''An entry within an archive. Represents the header data and it's location within the archive.''' - def __init__(self, pathname=None, size=None, mtime=None, mode=None, hpos=None, encoding=ENCODING): - self.pathname = pathname - self.size = size - self.mtime = mtime - self.mode = mode - self.hpos = hpos - self.encoding = encoding - self.linkname = None - self.id = None - self.hardlink = None - - @property - def header_position(self): - return self.hpos - - @classmethod - def from_archive(cls, archive, encoding=ENCODING): - '''Instantiates an Entry class and sets all the properties from an archive header.''' - e = _libarchive.archive_entry_new() - try: - call_and_check(_libarchive.archive_read_next_header2, archive._a, archive._a, e) - mode = _libarchive.archive_entry_filetype(e) - mode |= _libarchive.archive_entry_perm(e) - mtime = _libarchive.archive_entry_mtime(e) + _libarchive.archive_entry_mtime_nsec(e) / 1000000000.0 - # use current time as mtime if stored mtime is equal to 0 - mtime = mtime or time.time() - entry = cls( - pathname=_libarchive.archive_entry_pathname(e).decode(encoding), - size=_libarchive.archive_entry_size(e), - mtime=mtime, - mode=mode, - hpos=archive.header_position, - ) - # check hardlinkness first to processes hardlinks to the symlinks correctly - hardlink = _libarchive.archive_entry_hardlink(e) - if hardlink: - entry.hardlink = hardlink - elif entry.issym(): - entry.linkname = _libarchive.archive_entry_symlink(e) - finally: - _libarchive.archive_entry_free(e) - return entry - - @classmethod - def from_file(cls, f, entry=None, encoding=ENCODING, mtime=None): - '''Instantiates an Entry class and sets all the properties from a file on the file system. - f can be a file-like object or a path.''' - if entry is None: - entry = cls(encoding=encoding) - if entry.pathname is None: - if isinstance(f, six.string_types): - st = os.lstat(f) - entry.pathname = f - entry.size = st.st_size - entry.mtime = st.st_mtime if mtime is None else mtime - entry.mode = st.st_mode - entry.id = cls.get_entry_id(st) - if entry.issym(): - entry.linkname = os.readlink(f) - elif hasattr(f, 'fileno'): - st = os.fstat(f.fileno()) - entry.pathname = getattr(f, 'name', None) - entry.size = st.st_size - entry.mtime = st.st_mtime if mtime is None else mtime - entry.mode = st.st_mode - entry.id = cls.get_entry_id(st) - else: - entry.pathname = getattr(f, 'pathname', None) - entry.size = getattr(f, 'size', 0) - entry.mtime = getattr(f, 'mtime', time.time()) if mtime is None else mtime - entry.mode = getattr(f, 'mode', stat.S_IFREG) - return entry - - @staticmethod - def get_entry_id(st): - # windows doesn't have such information - if st.st_ino and st.st_dev: - return (st.st_dev, st.st_ino) - return None - - def to_archive(self, archive): - '''Creates an archive header and writes it to the given archive.''' - e = _libarchive.archive_entry_new() - try: - _libarchive.archive_entry_set_pathname(e, encode(self.pathname, self.encoding)) - _libarchive.archive_entry_set_filetype(e, stat.S_IFMT(self.mode)) - _libarchive.archive_entry_set_perm(e, stat.S_IMODE(self.mode)) - - nsec, sec = math.modf(self.mtime) - nsec *= 1000000000 - _libarchive.archive_entry_set_mtime(e, int(sec), int(nsec)) - - if self.ishardlink(): - _libarchive.archive_entry_set_size(e, 0) - _libarchive.archive_entry_set_hardlink(e, encode(self.hardlink, self.encoding)) - elif self.issym(): - _libarchive.archive_entry_set_size(e, 0) - _libarchive.archive_entry_set_symlink(e, encode(self.linkname, self.encoding)) - else: - _libarchive.archive_entry_set_size(e, self.size) - call_and_check(_libarchive.archive_write_header, archive._a, archive._a, e) - #self.hpos = archive.header_position - finally: - _libarchive.archive_entry_free(e) - - def isdir(self): - return stat.S_ISDIR(self.mode) - - def isfile(self): - return stat.S_ISREG(self.mode) - - def issym(self): - return stat.S_ISLNK(self.mode) - - def isfifo(self): - return stat.S_ISFIFO(self.mode) - - def ischr(self): - return stat.S_ISCHR(self.mode) - - def isblk(self): - return stat.S_ISBLK(self.mode) - - def ishardlink(self): - return bool(self.hardlink) - - -class Archive(object): - '''A low-level archive reader which provides forward-only iteration. Consider - this a light-weight pythonic libarchive wrapper.''' - def __init__(self, f, mode='rb', format=None, filter=None, entry_class=Entry, encoding=ENCODING, blocksize=BLOCK_SIZE, filter_opts=None, format_opts=None, fsync=False, fixed_mtime=None): - if six.PY2: - assert mode in ('r', 'rb', 'w', 'wb', 'a', 'ab'), 'Mode should be "r[b]", "w[b]" or "a[b]".' - else: - assert mode in ('rb', 'wb', 'ab'), 'Mode should be "rb", "wb", or "ab".' - self._stream = None - self.encoding = encoding - self.blocksize = blocksize - self.file_handle = None - self.fd = None - self.filename = None - self.fsync = fsync - if isinstance(f, six.string_types): - self.filename = f - self.file_handle = open(f, mode) - self.fd = self.file_handle.fileno() - # Only close it if we opened it... - self._defer_close = True - elif hasattr(f, 'fileno'): - self.filename = getattr(f, 'name', None) - self.file_handle = f - self.fd = self.file_handle.fileno() - # Leave the fd alone, caller should manage it... - self._defer_close = False - elif isinstance(f, int): - assert f >= 0, f - self.fd = f - # Leave the fd alone, caller should manage it... - self._defer_close = False - else: - raise Exception('Provided file is not path or open file.') - self.mode = mode - # Guess the format/filter from file name (if not provided) - if self.filename: - if format is None: - format = guess_format(self.filename)[0] - if filter is None: - filter = guess_format(self.filename)[1] - self.format = format - self.filter = filter - # The class to use for entries. - self.entry_class = entry_class - self.fixed_mtime = fixed_mtime - # Select filter/format functions. - if self.mode.startswith('r'): - self.format_func = get_func(self.format, FORMATS, 0) - if self.format_func is None: - raise Exception('Unsupported format %s' % format) - self.filter_func = get_func(self.filter, FILTERS, 0) - if self.filter_func is None: - raise Exception('Unsupported filter %s' % filter) - else: - # TODO: how to support appending? - if self.format is None: - raise Exception('You must specify a format for writing.') - self.format_func = get_func(self.format, FORMATS, 1) - if self.format_func is None: - raise Exception('Unsupported format %s' % format) - self.filter_func = get_func(self.filter, FILTERS, 1) - if self.filter_func is None: - raise Exception('Unsupported filter %s' % filter) - # Open the archive, apply filter/format functions. - self.filter_opts = filter_opts - self.format_opts = format_opts - # Stores every added entry's id to handle hardlinks properly - self.members = {} - self.init() - - def __iter__(self): - while True: - try: - yield self.entry_class.from_archive(self, encoding=self.encoding) - except EOF: - break - - def __enter__(self): - return self - - def __exit__(self, type, value, traceback): - self.close() - - def __del__(self): - self.close() - - def init(self): - def _apply_opts(f, opts): - if opts: - for opt_name, opt_val in opts.items(): - call_and_check(f, self._a, self._a, None, encode(opt_name, self.encoding), encode(opt_val, self.encoding)) - - if self.mode.startswith('r'): - self._a = _libarchive.archive_read_new() - else: - self._a = _libarchive.archive_write_new() - self.format_func(self._a) - self.filter_func(self._a) - if self.mode.startswith('r'): - _apply_opts(_libarchive.archive_read_set_format_option, self.format_opts) - _apply_opts(_libarchive.archive_read_set_filter_option, self.filter_opts) - call_and_check(_libarchive.archive_read_open_fd, self._a, self._a, self.fd, self.blocksize) - else: - _apply_opts(_libarchive.archive_write_set_format_option, self.format_opts) - _apply_opts(_libarchive.archive_write_set_filter_option, self.filter_opts) - call_and_check(_libarchive.archive_write_open_fd, self._a, self._a, self.fd) - # XXX Don't pad the last block to avoid badly formed archive with zstd filter - call_and_check(_libarchive.archive_write_set_bytes_in_last_block, self._a, self._a, 1) - - def denit(self): - '''Closes and deallocates the archive reader/writer.''' - if getattr(self, '_a', None) is None: - return - try: - if self.mode.startswith('r'): - _libarchive.archive_read_close(self._a) - _libarchive.archive_read_free(self._a) - else: - _libarchive.archive_write_close(self._a) - _libarchive.archive_write_free(self._a) - finally: - # We only want one try at this... - self._a = None - - def close(self, _defer=False): - # _defer == True is how a stream can notify Archive that the stream is - # now closed. Calling it directly in not recommended. - if _defer: - # This call came from our open stream. - self._stream = None - if not self._defer_close: - # We are not yet ready to close. - return - if self._stream is not None: - # We have a stream open! don't close, but remember we were asked to. - self._defer_close = True - return - self.denit() - # If there is a file attached... - if getattr(self, 'file_handle', None): - # Make sure it is not already closed... - if getattr(self.file_handle, 'closed', False): - return - # Flush it if not read-only... - if not self.file_handle.mode.startswith('r'): - self.file_handle.flush() - if self.fsync: - os.fsync(self.fd) - # and then close it, if we opened it... - if getattr(self, 'close', None): - self.file_handle.close() - - @property - def header_position(self): - '''The position within the file.''' - return _libarchive.archive_read_header_position(self._a) - - def iterpaths(self): - for entry in self: - yield entry.pathname - - def read(self, size): - '''Read current archive entry contents into string.''' - return _libarchive.archive_read_data_into_str(self._a, size) - - def readpath(self, f): - '''Write current archive entry contents to file. f can be a file-like object or - a path.''' - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - basedir = os.path.basename(f) - if not os.path.exists(basedir): - os.makedirs(basedir) - f = exit_stack.enter_context(open(f, 'wb')) - return _libarchive.archive_read_data_into_fd(self._a, f.fileno()) - - def readstream(self, size): - '''Returns a file-like object for reading current archive entry contents.''' - self._stream = EntryReadStream(self, size) - return self._stream - - def write(self, member, data=None): - '''Writes a string buffer to the archive as the given entry.''' - if isinstance(member, six.string_types): - if self.fixed_mtime is None: - mtime = time.time() - else: - mtime = self.fixed_mtime - # Use default mode - member = self.entry_class(pathname=member, encoding=self.encoding, mtime=mtime, mode=stat.S_IFREG | 0o755) - if data: - member.size = len(data) - member.to_archive(self) - if data: - _libarchive.archive_write_data_from_str(self._a, data) - _libarchive.archive_write_finish_entry(self._a) - - def writepath(self, f, pathname=None): - '''Writes a file to the archive. f can be a file-like object or a path. Uses - write() to do the actual writing.''' - member = self.entry_class.from_file(f, encoding=self.encoding, mtime=self.fixed_mtime) - - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - if os.path.isfile(f): - f = exit_stack.enter_context(open(f, 'rb')) - if pathname: - member.pathname = pathname - - # hardlinks and symlink has no data to be written - if member.id in self.members: - member.hardlink = self.members[member.id] - self.write(member) - return - elif member.issym(): - self.write(member) - elif hasattr(f, 'read') and hasattr(f, 'seek') and hasattr(f, 'tell'): - self.write_from_file_object(member, f) - elif hasattr(f, 'read'): - # TODO: optimize this to write directly from f to archive. - self.write(member, data=f.read()) - else: - self.write(member) - - if member.id: - self.members[member.id] = member.pathname - - def write_from_file_object(self, member, fileobj): - if isinstance(member, six.string_types): - member = self.entry_class(pathname=member, encoding=self.encoding, mtime=self.fixed_mtime) - - start = fileobj.tell() - fileobj.seek(0, os.SEEK_END) - size = fileobj.tell() - start - fileobj.seek(start, os.SEEK_SET) - - if size: - member.size = size - member.to_archive(self) - - while size: - data = fileobj.read(BLOCK_SIZE) - if not data: - break - - size -= len(data) - if size < 0: - msg = "File ({}) size has changed. Can't write more data than was declared in the tar header ({}). " \ - "(probably file was changed during archiving)".format(member.pathname, member.size) - logger.warning(msg) - # write rest expected data (size is negative) - _libarchive.archive_write_data_from_str(self._a, data[:size]) - break - - _libarchive.archive_write_data_from_str(self._a, data) - - _libarchive.archive_write_finish_entry(self._a) - - def writestream(self, pathname, size=None): - '''Returns a file-like object for writing a new entry.''' - self._stream = EntryWriteStream(self, pathname, size) - return self._stream - - def printlist(self, s=sys.stdout): - for entry in self: - s.write(entry.size) - s.write('\t') - s.write(entry.mtime.strftime(MTIME_FORMAT)) - s.write('\t') - s.write(entry.pathname) - s.flush() - - -class SeekableArchive(Archive): - '''A class that provides random-access to archive entries. It does this by using one - or many Archive instances to seek to the correct location. The best performance will - occur when reading archive entries in the order in which they appear in the archive. - Reading out of order will cause the archive to be closed and opened each time a - reverse seek is needed.''' - def __init__(self, f, **kwargs): - self._stream = None - # Convert file to open file. We need this to reopen the archive. - mode = kwargs.setdefault('mode', 'rb') - if isinstance(f, six.string_types): - f = open(f, mode) - super(SeekableArchive, self).__init__(f, **kwargs) - self.entries = [] - self.eof = False - - def __iter__(self): - for entry in self.entries: - yield entry - if not self.eof: - try: - for entry in super(SeekableArchive, self).__iter__(): - self.entries.append(entry) - yield entry - except StopIteration: - self.eof = True - - def reopen(self): - '''Seeks the underlying fd to 0 position, then opens the archive. If the archive - is already open, this will effectively re-open it (rewind to the beginning).''' - self.denit() - self.file_handle.seek(0) - self.init() - - def getentry(self, pathname): - '''Take a name or entry object and returns an entry object.''' - for entry in self: - if entry.pathname == pathname: - return entry - raise KeyError(pathname) - - def seek(self, entry): - '''Seeks the archive to the requested entry. Will reopen if necessary.''' - move = entry.header_position - self.header_position - if move != 0: - if move < 0: - # can't move back, re-open archive: - self.reopen() - # move to proper position in stream - for curr in super(SeekableArchive, self).__iter__(): - if curr.header_position == entry.header_position: - break - - def read(self, member): - '''Return the requested archive entry contents as a string.''' - entry = self.getentry(member) - self.seek(entry) - return super(SeekableArchive, self).read(entry.size) - - def readpath(self, member, f): - entry = self.getentry(member) - self.seek(entry) - return super(SeekableArchive, self).readpath(f) - - def readstream(self, member): - '''Returns a file-like object for reading requested archive entry contents.''' - entry = self.getentry(member) - self.seek(entry) - self._stream = EntryReadStream(self, entry.size) - return self._stream diff --git a/contrib/python/python-libarchive/py2/libarchive/_libarchive.swg b/contrib/python/python-libarchive/py2/libarchive/_libarchive.swg deleted file mode 100644 index 2fcb05420e..0000000000 --- a/contrib/python/python-libarchive/py2/libarchive/_libarchive.swg +++ /dev/null @@ -1,339 +0,0 @@ -/* Copyright (c) 2011, SmartFile <btimby@smartfile.com> - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are met: - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the organization nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND - ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY - DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ - -%module _libarchive - -%{ -#define SWIG_PYTHON_STRICT_BYTE_CHAR - -#include <archive.h> -#include <archive_entry.h> -%} - -%include "typemaps.i" - -%typemap(in) time_t -{ - if (PyLong_Check($input)) - $1 = (time_t) PyLong_AsLong($input); - else if (PyInt_Check($input)) - $1 = (time_t) PyInt_AsLong($input); - else if (PyFloat_Check($input)) - $1 = (time_t) PyFloat_AsDouble($input); - else { - PyErr_SetString(PyExc_TypeError,"Expected a large number"); - return NULL; - } -} - -%typemap(out) time_t -{ - $result = PyLong_FromLong((long)$1); -} - -%typemap(in) int64_t -{ - if (PyLong_Check($input)) - $1 = (int64_t) PyLong_AsLong($input); - else if (PyInt_Check($input)) - $1 = (int64_t) PyInt_AsLong($input); - else if (PyFloat_Check($input)) - $1 = (int64_t) PyFloat_AsDouble($input); - else { - PyErr_SetString(PyExc_TypeError,"Expected a large number"); - return NULL; - } -} - -%typemap(out) int64_t -{ - $result = PyLong_FromLong((long)$1); -} - -#define __LA_INT64_T long long -#define __LA_MODE_T int - -/* STRUCTURES */ -struct archive; -struct archive_entry; - -/* ARCHIVE READING */ -extern struct archive *archive_read_new(void); -extern int archive_read_free(struct archive *); - -/* opening */ -extern int archive_read_open_filename(struct archive *, - const char *_filename, size_t _block_size); -extern int archive_read_open_memory(struct archive *, - void * buff, size_t size); -extern int archive_read_open_memory2(struct archive *a, void *buff, - size_t size, size_t read_size); -extern int archive_read_open_fd(struct archive *, int _fd, - size_t _block_size); - -/* closing */ -extern int archive_read_close(struct archive *); -extern int archive_format(struct archive *); - -/* headers */ -extern int archive_read_next_header2(struct archive *, - struct archive_entry *); -extern const struct stat *archive_entry_stat(struct archive_entry *); -extern __LA_INT64_T archive_read_header_position(struct archive *); - -/* data */ -extern int archive_read_data_skip(struct archive *); -extern int archive_read_data_into_fd(struct archive *, int fd); - -/* FILTERS */ -extern int archive_read_support_filter_all(struct archive *); -extern int archive_read_support_filter_bzip2(struct archive *); -extern int archive_read_support_filter_compress(struct archive *); -extern int archive_read_support_filter_gzip(struct archive *); -extern int archive_read_support_filter_lzip(struct archive *); -extern int archive_read_support_filter_lzma(struct archive *); -extern int archive_read_support_filter_none(struct archive *); -extern int archive_read_support_filter_rpm(struct archive *); -extern int archive_read_support_filter_uu(struct archive *); -extern int archive_read_support_filter_xz(struct archive *); -extern int archive_read_support_filter_zstd(struct archive *); - -extern int archive_filter_count(struct archive *); -extern const char * archive_filter_name(struct archive *, int); - -/* FORMATS */ -extern int archive_read_support_format_all(struct archive *); -extern int archive_read_support_format_7zip(struct archive *); -extern int archive_read_support_format_ar(struct archive *); -extern int archive_read_support_format_cab(struct archive *); -extern int archive_read_support_format_cpio(struct archive *); -extern int archive_read_support_format_empty(struct archive *); -extern int archive_read_support_format_gnutar(struct archive *); -extern int archive_read_support_format_iso9660(struct archive *); -extern int archive_read_support_format_lha(struct archive *); -/*extern int archive_read_support_format_mtree(struct archive *);*/ -extern int archive_read_support_format_rar(struct archive *); -extern int archive_read_support_format_raw(struct archive *); -extern int archive_read_support_format_tar(struct archive *); -extern int archive_read_support_format_xar(struct archive *); -extern int archive_read_support_format_zip(struct archive *); -/*extern int archive_read_support_format_by_code(struct archive *, int);*/ - -/* OPTIONS */ -extern int archive_write_set_bytes_in_last_block(struct archive *_a, int bytes_in_last_block); -extern int archive_write_set_filter_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_write_zip_set_compression_deflate(struct archive *_a); -extern int archive_write_set_format_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_read_set_filter_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_read_set_format_option(struct archive *_a, const char *m, const char *o, const char *v); - -/* ARCHIVE WRITING */ -extern struct archive *archive_write_new(void); -extern int archive_write_free(struct archive *); - -/* opening */ -extern int archive_write_open(struct archive *, void *, - archive_open_callback *, archive_write_callback *, - archive_close_callback *); -extern int archive_write_open_fd(struct archive *, int _fd); -extern int archive_write_open_filename(struct archive *, const char *_file); -extern int archive_write_open_filename_w(struct archive *, - const wchar_t *_file); -extern int archive_write_open_memory(struct archive *, - void *_buffer, size_t _buffSize, size_t *_used); - -/* closing */ -extern int archive_write_close(struct archive *); - -/* headers */ -extern int archive_write_header(struct archive *, - struct archive_entry *); - -/* data */ - -/* commit */ -extern int archive_write_finish_entry(struct archive *); - -/* FILTERS */ -extern int archive_write_add_filter_bzip2(struct archive *); -extern int archive_write_add_filter_compress(struct archive *); -extern int archive_write_add_filter_gzip(struct archive *); -extern int archive_write_add_filter_lzip(struct archive *); -extern int archive_write_add_filter_lzma(struct archive *); -extern int archive_write_add_filter_none(struct archive *); -extern int archive_write_add_filter_xz(struct archive *); -extern int archive_write_add_filter_zstd(struct archive *); - - -/* FORMATS */ -/* A convenience function to set the format based on the code or name. */ -extern int archive_write_set_format(struct archive *, int format_code); -extern int archive_write_set_format_by_name(struct archive *, - const char *name); -/* To minimize link pollution, use one or more of the following. */ -extern int archive_write_set_format_ar_bsd(struct archive *); -extern int archive_write_set_format_ar_svr4(struct archive *); -extern int archive_write_set_format_cpio(struct archive *); -extern int archive_write_set_format_cpio_newc(struct archive *); -extern int archive_write_set_format_gnutar(struct archive *); -extern int archive_write_set_format_iso9660(struct archive *); -/*extern int archive_write_set_format_mtree(struct archive *);*/ -/* TODO: int archive_write_set_format_old_tar(struct archive *); */ -extern int archive_write_set_format_pax(struct archive *); -extern int archive_write_set_format_pax_restricted(struct archive *); -extern int archive_write_set_format_shar(struct archive *); -extern int archive_write_set_format_shar_dump(struct archive *); -extern int archive_write_set_format_ustar(struct archive *); -extern int archive_write_set_format_xar(struct archive *); -extern int archive_write_set_format_zip(struct archive *); - -/* ARCHIVE ENTRY */ -extern struct archive_entry *archive_entry_new(void); -extern void archive_entry_free(struct archive_entry *); -extern const char *archive_entry_symlink(struct archive_entry *); -extern void archive_entry_set_symlink(struct archive_entry *, const char *); -extern const char *archive_entry_hardlink(struct archive_entry *); -extern void archive_entry_set_hardlink(struct archive_entry *, const char *); - -/* ARCHIVE ENTRY PROPERTY ACCESS */ -/* reading */ -extern const char *archive_entry_pathname(struct archive_entry *); -extern const wchar_t *archive_entry_pathname_w(struct archive_entry *); -extern __LA_INT64_T archive_entry_size(struct archive_entry *); -extern time_t archive_entry_mtime(struct archive_entry *); -extern time_t archive_entry_mtime_nsec(struct archive_entry *); -extern __LA_MODE_T archive_entry_filetype(struct archive_entry *); -extern __LA_MODE_T archive_entry_perm(struct archive_entry *); - -/* writing */ -extern void archive_entry_set_pathname(struct archive_entry *, const char *); -extern void archive_entry_set_size(struct archive_entry *, __LA_INT64_T); -extern void archive_entry_set_mtime(struct archive_entry *, time_t, long); -extern void archive_entry_set_filetype(struct archive_entry *, unsigned int); -extern void archive_entry_set_perm(struct archive_entry *, __LA_MODE_T); - - -/* ERROR HANDLING */ -extern int archive_errno(struct archive *); -extern const char *archive_error_string(struct archive *); - - -/* CONSTANTS */ -#define ARCHIVE_VERSION_NUMBER 3000001 -#define ARCHIVE_VERSION_STRING "libarchive 3.0.1b" -#define ARCHIVE_EOF 1 /* Found end of archive. */ -#define ARCHIVE_OK 0 /* Operation was successful. */ -#define ARCHIVE_RETRY (-10) /* Retry might succeed. */ -#define ARCHIVE_WARN (-20) /* Partial success. */ -#define ARCHIVE_FAILED (-25) /* Current operation cannot complete. */ -#define ARCHIVE_FATAL (-30) /* No more operations are possible. */ - -#define ARCHIVE_FILTER_NONE 0 -#define ARCHIVE_FILTER_GZIP 1 -#define ARCHIVE_FILTER_BZIP2 2 -#define ARCHIVE_FILTER_COMPRESS 3 -#define ARCHIVE_FILTER_PROGRAM 4 -#define ARCHIVE_FILTER_LZMA 5 -#define ARCHIVE_FILTER_XZ 6 -#define ARCHIVE_FILTER_UU 7 -#define ARCHIVE_FILTER_RPM 8 -#define ARCHIVE_FILTER_LZIP 9 - -#define ARCHIVE_FORMAT_BASE_MASK 0xff0000 -#define ARCHIVE_FORMAT_CPIO 0x10000 -#define ARCHIVE_FORMAT_CPIO_POSIX (ARCHIVE_FORMAT_CPIO | 1) -#define ARCHIVE_FORMAT_CPIO_BIN_LE (ARCHIVE_FORMAT_CPIO | 2) -#define ARCHIVE_FORMAT_CPIO_BIN_BE (ARCHIVE_FORMAT_CPIO | 3) -#define ARCHIVE_FORMAT_CPIO_SVR4_NOCRC (ARCHIVE_FORMAT_CPIO | 4) -#define ARCHIVE_FORMAT_CPIO_SVR4_CRC (ARCHIVE_FORMAT_CPIO | 5) -#define ARCHIVE_FORMAT_CPIO_AFIO_LARGE (ARCHIVE_FORMAT_CPIO | 6) -#define ARCHIVE_FORMAT_SHAR 0x20000 -#define ARCHIVE_FORMAT_SHAR_BASE (ARCHIVE_FORMAT_SHAR | 1) -#define ARCHIVE_FORMAT_SHAR_DUMP (ARCHIVE_FORMAT_SHAR | 2) -#define ARCHIVE_FORMAT_TAR 0x30000 -#define ARCHIVE_FORMAT_TAR_USTAR (ARCHIVE_FORMAT_TAR | 1) -#define ARCHIVE_FORMAT_TAR_PAX_INTERCHANGE (ARCHIVE_FORMAT_TAR | 2) -#define ARCHIVE_FORMAT_TAR_PAX_RESTRICTED (ARCHIVE_FORMAT_TAR | 3) -#define ARCHIVE_FORMAT_TAR_GNUTAR (ARCHIVE_FORMAT_TAR | 4) -#define ARCHIVE_FORMAT_ISO9660 0x40000 -#define ARCHIVE_FORMAT_ISO9660_ROCKRIDGE (ARCHIVE_FORMAT_ISO9660 | 1) -#define ARCHIVE_FORMAT_ZIP 0x50000 -#define ARCHIVE_FORMAT_EMPTY 0x60000 -#define ARCHIVE_FORMAT_AR 0x70000 -#define ARCHIVE_FORMAT_AR_GNU (ARCHIVE_FORMAT_AR | 1) -#define ARCHIVE_FORMAT_AR_BSD (ARCHIVE_FORMAT_AR | 2) -#define ARCHIVE_FORMAT_MTREE 0x80000 -#define ARCHIVE_FORMAT_RAW 0x90000 -#define ARCHIVE_FORMAT_XAR 0xA0000 -#define ARCHIVE_FORMAT_LHA 0xB0000 -#define ARCHIVE_FORMAT_CAB 0xC0000 -#define ARCHIVE_FORMAT_RAR 0xD0000 -#define ARCHIVE_FORMAT_7ZIP 0xE0000 - -#define ARCHIVE_EXTRACT_OWNER (0x0001) -#define ARCHIVE_EXTRACT_PERM (0x0002) -#define ARCHIVE_EXTRACT_TIME (0x0004) -#define ARCHIVE_EXTRACT_NO_OVERWRITE (0x0008) -#define ARCHIVE_EXTRACT_UNLINK (0x0010) -#define ARCHIVE_EXTRACT_ACL (0x0020) -#define ARCHIVE_EXTRACT_FFLAGS (0x0040) -#define ARCHIVE_EXTRACT_XATTR (0x0080) -#define ARCHIVE_EXTRACT_SECURE_SYMLINKS (0x0100) -#define ARCHIVE_EXTRACT_SECURE_NODOTDOT (0x0200) -#define ARCHIVE_EXTRACT_NO_AUTODIR (0x0400) -#define ARCHIVE_EXTRACT_NO_OVERWRITE_NEWER (0x0800) -#define ARCHIVE_EXTRACT_SPARSE (0x1000) -#define ARCHIVE_EXTRACT_MAC_METADATA (0x2000) - -%inline %{ -PyObject *archive_read_data_into_str(struct archive *archive, int len) { - PyObject *str = NULL; - if (!(str = PyBytes_FromStringAndSize(NULL, len))) { - PyErr_SetString(PyExc_MemoryError, "could not allocate string."); - return NULL; - } - if (len != archive_read_data(archive, PyBytes_AS_STRING(str), len)) { - PyErr_SetString(PyExc_RuntimeError, "could not read requested data."); - return NULL; - } - return str; -} - -PyObject *archive_write_data_from_str(struct archive *archive, PyObject *str) { - int len = PyBytes_Size(str); - if (len == 0) - return PyInt_FromLong(len); - int ret = archive_write_data(archive, PyBytes_AS_STRING(str), len); - if (ret == ARCHIVE_FATAL) { - PyErr_Format(PyExc_RuntimeError, "Could not write requested data - most likely no space left on device (error code: %d)", ret); - return NULL; - } - else if (ret <= 0) { - PyErr_Format(PyExc_RuntimeError, "Could not write requested data (error code: %d)", ret); - return NULL; - } - return PyInt_FromLong(len); -} -%} diff --git a/contrib/python/python-libarchive/py2/libarchive/tar.py b/contrib/python/python-libarchive/py2/libarchive/tar.py deleted file mode 100644 index f14149804b..0000000000 --- a/contrib/python/python-libarchive/py2/libarchive/tar.py +++ /dev/null @@ -1,135 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import time -from libarchive import is_archive, Entry, SeekableArchive -from tarfile import DEFAULT_FORMAT, USTAR_FORMAT, GNU_FORMAT, PAX_FORMAT, ENCODING -from tarfile import REGTYPE, AREGTYPE, LNKTYPE, SYMTYPE, DIRTYPE, FIFOTYPE, CONTTYPE, CHRTYPE, BLKTYPE, GNUTYPE_SPARSE - -FORMAT_CONVERSION = { - USTAR_FORMAT: 'tar', - GNU_FORMAT: 'gnu', - PAX_FORMAT: 'pax', -} - - -def is_tarfile(filename): - return is_archive(filename, formats=('tar', 'gnu', 'pax')) - - -def open(**kwargs): - return TarFile(**kwargs) - - -class TarInfo(Entry): - def __init__(self, name): - super(TarInfo, self).__init__(pathname=name) - - fromtarfile = Entry.from_archive - - def get_name(self): - return self.pathname - - def set_name(self, value): - self.pathname = value - - name = property(get_name, set_name) - - @property - def get_type(self): - for attr, type in ( - ('isdir', DIRTYPE), ('isfile', REGTYPE), ('issym', SYMTYPE), - ('isfifo', FIFOTYPE), ('ischr', CHRTYPE), ('isblk', BLKTYPE), - ): - if getattr(self, attr)(): - return type - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - pax_headers = property(_get_missing, _set_missing) - - -class TarFile(SeekableArchive): - def __init__(self, name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, encoding=ENCODING): - if name: - f = name - elif fileobj: - f = fileobj - try: - format = FORMAT_CONVERSON.get(format) - except KeyError: - raise Exception('Invalid tar format: %s' % format) - super(TarFile, self).__init__(f, mode=mode, format=format, entry_class=tarinfo, encoding=encoding) - - getmember = SeekableArchive.getentry - list = SeekableArchive.printlist - extract = SeekableArchive.readpath - extractfile = SeekableArchive.readstream - - def getmembers(self): - return list(self) - - def getnames(self): - return list(self.iterpaths) - - def next(self): - pass # TODO: how to do this? - - def extract(self, member, path=None): - if path is None: - path = os.getcwd() - if isinstance(member, basestring): - f = os.path.join(path, member) - else: - f = os.path.join(path, member.pathname) - return self.readpath(member, f) - - def add(self, name, arcname, recursive=True, exclude=None, filter=None): - pass # TODO: implement this. - - def addfile(tarinfo, fileobj): - return self.writepath(fileobj, tarinfo) - - def gettarinfo(name=None, arcname=None, fileobj=None): - if name: - f = name - elif fileobj: - f = fileobj - entry = self.entry_class.from_file(f) - if arcname: - entry.pathname = arcname - return entry - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - pax_headers = property(_get_missing, _set_missing) diff --git a/contrib/python/python-libarchive/py2/libarchive/zip.py b/contrib/python/python-libarchive/py2/libarchive/zip.py deleted file mode 100644 index 539f6dbcc4..0000000000 --- a/contrib/python/python-libarchive/py2/libarchive/zip.py +++ /dev/null @@ -1,151 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import os, time -from libarchive import is_archive, Entry, SeekableArchive -from zipfile import ZIP_STORED, ZIP_DEFLATED - - -def is_zipfile(filename): - return is_archive(filename, formats=('zip', )) - - -class ZipEntry(Entry): - def __init__(self, *args, **kwargs): - super(ZipEntry, self).__init__(*args, **kwargs) - - def get_filename(self): - return self.pathname - - def set_filename(self, value): - self.pathname = value - - filename = property(get_filename, set_filename) - - def get_file_size(self): - return self.size - - def set_file_size(self, value): - assert isinstance(size, (int, long)), 'Please provide size as int or long.' - self.size = value - - file_size = property(get_file_size, set_file_size) - - def get_date_time(self): - return time.localtime(self.mtime)[0:6] - - def set_date_time(self, value): - assert isinstance(value, tuple), 'mtime should be tuple (year, month, day, hour, minute, second).' - assert len(value) == 6, 'mtime should be tuple (year, month, day, hour, minute, second).' - self.mtime = time.mktime(value + (0, 0, 0)) - - date_time = property(get_date_time, set_date_time) - - header_offset = Entry.header_position - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - compress_type = property(_get_missing, _set_missing) - comment = property(_get_missing, _set_missing) - extra = property(_get_missing, _set_missing) - create_system = property(_get_missing, _set_missing) - create_version = property(_get_missing, _set_missing) - extract_version = property(_get_missing, _set_missing) - reserved = property(_get_missing, _set_missing) - flag_bits = property(_get_missing, _set_missing) - volume = property(_get_missing, _set_missing) - internal_attr = property(_get_missing, _set_missing) - external_attr = property(_get_missing, _set_missing) - CRC = property(_get_missing, _set_missing) - compress_size = property(_get_missing, _set_missing) - - -class ZipFile(SeekableArchive): - def __init__(self, f, mode='r', compression=ZIP_DEFLATED, allowZip64=False): - super(ZipFile, self).__init__(f, mode=mode, format='zip', entry_class=ZipEntry, encoding='CP437') - if mode == 'w' and compression == ZIP_STORED: - # Disable compression for writing. - _libarchive.archive_write_set_format_option(self.archive._a, "zip", "compression", "store") - self.compression = compression - - getinfo = SeekableArchive.getentry - - def namelist(self): - return list(self.iterpaths) - - def infolist(self): - return list(self) - - def open(self, name, mode, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if mode == 'r': - return self.readstream(name) - else: - return self.writestream(name) - - def extract(self, name, path=None, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if not path: - path = os.getcwd() - return self.readpath(name, os.path.join(path, name)) - - def extractall(self, path, names=None, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if not names: - names = self.namelist() - if names: - for name in names: - self.extract(name, path) - - def read(self, name, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - return self.read(name) - - def writestr(self, member, data, compress_type=None): - if compress_type != self.compression: - raise Exception('Cannot change compression type for individual entries.') - return self.write(member, data) - - def setpassword(self, pwd): - raise NotImplemented('Encryption not supported.') - - def testzip(self): - raise NotImplemented() - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - comment = property(_get_missing, _set_missing) diff --git a/contrib/python/python-libarchive/py2/ya.make b/contrib/python/python-libarchive/py2/ya.make deleted file mode 100644 index 3e73181b64..0000000000 --- a/contrib/python/python-libarchive/py2/ya.make +++ /dev/null @@ -1,28 +0,0 @@ -PY2_LIBRARY() - -LICENSE(BSD-3-Clause) - -VERSION(3.1.2.post1) - -PEERDIR( - contrib/libs/libarchive - contrib/python/contextlib2 - contrib/python/six -) - -ADDINCL( - contrib/libs/libarchive/libarchive -) - -NO_LINT() - -PY_SRCS( - SWIG_C - TOP_LEVEL - libarchive/__init__.py - libarchive/tar.py - libarchive/zip.py - libarchive/_libarchive.swg -) - -END() diff --git a/contrib/python/python-libarchive/py3/libarchive/__init__.py b/contrib/python/python-libarchive/py3/libarchive/__init__.py deleted file mode 100644 index 0c0c63359a..0000000000 --- a/contrib/python/python-libarchive/py3/libarchive/__init__.py +++ /dev/null @@ -1,800 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import os -import stat -import sys -import math -import time -import logging -import warnings - -import contextlib2 - -from libarchive import _libarchive -import six - -logger = logging.getLogger(__name__) - -# Suggested block size for libarchive. Libarchive may adjust it. -BLOCK_SIZE = 10240 - -MTIME_FORMAT = '' - -# Default encoding scheme. -ENCODING = 'utf-8' - -if six.PY2: - def encode(value, encoding): - if type(value) == str: - value = value.decode(encoding, errors='ignore') - return value.encode(encoding) -else: - def encode(value, encoding): - return value.encode(encoding) - - -# Functions to initialize read/write for various libarchive supported formats and filters. -FORMATS = { - None: (_libarchive.archive_read_support_format_all, None), - 'tar': (_libarchive.archive_read_support_format_tar, _libarchive.archive_write_set_format_ustar), - 'pax': (_libarchive.archive_read_support_format_tar, _libarchive.archive_write_set_format_pax), - 'gnu': (_libarchive.archive_read_support_format_gnutar, _libarchive.archive_write_set_format_gnutar), - 'zip': (_libarchive.archive_read_support_format_zip, _libarchive.archive_write_set_format_zip), - 'rar': (_libarchive.archive_read_support_format_rar, None), - '7zip': (_libarchive.archive_read_support_format_7zip, None), - 'ar': (_libarchive.archive_read_support_format_ar, None), - 'cab': (_libarchive.archive_read_support_format_cab, None), - 'cpio': (_libarchive.archive_read_support_format_cpio, _libarchive.archive_write_set_format_cpio_newc), - 'iso': (_libarchive.archive_read_support_format_iso9660, _libarchive.archive_write_set_format_iso9660), - 'lha': (_libarchive.archive_read_support_format_lha, None), - 'xar': (_libarchive.archive_read_support_format_xar, _libarchive.archive_write_set_format_xar), -} - -FILTERS = { - None: (_libarchive.archive_read_support_filter_all, _libarchive.archive_write_add_filter_none), - 'bzip2': (_libarchive.archive_read_support_filter_bzip2, _libarchive.archive_write_add_filter_bzip2), - 'gzip': (_libarchive.archive_read_support_filter_gzip, _libarchive.archive_write_add_filter_gzip), - 'zstd': (_libarchive.archive_read_support_filter_zstd, _libarchive.archive_write_add_filter_zstd), -} - -# Map file extensions to formats and filters. To support quick detection. -FORMAT_EXTENSIONS = { - '.tar': 'tar', - '.zip': 'zip', - '.rar': 'rar', - '.7z': '7zip', - '.ar': 'ar', - '.cab': 'cab', - '.rpm': 'cpio', - '.cpio': 'cpio', - '.iso': 'iso', - '.lha': 'lha', - '.xar': 'xar', -} -FILTER_EXTENSIONS = { - '.bz2': 'bzip2', - '.gz': 'gzip', - '.zst': 'zstd', -} - - -class EOF(Exception): - '''Raised by ArchiveInfo.from_archive() when unable to read the next - archive header.''' - pass - - -def get_error(archive): - '''Retrieves the last error description for the given archive instance.''' - return _libarchive.archive_error_string(archive) - - -def call_and_check(func, archive, *args): - '''Executes a libarchive function and raises an exception when appropriate.''' - ret = func(*args) - if ret == _libarchive.ARCHIVE_OK: - return - elif ret == _libarchive.ARCHIVE_WARN: - warnings.warn('Warning executing function: %s.' % get_error(archive), RuntimeWarning) - elif ret == _libarchive.ARCHIVE_EOF: - raise EOF() - else: - raise Exception('Fatal error executing function, message is: %s.' % get_error(archive)) - - -def get_func(name, items, index): - item = items.get(name, None) - if item is None: - return None - return item[index] - - -def guess_format(filename): - filename, ext = os.path.splitext(filename) - filter = FILTER_EXTENSIONS.get(ext) - if filter: - filename, ext = os.path.splitext(filename) - format = FORMAT_EXTENSIONS.get(ext) - return format, filter - - -def is_archive_name(filename, formats=None): - '''Quick check to see if the given file has an extension indiciating that it is - an archive. The format parameter can be used to limit what archive format is acceptable. - If omitted, all supported archive formats will be checked. - - This function will return the name of the most likely archive format, None if the file is - unlikely to be an archive.''' - if formats is None: - formats = FORMAT_EXTENSIONS.values() - format, filter = guess_format(filename) - if format in formats: - return format - - -def is_archive(f, formats=(None, ), filters=(None, )): - '''Check to see if the given file is actually an archive. The format parameter - can be used to specify which archive format is acceptable. If ommitted, all supported - archive formats will be checked. It opens the file using libarchive. If no error is - received, the file was successfully detected by the libarchive bidding process. - - This procedure is quite costly, so you should avoid calling it unless you are reasonably - sure that the given file is an archive. In other words, you may wish to filter large - numbers of file names using is_archive_name() before double-checking the positives with - this function. - - This function will return True if the file can be opened as an archive using the given - format(s)/filter(s).''' - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - f = exit_stack.enter_context(open(f, 'rb')) - a = _libarchive.archive_read_new() - for format in formats: - format = get_func(format, FORMATS, 0) - if format is None: - return False - format(a) - for filter in filters: - filter = get_func(filter, FILTERS, 0) - if filter is None: - return False - filter(a) - try: - try: - call_and_check(_libarchive.archive_read_open_fd, a, a, f.fileno(), BLOCK_SIZE) - return True - except: - return False - finally: - _libarchive.archive_read_close(a) - _libarchive.archive_read_free(a) - - -def get_archive_filter_names(filename): - with open(filename, 'rb') as afile: - a = _libarchive.archive_read_new() - try: - format_func = get_func(None, FORMATS, 0) - format_func(a) - filter_func = get_func(None, FILTERS, 0) - filter_func(a) - if _libarchive.archive_read_open_fd(a, afile.fileno(), BLOCK_SIZE) == _libarchive.ARCHIVE_OK: - try: - nfilter = _libarchive.archive_filter_count(a) - return [_libarchive.archive_filter_name(a, i).decode(ENCODING) for i in range(nfilter)] - finally: - _libarchive.archive_read_close(a) - finally: - _libarchive.archive_read_free(a) - return [] - - -class EntryReadStream(object): - '''A file-like object for reading an entry from the archive.''' - def __init__(self, archive, size): - self.archive = archive - self.closed = False - self.size = size - self.bytes = 0 - - def __enter__(self): - return self - - def __exit__(self, *args): - return - - def __iter__(self): - if self.closed: - return - while True: - data = self.read(BLOCK_SIZE) - if not data: - break - yield data - - def __len__(self): - return self.size - - def tell(self): - return self.bytes - - def read(self, bytes=-1): - if self.closed: - return - if self.bytes == self.size: - # EOF already reached. - return - if bytes < 0: - bytes = self.size - self.bytes - elif self.bytes + bytes > self.size: - # Limit read to remaining bytes - bytes = self.size - self.bytes - # Read requested bytes - data = _libarchive.archive_read_data_into_str(self.archive._a, bytes) - self.bytes += len(data) - return data - - def close(self): - if self.closed: - return - # Call archive.close() with _defer True to let it know we have been - # closed and it is now safe to actually close. - self.archive.close(_defer=True) - self.archive = None - self.closed = True - - -class EntryWriteStream(object): - '''A file-like object for writing an entry to an archive. - - If the size is known ahead of time and provided, then the file contents - are not buffered but flushed directly to the archive. If size is omitted, - then the file contents are buffered and flushed in the close() method.''' - def __init__(self, archive, pathname, size=None): - self.archive = archive - self.entry = Entry(pathname=pathname, mtime=time.time(), mode=stat.S_IFREG) - if size is None: - self.buffer = six.StringIO() - else: - self.buffer = None - self.entry.size = size - self.entry.to_archive(self.archive) - self.bytes = 0 - self.closed = False - - def __enter__(self): - return self - - def __exit__(self, *args): - self.close() - - def __del__(self): - self.close() - - def __len__(self): - return self.bytes - - def tell(self): - return self.bytes - - def write(self, data): - if self.closed: - raise Exception('Cannot write to closed stream.') - if self.buffer: - self.buffer.write(data) - else: - _libarchive.archive_write_data_from_str(self.archive._a, data) - self.bytes += len(data) - - def close(self): - if self.closed: - return - if self.buffer: - self.entry.size = self.buffer.tell() - self.entry.to_archive(self.archive) - _libarchive.archive_write_data_from_str(self.archive._a, self.buffer.getvalue()) - _libarchive.archive_write_finish_entry(self.archive._a) - - # Call archive.close() with _defer True to let it know we have been - # closed and it is now safe to actually close. - self.archive.close(_defer=True) - self.archive = None - self.closed = True - - -class Entry(object): - '''An entry within an archive. Represents the header data and it's location within the archive.''' - def __init__(self, pathname=None, size=None, mtime=None, mode=None, hpos=None, encoding=ENCODING): - self.pathname = pathname - self.size = size - self.mtime = mtime - self.mode = mode - self.hpos = hpos - self.encoding = encoding - self.linkname = None - self.id = None - self.hardlink = None - - @property - def header_position(self): - return self.hpos - - @classmethod - def from_archive(cls, archive, encoding=ENCODING): - '''Instantiates an Entry class and sets all the properties from an archive header.''' - e = _libarchive.archive_entry_new() - try: - call_and_check(_libarchive.archive_read_next_header2, archive._a, archive._a, e) - mode = _libarchive.archive_entry_filetype(e) - mode |= _libarchive.archive_entry_perm(e) - mtime = _libarchive.archive_entry_mtime(e) + _libarchive.archive_entry_mtime_nsec(e) / 1000000000.0 - # use current time as mtime if stored mtime is equal to 0 - mtime = mtime or time.time() - entry = cls( - pathname=_libarchive.archive_entry_pathname(e).decode(encoding), - size=_libarchive.archive_entry_size(e), - mtime=mtime, - mode=mode, - hpos=archive.header_position, - ) - # check hardlinkness first to processes hardlinks to the symlinks correctly - hardlink = _libarchive.archive_entry_hardlink(e) - if hardlink: - entry.hardlink = hardlink - elif entry.issym(): - entry.linkname = _libarchive.archive_entry_symlink(e) - finally: - _libarchive.archive_entry_free(e) - return entry - - @classmethod - def from_file(cls, f, entry=None, encoding=ENCODING, mtime=None): - '''Instantiates an Entry class and sets all the properties from a file on the file system. - f can be a file-like object or a path.''' - if entry is None: - entry = cls(encoding=encoding) - if entry.pathname is None: - if isinstance(f, six.string_types): - st = os.lstat(f) - entry.pathname = f - entry.size = st.st_size - entry.mtime = st.st_mtime if mtime is None else mtime - entry.mode = st.st_mode - entry.id = cls.get_entry_id(st) - if entry.issym(): - entry.linkname = os.readlink(f) - elif hasattr(f, 'fileno'): - st = os.fstat(f.fileno()) - entry.pathname = getattr(f, 'name', None) - entry.size = st.st_size - entry.mtime = st.st_mtime if mtime is None else mtime - entry.mode = st.st_mode - entry.id = cls.get_entry_id(st) - else: - entry.pathname = getattr(f, 'pathname', None) - entry.size = getattr(f, 'size', 0) - entry.mtime = getattr(f, 'mtime', time.time()) if mtime is None else mtime - entry.mode = getattr(f, 'mode', stat.S_IFREG) - return entry - - @staticmethod - def get_entry_id(st): - # windows doesn't have such information - if st.st_ino and st.st_dev: - return (st.st_dev, st.st_ino) - return None - - def to_archive(self, archive): - '''Creates an archive header and writes it to the given archive.''' - e = _libarchive.archive_entry_new() - try: - _libarchive.archive_entry_set_pathname(e, encode(self.pathname, self.encoding)) - _libarchive.archive_entry_set_filetype(e, stat.S_IFMT(self.mode)) - _libarchive.archive_entry_set_perm(e, stat.S_IMODE(self.mode)) - - nsec, sec = math.modf(self.mtime) - nsec *= 1000000000 - _libarchive.archive_entry_set_mtime(e, int(sec), int(nsec)) - - if self.ishardlink(): - _libarchive.archive_entry_set_size(e, 0) - _libarchive.archive_entry_set_hardlink(e, encode(self.hardlink, self.encoding)) - elif self.issym(): - _libarchive.archive_entry_set_size(e, 0) - _libarchive.archive_entry_set_symlink(e, encode(self.linkname, self.encoding)) - else: - _libarchive.archive_entry_set_size(e, self.size) - call_and_check(_libarchive.archive_write_header, archive._a, archive._a, e) - #self.hpos = archive.header_position - finally: - _libarchive.archive_entry_free(e) - - def isdir(self): - return stat.S_ISDIR(self.mode) - - def isfile(self): - return stat.S_ISREG(self.mode) - - def issym(self): - return stat.S_ISLNK(self.mode) - - def isfifo(self): - return stat.S_ISFIFO(self.mode) - - def ischr(self): - return stat.S_ISCHR(self.mode) - - def isblk(self): - return stat.S_ISBLK(self.mode) - - def ishardlink(self): - return bool(self.hardlink) - - -class Archive(object): - '''A low-level archive reader which provides forward-only iteration. Consider - this a light-weight pythonic libarchive wrapper.''' - def __init__(self, f, mode='rb', format=None, filter=None, entry_class=Entry, encoding=ENCODING, blocksize=BLOCK_SIZE, filter_opts=None, format_opts=None, fsync=False, fixed_mtime=None): - if six.PY2: - assert mode in ('r', 'rb', 'w', 'wb', 'a', 'ab'), 'Mode should be "r[b]", "w[b]" or "a[b]".' - else: - assert mode in ('rb', 'wb', 'ab'), 'Mode should be "rb", "wb", or "ab".' - self._stream = None - self.encoding = encoding - self.blocksize = blocksize - self.file_handle = None - self.fd = None - self.filename = None - self.fsync = fsync - if isinstance(f, six.string_types): - self.filename = f - self.file_handle = open(f, mode) - self.fd = self.file_handle.fileno() - # Only close it if we opened it... - self._defer_close = True - elif hasattr(f, 'fileno'): - self.filename = getattr(f, 'name', None) - self.file_handle = f - self.fd = self.file_handle.fileno() - # Leave the fd alone, caller should manage it... - self._defer_close = False - elif isinstance(f, int): - assert f >= 0, f - self.fd = f - # Leave the fd alone, caller should manage it... - self._defer_close = False - else: - raise Exception('Provided file is not path or open file.') - self.mode = mode - # Guess the format/filter from file name (if not provided) - if self.filename: - if format is None: - format = guess_format(self.filename)[0] - if filter is None: - filter = guess_format(self.filename)[1] - self.format = format - self.filter = filter - # The class to use for entries. - self.entry_class = entry_class - self.fixed_mtime = fixed_mtime - # Select filter/format functions. - if self.mode.startswith('r'): - self.format_func = get_func(self.format, FORMATS, 0) - if self.format_func is None: - raise Exception('Unsupported format %s' % format) - self.filter_func = get_func(self.filter, FILTERS, 0) - if self.filter_func is None: - raise Exception('Unsupported filter %s' % filter) - else: - # TODO: how to support appending? - if self.format is None: - raise Exception('You must specify a format for writing.') - self.format_func = get_func(self.format, FORMATS, 1) - if self.format_func is None: - raise Exception('Unsupported format %s' % format) - self.filter_func = get_func(self.filter, FILTERS, 1) - if self.filter_func is None: - raise Exception('Unsupported filter %s' % filter) - # Open the archive, apply filter/format functions. - self.filter_opts = filter_opts - self.format_opts = format_opts - # Stores every added entry's id to handle hardlinks properly - self.members = {} - self.init() - - def __iter__(self): - while True: - try: - yield self.entry_class.from_archive(self, encoding=self.encoding) - except EOF: - break - - def __enter__(self): - return self - - def __exit__(self, type, value, traceback): - self.close() - - def __del__(self): - self.close() - - def init(self): - def _apply_opts(f, opts): - if opts: - for opt_name, opt_val in opts.items(): - call_and_check(f, self._a, self._a, None, encode(opt_name, self.encoding), encode(opt_val, self.encoding)) - - if self.mode.startswith('r'): - self._a = _libarchive.archive_read_new() - else: - self._a = _libarchive.archive_write_new() - self.format_func(self._a) - self.filter_func(self._a) - if self.mode.startswith('r'): - _apply_opts(_libarchive.archive_read_set_format_option, self.format_opts) - _apply_opts(_libarchive.archive_read_set_filter_option, self.filter_opts) - call_and_check(_libarchive.archive_read_open_fd, self._a, self._a, self.fd, self.blocksize) - else: - _apply_opts(_libarchive.archive_write_set_format_option, self.format_opts) - _apply_opts(_libarchive.archive_write_set_filter_option, self.filter_opts) - call_and_check(_libarchive.archive_write_open_fd, self._a, self._a, self.fd) - # XXX Don't pad the last block to avoid badly formed archive with zstd filter - call_and_check(_libarchive.archive_write_set_bytes_in_last_block, self._a, self._a, 1) - - def denit(self): - '''Closes and deallocates the archive reader/writer.''' - if getattr(self, '_a', None) is None: - return - try: - if self.mode.startswith('r'): - _libarchive.archive_read_close(self._a) - _libarchive.archive_read_free(self._a) - else: - _libarchive.archive_write_close(self._a) - _libarchive.archive_write_free(self._a) - finally: - # We only want one try at this... - self._a = None - - def close(self, _defer=False): - # _defer == True is how a stream can notify Archive that the stream is - # now closed. Calling it directly in not recommended. - if _defer: - # This call came from our open stream. - self._stream = None - if not self._defer_close: - # We are not yet ready to close. - return - if self._stream is not None: - # We have a stream open! don't close, but remember we were asked to. - self._defer_close = True - return - self.denit() - # If there is a file attached... - if getattr(self, 'file_handle', None): - # Make sure it is not already closed... - if getattr(self.file_handle, 'closed', False): - return - # Flush it if not read-only... - if not self.file_handle.mode.startswith('r'): - self.file_handle.flush() - if self.fsync: - os.fsync(self.fd) - # and then close it, if we opened it... - if getattr(self, 'close', None): - self.file_handle.close() - - @property - def header_position(self): - '''The position within the file.''' - return _libarchive.archive_read_header_position(self._a) - - def iterpaths(self): - for entry in self: - yield entry.pathname - - def read(self, size): - '''Read current archive entry contents into string.''' - return _libarchive.archive_read_data_into_str(self._a, size) - - def readpath(self, f): - '''Write current archive entry contents to file. f can be a file-like object or - a path.''' - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - basedir = os.path.basename(f) - if not os.path.exists(basedir): - os.makedirs(basedir) - f = exit_stack.enter_context(open(f, 'wb')) - return _libarchive.archive_read_data_into_fd(self._a, f.fileno()) - - def readstream(self, size): - '''Returns a file-like object for reading current archive entry contents.''' - self._stream = EntryReadStream(self, size) - return self._stream - - def write(self, member, data=None): - '''Writes a string buffer to the archive as the given entry.''' - if isinstance(member, six.string_types): - if self.fixed_mtime is None: - mtime = time.time() - else: - mtime = self.fixed_mtime - # Use default mode - member = self.entry_class(pathname=member, encoding=self.encoding, mtime=mtime, mode=stat.S_IFREG | 0o755) - if data: - member.size = len(data) - member.to_archive(self) - if data: - _libarchive.archive_write_data_from_str(self._a, data) - _libarchive.archive_write_finish_entry(self._a) - - def writepath(self, f, pathname=None): - '''Writes a file to the archive. f can be a file-like object or a path. Uses - write() to do the actual writing.''' - member = self.entry_class.from_file(f, encoding=self.encoding, mtime=self.fixed_mtime) - - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - if os.path.isfile(f): - f = exit_stack.enter_context(open(f, 'rb')) - if pathname: - member.pathname = pathname - - # hardlinks and symlink has no data to be written - if member.id in self.members: - member.hardlink = self.members[member.id] - self.write(member) - return - elif member.issym(): - self.write(member) - elif hasattr(f, 'read') and hasattr(f, 'seek') and hasattr(f, 'tell'): - self.write_from_file_object(member, f) - elif hasattr(f, 'read'): - # TODO: optimize this to write directly from f to archive. - self.write(member, data=f.read()) - else: - self.write(member) - - if member.id: - self.members[member.id] = member.pathname - - def write_from_file_object(self, member, fileobj): - if isinstance(member, six.string_types): - member = self.entry_class(pathname=member, encoding=self.encoding, mtime=self.fixed_mtime) - - start = fileobj.tell() - fileobj.seek(0, os.SEEK_END) - size = fileobj.tell() - start - fileobj.seek(start, os.SEEK_SET) - - if size: - member.size = size - member.to_archive(self) - - while size: - data = fileobj.read(BLOCK_SIZE) - if not data: - break - - size -= len(data) - if size < 0: - msg = "File ({}) size has changed. Can't write more data than was declared in the tar header ({}). " \ - "(probably file was changed during archiving)".format(member.pathname, member.size) - logger.warning(msg) - # write rest expected data (size is negative) - _libarchive.archive_write_data_from_str(self._a, data[:size]) - break - - _libarchive.archive_write_data_from_str(self._a, data) - - _libarchive.archive_write_finish_entry(self._a) - - def writestream(self, pathname, size=None): - '''Returns a file-like object for writing a new entry.''' - self._stream = EntryWriteStream(self, pathname, size) - return self._stream - - def printlist(self, s=sys.stdout): - for entry in self: - s.write(entry.size) - s.write('\t') - s.write(entry.mtime.strftime(MTIME_FORMAT)) - s.write('\t') - s.write(entry.pathname) - s.flush() - - -class SeekableArchive(Archive): - '''A class that provides random-access to archive entries. It does this by using one - or many Archive instances to seek to the correct location. The best performance will - occur when reading archive entries in the order in which they appear in the archive. - Reading out of order will cause the archive to be closed and opened each time a - reverse seek is needed.''' - def __init__(self, f, **kwargs): - self._stream = None - # Convert file to open file. We need this to reopen the archive. - mode = kwargs.setdefault('mode', 'rb') - if isinstance(f, six.string_types): - f = open(f, mode) - super(SeekableArchive, self).__init__(f, **kwargs) - self.entries = [] - self.eof = False - - def __iter__(self): - for entry in self.entries: - yield entry - if not self.eof: - try: - for entry in super(SeekableArchive, self).__iter__(): - self.entries.append(entry) - yield entry - except StopIteration: - self.eof = True - - def reopen(self): - '''Seeks the underlying fd to 0 position, then opens the archive. If the archive - is already open, this will effectively re-open it (rewind to the beginning).''' - self.denit() - self.file_handle.seek(0) - self.init() - - def getentry(self, pathname): - '''Take a name or entry object and returns an entry object.''' - for entry in self: - if entry.pathname == pathname: - return entry - raise KeyError(pathname) - - def seek(self, entry): - '''Seeks the archive to the requested entry. Will reopen if necessary.''' - move = entry.header_position - self.header_position - if move != 0: - if move < 0: - # can't move back, re-open archive: - self.reopen() - # move to proper position in stream - for curr in super(SeekableArchive, self).__iter__(): - if curr.header_position == entry.header_position: - break - - def read(self, member): - '''Return the requested archive entry contents as a string.''' - entry = self.getentry(member) - self.seek(entry) - return super(SeekableArchive, self).read(entry.size) - - def readpath(self, member, f): - entry = self.getentry(member) - self.seek(entry) - return super(SeekableArchive, self).readpath(f) - - def readstream(self, member): - '''Returns a file-like object for reading requested archive entry contents.''' - entry = self.getentry(member) - self.seek(entry) - self._stream = EntryReadStream(self, entry.size) - return self._stream diff --git a/contrib/python/python-libarchive/py3/libarchive/_libarchive.swg b/contrib/python/python-libarchive/py3/libarchive/_libarchive.swg deleted file mode 100644 index 2fcb05420e..0000000000 --- a/contrib/python/python-libarchive/py3/libarchive/_libarchive.swg +++ /dev/null @@ -1,339 +0,0 @@ -/* Copyright (c) 2011, SmartFile <btimby@smartfile.com> - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are met: - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the organization nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND - ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY - DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ - -%module _libarchive - -%{ -#define SWIG_PYTHON_STRICT_BYTE_CHAR - -#include <archive.h> -#include <archive_entry.h> -%} - -%include "typemaps.i" - -%typemap(in) time_t -{ - if (PyLong_Check($input)) - $1 = (time_t) PyLong_AsLong($input); - else if (PyInt_Check($input)) - $1 = (time_t) PyInt_AsLong($input); - else if (PyFloat_Check($input)) - $1 = (time_t) PyFloat_AsDouble($input); - else { - PyErr_SetString(PyExc_TypeError,"Expected a large number"); - return NULL; - } -} - -%typemap(out) time_t -{ - $result = PyLong_FromLong((long)$1); -} - -%typemap(in) int64_t -{ - if (PyLong_Check($input)) - $1 = (int64_t) PyLong_AsLong($input); - else if (PyInt_Check($input)) - $1 = (int64_t) PyInt_AsLong($input); - else if (PyFloat_Check($input)) - $1 = (int64_t) PyFloat_AsDouble($input); - else { - PyErr_SetString(PyExc_TypeError,"Expected a large number"); - return NULL; - } -} - -%typemap(out) int64_t -{ - $result = PyLong_FromLong((long)$1); -} - -#define __LA_INT64_T long long -#define __LA_MODE_T int - -/* STRUCTURES */ -struct archive; -struct archive_entry; - -/* ARCHIVE READING */ -extern struct archive *archive_read_new(void); -extern int archive_read_free(struct archive *); - -/* opening */ -extern int archive_read_open_filename(struct archive *, - const char *_filename, size_t _block_size); -extern int archive_read_open_memory(struct archive *, - void * buff, size_t size); -extern int archive_read_open_memory2(struct archive *a, void *buff, - size_t size, size_t read_size); -extern int archive_read_open_fd(struct archive *, int _fd, - size_t _block_size); - -/* closing */ -extern int archive_read_close(struct archive *); -extern int archive_format(struct archive *); - -/* headers */ -extern int archive_read_next_header2(struct archive *, - struct archive_entry *); -extern const struct stat *archive_entry_stat(struct archive_entry *); -extern __LA_INT64_T archive_read_header_position(struct archive *); - -/* data */ -extern int archive_read_data_skip(struct archive *); -extern int archive_read_data_into_fd(struct archive *, int fd); - -/* FILTERS */ -extern int archive_read_support_filter_all(struct archive *); -extern int archive_read_support_filter_bzip2(struct archive *); -extern int archive_read_support_filter_compress(struct archive *); -extern int archive_read_support_filter_gzip(struct archive *); -extern int archive_read_support_filter_lzip(struct archive *); -extern int archive_read_support_filter_lzma(struct archive *); -extern int archive_read_support_filter_none(struct archive *); -extern int archive_read_support_filter_rpm(struct archive *); -extern int archive_read_support_filter_uu(struct archive *); -extern int archive_read_support_filter_xz(struct archive *); -extern int archive_read_support_filter_zstd(struct archive *); - -extern int archive_filter_count(struct archive *); -extern const char * archive_filter_name(struct archive *, int); - -/* FORMATS */ -extern int archive_read_support_format_all(struct archive *); -extern int archive_read_support_format_7zip(struct archive *); -extern int archive_read_support_format_ar(struct archive *); -extern int archive_read_support_format_cab(struct archive *); -extern int archive_read_support_format_cpio(struct archive *); -extern int archive_read_support_format_empty(struct archive *); -extern int archive_read_support_format_gnutar(struct archive *); -extern int archive_read_support_format_iso9660(struct archive *); -extern int archive_read_support_format_lha(struct archive *); -/*extern int archive_read_support_format_mtree(struct archive *);*/ -extern int archive_read_support_format_rar(struct archive *); -extern int archive_read_support_format_raw(struct archive *); -extern int archive_read_support_format_tar(struct archive *); -extern int archive_read_support_format_xar(struct archive *); -extern int archive_read_support_format_zip(struct archive *); -/*extern int archive_read_support_format_by_code(struct archive *, int);*/ - -/* OPTIONS */ -extern int archive_write_set_bytes_in_last_block(struct archive *_a, int bytes_in_last_block); -extern int archive_write_set_filter_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_write_zip_set_compression_deflate(struct archive *_a); -extern int archive_write_set_format_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_read_set_filter_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_read_set_format_option(struct archive *_a, const char *m, const char *o, const char *v); - -/* ARCHIVE WRITING */ -extern struct archive *archive_write_new(void); -extern int archive_write_free(struct archive *); - -/* opening */ -extern int archive_write_open(struct archive *, void *, - archive_open_callback *, archive_write_callback *, - archive_close_callback *); -extern int archive_write_open_fd(struct archive *, int _fd); -extern int archive_write_open_filename(struct archive *, const char *_file); -extern int archive_write_open_filename_w(struct archive *, - const wchar_t *_file); -extern int archive_write_open_memory(struct archive *, - void *_buffer, size_t _buffSize, size_t *_used); - -/* closing */ -extern int archive_write_close(struct archive *); - -/* headers */ -extern int archive_write_header(struct archive *, - struct archive_entry *); - -/* data */ - -/* commit */ -extern int archive_write_finish_entry(struct archive *); - -/* FILTERS */ -extern int archive_write_add_filter_bzip2(struct archive *); -extern int archive_write_add_filter_compress(struct archive *); -extern int archive_write_add_filter_gzip(struct archive *); -extern int archive_write_add_filter_lzip(struct archive *); -extern int archive_write_add_filter_lzma(struct archive *); -extern int archive_write_add_filter_none(struct archive *); -extern int archive_write_add_filter_xz(struct archive *); -extern int archive_write_add_filter_zstd(struct archive *); - - -/* FORMATS */ -/* A convenience function to set the format based on the code or name. */ -extern int archive_write_set_format(struct archive *, int format_code); -extern int archive_write_set_format_by_name(struct archive *, - const char *name); -/* To minimize link pollution, use one or more of the following. */ -extern int archive_write_set_format_ar_bsd(struct archive *); -extern int archive_write_set_format_ar_svr4(struct archive *); -extern int archive_write_set_format_cpio(struct archive *); -extern int archive_write_set_format_cpio_newc(struct archive *); -extern int archive_write_set_format_gnutar(struct archive *); -extern int archive_write_set_format_iso9660(struct archive *); -/*extern int archive_write_set_format_mtree(struct archive *);*/ -/* TODO: int archive_write_set_format_old_tar(struct archive *); */ -extern int archive_write_set_format_pax(struct archive *); -extern int archive_write_set_format_pax_restricted(struct archive *); -extern int archive_write_set_format_shar(struct archive *); -extern int archive_write_set_format_shar_dump(struct archive *); -extern int archive_write_set_format_ustar(struct archive *); -extern int archive_write_set_format_xar(struct archive *); -extern int archive_write_set_format_zip(struct archive *); - -/* ARCHIVE ENTRY */ -extern struct archive_entry *archive_entry_new(void); -extern void archive_entry_free(struct archive_entry *); -extern const char *archive_entry_symlink(struct archive_entry *); -extern void archive_entry_set_symlink(struct archive_entry *, const char *); -extern const char *archive_entry_hardlink(struct archive_entry *); -extern void archive_entry_set_hardlink(struct archive_entry *, const char *); - -/* ARCHIVE ENTRY PROPERTY ACCESS */ -/* reading */ -extern const char *archive_entry_pathname(struct archive_entry *); -extern const wchar_t *archive_entry_pathname_w(struct archive_entry *); -extern __LA_INT64_T archive_entry_size(struct archive_entry *); -extern time_t archive_entry_mtime(struct archive_entry *); -extern time_t archive_entry_mtime_nsec(struct archive_entry *); -extern __LA_MODE_T archive_entry_filetype(struct archive_entry *); -extern __LA_MODE_T archive_entry_perm(struct archive_entry *); - -/* writing */ -extern void archive_entry_set_pathname(struct archive_entry *, const char *); -extern void archive_entry_set_size(struct archive_entry *, __LA_INT64_T); -extern void archive_entry_set_mtime(struct archive_entry *, time_t, long); -extern void archive_entry_set_filetype(struct archive_entry *, unsigned int); -extern void archive_entry_set_perm(struct archive_entry *, __LA_MODE_T); - - -/* ERROR HANDLING */ -extern int archive_errno(struct archive *); -extern const char *archive_error_string(struct archive *); - - -/* CONSTANTS */ -#define ARCHIVE_VERSION_NUMBER 3000001 -#define ARCHIVE_VERSION_STRING "libarchive 3.0.1b" -#define ARCHIVE_EOF 1 /* Found end of archive. */ -#define ARCHIVE_OK 0 /* Operation was successful. */ -#define ARCHIVE_RETRY (-10) /* Retry might succeed. */ -#define ARCHIVE_WARN (-20) /* Partial success. */ -#define ARCHIVE_FAILED (-25) /* Current operation cannot complete. */ -#define ARCHIVE_FATAL (-30) /* No more operations are possible. */ - -#define ARCHIVE_FILTER_NONE 0 -#define ARCHIVE_FILTER_GZIP 1 -#define ARCHIVE_FILTER_BZIP2 2 -#define ARCHIVE_FILTER_COMPRESS 3 -#define ARCHIVE_FILTER_PROGRAM 4 -#define ARCHIVE_FILTER_LZMA 5 -#define ARCHIVE_FILTER_XZ 6 -#define ARCHIVE_FILTER_UU 7 -#define ARCHIVE_FILTER_RPM 8 -#define ARCHIVE_FILTER_LZIP 9 - -#define ARCHIVE_FORMAT_BASE_MASK 0xff0000 -#define ARCHIVE_FORMAT_CPIO 0x10000 -#define ARCHIVE_FORMAT_CPIO_POSIX (ARCHIVE_FORMAT_CPIO | 1) -#define ARCHIVE_FORMAT_CPIO_BIN_LE (ARCHIVE_FORMAT_CPIO | 2) -#define ARCHIVE_FORMAT_CPIO_BIN_BE (ARCHIVE_FORMAT_CPIO | 3) -#define ARCHIVE_FORMAT_CPIO_SVR4_NOCRC (ARCHIVE_FORMAT_CPIO | 4) -#define ARCHIVE_FORMAT_CPIO_SVR4_CRC (ARCHIVE_FORMAT_CPIO | 5) -#define ARCHIVE_FORMAT_CPIO_AFIO_LARGE (ARCHIVE_FORMAT_CPIO | 6) -#define ARCHIVE_FORMAT_SHAR 0x20000 -#define ARCHIVE_FORMAT_SHAR_BASE (ARCHIVE_FORMAT_SHAR | 1) -#define ARCHIVE_FORMAT_SHAR_DUMP (ARCHIVE_FORMAT_SHAR | 2) -#define ARCHIVE_FORMAT_TAR 0x30000 -#define ARCHIVE_FORMAT_TAR_USTAR (ARCHIVE_FORMAT_TAR | 1) -#define ARCHIVE_FORMAT_TAR_PAX_INTERCHANGE (ARCHIVE_FORMAT_TAR | 2) -#define ARCHIVE_FORMAT_TAR_PAX_RESTRICTED (ARCHIVE_FORMAT_TAR | 3) -#define ARCHIVE_FORMAT_TAR_GNUTAR (ARCHIVE_FORMAT_TAR | 4) -#define ARCHIVE_FORMAT_ISO9660 0x40000 -#define ARCHIVE_FORMAT_ISO9660_ROCKRIDGE (ARCHIVE_FORMAT_ISO9660 | 1) -#define ARCHIVE_FORMAT_ZIP 0x50000 -#define ARCHIVE_FORMAT_EMPTY 0x60000 -#define ARCHIVE_FORMAT_AR 0x70000 -#define ARCHIVE_FORMAT_AR_GNU (ARCHIVE_FORMAT_AR | 1) -#define ARCHIVE_FORMAT_AR_BSD (ARCHIVE_FORMAT_AR | 2) -#define ARCHIVE_FORMAT_MTREE 0x80000 -#define ARCHIVE_FORMAT_RAW 0x90000 -#define ARCHIVE_FORMAT_XAR 0xA0000 -#define ARCHIVE_FORMAT_LHA 0xB0000 -#define ARCHIVE_FORMAT_CAB 0xC0000 -#define ARCHIVE_FORMAT_RAR 0xD0000 -#define ARCHIVE_FORMAT_7ZIP 0xE0000 - -#define ARCHIVE_EXTRACT_OWNER (0x0001) -#define ARCHIVE_EXTRACT_PERM (0x0002) -#define ARCHIVE_EXTRACT_TIME (0x0004) -#define ARCHIVE_EXTRACT_NO_OVERWRITE (0x0008) -#define ARCHIVE_EXTRACT_UNLINK (0x0010) -#define ARCHIVE_EXTRACT_ACL (0x0020) -#define ARCHIVE_EXTRACT_FFLAGS (0x0040) -#define ARCHIVE_EXTRACT_XATTR (0x0080) -#define ARCHIVE_EXTRACT_SECURE_SYMLINKS (0x0100) -#define ARCHIVE_EXTRACT_SECURE_NODOTDOT (0x0200) -#define ARCHIVE_EXTRACT_NO_AUTODIR (0x0400) -#define ARCHIVE_EXTRACT_NO_OVERWRITE_NEWER (0x0800) -#define ARCHIVE_EXTRACT_SPARSE (0x1000) -#define ARCHIVE_EXTRACT_MAC_METADATA (0x2000) - -%inline %{ -PyObject *archive_read_data_into_str(struct archive *archive, int len) { - PyObject *str = NULL; - if (!(str = PyBytes_FromStringAndSize(NULL, len))) { - PyErr_SetString(PyExc_MemoryError, "could not allocate string."); - return NULL; - } - if (len != archive_read_data(archive, PyBytes_AS_STRING(str), len)) { - PyErr_SetString(PyExc_RuntimeError, "could not read requested data."); - return NULL; - } - return str; -} - -PyObject *archive_write_data_from_str(struct archive *archive, PyObject *str) { - int len = PyBytes_Size(str); - if (len == 0) - return PyInt_FromLong(len); - int ret = archive_write_data(archive, PyBytes_AS_STRING(str), len); - if (ret == ARCHIVE_FATAL) { - PyErr_Format(PyExc_RuntimeError, "Could not write requested data - most likely no space left on device (error code: %d)", ret); - return NULL; - } - else if (ret <= 0) { - PyErr_Format(PyExc_RuntimeError, "Could not write requested data (error code: %d)", ret); - return NULL; - } - return PyInt_FromLong(len); -} -%} diff --git a/contrib/python/python-libarchive/py3/libarchive/tar.py b/contrib/python/python-libarchive/py3/libarchive/tar.py deleted file mode 100644 index f14149804b..0000000000 --- a/contrib/python/python-libarchive/py3/libarchive/tar.py +++ /dev/null @@ -1,135 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import time -from libarchive import is_archive, Entry, SeekableArchive -from tarfile import DEFAULT_FORMAT, USTAR_FORMAT, GNU_FORMAT, PAX_FORMAT, ENCODING -from tarfile import REGTYPE, AREGTYPE, LNKTYPE, SYMTYPE, DIRTYPE, FIFOTYPE, CONTTYPE, CHRTYPE, BLKTYPE, GNUTYPE_SPARSE - -FORMAT_CONVERSION = { - USTAR_FORMAT: 'tar', - GNU_FORMAT: 'gnu', - PAX_FORMAT: 'pax', -} - - -def is_tarfile(filename): - return is_archive(filename, formats=('tar', 'gnu', 'pax')) - - -def open(**kwargs): - return TarFile(**kwargs) - - -class TarInfo(Entry): - def __init__(self, name): - super(TarInfo, self).__init__(pathname=name) - - fromtarfile = Entry.from_archive - - def get_name(self): - return self.pathname - - def set_name(self, value): - self.pathname = value - - name = property(get_name, set_name) - - @property - def get_type(self): - for attr, type in ( - ('isdir', DIRTYPE), ('isfile', REGTYPE), ('issym', SYMTYPE), - ('isfifo', FIFOTYPE), ('ischr', CHRTYPE), ('isblk', BLKTYPE), - ): - if getattr(self, attr)(): - return type - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - pax_headers = property(_get_missing, _set_missing) - - -class TarFile(SeekableArchive): - def __init__(self, name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, encoding=ENCODING): - if name: - f = name - elif fileobj: - f = fileobj - try: - format = FORMAT_CONVERSON.get(format) - except KeyError: - raise Exception('Invalid tar format: %s' % format) - super(TarFile, self).__init__(f, mode=mode, format=format, entry_class=tarinfo, encoding=encoding) - - getmember = SeekableArchive.getentry - list = SeekableArchive.printlist - extract = SeekableArchive.readpath - extractfile = SeekableArchive.readstream - - def getmembers(self): - return list(self) - - def getnames(self): - return list(self.iterpaths) - - def next(self): - pass # TODO: how to do this? - - def extract(self, member, path=None): - if path is None: - path = os.getcwd() - if isinstance(member, basestring): - f = os.path.join(path, member) - else: - f = os.path.join(path, member.pathname) - return self.readpath(member, f) - - def add(self, name, arcname, recursive=True, exclude=None, filter=None): - pass # TODO: implement this. - - def addfile(tarinfo, fileobj): - return self.writepath(fileobj, tarinfo) - - def gettarinfo(name=None, arcname=None, fileobj=None): - if name: - f = name - elif fileobj: - f = fileobj - entry = self.entry_class.from_file(f) - if arcname: - entry.pathname = arcname - return entry - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - pax_headers = property(_get_missing, _set_missing) diff --git a/contrib/python/python-libarchive/py3/libarchive/zip.py b/contrib/python/python-libarchive/py3/libarchive/zip.py deleted file mode 100644 index 539f6dbcc4..0000000000 --- a/contrib/python/python-libarchive/py3/libarchive/zip.py +++ /dev/null @@ -1,151 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import os, time -from libarchive import is_archive, Entry, SeekableArchive -from zipfile import ZIP_STORED, ZIP_DEFLATED - - -def is_zipfile(filename): - return is_archive(filename, formats=('zip', )) - - -class ZipEntry(Entry): - def __init__(self, *args, **kwargs): - super(ZipEntry, self).__init__(*args, **kwargs) - - def get_filename(self): - return self.pathname - - def set_filename(self, value): - self.pathname = value - - filename = property(get_filename, set_filename) - - def get_file_size(self): - return self.size - - def set_file_size(self, value): - assert isinstance(size, (int, long)), 'Please provide size as int or long.' - self.size = value - - file_size = property(get_file_size, set_file_size) - - def get_date_time(self): - return time.localtime(self.mtime)[0:6] - - def set_date_time(self, value): - assert isinstance(value, tuple), 'mtime should be tuple (year, month, day, hour, minute, second).' - assert len(value) == 6, 'mtime should be tuple (year, month, day, hour, minute, second).' - self.mtime = time.mktime(value + (0, 0, 0)) - - date_time = property(get_date_time, set_date_time) - - header_offset = Entry.header_position - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - compress_type = property(_get_missing, _set_missing) - comment = property(_get_missing, _set_missing) - extra = property(_get_missing, _set_missing) - create_system = property(_get_missing, _set_missing) - create_version = property(_get_missing, _set_missing) - extract_version = property(_get_missing, _set_missing) - reserved = property(_get_missing, _set_missing) - flag_bits = property(_get_missing, _set_missing) - volume = property(_get_missing, _set_missing) - internal_attr = property(_get_missing, _set_missing) - external_attr = property(_get_missing, _set_missing) - CRC = property(_get_missing, _set_missing) - compress_size = property(_get_missing, _set_missing) - - -class ZipFile(SeekableArchive): - def __init__(self, f, mode='r', compression=ZIP_DEFLATED, allowZip64=False): - super(ZipFile, self).__init__(f, mode=mode, format='zip', entry_class=ZipEntry, encoding='CP437') - if mode == 'w' and compression == ZIP_STORED: - # Disable compression for writing. - _libarchive.archive_write_set_format_option(self.archive._a, "zip", "compression", "store") - self.compression = compression - - getinfo = SeekableArchive.getentry - - def namelist(self): - return list(self.iterpaths) - - def infolist(self): - return list(self) - - def open(self, name, mode, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if mode == 'r': - return self.readstream(name) - else: - return self.writestream(name) - - def extract(self, name, path=None, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if not path: - path = os.getcwd() - return self.readpath(name, os.path.join(path, name)) - - def extractall(self, path, names=None, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if not names: - names = self.namelist() - if names: - for name in names: - self.extract(name, path) - - def read(self, name, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - return self.read(name) - - def writestr(self, member, data, compress_type=None): - if compress_type != self.compression: - raise Exception('Cannot change compression type for individual entries.') - return self.write(member, data) - - def setpassword(self, pwd): - raise NotImplemented('Encryption not supported.') - - def testzip(self): - raise NotImplemented() - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - comment = property(_get_missing, _set_missing) diff --git a/contrib/python/python-libarchive/py3/ya.make b/contrib/python/python-libarchive/py3/ya.make deleted file mode 100644 index a905f47a12..0000000000 --- a/contrib/python/python-libarchive/py3/ya.make +++ /dev/null @@ -1,28 +0,0 @@ -PY3_LIBRARY() - -LICENSE(BSD-3-Clause) - -VERSION(3.1.2.post1) - -PEERDIR( - contrib/libs/libarchive - contrib/python/contextlib2 - contrib/python/six -) - -ADDINCL( - contrib/libs/libarchive/libarchive -) - -NO_LINT() - -PY_SRCS( - SWIG_C - TOP_LEVEL - libarchive/__init__.py - libarchive/tar.py - libarchive/zip.py - libarchive/_libarchive.swg -) - -END() diff --git a/contrib/python/python-libarchive/ya.make b/contrib/python/python-libarchive/ya.make deleted file mode 100644 index 112b869160..0000000000 --- a/contrib/python/python-libarchive/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/python-libarchive/py2) -ELSE() - PEERDIR(contrib/python/python-libarchive/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/termcolor/py2/.dist-info/METADATA b/contrib/python/termcolor/py2/.dist-info/METADATA deleted file mode 100644 index cbcd7281c6..0000000000 --- a/contrib/python/termcolor/py2/.dist-info/METADATA +++ /dev/null @@ -1,135 +0,0 @@ -Metadata-Version: 2.1 -Name: termcolor -Version: 1.1.0 -Summary: ANSII Color formatting for output in terminal. -Home-page: http://pypi.python.org/pypi/termcolor -Author: Konstantin Lepa -Author-email: konstantin.lepa@gmail.com -License: MIT -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: Environment :: Console -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: MIT License -Classifier: Operating System :: OS Independent -Classifier: Programming Language :: Python -Classifier: Topic :: Terminals - -Example -======= - :: - - import sys - from termcolor import colored, cprint - - text = colored('Hello, World!', 'red', attrs=['reverse', 'blink']) - print(text) - cprint('Hello, World!', 'green', 'on_red') - - print_red_on_cyan = lambda x: cprint(x, 'red', 'on_cyan') - print_red_on_cyan('Hello, World!') - print_red_on_cyan('Hello, Universe!') - - for i in range(10): - cprint(i, 'magenta', end=' ') - - cprint("Attention!", 'red', attrs=['bold'], file=sys.stderr) - -Text Properties -=============== - - Text colors: - - - grey - - red - - green - - yellow - - blue - - magenta - - cyan - - white - - Text highlights: - - - on_grey - - on_red - - on_green - - on_yellow - - on_blue - - on_magenta - - on_cyan - - on_white - - Attributes: - - - bold - - dark - - underline - - blink - - reverse - - concealed - -Terminal properties -=================== - - ============ ======= ==== ========= ========== ======= ========= - Terminal bold dark underline blink reverse concealed - ------------ ------- ---- --------- ---------- ------- --------- - xterm yes no yes bold yes yes - linux yes yes bold yes yes no - rxvt yes no yes bold/black yes no - dtterm yes yes yes reverse yes yes - teraterm reverse no yes rev/red yes no - aixterm normal no yes no yes yes - PuTTY color no yes no yes no - Windows no no no no yes no - Cygwin SSH yes no color color color yes - Mac Terminal yes no yes yes yes yes - ============ ======= ==== ========= ========== ======= ========= - - -CHANGES -======= - -1.1.0 (13.01.2011) ------------------- - -- Added cprint function. - -1.0.1 (13.01.2011) ------------------- - -- Updated README.rst. - -1.0.0 (13.01.2011) ------------------- - -- Changed license to MIT. -- Updated copyright. -- Refactored source code. - -0.2 (07.09.2010) ----------------- - -- Added support of Python 3.x. - -0.1.2 (04.06.2009) ------------------- - -- Fixed bold characters. (Thanks Tibor Fekete) - -0.1.1 (05.03.2009) ------------------- - -- Some refactoring. -- Updated copyright. -- Fixed reset colors. -- Updated documentation. - -0.1 (09.06.2008) ----------------- - -- Initial release. - - - diff --git a/contrib/python/termcolor/py2/.dist-info/top_level.txt b/contrib/python/termcolor/py2/.dist-info/top_level.txt deleted file mode 100644 index f08cca1411..0000000000 --- a/contrib/python/termcolor/py2/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -termcolor diff --git a/contrib/python/termcolor/py2/COPYING.txt b/contrib/python/termcolor/py2/COPYING.txt deleted file mode 100644 index d5df97633a..0000000000 --- a/contrib/python/termcolor/py2/COPYING.txt +++ /dev/null @@ -1,20 +0,0 @@ -Copyright (c) 2008-2011 Volvox Development Team - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - diff --git a/contrib/python/termcolor/py2/README.rst b/contrib/python/termcolor/py2/README.rst deleted file mode 100644 index 95aa524367..0000000000 --- a/contrib/python/termcolor/py2/README.rst +++ /dev/null @@ -1,72 +0,0 @@ -Example -======= - :: - - import sys - from termcolor import colored, cprint - - text = colored('Hello, World!', 'red', attrs=['reverse', 'blink']) - print(text) - cprint('Hello, World!', 'green', 'on_red') - - print_red_on_cyan = lambda x: cprint(x, 'red', 'on_cyan') - print_red_on_cyan('Hello, World!') - print_red_on_cyan('Hello, Universe!') - - for i in range(10): - cprint(i, 'magenta', end=' ') - - cprint("Attention!", 'red', attrs=['bold'], file=sys.stderr) - -Text Properties -=============== - - Text colors: - - - grey - - red - - green - - yellow - - blue - - magenta - - cyan - - white - - Text highlights: - - - on_grey - - on_red - - on_green - - on_yellow - - on_blue - - on_magenta - - on_cyan - - on_white - - Attributes: - - - bold - - dark - - underline - - blink - - reverse - - concealed - -Terminal properties -=================== - - ============ ======= ==== ========= ========== ======= ========= - Terminal bold dark underline blink reverse concealed - ------------ ------- ---- --------- ---------- ------- --------- - xterm yes no yes bold yes yes - linux yes yes bold yes yes no - rxvt yes no yes bold/black yes no - dtterm yes yes yes reverse yes yes - teraterm reverse no yes rev/red yes no - aixterm normal no yes no yes yes - PuTTY color no yes no yes no - Windows no no no no yes no - Cygwin SSH yes no color color color yes - Mac Terminal yes no yes yes yes yes - ============ ======= ==== ========= ========== ======= ========= - diff --git a/contrib/python/termcolor/py2/termcolor.py b/contrib/python/termcolor/py2/termcolor.py deleted file mode 100644 index db63ea0200..0000000000 --- a/contrib/python/termcolor/py2/termcolor.py +++ /dev/null @@ -1,217 +0,0 @@ -# coding: utf-8 -# Copyright (c) 2008-2011 Volvox Development Team -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -# THE SOFTWARE. -# -# Author: Konstantin Lepa <konstantin.lepa@gmail.com> - -"""ANSI color formatting for output in terminal.""" - -from __future__ import print_function -import os -import sys - - -__ALL__ = [ "ATTRIBUTES", "COLORS", "HIGHLIGHTS", "RESET", "colored", "cprint" ] - -ATTRIBUTES = { - "bold": 1, - "dark": 2, - "underline": 4, - "blink": 5, - "reverse": 7, - "concealed": 8, -} - - -HIGHLIGHTS = { - "on_black": 40, - "on_grey": 40, # Actually black but kept for backwards compatibility - "on_red": 41, - "on_green": 42, - "on_yellow": 43, - "on_blue": 44, - "on_magenta": 45, - "on_cyan": 46, - "on_light_grey": 47, - "on_dark_grey": 100, - "on_light_red": 101, - "on_light_green": 102, - "on_light_yellow": 103, - "on_light_blue": 104, - "on_light_magenta": 105, - "on_light_cyan": 106, - "on_white": 107, -} - -COLORS = { - "black": 30, - "grey": 30, # Actually black but kept for backwards compatibility - "red": 31, - "green": 32, - "yellow": 33, - "blue": 34, - "magenta": 35, - "cyan": 36, - "light_grey": 37, - "dark_grey": 90, - "light_red": 91, - "light_green": 92, - "light_yellow": 93, - "light_blue": 94, - "light_magenta": 95, - "light_cyan": 96, - "white": 97, -} - - -RESET = "\033[0m" - - -def _can_do_colour(no_color, force_color=None): - """Check env vars and for tty/dumb terminal""" - # First check overrides: - # "User-level configuration files and per-instance command-line arguments should - # override $NO_COLOR. A user should be able to export $NO_COLOR in their shell - # configuration file as a default, but configure a specific program in its - # configuration file to specifically enable color." - # https://no-color.org - if no_color is not None and no_color: - return False - if force_color is not None and force_color: - return True - - # Then check env vars: - if "ANSI_COLORS_DISABLED" in os.environ: - return False - if "NO_COLOR" in os.environ: - return False - if "FORCE_COLOR" in os.environ: - return True - return hasattr(sys.stdout, "isatty") and sys.stdout.isatty() and os.environ.get("TERM") != "dumb" - - -def colored(text, color=None, on_color=None, attrs=None, no_color=None, force_color=None): - """Colorize text. - - Available text colors: - black, red, green, yellow, blue, magenta, cyan, white, - light_grey, dark_grey, light_red, light_green, light_yellow, light_blue, - light_magenta, light_cyan. - - Available text highlights: - on_black, on_red, on_green, on_yellow, on_blue, on_magenta, on_cyan, on_white, - on_light_grey, on_dark_grey, on_light_red, on_light_green, on_light_yellow, - on_light_blue, on_light_magenta, on_light_cyan. - - Available attributes: - bold, dark, underline, blink, reverse, concealed. - - Example: - colored('Hello, World!', 'red', 'on_black', ['bold', 'blink']) - colored('Hello, World!', 'green') - """ - if not _can_do_colour(no_color=no_color, force_color=force_color): - return text - - fmt_str = "\033[%dm%s" - if color is not None: - text = fmt_str % (COLORS[color], text) - - if on_color is not None: - text = fmt_str % (HIGHLIGHTS[on_color], text) - - if attrs is not None: - for attr in attrs: - text = fmt_str % (ATTRIBUTES[attr], text) - - return text + RESET - - -def cprint(text, color=None, on_color=None, attrs=None, no_color=None, force_color=None, **kwargs): - """Print colorized text. - - It accepts arguments of print function. - """ - - print((colored(text, color, on_color, attrs, no_color=no_color, force_color=force_color)), **kwargs) - - -if __name__ == "__main__": - print("Current terminal type: %s" % os.getenv("TERM")) - print("Test basic colors:") - cprint("Black color", "black") - cprint("Red color", "red") - cprint("Green color", "green") - cprint("Yellow color", "yellow") - cprint("Blue color", "blue") - cprint("Magenta color", "magenta") - cprint("Cyan color", "cyan") - cprint("White color", "white") - cprint("Light grey color", "light_grey") - cprint("Dark grey color", "dark_grey") - cprint("Light red color", "light_red") - cprint("Light green color", "light_green") - cprint("Light yellow color", "light_yellow") - cprint("Light blue color", "light_blue") - cprint("Light magenta color", "light_magenta") - cprint("Light cyan color", "light_cyan") - print("-" * 78) - - print("Test highlights:") - cprint("On black color", on_color="on_black") - cprint("On red color", on_color="on_red") - cprint("On green color", on_color="on_green") - cprint("On yellow color", on_color="on_yellow") - cprint("On blue color", on_color="on_blue") - cprint("On magenta color", on_color="on_magenta") - cprint("On cyan color", on_color="on_cyan") - cprint("On white color", color="black", on_color="on_white") - cprint("On light grey color", on_color="on_light_grey") - cprint("On dark grey color", on_color="on_dark_grey") - cprint("On light red color", on_color="on_light_red") - cprint("On light green color", on_color="on_light_green") - cprint("On light yellow color", on_color="on_light_yellow") - cprint("On light blue color", on_color="on_light_blue") - cprint("On light magenta color", on_color="on_light_magenta") - cprint("On light cyan color", on_color="on_light_cyan") - print("-" * 78) - - print("Test attributes:") - cprint("Bold black color", "black", attrs=["bold"]) - cprint("Dark red color", "red", attrs=["dark"]) - cprint("Underline green color", "green", attrs=["underline"]) - cprint("Blink yellow color", "yellow", attrs=["blink"]) - cprint("Reversed blue color", "blue", attrs=["reverse"]) - cprint("Concealed Magenta color", "magenta", attrs=["concealed"]) - cprint( - "Bold underline reverse cyan color", - "cyan", - attrs=["bold", "underline", "reverse"], - ) - cprint( - "Dark blink concealed white color", - "white", - attrs=["dark", "blink", "concealed"], - ) - print("-" * 78) - - print("Test mixing:") - cprint("Underline red on black color", "red", "on_black", ["underline"]) - cprint("Reversed green on red color", "green", "on_red", ["reverse"]) diff --git a/contrib/python/termcolor/py2/ya.make b/contrib/python/termcolor/py2/ya.make deleted file mode 100644 index 0c30233373..0000000000 --- a/contrib/python/termcolor/py2/ya.make +++ /dev/null @@ -1,22 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(1.1.0) - -LICENSE(MIT) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - termcolor.py -) - -RESOURCE_FILES( - PREFIX contrib/python/termcolor/py2/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() diff --git a/contrib/python/termcolor/py3/.dist-info/METADATA b/contrib/python/termcolor/py3/.dist-info/METADATA deleted file mode 100644 index d8dea360f4..0000000000 --- a/contrib/python/termcolor/py3/.dist-info/METADATA +++ /dev/null @@ -1,126 +0,0 @@ -Metadata-Version: 2.1 -Name: termcolor -Version: 2.3.0 -Summary: ANSI color formatting for output in terminal -Project-URL: Changelog, https://github.com/termcolor/termcolor/releases -Project-URL: Homepage, https://github.com/termcolor/termcolor -Project-URL: Source, https://github.com/termcolor/termcolor -Author-email: Konstantin Lepa <konstantin.lepa@gmail.com> -Maintainer: Hugo van Kemenade -License: MIT -License-File: COPYING.txt -Keywords: ANSI,ANSI color,ANSI colour,color,colour,formatting,termcolor,terminal -Classifier: Development Status :: 5 - Production/Stable -Classifier: Environment :: Console -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: MIT License -Classifier: Operating System :: OS Independent -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3 :: Only -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Programming Language :: Python :: 3.11 -Classifier: Programming Language :: Python :: 3.12 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Classifier: Topic :: Terminals -Requires-Python: >=3.7 -Provides-Extra: tests -Requires-Dist: pytest; extra == 'tests' -Requires-Dist: pytest-cov; extra == 'tests' -Description-Content-Type: text/markdown - -# termcolor - -[](https://pypi.org/project/termcolor) -[](https://pypi.org/project/termcolor) -[](https://pypistats.org/packages/termcolor) -[](https://github.com/termcolor/termcolor/actions) -[](https://codecov.io/gh/termcolor/termcolor) -[](COPYING.txt) -[](https://github.com/psf/black) -[](https://tidelift.com/subscription/pkg/pypi-termcolor?utm_source=pypi-termcolor&utm_medium=referral&utm_campaign=readme) - -## Installation - -### From PyPI - -```bash -python3 -m pip install --upgrade termcolor -``` - -### From source - -```bash -git clone https://github.com/termcolor/termcolor -cd termcolor -python3 -m pip install . -``` - -### Demo - -To see demo output, run: - -```bash -python3 -m termcolor -``` - -## Example - -```python -import sys - -from termcolor import colored, cprint - -text = colored("Hello, World!", "red", attrs=["reverse", "blink"]) -print(text) -cprint("Hello, World!", "green", "on_red") - -print_red_on_cyan = lambda x: cprint(x, "red", "on_cyan") -print_red_on_cyan("Hello, World!") -print_red_on_cyan("Hello, Universe!") - -for i in range(10): - cprint(i, "magenta", end=" ") - -cprint("Attention!", "red", attrs=["bold"], file=sys.stderr) -``` - -## Text properties - -| Text colors | Text highlights | Attributes | -| --------------- | ------------------ | ----------- | -| `black` | `on_black` | `bold` | -| `red` | `on_red` | `dark` | -| `green` | `on_green` | `underline` | -| `yellow` | `on_yellow` | `blink` | -| `blue` | `on_blue` | `reverse` | -| `magenta` | `on_magenta` | `concealed` | -| `cyan` | `on_cyan` | | -| `white` | `on_white` | | -| `light_grey` | `on_light_grey` | | -| `dark_grey` | `on_dark_grey` | | -| `light_red` | `on_light_red` | | -| `light_green` | `on_light_green` | | -| `light_yellow` | `on_light_yellow` | | -| `light_blue` | `on_light_blue` | | -| `light_magenta` | `on_light_magenta` | | -| `light_cyan` | `on_light_cyan` | | - -## Terminal properties - -| Terminal | bold | dark | underline | blink | reverse | concealed | -| ------------ | ------- | ---- | --------- | ---------- | ------- | --------- | -| xterm | yes | no | yes | bold | yes | yes | -| linux | yes | yes | bold | yes | yes | no | -| rxvt | yes | no | yes | bold/black | yes | no | -| dtterm | yes | yes | yes | reverse | yes | yes | -| teraterm | reverse | no | yes | rev/red | yes | no | -| aixterm | normal | no | yes | no | yes | yes | -| PuTTY | color | no | yes | no | yes | no | -| Windows | no | no | no | no | yes | no | -| Cygwin SSH | yes | no | color | color | color | yes | -| Mac Terminal | yes | no | yes | yes | yes | yes | diff --git a/contrib/python/termcolor/py3/.dist-info/top_level.txt b/contrib/python/termcolor/py3/.dist-info/top_level.txt deleted file mode 100644 index f08cca1411..0000000000 --- a/contrib/python/termcolor/py3/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -termcolor diff --git a/contrib/python/termcolor/py3/COPYING.txt b/contrib/python/termcolor/py3/COPYING.txt deleted file mode 100644 index d0b79705c3..0000000000 --- a/contrib/python/termcolor/py3/COPYING.txt +++ /dev/null @@ -1,19 +0,0 @@ -Copyright (c) 2008-2011 Volvox Development Team - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. diff --git a/contrib/python/termcolor/py3/README.md b/contrib/python/termcolor/py3/README.md deleted file mode 100644 index 7ccfd9f5b0..0000000000 --- a/contrib/python/termcolor/py3/README.md +++ /dev/null @@ -1,91 +0,0 @@ -# termcolor - -[](https://pypi.org/project/termcolor) -[](https://pypi.org/project/termcolor) -[](https://pypistats.org/packages/termcolor) -[](https://github.com/termcolor/termcolor/actions) -[](https://codecov.io/gh/termcolor/termcolor) -[](COPYING.txt) -[](https://github.com/psf/black) -[](https://tidelift.com/subscription/pkg/pypi-termcolor?utm_source=pypi-termcolor&utm_medium=referral&utm_campaign=readme) - -## Installation - -### From PyPI - -```bash -python3 -m pip install --upgrade termcolor -``` - -### From source - -```bash -git clone https://github.com/termcolor/termcolor -cd termcolor -python3 -m pip install . -``` - -### Demo - -To see demo output, run: - -```bash -python3 -m termcolor -``` - -## Example - -```python -import sys - -from termcolor import colored, cprint - -text = colored("Hello, World!", "red", attrs=["reverse", "blink"]) -print(text) -cprint("Hello, World!", "green", "on_red") - -print_red_on_cyan = lambda x: cprint(x, "red", "on_cyan") -print_red_on_cyan("Hello, World!") -print_red_on_cyan("Hello, Universe!") - -for i in range(10): - cprint(i, "magenta", end=" ") - -cprint("Attention!", "red", attrs=["bold"], file=sys.stderr) -``` - -## Text properties - -| Text colors | Text highlights | Attributes | -| --------------- | ------------------ | ----------- | -| `black` | `on_black` | `bold` | -| `red` | `on_red` | `dark` | -| `green` | `on_green` | `underline` | -| `yellow` | `on_yellow` | `blink` | -| `blue` | `on_blue` | `reverse` | -| `magenta` | `on_magenta` | `concealed` | -| `cyan` | `on_cyan` | | -| `white` | `on_white` | | -| `light_grey` | `on_light_grey` | | -| `dark_grey` | `on_dark_grey` | | -| `light_red` | `on_light_red` | | -| `light_green` | `on_light_green` | | -| `light_yellow` | `on_light_yellow` | | -| `light_blue` | `on_light_blue` | | -| `light_magenta` | `on_light_magenta` | | -| `light_cyan` | `on_light_cyan` | | - -## Terminal properties - -| Terminal | bold | dark | underline | blink | reverse | concealed | -| ------------ | ------- | ---- | --------- | ---------- | ------- | --------- | -| xterm | yes | no | yes | bold | yes | yes | -| linux | yes | yes | bold | yes | yes | no | -| rxvt | yes | no | yes | bold/black | yes | no | -| dtterm | yes | yes | yes | reverse | yes | yes | -| teraterm | reverse | no | yes | rev/red | yes | no | -| aixterm | normal | no | yes | no | yes | yes | -| PuTTY | color | no | yes | no | yes | no | -| Windows | no | no | no | no | yes | no | -| Cygwin SSH | yes | no | color | color | color | yes | -| Mac Terminal | yes | no | yes | yes | yes | yes | diff --git a/contrib/python/termcolor/py3/termcolor/__init__.py b/contrib/python/termcolor/py3/termcolor/__init__.py deleted file mode 100644 index 86ace41518..0000000000 --- a/contrib/python/termcolor/py3/termcolor/__init__.py +++ /dev/null @@ -1,13 +0,0 @@ -"""ANSI color formatting for output in terminal.""" -from __future__ import annotations - -from termcolor.termcolor import ATTRIBUTES, COLORS, HIGHLIGHTS, RESET, colored, cprint - -__all__ = [ - "ATTRIBUTES", - "COLORS", - "HIGHLIGHTS", - "RESET", - "colored", - "cprint", -] diff --git a/contrib/python/termcolor/py3/termcolor/__main__.py b/contrib/python/termcolor/py3/termcolor/__main__.py deleted file mode 100644 index c0620ab389..0000000000 --- a/contrib/python/termcolor/py3/termcolor/__main__.py +++ /dev/null @@ -1,68 +0,0 @@ -from __future__ import annotations - -import os - -from termcolor import cprint - -if __name__ == "__main__": - print(f"Current terminal type: {os.getenv('TERM')}") - print("Test basic colors:") - cprint("Black color", "black") - cprint("Red color", "red") - cprint("Green color", "green") - cprint("Yellow color", "yellow") - cprint("Blue color", "blue") - cprint("Magenta color", "magenta") - cprint("Cyan color", "cyan") - cprint("White color", "white") - cprint("Light grey color", "light_grey") - cprint("Dark grey color", "dark_grey") - cprint("Light red color", "light_red") - cprint("Light green color", "light_green") - cprint("Light yellow color", "light_yellow") - cprint("Light blue color", "light_blue") - cprint("Light magenta color", "light_magenta") - cprint("Light cyan color", "light_cyan") - print("-" * 78) - - print("Test highlights:") - cprint("On black color", on_color="on_black") - cprint("On red color", on_color="on_red") - cprint("On green color", on_color="on_green") - cprint("On yellow color", on_color="on_yellow") - cprint("On blue color", on_color="on_blue") - cprint("On magenta color", on_color="on_magenta") - cprint("On cyan color", on_color="on_cyan") - cprint("On white color", color="black", on_color="on_white") - cprint("On light grey color", on_color="on_light_grey") - cprint("On dark grey color", on_color="on_dark_grey") - cprint("On light red color", on_color="on_light_red") - cprint("On light green color", on_color="on_light_green") - cprint("On light yellow color", on_color="on_light_yellow") - cprint("On light blue color", on_color="on_light_blue") - cprint("On light magenta color", on_color="on_light_magenta") - cprint("On light cyan color", on_color="on_light_cyan") - print("-" * 78) - - print("Test attributes:") - cprint("Bold black color", "black", attrs=["bold"]) - cprint("Dark red color", "red", attrs=["dark"]) - cprint("Underline green color", "green", attrs=["underline"]) - cprint("Blink yellow color", "yellow", attrs=["blink"]) - cprint("Reversed blue color", "blue", attrs=["reverse"]) - cprint("Concealed Magenta color", "magenta", attrs=["concealed"]) - cprint( - "Bold underline reverse cyan color", - "cyan", - attrs=["bold", "underline", "reverse"], - ) - cprint( - "Dark blink concealed white color", - "white", - attrs=["dark", "blink", "concealed"], - ) - print("-" * 78) - - print("Test mixing:") - cprint("Underline red on black color", "red", "on_black", ["underline"]) - cprint("Reversed green on red color", "green", "on_red", ["reverse"]) diff --git a/contrib/python/termcolor/py3/termcolor/py.typed b/contrib/python/termcolor/py3/termcolor/py.typed deleted file mode 100644 index e69de29bb2..0000000000 --- a/contrib/python/termcolor/py3/termcolor/py.typed +++ /dev/null diff --git a/contrib/python/termcolor/py3/termcolor/termcolor.py b/contrib/python/termcolor/py3/termcolor/termcolor.py deleted file mode 100644 index 2c8db2dc4e..0000000000 --- a/contrib/python/termcolor/py3/termcolor/termcolor.py +++ /dev/null @@ -1,201 +0,0 @@ -# Copyright (c) 2008-2011 Volvox Development Team -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -# THE SOFTWARE. -# -# Author: Konstantin Lepa <konstantin.lepa@gmail.com> - -"""ANSI color formatting for output in terminal.""" - -from __future__ import annotations - -import os -import sys -import warnings -from typing import Any, Iterable - - -def __getattr__(name: str) -> list[str]: - if name == "__ALL__": - warnings.warn( - "__ALL__ is deprecated and will be removed in termcolor 3. " - "Use __all__ instead.", - DeprecationWarning, - stacklevel=2, - ) - return ["colored", "cprint"] - msg = f"module '{__name__}' has no attribute '{name}'" - raise AttributeError(msg) - - -ATTRIBUTES = { - "bold": 1, - "dark": 2, - "underline": 4, - "blink": 5, - "reverse": 7, - "concealed": 8, -} - - -HIGHLIGHTS = { - "on_black": 40, - "on_grey": 40, # Actually black but kept for backwards compatibility - "on_red": 41, - "on_green": 42, - "on_yellow": 43, - "on_blue": 44, - "on_magenta": 45, - "on_cyan": 46, - "on_light_grey": 47, - "on_dark_grey": 100, - "on_light_red": 101, - "on_light_green": 102, - "on_light_yellow": 103, - "on_light_blue": 104, - "on_light_magenta": 105, - "on_light_cyan": 106, - "on_white": 107, -} - -COLORS = { - "black": 30, - "grey": 30, # Actually black but kept for backwards compatibility - "red": 31, - "green": 32, - "yellow": 33, - "blue": 34, - "magenta": 35, - "cyan": 36, - "light_grey": 37, - "dark_grey": 90, - "light_red": 91, - "light_green": 92, - "light_yellow": 93, - "light_blue": 94, - "light_magenta": 95, - "light_cyan": 96, - "white": 97, -} - - -RESET = "\033[0m" - - -def _can_do_colour( - *, no_color: bool | None = None, force_color: bool | None = None -) -> bool: - """Check env vars and for tty/dumb terminal""" - # First check overrides: - # "User-level configuration files and per-instance command-line arguments should - # override $NO_COLOR. A user should be able to export $NO_COLOR in their shell - # configuration file as a default, but configure a specific program in its - # configuration file to specifically enable color." - # https://no-color.org - if no_color is not None and no_color: - return False - if force_color is not None and force_color: - return True - - # Then check env vars: - if "ANSI_COLORS_DISABLED" in os.environ: - return False - if "NO_COLOR" in os.environ: - return False - if "FORCE_COLOR" in os.environ: - return True - return ( - hasattr(sys.stdout, "isatty") - and sys.stdout.isatty() - and os.environ.get("TERM") != "dumb" - ) - - -def colored( - text: str, - color: str | None = None, - on_color: str | None = None, - attrs: Iterable[str] | None = None, - *, - no_color: bool | None = None, - force_color: bool | None = None, -) -> str: - """Colorize text. - - Available text colors: - black, red, green, yellow, blue, magenta, cyan, white, - light_grey, dark_grey, light_red, light_green, light_yellow, light_blue, - light_magenta, light_cyan. - - Available text highlights: - on_black, on_red, on_green, on_yellow, on_blue, on_magenta, on_cyan, on_white, - on_light_grey, on_dark_grey, on_light_red, on_light_green, on_light_yellow, - on_light_blue, on_light_magenta, on_light_cyan. - - Available attributes: - bold, dark, underline, blink, reverse, concealed. - - Example: - colored('Hello, World!', 'red', 'on_black', ['bold', 'blink']) - colored('Hello, World!', 'green') - """ - if not _can_do_colour(no_color=no_color, force_color=force_color): - return text - - fmt_str = "\033[%dm%s" - if color is not None: - text = fmt_str % (COLORS[color], text) - - if on_color is not None: - text = fmt_str % (HIGHLIGHTS[on_color], text) - - if attrs is not None: - for attr in attrs: - text = fmt_str % (ATTRIBUTES[attr], text) - - return text + RESET - - -def cprint( - text: str, - color: str | None = None, - on_color: str | None = None, - attrs: Iterable[str] | None = None, - *, - no_color: bool | None = None, - force_color: bool | None = None, - **kwargs: Any, -) -> None: - """Print colorized text. - - It accepts arguments of print function. - """ - - print( - ( - colored( - text, - color, - on_color, - attrs, - no_color=no_color, - force_color=force_color, - ) - ), - **kwargs, - ) diff --git a/contrib/python/termcolor/py3/ya.make b/contrib/python/termcolor/py3/ya.make deleted file mode 100644 index afbfca8bd5..0000000000 --- a/contrib/python/termcolor/py3/ya.make +++ /dev/null @@ -1,25 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY3_LIBRARY() - -VERSION(2.3.0) - -LICENSE(MIT) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - termcolor/__init__.py - termcolor/__main__.py - termcolor/termcolor.py -) - -RESOURCE_FILES( - PREFIX contrib/python/termcolor/py3/ - .dist-info/METADATA - .dist-info/top_level.txt - termcolor/py.typed -) - -END() diff --git a/contrib/python/termcolor/ya.make b/contrib/python/termcolor/ya.make deleted file mode 100644 index 7f7866e83f..0000000000 --- a/contrib/python/termcolor/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/termcolor/py2) -ELSE() - PEERDIR(contrib/python/termcolor/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/toolz/py2/.dist-info/METADATA b/contrib/python/toolz/py2/.dist-info/METADATA deleted file mode 100644 index c43bc308d4..0000000000 --- a/contrib/python/toolz/py2/.dist-info/METADATA +++ /dev/null @@ -1,159 +0,0 @@ -Metadata-Version: 2.1 -Name: toolz -Version: 0.10.0 -Summary: List processing tools and functional utilities -Home-page: https://github.com/pytoolz/toolz/ -Author: https://raw.github.com/pytoolz/toolz/master/AUTHORS.md -Maintainer: Matthew Rocklin -Maintainer-email: mrocklin@gmail.com -License: BSD -Keywords: functional utility itertools functools -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: License :: OSI Approved :: BSD License -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* - -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies, partial - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 2.7 and Python 3.4+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://travis-ci.org/pytoolz/toolz.svg?branch=master - :target: https://travis-ci.org/pytoolz/toolz -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz - - diff --git a/contrib/python/toolz/py2/.dist-info/top_level.txt b/contrib/python/toolz/py2/.dist-info/top_level.txt deleted file mode 100644 index e58ef014ac..0000000000 --- a/contrib/python/toolz/py2/.dist-info/top_level.txt +++ /dev/null @@ -1,2 +0,0 @@ -tlz -toolz diff --git a/contrib/python/toolz/py2/LICENSE.txt b/contrib/python/toolz/py2/LICENSE.txt deleted file mode 100644 index eeb91b202c..0000000000 --- a/contrib/python/toolz/py2/LICENSE.txt +++ /dev/null @@ -1,28 +0,0 @@ -Copyright (c) 2013 Matthew Rocklin - -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - - a. Redistributions of source code must retain the above copyright notice, - this list of conditions and the following disclaimer. - b. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - c. Neither the name of toolz nor the names of its contributors - may be used to endorse or promote products derived from this software - without specific prior written permission. - - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR -ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL -DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR -SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER -CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY -OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH -DAMAGE. diff --git a/contrib/python/toolz/py2/README.rst b/contrib/python/toolz/py2/README.rst deleted file mode 100644 index 099c3ff807..0000000000 --- a/contrib/python/toolz/py2/README.rst +++ /dev/null @@ -1,132 +0,0 @@ -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies, partial - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 2.7 and Python 3.4+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://travis-ci.org/pytoolz/toolz.svg?branch=master - :target: https://travis-ci.org/pytoolz/toolz -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz diff --git a/contrib/python/toolz/py2/tlz/__init__.py b/contrib/python/toolz/py2/tlz/__init__.py deleted file mode 100644 index 9c9c84afe1..0000000000 --- a/contrib/python/toolz/py2/tlz/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -"""``tlz`` mirrors the ``toolz`` API and uses ``cytoolz`` if possible. - -The ``tlz`` package is installed when ``toolz`` is installed. It provides -a convenient way to use functions from ``cytoolz``--a faster Cython -implementation of ``toolz``--if it is installed, otherwise it uses -functions from ``toolz``. -""" - -from . import _build_tlz diff --git a/contrib/python/toolz/py2/tlz/_build_tlz.py b/contrib/python/toolz/py2/tlz/_build_tlz.py deleted file mode 100644 index 3c017a542c..0000000000 --- a/contrib/python/toolz/py2/tlz/_build_tlz.py +++ /dev/null @@ -1,100 +0,0 @@ -import sys -import types -import toolz -from importlib import import_module - - -class TlzLoader(object): - """ Finds and loads ``tlz`` modules when added to sys.meta_path""" - def __init__(self): - self.always_from_toolz = { - toolz.pipe, - } - - def _load_toolz(self, fullname): - rv = {} - package, dot, submodules = fullname.partition('.') - try: - module_name = ''.join(['cytoolz', dot, submodules]) - rv['cytoolz'] = import_module(module_name) - except ImportError: - pass - try: - module_name = ''.join(['toolz', dot, submodules]) - rv['toolz'] = import_module(module_name) - except ImportError: - pass - if not rv: - raise ImportError(fullname) - return rv - - def find_module(self, fullname, path=None): # pragma: py3 no cover - package, dot, submodules = fullname.partition('.') - if package == 'tlz': - return self - - def load_module(self, fullname): # pragma: py3 no cover - if fullname in sys.modules: # pragma: no cover - return sys.modules[fullname] - spec = TlzSpec(fullname, self) - module = self.create_module(spec) - sys.modules[fullname] = module - self.exec_module(module) - return module - - def find_spec(self, fullname, path, target=None): # pragma: no cover - package, dot, submodules = fullname.partition('.') - if package == 'tlz': - return TlzSpec(fullname, self) - - def create_module(self, spec): - return types.ModuleType(spec.name) - - def exec_module(self, module): - toolz_mods = self._load_toolz(module.__name__) - fast_mod = toolz_mods.get('cytoolz') or toolz_mods['toolz'] - slow_mod = toolz_mods.get('toolz') or toolz_mods['cytoolz'] - module.__dict__.update(toolz.merge(fast_mod.__dict__, module.__dict__)) - package = fast_mod.__package__ - if package is not None: - package, dot, submodules = package.partition('.') - module.__package__ = ''.join(['tlz', dot, submodules]) - if not module.__doc__: - module.__doc__ = fast_mod.__doc__ - - # show file from toolz during introspection - module.__file__ = slow_mod.__file__ - - for k, v in fast_mod.__dict__.items(): - tv = slow_mod.__dict__.get(k) - try: - hash(tv) - except TypeError: - tv = None - if tv in self.always_from_toolz: - module.__dict__[k] = tv - elif ( - isinstance(v, types.ModuleType) - and v.__package__ == fast_mod.__name__ - ): - package, dot, submodules = v.__name__.partition('.') - module_name = ''.join(['tlz', dot, submodules]) - submodule = import_module(module_name) - module.__dict__[k] = submodule - - -class TlzSpec(object): - def __init__(self, name, loader): - self.name = name - self.loader = loader - self.origin = None - self.submodule_search_locations = [] - self.loader_state = None - self.cached = None - self.parent = None - self.has_location = False - - -tlz_loader = TlzLoader() -sys.meta_path.append(tlz_loader) -tlz_loader.exec_module(sys.modules['tlz']) diff --git a/contrib/python/toolz/py2/toolz/__init__.py b/contrib/python/toolz/py2/toolz/__init__.py deleted file mode 100644 index 7fa86ab473..0000000000 --- a/contrib/python/toolz/py2/toolz/__init__.py +++ /dev/null @@ -1,22 +0,0 @@ -from .itertoolz import * - -from .functoolz import * - -from .dicttoolz import * - -from .recipes import * - -from .compatibility import map, filter - -from functools import partial, reduce - -sorted = sorted - -# Aliases -comp = compose - -from . import curried, sandbox - -functoolz._sigs.create_signature_registry() - -__version__ = '0.10.0' diff --git a/contrib/python/toolz/py2/toolz/_signatures.py b/contrib/python/toolz/py2/toolz/_signatures.py deleted file mode 100644 index c55a778b3b..0000000000 --- a/contrib/python/toolz/py2/toolz/_signatures.py +++ /dev/null @@ -1,832 +0,0 @@ -"""Internal module for better introspection of builtins. - -The main functions are ``is_builtin_valid_args``, ``is_builtin_partial_args``, -and ``has_unknown_args``. Other functions in this module support these three. - -Notably, we create a ``signatures`` registry to enable introspection of -builtin functions in any Python version. This includes builtins that -have more than one valid signature. Currently, the registry includes -builtins from ``builtins``, ``functools``, ``itertools``, and ``operator`` -modules. More can be added as requested. We don't guarantee full coverage. - -Everything in this module should be regarded as implementation details. -Users should try to not use this module directly. -""" -import functools -import inspect -import itertools -import operator -from importlib import import_module - -from .compatibility import PY3 -from .functoolz import (is_partial_args, is_arity, has_varargs, - has_keywords, num_required_args) - -if PY3: # pragma: py2 no cover - import builtins -else: # pragma: py3 no cover - import __builtin__ as builtins - -# We mock builtin callables using lists of tuples with lambda functions. -# -# The tuple spec is (num_position_args, lambda_func, keyword_only_args). -# -# num_position_args: -# - The number of positional-only arguments. If not specified, -# all positional arguments are considered positional-only. -# -# lambda_func: -# - lambda function that matches a signature of a builtin, but does -# not include keyword-only arguments. -# -# keyword_only_args: (optional) -# - Tuple of keyword-only argumemts. - -module_info = {} - -module_info[builtins] = dict( - abs=[ - lambda x: None], - all=[ - lambda iterable: None], - any=[ - lambda iterable: None], - apply=[ - lambda object: None, - lambda object, args: None, - lambda object, args, kwargs: None], - ascii=[ - lambda obj: None], - bin=[ - lambda number: None], - bool=[ - lambda x=False: None], - buffer=[ - lambda object: None, - lambda object, offset: None, - lambda object, offset, size: None], - bytearray=[ - lambda: None, - lambda int: None, - lambda string, encoding='utf8', errors='strict': None], - callable=[ - lambda obj: None], - chr=[ - lambda i: None], - classmethod=[ - lambda function: None], - cmp=[ - lambda x, y: None], - coerce=[ - lambda x, y: None], - complex=[ - lambda real=0, imag=0: None], - delattr=[ - lambda obj, name: None], - dict=[ - lambda **kwargs: None, - lambda mapping, **kwargs: None], - dir=[ - lambda: None, - lambda object: None], - divmod=[ - lambda x, y: None], - enumerate=[ - (0, lambda iterable, start=0: None)], - eval=[ - lambda source: None, - lambda source, globals: None, - lambda source, globals, locals: None], - execfile=[ - lambda filename: None, - lambda filename, globals: None, - lambda filename, globals, locals: None], - file=[ - (0, lambda name, mode='r', buffering=-1: None)], - filter=[ - lambda function, iterable: None], - float=[ - lambda x=0.0: None], - format=[ - lambda value: None, - lambda value, format_spec: None], - frozenset=[ - lambda: None, - lambda iterable: None], - getattr=[ - lambda object, name: None, - lambda object, name, default: None], - globals=[ - lambda: None], - hasattr=[ - lambda obj, name: None], - hash=[ - lambda obj: None], - hex=[ - lambda number: None], - id=[ - lambda obj: None], - input=[ - lambda: None, - lambda prompt: None], - int=[ - lambda x=0: None, - (0, lambda x, base=10: None)], - intern=[ - lambda string: None], - isinstance=[ - lambda obj, class_or_tuple: None], - issubclass=[ - lambda cls, class_or_tuple: None], - iter=[ - lambda iterable: None, - lambda callable, sentinel: None], - len=[ - lambda obj: None], - list=[ - lambda: None, - lambda iterable: None], - locals=[ - lambda: None], - long=[ - lambda x=0: None, - (0, lambda x, base=10: None)], - map=[ - lambda func, sequence, *iterables: None], - memoryview=[ - (0, lambda object: None)], - next=[ - lambda iterator: None, - lambda iterator, default: None], - object=[ - lambda: None], - oct=[ - lambda number: None], - ord=[ - lambda c: None], - pow=[ - lambda x, y: None, - lambda x, y, z: None], - property=[ - lambda fget=None, fset=None, fdel=None, doc=None: None], - range=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - raw_input=[ - lambda: None, - lambda prompt: None], - reduce=[ - lambda function, sequence: None, - lambda function, sequence, initial: None], - reload=[ - lambda module: None], - repr=[ - lambda obj: None], - reversed=[ - lambda sequence: None], - round=[ - (0, lambda number, ndigits=0: None)], - set=[ - lambda: None, - lambda iterable: None], - setattr=[ - lambda obj, name, value: None], - slice=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - staticmethod=[ - lambda function: None], - sum=[ - lambda iterable: None, - lambda iterable, start: None], - super=[ - lambda type: None, - lambda type, obj: None], - tuple=[ - lambda: None, - lambda iterable: None], - type=[ - lambda object: None, - lambda name, bases, dict: None], - unichr=[ - lambda i: None], - unicode=[ - lambda object: None, - lambda string='', encoding='utf8', errors='strict': None], - vars=[ - lambda: None, - lambda object: None], - xrange=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - zip=[ - lambda *iterables: None], - __build_class__=[ - (2, lambda func, name, *bases, **kwds: None, ('metaclass',))], - __import__=[ - (0, lambda name, globals=None, locals=None, fromlist=None, - level=None: None)], -) -module_info[builtins]['exec'] = [ - lambda source: None, - lambda source, globals: None, - lambda source, globals, locals: None] - -if PY3: # pragma: py2 no cover - module_info[builtins].update( - breakpoint=[ - lambda *args, **kws: None], - bytes=[ - lambda: None, - lambda int: None, - lambda string, encoding='utf8', errors='strict': None], - compile=[ - (0, lambda source, filename, mode, flags=0, - dont_inherit=False, optimize=-1: None)], - max=[ - (1, lambda iterable: None, ('default', 'key',)), - (1, lambda arg1, arg2, *args: None, ('key',))], - min=[ - (1, lambda iterable: None, ('default', 'key',)), - (1, lambda arg1, arg2, *args: None, ('key',))], - open=[ - (0, lambda file, mode='r', buffering=-1, encoding=None, - errors=None, newline=None, closefd=True, opener=None: None)], - sorted=[ - (1, lambda iterable: None, ('key', 'reverse'))], - str=[ - lambda object='', encoding='utf', errors='strict': None], - ) - module_info[builtins]['print'] = [ - (0, lambda *args: None, ('sep', 'end', 'file', 'flush',))] - -else: # pragma: py3 no cover - module_info[builtins].update( - bytes=[ - lambda object='': None], - compile=[ - (0, lambda source, filename, mode, flags=0, - dont_inherit=False: None)], - max=[ - (1, lambda iterable, *args: None, ('key',))], - min=[ - (1, lambda iterable, *args: None, ('key',))], - open=[ - (0, lambda file, mode='r', buffering=-1: None)], - sorted=[ - lambda iterable, cmp=None, key=None, reverse=False: None], - str=[ - lambda object='': None], - ) - module_info[builtins]['print'] = [ - (0, lambda *args: None, ('sep', 'end', 'file',))] - -module_info[functools] = dict( - cmp_to_key=[ - (0, lambda mycmp: None)], - partial=[ - lambda func, *args, **kwargs: None], - partialmethod=[ - lambda func, *args, **kwargs: None], - reduce=[ - lambda function, sequence: None, - lambda function, sequence, initial: None], -) - -module_info[itertools] = dict( - accumulate=[ - (0, lambda iterable, func=None: None)], - chain=[ - lambda *iterables: None], - combinations=[ - (0, lambda iterable, r: None)], - combinations_with_replacement=[ - (0, lambda iterable, r: None)], - compress=[ - (0, lambda data, selectors: None)], - count=[ - lambda start=0, step=1: None], - cycle=[ - lambda iterable: None], - dropwhile=[ - lambda predicate, iterable: None], - filterfalse=[ - lambda function, sequence: None], - groupby=[ - (0, lambda iterable, key=None: None)], - ifilter=[ - lambda function, sequence: None], - ifilterfalse=[ - lambda function, sequence: None], - imap=[ - lambda func, sequence, *iterables: None], - islice=[ - lambda iterable, stop: None, - lambda iterable, start, stop: None, - lambda iterable, start, stop, step: None], - izip=[ - lambda *iterables: None], - izip_longest=[ - (0, lambda *iterables: None, ('fillvalue',))], - permutations=[ - (0, lambda iterable, r=0: None)], - repeat=[ - (0, lambda object, times=0: None)], - starmap=[ - lambda function, sequence: None], - takewhile=[ - lambda predicate, iterable: None], - tee=[ - lambda iterable: None, - lambda iterable, n: None], - zip_longest=[ - (0, lambda *iterables: None, ('fillvalue',))], -) - -if PY3: # pragma: py2 no cover - module_info[itertools].update( - product=[ - (0, lambda *iterables: None, ('repeat',))], - ) -else: # pragma: py3 no cover - module_info[itertools].update( - product=[ - lambda *iterables: None], - ) - -module_info[operator] = dict( - __abs__=[ - lambda a: None], - __add__=[ - lambda a, b: None], - __and__=[ - lambda a, b: None], - __concat__=[ - lambda a, b: None], - __contains__=[ - lambda a, b: None], - __delitem__=[ - lambda a, b: None], - __delslice__=[ - lambda a, b, c: None], - __div__=[ - lambda a, b: None], - __eq__=[ - lambda a, b: None], - __floordiv__=[ - lambda a, b: None], - __ge__=[ - lambda a, b: None], - __getitem__=[ - lambda a, b: None], - __getslice__=[ - lambda a, b, c: None], - __gt__=[ - lambda a, b: None], - __iadd__=[ - lambda a, b: None], - __iand__=[ - lambda a, b: None], - __iconcat__=[ - lambda a, b: None], - __idiv__=[ - lambda a, b: None], - __ifloordiv__=[ - lambda a, b: None], - __ilshift__=[ - lambda a, b: None], - __imatmul__=[ - lambda a, b: None], - __imod__=[ - lambda a, b: None], - __imul__=[ - lambda a, b: None], - __index__=[ - lambda a: None], - __inv__=[ - lambda a: None], - __invert__=[ - lambda a: None], - __ior__=[ - lambda a, b: None], - __ipow__=[ - lambda a, b: None], - __irepeat__=[ - lambda a, b: None], - __irshift__=[ - lambda a, b: None], - __isub__=[ - lambda a, b: None], - __itruediv__=[ - lambda a, b: None], - __ixor__=[ - lambda a, b: None], - __le__=[ - lambda a, b: None], - __lshift__=[ - lambda a, b: None], - __lt__=[ - lambda a, b: None], - __matmul__=[ - lambda a, b: None], - __mod__=[ - lambda a, b: None], - __mul__=[ - lambda a, b: None], - __ne__=[ - lambda a, b: None], - __neg__=[ - lambda a: None], - __not__=[ - lambda a: None], - __or__=[ - lambda a, b: None], - __pos__=[ - lambda a: None], - __pow__=[ - lambda a, b: None], - __repeat__=[ - lambda a, b: None], - __rshift__=[ - lambda a, b: None], - __setitem__=[ - lambda a, b, c: None], - __setslice__=[ - lambda a, b, c, d: None], - __sub__=[ - lambda a, b: None], - __truediv__=[ - lambda a, b: None], - __xor__=[ - lambda a, b: None], - _abs=[ - lambda x: None], - _compare_digest=[ - lambda a, b: None], - abs=[ - lambda a: None], - add=[ - lambda a, b: None], - and_=[ - lambda a, b: None], - attrgetter=[ - lambda attr, *args: None], - concat=[ - lambda a, b: None], - contains=[ - lambda a, b: None], - countOf=[ - lambda a, b: None], - delitem=[ - lambda a, b: None], - delslice=[ - lambda a, b, c: None], - div=[ - lambda a, b: None], - eq=[ - lambda a, b: None], - floordiv=[ - lambda a, b: None], - ge=[ - lambda a, b: None], - getitem=[ - lambda a, b: None], - getslice=[ - lambda a, b, c: None], - gt=[ - lambda a, b: None], - iadd=[ - lambda a, b: None], - iand=[ - lambda a, b: None], - iconcat=[ - lambda a, b: None], - idiv=[ - lambda a, b: None], - ifloordiv=[ - lambda a, b: None], - ilshift=[ - lambda a, b: None], - imatmul=[ - lambda a, b: None], - imod=[ - lambda a, b: None], - imul=[ - lambda a, b: None], - index=[ - lambda a: None], - indexOf=[ - lambda a, b: None], - inv=[ - lambda a: None], - invert=[ - lambda a: None], - ior=[ - lambda a, b: None], - ipow=[ - lambda a, b: None], - irepeat=[ - lambda a, b: None], - irshift=[ - lambda a, b: None], - is_=[ - lambda a, b: None], - is_not=[ - lambda a, b: None], - isCallable=[ - lambda a: None], - isMappingType=[ - lambda a: None], - isNumberType=[ - lambda a: None], - isSequenceType=[ - lambda a: None], - isub=[ - lambda a, b: None], - itemgetter=[ - lambda item, *args: None], - itruediv=[ - lambda a, b: None], - ixor=[ - lambda a, b: None], - le=[ - lambda a, b: None], - length_hint=[ - lambda obj: None, - lambda obj, default: None], - lshift=[ - lambda a, b: None], - lt=[ - lambda a, b: None], - matmul=[ - lambda a, b: None], - methodcaller=[ - lambda name, *args, **kwargs: None], - mod=[ - lambda a, b: None], - mul=[ - lambda a, b: None], - ne=[ - lambda a, b: None], - neg=[ - lambda a: None], - not_=[ - lambda a: None], - or_=[ - lambda a, b: None], - pos=[ - lambda a: None], - pow=[ - lambda a, b: None], - repeat=[ - lambda a, b: None], - rshift=[ - lambda a, b: None], - sequenceIncludes=[ - lambda a, b: None], - setitem=[ - lambda a, b, c: None], - setslice=[ - lambda a, b, c, d: None], - sub=[ - lambda a, b: None], - truediv=[ - lambda a, b: None], - truth=[ - lambda a: None], - xor=[ - lambda a, b: None], -) - -module_info['toolz'] = dict( - curry=[ - (0, lambda *args, **kwargs: None)], - excepts=[ - (0, lambda exc, func, handler=None: None)], - flip=[ - (0, lambda func=None, a=None, b=None: None)], - juxt=[ - (0, lambda *funcs: None)], - memoize=[ - (0, lambda func=None, cache=None, key=None: None)], -) - -module_info['toolz.functoolz'] = dict( - Compose=[ - (0, lambda funcs: None)], - InstanceProperty=[ - (0, lambda fget=None, fset=None, fdel=None, doc=None, - classval=None: None)], -) - -if PY3: # pragma: py2 no cover - def num_pos_args(sigspec): - """ Return the number of positional arguments. ``f(x, y=1)`` has 1""" - return sum(1 for x in sigspec.parameters.values() - if x.kind == x.POSITIONAL_OR_KEYWORD - and x.default is x.empty) - - def get_exclude_keywords(num_pos_only, sigspec): - """ Return the names of position-only arguments if func has **kwargs""" - if num_pos_only == 0: - return () - has_kwargs = any(x.kind == x.VAR_KEYWORD - for x in sigspec.parameters.values()) - if not has_kwargs: - return () - pos_args = list(sigspec.parameters.values())[:num_pos_only] - return tuple(x.name for x in pos_args) - - def signature_or_spec(func): - try: - return inspect.signature(func) - except (ValueError, TypeError): - return None - -else: # pragma: py3 no cover - def num_pos_args(sigspec): - """ Return the number of positional arguments. ``f(x, y=1)`` has 1""" - if sigspec.defaults: - return len(sigspec.args) - len(sigspec.defaults) - return len(sigspec.args) - - def get_exclude_keywords(num_pos_only, sigspec): - """ Return the names of position-only arguments if func has **kwargs""" - if num_pos_only == 0: - return () - has_kwargs = sigspec.keywords is not None - if not has_kwargs: - return () - return tuple(sigspec.args[:num_pos_only]) - - def signature_or_spec(func): - try: - return inspect.getargspec(func) - except TypeError: - return None - - -def expand_sig(sig): - """ Convert the signature spec in ``module_info`` to add to ``signatures`` - - The input signature spec is one of: - - ``lambda_func`` - - ``(num_position_args, lambda_func)`` - - ``(num_position_args, lambda_func, keyword_only_args)`` - - The output signature spec is: - ``(num_position_args, lambda_func, keyword_exclude, sigspec)`` - - where ``keyword_exclude`` includes keyword only arguments and, if variadic - keywords is present, the names of position-only argument. The latter is - included to support builtins such as ``partial(func, *args, **kwargs)``, - which allows ``func=`` to be used as a keyword even though it's the name - of a positional argument. - """ - if isinstance(sig, tuple): - if len(sig) == 3: - num_pos_only, func, keyword_only = sig - assert isinstance(sig[-1], tuple) - else: - num_pos_only, func = sig - keyword_only = () - sigspec = signature_or_spec(func) - else: - func = sig - sigspec = signature_or_spec(func) - num_pos_only = num_pos_args(sigspec) - keyword_only = () - keyword_exclude = get_exclude_keywords(num_pos_only, sigspec) - return num_pos_only, func, keyword_only + keyword_exclude, sigspec - - -signatures = {} - - -def create_signature_registry(module_info=module_info, signatures=signatures): - for module, info in module_info.items(): - if isinstance(module, str): - module = import_module(module) - for name, sigs in info.items(): - if hasattr(module, name): - new_sigs = tuple(expand_sig(sig) for sig in sigs) - signatures[getattr(module, name)] = new_sigs - - -def check_valid(sig, args, kwargs): - """ Like ``is_valid_args`` for the given signature spec""" - num_pos_only, func, keyword_exclude, sigspec = sig - if len(args) < num_pos_only: - return False - if keyword_exclude: - kwargs = dict(kwargs) - for item in keyword_exclude: - kwargs.pop(item, None) - try: - func(*args, **kwargs) - return True - except TypeError: - return False - - -def _is_valid_args(func, args, kwargs): - """ Like ``is_valid_args`` for builtins in our ``signatures`` registry""" - if func not in signatures: - return None - sigs = signatures[func] - return any(check_valid(sig, args, kwargs) for sig in sigs) - - -def check_partial(sig, args, kwargs): - """ Like ``is_partial_args`` for the given signature spec""" - num_pos_only, func, keyword_exclude, sigspec = sig - if len(args) < num_pos_only: - pad = (None,) * (num_pos_only - len(args)) - args = args + pad - if keyword_exclude: - kwargs = dict(kwargs) - for item in keyword_exclude: - kwargs.pop(item, None) - return is_partial_args(func, args, kwargs, sigspec) - - -def _is_partial_args(func, args, kwargs): - """ Like ``is_partial_args`` for builtins in our ``signatures`` registry""" - if func not in signatures: - return None - sigs = signatures[func] - return any(check_partial(sig, args, kwargs) for sig in sigs) - - -def check_arity(n, sig): - num_pos_only, func, keyword_exclude, sigspec = sig - if keyword_exclude or num_pos_only > n: - return False - return is_arity(n, func, sigspec) - - -def _is_arity(n, func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_arity(n, sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_varargs(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - return has_varargs(func, sigspec) - - -def _has_varargs(func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_varargs(sig) for sig in sigs] - if all(checks): - return True - elif any(checks): # pragma: py2 no cover - return None - return False - - -def check_keywords(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - if keyword_exclude: - return True - return has_keywords(func, sigspec) - - -def _has_keywords(func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_keywords(sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_required_args(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - return num_required_args(func, sigspec) - - -def _num_required_args(func): - if func not in signatures: - return None - sigs = signatures[func] - vals = [check_required_args(sig) for sig in sigs] - val = vals[0] - if all(x == val for x in vals): - return val - return None diff --git a/contrib/python/toolz/py2/toolz/compatibility.py b/contrib/python/toolz/py2/toolz/compatibility.py deleted file mode 100644 index 51e3673fad..0000000000 --- a/contrib/python/toolz/py2/toolz/compatibility.py +++ /dev/null @@ -1,34 +0,0 @@ -import operator -import sys -PY3 = sys.version_info[0] > 2 -PY34 = sys.version_info[0] == 3 and sys.version_info[1] == 4 -PYPY = hasattr(sys, 'pypy_version_info') - -__all__ = ('map', 'filter', 'range', 'zip', 'reduce', 'zip_longest', - 'iteritems', 'iterkeys', 'itervalues', 'filterfalse', - 'PY3', 'PY34', 'PYPY') - -if PY3: - map = map - filter = filter - range = range - zip = zip - from functools import reduce - from itertools import zip_longest - from itertools import filterfalse - iteritems = operator.methodcaller('items') - iterkeys = operator.methodcaller('keys') - itervalues = operator.methodcaller('values') - from collections.abc import Sequence -else: - range = xrange - reduce = reduce - from itertools import imap as map - from itertools import ifilter as filter - from itertools import ifilterfalse as filterfalse - from itertools import izip as zip - from itertools import izip_longest as zip_longest - iteritems = operator.methodcaller('iteritems') - iterkeys = operator.methodcaller('iterkeys') - itervalues = operator.methodcaller('itervalues') - from collections import Sequence diff --git a/contrib/python/toolz/py2/toolz/curried/__init__.py b/contrib/python/toolz/py2/toolz/curried/__init__.py deleted file mode 100644 index 356eddbd3b..0000000000 --- a/contrib/python/toolz/py2/toolz/curried/__init__.py +++ /dev/null @@ -1,103 +0,0 @@ -""" -Alternate namespace for toolz such that all functions are curried - -Currying provides implicit partial evaluation of all functions - -Example: - - Get usually requires two arguments, an index and a collection - >>> from toolz.curried import get - >>> get(0, ('a', 'b')) - 'a' - - When we use it in higher order functions we often want to pass a partially - evaluated form - >>> data = [(1, 2), (11, 22), (111, 222)] - >>> list(map(lambda seq: get(0, seq), data)) - [1, 11, 111] - - The curried version allows simple expression of partial evaluation - >>> list(map(get(0), data)) - [1, 11, 111] - -See Also: - toolz.functoolz.curry -""" -import toolz -from . import operator -from toolz import ( - apply, - comp, - complement, - compose, - compose_left, - concat, - concatv, - count, - curry, - diff, - first, - flip, - frequencies, - identity, - interleave, - isdistinct, - isiterable, - juxt, - last, - memoize, - merge_sorted, - peek, - pipe, - second, - thread_first, - thread_last, -) -from .exceptions import merge, merge_with - -accumulate = toolz.curry(toolz.accumulate) -assoc = toolz.curry(toolz.assoc) -assoc_in = toolz.curry(toolz.assoc_in) -cons = toolz.curry(toolz.cons) -countby = toolz.curry(toolz.countby) -dissoc = toolz.curry(toolz.dissoc) -do = toolz.curry(toolz.do) -drop = toolz.curry(toolz.drop) -excepts = toolz.curry(toolz.excepts) -filter = toolz.curry(toolz.filter) -get = toolz.curry(toolz.get) -get_in = toolz.curry(toolz.get_in) -groupby = toolz.curry(toolz.groupby) -interpose = toolz.curry(toolz.interpose) -itemfilter = toolz.curry(toolz.itemfilter) -itemmap = toolz.curry(toolz.itemmap) -iterate = toolz.curry(toolz.iterate) -join = toolz.curry(toolz.join) -keyfilter = toolz.curry(toolz.keyfilter) -keymap = toolz.curry(toolz.keymap) -map = toolz.curry(toolz.map) -mapcat = toolz.curry(toolz.mapcat) -nth = toolz.curry(toolz.nth) -partial = toolz.curry(toolz.partial) -partition = toolz.curry(toolz.partition) -partition_all = toolz.curry(toolz.partition_all) -partitionby = toolz.curry(toolz.partitionby) -peekn = toolz.curry(toolz.peekn) -pluck = toolz.curry(toolz.pluck) -random_sample = toolz.curry(toolz.random_sample) -reduce = toolz.curry(toolz.reduce) -reduceby = toolz.curry(toolz.reduceby) -remove = toolz.curry(toolz.remove) -sliding_window = toolz.curry(toolz.sliding_window) -sorted = toolz.curry(toolz.sorted) -tail = toolz.curry(toolz.tail) -take = toolz.curry(toolz.take) -take_nth = toolz.curry(toolz.take_nth) -topk = toolz.curry(toolz.topk) -unique = toolz.curry(toolz.unique) -update_in = toolz.curry(toolz.update_in) -valfilter = toolz.curry(toolz.valfilter) -valmap = toolz.curry(toolz.valmap) - -del exceptions -del toolz diff --git a/contrib/python/toolz/py2/toolz/curried/exceptions.py b/contrib/python/toolz/py2/toolz/curried/exceptions.py deleted file mode 100644 index 75a52bbbf2..0000000000 --- a/contrib/python/toolz/py2/toolz/curried/exceptions.py +++ /dev/null @@ -1,18 +0,0 @@ -import toolz - - -__all__ = ['merge_with', 'merge'] - - -@toolz.curry -def merge_with(func, d, *dicts, **kwargs): - return toolz.merge_with(func, d, *dicts, **kwargs) - - -@toolz.curry -def merge(d, *dicts, **kwargs): - return toolz.merge(d, *dicts, **kwargs) - - -merge_with.__doc__ = toolz.merge_with.__doc__ -merge.__doc__ = toolz.merge.__doc__ diff --git a/contrib/python/toolz/py2/toolz/curried/operator.py b/contrib/python/toolz/py2/toolz/curried/operator.py deleted file mode 100644 index 8bc9e52317..0000000000 --- a/contrib/python/toolz/py2/toolz/curried/operator.py +++ /dev/null @@ -1,23 +0,0 @@ -from __future__ import absolute_import - -import operator - -from toolz.functoolz import curry, num_required_args, has_keywords - - -def should_curry(f): - num = num_required_args(f) - return num is None or num > 1 or num == 1 and has_keywords(f) is not False - - -locals().update( - {name: curry(f) if should_curry(f) else f - for name, f in vars(operator).items() if callable(f)}, -) - -# Clean up the namespace. -del curry -del num_required_args -del has_keywords -del operator -del should_curry diff --git a/contrib/python/toolz/py2/toolz/dicttoolz.py b/contrib/python/toolz/py2/toolz/dicttoolz.py deleted file mode 100644 index 91bff23cef..0000000000 --- a/contrib/python/toolz/py2/toolz/dicttoolz.py +++ /dev/null @@ -1,337 +0,0 @@ -import operator -from toolz.compatibility import (map, zip, iteritems, iterkeys, itervalues, - reduce) - -__all__ = ('merge', 'merge_with', 'valmap', 'keymap', 'itemmap', - 'valfilter', 'keyfilter', 'itemfilter', - 'assoc', 'dissoc', 'assoc_in', 'update_in', 'get_in') - - -def _get_factory(f, kwargs): - factory = kwargs.pop('factory', dict) - if kwargs: - raise TypeError("{}() got an unexpected keyword argument " - "'{}'".format(f.__name__, kwargs.popitem()[0])) - return factory - - -def merge(*dicts, **kwargs): - """ Merge a collection of dictionaries - - >>> merge({1: 'one'}, {2: 'two'}) - {1: 'one', 2: 'two'} - - Later dictionaries have precedence - - >>> merge({1: 2, 3: 4}, {3: 3, 4: 4}) - {1: 2, 3: 3, 4: 4} - - See Also: - merge_with - """ - if len(dicts) == 1 and not isinstance(dicts[0], dict): - dicts = dicts[0] - factory = _get_factory(merge, kwargs) - - rv = factory() - for d in dicts: - rv.update(d) - return rv - - -def merge_with(func, *dicts, **kwargs): - """ Merge dictionaries and apply function to combined values - - A key may occur in more than one dict, and all values mapped from the key - will be passed to the function as a list, such as func([val1, val2, ...]). - - >>> merge_with(sum, {1: 1, 2: 2}, {1: 10, 2: 20}) - {1: 11, 2: 22} - - >>> merge_with(first, {1: 1, 2: 2}, {2: 20, 3: 30}) # doctest: +SKIP - {1: 1, 2: 2, 3: 30} - - See Also: - merge - """ - if len(dicts) == 1 and not isinstance(dicts[0], dict): - dicts = dicts[0] - factory = _get_factory(merge_with, kwargs) - - result = factory() - for d in dicts: - for k, v in iteritems(d): - if k not in result: - result[k] = [v] - else: - result[k].append(v) - return valmap(func, result, factory) - - -def valmap(func, d, factory=dict): - """ Apply function to values of dictionary - - >>> bills = {"Alice": [20, 15, 30], "Bob": [10, 35]} - >>> valmap(sum, bills) # doctest: +SKIP - {'Alice': 65, 'Bob': 45} - - See Also: - keymap - itemmap - """ - rv = factory() - rv.update(zip(iterkeys(d), map(func, itervalues(d)))) - return rv - - -def keymap(func, d, factory=dict): - """ Apply function to keys of dictionary - - >>> bills = {"Alice": [20, 15, 30], "Bob": [10, 35]} - >>> keymap(str.lower, bills) # doctest: +SKIP - {'alice': [20, 15, 30], 'bob': [10, 35]} - - See Also: - valmap - itemmap - """ - rv = factory() - rv.update(zip(map(func, iterkeys(d)), itervalues(d))) - return rv - - -def itemmap(func, d, factory=dict): - """ Apply function to items of dictionary - - >>> accountids = {"Alice": 10, "Bob": 20} - >>> itemmap(reversed, accountids) # doctest: +SKIP - {10: "Alice", 20: "Bob"} - - See Also: - keymap - valmap - """ - rv = factory() - rv.update(map(func, iteritems(d))) - return rv - - -def valfilter(predicate, d, factory=dict): - """ Filter items in dictionary by value - - >>> iseven = lambda x: x % 2 == 0 - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> valfilter(iseven, d) - {1: 2, 3: 4} - - See Also: - keyfilter - itemfilter - valmap - """ - rv = factory() - for k, v in iteritems(d): - if predicate(v): - rv[k] = v - return rv - - -def keyfilter(predicate, d, factory=dict): - """ Filter items in dictionary by key - - >>> iseven = lambda x: x % 2 == 0 - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> keyfilter(iseven, d) - {2: 3, 4: 5} - - See Also: - valfilter - itemfilter - keymap - """ - rv = factory() - for k, v in iteritems(d): - if predicate(k): - rv[k] = v - return rv - - -def itemfilter(predicate, d, factory=dict): - """ Filter items in dictionary by item - - >>> def isvalid(item): - ... k, v = item - ... return k % 2 == 0 and v < 4 - - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> itemfilter(isvalid, d) - {2: 3} - - See Also: - keyfilter - valfilter - itemmap - """ - rv = factory() - for item in iteritems(d): - if predicate(item): - k, v = item - rv[k] = v - return rv - - -def assoc(d, key, value, factory=dict): - """ Return a new dict with new key value pair - - New dict has d[key] set to value. Does not modify the initial dictionary. - - >>> assoc({'x': 1}, 'x', 2) - {'x': 2} - >>> assoc({'x': 1}, 'y', 3) # doctest: +SKIP - {'x': 1, 'y': 3} - """ - d2 = factory() - d2.update(d) - d2[key] = value - return d2 - - -def dissoc(d, *keys, **kwargs): - """ Return a new dict with the given key(s) removed. - - New dict has d[key] deleted for each supplied key. - Does not modify the initial dictionary. - - >>> dissoc({'x': 1, 'y': 2}, 'y') - {'x': 1} - >>> dissoc({'x': 1, 'y': 2}, 'y', 'x') - {} - >>> dissoc({'x': 1}, 'y') # Ignores missing keys - {'x': 1} - """ - factory = _get_factory(dissoc, kwargs) - d2 = factory() - - if len(keys) < len(d) * .6: - d2.update(d) - for key in keys: - if key in d2: - del d2[key] - else: - remaining = set(d) - remaining.difference_update(keys) - for k in remaining: - d2[k] = d[k] - return d2 - - -def assoc_in(d, keys, value, factory=dict): - """ Return a new dict with new, potentially nested, key value pair - - >>> purchase = {'name': 'Alice', - ... 'order': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> assoc_in(purchase, ['order', 'costs'], [0.25, 1.00]) # doctest: +SKIP - {'credit card': '5555-1234-1234-1234', - 'name': 'Alice', - 'order': {'costs': [0.25, 1.00], 'items': ['Apple', 'Orange']}} - """ - return update_in(d, keys, lambda x: value, value, factory) - - -def update_in(d, keys, func, default=None, factory=dict): - """ Update value in a (potentially) nested dictionary - - inputs: - d - dictionary on which to operate - keys - list or tuple giving the location of the value to be changed in d - func - function to operate on that value - - If keys == [k0,..,kX] and d[k0]..[kX] == v, update_in returns a copy of the - original dictionary with v replaced by func(v), but does not mutate the - original dictionary. - - If k0 is not a key in d, update_in creates nested dictionaries to the depth - specified by the keys, with the innermost value set to func(default). - - >>> inc = lambda x: x + 1 - >>> update_in({'a': 0}, ['a'], inc) - {'a': 1} - - >>> transaction = {'name': 'Alice', - ... 'purchase': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> update_in(transaction, ['purchase', 'costs'], sum) # doctest: +SKIP - {'credit card': '5555-1234-1234-1234', - 'name': 'Alice', - 'purchase': {'costs': 1.75, 'items': ['Apple', 'Orange']}} - - >>> # updating a value when k0 is not in d - >>> update_in({}, [1, 2, 3], str, default="bar") - {1: {2: {3: 'bar'}}} - >>> update_in({1: 'foo'}, [2, 3, 4], inc, 0) - {1: 'foo', 2: {3: {4: 1}}} - """ - ks = iter(keys) - k = next(ks) - - rv = inner = factory() - rv.update(d) - - for key in ks: - if k in d: - d = d[k] - dtemp = factory() - dtemp.update(d) - else: - d = dtemp = factory() - - inner[k] = inner = dtemp - k = key - - if k in d: - inner[k] = func(d[k]) - else: - inner[k] = func(default) - return rv - - -def get_in(keys, coll, default=None, no_default=False): - """ Returns coll[i0][i1]...[iX] where [i0, i1, ..., iX]==keys. - - If coll[i0][i1]...[iX] cannot be found, returns ``default``, unless - ``no_default`` is specified, then it raises KeyError or IndexError. - - ``get_in`` is a generalization of ``operator.getitem`` for nested data - structures such as dictionaries and lists. - - >>> transaction = {'name': 'Alice', - ... 'purchase': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> get_in(['purchase', 'items', 0], transaction) - 'Apple' - >>> get_in(['name'], transaction) - 'Alice' - >>> get_in(['purchase', 'total'], transaction) - >>> get_in(['purchase', 'items', 'apple'], transaction) - >>> get_in(['purchase', 'items', 10], transaction) - >>> get_in(['purchase', 'total'], transaction, 0) - 0 - >>> get_in(['y'], {}, no_default=True) - Traceback (most recent call last): - ... - KeyError: 'y' - - See Also: - itertoolz.get - operator.getitem - """ - try: - return reduce(operator.getitem, keys, coll) - except (KeyError, IndexError, TypeError): - if no_default: - raise - return default diff --git a/contrib/python/toolz/py2/toolz/functoolz.py b/contrib/python/toolz/py2/toolz/functoolz.py deleted file mode 100644 index 01d3857a19..0000000000 --- a/contrib/python/toolz/py2/toolz/functoolz.py +++ /dev/null @@ -1,1152 +0,0 @@ -from functools import reduce, partial -import inspect -import operator -from operator import attrgetter -from importlib import import_module -from textwrap import dedent -from types import MethodType - -from .compatibility import PY3, PY34, PYPY -from .utils import no_default - - -__all__ = ('identity', 'apply', 'thread_first', 'thread_last', 'memoize', - 'compose', 'compose_left', 'pipe', 'complement', 'juxt', 'do', - 'curry', 'flip', 'excepts') - - -def identity(x): - """ Identity function. Return x - - >>> identity(3) - 3 - """ - return x - - -def apply(*func_and_args, **kwargs): - """ Applies a function and returns the results - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> apply(double, 5) - 10 - - >>> tuple(map(apply, [double, inc, double], [10, 500, 8000])) - (20, 501, 16000) - """ - if not func_and_args: - raise TypeError('func argument is required') - func, args = func_and_args[0], func_and_args[1:] - return func(*args, **kwargs) - - -def thread_first(val, *forms): - """ Thread value through a sequence of functions/forms - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> thread_first(1, inc, double) - 4 - - If the function expects more than one input you can specify those inputs - in a tuple. The value is used as the first input. - - >>> def add(x, y): return x + y - >>> def pow(x, y): return x**y - >>> thread_first(1, (add, 4), (pow, 2)) # pow(add(1, 4), 2) - 25 - - So in general - thread_first(x, f, (g, y, z)) - expands to - g(f(x), y, z) - - See Also: - thread_last - """ - def evalform_front(val, form): - if callable(form): - return form(val) - if isinstance(form, tuple): - func, args = form[0], form[1:] - args = (val,) + args - return func(*args) - return reduce(evalform_front, forms, val) - - -def thread_last(val, *forms): - """ Thread value through a sequence of functions/forms - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> thread_last(1, inc, double) - 4 - - If the function expects more than one input you can specify those inputs - in a tuple. The value is used as the last input. - - >>> def add(x, y): return x + y - >>> def pow(x, y): return x**y - >>> thread_last(1, (add, 4), (pow, 2)) # pow(2, add(4, 1)) - 32 - - So in general - thread_last(x, f, (g, y, z)) - expands to - g(y, z, f(x)) - - >>> def iseven(x): - ... return x % 2 == 0 - >>> list(thread_last([1, 2, 3], (map, inc), (filter, iseven))) - [2, 4] - - See Also: - thread_first - """ - def evalform_back(val, form): - if callable(form): - return form(val) - if isinstance(form, tuple): - func, args = form[0], form[1:] - args = args + (val,) - return func(*args) - return reduce(evalform_back, forms, val) - - -def instanceproperty(fget=None, fset=None, fdel=None, doc=None, classval=None): - """ Like @property, but returns ``classval`` when used as a class attribute - - >>> class MyClass(object): - ... '''The class docstring''' - ... @instanceproperty(classval=__doc__) - ... def __doc__(self): - ... return 'An object docstring' - ... @instanceproperty - ... def val(self): - ... return 42 - ... - >>> MyClass.__doc__ - 'The class docstring' - >>> MyClass.val is None - True - >>> obj = MyClass() - >>> obj.__doc__ - 'An object docstring' - >>> obj.val - 42 - """ - if fget is None: - return partial(instanceproperty, fset=fset, fdel=fdel, doc=doc, - classval=classval) - return InstanceProperty(fget=fget, fset=fset, fdel=fdel, doc=doc, - classval=classval) - - -class InstanceProperty(property): - """ Like @property, but returns ``classval`` when used as a class attribute - - Should not be used directly. Use ``instanceproperty`` instead. - """ - def __init__(self, fget=None, fset=None, fdel=None, doc=None, - classval=None): - self.classval = classval - property.__init__(self, fget=fget, fset=fset, fdel=fdel, doc=doc) - - def __get__(self, obj, type=None): - if obj is None: - return self.classval - return property.__get__(self, obj, type) - - def __reduce__(self): - state = (self.fget, self.fset, self.fdel, self.__doc__, self.classval) - return InstanceProperty, state - - -class curry(object): - """ Curry a callable function - - Enables partial application of arguments through calling a function with an - incomplete set of arguments. - - >>> def mul(x, y): - ... return x * y - >>> mul = curry(mul) - - >>> double = mul(2) - >>> double(10) - 20 - - Also supports keyword arguments - - >>> @curry # Can use curry as a decorator - ... def f(x, y, a=10): - ... return a * (x + y) - - >>> add = f(a=1) - >>> add(2, 3) - 5 - - See Also: - toolz.curried - namespace of curried functions - https://toolz.readthedocs.io/en/latest/curry.html - """ - def __init__(self, *args, **kwargs): - if not args: - raise TypeError('__init__() takes at least 2 arguments (1 given)') - func, args = args[0], args[1:] - if not callable(func): - raise TypeError("Input must be callable") - - # curry- or functools.partial-like object? Unpack and merge arguments - if ( - hasattr(func, 'func') - and hasattr(func, 'args') - and hasattr(func, 'keywords') - and isinstance(func.args, tuple) - ): - _kwargs = {} - if func.keywords: - _kwargs.update(func.keywords) - _kwargs.update(kwargs) - kwargs = _kwargs - args = func.args + args - func = func.func - - if kwargs: - self._partial = partial(func, *args, **kwargs) - else: - self._partial = partial(func, *args) - - self.__doc__ = getattr(func, '__doc__', None) - self.__name__ = getattr(func, '__name__', '<curry>') - self.__module__ = getattr(func, '__module__', None) - self.__qualname__ = getattr(func, '__qualname__', None) - self._sigspec = None - self._has_unknown_args = None - - @instanceproperty - def func(self): - return self._partial.func - - if PY3: # pragma: py2 no cover - @instanceproperty - def __signature__(self): - sig = inspect.signature(self.func) - args = self.args or () - keywords = self.keywords or {} - if is_partial_args(self.func, args, keywords, sig) is False: - raise TypeError('curry object has incorrect arguments') - - params = list(sig.parameters.values()) - skip = 0 - for param in params[:len(args)]: - if param.kind == param.VAR_POSITIONAL: - break - skip += 1 - - kwonly = False - newparams = [] - for param in params[skip:]: - kind = param.kind - default = param.default - if kind == param.VAR_KEYWORD: - pass - elif kind == param.VAR_POSITIONAL: - if kwonly: - continue - elif param.name in keywords: - default = keywords[param.name] - kind = param.KEYWORD_ONLY - kwonly = True - else: - if kwonly: - kind = param.KEYWORD_ONLY - if default is param.empty: - default = no_default - newparams.append(param.replace(default=default, kind=kind)) - - return sig.replace(parameters=newparams) - - @instanceproperty - def args(self): - return self._partial.args - - @instanceproperty - def keywords(self): - return self._partial.keywords - - @instanceproperty - def func_name(self): - return self.__name__ - - def __str__(self): - return str(self.func) - - def __repr__(self): - return repr(self.func) - - def __hash__(self): - return hash((self.func, self.args, - frozenset(self.keywords.items()) if self.keywords - else None)) - - def __eq__(self, other): - return (isinstance(other, curry) and self.func == other.func and - self.args == other.args and self.keywords == other.keywords) - - def __ne__(self, other): - return not self.__eq__(other) - - def __call__(self, *args, **kwargs): - try: - return self._partial(*args, **kwargs) - except TypeError as exc: - if self._should_curry(args, kwargs, exc): - return self.bind(*args, **kwargs) - raise - - def _should_curry(self, args, kwargs, exc=None): - func = self.func - args = self.args + args - if self.keywords: - kwargs = dict(self.keywords, **kwargs) - if self._sigspec is None: - sigspec = self._sigspec = _sigs.signature_or_spec(func) - self._has_unknown_args = has_varargs(func, sigspec) is not False - else: - sigspec = self._sigspec - - if is_partial_args(func, args, kwargs, sigspec) is False: - # Nothing can make the call valid - return False - elif self._has_unknown_args: - # The call may be valid and raised a TypeError, but we curry - # anyway because the function may have `*args`. This is useful - # for decorators with signature `func(*args, **kwargs)`. - return True - elif not is_valid_args(func, args, kwargs, sigspec): - # Adding more arguments may make the call valid - return True - else: - # There was a genuine TypeError - return False - - def bind(self, *args, **kwargs): - return type(self)(self, *args, **kwargs) - - def call(self, *args, **kwargs): - return self._partial(*args, **kwargs) - - def __get__(self, instance, owner): - if instance is None: - return self - return curry(self, instance) - - def __reduce__(self): - func = self.func - modname = getattr(func, '__module__', None) - qualname = getattr(func, '__qualname__', None) - if qualname is None: # pragma: py3 no cover - qualname = getattr(func, '__name__', None) - is_decorated = None - if modname and qualname: - attrs = [] - obj = import_module(modname) - for attr in qualname.split('.'): - if isinstance(obj, curry): # pragma: py2 no cover - attrs.append('func') - obj = obj.func - obj = getattr(obj, attr, None) - if obj is None: - break - attrs.append(attr) - if isinstance(obj, curry) and obj.func is func: - is_decorated = obj is self - qualname = '.'.join(attrs) - func = '%s:%s' % (modname, qualname) - - # functools.partial objects can't be pickled - userdict = tuple((k, v) for k, v in self.__dict__.items() - if k not in ('_partial', '_sigspec')) - state = (type(self), func, self.args, self.keywords, userdict, - is_decorated) - return _restore_curry, state - - -def _restore_curry(cls, func, args, kwargs, userdict, is_decorated): - if isinstance(func, str): - modname, qualname = func.rsplit(':', 1) - obj = import_module(modname) - for attr in qualname.split('.'): - obj = getattr(obj, attr) - if is_decorated: - return obj - func = obj.func - obj = cls(func, *args, **(kwargs or {})) - obj.__dict__.update(userdict) - return obj - - -@curry -def memoize(func, cache=None, key=None): - """ Cache a function's result for speedy future evaluation - - Considerations: - Trades memory for speed. - Only use on pure functions. - - >>> def add(x, y): return x + y - >>> add = memoize(add) - - Or use as a decorator - - >>> @memoize - ... def add(x, y): - ... return x + y - - Use the ``cache`` keyword to provide a dict-like object as an initial cache - - >>> @memoize(cache={(1, 2): 3}) - ... def add(x, y): - ... return x + y - - Note that the above works as a decorator because ``memoize`` is curried. - - It is also possible to provide a ``key(args, kwargs)`` function that - calculates keys used for the cache, which receives an ``args`` tuple and - ``kwargs`` dict as input, and must return a hashable value. However, - the default key function should be sufficient most of the time. - - >>> # Use key function that ignores extraneous keyword arguments - >>> @memoize(key=lambda args, kwargs: args) - ... def add(x, y, verbose=False): - ... if verbose: - ... print('Calculating %s + %s' % (x, y)) - ... return x + y - """ - if cache is None: - cache = {} - - try: - may_have_kwargs = has_keywords(func) is not False - # Is unary function (single arg, no variadic argument or keywords)? - is_unary = is_arity(1, func) - except TypeError: # pragma: no cover - may_have_kwargs = True - is_unary = False - - if key is None: - if is_unary: - def key(args, kwargs): - return args[0] - elif may_have_kwargs: - def key(args, kwargs): - return ( - args or None, - frozenset(kwargs.items()) if kwargs else None, - ) - else: - def key(args, kwargs): - return args - - def memof(*args, **kwargs): - k = key(args, kwargs) - try: - return cache[k] - except TypeError: - raise TypeError("Arguments to memoized function must be hashable") - except KeyError: - cache[k] = result = func(*args, **kwargs) - return result - - try: - memof.__name__ = func.__name__ - except AttributeError: - pass - memof.__doc__ = func.__doc__ - memof.__wrapped__ = func - return memof - - -class Compose(object): - """ A composition of functions - - See Also: - compose - """ - __slots__ = 'first', 'funcs' - - def __init__(self, funcs): - funcs = tuple(reversed(funcs)) - self.first = funcs[0] - self.funcs = funcs[1:] - - def __call__(self, *args, **kwargs): - ret = self.first(*args, **kwargs) - for f in self.funcs: - ret = f(ret) - return ret - - def __getstate__(self): - return self.first, self.funcs - - def __setstate__(self, state): - self.first, self.funcs = state - - @instanceproperty(classval=__doc__) - def __doc__(self): - def composed_doc(*fs): - """Generate a docstring for the composition of fs. - """ - if not fs: - # Argument name for the docstring. - return '*args, **kwargs' - - return '{f}({g})'.format(f=fs[0].__name__, g=composed_doc(*fs[1:])) - - try: - return ( - 'lambda *args, **kwargs: ' + - composed_doc(*reversed((self.first,) + self.funcs)) - ) - except AttributeError: - # One of our callables does not have a `__name__`, whatever. - return 'A composition of functions' - - @property - def __name__(self): - try: - return '_of_'.join( - (f.__name__ for f in reversed((self.first,) + self.funcs)) - ) - except AttributeError: - return type(self).__name__ - - def __repr__(self): - return '{.__class__.__name__}{!r}'.format( - self, tuple(reversed((self.first, ) + self.funcs))) - - def __eq__(self, other): - if isinstance(other, Compose): - return other.first == self.first and other.funcs == self.funcs - return NotImplemented - - def __ne__(self, other): - equality = self.__eq__(other) - return NotImplemented if equality is NotImplemented else not equality - - def __hash__(self): - return hash(self.first) ^ hash(self.funcs) - - # Mimic the descriptor behavior of python functions. - # i.e. let Compose be called as a method when bound to a class. - if PY3: # pragma: py2 no cover - # adapted from - # docs.python.org/3/howto/descriptor.html#functions-and-methods - def __get__(self, obj, objtype=None): - return self if obj is None else MethodType(self, obj) - else: # pragma: py3 no cover - # adapted from - # docs.python.org/2/howto/descriptor.html#functions-and-methods - def __get__(self, obj, objtype=None): - return self if obj is None else MethodType(self, obj, objtype) - - # introspection with Signature is only possible from py3.3+ - if PY3: # pragma: py2 no cover - @instanceproperty - def __signature__(self): - base = inspect.signature(self.first) - last = inspect.signature(self.funcs[-1]) - return base.replace(return_annotation=last.return_annotation) - - __wrapped__ = instanceproperty(attrgetter('first')) - - -def compose(*funcs): - """ Compose functions to operate in series. - - Returns a function that applies other functions in sequence. - - Functions are applied from right to left so that - ``compose(f, g, h)(x, y)`` is the same as ``f(g(h(x, y)))``. - - If no arguments are provided, the identity function (f(x) = x) is returned. - - >>> inc = lambda i: i + 1 - >>> compose(str, inc)(3) - '4' - - See Also: - compose_left - pipe - """ - if not funcs: - return identity - if len(funcs) == 1: - return funcs[0] - else: - return Compose(funcs) - - -def compose_left(*funcs): - """ Compose functions to operate in series. - - Returns a function that applies other functions in sequence. - - Functions are applied from left to right so that - ``compose_left(f, g, h)(x, y)`` is the same as ``h(g(f(x, y)))``. - - If no arguments are provided, the identity function (f(x) = x) is returned. - - >>> inc = lambda i: i + 1 - >>> compose_left(inc, str)(3) - '4' - - See Also: - compose - pipe - """ - return compose(*reversed(funcs)) - - -def pipe(data, *funcs): - """ Pipe a value through a sequence of functions - - I.e. ``pipe(data, f, g, h)`` is equivalent to ``h(g(f(data)))`` - - We think of the value as progressing through a pipe of several - transformations, much like pipes in UNIX - - ``$ cat data | f | g | h`` - - >>> double = lambda i: 2 * i - >>> pipe(3, double, str) - '6' - - See Also: - compose - compose_left - thread_first - thread_last - """ - for func in funcs: - data = func(data) - return data - - -def complement(func): - """ Convert a predicate function to its logical complement. - - In other words, return a function that, for inputs that normally - yield True, yields False, and vice-versa. - - >>> def iseven(n): return n % 2 == 0 - >>> isodd = complement(iseven) - >>> iseven(2) - True - >>> isodd(2) - False - """ - return compose(operator.not_, func) - - -class juxt(object): - """ Creates a function that calls several functions with the same arguments - - Takes several functions and returns a function that applies its arguments - to each of those functions then returns a tuple of the results. - - Name comes from juxtaposition: the fact of two things being seen or placed - close together with contrasting effect. - - >>> inc = lambda x: x + 1 - >>> double = lambda x: x * 2 - >>> juxt(inc, double)(10) - (11, 20) - >>> juxt([inc, double])(10) - (11, 20) - """ - __slots__ = ['funcs'] - - def __init__(self, *funcs): - if len(funcs) == 1 and not callable(funcs[0]): - funcs = funcs[0] - self.funcs = tuple(funcs) - - def __call__(self, *args, **kwargs): - return tuple(func(*args, **kwargs) for func in self.funcs) - - def __getstate__(self): - return self.funcs - - def __setstate__(self, state): - self.funcs = state - - -def do(func, x): - """ Runs ``func`` on ``x``, returns ``x`` - - Because the results of ``func`` are not returned, only the side - effects of ``func`` are relevant. - - Logging functions can be made by composing ``do`` with a storage function - like ``list.append`` or ``file.write`` - - >>> from toolz import compose - >>> from toolz.curried import do - - >>> log = [] - >>> inc = lambda x: x + 1 - >>> inc = compose(inc, do(log.append)) - >>> inc(1) - 2 - >>> inc(11) - 12 - >>> log - [1, 11] - """ - func(x) - return x - - -@curry -def flip(func, a, b): - """ Call the function call with the arguments flipped - - This function is curried. - - >>> def div(a, b): - ... return a // b - ... - >>> flip(div, 2, 6) - 3 - >>> div_by_two = flip(div, 2) - >>> div_by_two(4) - 2 - - This is particularly useful for built in functions and functions defined - in C extensions that accept positional only arguments. For example: - isinstance, issubclass. - - >>> data = [1, 'a', 'b', 2, 1.5, object(), 3] - >>> only_ints = list(filter(flip(isinstance, int), data)) - >>> only_ints - [1, 2, 3] - """ - return func(b, a) - - -def return_none(exc): - """ Returns None. - """ - return None - - -class excepts(object): - """A wrapper around a function to catch exceptions and - dispatch to a handler. - - This is like a functional try/except block, in the same way that - ifexprs are functional if/else blocks. - - Examples - -------- - >>> excepting = excepts( - ... ValueError, - ... lambda a: [1, 2].index(a), - ... lambda _: -1, - ... ) - >>> excepting(1) - 0 - >>> excepting(3) - -1 - - Multiple exceptions and default except clause. - >>> excepting = excepts((IndexError, KeyError), lambda a: a[0]) - >>> excepting([]) - >>> excepting([1]) - 1 - >>> excepting({}) - >>> excepting({0: 1}) - 1 - """ - def __init__(self, exc, func, handler=return_none): - self.exc = exc - self.func = func - self.handler = handler - - def __call__(self, *args, **kwargs): - try: - return self.func(*args, **kwargs) - except self.exc as e: - return self.handler(e) - - @instanceproperty(classval=__doc__) - def __doc__(self): - exc = self.exc - try: - if isinstance(exc, tuple): - exc_name = '(%s)' % ', '.join( - map(attrgetter('__name__'), exc), - ) - else: - exc_name = exc.__name__ - - return dedent( - """\ - A wrapper around {inst.func.__name__!r} that will except: - {exc} - and handle any exceptions with {inst.handler.__name__!r}. - - Docs for {inst.func.__name__!r}: - {inst.func.__doc__} - - Docs for {inst.handler.__name__!r}: - {inst.handler.__doc__} - """ - ).format( - inst=self, - exc=exc_name, - ) - except AttributeError: - return type(self).__doc__ - - @property - def __name__(self): - exc = self.exc - try: - if isinstance(exc, tuple): - exc_name = '_or_'.join(map(attrgetter('__name__'), exc)) - else: - exc_name = exc.__name__ - return '%s_excepting_%s' % (self.func.__name__, exc_name) - except AttributeError: - return 'excepting' - - -if PY3: # pragma: py2 no cover - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - if sigspec is None: - try: - sigspec = inspect.signature(func) - except (ValueError, TypeError) as e: - sigspec = e - if isinstance(sigspec, ValueError): - return None, builtin_func(*builtin_args) - elif not isinstance(sigspec, inspect.Signature): - if ( - func in _sigs.signatures - and (( - hasattr(func, '__signature__') - and hasattr(func.__signature__, '__get__') - )) - ): # pragma: no cover (not covered in Python 3.4) - val = builtin_func(*builtin_args) - return None, val - return None, False - return sigspec, None - -else: # pragma: py3 no cover - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - if sigspec is None: - try: - sigspec = inspect.getargspec(func) - except TypeError as e: - sigspec = e - if isinstance(sigspec, TypeError): - if not callable(func): - return None, False - return None, builtin_func(*builtin_args) - return sigspec, None - - -if PY34 or PYPY: # pragma: no cover - _check_sigspec_orig = _check_sigspec - - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - # Python 3.4 and PyPy may lie, so use our registry for builtins instead - if func in _sigs.signatures: - val = builtin_func(*builtin_args) - return None, val - return _check_sigspec_orig(sigspec, func, builtin_func, *builtin_args) - -_check_sigspec.__doc__ = """ \ -Private function to aid in introspection compatibly across Python versions. - -If a callable doesn't have a signature (Python 3) or an argspec (Python 2), -the signature registry in toolz._signatures is used. -""" - -if PY3: # pragma: py2 no cover - def num_required_args(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._num_required_args, - func) - if sigspec is None: - return rv - return sum(1 for p in sigspec.parameters.values() - if p.default is p.empty - and p.kind in (p.POSITIONAL_OR_KEYWORD, p.POSITIONAL_ONLY)) - - def has_varargs(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_varargs, func) - if sigspec is None: - return rv - return any(p.kind == p.VAR_POSITIONAL - for p in sigspec.parameters.values()) - - def has_keywords(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_keywords, func) - if sigspec is None: - return rv - return any(p.default is not p.empty - or p.kind in (p.KEYWORD_ONLY, p.VAR_KEYWORD) - for p in sigspec.parameters.values()) - - def is_valid_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_valid_args, - func, args, kwargs) - if sigspec is None: - return rv - try: - sigspec.bind(*args, **kwargs) - except TypeError: - return False - return True - - def is_partial_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_partial_args, - func, args, kwargs) - if sigspec is None: - return rv - try: - sigspec.bind_partial(*args, **kwargs) - except TypeError: - return False - return True - -else: # pragma: py3 no cover - def num_required_args(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._num_required_args, - func) - if sigspec is None: - return rv - num_defaults = len(sigspec.defaults) if sigspec.defaults else 0 - return len(sigspec.args) - num_defaults - - def has_varargs(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_varargs, func) - if sigspec is None: - return rv - return sigspec.varargs is not None - - def has_keywords(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_keywords, func) - if sigspec is None: - return rv - return sigspec.defaults is not None or sigspec.keywords is not None - - def is_valid_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_valid_args, - func, args, kwargs) - if sigspec is None: - return rv - spec = sigspec - defaults = spec.defaults or () - num_pos = len(spec.args) - len(defaults) - missing_pos = spec.args[len(args):num_pos] - if any(arg not in kwargs for arg in missing_pos): - return False - - if spec.varargs is None: - num_extra_pos = max(0, len(args) - num_pos) - else: - num_extra_pos = 0 - - kwargs = dict(kwargs) - - # Add missing keyword arguments (unless already included in `args`) - missing_kwargs = spec.args[num_pos + num_extra_pos:] - kwargs.update(zip(missing_kwargs, defaults[num_extra_pos:])) - - # Convert call to use positional arguments - args = args + tuple(kwargs.pop(key) for key in spec.args[len(args):]) - - if ( - not spec.keywords and kwargs - or not spec.varargs and len(args) > len(spec.args) - or set(spec.args[:len(args)]) & set(kwargs) - ): - return False - else: - return True - - def is_partial_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_partial_args, - func, args, kwargs) - if sigspec is None: - return rv - spec = sigspec - defaults = spec.defaults or () - num_pos = len(spec.args) - len(defaults) - if spec.varargs is None: - num_extra_pos = max(0, len(args) - num_pos) - else: - num_extra_pos = 0 - - kwargs = dict(kwargs) - - # Add missing keyword arguments (unless already included in `args`) - missing_kwargs = spec.args[num_pos + num_extra_pos:] - kwargs.update(zip(missing_kwargs, defaults[num_extra_pos:])) - - # Add missing position arguments as keywords (may already be in kwargs) - missing_args = spec.args[len(args):num_pos + num_extra_pos] - kwargs.update((x, None) for x in missing_args) - - # Convert call to use positional arguments - args = args + tuple(kwargs.pop(key) for key in spec.args[len(args):]) - - if ( - not spec.keywords and kwargs - or not spec.varargs and len(args) > len(spec.args) - or set(spec.args[:len(args)]) & set(kwargs) - ): - return False - else: - return True - - -def is_arity(n, func, sigspec=None): - """ Does a function have only n positional arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x): - ... return x - >>> is_arity(1, f) - True - >>> def g(x, y=1): - ... return x + y - >>> is_arity(1, g) - False - """ - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_arity, n, func) - if sigspec is None: - return rv - num = num_required_args(func, sigspec) - if num is not None: - num = num == n - if not num: - return False - varargs = has_varargs(func, sigspec) - if varargs: - return False - keywords = has_keywords(func, sigspec) - if keywords: - return False - if num is None or varargs is None or keywords is None: # pragma: no cover - return None - return True - - -num_required_args.__doc__ = """ \ -Number of required positional arguments - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x, y, z=3): - ... return x + y + z - >>> num_required_args(f) - 2 - >>> def g(*args, **kwargs): - ... pass - >>> num_required_args(g) - 0 - """ - -has_varargs.__doc__ = """ \ -Does a function have variadic positional arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(*args): - ... return args - >>> has_varargs(f) - True - >>> def g(**kwargs): - ... return kwargs - >>> has_varargs(g) - False - """ - -has_keywords.__doc__ = """ \ -Does a function have keyword arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x, y=0): - ... return x + y - - >>> has_keywords(f) - True - """ - -is_valid_args.__doc__ = """ \ -Is ``func(*args, **kwargs)`` a valid function call? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def add(x, y): - ... return x + y - - >>> is_valid_args(add, (1,), {}) - False - >>> is_valid_args(add, (1, 2), {}) - True - >>> is_valid_args(map, (), {}) - False - - **Implementation notes** - Python 2 relies on ``inspect.getargspec``, which only works for - user-defined functions. Python 3 uses ``inspect.signature``, which - works for many more types of callables. - - Many builtins in the standard library are also supported. - """ - -is_partial_args.__doc__ = """ \ -Can partial(func, *args, **kwargs)(*args2, **kwargs2) be a valid call? - - Returns True *only* if the call is valid or if it is possible for the - call to become valid by adding more positional or keyword arguments. - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def add(x, y): - ... return x + y - - >>> is_partial_args(add, (1,), {}) - True - >>> is_partial_args(add, (1, 2), {}) - True - >>> is_partial_args(add, (1, 2, 3), {}) - False - >>> is_partial_args(map, (), {}) - True - - **Implementation notes** - Python 2 relies on ``inspect.getargspec``, which only works for - user-defined functions. Python 3 uses ``inspect.signature``, which - works for many more types of callables. - - Many builtins in the standard library are also supported. - """ - -from . import _signatures as _sigs diff --git a/contrib/python/toolz/py2/toolz/itertoolz.py b/contrib/python/toolz/py2/toolz/itertoolz.py deleted file mode 100644 index e71f1eeef0..0000000000 --- a/contrib/python/toolz/py2/toolz/itertoolz.py +++ /dev/null @@ -1,1056 +0,0 @@ -import itertools -import heapq -import collections -import operator -from functools import partial -from random import Random -from toolz.compatibility import (map, filterfalse, zip, zip_longest, iteritems, - filter, Sequence) -from toolz.utils import no_default - - -__all__ = ('remove', 'accumulate', 'groupby', 'merge_sorted', 'interleave', - 'unique', 'isiterable', 'isdistinct', 'take', 'drop', 'take_nth', - 'first', 'second', 'nth', 'last', 'get', 'concat', 'concatv', - 'mapcat', 'cons', 'interpose', 'frequencies', 'reduceby', 'iterate', - 'sliding_window', 'partition', 'partition_all', 'count', 'pluck', - 'join', 'tail', 'diff', 'topk', 'peek', 'peekn', 'random_sample') - - -def remove(predicate, seq): - """ Return those items of sequence for which predicate(item) is False - - >>> def iseven(x): - ... return x % 2 == 0 - >>> list(remove(iseven, [1, 2, 3, 4])) - [1, 3] - """ - return filterfalse(predicate, seq) - - -def accumulate(binop, seq, initial=no_default): - """ Repeatedly apply binary function to a sequence, accumulating results - - >>> from operator import add, mul - >>> list(accumulate(add, [1, 2, 3, 4, 5])) - [1, 3, 6, 10, 15] - >>> list(accumulate(mul, [1, 2, 3, 4, 5])) - [1, 2, 6, 24, 120] - - Accumulate is similar to ``reduce`` and is good for making functions like - cumulative sum: - - >>> from functools import partial, reduce - >>> sum = partial(reduce, add) - >>> cumsum = partial(accumulate, add) - - Accumulate also takes an optional argument that will be used as the first - value. This is similar to reduce. - - >>> list(accumulate(add, [1, 2, 3], -1)) - [-1, 0, 2, 5] - >>> list(accumulate(add, [], 1)) - [1] - - See Also: - itertools.accumulate : In standard itertools for Python 3.2+ - """ - seq = iter(seq) - if initial == no_default: - try: - result = next(seq) - except StopIteration: - return - else: - result = initial - yield result - for elem in seq: - result = binop(result, elem) - yield result - - -def groupby(key, seq): - """ Group a collection by a key function - - >>> names = ['Alice', 'Bob', 'Charlie', 'Dan', 'Edith', 'Frank'] - >>> groupby(len, names) # doctest: +SKIP - {3: ['Bob', 'Dan'], 5: ['Alice', 'Edith', 'Frank'], 7: ['Charlie']} - - >>> iseven = lambda x: x % 2 == 0 - >>> groupby(iseven, [1, 2, 3, 4, 5, 6, 7, 8]) # doctest: +SKIP - {False: [1, 3, 5, 7], True: [2, 4, 6, 8]} - - Non-callable keys imply grouping on a member. - - >>> groupby('gender', [{'name': 'Alice', 'gender': 'F'}, - ... {'name': 'Bob', 'gender': 'M'}, - ... {'name': 'Charlie', 'gender': 'M'}]) # doctest:+SKIP - {'F': [{'gender': 'F', 'name': 'Alice'}], - 'M': [{'gender': 'M', 'name': 'Bob'}, - {'gender': 'M', 'name': 'Charlie'}]} - - Not to be confused with ``itertools.groupby`` - - See Also: - countby - """ - if not callable(key): - key = getter(key) - d = collections.defaultdict(lambda: [].append) - for item in seq: - d[key(item)](item) - rv = {} - for k, v in iteritems(d): - rv[k] = v.__self__ - return rv - - -def merge_sorted(*seqs, **kwargs): - """ Merge and sort a collection of sorted collections - - This works lazily and only keeps one value from each iterable in memory. - - >>> list(merge_sorted([1, 3, 5], [2, 4, 6])) - [1, 2, 3, 4, 5, 6] - - >>> ''.join(merge_sorted('abc', 'abc', 'abc')) - 'aaabbbccc' - - The "key" function used to sort the input may be passed as a keyword. - - >>> list(merge_sorted([2, 3], [1, 3], key=lambda x: x // 3)) - [2, 1, 3, 3] - """ - if len(seqs) == 0: - return iter([]) - elif len(seqs) == 1: - return iter(seqs[0]) - - key = kwargs.get('key', None) - if key is None: - return _merge_sorted_binary(seqs) - else: - return _merge_sorted_binary_key(seqs, key) - - -def _merge_sorted_binary(seqs): - mid = len(seqs) // 2 - L1 = seqs[:mid] - if len(L1) == 1: - seq1 = iter(L1[0]) - else: - seq1 = _merge_sorted_binary(L1) - L2 = seqs[mid:] - if len(L2) == 1: - seq2 = iter(L2[0]) - else: - seq2 = _merge_sorted_binary(L2) - - try: - val2 = next(seq2) - except StopIteration: - for val1 in seq1: - yield val1 - return - - for val1 in seq1: - if val2 < val1: - yield val2 - for val2 in seq2: - if val2 < val1: - yield val2 - else: - yield val1 - break - else: - break - else: - yield val1 - else: - yield val2 - for val2 in seq2: - yield val2 - return - yield val1 - for val1 in seq1: - yield val1 - - -def _merge_sorted_binary_key(seqs, key): - mid = len(seqs) // 2 - L1 = seqs[:mid] - if len(L1) == 1: - seq1 = iter(L1[0]) - else: - seq1 = _merge_sorted_binary_key(L1, key) - L2 = seqs[mid:] - if len(L2) == 1: - seq2 = iter(L2[0]) - else: - seq2 = _merge_sorted_binary_key(L2, key) - - try: - val2 = next(seq2) - except StopIteration: - for val1 in seq1: - yield val1 - return - key2 = key(val2) - - for val1 in seq1: - key1 = key(val1) - if key2 < key1: - yield val2 - for val2 in seq2: - key2 = key(val2) - if key2 < key1: - yield val2 - else: - yield val1 - break - else: - break - else: - yield val1 - else: - yield val2 - for val2 in seq2: - yield val2 - return - yield val1 - for val1 in seq1: - yield val1 - - -def interleave(seqs): - """ Interleave a sequence of sequences - - >>> list(interleave([[1, 2], [3, 4]])) - [1, 3, 2, 4] - - >>> ''.join(interleave(('ABC', 'XY'))) - 'AXBYC' - - Both the individual sequences and the sequence of sequences may be infinite - - Returns a lazy iterator - """ - iters = itertools.cycle(map(iter, seqs)) - while True: - try: - for itr in iters: - yield next(itr) - return - except StopIteration: - predicate = partial(operator.is_not, itr) - iters = itertools.cycle(itertools.takewhile(predicate, iters)) - - -def unique(seq, key=None): - """ Return only unique elements of a sequence - - >>> tuple(unique((1, 2, 3))) - (1, 2, 3) - >>> tuple(unique((1, 2, 1, 3))) - (1, 2, 3) - - Uniqueness can be defined by key keyword - - >>> tuple(unique(['cat', 'mouse', 'dog', 'hen'], key=len)) - ('cat', 'mouse') - """ - seen = set() - seen_add = seen.add - if key is None: - for item in seq: - if item not in seen: - seen_add(item) - yield item - else: # calculate key - for item in seq: - val = key(item) - if val not in seen: - seen_add(val) - yield item - - -def isiterable(x): - """ Is x iterable? - - >>> isiterable([1, 2, 3]) - True - >>> isiterable('abc') - True - >>> isiterable(5) - False - """ - try: - iter(x) - return True - except TypeError: - return False - - -def isdistinct(seq): - """ All values in sequence are distinct - - >>> isdistinct([1, 2, 3]) - True - >>> isdistinct([1, 2, 1]) - False - - >>> isdistinct("Hello") - False - >>> isdistinct("World") - True - """ - if iter(seq) is seq: - seen = set() - seen_add = seen.add - for item in seq: - if item in seen: - return False - seen_add(item) - return True - else: - return len(seq) == len(set(seq)) - - -def take(n, seq): - """ The first n elements of a sequence - - >>> list(take(2, [10, 20, 30, 40, 50])) - [10, 20] - - See Also: - drop - tail - """ - return itertools.islice(seq, n) - - -def tail(n, seq): - """ The last n elements of a sequence - - >>> tail(2, [10, 20, 30, 40, 50]) - [40, 50] - - See Also: - drop - take - """ - try: - return seq[-n:] - except (TypeError, KeyError): - return tuple(collections.deque(seq, n)) - - -def drop(n, seq): - """ The sequence following the first n elements - - >>> list(drop(2, [10, 20, 30, 40, 50])) - [30, 40, 50] - - See Also: - take - tail - """ - return itertools.islice(seq, n, None) - - -def take_nth(n, seq): - """ Every nth item in seq - - >>> list(take_nth(2, [10, 20, 30, 40, 50])) - [10, 30, 50] - """ - return itertools.islice(seq, 0, None, n) - - -def first(seq): - """ The first element in a sequence - - >>> first('ABC') - 'A' - """ - return next(iter(seq)) - - -def second(seq): - """ The second element in a sequence - - >>> second('ABC') - 'B' - """ - seq = iter(seq) - next(seq) - return next(seq) - - -def nth(n, seq): - """ The nth element in a sequence - - >>> nth(1, 'ABC') - 'B' - """ - if isinstance(seq, (tuple, list, Sequence)): - return seq[n] - else: - return next(itertools.islice(seq, n, None)) - - -def last(seq): - """ The last element in a sequence - - >>> last('ABC') - 'C' - """ - return tail(1, seq)[0] - - -rest = partial(drop, 1) - - -def _get(ind, seq, default): - try: - return seq[ind] - except (KeyError, IndexError): - return default - - -def get(ind, seq, default=no_default): - """ Get element in a sequence or dict - - Provides standard indexing - - >>> get(1, 'ABC') # Same as 'ABC'[1] - 'B' - - Pass a list to get multiple values - - >>> get([1, 2], 'ABC') # ('ABC'[1], 'ABC'[2]) - ('B', 'C') - - Works on any value that supports indexing/getitem - For example here we see that it works with dictionaries - - >>> phonebook = {'Alice': '555-1234', - ... 'Bob': '555-5678', - ... 'Charlie':'555-9999'} - >>> get('Alice', phonebook) - '555-1234' - - >>> get(['Alice', 'Bob'], phonebook) - ('555-1234', '555-5678') - - Provide a default for missing values - - >>> get(['Alice', 'Dennis'], phonebook, None) - ('555-1234', None) - - See Also: - pluck - """ - try: - return seq[ind] - except TypeError: # `ind` may be a list - if isinstance(ind, list): - if default == no_default: - if len(ind) > 1: - return operator.itemgetter(*ind)(seq) - elif ind: - return seq[ind[0]], - else: - return () - else: - return tuple(_get(i, seq, default) for i in ind) - elif default != no_default: - return default - else: - raise - except (KeyError, IndexError): # we know `ind` is not a list - if default == no_default: - raise - else: - return default - - -def concat(seqs): - """ Concatenate zero or more iterables, any of which may be infinite. - - An infinite sequence will prevent the rest of the arguments from - being included. - - We use chain.from_iterable rather than ``chain(*seqs)`` so that seqs - can be a generator. - - >>> list(concat([[], [1], [2, 3]])) - [1, 2, 3] - - See also: - itertools.chain.from_iterable equivalent - """ - return itertools.chain.from_iterable(seqs) - - -def concatv(*seqs): - """ Variadic version of concat - - >>> list(concatv([], ["a"], ["b", "c"])) - ['a', 'b', 'c'] - - See also: - itertools.chain - """ - return concat(seqs) - - -def mapcat(func, seqs): - """ Apply func to each sequence in seqs, concatenating results. - - >>> list(mapcat(lambda s: [c.upper() for c in s], - ... [["a", "b"], ["c", "d", "e"]])) - ['A', 'B', 'C', 'D', 'E'] - """ - return concat(map(func, seqs)) - - -def cons(el, seq): - """ Add el to beginning of (possibly infinite) sequence seq. - - >>> list(cons(1, [2, 3])) - [1, 2, 3] - """ - return itertools.chain([el], seq) - - -def interpose(el, seq): - """ Introduce element between each pair of elements in seq - - >>> list(interpose("a", [1, 2, 3])) - [1, 'a', 2, 'a', 3] - """ - inposed = concat(zip(itertools.repeat(el), seq)) - next(inposed) - return inposed - - -def frequencies(seq): - """ Find number of occurrences of each value in seq - - >>> frequencies(['cat', 'cat', 'ox', 'pig', 'pig', 'cat']) #doctest: +SKIP - {'cat': 3, 'ox': 1, 'pig': 2} - - See Also: - countby - groupby - """ - d = collections.defaultdict(int) - for item in seq: - d[item] += 1 - return dict(d) - - -def reduceby(key, binop, seq, init=no_default): - """ Perform a simultaneous groupby and reduction - - The computation: - - >>> result = reduceby(key, binop, seq, init) # doctest: +SKIP - - is equivalent to the following: - - >>> def reduction(group): # doctest: +SKIP - ... return reduce(binop, group, init) # doctest: +SKIP - - >>> groups = groupby(key, seq) # doctest: +SKIP - >>> result = valmap(reduction, groups) # doctest: +SKIP - - But the former does not build the intermediate groups, allowing it to - operate in much less space. This makes it suitable for larger datasets - that do not fit comfortably in memory - - The ``init`` keyword argument is the default initialization of the - reduction. This can be either a constant value like ``0`` or a callable - like ``lambda : 0`` as might be used in ``defaultdict``. - - Simple Examples - --------------- - - >>> from operator import add, mul - >>> iseven = lambda x: x % 2 == 0 - - >>> data = [1, 2, 3, 4, 5] - - >>> reduceby(iseven, add, data) # doctest: +SKIP - {False: 9, True: 6} - - >>> reduceby(iseven, mul, data) # doctest: +SKIP - {False: 15, True: 8} - - Complex Example - --------------- - - >>> projects = [{'name': 'build roads', 'state': 'CA', 'cost': 1000000}, - ... {'name': 'fight crime', 'state': 'IL', 'cost': 100000}, - ... {'name': 'help farmers', 'state': 'IL', 'cost': 2000000}, - ... {'name': 'help farmers', 'state': 'CA', 'cost': 200000}] - - >>> reduceby('state', # doctest: +SKIP - ... lambda acc, x: acc + x['cost'], - ... projects, 0) - {'CA': 1200000, 'IL': 2100000} - - Example Using ``init`` - ---------------------- - - >>> def set_add(s, i): - ... s.add(i) - ... return s - - >>> reduceby(iseven, set_add, [1, 2, 3, 4, 1, 2, 3], set) # doctest: +SKIP - {True: set([2, 4]), - False: set([1, 3])} - """ - is_no_default = init == no_default - if not is_no_default and not callable(init): - _init = init - init = lambda: _init - if not callable(key): - key = getter(key) - d = {} - for item in seq: - k = key(item) - if k not in d: - if is_no_default: - d[k] = item - continue - else: - d[k] = init() - d[k] = binop(d[k], item) - return d - - -def iterate(func, x): - """ Repeatedly apply a function func onto an original input - - Yields x, then func(x), then func(func(x)), then func(func(func(x))), etc.. - - >>> def inc(x): return x + 1 - >>> counter = iterate(inc, 0) - >>> next(counter) - 0 - >>> next(counter) - 1 - >>> next(counter) - 2 - - >>> double = lambda x: x * 2 - >>> powers_of_two = iterate(double, 1) - >>> next(powers_of_two) - 1 - >>> next(powers_of_two) - 2 - >>> next(powers_of_two) - 4 - >>> next(powers_of_two) - 8 - """ - while True: - yield x - x = func(x) - - -def sliding_window(n, seq): - """ A sequence of overlapping subsequences - - >>> list(sliding_window(2, [1, 2, 3, 4])) - [(1, 2), (2, 3), (3, 4)] - - This function creates a sliding window suitable for transformations like - sliding means / smoothing - - >>> mean = lambda seq: float(sum(seq)) / len(seq) - >>> list(map(mean, sliding_window(2, [1, 2, 3, 4]))) - [1.5, 2.5, 3.5] - """ - return zip(*(collections.deque(itertools.islice(it, i), 0) or it - for i, it in enumerate(itertools.tee(seq, n)))) - - -no_pad = '__no__pad__' - - -def partition(n, seq, pad=no_pad): - """ Partition sequence into tuples of length n - - >>> list(partition(2, [1, 2, 3, 4])) - [(1, 2), (3, 4)] - - If the length of ``seq`` is not evenly divisible by ``n``, the final tuple - is dropped if ``pad`` is not specified, or filled to length ``n`` by pad: - - >>> list(partition(2, [1, 2, 3, 4, 5])) - [(1, 2), (3, 4)] - - >>> list(partition(2, [1, 2, 3, 4, 5], pad=None)) - [(1, 2), (3, 4), (5, None)] - - See Also: - partition_all - """ - args = [iter(seq)] * n - if pad is no_pad: - return zip(*args) - else: - return zip_longest(*args, fillvalue=pad) - - -def partition_all(n, seq): - """ Partition all elements of sequence into tuples of length at most n - - The final tuple may be shorter to accommodate extra elements. - - >>> list(partition_all(2, [1, 2, 3, 4])) - [(1, 2), (3, 4)] - - >>> list(partition_all(2, [1, 2, 3, 4, 5])) - [(1, 2), (3, 4), (5,)] - - See Also: - partition - """ - args = [iter(seq)] * n - it = zip_longest(*args, fillvalue=no_pad) - try: - prev = next(it) - except StopIteration: - return - for item in it: - yield prev - prev = item - if prev[-1] is no_pad: - try: - # If seq defines __len__, then - # we can quickly calculate where no_pad starts - yield prev[:len(seq) % n] - except TypeError: - # Get first index of no_pad without using .index() - # https://github.com/pytoolz/toolz/issues/387 - # Binary search from CPython's bisect module, - # modified for identity testing. - lo, hi = 0, n - while lo < hi: - mid = (lo + hi) // 2 - if prev[mid] is no_pad: - hi = mid - else: - lo = mid + 1 - yield prev[:lo] - else: - yield prev - - -def count(seq): - """ Count the number of items in seq - - Like the builtin ``len`` but works on lazy sequencies. - - Not to be confused with ``itertools.count`` - - See also: - len - """ - if hasattr(seq, '__len__'): - return len(seq) - return sum(1 for i in seq) - - -def pluck(ind, seqs, default=no_default): - """ plucks an element or several elements from each item in a sequence. - - ``pluck`` maps ``itertoolz.get`` over a sequence and returns one or more - elements of each item in the sequence. - - This is equivalent to running `map(curried.get(ind), seqs)` - - ``ind`` can be either a single string/index or a list of strings/indices. - ``seqs`` should be sequence containing sequences or dicts. - - e.g. - - >>> data = [{'id': 1, 'name': 'Cheese'}, {'id': 2, 'name': 'Pies'}] - >>> list(pluck('name', data)) - ['Cheese', 'Pies'] - >>> list(pluck([0, 1], [[1, 2, 3], [4, 5, 7]])) - [(1, 2), (4, 5)] - - See Also: - get - map - """ - if default == no_default: - get = getter(ind) - return map(get, seqs) - elif isinstance(ind, list): - return (tuple(_get(item, seq, default) for item in ind) - for seq in seqs) - return (_get(ind, seq, default) for seq in seqs) - - -def getter(index): - if isinstance(index, list): - if len(index) == 1: - index = index[0] - return lambda x: (x[index],) - elif index: - return operator.itemgetter(*index) - else: - return lambda x: () - else: - return operator.itemgetter(index) - - -def join(leftkey, leftseq, rightkey, rightseq, - left_default=no_default, right_default=no_default): - """ Join two sequences on common attributes - - This is a semi-streaming operation. The LEFT sequence is fully evaluated - and placed into memory. The RIGHT sequence is evaluated lazily and so can - be arbitrarily large. - (Note: If right_default is defined, then unique keys of rightseq - will also be stored in memory.) - - >>> friends = [('Alice', 'Edith'), - ... ('Alice', 'Zhao'), - ... ('Edith', 'Alice'), - ... ('Zhao', 'Alice'), - ... ('Zhao', 'Edith')] - - >>> cities = [('Alice', 'NYC'), - ... ('Alice', 'Chicago'), - ... ('Dan', 'Syndey'), - ... ('Edith', 'Paris'), - ... ('Edith', 'Berlin'), - ... ('Zhao', 'Shanghai')] - - >>> # Vacation opportunities - >>> # In what cities do people have friends? - >>> result = join(second, friends, - ... first, cities) - >>> for ((a, b), (c, d)) in sorted(unique(result)): - ... print((a, d)) - ('Alice', 'Berlin') - ('Alice', 'Paris') - ('Alice', 'Shanghai') - ('Edith', 'Chicago') - ('Edith', 'NYC') - ('Zhao', 'Chicago') - ('Zhao', 'NYC') - ('Zhao', 'Berlin') - ('Zhao', 'Paris') - - Specify outer joins with keyword arguments ``left_default`` and/or - ``right_default``. Here is a full outer join in which unmatched elements - are paired with None. - - >>> identity = lambda x: x - >>> list(join(identity, [1, 2, 3], - ... identity, [2, 3, 4], - ... left_default=None, right_default=None)) - [(2, 2), (3, 3), (None, 4), (1, None)] - - Usually the key arguments are callables to be applied to the sequences. If - the keys are not obviously callable then it is assumed that indexing was - intended, e.g. the following is a legal change. - The join is implemented as a hash join and the keys of leftseq must be - hashable. Additionally, if right_default is defined, then keys of rightseq - must also be hashable. - - >>> # result = join(second, friends, first, cities) - >>> result = join(1, friends, 0, cities) # doctest: +SKIP - """ - if not callable(leftkey): - leftkey = getter(leftkey) - if not callable(rightkey): - rightkey = getter(rightkey) - - d = groupby(leftkey, leftseq) - - if left_default == no_default and right_default == no_default: - # Inner Join - for item in rightseq: - key = rightkey(item) - if key in d: - for left_match in d[key]: - yield (left_match, item) - elif left_default != no_default and right_default == no_default: - # Right Join - for item in rightseq: - key = rightkey(item) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - yield (left_default, item) - elif right_default != no_default: - seen_keys = set() - seen = seen_keys.add - - if left_default == no_default: - # Left Join - for item in rightseq: - key = rightkey(item) - seen(key) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - # Full Join - for item in rightseq: - key = rightkey(item) - seen(key) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - yield (left_default, item) - - for key, matches in iteritems(d): - if key not in seen_keys: - for match in matches: - yield (match, right_default) - - -def diff(*seqs, **kwargs): - """ Return those items that differ between sequences - - >>> list(diff([1, 2, 3], [1, 2, 10, 100])) - [(3, 10)] - - Shorter sequences may be padded with a ``default`` value: - - >>> list(diff([1, 2, 3], [1, 2, 10, 100], default=None)) - [(3, 10), (None, 100)] - - A ``key`` function may also be applied to each item to use during - comparisons: - - >>> list(diff(['apples', 'bananas'], ['Apples', 'Oranges'], key=str.lower)) - [('bananas', 'Oranges')] - """ - N = len(seqs) - if N == 1 and isinstance(seqs[0], list): - seqs = seqs[0] - N = len(seqs) - if N < 2: - raise TypeError('Too few sequences given (min 2 required)') - default = kwargs.get('default', no_default) - if default == no_default: - iters = zip(*seqs) - else: - iters = zip_longest(*seqs, fillvalue=default) - key = kwargs.get('key', None) - if key is None: - for items in iters: - if items.count(items[0]) != N: - yield items - else: - for items in iters: - vals = tuple(map(key, items)) - if vals.count(vals[0]) != N: - yield items - - -def topk(k, seq, key=None): - """ Find the k largest elements of a sequence - - Operates lazily in ``n*log(k)`` time - - >>> topk(2, [1, 100, 10, 1000]) - (1000, 100) - - Use a key function to change sorted order - - >>> topk(2, ['Alice', 'Bob', 'Charlie', 'Dan'], key=len) - ('Charlie', 'Alice') - - See also: - heapq.nlargest - """ - if key is not None and not callable(key): - key = getter(key) - return tuple(heapq.nlargest(k, seq, key=key)) - - -def peek(seq): - """ Retrieve the next element of a sequence - - Returns the first element and an iterable equivalent to the original - sequence, still having the element retrieved. - - >>> seq = [0, 1, 2, 3, 4] - >>> first, seq = peek(seq) - >>> first - 0 - >>> list(seq) - [0, 1, 2, 3, 4] - """ - iterator = iter(seq) - item = next(iterator) - return item, itertools.chain((item,), iterator) - - -def peekn(n, seq): - """ Retrieve the next n elements of a sequence - - Returns a tuple of the first n elements and an iterable equivalent - to the original, still having the elements retrieved. - - >>> seq = [0, 1, 2, 3, 4] - >>> first_two, seq = peekn(2, seq) - >>> first_two - (0, 1) - >>> list(seq) - [0, 1, 2, 3, 4] - """ - iterator = iter(seq) - peeked = tuple(take(n, iterator)) - return peeked, itertools.chain(iter(peeked), iterator) - - -def random_sample(prob, seq, random_state=None): - """ Return elements from a sequence with probability of prob - - Returns a lazy iterator of random items from seq. - - ``random_sample`` considers each item independently and without - replacement. See below how the first time it returned 13 items and the - next time it returned 6 items. - - >>> seq = list(range(100)) - >>> list(random_sample(0.1, seq)) # doctest: +SKIP - [6, 9, 19, 35, 45, 50, 58, 62, 68, 72, 78, 86, 95] - >>> list(random_sample(0.1, seq)) # doctest: +SKIP - [6, 44, 54, 61, 69, 94] - - Providing an integer seed for ``random_state`` will result in - deterministic sampling. Given the same seed it will return the same sample - every time. - - >>> list(random_sample(0.1, seq, random_state=2016)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - >>> list(random_sample(0.1, seq, random_state=2016)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - - ``random_state`` can also be any object with a method ``random`` that - returns floats between 0.0 and 1.0 (exclusive). - - >>> from random import Random - >>> randobj = Random(2016) - >>> list(random_sample(0.1, seq, random_state=randobj)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - """ - if not hasattr(random_state, 'random'): - random_state = Random(random_state) - return filter(lambda _: random_state.random() < prob, seq) diff --git a/contrib/python/toolz/py2/toolz/recipes.py b/contrib/python/toolz/py2/toolz/recipes.py deleted file mode 100644 index 08c6c8c1e2..0000000000 --- a/contrib/python/toolz/py2/toolz/recipes.py +++ /dev/null @@ -1,47 +0,0 @@ -import itertools -from .itertoolz import frequencies, pluck, getter -from .compatibility import map - - -__all__ = ('countby', 'partitionby') - - -def countby(key, seq): - """ Count elements of a collection by a key function - - >>> countby(len, ['cat', 'mouse', 'dog']) - {3: 2, 5: 1} - - >>> def iseven(x): return x % 2 == 0 - >>> countby(iseven, [1, 2, 3]) # doctest:+SKIP - {True: 1, False: 2} - - See Also: - groupby - """ - if not callable(key): - key = getter(key) - return frequencies(map(key, seq)) - - -def partitionby(func, seq): - """ Partition a sequence according to a function - - Partition `s` into a sequence of lists such that, when traversing - `s`, every time the output of `func` changes a new list is started - and that and subsequent items are collected into that list. - - >>> is_space = lambda c: c == " " - >>> list(partitionby(is_space, "I have space")) - [('I',), (' ',), ('h', 'a', 'v', 'e'), (' ',), ('s', 'p', 'a', 'c', 'e')] - - >>> is_large = lambda x: x > 10 - >>> list(partitionby(is_large, [1, 2, 1, 99, 88, 33, 99, -1, 5])) - [(1, 2, 1), (99, 88, 33, 99), (-1, 5)] - - See also: - partition - groupby - itertools.groupby - """ - return map(tuple, pluck(1, itertools.groupby(seq, key=func))) diff --git a/contrib/python/toolz/py2/toolz/sandbox/__init__.py b/contrib/python/toolz/py2/toolz/sandbox/__init__.py deleted file mode 100644 index 0abda1cb42..0000000000 --- a/contrib/python/toolz/py2/toolz/sandbox/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -from .core import EqualityHashKey, unzip -from .parallel import fold diff --git a/contrib/python/toolz/py2/toolz/sandbox/core.py b/contrib/python/toolz/py2/toolz/sandbox/core.py deleted file mode 100644 index 915f06c213..0000000000 --- a/contrib/python/toolz/py2/toolz/sandbox/core.py +++ /dev/null @@ -1,133 +0,0 @@ -from toolz.itertoolz import getter, cons, pluck -from itertools import tee, starmap - - -# See #166: https://github.com/pytoolz/toolz/issues/166 -# See #173: https://github.com/pytoolz/toolz/pull/173 -class EqualityHashKey(object): - """ Create a hash key that uses equality comparisons between items. - - This may be used to create hash keys for otherwise unhashable types: - - >>> from toolz import curry - >>> EqualityHashDefault = curry(EqualityHashKey, None) - >>> set(map(EqualityHashDefault, [[], (), [1], [1]])) # doctest: +SKIP - {=[]=, =()=, =[1]=} - - **Caution:** adding N ``EqualityHashKey`` items to a hash container - may require O(N**2) operations, not O(N) as for typical hashable types. - Therefore, a suitable key function such as ``tuple`` or ``frozenset`` - is usually preferred over using ``EqualityHashKey`` if possible. - - The ``key`` argument to ``EqualityHashKey`` should be a function or - index that returns a hashable object that effectively distinguishes - unequal items. This helps avoid the poor scaling that occurs when - using the default key. For example, the above example can be improved - by using a key function that distinguishes items by length or type: - - >>> EqualityHashLen = curry(EqualityHashKey, len) - >>> EqualityHashType = curry(EqualityHashKey, type) # this works too - >>> set(map(EqualityHashLen, [[], (), [1], [1]])) # doctest: +SKIP - {=[]=, =()=, =[1]=} - - ``EqualityHashKey`` is convenient to use when a suitable key function - is complicated or unavailable. For example, the following returns all - unique values based on equality: - - >>> from toolz import unique - >>> vals = [[], [], (), [1], [1], [2], {}, {}, {}] - >>> list(unique(vals, key=EqualityHashDefault)) - [[], (), [1], [2], {}] - - **Warning:** don't change the equality value of an item already in a hash - containter. Unhashable types are unhashable for a reason. For example: - - >>> L1 = [1] ; L2 = [2] - >>> s = set(map(EqualityHashDefault, [L1, L2])) - >>> s # doctest: +SKIP - {=[1]=, =[2]=} - - >>> L1[0] = 2 # Don't do this! ``s`` now has duplicate items! - >>> s # doctest: +SKIP - {=[2]=, =[2]=} - - Although this may appear problematic, immutable data types is a common - idiom in functional programming, and``EqualityHashKey`` easily allows - the same idiom to be used by convention rather than strict requirement. - - See Also: - identity - """ - __slots__ = ['item', 'key'] - _default_hashkey = '__default__hashkey__' - - def __init__(self, key, item): - if key is None: - self.key = self._default_hashkey - elif not callable(key): - self.key = getter(key) - else: - self.key = key - self.item = item - - def __hash__(self): - if self.key == self._default_hashkey: - val = self.key - else: - val = self.key(self.item) - return hash(val) - - def __eq__(self, other): - try: - return (self._default_hashkey == other._default_hashkey and - self.item == other.item) - except AttributeError: - return False - - def __ne__(self, other): - return not self.__eq__(other) - - def __str__(self): - return '=%s=' % str(self.item) - - def __repr__(self): - return '=%s=' % repr(self.item) - - -# See issue #293: https://github.com/pytoolz/toolz/issues/239 -def unzip(seq): - """Inverse of ``zip`` - - >>> a, b = unzip([('a', 1), ('b', 2)]) - >>> list(a) - ['a', 'b'] - >>> list(b) - [1, 2] - - Unlike the naive implementation ``def unzip(seq): zip(*seq)`` this - implementation can handle an infinite sequence ``seq``. - - Caveats: - - * The implementation uses ``tee``, and so can use a significant amount - of auxiliary storage if the resulting iterators are consumed at - different times. - - * The inner sequence cannot be infinite. In Python 3 ``zip(*seq)`` can be - used if ``seq`` is a finite sequence of infinite sequences. - - """ - - seq = iter(seq) - - # Check how many iterators we need - try: - first = tuple(next(seq)) - except StopIteration: - return tuple() - - # and create them - niters = len(first) - seqs = tee(cons(first, seq), niters) - - return tuple(starmap(pluck, enumerate(seqs))) diff --git a/contrib/python/toolz/py2/toolz/sandbox/parallel.py b/contrib/python/toolz/py2/toolz/sandbox/parallel.py deleted file mode 100644 index ef8ed39dbd..0000000000 --- a/contrib/python/toolz/py2/toolz/sandbox/parallel.py +++ /dev/null @@ -1,76 +0,0 @@ -import functools -from toolz.itertoolz import partition_all -from toolz.compatibility import reduce, map -from toolz.utils import no_default - - -def _reduce(func, seq, initial=None): - if initial is None: - return functools.reduce(func, seq) - else: - return functools.reduce(func, seq, initial) - - -def fold(binop, seq, default=no_default, map=map, chunksize=128, combine=None): - """ - Reduce without guarantee of ordered reduction. - - inputs: - - ``binop`` - associative operator. The associative property allows us to - leverage a parallel map to perform reductions in parallel. - ``seq`` - a sequence to be aggregated - ``default`` - an identity element like 0 for ``add`` or 1 for mul - - ``map`` - an implementation of ``map``. This may be parallel and - determines how work is distributed. - ``chunksize`` - Number of elements of ``seq`` that should be handled - within a single function call - ``combine`` - Binary operator to combine two intermediate results. - If ``binop`` is of type (total, item) -> total - then ``combine`` is of type (total, total) -> total - Defaults to ``binop`` for common case of operators like add - - Fold chunks up the collection into blocks of size ``chunksize`` and then - feeds each of these to calls to ``reduce``. This work is distributed - with a call to ``map``, gathered back and then refolded to finish the - computation. In this way ``fold`` specifies only how to chunk up data but - leaves the distribution of this work to an externally provided ``map`` - function. This function can be sequential or rely on multithreading, - multiprocessing, or even distributed solutions. - - If ``map`` intends to serialize functions it should be prepared to accept - and serialize lambdas. Note that the standard ``pickle`` module fails - here. - - Example - ------- - - >>> # Provide a parallel map to accomplish a parallel sum - >>> from operator import add - >>> fold(add, [1, 2, 3, 4], chunksize=2, map=map) - 10 - """ - assert chunksize > 1 - - if combine is None: - combine = binop - - chunks = partition_all(chunksize, seq) - - # Evaluate sequence in chunks via map - if default == no_default: - results = map( - functools.partial(_reduce, binop), - chunks) - else: - results = map( - functools.partial(_reduce, binop, initial=default), - chunks) - - results = list(results) # TODO: Support complete laziness - - if len(results) == 1: # Return completed result - return results[0] - else: # Recurse to reaggregate intermediate results - return fold(combine, results, map=map, chunksize=chunksize) diff --git a/contrib/python/toolz/py2/toolz/utils.py b/contrib/python/toolz/py2/toolz/utils.py deleted file mode 100644 index 1002c4649f..0000000000 --- a/contrib/python/toolz/py2/toolz/utils.py +++ /dev/null @@ -1,9 +0,0 @@ -def raises(err, lamda): - try: - lamda() - return False - except err: - return True - - -no_default = '__no__default__' diff --git a/contrib/python/toolz/py2/ya.make b/contrib/python/toolz/py2/ya.make deleted file mode 100644 index f64a13da67..0000000000 --- a/contrib/python/toolz/py2/ya.make +++ /dev/null @@ -1,41 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(0.10.0) - -LICENSE(BSD-3-Clause) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - tlz/__init__.py - tlz/_build_tlz.py - toolz/__init__.py - toolz/_signatures.py - toolz/compatibility.py - toolz/curried/__init__.py - toolz/curried/exceptions.py - toolz/curried/operator.py - toolz/dicttoolz.py - toolz/functoolz.py - toolz/itertoolz.py - toolz/recipes.py - toolz/sandbox/__init__.py - toolz/sandbox/core.py - toolz/sandbox/parallel.py - toolz/utils.py -) - -RESOURCE_FILES( - PREFIX contrib/python/toolz/py2/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/toolz/py3/.dist-info/METADATA b/contrib/python/toolz/py3/.dist-info/METADATA deleted file mode 100644 index 0af60db4cf..0000000000 --- a/contrib/python/toolz/py3/.dist-info/METADATA +++ /dev/null @@ -1,159 +0,0 @@ -Metadata-Version: 2.1 -Name: toolz -Version: 0.12.0 -Summary: List processing tools and functional utilities -Home-page: https://github.com/pytoolz/toolz/ -Author: https://raw.github.com/pytoolz/toolz/master/AUTHORS.md -Maintainer: Erik Welch -Maintainer-email: erik.n.welch@gmail.com -License: BSD -Keywords: functional utility itertools functools -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: License :: OSI Approved :: BSD License -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Requires-Python: >=3.5 - -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 3.5+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://github.com/pytoolz/toolz/workflows/Test/badge.svg - :target: https://github.com/pytoolz/toolz/actions -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz - - diff --git a/contrib/python/toolz/py3/.dist-info/top_level.txt b/contrib/python/toolz/py3/.dist-info/top_level.txt deleted file mode 100644 index e58ef014ac..0000000000 --- a/contrib/python/toolz/py3/.dist-info/top_level.txt +++ /dev/null @@ -1,2 +0,0 @@ -tlz -toolz diff --git a/contrib/python/toolz/py3/AUTHORS.md b/contrib/python/toolz/py3/AUTHORS.md deleted file mode 100644 index bd4a563d9b..0000000000 --- a/contrib/python/toolz/py3/AUTHORS.md +++ /dev/null @@ -1,33 +0,0 @@ -[Matthew Rocklin](http://matthewrocklin.com) [@mrocklin](http://github.com/mrocklin/) - -[John Jacobsen](http://eigenhombre.com) [@eigenhombre](http://github.com/eigenhombre/) - -Erik Welch [@eriknw](https://github.com/eriknw/) - -John Crichton [@jcrichton](https://github.com/jcrichton/) - -Han Semaj [@microamp](https://github.com/microamp/) - -[Graeme Coupar](https://twitter.com/obmarg) [@obmarg](https://github.com/obmarg/) - -[Leonid Shvechikov](http://brainstorage.me/shvechikov) [@shvechikov](https://github.com/shvechikov) - -Lars Buitinck [@larsmans](http://github.com/larsmans) - -José Ricardo [@josericardo](https://github.com/josericardo) - -Tom Prince [@tomprince](https://github.com/tomprince) - -Bart van Merriënboer [@bartvm](https://github.com/bartvm) - -Nikolaos-Digenis Karagiannis [@digenis](https://github.com/digenis/) - -[Antonio Lima](https://twitter.com/themiurgo) [@themiurgo](https://github.com/themiurgo/) - -Joe Jevnik [@llllllllll](https://github.com/llllllllll) - -Rory Kirchner [@roryk](https://github.com/roryk) - -[Steven Cutting](http://steven-cutting.github.io) [@steven_cutting](https://github.com/steven-cutting) - -Aric Coady [@coady](https://github.com/coady) diff --git a/contrib/python/toolz/py3/LICENSE.txt b/contrib/python/toolz/py3/LICENSE.txt deleted file mode 100644 index eeb91b202c..0000000000 --- a/contrib/python/toolz/py3/LICENSE.txt +++ /dev/null @@ -1,28 +0,0 @@ -Copyright (c) 2013 Matthew Rocklin - -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - - a. Redistributions of source code must retain the above copyright notice, - this list of conditions and the following disclaimer. - b. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - c. Neither the name of toolz nor the names of its contributors - may be used to endorse or promote products derived from this software - without specific prior written permission. - - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR -ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL -DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR -SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER -CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY -OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH -DAMAGE. diff --git a/contrib/python/toolz/py3/README.rst b/contrib/python/toolz/py3/README.rst deleted file mode 100644 index e62ac7917e..0000000000 --- a/contrib/python/toolz/py3/README.rst +++ /dev/null @@ -1,132 +0,0 @@ -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 3.5+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://github.com/pytoolz/toolz/workflows/Test/badge.svg - :target: https://github.com/pytoolz/toolz/actions -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz diff --git a/contrib/python/toolz/py3/tlz/__init__.py b/contrib/python/toolz/py3/tlz/__init__.py deleted file mode 100644 index 9c9c84afe1..0000000000 --- a/contrib/python/toolz/py3/tlz/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -"""``tlz`` mirrors the ``toolz`` API and uses ``cytoolz`` if possible. - -The ``tlz`` package is installed when ``toolz`` is installed. It provides -a convenient way to use functions from ``cytoolz``--a faster Cython -implementation of ``toolz``--if it is installed, otherwise it uses -functions from ``toolz``. -""" - -from . import _build_tlz diff --git a/contrib/python/toolz/py3/tlz/_build_tlz.py b/contrib/python/toolz/py3/tlz/_build_tlz.py deleted file mode 100644 index 3ac783699e..0000000000 --- a/contrib/python/toolz/py3/tlz/_build_tlz.py +++ /dev/null @@ -1,92 +0,0 @@ -import sys -import types -import toolz -from importlib import import_module -from importlib.machinery import ModuleSpec - - -class TlzLoader: - """ Finds and loads ``tlz`` modules when added to sys.meta_path""" - def __init__(self): - self.always_from_toolz = { - toolz.pipe, - } - - def _load_toolz(self, fullname): - rv = {} - package, dot, submodules = fullname.partition('.') - try: - module_name = ''.join(['cytoolz', dot, submodules]) - rv['cytoolz'] = import_module(module_name) - except ImportError: - pass - try: - module_name = ''.join(['toolz', dot, submodules]) - rv['toolz'] = import_module(module_name) - except ImportError: - pass - if not rv: - raise ImportError(fullname) - return rv - - def find_module(self, fullname, path=None): # pragma: py3 no cover - package, dot, submodules = fullname.partition('.') - if package == 'tlz': - return self - - def load_module(self, fullname): # pragma: py3 no cover - if fullname in sys.modules: # pragma: no cover - return sys.modules[fullname] - spec = ModuleSpec(fullname, self) - module = self.create_module(spec) - sys.modules[fullname] = module - self.exec_module(module) - return module - - def find_spec(self, fullname, path, target=None): # pragma: no cover - package, dot, submodules = fullname.partition('.') - if package == 'tlz': - return ModuleSpec(fullname, self) - - def create_module(self, spec): - return types.ModuleType(spec.name) - - def exec_module(self, module): - toolz_mods = self._load_toolz(module.__name__) - fast_mod = toolz_mods.get('cytoolz') or toolz_mods['toolz'] - slow_mod = toolz_mods.get('toolz') or toolz_mods['cytoolz'] - module.__dict__.update(toolz.merge(fast_mod.__dict__, module.__dict__)) - package = fast_mod.__package__ - if package is not None: - package, dot, submodules = package.partition('.') - module.__package__ = ''.join(['tlz', dot, submodules]) - if not module.__doc__: - module.__doc__ = fast_mod.__doc__ - - # show file from toolz during introspection - try: - module.__file__ = slow_mod.__file__ - except AttributeError: - pass - - for k, v in fast_mod.__dict__.items(): - tv = slow_mod.__dict__.get(k) - try: - hash(tv) - except TypeError: - tv = None - if tv in self.always_from_toolz: - module.__dict__[k] = tv - elif ( - isinstance(v, types.ModuleType) - and v.__package__ == fast_mod.__name__ - ): - package, dot, submodules = v.__name__.partition('.') - module_name = ''.join(['tlz', dot, submodules]) - submodule = import_module(module_name) - module.__dict__[k] = submodule - - -tlz_loader = TlzLoader() -sys.meta_path.append(tlz_loader) -tlz_loader.exec_module(sys.modules['tlz']) diff --git a/contrib/python/toolz/py3/toolz/__init__.py b/contrib/python/toolz/py3/toolz/__init__.py deleted file mode 100644 index ba49a662fc..0000000000 --- a/contrib/python/toolz/py3/toolz/__init__.py +++ /dev/null @@ -1,26 +0,0 @@ -from .itertoolz import * - -from .functoolz import * - -from .dicttoolz import * - -from .recipes import * - -from functools import partial, reduce - -sorted = sorted - -map = map - -filter = filter - -# Aliases -comp = compose - -from . import curried, sandbox - -functoolz._sigs.create_signature_registry() - -from ._version import get_versions -__version__ = get_versions()['version'] -del get_versions diff --git a/contrib/python/toolz/py3/toolz/_signatures.py b/contrib/python/toolz/py3/toolz/_signatures.py deleted file mode 100644 index 3ce1616a85..0000000000 --- a/contrib/python/toolz/py3/toolz/_signatures.py +++ /dev/null @@ -1,785 +0,0 @@ -"""Internal module for better introspection of builtins. - -The main functions are ``is_builtin_valid_args``, ``is_builtin_partial_args``, -and ``has_unknown_args``. Other functions in this module support these three. - -Notably, we create a ``signatures`` registry to enable introspection of -builtin functions in any Python version. This includes builtins that -have more than one valid signature. Currently, the registry includes -builtins from ``builtins``, ``functools``, ``itertools``, and ``operator`` -modules. More can be added as requested. We don't guarantee full coverage. - -Everything in this module should be regarded as implementation details. -Users should try to not use this module directly. -""" -import functools -import inspect -import itertools -import operator -from importlib import import_module - -from .functoolz import (is_partial_args, is_arity, has_varargs, - has_keywords, num_required_args) - -import builtins - -# We mock builtin callables using lists of tuples with lambda functions. -# -# The tuple spec is (num_position_args, lambda_func, keyword_only_args). -# -# num_position_args: -# - The number of positional-only arguments. If not specified, -# all positional arguments are considered positional-only. -# -# lambda_func: -# - lambda function that matches a signature of a builtin, but does -# not include keyword-only arguments. -# -# keyword_only_args: (optional) -# - Tuple of keyword-only argumemts. - -module_info = {} - -module_info[builtins] = dict( - abs=[ - lambda x: None], - all=[ - lambda iterable: None], - anext=[ - lambda aiterator: None, - lambda aiterator, default: None], - any=[ - lambda iterable: None], - apply=[ - lambda object: None, - lambda object, args: None, - lambda object, args, kwargs: None], - ascii=[ - lambda obj: None], - bin=[ - lambda number: None], - bool=[ - lambda x=False: None], - buffer=[ - lambda object: None, - lambda object, offset: None, - lambda object, offset, size: None], - bytearray=[ - lambda: None, - lambda int: None, - lambda string, encoding='utf8', errors='strict': None], - callable=[ - lambda obj: None], - chr=[ - lambda i: None], - classmethod=[ - lambda function: None], - cmp=[ - lambda x, y: None], - coerce=[ - lambda x, y: None], - complex=[ - lambda real=0, imag=0: None], - delattr=[ - lambda obj, name: None], - dict=[ - lambda **kwargs: None, - lambda mapping, **kwargs: None], - dir=[ - lambda: None, - lambda object: None], - divmod=[ - lambda x, y: None], - enumerate=[ - (0, lambda iterable, start=0: None)], - eval=[ - lambda source: None, - lambda source, globals: None, - lambda source, globals, locals: None], - execfile=[ - lambda filename: None, - lambda filename, globals: None, - lambda filename, globals, locals: None], - file=[ - (0, lambda name, mode='r', buffering=-1: None)], - filter=[ - lambda function, iterable: None], - float=[ - lambda x=0.0: None], - format=[ - lambda value: None, - lambda value, format_spec: None], - frozenset=[ - lambda: None, - lambda iterable: None], - getattr=[ - lambda object, name: None, - lambda object, name, default: None], - globals=[ - lambda: None], - hasattr=[ - lambda obj, name: None], - hash=[ - lambda obj: None], - hex=[ - lambda number: None], - id=[ - lambda obj: None], - input=[ - lambda: None, - lambda prompt: None], - int=[ - lambda x=0: None, - (0, lambda x, base=10: None)], - intern=[ - lambda string: None], - isinstance=[ - lambda obj, class_or_tuple: None], - issubclass=[ - lambda cls, class_or_tuple: None], - iter=[ - lambda iterable: None, - lambda callable, sentinel: None], - len=[ - lambda obj: None], - list=[ - lambda: None, - lambda iterable: None], - locals=[ - lambda: None], - long=[ - lambda x=0: None, - (0, lambda x, base=10: None)], - map=[ - lambda func, sequence, *iterables: None], - memoryview=[ - (0, lambda object: None)], - next=[ - lambda iterator: None, - lambda iterator, default: None], - object=[ - lambda: None], - oct=[ - lambda number: None], - ord=[ - lambda c: None], - pow=[ - lambda x, y: None, - lambda x, y, z: None], - property=[ - lambda fget=None, fset=None, fdel=None, doc=None: None], - range=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - raw_input=[ - lambda: None, - lambda prompt: None], - reduce=[ - lambda function, sequence: None, - lambda function, sequence, initial: None], - reload=[ - lambda module: None], - repr=[ - lambda obj: None], - reversed=[ - lambda sequence: None], - round=[ - (0, lambda number, ndigits=0: None)], - set=[ - lambda: None, - lambda iterable: None], - setattr=[ - lambda obj, name, value: None], - slice=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - staticmethod=[ - lambda function: None], - sum=[ - lambda iterable: None, - lambda iterable, start: None], - super=[ - lambda type: None, - lambda type, obj: None], - tuple=[ - lambda: None, - lambda iterable: None], - type=[ - lambda object: None, - lambda name, bases, dict: None], - unichr=[ - lambda i: None], - unicode=[ - lambda object: None, - lambda string='', encoding='utf8', errors='strict': None], - vars=[ - lambda: None, - lambda object: None], - xrange=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - zip=[ - lambda *iterables: None], - __build_class__=[ - (2, lambda func, name, *bases, **kwds: None, ('metaclass',))], - __import__=[ - (0, lambda name, globals=None, locals=None, fromlist=None, - level=None: None)], -) -module_info[builtins]['exec'] = [ - lambda source: None, - lambda source, globals: None, - lambda source, globals, locals: None] - -module_info[builtins].update( - breakpoint=[ - lambda *args, **kws: None], - bytes=[ - lambda: None, - lambda int: None, - lambda string, encoding='utf8', errors='strict': None], - compile=[ - (0, lambda source, filename, mode, flags=0, - dont_inherit=False, optimize=-1: None)], - max=[ - (1, lambda iterable: None, ('default', 'key',)), - (1, lambda arg1, arg2, *args: None, ('key',))], - min=[ - (1, lambda iterable: None, ('default', 'key',)), - (1, lambda arg1, arg2, *args: None, ('key',))], - open=[ - (0, lambda file, mode='r', buffering=-1, encoding=None, - errors=None, newline=None, closefd=True, opener=None: None)], - sorted=[ - (1, lambda iterable: None, ('key', 'reverse'))], - str=[ - lambda object='', encoding='utf', errors='strict': None], -) -module_info[builtins]['print'] = [ - (0, lambda *args: None, ('sep', 'end', 'file', 'flush',))] - - -module_info[functools] = dict( - cmp_to_key=[ - (0, lambda mycmp: None)], - partial=[ - lambda func, *args, **kwargs: None], - partialmethod=[ - lambda func, *args, **kwargs: None], - reduce=[ - lambda function, sequence: None, - lambda function, sequence, initial: None], -) - -module_info[itertools] = dict( - accumulate=[ - (0, lambda iterable, func=None: None)], - chain=[ - lambda *iterables: None], - combinations=[ - (0, lambda iterable, r: None)], - combinations_with_replacement=[ - (0, lambda iterable, r: None)], - compress=[ - (0, lambda data, selectors: None)], - count=[ - lambda start=0, step=1: None], - cycle=[ - lambda iterable: None], - dropwhile=[ - lambda predicate, iterable: None], - filterfalse=[ - lambda function, sequence: None], - groupby=[ - (0, lambda iterable, key=None: None)], - ifilter=[ - lambda function, sequence: None], - ifilterfalse=[ - lambda function, sequence: None], - imap=[ - lambda func, sequence, *iterables: None], - islice=[ - lambda iterable, stop: None, - lambda iterable, start, stop: None, - lambda iterable, start, stop, step: None], - izip=[ - lambda *iterables: None], - izip_longest=[ - (0, lambda *iterables: None, ('fillvalue',))], - permutations=[ - (0, lambda iterable, r=0: None)], - repeat=[ - (0, lambda object, times=0: None)], - starmap=[ - lambda function, sequence: None], - takewhile=[ - lambda predicate, iterable: None], - tee=[ - lambda iterable: None, - lambda iterable, n: None], - zip_longest=[ - (0, lambda *iterables: None, ('fillvalue',))], -) - -module_info[itertools].update( - product=[ - (0, lambda *iterables: None, ('repeat',))], -) - - -module_info[operator] = dict( - __abs__=[ - lambda a: None], - __add__=[ - lambda a, b: None], - __and__=[ - lambda a, b: None], - __concat__=[ - lambda a, b: None], - __contains__=[ - lambda a, b: None], - __delitem__=[ - lambda a, b: None], - __delslice__=[ - lambda a, b, c: None], - __div__=[ - lambda a, b: None], - __eq__=[ - lambda a, b: None], - __floordiv__=[ - lambda a, b: None], - __ge__=[ - lambda a, b: None], - __getitem__=[ - lambda a, b: None], - __getslice__=[ - lambda a, b, c: None], - __gt__=[ - lambda a, b: None], - __iadd__=[ - lambda a, b: None], - __iand__=[ - lambda a, b: None], - __iconcat__=[ - lambda a, b: None], - __idiv__=[ - lambda a, b: None], - __ifloordiv__=[ - lambda a, b: None], - __ilshift__=[ - lambda a, b: None], - __imatmul__=[ - lambda a, b: None], - __imod__=[ - lambda a, b: None], - __imul__=[ - lambda a, b: None], - __index__=[ - lambda a: None], - __inv__=[ - lambda a: None], - __invert__=[ - lambda a: None], - __ior__=[ - lambda a, b: None], - __ipow__=[ - lambda a, b: None], - __irepeat__=[ - lambda a, b: None], - __irshift__=[ - lambda a, b: None], - __isub__=[ - lambda a, b: None], - __itruediv__=[ - lambda a, b: None], - __ixor__=[ - lambda a, b: None], - __le__=[ - lambda a, b: None], - __lshift__=[ - lambda a, b: None], - __lt__=[ - lambda a, b: None], - __matmul__=[ - lambda a, b: None], - __mod__=[ - lambda a, b: None], - __mul__=[ - lambda a, b: None], - __ne__=[ - lambda a, b: None], - __neg__=[ - lambda a: None], - __not__=[ - lambda a: None], - __or__=[ - lambda a, b: None], - __pos__=[ - lambda a: None], - __pow__=[ - lambda a, b: None], - __repeat__=[ - lambda a, b: None], - __rshift__=[ - lambda a, b: None], - __setitem__=[ - lambda a, b, c: None], - __setslice__=[ - lambda a, b, c, d: None], - __sub__=[ - lambda a, b: None], - __truediv__=[ - lambda a, b: None], - __xor__=[ - lambda a, b: None], - _abs=[ - lambda x: None], - _compare_digest=[ - lambda a, b: None], - abs=[ - lambda a: None], - add=[ - lambda a, b: None], - and_=[ - lambda a, b: None], - attrgetter=[ - lambda attr, *args: None], - concat=[ - lambda a, b: None], - contains=[ - lambda a, b: None], - countOf=[ - lambda a, b: None], - delitem=[ - lambda a, b: None], - delslice=[ - lambda a, b, c: None], - div=[ - lambda a, b: None], - eq=[ - lambda a, b: None], - floordiv=[ - lambda a, b: None], - ge=[ - lambda a, b: None], - getitem=[ - lambda a, b: None], - getslice=[ - lambda a, b, c: None], - gt=[ - lambda a, b: None], - iadd=[ - lambda a, b: None], - iand=[ - lambda a, b: None], - iconcat=[ - lambda a, b: None], - idiv=[ - lambda a, b: None], - ifloordiv=[ - lambda a, b: None], - ilshift=[ - lambda a, b: None], - imatmul=[ - lambda a, b: None], - imod=[ - lambda a, b: None], - imul=[ - lambda a, b: None], - index=[ - lambda a: None], - indexOf=[ - lambda a, b: None], - inv=[ - lambda a: None], - invert=[ - lambda a: None], - ior=[ - lambda a, b: None], - ipow=[ - lambda a, b: None], - irepeat=[ - lambda a, b: None], - irshift=[ - lambda a, b: None], - is_=[ - lambda a, b: None], - is_not=[ - lambda a, b: None], - isCallable=[ - lambda a: None], - isMappingType=[ - lambda a: None], - isNumberType=[ - lambda a: None], - isSequenceType=[ - lambda a: None], - isub=[ - lambda a, b: None], - itemgetter=[ - lambda item, *args: None], - itruediv=[ - lambda a, b: None], - ixor=[ - lambda a, b: None], - le=[ - lambda a, b: None], - length_hint=[ - lambda obj: None, - lambda obj, default: None], - lshift=[ - lambda a, b: None], - lt=[ - lambda a, b: None], - matmul=[ - lambda a, b: None], - methodcaller=[ - lambda name, *args, **kwargs: None], - mod=[ - lambda a, b: None], - mul=[ - lambda a, b: None], - ne=[ - lambda a, b: None], - neg=[ - lambda a: None], - not_=[ - lambda a: None], - or_=[ - lambda a, b: None], - pos=[ - lambda a: None], - pow=[ - lambda a, b: None], - repeat=[ - lambda a, b: None], - rshift=[ - lambda a, b: None], - sequenceIncludes=[ - lambda a, b: None], - setitem=[ - lambda a, b, c: None], - setslice=[ - lambda a, b, c, d: None], - sub=[ - lambda a, b: None], - truediv=[ - lambda a, b: None], - truth=[ - lambda a: None], - xor=[ - lambda a, b: None], -) - -module_info['toolz'] = dict( - curry=[ - (0, lambda *args, **kwargs: None)], - excepts=[ - (0, lambda exc, func, handler=None: None)], - flip=[ - (0, lambda func=None, a=None, b=None: None)], - juxt=[ - (0, lambda *funcs: None)], - memoize=[ - (0, lambda func=None, cache=None, key=None: None)], -) - -module_info['toolz.functoolz'] = dict( - Compose=[ - (0, lambda funcs: None)], - InstanceProperty=[ - (0, lambda fget=None, fset=None, fdel=None, doc=None, - classval=None: None)], -) - - -def num_pos_args(sigspec): - """ Return the number of positional arguments. ``f(x, y=1)`` has 1""" - return sum(1 for x in sigspec.parameters.values() - if x.kind == x.POSITIONAL_OR_KEYWORD - and x.default is x.empty) - - -def get_exclude_keywords(num_pos_only, sigspec): - """ Return the names of position-only arguments if func has **kwargs""" - if num_pos_only == 0: - return () - has_kwargs = any(x.kind == x.VAR_KEYWORD - for x in sigspec.parameters.values()) - if not has_kwargs: - return () - pos_args = list(sigspec.parameters.values())[:num_pos_only] - return tuple(x.name for x in pos_args) - - -def signature_or_spec(func): - try: - return inspect.signature(func) - except (ValueError, TypeError): - return None - - -def expand_sig(sig): - """ Convert the signature spec in ``module_info`` to add to ``signatures`` - - The input signature spec is one of: - - ``lambda_func`` - - ``(num_position_args, lambda_func)`` - - ``(num_position_args, lambda_func, keyword_only_args)`` - - The output signature spec is: - ``(num_position_args, lambda_func, keyword_exclude, sigspec)`` - - where ``keyword_exclude`` includes keyword only arguments and, if variadic - keywords is present, the names of position-only argument. The latter is - included to support builtins such as ``partial(func, *args, **kwargs)``, - which allows ``func=`` to be used as a keyword even though it's the name - of a positional argument. - """ - if isinstance(sig, tuple): - if len(sig) == 3: - num_pos_only, func, keyword_only = sig - assert isinstance(sig[-1], tuple) - else: - num_pos_only, func = sig - keyword_only = () - sigspec = signature_or_spec(func) - else: - func = sig - sigspec = signature_or_spec(func) - num_pos_only = num_pos_args(sigspec) - keyword_only = () - keyword_exclude = get_exclude_keywords(num_pos_only, sigspec) - return num_pos_only, func, keyword_only + keyword_exclude, sigspec - - -signatures = {} - - -def create_signature_registry(module_info=module_info, signatures=signatures): - for module, info in module_info.items(): - if isinstance(module, str): - module = import_module(module) - for name, sigs in info.items(): - if hasattr(module, name): - new_sigs = tuple(expand_sig(sig) for sig in sigs) - signatures[getattr(module, name)] = new_sigs - - -def check_valid(sig, args, kwargs): - """ Like ``is_valid_args`` for the given signature spec""" - num_pos_only, func, keyword_exclude, sigspec = sig - if len(args) < num_pos_only: - return False - if keyword_exclude: - kwargs = dict(kwargs) - for item in keyword_exclude: - kwargs.pop(item, None) - try: - func(*args, **kwargs) - return True - except TypeError: - return False - - -def _is_valid_args(func, args, kwargs): - """ Like ``is_valid_args`` for builtins in our ``signatures`` registry""" - if func not in signatures: - return None - sigs = signatures[func] - return any(check_valid(sig, args, kwargs) for sig in sigs) - - -def check_partial(sig, args, kwargs): - """ Like ``is_partial_args`` for the given signature spec""" - num_pos_only, func, keyword_exclude, sigspec = sig - if len(args) < num_pos_only: - pad = (None,) * (num_pos_only - len(args)) - args = args + pad - if keyword_exclude: - kwargs = dict(kwargs) - for item in keyword_exclude: - kwargs.pop(item, None) - return is_partial_args(func, args, kwargs, sigspec) - - -def _is_partial_args(func, args, kwargs): - """ Like ``is_partial_args`` for builtins in our ``signatures`` registry""" - if func not in signatures: - return None - sigs = signatures[func] - return any(check_partial(sig, args, kwargs) for sig in sigs) - - -def check_arity(n, sig): - num_pos_only, func, keyword_exclude, sigspec = sig - if keyword_exclude or num_pos_only > n: - return False - return is_arity(n, func, sigspec) - - -def _is_arity(n, func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_arity(n, sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_varargs(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - return has_varargs(func, sigspec) - - -def _has_varargs(func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_varargs(sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_keywords(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - if keyword_exclude: - return True - return has_keywords(func, sigspec) - - -def _has_keywords(func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_keywords(sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_required_args(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - return num_required_args(func, sigspec) - - -def _num_required_args(func): - if func not in signatures: - return None - sigs = signatures[func] - vals = [check_required_args(sig) for sig in sigs] - val = vals[0] - if all(x == val for x in vals): - return val - return None diff --git a/contrib/python/toolz/py3/toolz/_version.py b/contrib/python/toolz/py3/toolz/_version.py deleted file mode 100644 index 6e979d1048..0000000000 --- a/contrib/python/toolz/py3/toolz/_version.py +++ /dev/null @@ -1,21 +0,0 @@ - -# This file was generated by 'versioneer.py' (0.18) from -# revision-control system data, or from the parent directory name of an -# unpacked source archive. Distribution tarballs contain a pre-generated copy -# of this file. - -import json - -version_json = ''' -{ - "date": "2022-07-09T23:15:45-0500", - "dirty": false, - "error": null, - "full-revisionid": "245b78e6320c41a4a9cdd15c6123681fbfb62843", - "version": "0.12.0" -} -''' # END VERSION_JSON - - -def get_versions(): - return json.loads(version_json) diff --git a/contrib/python/toolz/py3/toolz/compatibility.py b/contrib/python/toolz/py3/toolz/compatibility.py deleted file mode 100644 index 28bef91dc8..0000000000 --- a/contrib/python/toolz/py3/toolz/compatibility.py +++ /dev/null @@ -1,30 +0,0 @@ -import warnings -warnings.warn("The toolz.compatibility module is no longer " - "needed in Python 3 and has been deprecated. Please " - "import these utilities directly from the standard library. " - "This module will be removed in a future release.", - category=DeprecationWarning, stacklevel=2) - -import operator -import sys - -PY3 = sys.version_info[0] > 2 -PY34 = sys.version_info[0] == 3 and sys.version_info[1] == 4 -PYPY = hasattr(sys, 'pypy_version_info') and PY3 - -__all__ = ('map', 'filter', 'range', 'zip', 'reduce', 'zip_longest', - 'iteritems', 'iterkeys', 'itervalues', 'filterfalse', - 'PY3', 'PY34', 'PYPY') - - -map = map -filter = filter -range = range -zip = zip -from functools import reduce -from itertools import zip_longest -from itertools import filterfalse -iteritems = operator.methodcaller('items') -iterkeys = operator.methodcaller('keys') -itervalues = operator.methodcaller('values') -from collections.abc import Sequence diff --git a/contrib/python/toolz/py3/toolz/curried/__init__.py b/contrib/python/toolz/py3/toolz/curried/__init__.py deleted file mode 100644 index 356eddbd3b..0000000000 --- a/contrib/python/toolz/py3/toolz/curried/__init__.py +++ /dev/null @@ -1,103 +0,0 @@ -""" -Alternate namespace for toolz such that all functions are curried - -Currying provides implicit partial evaluation of all functions - -Example: - - Get usually requires two arguments, an index and a collection - >>> from toolz.curried import get - >>> get(0, ('a', 'b')) - 'a' - - When we use it in higher order functions we often want to pass a partially - evaluated form - >>> data = [(1, 2), (11, 22), (111, 222)] - >>> list(map(lambda seq: get(0, seq), data)) - [1, 11, 111] - - The curried version allows simple expression of partial evaluation - >>> list(map(get(0), data)) - [1, 11, 111] - -See Also: - toolz.functoolz.curry -""" -import toolz -from . import operator -from toolz import ( - apply, - comp, - complement, - compose, - compose_left, - concat, - concatv, - count, - curry, - diff, - first, - flip, - frequencies, - identity, - interleave, - isdistinct, - isiterable, - juxt, - last, - memoize, - merge_sorted, - peek, - pipe, - second, - thread_first, - thread_last, -) -from .exceptions import merge, merge_with - -accumulate = toolz.curry(toolz.accumulate) -assoc = toolz.curry(toolz.assoc) -assoc_in = toolz.curry(toolz.assoc_in) -cons = toolz.curry(toolz.cons) -countby = toolz.curry(toolz.countby) -dissoc = toolz.curry(toolz.dissoc) -do = toolz.curry(toolz.do) -drop = toolz.curry(toolz.drop) -excepts = toolz.curry(toolz.excepts) -filter = toolz.curry(toolz.filter) -get = toolz.curry(toolz.get) -get_in = toolz.curry(toolz.get_in) -groupby = toolz.curry(toolz.groupby) -interpose = toolz.curry(toolz.interpose) -itemfilter = toolz.curry(toolz.itemfilter) -itemmap = toolz.curry(toolz.itemmap) -iterate = toolz.curry(toolz.iterate) -join = toolz.curry(toolz.join) -keyfilter = toolz.curry(toolz.keyfilter) -keymap = toolz.curry(toolz.keymap) -map = toolz.curry(toolz.map) -mapcat = toolz.curry(toolz.mapcat) -nth = toolz.curry(toolz.nth) -partial = toolz.curry(toolz.partial) -partition = toolz.curry(toolz.partition) -partition_all = toolz.curry(toolz.partition_all) -partitionby = toolz.curry(toolz.partitionby) -peekn = toolz.curry(toolz.peekn) -pluck = toolz.curry(toolz.pluck) -random_sample = toolz.curry(toolz.random_sample) -reduce = toolz.curry(toolz.reduce) -reduceby = toolz.curry(toolz.reduceby) -remove = toolz.curry(toolz.remove) -sliding_window = toolz.curry(toolz.sliding_window) -sorted = toolz.curry(toolz.sorted) -tail = toolz.curry(toolz.tail) -take = toolz.curry(toolz.take) -take_nth = toolz.curry(toolz.take_nth) -topk = toolz.curry(toolz.topk) -unique = toolz.curry(toolz.unique) -update_in = toolz.curry(toolz.update_in) -valfilter = toolz.curry(toolz.valfilter) -valmap = toolz.curry(toolz.valmap) - -del exceptions -del toolz diff --git a/contrib/python/toolz/py3/toolz/curried/exceptions.py b/contrib/python/toolz/py3/toolz/curried/exceptions.py deleted file mode 100644 index 75a52bbbf2..0000000000 --- a/contrib/python/toolz/py3/toolz/curried/exceptions.py +++ /dev/null @@ -1,18 +0,0 @@ -import toolz - - -__all__ = ['merge_with', 'merge'] - - -@toolz.curry -def merge_with(func, d, *dicts, **kwargs): - return toolz.merge_with(func, d, *dicts, **kwargs) - - -@toolz.curry -def merge(d, *dicts, **kwargs): - return toolz.merge(d, *dicts, **kwargs) - - -merge_with.__doc__ = toolz.merge_with.__doc__ -merge.__doc__ = toolz.merge.__doc__ diff --git a/contrib/python/toolz/py3/toolz/curried/operator.py b/contrib/python/toolz/py3/toolz/curried/operator.py deleted file mode 100644 index 35979a6851..0000000000 --- a/contrib/python/toolz/py3/toolz/curried/operator.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import absolute_import - -import operator - -from toolz.functoolz import curry - - -# Tests will catch if/when this needs updated -IGNORE = { - "__abs__", "__index__", "__inv__", "__invert__", "__neg__", "__not__", - "__pos__", "_abs", "abs", "attrgetter", "index", "inv", "invert", - "itemgetter", "neg", "not_", "pos", "truth" -} -locals().update( - {name: f if name in IGNORE else curry(f) - for name, f in vars(operator).items() if callable(f)} -) - -# Clean up the namespace. -del IGNORE -del curry -del operator diff --git a/contrib/python/toolz/py3/toolz/dicttoolz.py b/contrib/python/toolz/py3/toolz/dicttoolz.py deleted file mode 100644 index 457bc26928..0000000000 --- a/contrib/python/toolz/py3/toolz/dicttoolz.py +++ /dev/null @@ -1,339 +0,0 @@ -import operator -import collections -from functools import reduce -from collections.abc import Mapping - -__all__ = ('merge', 'merge_with', 'valmap', 'keymap', 'itemmap', - 'valfilter', 'keyfilter', 'itemfilter', - 'assoc', 'dissoc', 'assoc_in', 'update_in', 'get_in') - - -def _get_factory(f, kwargs): - factory = kwargs.pop('factory', dict) - if kwargs: - raise TypeError("{}() got an unexpected keyword argument " - "'{}'".format(f.__name__, kwargs.popitem()[0])) - return factory - - -def merge(*dicts, **kwargs): - """ Merge a collection of dictionaries - - >>> merge({1: 'one'}, {2: 'two'}) - {1: 'one', 2: 'two'} - - Later dictionaries have precedence - - >>> merge({1: 2, 3: 4}, {3: 3, 4: 4}) - {1: 2, 3: 3, 4: 4} - - See Also: - merge_with - """ - if len(dicts) == 1 and not isinstance(dicts[0], Mapping): - dicts = dicts[0] - factory = _get_factory(merge, kwargs) - - rv = factory() - for d in dicts: - rv.update(d) - return rv - - -def merge_with(func, *dicts, **kwargs): - """ Merge dictionaries and apply function to combined values - - A key may occur in more than one dict, and all values mapped from the key - will be passed to the function as a list, such as func([val1, val2, ...]). - - >>> merge_with(sum, {1: 1, 2: 2}, {1: 10, 2: 20}) - {1: 11, 2: 22} - - >>> merge_with(first, {1: 1, 2: 2}, {2: 20, 3: 30}) # doctest: +SKIP - {1: 1, 2: 2, 3: 30} - - See Also: - merge - """ - if len(dicts) == 1 and not isinstance(dicts[0], Mapping): - dicts = dicts[0] - factory = _get_factory(merge_with, kwargs) - - values = collections.defaultdict(lambda: [].append) - for d in dicts: - for k, v in d.items(): - values[k](v) - - result = factory() - for k, v in values.items(): - result[k] = func(v.__self__) - return result - - -def valmap(func, d, factory=dict): - """ Apply function to values of dictionary - - >>> bills = {"Alice": [20, 15, 30], "Bob": [10, 35]} - >>> valmap(sum, bills) # doctest: +SKIP - {'Alice': 65, 'Bob': 45} - - See Also: - keymap - itemmap - """ - rv = factory() - rv.update(zip(d.keys(), map(func, d.values()))) - return rv - - -def keymap(func, d, factory=dict): - """ Apply function to keys of dictionary - - >>> bills = {"Alice": [20, 15, 30], "Bob": [10, 35]} - >>> keymap(str.lower, bills) # doctest: +SKIP - {'alice': [20, 15, 30], 'bob': [10, 35]} - - See Also: - valmap - itemmap - """ - rv = factory() - rv.update(zip(map(func, d.keys()), d.values())) - return rv - - -def itemmap(func, d, factory=dict): - """ Apply function to items of dictionary - - >>> accountids = {"Alice": 10, "Bob": 20} - >>> itemmap(reversed, accountids) # doctest: +SKIP - {10: "Alice", 20: "Bob"} - - See Also: - keymap - valmap - """ - rv = factory() - rv.update(map(func, d.items())) - return rv - - -def valfilter(predicate, d, factory=dict): - """ Filter items in dictionary by value - - >>> iseven = lambda x: x % 2 == 0 - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> valfilter(iseven, d) - {1: 2, 3: 4} - - See Also: - keyfilter - itemfilter - valmap - """ - rv = factory() - for k, v in d.items(): - if predicate(v): - rv[k] = v - return rv - - -def keyfilter(predicate, d, factory=dict): - """ Filter items in dictionary by key - - >>> iseven = lambda x: x % 2 == 0 - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> keyfilter(iseven, d) - {2: 3, 4: 5} - - See Also: - valfilter - itemfilter - keymap - """ - rv = factory() - for k, v in d.items(): - if predicate(k): - rv[k] = v - return rv - - -def itemfilter(predicate, d, factory=dict): - """ Filter items in dictionary by item - - >>> def isvalid(item): - ... k, v = item - ... return k % 2 == 0 and v < 4 - - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> itemfilter(isvalid, d) - {2: 3} - - See Also: - keyfilter - valfilter - itemmap - """ - rv = factory() - for item in d.items(): - if predicate(item): - k, v = item - rv[k] = v - return rv - - -def assoc(d, key, value, factory=dict): - """ Return a new dict with new key value pair - - New dict has d[key] set to value. Does not modify the initial dictionary. - - >>> assoc({'x': 1}, 'x', 2) - {'x': 2} - >>> assoc({'x': 1}, 'y', 3) # doctest: +SKIP - {'x': 1, 'y': 3} - """ - d2 = factory() - d2.update(d) - d2[key] = value - return d2 - - -def dissoc(d, *keys, **kwargs): - """ Return a new dict with the given key(s) removed. - - New dict has d[key] deleted for each supplied key. - Does not modify the initial dictionary. - - >>> dissoc({'x': 1, 'y': 2}, 'y') - {'x': 1} - >>> dissoc({'x': 1, 'y': 2}, 'y', 'x') - {} - >>> dissoc({'x': 1}, 'y') # Ignores missing keys - {'x': 1} - """ - factory = _get_factory(dissoc, kwargs) - d2 = factory() - - if len(keys) < len(d) * .6: - d2.update(d) - for key in keys: - if key in d2: - del d2[key] - else: - remaining = set(d) - remaining.difference_update(keys) - for k in remaining: - d2[k] = d[k] - return d2 - - -def assoc_in(d, keys, value, factory=dict): - """ Return a new dict with new, potentially nested, key value pair - - >>> purchase = {'name': 'Alice', - ... 'order': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> assoc_in(purchase, ['order', 'costs'], [0.25, 1.00]) # doctest: +SKIP - {'credit card': '5555-1234-1234-1234', - 'name': 'Alice', - 'order': {'costs': [0.25, 1.00], 'items': ['Apple', 'Orange']}} - """ - return update_in(d, keys, lambda x: value, value, factory) - - -def update_in(d, keys, func, default=None, factory=dict): - """ Update value in a (potentially) nested dictionary - - inputs: - d - dictionary on which to operate - keys - list or tuple giving the location of the value to be changed in d - func - function to operate on that value - - If keys == [k0,..,kX] and d[k0]..[kX] == v, update_in returns a copy of the - original dictionary with v replaced by func(v), but does not mutate the - original dictionary. - - If k0 is not a key in d, update_in creates nested dictionaries to the depth - specified by the keys, with the innermost value set to func(default). - - >>> inc = lambda x: x + 1 - >>> update_in({'a': 0}, ['a'], inc) - {'a': 1} - - >>> transaction = {'name': 'Alice', - ... 'purchase': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> update_in(transaction, ['purchase', 'costs'], sum) # doctest: +SKIP - {'credit card': '5555-1234-1234-1234', - 'name': 'Alice', - 'purchase': {'costs': 1.75, 'items': ['Apple', 'Orange']}} - - >>> # updating a value when k0 is not in d - >>> update_in({}, [1, 2, 3], str, default="bar") - {1: {2: {3: 'bar'}}} - >>> update_in({1: 'foo'}, [2, 3, 4], inc, 0) - {1: 'foo', 2: {3: {4: 1}}} - """ - ks = iter(keys) - k = next(ks) - - rv = inner = factory() - rv.update(d) - - for key in ks: - if k in d: - d = d[k] - dtemp = factory() - dtemp.update(d) - else: - d = dtemp = factory() - - inner[k] = inner = dtemp - k = key - - if k in d: - inner[k] = func(d[k]) - else: - inner[k] = func(default) - return rv - - -def get_in(keys, coll, default=None, no_default=False): - """ Returns coll[i0][i1]...[iX] where [i0, i1, ..., iX]==keys. - - If coll[i0][i1]...[iX] cannot be found, returns ``default``, unless - ``no_default`` is specified, then it raises KeyError or IndexError. - - ``get_in`` is a generalization of ``operator.getitem`` for nested data - structures such as dictionaries and lists. - - >>> transaction = {'name': 'Alice', - ... 'purchase': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> get_in(['purchase', 'items', 0], transaction) - 'Apple' - >>> get_in(['name'], transaction) - 'Alice' - >>> get_in(['purchase', 'total'], transaction) - >>> get_in(['purchase', 'items', 'apple'], transaction) - >>> get_in(['purchase', 'items', 10], transaction) - >>> get_in(['purchase', 'total'], transaction, 0) - 0 - >>> get_in(['y'], {}, no_default=True) - Traceback (most recent call last): - ... - KeyError: 'y' - - See Also: - itertoolz.get - operator.getitem - """ - try: - return reduce(operator.getitem, keys, coll) - except (KeyError, IndexError, TypeError): - if no_default: - raise - return default diff --git a/contrib/python/toolz/py3/toolz/functoolz.py b/contrib/python/toolz/py3/toolz/functoolz.py deleted file mode 100644 index 2c75d3a42a..0000000000 --- a/contrib/python/toolz/py3/toolz/functoolz.py +++ /dev/null @@ -1,1048 +0,0 @@ -from functools import reduce, partial -import inspect -import sys -from operator import attrgetter, not_ -from importlib import import_module -from types import MethodType - -from .utils import no_default - -PYPY = hasattr(sys, 'pypy_version_info') and sys.version_info[0] > 2 - - -__all__ = ('identity', 'apply', 'thread_first', 'thread_last', 'memoize', - 'compose', 'compose_left', 'pipe', 'complement', 'juxt', 'do', - 'curry', 'flip', 'excepts') - -PYPY = hasattr(sys, 'pypy_version_info') - - -def identity(x): - """ Identity function. Return x - - >>> identity(3) - 3 - """ - return x - - -def apply(*func_and_args, **kwargs): - """ Applies a function and returns the results - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> apply(double, 5) - 10 - - >>> tuple(map(apply, [double, inc, double], [10, 500, 8000])) - (20, 501, 16000) - """ - if not func_and_args: - raise TypeError('func argument is required') - func, args = func_and_args[0], func_and_args[1:] - return func(*args, **kwargs) - - -def thread_first(val, *forms): - """ Thread value through a sequence of functions/forms - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> thread_first(1, inc, double) - 4 - - If the function expects more than one input you can specify those inputs - in a tuple. The value is used as the first input. - - >>> def add(x, y): return x + y - >>> def pow(x, y): return x**y - >>> thread_first(1, (add, 4), (pow, 2)) # pow(add(1, 4), 2) - 25 - - So in general - thread_first(x, f, (g, y, z)) - expands to - g(f(x), y, z) - - See Also: - thread_last - """ - def evalform_front(val, form): - if callable(form): - return form(val) - if isinstance(form, tuple): - func, args = form[0], form[1:] - args = (val,) + args - return func(*args) - return reduce(evalform_front, forms, val) - - -def thread_last(val, *forms): - """ Thread value through a sequence of functions/forms - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> thread_last(1, inc, double) - 4 - - If the function expects more than one input you can specify those inputs - in a tuple. The value is used as the last input. - - >>> def add(x, y): return x + y - >>> def pow(x, y): return x**y - >>> thread_last(1, (add, 4), (pow, 2)) # pow(2, add(4, 1)) - 32 - - So in general - thread_last(x, f, (g, y, z)) - expands to - g(y, z, f(x)) - - >>> def iseven(x): - ... return x % 2 == 0 - >>> list(thread_last([1, 2, 3], (map, inc), (filter, iseven))) - [2, 4] - - See Also: - thread_first - """ - def evalform_back(val, form): - if callable(form): - return form(val) - if isinstance(form, tuple): - func, args = form[0], form[1:] - args = args + (val,) - return func(*args) - return reduce(evalform_back, forms, val) - - -def instanceproperty(fget=None, fset=None, fdel=None, doc=None, classval=None): - """ Like @property, but returns ``classval`` when used as a class attribute - - >>> class MyClass(object): - ... '''The class docstring''' - ... @instanceproperty(classval=__doc__) - ... def __doc__(self): - ... return 'An object docstring' - ... @instanceproperty - ... def val(self): - ... return 42 - ... - >>> MyClass.__doc__ - 'The class docstring' - >>> MyClass.val is None - True - >>> obj = MyClass() - >>> obj.__doc__ - 'An object docstring' - >>> obj.val - 42 - """ - if fget is None: - return partial(instanceproperty, fset=fset, fdel=fdel, doc=doc, - classval=classval) - return InstanceProperty(fget=fget, fset=fset, fdel=fdel, doc=doc, - classval=classval) - - -class InstanceProperty(property): - """ Like @property, but returns ``classval`` when used as a class attribute - - Should not be used directly. Use ``instanceproperty`` instead. - """ - def __init__(self, fget=None, fset=None, fdel=None, doc=None, - classval=None): - self.classval = classval - property.__init__(self, fget=fget, fset=fset, fdel=fdel, doc=doc) - - def __get__(self, obj, type=None): - if obj is None: - return self.classval - return property.__get__(self, obj, type) - - def __reduce__(self): - state = (self.fget, self.fset, self.fdel, self.__doc__, self.classval) - return InstanceProperty, state - - -class curry(object): - """ Curry a callable function - - Enables partial application of arguments through calling a function with an - incomplete set of arguments. - - >>> def mul(x, y): - ... return x * y - >>> mul = curry(mul) - - >>> double = mul(2) - >>> double(10) - 20 - - Also supports keyword arguments - - >>> @curry # Can use curry as a decorator - ... def f(x, y, a=10): - ... return a * (x + y) - - >>> add = f(a=1) - >>> add(2, 3) - 5 - - See Also: - toolz.curried - namespace of curried functions - https://toolz.readthedocs.io/en/latest/curry.html - """ - def __init__(self, *args, **kwargs): - if not args: - raise TypeError('__init__() takes at least 2 arguments (1 given)') - func, args = args[0], args[1:] - if not callable(func): - raise TypeError("Input must be callable") - - # curry- or functools.partial-like object? Unpack and merge arguments - if ( - hasattr(func, 'func') - and hasattr(func, 'args') - and hasattr(func, 'keywords') - and isinstance(func.args, tuple) - ): - _kwargs = {} - if func.keywords: - _kwargs.update(func.keywords) - _kwargs.update(kwargs) - kwargs = _kwargs - args = func.args + args - func = func.func - - if kwargs: - self._partial = partial(func, *args, **kwargs) - else: - self._partial = partial(func, *args) - - self.__doc__ = getattr(func, '__doc__', None) - self.__name__ = getattr(func, '__name__', '<curry>') - self.__module__ = getattr(func, '__module__', None) - self.__qualname__ = getattr(func, '__qualname__', None) - self._sigspec = None - self._has_unknown_args = None - - @instanceproperty - def func(self): - return self._partial.func - - @instanceproperty - def __signature__(self): - sig = inspect.signature(self.func) - args = self.args or () - keywords = self.keywords or {} - if is_partial_args(self.func, args, keywords, sig) is False: - raise TypeError('curry object has incorrect arguments') - - params = list(sig.parameters.values()) - skip = 0 - for param in params[:len(args)]: - if param.kind == param.VAR_POSITIONAL: - break - skip += 1 - - kwonly = False - newparams = [] - for param in params[skip:]: - kind = param.kind - default = param.default - if kind == param.VAR_KEYWORD: - pass - elif kind == param.VAR_POSITIONAL: - if kwonly: - continue - elif param.name in keywords: - default = keywords[param.name] - kind = param.KEYWORD_ONLY - kwonly = True - else: - if kwonly: - kind = param.KEYWORD_ONLY - if default is param.empty: - default = no_default - newparams.append(param.replace(default=default, kind=kind)) - - return sig.replace(parameters=newparams) - - @instanceproperty - def args(self): - return self._partial.args - - @instanceproperty - def keywords(self): - return self._partial.keywords - - @instanceproperty - def func_name(self): - return self.__name__ - - def __str__(self): - return str(self.func) - - def __repr__(self): - return repr(self.func) - - def __hash__(self): - return hash((self.func, self.args, - frozenset(self.keywords.items()) if self.keywords - else None)) - - def __eq__(self, other): - return (isinstance(other, curry) and self.func == other.func and - self.args == other.args and self.keywords == other.keywords) - - def __ne__(self, other): - return not self.__eq__(other) - - def __call__(self, *args, **kwargs): - try: - return self._partial(*args, **kwargs) - except TypeError as exc: - if self._should_curry(args, kwargs, exc): - return self.bind(*args, **kwargs) - raise - - def _should_curry(self, args, kwargs, exc=None): - func = self.func - args = self.args + args - if self.keywords: - kwargs = dict(self.keywords, **kwargs) - if self._sigspec is None: - sigspec = self._sigspec = _sigs.signature_or_spec(func) - self._has_unknown_args = has_varargs(func, sigspec) is not False - else: - sigspec = self._sigspec - - if is_partial_args(func, args, kwargs, sigspec) is False: - # Nothing can make the call valid - return False - elif self._has_unknown_args: - # The call may be valid and raised a TypeError, but we curry - # anyway because the function may have `*args`. This is useful - # for decorators with signature `func(*args, **kwargs)`. - return True - elif not is_valid_args(func, args, kwargs, sigspec): - # Adding more arguments may make the call valid - return True - else: - # There was a genuine TypeError - return False - - def bind(self, *args, **kwargs): - return type(self)(self, *args, **kwargs) - - def call(self, *args, **kwargs): - return self._partial(*args, **kwargs) - - def __get__(self, instance, owner): - if instance is None: - return self - return curry(self, instance) - - def __reduce__(self): - func = self.func - modname = getattr(func, '__module__', None) - qualname = getattr(func, '__qualname__', None) - if qualname is None: # pragma: no cover - qualname = getattr(func, '__name__', None) - is_decorated = None - if modname and qualname: - attrs = [] - obj = import_module(modname) - for attr in qualname.split('.'): - if isinstance(obj, curry): - attrs.append('func') - obj = obj.func - obj = getattr(obj, attr, None) - if obj is None: - break - attrs.append(attr) - if isinstance(obj, curry) and obj.func is func: - is_decorated = obj is self - qualname = '.'.join(attrs) - func = '%s:%s' % (modname, qualname) - - # functools.partial objects can't be pickled - userdict = tuple((k, v) for k, v in self.__dict__.items() - if k not in ('_partial', '_sigspec')) - state = (type(self), func, self.args, self.keywords, userdict, - is_decorated) - return _restore_curry, state - - -def _restore_curry(cls, func, args, kwargs, userdict, is_decorated): - if isinstance(func, str): - modname, qualname = func.rsplit(':', 1) - obj = import_module(modname) - for attr in qualname.split('.'): - obj = getattr(obj, attr) - if is_decorated: - return obj - func = obj.func - obj = cls(func, *args, **(kwargs or {})) - obj.__dict__.update(userdict) - return obj - - -@curry -def memoize(func, cache=None, key=None): - """ Cache a function's result for speedy future evaluation - - Considerations: - Trades memory for speed. - Only use on pure functions. - - >>> def add(x, y): return x + y - >>> add = memoize(add) - - Or use as a decorator - - >>> @memoize - ... def add(x, y): - ... return x + y - - Use the ``cache`` keyword to provide a dict-like object as an initial cache - - >>> @memoize(cache={(1, 2): 3}) - ... def add(x, y): - ... return x + y - - Note that the above works as a decorator because ``memoize`` is curried. - - It is also possible to provide a ``key(args, kwargs)`` function that - calculates keys used for the cache, which receives an ``args`` tuple and - ``kwargs`` dict as input, and must return a hashable value. However, - the default key function should be sufficient most of the time. - - >>> # Use key function that ignores extraneous keyword arguments - >>> @memoize(key=lambda args, kwargs: args) - ... def add(x, y, verbose=False): - ... if verbose: - ... print('Calculating %s + %s' % (x, y)) - ... return x + y - """ - if cache is None: - cache = {} - - try: - may_have_kwargs = has_keywords(func) is not False - # Is unary function (single arg, no variadic argument or keywords)? - is_unary = is_arity(1, func) - except TypeError: # pragma: no cover - may_have_kwargs = True - is_unary = False - - if key is None: - if is_unary: - def key(args, kwargs): - return args[0] - elif may_have_kwargs: - def key(args, kwargs): - return ( - args or None, - frozenset(kwargs.items()) if kwargs else None, - ) - else: - def key(args, kwargs): - return args - - def memof(*args, **kwargs): - k = key(args, kwargs) - try: - return cache[k] - except TypeError: - raise TypeError("Arguments to memoized function must be hashable") - except KeyError: - cache[k] = result = func(*args, **kwargs) - return result - - try: - memof.__name__ = func.__name__ - except AttributeError: - pass - memof.__doc__ = func.__doc__ - memof.__wrapped__ = func - return memof - - -class Compose(object): - """ A composition of functions - - See Also: - compose - """ - __slots__ = 'first', 'funcs' - - def __init__(self, funcs): - funcs = tuple(reversed(funcs)) - self.first = funcs[0] - self.funcs = funcs[1:] - - def __call__(self, *args, **kwargs): - ret = self.first(*args, **kwargs) - for f in self.funcs: - ret = f(ret) - return ret - - def __getstate__(self): - return self.first, self.funcs - - def __setstate__(self, state): - self.first, self.funcs = state - - @instanceproperty(classval=__doc__) - def __doc__(self): - def composed_doc(*fs): - """Generate a docstring for the composition of fs. - """ - if not fs: - # Argument name for the docstring. - return '*args, **kwargs' - - return '{f}({g})'.format(f=fs[0].__name__, g=composed_doc(*fs[1:])) - - try: - return ( - 'lambda *args, **kwargs: ' + - composed_doc(*reversed((self.first,) + self.funcs)) - ) - except AttributeError: - # One of our callables does not have a `__name__`, whatever. - return 'A composition of functions' - - @property - def __name__(self): - try: - return '_of_'.join( - (f.__name__ for f in reversed((self.first,) + self.funcs)) - ) - except AttributeError: - return type(self).__name__ - - def __repr__(self): - return '{.__class__.__name__}{!r}'.format( - self, tuple(reversed((self.first, ) + self.funcs))) - - def __eq__(self, other): - if isinstance(other, Compose): - return other.first == self.first and other.funcs == self.funcs - return NotImplemented - - def __ne__(self, other): - equality = self.__eq__(other) - return NotImplemented if equality is NotImplemented else not equality - - def __hash__(self): - return hash(self.first) ^ hash(self.funcs) - - # Mimic the descriptor behavior of python functions. - # i.e. let Compose be called as a method when bound to a class. - # adapted from - # docs.python.org/3/howto/descriptor.html#functions-and-methods - def __get__(self, obj, objtype=None): - return self if obj is None else MethodType(self, obj) - - # introspection with Signature is only possible from py3.3+ - @instanceproperty - def __signature__(self): - base = inspect.signature(self.first) - last = inspect.signature(self.funcs[-1]) - return base.replace(return_annotation=last.return_annotation) - - __wrapped__ = instanceproperty(attrgetter('first')) - - -def compose(*funcs): - """ Compose functions to operate in series. - - Returns a function that applies other functions in sequence. - - Functions are applied from right to left so that - ``compose(f, g, h)(x, y)`` is the same as ``f(g(h(x, y)))``. - - If no arguments are provided, the identity function (f(x) = x) is returned. - - >>> inc = lambda i: i + 1 - >>> compose(str, inc)(3) - '4' - - See Also: - compose_left - pipe - """ - if not funcs: - return identity - if len(funcs) == 1: - return funcs[0] - else: - return Compose(funcs) - - -def compose_left(*funcs): - """ Compose functions to operate in series. - - Returns a function that applies other functions in sequence. - - Functions are applied from left to right so that - ``compose_left(f, g, h)(x, y)`` is the same as ``h(g(f(x, y)))``. - - If no arguments are provided, the identity function (f(x) = x) is returned. - - >>> inc = lambda i: i + 1 - >>> compose_left(inc, str)(3) - '4' - - See Also: - compose - pipe - """ - return compose(*reversed(funcs)) - - -def pipe(data, *funcs): - """ Pipe a value through a sequence of functions - - I.e. ``pipe(data, f, g, h)`` is equivalent to ``h(g(f(data)))`` - - We think of the value as progressing through a pipe of several - transformations, much like pipes in UNIX - - ``$ cat data | f | g | h`` - - >>> double = lambda i: 2 * i - >>> pipe(3, double, str) - '6' - - See Also: - compose - compose_left - thread_first - thread_last - """ - for func in funcs: - data = func(data) - return data - - -def complement(func): - """ Convert a predicate function to its logical complement. - - In other words, return a function that, for inputs that normally - yield True, yields False, and vice-versa. - - >>> def iseven(n): return n % 2 == 0 - >>> isodd = complement(iseven) - >>> iseven(2) - True - >>> isodd(2) - False - """ - return compose(not_, func) - - -class juxt(object): - """ Creates a function that calls several functions with the same arguments - - Takes several functions and returns a function that applies its arguments - to each of those functions then returns a tuple of the results. - - Name comes from juxtaposition: the fact of two things being seen or placed - close together with contrasting effect. - - >>> inc = lambda x: x + 1 - >>> double = lambda x: x * 2 - >>> juxt(inc, double)(10) - (11, 20) - >>> juxt([inc, double])(10) - (11, 20) - """ - __slots__ = ['funcs'] - - def __init__(self, *funcs): - if len(funcs) == 1 and not callable(funcs[0]): - funcs = funcs[0] - self.funcs = tuple(funcs) - - def __call__(self, *args, **kwargs): - return tuple(func(*args, **kwargs) for func in self.funcs) - - def __getstate__(self): - return self.funcs - - def __setstate__(self, state): - self.funcs = state - - -def do(func, x): - """ Runs ``func`` on ``x``, returns ``x`` - - Because the results of ``func`` are not returned, only the side - effects of ``func`` are relevant. - - Logging functions can be made by composing ``do`` with a storage function - like ``list.append`` or ``file.write`` - - >>> from toolz import compose - >>> from toolz.curried import do - - >>> log = [] - >>> inc = lambda x: x + 1 - >>> inc = compose(inc, do(log.append)) - >>> inc(1) - 2 - >>> inc(11) - 12 - >>> log - [1, 11] - """ - func(x) - return x - - -@curry -def flip(func, a, b): - """ Call the function call with the arguments flipped - - This function is curried. - - >>> def div(a, b): - ... return a // b - ... - >>> flip(div, 2, 6) - 3 - >>> div_by_two = flip(div, 2) - >>> div_by_two(4) - 2 - - This is particularly useful for built in functions and functions defined - in C extensions that accept positional only arguments. For example: - isinstance, issubclass. - - >>> data = [1, 'a', 'b', 2, 1.5, object(), 3] - >>> only_ints = list(filter(flip(isinstance, int), data)) - >>> only_ints - [1, 2, 3] - """ - return func(b, a) - - -def return_none(exc): - """ Returns None. - """ - return None - - -class excepts(object): - """A wrapper around a function to catch exceptions and - dispatch to a handler. - - This is like a functional try/except block, in the same way that - ifexprs are functional if/else blocks. - - Examples - -------- - >>> excepting = excepts( - ... ValueError, - ... lambda a: [1, 2].index(a), - ... lambda _: -1, - ... ) - >>> excepting(1) - 0 - >>> excepting(3) - -1 - - Multiple exceptions and default except clause. - >>> excepting = excepts((IndexError, KeyError), lambda a: a[0]) - >>> excepting([]) - >>> excepting([1]) - 1 - >>> excepting({}) - >>> excepting({0: 1}) - 1 - """ - def __init__(self, exc, func, handler=return_none): - self.exc = exc - self.func = func - self.handler = handler - - def __call__(self, *args, **kwargs): - try: - return self.func(*args, **kwargs) - except self.exc as e: - return self.handler(e) - - @instanceproperty(classval=__doc__) - def __doc__(self): - from textwrap import dedent - - exc = self.exc - try: - if isinstance(exc, tuple): - exc_name = '(%s)' % ', '.join( - map(attrgetter('__name__'), exc), - ) - else: - exc_name = exc.__name__ - - return dedent( - """\ - A wrapper around {inst.func.__name__!r} that will except: - {exc} - and handle any exceptions with {inst.handler.__name__!r}. - - Docs for {inst.func.__name__!r}: - {inst.func.__doc__} - - Docs for {inst.handler.__name__!r}: - {inst.handler.__doc__} - """ - ).format( - inst=self, - exc=exc_name, - ) - except AttributeError: - return type(self).__doc__ - - @property - def __name__(self): - exc = self.exc - try: - if isinstance(exc, tuple): - exc_name = '_or_'.join(map(attrgetter('__name__'), exc)) - else: - exc_name = exc.__name__ - return '%s_excepting_%s' % (self.func.__name__, exc_name) - except AttributeError: - return 'excepting' - - -def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - if sigspec is None: - try: - sigspec = inspect.signature(func) - except (ValueError, TypeError) as e: - sigspec = e - if isinstance(sigspec, ValueError): - return None, builtin_func(*builtin_args) - elif not isinstance(sigspec, inspect.Signature): - if ( - func in _sigs.signatures - and (( - hasattr(func, '__signature__') - and hasattr(func.__signature__, '__get__') - )) - ): - val = builtin_func(*builtin_args) - return None, val - return None, False - return sigspec, None - - -if PYPY: # pragma: no cover - _check_sigspec_orig = _check_sigspec - - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - # PyPy may lie, so use our registry for builtins instead - if func in _sigs.signatures: - val = builtin_func(*builtin_args) - return None, val - return _check_sigspec_orig(sigspec, func, builtin_func, *builtin_args) - - -_check_sigspec.__doc__ = """ \ -Private function to aid in introspection compatibly across Python versions. - -If a callable doesn't have a signature (Python 3) or an argspec (Python 2), -the signature registry in toolz._signatures is used. -""" - - -def num_required_args(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._num_required_args, - func) - if sigspec is None: - return rv - return sum(1 for p in sigspec.parameters.values() - if p.default is p.empty - and p.kind in (p.POSITIONAL_OR_KEYWORD, p.POSITIONAL_ONLY)) - - -def has_varargs(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_varargs, func) - if sigspec is None: - return rv - return any(p.kind == p.VAR_POSITIONAL - for p in sigspec.parameters.values()) - - -def has_keywords(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_keywords, func) - if sigspec is None: - return rv - return any(p.default is not p.empty - or p.kind in (p.KEYWORD_ONLY, p.VAR_KEYWORD) - for p in sigspec.parameters.values()) - - -def is_valid_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_valid_args, - func, args, kwargs) - if sigspec is None: - return rv - try: - sigspec.bind(*args, **kwargs) - except TypeError: - return False - return True - - -def is_partial_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_partial_args, - func, args, kwargs) - if sigspec is None: - return rv - try: - sigspec.bind_partial(*args, **kwargs) - except TypeError: - return False - return True - - -def is_arity(n, func, sigspec=None): - """ Does a function have only n positional arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x): - ... return x - >>> is_arity(1, f) - True - >>> def g(x, y=1): - ... return x + y - >>> is_arity(1, g) - False - """ - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_arity, n, func) - if sigspec is None: - return rv - num = num_required_args(func, sigspec) - if num is not None: - num = num == n - if not num: - return False - varargs = has_varargs(func, sigspec) - if varargs: - return False - keywords = has_keywords(func, sigspec) - if keywords: - return False - if num is None or varargs is None or keywords is None: # pragma: no cover - return None - return True - - -num_required_args.__doc__ = """ \ -Number of required positional arguments - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x, y, z=3): - ... return x + y + z - >>> num_required_args(f) - 2 - >>> def g(*args, **kwargs): - ... pass - >>> num_required_args(g) - 0 - """ - -has_varargs.__doc__ = """ \ -Does a function have variadic positional arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(*args): - ... return args - >>> has_varargs(f) - True - >>> def g(**kwargs): - ... return kwargs - >>> has_varargs(g) - False - """ - -has_keywords.__doc__ = """ \ -Does a function have keyword arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x, y=0): - ... return x + y - - >>> has_keywords(f) - True - """ - -is_valid_args.__doc__ = """ \ -Is ``func(*args, **kwargs)`` a valid function call? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def add(x, y): - ... return x + y - - >>> is_valid_args(add, (1,), {}) - False - >>> is_valid_args(add, (1, 2), {}) - True - >>> is_valid_args(map, (), {}) - False - - **Implementation notes** - Python 2 relies on ``inspect.getargspec``, which only works for - user-defined functions. Python 3 uses ``inspect.signature``, which - works for many more types of callables. - - Many builtins in the standard library are also supported. - """ - -is_partial_args.__doc__ = """ \ -Can partial(func, *args, **kwargs)(*args2, **kwargs2) be a valid call? - - Returns True *only* if the call is valid or if it is possible for the - call to become valid by adding more positional or keyword arguments. - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def add(x, y): - ... return x + y - - >>> is_partial_args(add, (1,), {}) - True - >>> is_partial_args(add, (1, 2), {}) - True - >>> is_partial_args(add, (1, 2, 3), {}) - False - >>> is_partial_args(map, (), {}) - True - - **Implementation notes** - Python 2 relies on ``inspect.getargspec``, which only works for - user-defined functions. Python 3 uses ``inspect.signature``, which - works for many more types of callables. - - Many builtins in the standard library are also supported. - """ - -from . import _signatures as _sigs diff --git a/contrib/python/toolz/py3/toolz/itertoolz.py b/contrib/python/toolz/py3/toolz/itertoolz.py deleted file mode 100644 index 5049e5eb4b..0000000000 --- a/contrib/python/toolz/py3/toolz/itertoolz.py +++ /dev/null @@ -1,1057 +0,0 @@ -import itertools -import heapq -import collections -import operator -from functools import partial -from itertools import filterfalse, zip_longest -from collections.abc import Sequence -from toolz.utils import no_default - - -__all__ = ('remove', 'accumulate', 'groupby', 'merge_sorted', 'interleave', - 'unique', 'isiterable', 'isdistinct', 'take', 'drop', 'take_nth', - 'first', 'second', 'nth', 'last', 'get', 'concat', 'concatv', - 'mapcat', 'cons', 'interpose', 'frequencies', 'reduceby', 'iterate', - 'sliding_window', 'partition', 'partition_all', 'count', 'pluck', - 'join', 'tail', 'diff', 'topk', 'peek', 'peekn', 'random_sample') - - -def remove(predicate, seq): - """ Return those items of sequence for which predicate(item) is False - - >>> def iseven(x): - ... return x % 2 == 0 - >>> list(remove(iseven, [1, 2, 3, 4])) - [1, 3] - """ - return filterfalse(predicate, seq) - - -def accumulate(binop, seq, initial=no_default): - """ Repeatedly apply binary function to a sequence, accumulating results - - >>> from operator import add, mul - >>> list(accumulate(add, [1, 2, 3, 4, 5])) - [1, 3, 6, 10, 15] - >>> list(accumulate(mul, [1, 2, 3, 4, 5])) - [1, 2, 6, 24, 120] - - Accumulate is similar to ``reduce`` and is good for making functions like - cumulative sum: - - >>> from functools import partial, reduce - >>> sum = partial(reduce, add) - >>> cumsum = partial(accumulate, add) - - Accumulate also takes an optional argument that will be used as the first - value. This is similar to reduce. - - >>> list(accumulate(add, [1, 2, 3], -1)) - [-1, 0, 2, 5] - >>> list(accumulate(add, [], 1)) - [1] - - See Also: - itertools.accumulate : In standard itertools for Python 3.2+ - """ - seq = iter(seq) - if initial == no_default: - try: - result = next(seq) - except StopIteration: - return - else: - result = initial - yield result - for elem in seq: - result = binop(result, elem) - yield result - - -def groupby(key, seq): - """ Group a collection by a key function - - >>> names = ['Alice', 'Bob', 'Charlie', 'Dan', 'Edith', 'Frank'] - >>> groupby(len, names) # doctest: +SKIP - {3: ['Bob', 'Dan'], 5: ['Alice', 'Edith', 'Frank'], 7: ['Charlie']} - - >>> iseven = lambda x: x % 2 == 0 - >>> groupby(iseven, [1, 2, 3, 4, 5, 6, 7, 8]) # doctest: +SKIP - {False: [1, 3, 5, 7], True: [2, 4, 6, 8]} - - Non-callable keys imply grouping on a member. - - >>> groupby('gender', [{'name': 'Alice', 'gender': 'F'}, - ... {'name': 'Bob', 'gender': 'M'}, - ... {'name': 'Charlie', 'gender': 'M'}]) # doctest:+SKIP - {'F': [{'gender': 'F', 'name': 'Alice'}], - 'M': [{'gender': 'M', 'name': 'Bob'}, - {'gender': 'M', 'name': 'Charlie'}]} - - Not to be confused with ``itertools.groupby`` - - See Also: - countby - """ - if not callable(key): - key = getter(key) - d = collections.defaultdict(lambda: [].append) - for item in seq: - d[key(item)](item) - rv = {} - for k, v in d.items(): - rv[k] = v.__self__ - return rv - - -def merge_sorted(*seqs, **kwargs): - """ Merge and sort a collection of sorted collections - - This works lazily and only keeps one value from each iterable in memory. - - >>> list(merge_sorted([1, 3, 5], [2, 4, 6])) - [1, 2, 3, 4, 5, 6] - - >>> ''.join(merge_sorted('abc', 'abc', 'abc')) - 'aaabbbccc' - - The "key" function used to sort the input may be passed as a keyword. - - >>> list(merge_sorted([2, 3], [1, 3], key=lambda x: x // 3)) - [2, 1, 3, 3] - """ - if len(seqs) == 0: - return iter([]) - elif len(seqs) == 1: - return iter(seqs[0]) - - key = kwargs.get('key', None) - if key is None: - return _merge_sorted_binary(seqs) - else: - return _merge_sorted_binary_key(seqs, key) - - -def _merge_sorted_binary(seqs): - mid = len(seqs) // 2 - L1 = seqs[:mid] - if len(L1) == 1: - seq1 = iter(L1[0]) - else: - seq1 = _merge_sorted_binary(L1) - L2 = seqs[mid:] - if len(L2) == 1: - seq2 = iter(L2[0]) - else: - seq2 = _merge_sorted_binary(L2) - - try: - val2 = next(seq2) - except StopIteration: - for val1 in seq1: - yield val1 - return - - for val1 in seq1: - if val2 < val1: - yield val2 - for val2 in seq2: - if val2 < val1: - yield val2 - else: - yield val1 - break - else: - break - else: - yield val1 - else: - yield val2 - for val2 in seq2: - yield val2 - return - yield val1 - for val1 in seq1: - yield val1 - - -def _merge_sorted_binary_key(seqs, key): - mid = len(seqs) // 2 - L1 = seqs[:mid] - if len(L1) == 1: - seq1 = iter(L1[0]) - else: - seq1 = _merge_sorted_binary_key(L1, key) - L2 = seqs[mid:] - if len(L2) == 1: - seq2 = iter(L2[0]) - else: - seq2 = _merge_sorted_binary_key(L2, key) - - try: - val2 = next(seq2) - except StopIteration: - for val1 in seq1: - yield val1 - return - key2 = key(val2) - - for val1 in seq1: - key1 = key(val1) - if key2 < key1: - yield val2 - for val2 in seq2: - key2 = key(val2) - if key2 < key1: - yield val2 - else: - yield val1 - break - else: - break - else: - yield val1 - else: - yield val2 - for val2 in seq2: - yield val2 - return - yield val1 - for val1 in seq1: - yield val1 - - -def interleave(seqs): - """ Interleave a sequence of sequences - - >>> list(interleave([[1, 2], [3, 4]])) - [1, 3, 2, 4] - - >>> ''.join(interleave(('ABC', 'XY'))) - 'AXBYC' - - Both the individual sequences and the sequence of sequences may be infinite - - Returns a lazy iterator - """ - iters = itertools.cycle(map(iter, seqs)) - while True: - try: - for itr in iters: - yield next(itr) - return - except StopIteration: - predicate = partial(operator.is_not, itr) - iters = itertools.cycle(itertools.takewhile(predicate, iters)) - - -def unique(seq, key=None): - """ Return only unique elements of a sequence - - >>> tuple(unique((1, 2, 3))) - (1, 2, 3) - >>> tuple(unique((1, 2, 1, 3))) - (1, 2, 3) - - Uniqueness can be defined by key keyword - - >>> tuple(unique(['cat', 'mouse', 'dog', 'hen'], key=len)) - ('cat', 'mouse') - """ - seen = set() - seen_add = seen.add - if key is None: - for item in seq: - if item not in seen: - seen_add(item) - yield item - else: # calculate key - for item in seq: - val = key(item) - if val not in seen: - seen_add(val) - yield item - - -def isiterable(x): - """ Is x iterable? - - >>> isiterable([1, 2, 3]) - True - >>> isiterable('abc') - True - >>> isiterable(5) - False - """ - try: - iter(x) - return True - except TypeError: - return False - - -def isdistinct(seq): - """ All values in sequence are distinct - - >>> isdistinct([1, 2, 3]) - True - >>> isdistinct([1, 2, 1]) - False - - >>> isdistinct("Hello") - False - >>> isdistinct("World") - True - """ - if iter(seq) is seq: - seen = set() - seen_add = seen.add - for item in seq: - if item in seen: - return False - seen_add(item) - return True - else: - return len(seq) == len(set(seq)) - - -def take(n, seq): - """ The first n elements of a sequence - - >>> list(take(2, [10, 20, 30, 40, 50])) - [10, 20] - - See Also: - drop - tail - """ - return itertools.islice(seq, n) - - -def tail(n, seq): - """ The last n elements of a sequence - - >>> tail(2, [10, 20, 30, 40, 50]) - [40, 50] - - See Also: - drop - take - """ - try: - return seq[-n:] - except (TypeError, KeyError): - return tuple(collections.deque(seq, n)) - - -def drop(n, seq): - """ The sequence following the first n elements - - >>> list(drop(2, [10, 20, 30, 40, 50])) - [30, 40, 50] - - See Also: - take - tail - """ - return itertools.islice(seq, n, None) - - -def take_nth(n, seq): - """ Every nth item in seq - - >>> list(take_nth(2, [10, 20, 30, 40, 50])) - [10, 30, 50] - """ - return itertools.islice(seq, 0, None, n) - - -def first(seq): - """ The first element in a sequence - - >>> first('ABC') - 'A' - """ - return next(iter(seq)) - - -def second(seq): - """ The second element in a sequence - - >>> second('ABC') - 'B' - """ - seq = iter(seq) - next(seq) - return next(seq) - - -def nth(n, seq): - """ The nth element in a sequence - - >>> nth(1, 'ABC') - 'B' - """ - if isinstance(seq, (tuple, list, Sequence)): - return seq[n] - else: - return next(itertools.islice(seq, n, None)) - - -def last(seq): - """ The last element in a sequence - - >>> last('ABC') - 'C' - """ - return tail(1, seq)[0] - - -rest = partial(drop, 1) - - -def _get(ind, seq, default): - try: - return seq[ind] - except (KeyError, IndexError): - return default - - -def get(ind, seq, default=no_default): - """ Get element in a sequence or dict - - Provides standard indexing - - >>> get(1, 'ABC') # Same as 'ABC'[1] - 'B' - - Pass a list to get multiple values - - >>> get([1, 2], 'ABC') # ('ABC'[1], 'ABC'[2]) - ('B', 'C') - - Works on any value that supports indexing/getitem - For example here we see that it works with dictionaries - - >>> phonebook = {'Alice': '555-1234', - ... 'Bob': '555-5678', - ... 'Charlie':'555-9999'} - >>> get('Alice', phonebook) - '555-1234' - - >>> get(['Alice', 'Bob'], phonebook) - ('555-1234', '555-5678') - - Provide a default for missing values - - >>> get(['Alice', 'Dennis'], phonebook, None) - ('555-1234', None) - - See Also: - pluck - """ - try: - return seq[ind] - except TypeError: # `ind` may be a list - if isinstance(ind, list): - if default == no_default: - if len(ind) > 1: - return operator.itemgetter(*ind)(seq) - elif ind: - return seq[ind[0]], - else: - return () - else: - return tuple(_get(i, seq, default) for i in ind) - elif default != no_default: - return default - else: - raise - except (KeyError, IndexError): # we know `ind` is not a list - if default == no_default: - raise - else: - return default - - -def concat(seqs): - """ Concatenate zero or more iterables, any of which may be infinite. - - An infinite sequence will prevent the rest of the arguments from - being included. - - We use chain.from_iterable rather than ``chain(*seqs)`` so that seqs - can be a generator. - - >>> list(concat([[], [1], [2, 3]])) - [1, 2, 3] - - See also: - itertools.chain.from_iterable equivalent - """ - return itertools.chain.from_iterable(seqs) - - -def concatv(*seqs): - """ Variadic version of concat - - >>> list(concatv([], ["a"], ["b", "c"])) - ['a', 'b', 'c'] - - See also: - itertools.chain - """ - return concat(seqs) - - -def mapcat(func, seqs): - """ Apply func to each sequence in seqs, concatenating results. - - >>> list(mapcat(lambda s: [c.upper() for c in s], - ... [["a", "b"], ["c", "d", "e"]])) - ['A', 'B', 'C', 'D', 'E'] - """ - return concat(map(func, seqs)) - - -def cons(el, seq): - """ Add el to beginning of (possibly infinite) sequence seq. - - >>> list(cons(1, [2, 3])) - [1, 2, 3] - """ - return itertools.chain([el], seq) - - -def interpose(el, seq): - """ Introduce element between each pair of elements in seq - - >>> list(interpose("a", [1, 2, 3])) - [1, 'a', 2, 'a', 3] - """ - inposed = concat(zip(itertools.repeat(el), seq)) - next(inposed) - return inposed - - -def frequencies(seq): - """ Find number of occurrences of each value in seq - - >>> frequencies(['cat', 'cat', 'ox', 'pig', 'pig', 'cat']) #doctest: +SKIP - {'cat': 3, 'ox': 1, 'pig': 2} - - See Also: - countby - groupby - """ - d = collections.defaultdict(int) - for item in seq: - d[item] += 1 - return dict(d) - - -def reduceby(key, binop, seq, init=no_default): - """ Perform a simultaneous groupby and reduction - - The computation: - - >>> result = reduceby(key, binop, seq, init) # doctest: +SKIP - - is equivalent to the following: - - >>> def reduction(group): # doctest: +SKIP - ... return reduce(binop, group, init) # doctest: +SKIP - - >>> groups = groupby(key, seq) # doctest: +SKIP - >>> result = valmap(reduction, groups) # doctest: +SKIP - - But the former does not build the intermediate groups, allowing it to - operate in much less space. This makes it suitable for larger datasets - that do not fit comfortably in memory - - The ``init`` keyword argument is the default initialization of the - reduction. This can be either a constant value like ``0`` or a callable - like ``lambda : 0`` as might be used in ``defaultdict``. - - Simple Examples - --------------- - - >>> from operator import add, mul - >>> iseven = lambda x: x % 2 == 0 - - >>> data = [1, 2, 3, 4, 5] - - >>> reduceby(iseven, add, data) # doctest: +SKIP - {False: 9, True: 6} - - >>> reduceby(iseven, mul, data) # doctest: +SKIP - {False: 15, True: 8} - - Complex Example - --------------- - - >>> projects = [{'name': 'build roads', 'state': 'CA', 'cost': 1000000}, - ... {'name': 'fight crime', 'state': 'IL', 'cost': 100000}, - ... {'name': 'help farmers', 'state': 'IL', 'cost': 2000000}, - ... {'name': 'help farmers', 'state': 'CA', 'cost': 200000}] - - >>> reduceby('state', # doctest: +SKIP - ... lambda acc, x: acc + x['cost'], - ... projects, 0) - {'CA': 1200000, 'IL': 2100000} - - Example Using ``init`` - ---------------------- - - >>> def set_add(s, i): - ... s.add(i) - ... return s - - >>> reduceby(iseven, set_add, [1, 2, 3, 4, 1, 2, 3], set) # doctest: +SKIP - {True: set([2, 4]), - False: set([1, 3])} - """ - is_no_default = init == no_default - if not is_no_default and not callable(init): - _init = init - init = lambda: _init - if not callable(key): - key = getter(key) - d = {} - for item in seq: - k = key(item) - if k not in d: - if is_no_default: - d[k] = item - continue - else: - d[k] = init() - d[k] = binop(d[k], item) - return d - - -def iterate(func, x): - """ Repeatedly apply a function func onto an original input - - Yields x, then func(x), then func(func(x)), then func(func(func(x))), etc.. - - >>> def inc(x): return x + 1 - >>> counter = iterate(inc, 0) - >>> next(counter) - 0 - >>> next(counter) - 1 - >>> next(counter) - 2 - - >>> double = lambda x: x * 2 - >>> powers_of_two = iterate(double, 1) - >>> next(powers_of_two) - 1 - >>> next(powers_of_two) - 2 - >>> next(powers_of_two) - 4 - >>> next(powers_of_two) - 8 - """ - while True: - yield x - x = func(x) - - -def sliding_window(n, seq): - """ A sequence of overlapping subsequences - - >>> list(sliding_window(2, [1, 2, 3, 4])) - [(1, 2), (2, 3), (3, 4)] - - This function creates a sliding window suitable for transformations like - sliding means / smoothing - - >>> mean = lambda seq: float(sum(seq)) / len(seq) - >>> list(map(mean, sliding_window(2, [1, 2, 3, 4]))) - [1.5, 2.5, 3.5] - """ - return zip(*(collections.deque(itertools.islice(it, i), 0) or it - for i, it in enumerate(itertools.tee(seq, n)))) - - -no_pad = '__no__pad__' - - -def partition(n, seq, pad=no_pad): - """ Partition sequence into tuples of length n - - >>> list(partition(2, [1, 2, 3, 4])) - [(1, 2), (3, 4)] - - If the length of ``seq`` is not evenly divisible by ``n``, the final tuple - is dropped if ``pad`` is not specified, or filled to length ``n`` by pad: - - >>> list(partition(2, [1, 2, 3, 4, 5])) - [(1, 2), (3, 4)] - - >>> list(partition(2, [1, 2, 3, 4, 5], pad=None)) - [(1, 2), (3, 4), (5, None)] - - See Also: - partition_all - """ - args = [iter(seq)] * n - if pad is no_pad: - return zip(*args) - else: - return zip_longest(*args, fillvalue=pad) - - -def partition_all(n, seq): - """ Partition all elements of sequence into tuples of length at most n - - The final tuple may be shorter to accommodate extra elements. - - >>> list(partition_all(2, [1, 2, 3, 4])) - [(1, 2), (3, 4)] - - >>> list(partition_all(2, [1, 2, 3, 4, 5])) - [(1, 2), (3, 4), (5,)] - - See Also: - partition - """ - args = [iter(seq)] * n - it = zip_longest(*args, fillvalue=no_pad) - try: - prev = next(it) - except StopIteration: - return - for item in it: - yield prev - prev = item - if prev[-1] is no_pad: - try: - # If seq defines __len__, then - # we can quickly calculate where no_pad starts - yield prev[:len(seq) % n] - except TypeError: - # Get first index of no_pad without using .index() - # https://github.com/pytoolz/toolz/issues/387 - # Binary search from CPython's bisect module, - # modified for identity testing. - lo, hi = 0, n - while lo < hi: - mid = (lo + hi) // 2 - if prev[mid] is no_pad: - hi = mid - else: - lo = mid + 1 - yield prev[:lo] - else: - yield prev - - -def count(seq): - """ Count the number of items in seq - - Like the builtin ``len`` but works on lazy sequences. - - Not to be confused with ``itertools.count`` - - See also: - len - """ - if hasattr(seq, '__len__'): - return len(seq) - return sum(1 for i in seq) - - -def pluck(ind, seqs, default=no_default): - """ plucks an element or several elements from each item in a sequence. - - ``pluck`` maps ``itertoolz.get`` over a sequence and returns one or more - elements of each item in the sequence. - - This is equivalent to running `map(curried.get(ind), seqs)` - - ``ind`` can be either a single string/index or a list of strings/indices. - ``seqs`` should be sequence containing sequences or dicts. - - e.g. - - >>> data = [{'id': 1, 'name': 'Cheese'}, {'id': 2, 'name': 'Pies'}] - >>> list(pluck('name', data)) - ['Cheese', 'Pies'] - >>> list(pluck([0, 1], [[1, 2, 3], [4, 5, 7]])) - [(1, 2), (4, 5)] - - See Also: - get - map - """ - if default == no_default: - get = getter(ind) - return map(get, seqs) - elif isinstance(ind, list): - return (tuple(_get(item, seq, default) for item in ind) - for seq in seqs) - return (_get(ind, seq, default) for seq in seqs) - - -def getter(index): - if isinstance(index, list): - if len(index) == 1: - index = index[0] - return lambda x: (x[index],) - elif index: - return operator.itemgetter(*index) - else: - return lambda x: () - else: - return operator.itemgetter(index) - - -def join(leftkey, leftseq, rightkey, rightseq, - left_default=no_default, right_default=no_default): - """ Join two sequences on common attributes - - This is a semi-streaming operation. The LEFT sequence is fully evaluated - and placed into memory. The RIGHT sequence is evaluated lazily and so can - be arbitrarily large. - (Note: If right_default is defined, then unique keys of rightseq - will also be stored in memory.) - - >>> friends = [('Alice', 'Edith'), - ... ('Alice', 'Zhao'), - ... ('Edith', 'Alice'), - ... ('Zhao', 'Alice'), - ... ('Zhao', 'Edith')] - - >>> cities = [('Alice', 'NYC'), - ... ('Alice', 'Chicago'), - ... ('Dan', 'Syndey'), - ... ('Edith', 'Paris'), - ... ('Edith', 'Berlin'), - ... ('Zhao', 'Shanghai')] - - >>> # Vacation opportunities - >>> # In what cities do people have friends? - >>> result = join(second, friends, - ... first, cities) - >>> for ((a, b), (c, d)) in sorted(unique(result)): - ... print((a, d)) - ('Alice', 'Berlin') - ('Alice', 'Paris') - ('Alice', 'Shanghai') - ('Edith', 'Chicago') - ('Edith', 'NYC') - ('Zhao', 'Chicago') - ('Zhao', 'NYC') - ('Zhao', 'Berlin') - ('Zhao', 'Paris') - - Specify outer joins with keyword arguments ``left_default`` and/or - ``right_default``. Here is a full outer join in which unmatched elements - are paired with None. - - >>> identity = lambda x: x - >>> list(join(identity, [1, 2, 3], - ... identity, [2, 3, 4], - ... left_default=None, right_default=None)) - [(2, 2), (3, 3), (None, 4), (1, None)] - - Usually the key arguments are callables to be applied to the sequences. If - the keys are not obviously callable then it is assumed that indexing was - intended, e.g. the following is a legal change. - The join is implemented as a hash join and the keys of leftseq must be - hashable. Additionally, if right_default is defined, then keys of rightseq - must also be hashable. - - >>> # result = join(second, friends, first, cities) - >>> result = join(1, friends, 0, cities) # doctest: +SKIP - """ - if not callable(leftkey): - leftkey = getter(leftkey) - if not callable(rightkey): - rightkey = getter(rightkey) - - d = groupby(leftkey, leftseq) - - if left_default == no_default and right_default == no_default: - # Inner Join - for item in rightseq: - key = rightkey(item) - if key in d: - for left_match in d[key]: - yield (left_match, item) - elif left_default != no_default and right_default == no_default: - # Right Join - for item in rightseq: - key = rightkey(item) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - yield (left_default, item) - elif right_default != no_default: - seen_keys = set() - seen = seen_keys.add - - if left_default == no_default: - # Left Join - for item in rightseq: - key = rightkey(item) - seen(key) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - # Full Join - for item in rightseq: - key = rightkey(item) - seen(key) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - yield (left_default, item) - - for key, matches in d.items(): - if key not in seen_keys: - for match in matches: - yield (match, right_default) - - -def diff(*seqs, **kwargs): - """ Return those items that differ between sequences - - >>> list(diff([1, 2, 3], [1, 2, 10, 100])) - [(3, 10)] - - Shorter sequences may be padded with a ``default`` value: - - >>> list(diff([1, 2, 3], [1, 2, 10, 100], default=None)) - [(3, 10), (None, 100)] - - A ``key`` function may also be applied to each item to use during - comparisons: - - >>> list(diff(['apples', 'bananas'], ['Apples', 'Oranges'], key=str.lower)) - [('bananas', 'Oranges')] - """ - N = len(seqs) - if N == 1 and isinstance(seqs[0], list): - seqs = seqs[0] - N = len(seqs) - if N < 2: - raise TypeError('Too few sequences given (min 2 required)') - default = kwargs.get('default', no_default) - if default == no_default: - iters = zip(*seqs) - else: - iters = zip_longest(*seqs, fillvalue=default) - key = kwargs.get('key', None) - if key is None: - for items in iters: - if items.count(items[0]) != N: - yield items - else: - for items in iters: - vals = tuple(map(key, items)) - if vals.count(vals[0]) != N: - yield items - - -def topk(k, seq, key=None): - """ Find the k largest elements of a sequence - - Operates lazily in ``n*log(k)`` time - - >>> topk(2, [1, 100, 10, 1000]) - (1000, 100) - - Use a key function to change sorted order - - >>> topk(2, ['Alice', 'Bob', 'Charlie', 'Dan'], key=len) - ('Charlie', 'Alice') - - See also: - heapq.nlargest - """ - if key is not None and not callable(key): - key = getter(key) - return tuple(heapq.nlargest(k, seq, key=key)) - - -def peek(seq): - """ Retrieve the next element of a sequence - - Returns the first element and an iterable equivalent to the original - sequence, still having the element retrieved. - - >>> seq = [0, 1, 2, 3, 4] - >>> first, seq = peek(seq) - >>> first - 0 - >>> list(seq) - [0, 1, 2, 3, 4] - """ - iterator = iter(seq) - item = next(iterator) - return item, itertools.chain((item,), iterator) - - -def peekn(n, seq): - """ Retrieve the next n elements of a sequence - - Returns a tuple of the first n elements and an iterable equivalent - to the original, still having the elements retrieved. - - >>> seq = [0, 1, 2, 3, 4] - >>> first_two, seq = peekn(2, seq) - >>> first_two - (0, 1) - >>> list(seq) - [0, 1, 2, 3, 4] - """ - iterator = iter(seq) - peeked = tuple(take(n, iterator)) - return peeked, itertools.chain(iter(peeked), iterator) - - -def random_sample(prob, seq, random_state=None): - """ Return elements from a sequence with probability of prob - - Returns a lazy iterator of random items from seq. - - ``random_sample`` considers each item independently and without - replacement. See below how the first time it returned 13 items and the - next time it returned 6 items. - - >>> seq = list(range(100)) - >>> list(random_sample(0.1, seq)) # doctest: +SKIP - [6, 9, 19, 35, 45, 50, 58, 62, 68, 72, 78, 86, 95] - >>> list(random_sample(0.1, seq)) # doctest: +SKIP - [6, 44, 54, 61, 69, 94] - - Providing an integer seed for ``random_state`` will result in - deterministic sampling. Given the same seed it will return the same sample - every time. - - >>> list(random_sample(0.1, seq, random_state=2016)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - >>> list(random_sample(0.1, seq, random_state=2016)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - - ``random_state`` can also be any object with a method ``random`` that - returns floats between 0.0 and 1.0 (exclusive). - - >>> from random import Random - >>> randobj = Random(2016) - >>> list(random_sample(0.1, seq, random_state=randobj)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - """ - if not hasattr(random_state, 'random'): - from random import Random - - random_state = Random(random_state) - return filter(lambda _: random_state.random() < prob, seq) diff --git a/contrib/python/toolz/py3/toolz/recipes.py b/contrib/python/toolz/py3/toolz/recipes.py deleted file mode 100644 index 89de88db2b..0000000000 --- a/contrib/python/toolz/py3/toolz/recipes.py +++ /dev/null @@ -1,46 +0,0 @@ -import itertools -from .itertoolz import frequencies, pluck, getter - - -__all__ = ('countby', 'partitionby') - - -def countby(key, seq): - """ Count elements of a collection by a key function - - >>> countby(len, ['cat', 'mouse', 'dog']) - {3: 2, 5: 1} - - >>> def iseven(x): return x % 2 == 0 - >>> countby(iseven, [1, 2, 3]) # doctest:+SKIP - {True: 1, False: 2} - - See Also: - groupby - """ - if not callable(key): - key = getter(key) - return frequencies(map(key, seq)) - - -def partitionby(func, seq): - """ Partition a sequence according to a function - - Partition `s` into a sequence of lists such that, when traversing - `s`, every time the output of `func` changes a new list is started - and that and subsequent items are collected into that list. - - >>> is_space = lambda c: c == " " - >>> list(partitionby(is_space, "I have space")) - [('I',), (' ',), ('h', 'a', 'v', 'e'), (' ',), ('s', 'p', 'a', 'c', 'e')] - - >>> is_large = lambda x: x > 10 - >>> list(partitionby(is_large, [1, 2, 1, 99, 88, 33, 99, -1, 5])) - [(1, 2, 1), (99, 88, 33, 99), (-1, 5)] - - See also: - partition - groupby - itertools.groupby - """ - return map(tuple, pluck(1, itertools.groupby(seq, key=func))) diff --git a/contrib/python/toolz/py3/toolz/sandbox/__init__.py b/contrib/python/toolz/py3/toolz/sandbox/__init__.py deleted file mode 100644 index 0abda1cb42..0000000000 --- a/contrib/python/toolz/py3/toolz/sandbox/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -from .core import EqualityHashKey, unzip -from .parallel import fold diff --git a/contrib/python/toolz/py3/toolz/sandbox/core.py b/contrib/python/toolz/py3/toolz/sandbox/core.py deleted file mode 100644 index 55e09d74e9..0000000000 --- a/contrib/python/toolz/py3/toolz/sandbox/core.py +++ /dev/null @@ -1,133 +0,0 @@ -from toolz.itertoolz import getter, cons, pluck -from itertools import tee, starmap - - -# See #166: https://github.com/pytoolz/toolz/issues/166 -# See #173: https://github.com/pytoolz/toolz/pull/173 -class EqualityHashKey(object): - """ Create a hash key that uses equality comparisons between items. - - This may be used to create hash keys for otherwise unhashable types: - - >>> from toolz import curry - >>> EqualityHashDefault = curry(EqualityHashKey, None) - >>> set(map(EqualityHashDefault, [[], (), [1], [1]])) # doctest: +SKIP - {=[]=, =()=, =[1]=} - - **Caution:** adding N ``EqualityHashKey`` items to a hash container - may require O(N**2) operations, not O(N) as for typical hashable types. - Therefore, a suitable key function such as ``tuple`` or ``frozenset`` - is usually preferred over using ``EqualityHashKey`` if possible. - - The ``key`` argument to ``EqualityHashKey`` should be a function or - index that returns a hashable object that effectively distinguishes - unequal items. This helps avoid the poor scaling that occurs when - using the default key. For example, the above example can be improved - by using a key function that distinguishes items by length or type: - - >>> EqualityHashLen = curry(EqualityHashKey, len) - >>> EqualityHashType = curry(EqualityHashKey, type) # this works too - >>> set(map(EqualityHashLen, [[], (), [1], [1]])) # doctest: +SKIP - {=[]=, =()=, =[1]=} - - ``EqualityHashKey`` is convenient to use when a suitable key function - is complicated or unavailable. For example, the following returns all - unique values based on equality: - - >>> from toolz import unique - >>> vals = [[], [], (), [1], [1], [2], {}, {}, {}] - >>> list(unique(vals, key=EqualityHashDefault)) - [[], (), [1], [2], {}] - - **Warning:** don't change the equality value of an item already in a hash - container. Unhashable types are unhashable for a reason. For example: - - >>> L1 = [1] ; L2 = [2] - >>> s = set(map(EqualityHashDefault, [L1, L2])) - >>> s # doctest: +SKIP - {=[1]=, =[2]=} - - >>> L1[0] = 2 # Don't do this! ``s`` now has duplicate items! - >>> s # doctest: +SKIP - {=[2]=, =[2]=} - - Although this may appear problematic, immutable data types is a common - idiom in functional programming, and``EqualityHashKey`` easily allows - the same idiom to be used by convention rather than strict requirement. - - See Also: - identity - """ - __slots__ = ['item', 'key'] - _default_hashkey = '__default__hashkey__' - - def __init__(self, key, item): - if key is None: - self.key = self._default_hashkey - elif not callable(key): - self.key = getter(key) - else: - self.key = key - self.item = item - - def __hash__(self): - if self.key == self._default_hashkey: - val = self.key - else: - val = self.key(self.item) - return hash(val) - - def __eq__(self, other): - try: - return (self._default_hashkey == other._default_hashkey and - self.item == other.item) - except AttributeError: - return False - - def __ne__(self, other): - return not self.__eq__(other) - - def __str__(self): - return '=%s=' % str(self.item) - - def __repr__(self): - return '=%s=' % repr(self.item) - - -# See issue #293: https://github.com/pytoolz/toolz/issues/239 -def unzip(seq): - """Inverse of ``zip`` - - >>> a, b = unzip([('a', 1), ('b', 2)]) - >>> list(a) - ['a', 'b'] - >>> list(b) - [1, 2] - - Unlike the naive implementation ``def unzip(seq): zip(*seq)`` this - implementation can handle an infinite sequence ``seq``. - - Caveats: - - * The implementation uses ``tee``, and so can use a significant amount - of auxiliary storage if the resulting iterators are consumed at - different times. - - * The inner sequence cannot be infinite. In Python 3 ``zip(*seq)`` can be - used if ``seq`` is a finite sequence of infinite sequences. - - """ - - seq = iter(seq) - - # Check how many iterators we need - try: - first = tuple(next(seq)) - except StopIteration: - return tuple() - - # and create them - niters = len(first) - seqs = tee(cons(first, seq), niters) - - return tuple(starmap(pluck, enumerate(seqs))) diff --git a/contrib/python/toolz/py3/toolz/sandbox/parallel.py b/contrib/python/toolz/py3/toolz/sandbox/parallel.py deleted file mode 100644 index 114077d2ba..0000000000 --- a/contrib/python/toolz/py3/toolz/sandbox/parallel.py +++ /dev/null @@ -1,75 +0,0 @@ -import functools -from toolz.itertoolz import partition_all -from toolz.utils import no_default - - -def _reduce(func, seq, initial=None): - if initial is None: - return functools.reduce(func, seq) - else: - return functools.reduce(func, seq, initial) - - -def fold(binop, seq, default=no_default, map=map, chunksize=128, combine=None): - """ - Reduce without guarantee of ordered reduction. - - inputs: - - ``binop`` - associative operator. The associative property allows us to - leverage a parallel map to perform reductions in parallel. - ``seq`` - a sequence to be aggregated - ``default`` - an identity element like 0 for ``add`` or 1 for mul - - ``map`` - an implementation of ``map``. This may be parallel and - determines how work is distributed. - ``chunksize`` - Number of elements of ``seq`` that should be handled - within a single function call - ``combine`` - Binary operator to combine two intermediate results. - If ``binop`` is of type (total, item) -> total - then ``combine`` is of type (total, total) -> total - Defaults to ``binop`` for common case of operators like add - - Fold chunks up the collection into blocks of size ``chunksize`` and then - feeds each of these to calls to ``reduce``. This work is distributed - with a call to ``map``, gathered back and then refolded to finish the - computation. In this way ``fold`` specifies only how to chunk up data but - leaves the distribution of this work to an externally provided ``map`` - function. This function can be sequential or rely on multithreading, - multiprocessing, or even distributed solutions. - - If ``map`` intends to serialize functions it should be prepared to accept - and serialize lambdas. Note that the standard ``pickle`` module fails - here. - - Example - ------- - - >>> # Provide a parallel map to accomplish a parallel sum - >>> from operator import add - >>> fold(add, [1, 2, 3, 4], chunksize=2, map=map) - 10 - """ - assert chunksize > 1 - - if combine is None: - combine = binop - - chunks = partition_all(chunksize, seq) - - # Evaluate sequence in chunks via map - if default == no_default: - results = map( - functools.partial(_reduce, binop), - chunks) - else: - results = map( - functools.partial(_reduce, binop, initial=default), - chunks) - - results = list(results) # TODO: Support complete laziness - - if len(results) == 1: # Return completed result - return results[0] - else: # Recurse to reaggregate intermediate results - return fold(combine, results, map=map, chunksize=chunksize) diff --git a/contrib/python/toolz/py3/toolz/utils.py b/contrib/python/toolz/py3/toolz/utils.py deleted file mode 100644 index 1002c4649f..0000000000 --- a/contrib/python/toolz/py3/toolz/utils.py +++ /dev/null @@ -1,9 +0,0 @@ -def raises(err, lamda): - try: - lamda() - return False - except err: - return True - - -no_default = '__no__default__' diff --git a/contrib/python/toolz/py3/ya.make b/contrib/python/toolz/py3/ya.make deleted file mode 100644 index bc99c48e44..0000000000 --- a/contrib/python/toolz/py3/ya.make +++ /dev/null @@ -1,42 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY3_LIBRARY() - -VERSION(0.12.0) - -LICENSE(BSD-3-Clause) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - tlz/__init__.py - tlz/_build_tlz.py - toolz/__init__.py - toolz/_signatures.py - toolz/_version.py - toolz/compatibility.py - toolz/curried/__init__.py - toolz/curried/exceptions.py - toolz/curried/operator.py - toolz/dicttoolz.py - toolz/functoolz.py - toolz/itertoolz.py - toolz/recipes.py - toolz/sandbox/__init__.py - toolz/sandbox/core.py - toolz/sandbox/parallel.py - toolz/utils.py -) - -RESOURCE_FILES( - PREFIX contrib/python/toolz/py3/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/toolz/ya.make b/contrib/python/toolz/ya.make deleted file mode 100644 index 0c05eedeac..0000000000 --- a/contrib/python/toolz/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/toolz/py2) -ELSE() - PEERDIR(contrib/python/toolz/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) |