diff options
| author | robot-piglet <robot-piglet@yandex-team.com> | 2023-10-03 15:02:38 +0300 |
|---|---|---|
| committer | robot-piglet <robot-piglet@yandex-team.com> | 2023-10-03 16:04:35 +0300 |
| commit | 5478b8f55cc7055a4861c4030e0c401b5c72714c (patch) | |
| tree | 3d003e5b4c1800297fcc491faffc9a006d174289 /contrib/python | |
| parent | ca778ad9bfb31839b0f05a4995753bc61db648ad (diff) | |
| download | ydb-5478b8f55cc7055a4861c4030e0c401b5c72714c.tar.gz | |
Intermediate changes
Diffstat (limited to 'contrib/python')
79 files changed, 0 insertions, 18984 deletions
diff --git a/contrib/python/distro/py2/.dist-info/METADATA b/contrib/python/distro/py2/.dist-info/METADATA deleted file mode 100644 index bc42dfcf165..00000000000 --- a/contrib/python/distro/py2/.dist-info/METADATA +++ /dev/null @@ -1,169 +0,0 @@ -Metadata-Version: 2.1 -Name: distro -Version: 1.6.0 -Summary: Distro - an OS platform information API -Home-page: https://github.com/python-distro/distro -Author: Nir Cohen -Author-email: nir36g@gmail.com -License: Apache License, Version 2.0 -Platform: All -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: Intended Audience :: System Administrators -Classifier: License :: OSI Approved :: Apache Software License -Classifier: Operating System :: POSIX :: Linux -Classifier: Operating System :: POSIX :: BSD -Classifier: Operating System :: POSIX :: BSD :: FreeBSD -Classifier: Operating System :: POSIX :: BSD :: NetBSD -Classifier: Operating System :: POSIX :: BSD :: OpenBSD -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Topic :: Software Development :: Libraries :: Python Modules -Classifier: Topic :: System :: Operating System -Description-Content-Type: text/markdown -License-File: LICENSE - -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.linux_distribution(full_distribution_name=False) -('centos', '7.1.1503', 'Core') -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 2.7, 3.4+ and PyPy and on -any distribution that provides one or more of the data sources -covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/sid/python-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python2-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ - - diff --git a/contrib/python/distro/py2/.dist-info/entry_points.txt b/contrib/python/distro/py2/.dist-info/entry_points.txt deleted file mode 100644 index dd4023997f5..00000000000 --- a/contrib/python/distro/py2/.dist-info/entry_points.txt +++ /dev/null @@ -1,3 +0,0 @@ -[console_scripts] -distro = distro:main - diff --git a/contrib/python/distro/py2/.dist-info/top_level.txt b/contrib/python/distro/py2/.dist-info/top_level.txt deleted file mode 100644 index 0e0933171d3..00000000000 --- a/contrib/python/distro/py2/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -distro diff --git a/contrib/python/distro/py2/LICENSE b/contrib/python/distro/py2/LICENSE deleted file mode 100644 index e06d2081865..00000000000 --- a/contrib/python/distro/py2/LICENSE +++ /dev/null @@ -1,202 +0,0 @@ -Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - diff --git a/contrib/python/distro/py2/README.md b/contrib/python/distro/py2/README.md deleted file mode 100644 index 76eaef244df..00000000000 --- a/contrib/python/distro/py2/README.md +++ /dev/null @@ -1,135 +0,0 @@ -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.linux_distribution(full_distribution_name=False) -('centos', '7.1.1503', 'Core') -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 2.7, 3.4+ and PyPy and on -any distribution that provides one or more of the data sources -covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/sid/python-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python2-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ diff --git a/contrib/python/distro/py2/distro.py b/contrib/python/distro/py2/distro.py deleted file mode 100644 index 7892741347d..00000000000 --- a/contrib/python/distro/py2/distro.py +++ /dev/null @@ -1,1386 +0,0 @@ -# Copyright 2015,2016,2017 Nir Cohen -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -""" -The ``distro`` package (``distro`` stands for Linux Distribution) provides -information about the Linux distribution it runs on, such as a reliable -machine-readable distro ID, or version information. - -It is the recommended replacement for Python's original -:py:func:`platform.linux_distribution` function, but it provides much more -functionality. An alternative implementation became necessary because Python -3.5 deprecated this function, and Python 3.8 removed it altogether. Its -predecessor function :py:func:`platform.dist` was already deprecated since -Python 2.6 and removed in Python 3.8. Still, there are many cases in which -access to OS distribution information is needed. See `Python issue 1322 -<https://bugs.python.org/issue1322>`_ for more information. -""" - -import argparse -import json -import logging -import os -import re -import shlex -import subprocess -import sys -import warnings - -__version__ = "1.6.0" - -# Use `if False` to avoid an ImportError on Python 2. After dropping Python 2 -# support, can use typing.TYPE_CHECKING instead. See: -# https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING -if False: # pragma: nocover - from typing import ( - Any, - Callable, - Dict, - Iterable, - Optional, - Sequence, - TextIO, - Tuple, - Type, - TypedDict, - Union, - ) - - VersionDict = TypedDict( - "VersionDict", {"major": str, "minor": str, "build_number": str} - ) - InfoDict = TypedDict( - "InfoDict", - { - "id": str, - "version": str, - "version_parts": VersionDict, - "like": str, - "codename": str, - }, - ) - - -_UNIXCONFDIR = os.environ.get("UNIXCONFDIR", "/etc") -_UNIXUSRLIBDIR = os.environ.get("UNIXUSRLIBDIR", "/usr/lib") -_OS_RELEASE_BASENAME = "os-release" - -#: Translation table for normalizing the "ID" attribute defined in os-release -#: files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as defined in the os-release file, translated to lower case, -#: with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_OS_ID = { - "ol": "oracle", # Oracle Linux -} - -#: Translation table for normalizing the "Distributor ID" attribute returned by -#: the lsb_release command, for use by the :func:`distro.id` method. -#: -#: * Key: Value as returned by the lsb_release command, translated to lower -#: case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_LSB_ID = { - "enterpriseenterpriseas": "oracle", # Oracle Enterprise Linux 4 - "enterpriseenterpriseserver": "oracle", # Oracle Linux 5 - "redhatenterpriseworkstation": "rhel", # RHEL 6, 7 Workstation - "redhatenterpriseserver": "rhel", # RHEL 6, 7 Server - "redhatenterprisecomputenode": "rhel", # RHEL 6 ComputeNode -} - -#: Translation table for normalizing the distro ID derived from the file name -#: of distro release files, for use by the :func:`distro.id` method. -#: -#: * Key: Value as derived from the file name of a distro release file, -#: translated to lower case, with blanks translated to underscores. -#: -#: * Value: Normalized value. -NORMALIZED_DISTRO_ID = { - "redhat": "rhel", # RHEL 6.x, 7.x -} - -# Pattern for content of distro release file (reversed) -_DISTRO_RELEASE_CONTENT_REVERSED_PATTERN = re.compile( - r"(?:[^)]*\)(.*)\()? *(?:STL )?([\d.+\-a-z]*\d) *(?:esaeler *)?(.+)" -) - -# Pattern for base file name of distro release file -_DISTRO_RELEASE_BASENAME_PATTERN = re.compile(r"(\w+)[-_](release|version)$") - -# Base file names to be ignored when searching for distro release file -_DISTRO_RELEASE_IGNORE_BASENAMES = ( - "debian_version", - "lsb-release", - "oem-release", - _OS_RELEASE_BASENAME, - "system-release", - "plesk-release", - "iredmail-release", -) - - -def linux_distribution(full_distribution_name=True): - # type: (bool) -> Tuple[str, str, str] - """ - .. deprecated:: 1.6.0 - - :func:`distro.linux_distribution()` is deprecated. It should only be - used as a compatibility shim with Python's - :py:func:`platform.linux_distribution()`. Please use :func:`distro.id`, - :func:`distro.version` and :func:`distro.name` instead. - - Return information about the current OS distribution as a tuple - ``(id_name, version, codename)`` with items as follows: - - * ``id_name``: If *full_distribution_name* is false, the result of - :func:`distro.id`. Otherwise, the result of :func:`distro.name`. - - * ``version``: The result of :func:`distro.version`. - - * ``codename``: The result of :func:`distro.codename`. - - The interface of this function is compatible with the original - :py:func:`platform.linux_distribution` function, supporting a subset of - its parameters. - - The data it returns may not exactly be the same, because it uses more data - sources than the original function, and that may lead to different data if - the OS distribution is not consistent across multiple data sources it - provides (there are indeed such distributions ...). - - Another reason for differences is the fact that the :func:`distro.id` - method normalizes the distro ID string to a reliable machine-readable value - for a number of popular OS distributions. - """ - warnings.warn( - "distro.linux_distribution() is deprecated. It should only be used as a " - "compatibility shim with Python's platform.linux_distribution(). Please use " - "distro.id(), distro.version() and distro.name() instead.", - DeprecationWarning, - stacklevel=2, - ) - return _distro.linux_distribution(full_distribution_name) - - -def id(): - # type: () -> str - """ - Return the distro ID of the current distribution, as a - machine-readable string. - - For a number of OS distributions, the returned distro ID value is - *reliable*, in the sense that it is documented and that it does not change - across releases of the distribution. - - This package maintains the following reliable distro ID values: - - ============== ========================================= - Distro ID Distribution - ============== ========================================= - "ubuntu" Ubuntu - "debian" Debian - "rhel" RedHat Enterprise Linux - "centos" CentOS - "fedora" Fedora - "sles" SUSE Linux Enterprise Server - "opensuse" openSUSE - "amazon" Amazon Linux - "arch" Arch Linux - "cloudlinux" CloudLinux OS - "exherbo" Exherbo Linux - "gentoo" GenToo Linux - "ibm_powerkvm" IBM PowerKVM - "kvmibm" KVM for IBM z Systems - "linuxmint" Linux Mint - "mageia" Mageia - "mandriva" Mandriva Linux - "parallels" Parallels - "pidora" Pidora - "raspbian" Raspbian - "oracle" Oracle Linux (and Oracle Enterprise Linux) - "scientific" Scientific Linux - "slackware" Slackware - "xenserver" XenServer - "openbsd" OpenBSD - "netbsd" NetBSD - "freebsd" FreeBSD - "midnightbsd" MidnightBSD - ============== ========================================= - - If you have a need to get distros for reliable IDs added into this set, - or if you find that the :func:`distro.id` function returns a different - distro ID for one of the listed distros, please create an issue in the - `distro issue tracker`_. - - **Lookup hierarchy and transformations:** - - First, the ID is obtained from the following sources, in the specified - order. The first available and non-empty value is used: - - * the value of the "ID" attribute of the os-release file, - - * the value of the "Distributor ID" attribute returned by the lsb_release - command, - - * the first part of the file name of the distro release file, - - The so determined ID value then passes the following transformations, - before it is returned by this method: - - * it is translated to lower case, - - * blanks (which should not be there anyway) are translated to underscores, - - * a normalization of the ID is performed, based upon - `normalization tables`_. The purpose of this normalization is to ensure - that the ID is as reliable as possible, even across incompatible changes - in the OS distributions. A common reason for an incompatible change is - the addition of an os-release file, or the addition of the lsb_release - command, with ID values that differ from what was previously determined - from the distro release file name. - """ - return _distro.id() - - -def name(pretty=False): - # type: (bool) -> str - """ - Return the name of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the name is returned without version or codename. - (e.g. "CentOS Linux") - - If *pretty* is true, the version and codename are appended. - (e.g. "CentOS Linux 7.1.1503 (Core)") - - **Lookup hierarchy:** - - The name is obtained from the following sources, in the specified order. - The first available and non-empty value is used: - - * If *pretty* is false: - - - the value of the "NAME" attribute of the os-release file, - - - the value of the "Distributor ID" attribute returned by the lsb_release - command, - - - the value of the "<name>" field of the distro release file. - - * If *pretty* is true: - - - the value of the "PRETTY_NAME" attribute of the os-release file, - - - the value of the "Description" attribute returned by the lsb_release - command, - - - the value of the "<name>" field of the distro release file, appended - with the value of the pretty version ("<version_id>" and "<codename>" - fields) of the distro release file, if available. - """ - return _distro.name(pretty) - - -def version(pretty=False, best=False): - # type: (bool, bool) -> str - """ - Return the version of the current OS distribution, as a human-readable - string. - - If *pretty* is false, the version is returned without codename (e.g. - "7.0"). - - If *pretty* is true, the codename in parenthesis is appended, if the - codename is non-empty (e.g. "7.0 (Maipo)"). - - Some distributions provide version numbers with different precisions in - the different sources of distribution information. Examining the different - sources in a fixed priority order does not always yield the most precise - version (e.g. for Debian 8.2, or CentOS 7.1). - - The *best* parameter can be used to control the approach for the returned - version: - - If *best* is false, the first non-empty version number in priority order of - the examined sources is returned. - - If *best* is true, the most precise version number out of all examined - sources is returned. - - **Lookup hierarchy:** - - In all cases, the version number is obtained from the following sources. - If *best* is false, this order represents the priority order: - - * the value of the "VERSION_ID" attribute of the os-release file, - * the value of the "Release" attribute returned by the lsb_release - command, - * the version number parsed from the "<version_id>" field of the first line - of the distro release file, - * the version number parsed from the "PRETTY_NAME" attribute of the - os-release file, if it follows the format of the distro release files. - * the version number parsed from the "Description" attribute returned by - the lsb_release command, if it follows the format of the distro release - files. - """ - return _distro.version(pretty, best) - - -def version_parts(best=False): - # type: (bool) -> Tuple[str, str, str] - """ - Return the version of the current OS distribution as a tuple - ``(major, minor, build_number)`` with items as follows: - - * ``major``: The result of :func:`distro.major_version`. - - * ``minor``: The result of :func:`distro.minor_version`. - - * ``build_number``: The result of :func:`distro.build_number`. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.version_parts(best) - - -def major_version(best=False): - # type: (bool) -> str - """ - Return the major version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The major version is the first - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.major_version(best) - - -def minor_version(best=False): - # type: (bool) -> str - """ - Return the minor version of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The minor version is the second - part of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.minor_version(best) - - -def build_number(best=False): - # type: (bool) -> str - """ - Return the build number of the current OS distribution, as a string, - if provided. - Otherwise, the empty string is returned. The build number is the third part - of the dot-separated version string. - - For a description of the *best* parameter, see the :func:`distro.version` - method. - """ - return _distro.build_number(best) - - -def like(): - # type: () -> str - """ - Return a space-separated list of distro IDs of distributions that are - closely related to the current OS distribution in regards to packaging - and programming interfaces, for example distributions the current - distribution is a derivative from. - - **Lookup hierarchy:** - - This information item is only provided by the os-release file. - For details, see the description of the "ID_LIKE" attribute in the - `os-release man page - <http://www.freedesktop.org/software/systemd/man/os-release.html>`_. - """ - return _distro.like() - - -def codename(): - # type: () -> str - """ - Return the codename for the release of the current OS distribution, - as a string. - - If the distribution does not have a codename, an empty string is returned. - - Note that the returned codename is not always really a codename. For - example, openSUSE returns "x86_64". This function does not handle such - cases in any special way and just returns the string it finds, if any. - - **Lookup hierarchy:** - - * the codename within the "VERSION" attribute of the os-release file, if - provided, - - * the value of the "Codename" attribute returned by the lsb_release - command, - - * the value of the "<codename>" field of the distro release file. - """ - return _distro.codename() - - -def info(pretty=False, best=False): - # type: (bool, bool) -> InfoDict - """ - Return certain machine-readable information items about the current OS - distribution in a dictionary, as shown in the following example: - - .. sourcecode:: python - - { - 'id': 'rhel', - 'version': '7.0', - 'version_parts': { - 'major': '7', - 'minor': '0', - 'build_number': '' - }, - 'like': 'fedora', - 'codename': 'Maipo' - } - - The dictionary structure and keys are always the same, regardless of which - information items are available in the underlying data sources. The values - for the various keys are as follows: - - * ``id``: The result of :func:`distro.id`. - - * ``version``: The result of :func:`distro.version`. - - * ``version_parts -> major``: The result of :func:`distro.major_version`. - - * ``version_parts -> minor``: The result of :func:`distro.minor_version`. - - * ``version_parts -> build_number``: The result of - :func:`distro.build_number`. - - * ``like``: The result of :func:`distro.like`. - - * ``codename``: The result of :func:`distro.codename`. - - For a description of the *pretty* and *best* parameters, see the - :func:`distro.version` method. - """ - return _distro.info(pretty, best) - - -def os_release_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the os-release file data source of the current OS distribution. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_info() - - -def lsb_release_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the lsb_release command data source of the current OS distribution. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_info() - - -def distro_release_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_info() - - -def uname_info(): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information items - from the distro release file data source of the current OS distribution. - """ - return _distro.uname_info() - - -def os_release_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the os-release file data source - of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `os-release file`_ for details about these information items. - """ - return _distro.os_release_attr(attribute) - - -def lsb_release_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the lsb_release command output - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `lsb_release command output`_ for details about these information - items. - """ - return _distro.lsb_release_attr(attribute) - - -def distro_release_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - - See `distro release file`_ for details about these information items. - """ - return _distro.distro_release_attr(attribute) - - -def uname_attr(attribute): - # type: (str) -> str - """ - Return a single named information item from the distro release file - data source of the current OS distribution. - - Parameters: - - * ``attribute`` (string): Key of the information item. - - Returns: - - * (string): Value of the information item, if the item exists. - The empty string, if the item does not exist. - """ - return _distro.uname_attr(attribute) - - -try: - from functools import cached_property -except ImportError: - # Python < 3.8 - class cached_property(object): # type: ignore - """A version of @property which caches the value. On access, it calls the - underlying function and sets the value in `__dict__` so future accesses - will not re-call the property. - """ - - def __init__(self, f): - # type: (Callable[[Any], Any]) -> None - self._fname = f.__name__ - self._f = f - - def __get__(self, obj, owner): - # type: (Any, Type[Any]) -> Any - assert obj is not None, "call {} on an instance".format(self._fname) - ret = obj.__dict__[self._fname] = self._f(obj) - return ret - - -class LinuxDistribution(object): - """ - Provides information about a OS distribution. - - This package creates a private module-global instance of this class with - default initialization arguments, that is used by the - `consolidated accessor functions`_ and `single source accessor functions`_. - By using default initialization arguments, that module-global instance - returns data about the current OS distribution (i.e. the distro this - package runs on). - - Normally, it is not necessary to create additional instances of this class. - However, in situations where control is needed over the exact data sources - that are used, instances of this class can be created with a specific - distro release file, or a specific os-release file, or without invoking the - lsb_release command. - """ - - def __init__( - self, - include_lsb=True, - os_release_file="", - distro_release_file="", - include_uname=True, - root_dir=None, - ): - # type: (bool, str, str, bool, Optional[str]) -> None - """ - The initialization method of this class gathers information from the - available data sources, and stores that in private instance attributes. - Subsequent access to the information items uses these private instance - attributes, so that the data sources are read only once. - - Parameters: - - * ``include_lsb`` (bool): Controls whether the - `lsb_release command output`_ is included as a data source. - - If the lsb_release command is not available in the program execution - path, the data source for the lsb_release command will be empty. - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is to be used as a data source. - - An empty string (the default) will cause the default path name to - be used (see `os-release file`_ for details). - - If the specified or defaulted os-release file does not exist, the - data source for the os-release file will be empty. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is to be used as a data source. - - An empty string (the default) will cause a default search algorithm - to be used (see `distro release file`_ for details). - - If the specified distro release file does not exist, or if no default - distro release file can be found, the data source for the distro - release file will be empty. - - * ``include_uname`` (bool): Controls whether uname command output is - included as a data source. If the uname command is not available in - the program execution path the data source for the uname command will - be empty. - - * ``root_dir`` (string): The absolute path to the root directory to use - to find distro-related information files. - - Public instance attributes: - - * ``os_release_file`` (string): The path name of the - `os-release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``distro_release_file`` (string): The path name of the - `distro release file`_ that is actually used as a data source. The - empty string if no distro release file is used as a data source. - - * ``include_lsb`` (bool): The result of the ``include_lsb`` parameter. - This controls whether the lsb information will be loaded. - - * ``include_uname`` (bool): The result of the ``include_uname`` - parameter. This controls whether the uname information will - be loaded. - - Raises: - - * :py:exc:`IOError`: Some I/O issue with an os-release file or distro - release file. - - * :py:exc:`subprocess.CalledProcessError`: The lsb_release command had - some issue (other than not being available in the program execution - path). - - * :py:exc:`UnicodeError`: A data source has unexpected characters or - uses an unexpected encoding. - """ - self.root_dir = root_dir - self.etc_dir = os.path.join(root_dir, "etc") if root_dir else _UNIXCONFDIR - self.usr_lib_dir = ( - os.path.join(root_dir, "usr/lib") if root_dir else _UNIXUSRLIBDIR - ) - - if os_release_file: - self.os_release_file = os_release_file - else: - etc_dir_os_release_file = os.path.join(self.etc_dir, _OS_RELEASE_BASENAME) - usr_lib_os_release_file = os.path.join( - self.usr_lib_dir, _OS_RELEASE_BASENAME - ) - - # NOTE: The idea is to respect order **and** have it set - # at all times for API backwards compatibility. - if os.path.isfile(etc_dir_os_release_file) or not os.path.isfile( - usr_lib_os_release_file - ): - self.os_release_file = etc_dir_os_release_file - else: - self.os_release_file = usr_lib_os_release_file - - self.distro_release_file = distro_release_file or "" # updated later - self.include_lsb = include_lsb - self.include_uname = include_uname - - def __repr__(self): - # type: () -> str - """Return repr of all info""" - return ( - "LinuxDistribution(" - "os_release_file={self.os_release_file!r}, " - "distro_release_file={self.distro_release_file!r}, " - "include_lsb={self.include_lsb!r}, " - "include_uname={self.include_uname!r}, " - "_os_release_info={self._os_release_info!r}, " - "_lsb_release_info={self._lsb_release_info!r}, " - "_distro_release_info={self._distro_release_info!r}, " - "_uname_info={self._uname_info!r})".format(self=self) - ) - - def linux_distribution(self, full_distribution_name=True): - # type: (bool) -> Tuple[str, str, str] - """ - Return information about the OS distribution that is compatible - with Python's :func:`platform.linux_distribution`, supporting a subset - of its parameters. - - For details, see :func:`distro.linux_distribution`. - """ - return ( - self.name() if full_distribution_name else self.id(), - self.version(), - self.codename(), - ) - - def id(self): - # type: () -> str - """Return the distro ID of the OS distribution, as a string. - - For details, see :func:`distro.id`. - """ - - def normalize(distro_id, table): - # type: (str, Dict[str, str]) -> str - distro_id = distro_id.lower().replace(" ", "_") - return table.get(distro_id, distro_id) - - distro_id = self.os_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_OS_ID) - - distro_id = self.lsb_release_attr("distributor_id") - if distro_id: - return normalize(distro_id, NORMALIZED_LSB_ID) - - distro_id = self.distro_release_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - distro_id = self.uname_attr("id") - if distro_id: - return normalize(distro_id, NORMALIZED_DISTRO_ID) - - return "" - - def name(self, pretty=False): - # type: (bool) -> str - """ - Return the name of the OS distribution, as a string. - - For details, see :func:`distro.name`. - """ - name = ( - self.os_release_attr("name") - or self.lsb_release_attr("distributor_id") - or self.distro_release_attr("name") - or self.uname_attr("name") - ) - if pretty: - name = self.os_release_attr("pretty_name") or self.lsb_release_attr( - "description" - ) - if not name: - name = self.distro_release_attr("name") or self.uname_attr("name") - version = self.version(pretty=True) - if version: - name = name + " " + version - return name or "" - - def version(self, pretty=False, best=False): - # type: (bool, bool) -> str - """ - Return the version of the OS distribution, as a string. - - For details, see :func:`distro.version`. - """ - versions = [ - self.os_release_attr("version_id"), - self.lsb_release_attr("release"), - self.distro_release_attr("version_id"), - self._parse_distro_release_content(self.os_release_attr("pretty_name")).get( - "version_id", "" - ), - self._parse_distro_release_content( - self.lsb_release_attr("description") - ).get("version_id", ""), - self.uname_attr("release"), - ] - version = "" - if best: - # This algorithm uses the last version in priority order that has - # the best precision. If the versions are not in conflict, that - # does not matter; otherwise, using the last one instead of the - # first one might be considered a surprise. - for v in versions: - if v.count(".") > version.count(".") or version == "": - version = v - else: - for v in versions: - if v != "": - version = v - break - if pretty and version and self.codename(): - version = "{0} ({1})".format(version, self.codename()) - return version - - def version_parts(self, best=False): - # type: (bool) -> Tuple[str, str, str] - """ - Return the version of the OS distribution, as a tuple of version - numbers. - - For details, see :func:`distro.version_parts`. - """ - version_str = self.version(best=best) - if version_str: - version_regex = re.compile(r"(\d+)\.?(\d+)?\.?(\d+)?") - matches = version_regex.match(version_str) - if matches: - major, minor, build_number = matches.groups() - return major, minor or "", build_number or "" - return "", "", "" - - def major_version(self, best=False): - # type: (bool) -> str - """ - Return the major version number of the current distribution. - - For details, see :func:`distro.major_version`. - """ - return self.version_parts(best)[0] - - def minor_version(self, best=False): - # type: (bool) -> str - """ - Return the minor version number of the current distribution. - - For details, see :func:`distro.minor_version`. - """ - return self.version_parts(best)[1] - - def build_number(self, best=False): - # type: (bool) -> str - """ - Return the build number of the current distribution. - - For details, see :func:`distro.build_number`. - """ - return self.version_parts(best)[2] - - def like(self): - # type: () -> str - """ - Return the IDs of distributions that are like the OS distribution. - - For details, see :func:`distro.like`. - """ - return self.os_release_attr("id_like") or "" - - def codename(self): - # type: () -> str - """ - Return the codename of the OS distribution. - - For details, see :func:`distro.codename`. - """ - try: - # Handle os_release specially since distros might purposefully set - # this to empty string to have no codename - return self._os_release_info["codename"] - except KeyError: - return ( - self.lsb_release_attr("codename") - or self.distro_release_attr("codename") - or "" - ) - - def info(self, pretty=False, best=False): - # type: (bool, bool) -> InfoDict - """ - Return certain machine-readable information about the OS - distribution. - - For details, see :func:`distro.info`. - """ - return dict( - id=self.id(), - version=self.version(pretty, best), - version_parts=dict( - major=self.major_version(best), - minor=self.minor_version(best), - build_number=self.build_number(best), - ), - like=self.like(), - codename=self.codename(), - ) - - def os_release_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the os-release file data source of the OS distribution. - - For details, see :func:`distro.os_release_info`. - """ - return self._os_release_info - - def lsb_release_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the lsb_release command data source of the OS - distribution. - - For details, see :func:`distro.lsb_release_info`. - """ - return self._lsb_release_info - - def distro_release_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the distro release file data source of the OS - distribution. - - For details, see :func:`distro.distro_release_info`. - """ - return self._distro_release_info - - def uname_info(self): - # type: () -> Dict[str, str] - """ - Return a dictionary containing key-value pairs for the information - items from the uname command data source of the OS distribution. - - For details, see :func:`distro.uname_info`. - """ - return self._uname_info - - def os_release_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the os-release file data - source of the OS distribution. - - For details, see :func:`distro.os_release_attr`. - """ - return self._os_release_info.get(attribute, "") - - def lsb_release_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the lsb_release command - output data source of the OS distribution. - - For details, see :func:`distro.lsb_release_attr`. - """ - return self._lsb_release_info.get(attribute, "") - - def distro_release_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the distro release file - data source of the OS distribution. - - For details, see :func:`distro.distro_release_attr`. - """ - return self._distro_release_info.get(attribute, "") - - def uname_attr(self, attribute): - # type: (str) -> str - """ - Return a single named information item from the uname command - output data source of the OS distribution. - - For details, see :func:`distro.uname_attr`. - """ - return self._uname_info.get(attribute, "") - - @cached_property - def _os_release_info(self): - # type: () -> Dict[str, str] - """ - Get the information items from the specified os-release file. - - Returns: - A dictionary containing all information items. - """ - if os.path.isfile(self.os_release_file): - with open(self.os_release_file) as release_file: - return self._parse_os_release_content(release_file) - return {} - - @staticmethod - def _parse_os_release_content(lines): - # type: (TextIO) -> Dict[str, str] - """ - Parse the lines of an os-release file. - - Parameters: - - * lines: Iterable through the lines in the os-release file. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - lexer = shlex.shlex(lines, posix=True) - lexer.whitespace_split = True - - # The shlex module defines its `wordchars` variable using literals, - # making it dependent on the encoding of the Python source file. - # In Python 2.6 and 2.7, the shlex source file is encoded in - # 'iso-8859-1', and the `wordchars` variable is defined as a byte - # string. This causes a UnicodeDecodeError to be raised when the - # parsed content is a unicode object. The following fix resolves that - # (... but it should be fixed in shlex...): - if sys.version_info[0] == 2 and isinstance(lexer.wordchars, bytes): - lexer.wordchars = lexer.wordchars.decode("iso-8859-1") - - tokens = list(lexer) - for token in tokens: - # At this point, all shell-like parsing has been done (i.e. - # comments processed, quotes and backslash escape sequences - # processed, multi-line values assembled, trailing newlines - # stripped, etc.), so the tokens are now either: - # * variable assignments: var=value - # * commands or their arguments (not allowed in os-release) - if "=" in token: - k, v = token.split("=", 1) - props[k.lower()] = v - else: - # Ignore any tokens that are not variable assignments - pass - - if "version_codename" in props: - # os-release added a version_codename field. Use that in - # preference to anything else Note that some distros purposefully - # do not have code names. They should be setting - # version_codename="" - props["codename"] = props["version_codename"] - elif "ubuntu_codename" in props: - # Same as above but a non-standard field name used on older Ubuntus - props["codename"] = props["ubuntu_codename"] - elif "version" in props: - # If there is no version_codename, parse it from the version - match = re.search(r"(\(\D+\))|,(\s+)?\D+", props["version"]) - if match: - codename = match.group() - codename = codename.strip("()") - codename = codename.strip(",") - codename = codename.strip() - # codename appears within paranthese. - props["codename"] = codename - - return props - - @cached_property - def _lsb_release_info(self): - # type: () -> Dict[str, str] - """ - Get the information items from the lsb_release command output. - - Returns: - A dictionary containing all information items. - """ - if not self.include_lsb: - return {} - with open(os.devnull, "wb") as devnull: - try: - cmd = ("lsb_release", "-a") - stdout = subprocess.check_output(cmd, stderr=devnull) - # Command not found or lsb_release returned error - except (OSError, subprocess.CalledProcessError): - return {} - content = self._to_str(stdout).splitlines() - return self._parse_lsb_release_content(content) - - @staticmethod - def _parse_lsb_release_content(lines): - # type: (Iterable[str]) -> Dict[str, str] - """ - Parse the output of the lsb_release command. - - Parameters: - - * lines: Iterable through the lines of the lsb_release output. - Each line must be a unicode string or a UTF-8 encoded byte - string. - - Returns: - A dictionary containing all information items. - """ - props = {} - for line in lines: - kv = line.strip("\n").split(":", 1) - if len(kv) != 2: - # Ignore lines without colon. - continue - k, v = kv - props.update({k.replace(" ", "_").lower(): v.strip()}) - return props - - @cached_property - def _uname_info(self): - # type: () -> Dict[str, str] - with open(os.devnull, "wb") as devnull: - try: - cmd = ("uname", "-rs") - stdout = subprocess.check_output(cmd, stderr=devnull) - except OSError: - return {} - content = self._to_str(stdout).splitlines() - return self._parse_uname_content(content) - - @staticmethod - def _parse_uname_content(lines): - # type: (Sequence[str]) -> Dict[str, str] - props = {} - match = re.search(r"^([^\s]+)\s+([\d\.]+)", lines[0].strip()) - if match: - name, version = match.groups() - - # This is to prevent the Linux kernel version from - # appearing as the 'best' version on otherwise - # identifiable distributions. - if name == "Linux": - return {} - props["id"] = name.lower() - props["name"] = name - props["release"] = version - return props - - @staticmethod - def _to_str(text): - # type: (Union[bytes, str]) -> str - encoding = sys.getfilesystemencoding() - encoding = "utf-8" if encoding == "ascii" else encoding - - if sys.version_info[0] >= 3: - if isinstance(text, bytes): - return text.decode(encoding) - else: - if isinstance(text, unicode): # noqa - return text.encode(encoding) - - return text - - @cached_property - def _distro_release_info(self): - # type: () -> Dict[str, str] - """ - Get the information items from the specified distro release file. - - Returns: - A dictionary containing all information items. - """ - if self.distro_release_file: - # If it was specified, we use it and parse what we can, even if - # its file name or content does not match the expected pattern. - distro_info = self._parse_distro_release_file(self.distro_release_file) - basename = os.path.basename(self.distro_release_file) - # The file name pattern for user-specified distro release files - # is somewhat more tolerant (compared to when searching for the - # file), because we want to use what was specified as best as - # possible. - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - if "name" in distro_info and "cloudlinux" in distro_info["name"].lower(): - distro_info["id"] = "cloudlinux" - elif match: - distro_info["id"] = match.group(1) - return distro_info - else: - try: - basenames = os.listdir(self.etc_dir) - # We sort for repeatability in cases where there are multiple - # distro specific files; e.g. CentOS, Oracle, Enterprise all - # containing `redhat-release` on top of their own. - basenames.sort() - except OSError: - # This may occur when /etc is not readable but we can't be - # sure about the *-release files. Check common entries of - # /etc for information. If they turn out to not be there the - # error is handled in `_parse_distro_release_file()`. - basenames = [ - "SuSE-release", - "arch-release", - "base-release", - "centos-release", - "fedora-release", - "gentoo-release", - "mageia-release", - "mandrake-release", - "mandriva-release", - "mandrivalinux-release", - "manjaro-release", - "oracle-release", - "redhat-release", - "sl-release", - "slackware-version", - ] - for basename in basenames: - if basename in _DISTRO_RELEASE_IGNORE_BASENAMES: - continue - match = _DISTRO_RELEASE_BASENAME_PATTERN.match(basename) - if match: - filepath = os.path.join(self.etc_dir, basename) - distro_info = self._parse_distro_release_file(filepath) - if "name" in distro_info: - # The name is always present if the pattern matches - self.distro_release_file = filepath - distro_info["id"] = match.group(1) - if "cloudlinux" in distro_info["name"].lower(): - distro_info["id"] = "cloudlinux" - return distro_info - return {} - - def _parse_distro_release_file(self, filepath): - # type: (str) -> Dict[str, str] - """ - Parse a distro release file. - - Parameters: - - * filepath: Path name of the distro release file. - - Returns: - A dictionary containing all information items. - """ - try: - with open(filepath) as fp: - # Only parse the first line. For instance, on SLES there - # are multiple lines. We don't want them... - return self._parse_distro_release_content(fp.readline()) - except (OSError, IOError): - # Ignore not being able to read a specific, seemingly version - # related file. - # See https://github.com/python-distro/distro/issues/162 - return {} - - @staticmethod - def _parse_distro_release_content(line): - # type: (str) -> Dict[str, str] - """ - Parse a line from a distro release file. - - Parameters: - * line: Line from the distro release file. Must be a unicode string - or a UTF-8 encoded byte string. - - Returns: - A dictionary containing all information items. - """ - matches = _DISTRO_RELEASE_CONTENT_REVERSED_PATTERN.match(line.strip()[::-1]) - distro_info = {} - if matches: - # regexp ensures non-None - distro_info["name"] = matches.group(3)[::-1] - if matches.group(2): - distro_info["version_id"] = matches.group(2)[::-1] - if matches.group(1): - distro_info["codename"] = matches.group(1)[::-1] - elif line: - distro_info["name"] = line.strip() - return distro_info - - -_distro = LinuxDistribution() - - -def main(): - # type: () -> None - logger = logging.getLogger(__name__) - logger.setLevel(logging.DEBUG) - logger.addHandler(logging.StreamHandler(sys.stdout)) - - parser = argparse.ArgumentParser(description="OS distro info tool") - parser.add_argument( - "--json", "-j", help="Output in machine readable format", action="store_true" - ) - - parser.add_argument( - "--root-dir", - "-r", - type=str, - dest="root_dir", - help="Path to the root filesystem directory (defaults to /)", - ) - - args = parser.parse_args() - - if args.root_dir: - dist = LinuxDistribution( - include_lsb=False, include_uname=False, root_dir=args.root_dir - ) - else: - dist = _distro - - if args.json: - logger.info(json.dumps(dist.info(), indent=4, sort_keys=True)) - else: - logger.info("Name: %s", dist.name(pretty=True)) - distribution_version = dist.version(pretty=True) - logger.info("Version: %s", distribution_version) - distribution_codename = dist.codename() - logger.info("Codename: %s", distribution_codename) - - -if __name__ == "__main__": - main() diff --git a/contrib/python/distro/py2/ya.make b/contrib/python/distro/py2/ya.make deleted file mode 100644 index e2717536110..00000000000 --- a/contrib/python/distro/py2/ya.make +++ /dev/null @@ -1,31 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(1.6.0) - -LICENSE(Apache-2.0) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - distro.py -) - -RESOURCE_FILES( - PREFIX contrib/python/distro/py2/ - .dist-info/METADATA - .dist-info/entry_points.txt - .dist-info/top_level.txt -) - -END() - -RECURSE( - bin -) - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/distro/py3/LICENSE b/contrib/python/distro/py3/LICENSE deleted file mode 100644 index e06d2081865..00000000000 --- a/contrib/python/distro/py3/LICENSE +++ /dev/null @@ -1,202 +0,0 @@ -Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - diff --git a/contrib/python/distro/py3/README.md b/contrib/python/distro/py3/README.md deleted file mode 100644 index de6f3c79d38..00000000000 --- a/contrib/python/distro/py3/README.md +++ /dev/null @@ -1,152 +0,0 @@ -Distro - an OS platform information API -======================================= - -[](https://github.com/python-distro/distro/actions/workflows/ci.yaml) -[](https://pypi.python.org/pypi/distro) -[](https://img.shields.io/pypi/pyversions/distro.svg) -[](https://codecov.io/github/python-distro/distro?branch=master) -[](https://pypi.python.org/pypi/distro) -[](http://distro.readthedocs.io/en/latest/) -[](https://gitter.im/python-distro/distro?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -`distro` provides information about the -OS distribution it runs on, such as a reliable machine-readable ID, or -version information. - -It is the recommended replacement for Python's original -[`platform.linux_distribution`](https://docs.python.org/3.7/library/platform.html#platform.linux_distribution) -function (removed in Python 3.8). It also provides much more functionality -which isn't necessarily Python bound, like a command-line interface. - -Distro currently supports Linux and BSD based systems but [Windows and OS X support](https://github.com/python-distro/distro/issues/177) is also planned. - -For Python 2.6 support, see https://github.com/python-distro/distro/tree/python2.6-support - -## Installation - -Installation of the latest released version from PyPI: - -```shell -pip install distro -``` - -Installation of the latest development version: - -```shell -pip install https://github.com/python-distro/distro/archive/master.tar.gz -``` - -To use as a standalone script, download `distro.py` directly: - -```shell -curl -O https://raw.githubusercontent.com/python-distro/distro/master/src/distro/distro.py -python distro.py -``` - -``distro`` is safe to vendor within projects that do not wish to add -dependencies. - -```shell -cd myproject -curl -O https://raw.githubusercontent.com/python-distro/distro/master/src/distro/distro.py -``` - -## Usage - -```bash -$ distro -Name: Antergos Linux -Version: 2015.10 (ISO-Rolling) -Codename: ISO-Rolling - -$ distro -j -{ - "codename": "ISO-Rolling", - "id": "antergos", - "like": "arch", - "version": "16.9", - "version_parts": { - "build_number": "", - "major": "16", - "minor": "9" - } -} - - -$ python ->>> import distro ->>> distro.name(pretty=True) -'CentOS Linux 8' ->>> distro.id() -'centos' ->>> distro.version(best=True) -'8.4.2105' -``` - - -## Documentation - -On top of the aforementioned API, several more functions are available. For a complete description of the -API, see the [latest API documentation](http://distro.readthedocs.org/en/latest/). - -## Background - -An alternative implementation became necessary because Python 3.5 deprecated -this function, and Python 3.8 removed it altogether. Its predecessor function -[`platform.dist`](https://docs.python.org/3.7/library/platform.html#platform.dist) -was already deprecated since Python 2.6 and removed in Python 3.8. Still, there -are many cases in which access to that information is needed. See [Python issue -1322](https://bugs.python.org/issue1322) for more information. - -The `distro` package implements a robust and inclusive way of retrieving the -information about a distribution based on new standards and old methods, -namely from these data sources (from high to low precedence): - -* The os-release file `/etc/os-release` if present, with a fall-back on `/usr/lib/os-release` if needed. -* The output of the `lsb_release` command, if available. -* The distro release file (`/etc/*(-|_)(release|version)`), if present. -* The `uname` command for BSD based distrubtions. - - -## Python and Distribution Support - -`distro` is supported and tested on Python 3.6+ and PyPy and on any -distribution that provides one or more of the data sources covered. - -This package is tested with test data that mimics the exact behavior of the data sources of [a number of Linux distributions](https://github.com/python-distro/distro/tree/master/tests/resources/distros). - - -## Testing - -```shell -git clone git@github.com:python-distro/distro.git -cd distro -pip install tox -tox -``` - - -## Contributions - -Pull requests are always welcome to deal with specific distributions or just -for general merriment. - -See [CONTRIBUTIONS](https://github.com/python-distro/distro/blob/master/CONTRIBUTING.md) for contribution info. - -Reference implementations for supporting additional distributions and file -formats can be found here: - -* https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L1172 -* https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb -* https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/distribution.py -* https://github.com/puppetlabs/facter/blob/master/lib/src/facts/linux/os_linux.cc - -## Package manager distributions - -* https://src.fedoraproject.org/rpms/python-distro -* https://www.archlinux.org/packages/community/any/python-distro/ -* https://launchpad.net/ubuntu/+source/python-distro -* https://packages.debian.org/stable/python3-distro -* https://packages.gentoo.org/packages/dev-python/distro -* https://pkgs.org/download/python3-distro -* https://slackbuilds.org/repository/14.2/python/python-distro/ diff --git a/contrib/python/distro/ya.make b/contrib/python/distro/ya.make deleted file mode 100644 index 82a91a17a29..00000000000 --- a/contrib/python/distro/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/distro/py2) -ELSE() - PEERDIR(contrib/python/distro/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/portalocker/py2/.dist-info/METADATA b/contrib/python/portalocker/py2/.dist-info/METADATA deleted file mode 100644 index d01a6203e0a..00000000000 --- a/contrib/python/portalocker/py2/.dist-info/METADATA +++ /dev/null @@ -1,136 +0,0 @@ -Metadata-Version: 2.1 -Name: portalocker -Version: 1.7.1 -Summary: Wraps the portalocker recipe for easy usage -Home-page: https://github.com/WoLpH/portalocker -Author: Rick van Hattem -Author-email: wolph@wol.ph -License: PSF -Keywords: locking,locks,with statement,windows,linux,unix -Platform: any -Classifier: Intended Audience :: Developers -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3.3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Requires-Dist: pywin32 (!=226) ; platform_system == "Windows" -Provides-Extra: docs -Requires-Dist: sphinx (>=1.7.1) ; extra == 'docs' -Provides-Extra: tests -Requires-Dist: pytest (>=4.6.9) ; extra == 'tests' -Requires-Dist: pytest-cov (>=2.8.1) ; extra == 'tests' -Requires-Dist: sphinx (>=1.8.5) ; extra == 'tests' -Requires-Dist: pytest-flake8 (>=1.0.5) ; extra == 'tests' - -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://travis-ci.org/WoLpH/portalocker.svg?branch=master - :alt: Linux Test Status - :target: https://travis-ci.org/WoLpH/portalocker - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: - print >>fh, 'writing some stuff to my cache...' - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LOCK_EX) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking might not be needed in all cases: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -But can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - -Changelog ---------- - -See the `changelog <http://portalocker.readthedocs.io/en/latest/changelog.html>`_ page. - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - - - diff --git a/contrib/python/portalocker/py2/.dist-info/top_level.txt b/contrib/python/portalocker/py2/.dist-info/top_level.txt deleted file mode 100644 index 7bbc14e6fa6..00000000000 --- a/contrib/python/portalocker/py2/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -portalocker diff --git a/contrib/python/portalocker/py2/LICENSE b/contrib/python/portalocker/py2/LICENSE deleted file mode 100644 index adb8038169c..00000000000 --- a/contrib/python/portalocker/py2/LICENSE +++ /dev/null @@ -1,48 +0,0 @@ -PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 --------------------------------------------- - -1. This LICENSE AGREEMENT is between the Python Software Foundation -("PSF"), and the Individual or Organization ("Licensee") accessing and -otherwise using this software ("Python") in source or binary form and -its associated documentation. - -2. Subject to the terms and conditions of this License Agreement, PSF hereby -grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, -analyze, test, perform and/or display publicly, prepare derivative works, -distribute, and otherwise use Python alone or in any derivative version, -provided, however, that PSF's License Agreement and PSF's notice of copyright, -i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 -Python Software Foundation; All Rights Reserved" are retained in Python alone or -in any derivative version prepared by Licensee. - -3. In the event Licensee prepares a derivative work that is based on -or incorporates Python or any part thereof, and wants to make -the derivative work available to others as provided herein, then -Licensee hereby agrees to include in any such work a brief summary of -the changes made to Python. - -4. PSF is making Python available to Licensee on an "AS IS" -basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR -IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND -DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS -FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT -INFRINGE ANY THIRD PARTY RIGHTS. - -5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON -FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS -A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, -OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. - -6. This License Agreement will automatically terminate upon a material -breach of its terms and conditions. - -7. Nothing in this License Agreement shall be deemed to create any -relationship of agency, partnership, or joint venture between PSF and -Licensee. This License Agreement does not grant permission to use PSF -trademarks or trade name in a trademark sense to endorse or promote -products or services of Licensee, or any third party. - -8. By copying, installing or otherwise using Python, Licensee -agrees to be bound by the terms and conditions of this License -Agreement. - diff --git a/contrib/python/portalocker/py2/README.rst b/contrib/python/portalocker/py2/README.rst deleted file mode 100644 index c013490c765..00000000000 --- a/contrib/python/portalocker/py2/README.rst +++ /dev/null @@ -1,106 +0,0 @@ -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://travis-ci.org/WoLpH/portalocker.svg?branch=master - :alt: Linux Test Status - :target: https://travis-ci.org/WoLpH/portalocker - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: - print >>fh, 'writing some stuff to my cache...' - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LOCK_EX) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking might not be needed in all cases: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -But can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - -Changelog ---------- - -See the `changelog <http://portalocker.readthedocs.io/en/latest/changelog.html>`_ page. - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - diff --git a/contrib/python/portalocker/py2/portalocker/__about__.py b/contrib/python/portalocker/py2/portalocker/__about__.py deleted file mode 100644 index f16fe0cdf7d..00000000000 --- a/contrib/python/portalocker/py2/portalocker/__about__.py +++ /dev/null @@ -1,7 +0,0 @@ -__package_name__ = 'portalocker' -__author__ = 'Rick van Hattem' -__email__ = 'wolph@wol.ph' -__version__ = '1.7.1' -__description__ = '''Wraps the portalocker recipe for easy usage''' -__url__ = 'https://github.com/WoLpH/portalocker' - diff --git a/contrib/python/portalocker/py2/portalocker/__init__.py b/contrib/python/portalocker/py2/portalocker/__init__.py deleted file mode 100644 index 9bf27fee0fc..00000000000 --- a/contrib/python/portalocker/py2/portalocker/__init__.py +++ /dev/null @@ -1,67 +0,0 @@ -from . import __about__ -from . import constants -from . import exceptions -from . import portalocker -from . import utils - -#: The package name on Pypi -__package_name__ = __about__.__package_name__ -#: Current author and maintainer, view the git history for the previous ones -__author__ = __about__.__author__ -#: Current author's email address -__email__ = __about__.__email__ -#: Version number -__version__ = '1.7.1' -#: Package description for Pypi -__description__ = __about__.__description__ -#: Package homepage -__url__ = __about__.__url__ - - -#: Exception thrown when the file is already locked by someone else -AlreadyLocked = exceptions.AlreadyLocked -#: Exception thrown if an error occurred during locking -LockException = exceptions.LockException - - -#: Lock a file. Note that this is an advisory lock on Linux/Unix systems -lock = portalocker.lock -#: Unlock a file -unlock = portalocker.unlock - -#: Place an exclusive lock. -#: Only one process may hold an exclusive lock for a given file at a given -#: time. -LOCK_EX = constants.LOCK_EX - -#: Place a shared lock. -#: More than one process may hold a shared lock for a given file at a given -#: time. -LOCK_SH = constants.LOCK_SH - -#: Acquire the lock in a non-blocking fashion. -LOCK_NB = constants.LOCK_NB - -#: Remove an existing lock held by this process. -LOCK_UN = constants.LOCK_UN - -#: Locking utility class to automatically handle opening with timeouts and -#: context wrappers -Lock = utils.Lock -RLock = utils.RLock -TemporaryFileLock = utils.TemporaryFileLock -open_atomic = utils.open_atomic - -__all__ = [ - 'lock', - 'unlock', - 'LOCK_EX', - 'LOCK_SH', - 'LOCK_NB', - 'LOCK_UN', - 'LockException', - 'Lock', - 'AlreadyLocked', - 'open_atomic', -] - diff --git a/contrib/python/portalocker/py2/portalocker/constants.py b/contrib/python/portalocker/py2/portalocker/constants.py deleted file mode 100644 index fb0927e2da9..00000000000 --- a/contrib/python/portalocker/py2/portalocker/constants.py +++ /dev/null @@ -1,39 +0,0 @@ -''' -Locking constants - -Lock types: - -- `LOCK_EX` exclusive lock -- `LOCK_SH` shared lock - -Lock flags: - -- `LOCK_NB` non-blocking - -Manually unlock, only needed internally - -- `LOCK_UN` unlock -''' -import os - -# The actual tests will execute the code anyhow so the following code can -# safely be ignored from the coverage tests -if os.name == 'nt': # pragma: no cover - import msvcrt - - LOCK_EX = 0x1 #: exclusive lock - LOCK_SH = 0x2 #: shared lock - LOCK_NB = 0x4 #: non-blocking - LOCK_UN = msvcrt.LK_UNLCK #: unlock - -elif os.name == 'posix': # pragma: no cover - import fcntl - - LOCK_EX = fcntl.LOCK_EX #: exclusive lock - LOCK_SH = fcntl.LOCK_SH #: shared lock - LOCK_NB = fcntl.LOCK_NB #: non-blocking - LOCK_UN = fcntl.LOCK_UN #: unlock - -else: # pragma: no cover - raise RuntimeError('PortaLocker only defined for nt and posix platforms') - diff --git a/contrib/python/portalocker/py2/portalocker/exceptions.py b/contrib/python/portalocker/py2/portalocker/exceptions.py deleted file mode 100644 index bb2b35eb7bc..00000000000 --- a/contrib/python/portalocker/py2/portalocker/exceptions.py +++ /dev/null @@ -1,19 +0,0 @@ -class BaseLockException(Exception): - # Error codes: - LOCK_FAILED = 1 - - def __init__(self, *args, **kwargs): - self.fh = kwargs.pop('fh', None) - Exception.__init__(self, *args, **kwargs) - - -class LockException(BaseLockException): - pass - - -class AlreadyLocked(BaseLockException): - pass - - -class FileToLarge(BaseLockException): - pass diff --git a/contrib/python/portalocker/py2/portalocker/portalocker.py b/contrib/python/portalocker/py2/portalocker/portalocker.py deleted file mode 100644 index 460cf06e476..00000000000 --- a/contrib/python/portalocker/py2/portalocker/portalocker.py +++ /dev/null @@ -1,148 +0,0 @@ -import os -import sys -from . import exceptions -from . import constants - - -if os.name == 'nt': # pragma: no cover - import win32con - import win32file - import pywintypes - import winerror - import msvcrt - __overlapped = pywintypes.OVERLAPPED() - - if sys.version_info.major == 2: - lock_length = -1 - else: - lock_length = int(2**31 - 1) - - def lock(file_, flags): - if flags & constants.LOCK_SH: - if sys.version_info.major == 2: - if flags & constants.LOCK_NB: - mode = win32con.LOCKFILE_FAIL_IMMEDIATELY - else: - mode = 0 - - else: - if flags & constants.LOCK_NB: - mode = msvcrt.LK_NBRLCK - else: - mode = msvcrt.LK_RLCK - - # is there any reason not to reuse the following structure? - hfile = win32file._get_osfhandle(file_.fileno()) - try: - win32file.LockFileEx(hfile, mode, 0, -0x10000, __overlapped) - except pywintypes.error as exc_value: - # error: (33, 'LockFileEx', 'The process cannot access the file - # because another process has locked a portion of the file.') - if exc_value.winerror == winerror.ERROR_LOCK_VIOLATION: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_) - else: - # Q: Are there exceptions/codes we should be dealing with - # here? - raise - else: - mode = win32con.LOCKFILE_EXCLUSIVE_LOCK - if flags & constants.LOCK_NB: - mode |= win32con.LOCKFILE_FAIL_IMMEDIATELY - - if flags & constants.LOCK_NB: - mode = msvcrt.LK_NBLCK - else: - mode = msvcrt.LK_LOCK - - # windows locks byte ranges, so make sure to lock from file start - try: - savepos = file_.tell() - if savepos: - # [ ] test exclusive lock fails on seek here - # [ ] test if shared lock passes this point - file_.seek(0) - # [x] check if 0 param locks entire file (not documented in - # Python) - # [x] fails with "IOError: [Errno 13] Permission denied", - # but -1 seems to do the trick - - try: - msvcrt.locking(file_.fileno(), mode, lock_length) - except IOError as exc_value: - # [ ] be more specific here - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_) - finally: - if savepos: - file_.seek(savepos) - except IOError as exc_value: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, exc_value.strerror, - fh=file_) - - def unlock(file_): - try: - savepos = file_.tell() - if savepos: - file_.seek(0) - - try: - msvcrt.locking(file_.fileno(), constants.LOCK_UN, lock_length) - except IOError as exc_value: - if exc_value.strerror == 'Permission denied': - hfile = win32file._get_osfhandle(file_.fileno()) - try: - win32file.UnlockFileEx( - hfile, 0, -0x10000, __overlapped) - except pywintypes.error as exc_value: - if exc_value.winerror == winerror.ERROR_NOT_LOCKED: - # error: (158, 'UnlockFileEx', - # 'The segment is already unlocked.') - # To match the 'posix' implementation, silently - # ignore this error - pass - else: - # Q: Are there exceptions/codes we should be - # dealing with here? - raise - else: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, - exc_value.strerror, - fh=file_) - finally: - if savepos: - file_.seek(savepos) - except IOError as exc_value: - raise exceptions.LockException( - exceptions.LockException.LOCK_FAILED, exc_value.strerror, - fh=file_) - -elif os.name == 'posix': # pragma: no cover - import fcntl - - def lock(file_, flags): - locking_exceptions = IOError, - try: # pragma: no cover - locking_exceptions += BlockingIOError, - except NameError: # pragma: no cover - pass - - try: - fcntl.flock(file_.fileno(), flags) - except locking_exceptions as exc_value: - # The exception code varies on different systems so we'll catch - # every IO error - raise exceptions.LockException(exc_value, fh=file_) - - def unlock(file_): - fcntl.flock(file_.fileno(), constants.LOCK_UN) - -else: # pragma: no cover - raise RuntimeError('PortaLocker only defined for nt and posix platforms') - diff --git a/contrib/python/portalocker/py2/portalocker/utils.py b/contrib/python/portalocker/py2/portalocker/utils.py deleted file mode 100644 index 8baebc2b200..00000000000 --- a/contrib/python/portalocker/py2/portalocker/utils.py +++ /dev/null @@ -1,256 +0,0 @@ -import os -import time -import atexit -import tempfile -import contextlib -from . import exceptions -from . import constants -from . import portalocker - -current_time = getattr(time, "monotonic", time.time) - -DEFAULT_TIMEOUT = 5 -DEFAULT_CHECK_INTERVAL = 0.25 -LOCK_METHOD = constants.LOCK_EX | constants.LOCK_NB - -__all__ = [ - 'Lock', - 'open_atomic', -] - - -@contextlib.contextmanager -def open_atomic(filename, binary=True): - '''Open a file for atomic writing. Instead of locking this method allows - you to write the entire file and move it to the actual location. Note that - this makes the assumption that a rename is atomic on your platform which - is generally the case but not a guarantee. - - http://docs.python.org/library/os.html#os.rename - - >>> filename = 'test_file.txt' - >>> if os.path.exists(filename): - ... os.remove(filename) - - >>> with open_atomic(filename) as fh: - ... written = fh.write(b'test') - >>> assert os.path.exists(filename) - >>> os.remove(filename) - - ''' - assert not os.path.exists(filename), '%r exists' % filename - path, name = os.path.split(filename) - - # Create the parent directory if it doesn't exist - if path and not os.path.isdir(path): # pragma: no cover - os.makedirs(path) - - temp_fh = tempfile.NamedTemporaryFile( - mode=binary and 'wb' or 'w', - dir=path, - delete=False, - ) - yield temp_fh - temp_fh.flush() - os.fsync(temp_fh.fileno()) - temp_fh.close() - try: - os.rename(temp_fh.name, filename) - finally: - try: - os.remove(temp_fh.name) - except Exception: - pass - - -class Lock(object): - - def __init__( - self, filename, mode='a', timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=False, - flags=LOCK_METHOD, **file_open_kwargs): - '''Lock manager with build-in timeout - - filename -- filename - mode -- the open mode, 'a' or 'ab' should be used for writing - truncate -- use truncate to emulate 'w' mode, None is disabled, 0 is - truncate to 0 bytes - timeout -- timeout when trying to acquire a lock - check_interval -- check interval while waiting - fail_when_locked -- after the initial lock failed, return an error - or lock the file - **file_open_kwargs -- The kwargs for the `open(...)` call - - fail_when_locked is useful when multiple threads/processes can race - when creating a file. If set to true than the system will wait till - the lock was acquired and then return an AlreadyLocked exception. - - Note that the file is opened first and locked later. So using 'w' as - mode will result in truncate _BEFORE_ the lock is checked. - ''' - - if 'w' in mode: - truncate = True - mode = mode.replace('w', 'a') - else: - truncate = False - - self.fh = None - self.filename = filename - self.mode = mode - self.truncate = truncate - self.timeout = timeout - self.check_interval = check_interval - self.fail_when_locked = fail_when_locked - self.flags = flags - self.file_open_kwargs = file_open_kwargs - - def acquire( - self, timeout=None, check_interval=None, fail_when_locked=None): - '''Acquire the locked filehandle''' - if timeout is None: - timeout = self.timeout - if timeout is None: - timeout = 0 - - if check_interval is None: - check_interval = self.check_interval - - if fail_when_locked is None: - fail_when_locked = self.fail_when_locked - - # If we already have a filehandle, return it - fh = self.fh - if fh: - return fh - - # Get a new filehandler - fh = self._get_fh() - try: - # Try to lock - fh = self._get_lock(fh) - except exceptions.LockException as exception: - # Try till the timeout has passed - timeoutend = current_time() + timeout - while timeoutend > current_time(): - # Wait a bit - time.sleep(check_interval) - - # Try again - try: - - # We already tried to the get the lock - # If fail_when_locked is true, then stop trying - if fail_when_locked: - raise exceptions.AlreadyLocked(exception) - - else: # pragma: no cover - # We've got the lock - fh = self._get_lock(fh) - break - - except exceptions.LockException: - pass - - else: - fh.close() - # We got a timeout... reraising - raise exceptions.LockException(exception) - - # Prepare the filehandle (truncate if needed) - fh = self._prepare_fh(fh) - - self.fh = fh - return fh - - def release(self): - '''Releases the currently locked file handle''' - if self.fh: - portalocker.unlock(self.fh) - self.fh.close() - self.fh = None - - def _get_fh(self): - '''Get a new filehandle''' - return open(self.filename, self.mode, **self.file_open_kwargs) - - def _get_lock(self, fh): - ''' - Try to lock the given filehandle - - returns LockException if it fails''' - portalocker.lock(fh, self.flags) - return fh - - def _prepare_fh(self, fh): - ''' - Prepare the filehandle for usage - - If truncate is a number, the file will be truncated to that amount of - bytes - ''' - if self.truncate: - fh.seek(0) - fh.truncate(0) - - return fh - - def __enter__(self): - return self.acquire() - - def __exit__(self, type_, value, tb): - self.release() - - def __delete__(self, instance): # pragma: no cover - instance.release() - - -class RLock(Lock): - """ - A reentrant lock, functions in a similar way to threading.RLock in that it - can be acquired multiple times. When the corresponding number of release() - calls are made the lock will finally release the underlying file lock. - """ - def __init__( - self, filename, mode='a', timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=False, - flags=LOCK_METHOD): - super(RLock, self).__init__(filename, mode, timeout, check_interval, - fail_when_locked, flags) - self._acquire_count = 0 - - def acquire( - self, timeout=None, check_interval=None, fail_when_locked=None): - if self._acquire_count >= 1: - fh = self.fh - else: - fh = super(RLock, self).acquire(timeout, check_interval, - fail_when_locked) - self._acquire_count += 1 - return fh - - def release(self): - if self._acquire_count == 0: - raise exceptions.LockException( - "Cannot release more times than acquired") - - if self._acquire_count == 1: - super(RLock, self).release() - self._acquire_count -= 1 - - -class TemporaryFileLock(Lock): - - def __init__(self, filename='.lock', timeout=DEFAULT_TIMEOUT, - check_interval=DEFAULT_CHECK_INTERVAL, fail_when_locked=True, - flags=LOCK_METHOD): - - Lock.__init__(self, filename=filename, mode='w', timeout=timeout, - check_interval=check_interval, - fail_when_locked=fail_when_locked, flags=flags) - atexit.register(self.release) - - def release(self): - Lock.release(self) - if os.path.isfile(self.filename): # pragma: no branch - os.unlink(self.filename) diff --git a/contrib/python/portalocker/py2/ya.make b/contrib/python/portalocker/py2/ya.make deleted file mode 100644 index 0cf65137041..00000000000 --- a/contrib/python/portalocker/py2/ya.make +++ /dev/null @@ -1,31 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(1.7.1) - -LICENSE(PSF-2.0) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - portalocker/__about__.py - portalocker/__init__.py - portalocker/constants.py - portalocker/exceptions.py - portalocker/portalocker.py - portalocker/utils.py -) - -RESOURCE_FILES( - PREFIX contrib/python/portalocker/py2/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() - -RECURSE( - tests -) diff --git a/contrib/python/portalocker/py3/LICENSE b/contrib/python/portalocker/py3/LICENSE deleted file mode 100644 index b638bda0d3b..00000000000 --- a/contrib/python/portalocker/py3/LICENSE +++ /dev/null @@ -1,11 +0,0 @@ -Copyright 2022 Rick van Hattem - -Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - -1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - -2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - -3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/contrib/python/portalocker/py3/README.rst b/contrib/python/portalocker/py3/README.rst deleted file mode 100644 index c5ef42f614b..00000000000 --- a/contrib/python/portalocker/py3/README.rst +++ /dev/null @@ -1,193 +0,0 @@ -############################################ -portalocker - Cross-platform locking library -############################################ - -.. image:: https://github.com/WoLpH/portalocker/actions/workflows/python-package.yml/badge.svg?branch=master - :alt: Linux Test Status - :target: https://github.com/WoLpH/portalocker/actions/ - -.. image:: https://ci.appveyor.com/api/projects/status/mgqry98hgpy4prhh?svg=true - :alt: Windows Tests Status - :target: https://ci.appveyor.com/project/WoLpH/portalocker - -.. image:: https://coveralls.io/repos/WoLpH/portalocker/badge.svg?branch=master - :alt: Coverage Status - :target: https://coveralls.io/r/WoLpH/portalocker?branch=master - -Overview --------- - -Portalocker is a library to provide an easy API to file locking. - -An important detail to note is that on Linux and Unix systems the locks are -advisory by default. By specifying the `-o mand` option to the mount command it -is possible to enable mandatory file locking on Linux. This is generally not -recommended however. For more information about the subject: - - - https://en.wikipedia.org/wiki/File_locking - - http://stackoverflow.com/questions/39292051/portalocker-does-not-seem-to-lock - - https://stackoverflow.com/questions/12062466/mandatory-file-lock-on-linux - -The module is currently maintained by Rick van Hattem <Wolph@wol.ph>. -The project resides at https://github.com/WoLpH/portalocker . Bugs and feature -requests can be submitted there. Patches are also very welcome. - -Security contact information ------------------------------------------------------------------------------- - -To report a security vulnerability, please use the -`Tidelift security contact <https://tidelift.com/security>`_. -Tidelift will coordinate the fix and disclosure. - -Redis Locks ------------ - -This library now features a lock based on Redis which allows for locks across -multiple threads, processes and even distributed locks across multiple -computers. - -It is an extremely reliable Redis lock that is based on pubsub. - -As opposed to most Redis locking systems based on key/value pairs, -this locking method is based on the pubsub system. The big advantage is -that if the connection gets killed due to network issues, crashing -processes or otherwise, it will still immediately unlock instead of -waiting for a lock timeout. - -First make sure you have everything installed correctly: - -:: - - pip install "portalocker[redis]" - -Usage is really easy: - -:: - - import portalocker - - lock = portalocker.RedisLock('some_lock_channel_name') - - with lock: - print('do something here') - -The API is essentially identical to the other ``Lock`` classes so in addition -to the ``with`` statement you can also use ``lock.acquire(...)``. - -Python 2 --------- - -Python 2 was supported in versions before Portalocker 2.0. If you are still -using -Python 2, -you can run this to install: - -:: - - pip install "portalocker<2" - -Tips ----- - -On some networked filesystems it might be needed to force a `os.fsync()` before -closing the file so it's actually written before another client reads the file. -Effectively this comes down to: - -:: - - with portalocker.Lock('some_file', 'rb+', timeout=60) as fh: - # do what you need to do - ... - - # flush and sync to filesystem - fh.flush() - os.fsync(fh.fileno()) - -Links ------ - -* Documentation - - http://portalocker.readthedocs.org/en/latest/ -* Source - - https://github.com/WoLpH/portalocker -* Bug reports - - https://github.com/WoLpH/portalocker/issues -* Package homepage - - https://pypi.python.org/pypi/portalocker -* My blog - - http://w.wol.ph/ - -Examples --------- - -To make sure your cache generation scripts don't race, use the `Lock` class: - ->>> import portalocker ->>> with portalocker.Lock('somefile', timeout=1) as fh: -... print('writing some stuff to my cache...', file=fh) - -To customize the opening and locking a manual approach is also possible: - ->>> import portalocker ->>> file = open('somefile', 'r+') ->>> portalocker.lock(file, portalocker.LockFlags.EXCLUSIVE) ->>> file.seek(12) ->>> file.write('foo') ->>> file.close() - -Explicitly unlocking is not needed in most cases but omitting it has been known -to cause issues: -https://github.com/AzureAD/microsoft-authentication-extensions-for-python/issues/42#issuecomment-601108266 - -If needed, it can be done through: - ->>> portalocker.unlock(file) - -Do note that your data might still be in a buffer so it is possible that your -data is not available until you `flush()` or `close()`. - -To create a cross platform bounded semaphore across multiple processes you can -use the `BoundedSemaphore` class which functions somewhat similar to -`threading.BoundedSemaphore`: - ->>> import portalocker ->>> n = 2 ->>> timeout = 0.1 - ->>> semaphore_a = portalocker.BoundedSemaphore(n, timeout=timeout) ->>> semaphore_b = portalocker.BoundedSemaphore(n, timeout=timeout) ->>> semaphore_c = portalocker.BoundedSemaphore(n, timeout=timeout) - ->>> semaphore_a.acquire() -<portalocker.utils.Lock object at ...> ->>> semaphore_b.acquire() -<portalocker.utils.Lock object at ...> ->>> semaphore_c.acquire() -Traceback (most recent call last): - ... -portalocker.exceptions.AlreadyLocked - - -More examples can be found in the -`tests <http://portalocker.readthedocs.io/en/latest/_modules/tests/tests.html>`_. - - -Versioning ----------- - -This library follows `Semantic Versioning <http://semver.org/>`_. - - -Changelog ---------- - -Every release has a ``git tag`` with a commit message for the tag -explaining what was added and/or changed. The list of tags/releases -including the commit messages can be found here: -https://github.com/WoLpH/portalocker/releases - -License -------- - -See the `LICENSE <https://github.com/WoLpH/portalocker/blob/develop/LICENSE>`_ file. - diff --git a/contrib/python/portalocker/ya.make b/contrib/python/portalocker/ya.make deleted file mode 100644 index 4012370b1cc..00000000000 --- a/contrib/python/portalocker/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/portalocker/py2) -ELSE() - PEERDIR(contrib/python/portalocker/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/python-libarchive/libarchive/__init__.py b/contrib/python/python-libarchive/libarchive/__init__.py deleted file mode 100644 index 0c0c63359a3..00000000000 --- a/contrib/python/python-libarchive/libarchive/__init__.py +++ /dev/null @@ -1,800 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import os -import stat -import sys -import math -import time -import logging -import warnings - -import contextlib2 - -from libarchive import _libarchive -import six - -logger = logging.getLogger(__name__) - -# Suggested block size for libarchive. Libarchive may adjust it. -BLOCK_SIZE = 10240 - -MTIME_FORMAT = '' - -# Default encoding scheme. -ENCODING = 'utf-8' - -if six.PY2: - def encode(value, encoding): - if type(value) == str: - value = value.decode(encoding, errors='ignore') - return value.encode(encoding) -else: - def encode(value, encoding): - return value.encode(encoding) - - -# Functions to initialize read/write for various libarchive supported formats and filters. -FORMATS = { - None: (_libarchive.archive_read_support_format_all, None), - 'tar': (_libarchive.archive_read_support_format_tar, _libarchive.archive_write_set_format_ustar), - 'pax': (_libarchive.archive_read_support_format_tar, _libarchive.archive_write_set_format_pax), - 'gnu': (_libarchive.archive_read_support_format_gnutar, _libarchive.archive_write_set_format_gnutar), - 'zip': (_libarchive.archive_read_support_format_zip, _libarchive.archive_write_set_format_zip), - 'rar': (_libarchive.archive_read_support_format_rar, None), - '7zip': (_libarchive.archive_read_support_format_7zip, None), - 'ar': (_libarchive.archive_read_support_format_ar, None), - 'cab': (_libarchive.archive_read_support_format_cab, None), - 'cpio': (_libarchive.archive_read_support_format_cpio, _libarchive.archive_write_set_format_cpio_newc), - 'iso': (_libarchive.archive_read_support_format_iso9660, _libarchive.archive_write_set_format_iso9660), - 'lha': (_libarchive.archive_read_support_format_lha, None), - 'xar': (_libarchive.archive_read_support_format_xar, _libarchive.archive_write_set_format_xar), -} - -FILTERS = { - None: (_libarchive.archive_read_support_filter_all, _libarchive.archive_write_add_filter_none), - 'bzip2': (_libarchive.archive_read_support_filter_bzip2, _libarchive.archive_write_add_filter_bzip2), - 'gzip': (_libarchive.archive_read_support_filter_gzip, _libarchive.archive_write_add_filter_gzip), - 'zstd': (_libarchive.archive_read_support_filter_zstd, _libarchive.archive_write_add_filter_zstd), -} - -# Map file extensions to formats and filters. To support quick detection. -FORMAT_EXTENSIONS = { - '.tar': 'tar', - '.zip': 'zip', - '.rar': 'rar', - '.7z': '7zip', - '.ar': 'ar', - '.cab': 'cab', - '.rpm': 'cpio', - '.cpio': 'cpio', - '.iso': 'iso', - '.lha': 'lha', - '.xar': 'xar', -} -FILTER_EXTENSIONS = { - '.bz2': 'bzip2', - '.gz': 'gzip', - '.zst': 'zstd', -} - - -class EOF(Exception): - '''Raised by ArchiveInfo.from_archive() when unable to read the next - archive header.''' - pass - - -def get_error(archive): - '''Retrieves the last error description for the given archive instance.''' - return _libarchive.archive_error_string(archive) - - -def call_and_check(func, archive, *args): - '''Executes a libarchive function and raises an exception when appropriate.''' - ret = func(*args) - if ret == _libarchive.ARCHIVE_OK: - return - elif ret == _libarchive.ARCHIVE_WARN: - warnings.warn('Warning executing function: %s.' % get_error(archive), RuntimeWarning) - elif ret == _libarchive.ARCHIVE_EOF: - raise EOF() - else: - raise Exception('Fatal error executing function, message is: %s.' % get_error(archive)) - - -def get_func(name, items, index): - item = items.get(name, None) - if item is None: - return None - return item[index] - - -def guess_format(filename): - filename, ext = os.path.splitext(filename) - filter = FILTER_EXTENSIONS.get(ext) - if filter: - filename, ext = os.path.splitext(filename) - format = FORMAT_EXTENSIONS.get(ext) - return format, filter - - -def is_archive_name(filename, formats=None): - '''Quick check to see if the given file has an extension indiciating that it is - an archive. The format parameter can be used to limit what archive format is acceptable. - If omitted, all supported archive formats will be checked. - - This function will return the name of the most likely archive format, None if the file is - unlikely to be an archive.''' - if formats is None: - formats = FORMAT_EXTENSIONS.values() - format, filter = guess_format(filename) - if format in formats: - return format - - -def is_archive(f, formats=(None, ), filters=(None, )): - '''Check to see if the given file is actually an archive. The format parameter - can be used to specify which archive format is acceptable. If ommitted, all supported - archive formats will be checked. It opens the file using libarchive. If no error is - received, the file was successfully detected by the libarchive bidding process. - - This procedure is quite costly, so you should avoid calling it unless you are reasonably - sure that the given file is an archive. In other words, you may wish to filter large - numbers of file names using is_archive_name() before double-checking the positives with - this function. - - This function will return True if the file can be opened as an archive using the given - format(s)/filter(s).''' - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - f = exit_stack.enter_context(open(f, 'rb')) - a = _libarchive.archive_read_new() - for format in formats: - format = get_func(format, FORMATS, 0) - if format is None: - return False - format(a) - for filter in filters: - filter = get_func(filter, FILTERS, 0) - if filter is None: - return False - filter(a) - try: - try: - call_and_check(_libarchive.archive_read_open_fd, a, a, f.fileno(), BLOCK_SIZE) - return True - except: - return False - finally: - _libarchive.archive_read_close(a) - _libarchive.archive_read_free(a) - - -def get_archive_filter_names(filename): - with open(filename, 'rb') as afile: - a = _libarchive.archive_read_new() - try: - format_func = get_func(None, FORMATS, 0) - format_func(a) - filter_func = get_func(None, FILTERS, 0) - filter_func(a) - if _libarchive.archive_read_open_fd(a, afile.fileno(), BLOCK_SIZE) == _libarchive.ARCHIVE_OK: - try: - nfilter = _libarchive.archive_filter_count(a) - return [_libarchive.archive_filter_name(a, i).decode(ENCODING) for i in range(nfilter)] - finally: - _libarchive.archive_read_close(a) - finally: - _libarchive.archive_read_free(a) - return [] - - -class EntryReadStream(object): - '''A file-like object for reading an entry from the archive.''' - def __init__(self, archive, size): - self.archive = archive - self.closed = False - self.size = size - self.bytes = 0 - - def __enter__(self): - return self - - def __exit__(self, *args): - return - - def __iter__(self): - if self.closed: - return - while True: - data = self.read(BLOCK_SIZE) - if not data: - break - yield data - - def __len__(self): - return self.size - - def tell(self): - return self.bytes - - def read(self, bytes=-1): - if self.closed: - return - if self.bytes == self.size: - # EOF already reached. - return - if bytes < 0: - bytes = self.size - self.bytes - elif self.bytes + bytes > self.size: - # Limit read to remaining bytes - bytes = self.size - self.bytes - # Read requested bytes - data = _libarchive.archive_read_data_into_str(self.archive._a, bytes) - self.bytes += len(data) - return data - - def close(self): - if self.closed: - return - # Call archive.close() with _defer True to let it know we have been - # closed and it is now safe to actually close. - self.archive.close(_defer=True) - self.archive = None - self.closed = True - - -class EntryWriteStream(object): - '''A file-like object for writing an entry to an archive. - - If the size is known ahead of time and provided, then the file contents - are not buffered but flushed directly to the archive. If size is omitted, - then the file contents are buffered and flushed in the close() method.''' - def __init__(self, archive, pathname, size=None): - self.archive = archive - self.entry = Entry(pathname=pathname, mtime=time.time(), mode=stat.S_IFREG) - if size is None: - self.buffer = six.StringIO() - else: - self.buffer = None - self.entry.size = size - self.entry.to_archive(self.archive) - self.bytes = 0 - self.closed = False - - def __enter__(self): - return self - - def __exit__(self, *args): - self.close() - - def __del__(self): - self.close() - - def __len__(self): - return self.bytes - - def tell(self): - return self.bytes - - def write(self, data): - if self.closed: - raise Exception('Cannot write to closed stream.') - if self.buffer: - self.buffer.write(data) - else: - _libarchive.archive_write_data_from_str(self.archive._a, data) - self.bytes += len(data) - - def close(self): - if self.closed: - return - if self.buffer: - self.entry.size = self.buffer.tell() - self.entry.to_archive(self.archive) - _libarchive.archive_write_data_from_str(self.archive._a, self.buffer.getvalue()) - _libarchive.archive_write_finish_entry(self.archive._a) - - # Call archive.close() with _defer True to let it know we have been - # closed and it is now safe to actually close. - self.archive.close(_defer=True) - self.archive = None - self.closed = True - - -class Entry(object): - '''An entry within an archive. Represents the header data and it's location within the archive.''' - def __init__(self, pathname=None, size=None, mtime=None, mode=None, hpos=None, encoding=ENCODING): - self.pathname = pathname - self.size = size - self.mtime = mtime - self.mode = mode - self.hpos = hpos - self.encoding = encoding - self.linkname = None - self.id = None - self.hardlink = None - - @property - def header_position(self): - return self.hpos - - @classmethod - def from_archive(cls, archive, encoding=ENCODING): - '''Instantiates an Entry class and sets all the properties from an archive header.''' - e = _libarchive.archive_entry_new() - try: - call_and_check(_libarchive.archive_read_next_header2, archive._a, archive._a, e) - mode = _libarchive.archive_entry_filetype(e) - mode |= _libarchive.archive_entry_perm(e) - mtime = _libarchive.archive_entry_mtime(e) + _libarchive.archive_entry_mtime_nsec(e) / 1000000000.0 - # use current time as mtime if stored mtime is equal to 0 - mtime = mtime or time.time() - entry = cls( - pathname=_libarchive.archive_entry_pathname(e).decode(encoding), - size=_libarchive.archive_entry_size(e), - mtime=mtime, - mode=mode, - hpos=archive.header_position, - ) - # check hardlinkness first to processes hardlinks to the symlinks correctly - hardlink = _libarchive.archive_entry_hardlink(e) - if hardlink: - entry.hardlink = hardlink - elif entry.issym(): - entry.linkname = _libarchive.archive_entry_symlink(e) - finally: - _libarchive.archive_entry_free(e) - return entry - - @classmethod - def from_file(cls, f, entry=None, encoding=ENCODING, mtime=None): - '''Instantiates an Entry class and sets all the properties from a file on the file system. - f can be a file-like object or a path.''' - if entry is None: - entry = cls(encoding=encoding) - if entry.pathname is None: - if isinstance(f, six.string_types): - st = os.lstat(f) - entry.pathname = f - entry.size = st.st_size - entry.mtime = st.st_mtime if mtime is None else mtime - entry.mode = st.st_mode - entry.id = cls.get_entry_id(st) - if entry.issym(): - entry.linkname = os.readlink(f) - elif hasattr(f, 'fileno'): - st = os.fstat(f.fileno()) - entry.pathname = getattr(f, 'name', None) - entry.size = st.st_size - entry.mtime = st.st_mtime if mtime is None else mtime - entry.mode = st.st_mode - entry.id = cls.get_entry_id(st) - else: - entry.pathname = getattr(f, 'pathname', None) - entry.size = getattr(f, 'size', 0) - entry.mtime = getattr(f, 'mtime', time.time()) if mtime is None else mtime - entry.mode = getattr(f, 'mode', stat.S_IFREG) - return entry - - @staticmethod - def get_entry_id(st): - # windows doesn't have such information - if st.st_ino and st.st_dev: - return (st.st_dev, st.st_ino) - return None - - def to_archive(self, archive): - '''Creates an archive header and writes it to the given archive.''' - e = _libarchive.archive_entry_new() - try: - _libarchive.archive_entry_set_pathname(e, encode(self.pathname, self.encoding)) - _libarchive.archive_entry_set_filetype(e, stat.S_IFMT(self.mode)) - _libarchive.archive_entry_set_perm(e, stat.S_IMODE(self.mode)) - - nsec, sec = math.modf(self.mtime) - nsec *= 1000000000 - _libarchive.archive_entry_set_mtime(e, int(sec), int(nsec)) - - if self.ishardlink(): - _libarchive.archive_entry_set_size(e, 0) - _libarchive.archive_entry_set_hardlink(e, encode(self.hardlink, self.encoding)) - elif self.issym(): - _libarchive.archive_entry_set_size(e, 0) - _libarchive.archive_entry_set_symlink(e, encode(self.linkname, self.encoding)) - else: - _libarchive.archive_entry_set_size(e, self.size) - call_and_check(_libarchive.archive_write_header, archive._a, archive._a, e) - #self.hpos = archive.header_position - finally: - _libarchive.archive_entry_free(e) - - def isdir(self): - return stat.S_ISDIR(self.mode) - - def isfile(self): - return stat.S_ISREG(self.mode) - - def issym(self): - return stat.S_ISLNK(self.mode) - - def isfifo(self): - return stat.S_ISFIFO(self.mode) - - def ischr(self): - return stat.S_ISCHR(self.mode) - - def isblk(self): - return stat.S_ISBLK(self.mode) - - def ishardlink(self): - return bool(self.hardlink) - - -class Archive(object): - '''A low-level archive reader which provides forward-only iteration. Consider - this a light-weight pythonic libarchive wrapper.''' - def __init__(self, f, mode='rb', format=None, filter=None, entry_class=Entry, encoding=ENCODING, blocksize=BLOCK_SIZE, filter_opts=None, format_opts=None, fsync=False, fixed_mtime=None): - if six.PY2: - assert mode in ('r', 'rb', 'w', 'wb', 'a', 'ab'), 'Mode should be "r[b]", "w[b]" or "a[b]".' - else: - assert mode in ('rb', 'wb', 'ab'), 'Mode should be "rb", "wb", or "ab".' - self._stream = None - self.encoding = encoding - self.blocksize = blocksize - self.file_handle = None - self.fd = None - self.filename = None - self.fsync = fsync - if isinstance(f, six.string_types): - self.filename = f - self.file_handle = open(f, mode) - self.fd = self.file_handle.fileno() - # Only close it if we opened it... - self._defer_close = True - elif hasattr(f, 'fileno'): - self.filename = getattr(f, 'name', None) - self.file_handle = f - self.fd = self.file_handle.fileno() - # Leave the fd alone, caller should manage it... - self._defer_close = False - elif isinstance(f, int): - assert f >= 0, f - self.fd = f - # Leave the fd alone, caller should manage it... - self._defer_close = False - else: - raise Exception('Provided file is not path or open file.') - self.mode = mode - # Guess the format/filter from file name (if not provided) - if self.filename: - if format is None: - format = guess_format(self.filename)[0] - if filter is None: - filter = guess_format(self.filename)[1] - self.format = format - self.filter = filter - # The class to use for entries. - self.entry_class = entry_class - self.fixed_mtime = fixed_mtime - # Select filter/format functions. - if self.mode.startswith('r'): - self.format_func = get_func(self.format, FORMATS, 0) - if self.format_func is None: - raise Exception('Unsupported format %s' % format) - self.filter_func = get_func(self.filter, FILTERS, 0) - if self.filter_func is None: - raise Exception('Unsupported filter %s' % filter) - else: - # TODO: how to support appending? - if self.format is None: - raise Exception('You must specify a format for writing.') - self.format_func = get_func(self.format, FORMATS, 1) - if self.format_func is None: - raise Exception('Unsupported format %s' % format) - self.filter_func = get_func(self.filter, FILTERS, 1) - if self.filter_func is None: - raise Exception('Unsupported filter %s' % filter) - # Open the archive, apply filter/format functions. - self.filter_opts = filter_opts - self.format_opts = format_opts - # Stores every added entry's id to handle hardlinks properly - self.members = {} - self.init() - - def __iter__(self): - while True: - try: - yield self.entry_class.from_archive(self, encoding=self.encoding) - except EOF: - break - - def __enter__(self): - return self - - def __exit__(self, type, value, traceback): - self.close() - - def __del__(self): - self.close() - - def init(self): - def _apply_opts(f, opts): - if opts: - for opt_name, opt_val in opts.items(): - call_and_check(f, self._a, self._a, None, encode(opt_name, self.encoding), encode(opt_val, self.encoding)) - - if self.mode.startswith('r'): - self._a = _libarchive.archive_read_new() - else: - self._a = _libarchive.archive_write_new() - self.format_func(self._a) - self.filter_func(self._a) - if self.mode.startswith('r'): - _apply_opts(_libarchive.archive_read_set_format_option, self.format_opts) - _apply_opts(_libarchive.archive_read_set_filter_option, self.filter_opts) - call_and_check(_libarchive.archive_read_open_fd, self._a, self._a, self.fd, self.blocksize) - else: - _apply_opts(_libarchive.archive_write_set_format_option, self.format_opts) - _apply_opts(_libarchive.archive_write_set_filter_option, self.filter_opts) - call_and_check(_libarchive.archive_write_open_fd, self._a, self._a, self.fd) - # XXX Don't pad the last block to avoid badly formed archive with zstd filter - call_and_check(_libarchive.archive_write_set_bytes_in_last_block, self._a, self._a, 1) - - def denit(self): - '''Closes and deallocates the archive reader/writer.''' - if getattr(self, '_a', None) is None: - return - try: - if self.mode.startswith('r'): - _libarchive.archive_read_close(self._a) - _libarchive.archive_read_free(self._a) - else: - _libarchive.archive_write_close(self._a) - _libarchive.archive_write_free(self._a) - finally: - # We only want one try at this... - self._a = None - - def close(self, _defer=False): - # _defer == True is how a stream can notify Archive that the stream is - # now closed. Calling it directly in not recommended. - if _defer: - # This call came from our open stream. - self._stream = None - if not self._defer_close: - # We are not yet ready to close. - return - if self._stream is not None: - # We have a stream open! don't close, but remember we were asked to. - self._defer_close = True - return - self.denit() - # If there is a file attached... - if getattr(self, 'file_handle', None): - # Make sure it is not already closed... - if getattr(self.file_handle, 'closed', False): - return - # Flush it if not read-only... - if not self.file_handle.mode.startswith('r'): - self.file_handle.flush() - if self.fsync: - os.fsync(self.fd) - # and then close it, if we opened it... - if getattr(self, 'close', None): - self.file_handle.close() - - @property - def header_position(self): - '''The position within the file.''' - return _libarchive.archive_read_header_position(self._a) - - def iterpaths(self): - for entry in self: - yield entry.pathname - - def read(self, size): - '''Read current archive entry contents into string.''' - return _libarchive.archive_read_data_into_str(self._a, size) - - def readpath(self, f): - '''Write current archive entry contents to file. f can be a file-like object or - a path.''' - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - basedir = os.path.basename(f) - if not os.path.exists(basedir): - os.makedirs(basedir) - f = exit_stack.enter_context(open(f, 'wb')) - return _libarchive.archive_read_data_into_fd(self._a, f.fileno()) - - def readstream(self, size): - '''Returns a file-like object for reading current archive entry contents.''' - self._stream = EntryReadStream(self, size) - return self._stream - - def write(self, member, data=None): - '''Writes a string buffer to the archive as the given entry.''' - if isinstance(member, six.string_types): - if self.fixed_mtime is None: - mtime = time.time() - else: - mtime = self.fixed_mtime - # Use default mode - member = self.entry_class(pathname=member, encoding=self.encoding, mtime=mtime, mode=stat.S_IFREG | 0o755) - if data: - member.size = len(data) - member.to_archive(self) - if data: - _libarchive.archive_write_data_from_str(self._a, data) - _libarchive.archive_write_finish_entry(self._a) - - def writepath(self, f, pathname=None): - '''Writes a file to the archive. f can be a file-like object or a path. Uses - write() to do the actual writing.''' - member = self.entry_class.from_file(f, encoding=self.encoding, mtime=self.fixed_mtime) - - with contextlib2.ExitStack() as exit_stack: - if isinstance(f, six.string_types): - if os.path.isfile(f): - f = exit_stack.enter_context(open(f, 'rb')) - if pathname: - member.pathname = pathname - - # hardlinks and symlink has no data to be written - if member.id in self.members: - member.hardlink = self.members[member.id] - self.write(member) - return - elif member.issym(): - self.write(member) - elif hasattr(f, 'read') and hasattr(f, 'seek') and hasattr(f, 'tell'): - self.write_from_file_object(member, f) - elif hasattr(f, 'read'): - # TODO: optimize this to write directly from f to archive. - self.write(member, data=f.read()) - else: - self.write(member) - - if member.id: - self.members[member.id] = member.pathname - - def write_from_file_object(self, member, fileobj): - if isinstance(member, six.string_types): - member = self.entry_class(pathname=member, encoding=self.encoding, mtime=self.fixed_mtime) - - start = fileobj.tell() - fileobj.seek(0, os.SEEK_END) - size = fileobj.tell() - start - fileobj.seek(start, os.SEEK_SET) - - if size: - member.size = size - member.to_archive(self) - - while size: - data = fileobj.read(BLOCK_SIZE) - if not data: - break - - size -= len(data) - if size < 0: - msg = "File ({}) size has changed. Can't write more data than was declared in the tar header ({}). " \ - "(probably file was changed during archiving)".format(member.pathname, member.size) - logger.warning(msg) - # write rest expected data (size is negative) - _libarchive.archive_write_data_from_str(self._a, data[:size]) - break - - _libarchive.archive_write_data_from_str(self._a, data) - - _libarchive.archive_write_finish_entry(self._a) - - def writestream(self, pathname, size=None): - '''Returns a file-like object for writing a new entry.''' - self._stream = EntryWriteStream(self, pathname, size) - return self._stream - - def printlist(self, s=sys.stdout): - for entry in self: - s.write(entry.size) - s.write('\t') - s.write(entry.mtime.strftime(MTIME_FORMAT)) - s.write('\t') - s.write(entry.pathname) - s.flush() - - -class SeekableArchive(Archive): - '''A class that provides random-access to archive entries. It does this by using one - or many Archive instances to seek to the correct location. The best performance will - occur when reading archive entries in the order in which they appear in the archive. - Reading out of order will cause the archive to be closed and opened each time a - reverse seek is needed.''' - def __init__(self, f, **kwargs): - self._stream = None - # Convert file to open file. We need this to reopen the archive. - mode = kwargs.setdefault('mode', 'rb') - if isinstance(f, six.string_types): - f = open(f, mode) - super(SeekableArchive, self).__init__(f, **kwargs) - self.entries = [] - self.eof = False - - def __iter__(self): - for entry in self.entries: - yield entry - if not self.eof: - try: - for entry in super(SeekableArchive, self).__iter__(): - self.entries.append(entry) - yield entry - except StopIteration: - self.eof = True - - def reopen(self): - '''Seeks the underlying fd to 0 position, then opens the archive. If the archive - is already open, this will effectively re-open it (rewind to the beginning).''' - self.denit() - self.file_handle.seek(0) - self.init() - - def getentry(self, pathname): - '''Take a name or entry object and returns an entry object.''' - for entry in self: - if entry.pathname == pathname: - return entry - raise KeyError(pathname) - - def seek(self, entry): - '''Seeks the archive to the requested entry. Will reopen if necessary.''' - move = entry.header_position - self.header_position - if move != 0: - if move < 0: - # can't move back, re-open archive: - self.reopen() - # move to proper position in stream - for curr in super(SeekableArchive, self).__iter__(): - if curr.header_position == entry.header_position: - break - - def read(self, member): - '''Return the requested archive entry contents as a string.''' - entry = self.getentry(member) - self.seek(entry) - return super(SeekableArchive, self).read(entry.size) - - def readpath(self, member, f): - entry = self.getentry(member) - self.seek(entry) - return super(SeekableArchive, self).readpath(f) - - def readstream(self, member): - '''Returns a file-like object for reading requested archive entry contents.''' - entry = self.getentry(member) - self.seek(entry) - self._stream = EntryReadStream(self, entry.size) - return self._stream diff --git a/contrib/python/python-libarchive/libarchive/_libarchive.swg b/contrib/python/python-libarchive/libarchive/_libarchive.swg deleted file mode 100644 index 2fcb05420e4..00000000000 --- a/contrib/python/python-libarchive/libarchive/_libarchive.swg +++ /dev/null @@ -1,339 +0,0 @@ -/* Copyright (c) 2011, SmartFile <btimby@smartfile.com> - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are met: - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the organization nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND - ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY - DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ - -%module _libarchive - -%{ -#define SWIG_PYTHON_STRICT_BYTE_CHAR - -#include <archive.h> -#include <archive_entry.h> -%} - -%include "typemaps.i" - -%typemap(in) time_t -{ - if (PyLong_Check($input)) - $1 = (time_t) PyLong_AsLong($input); - else if (PyInt_Check($input)) - $1 = (time_t) PyInt_AsLong($input); - else if (PyFloat_Check($input)) - $1 = (time_t) PyFloat_AsDouble($input); - else { - PyErr_SetString(PyExc_TypeError,"Expected a large number"); - return NULL; - } -} - -%typemap(out) time_t -{ - $result = PyLong_FromLong((long)$1); -} - -%typemap(in) int64_t -{ - if (PyLong_Check($input)) - $1 = (int64_t) PyLong_AsLong($input); - else if (PyInt_Check($input)) - $1 = (int64_t) PyInt_AsLong($input); - else if (PyFloat_Check($input)) - $1 = (int64_t) PyFloat_AsDouble($input); - else { - PyErr_SetString(PyExc_TypeError,"Expected a large number"); - return NULL; - } -} - -%typemap(out) int64_t -{ - $result = PyLong_FromLong((long)$1); -} - -#define __LA_INT64_T long long -#define __LA_MODE_T int - -/* STRUCTURES */ -struct archive; -struct archive_entry; - -/* ARCHIVE READING */ -extern struct archive *archive_read_new(void); -extern int archive_read_free(struct archive *); - -/* opening */ -extern int archive_read_open_filename(struct archive *, - const char *_filename, size_t _block_size); -extern int archive_read_open_memory(struct archive *, - void * buff, size_t size); -extern int archive_read_open_memory2(struct archive *a, void *buff, - size_t size, size_t read_size); -extern int archive_read_open_fd(struct archive *, int _fd, - size_t _block_size); - -/* closing */ -extern int archive_read_close(struct archive *); -extern int archive_format(struct archive *); - -/* headers */ -extern int archive_read_next_header2(struct archive *, - struct archive_entry *); -extern const struct stat *archive_entry_stat(struct archive_entry *); -extern __LA_INT64_T archive_read_header_position(struct archive *); - -/* data */ -extern int archive_read_data_skip(struct archive *); -extern int archive_read_data_into_fd(struct archive *, int fd); - -/* FILTERS */ -extern int archive_read_support_filter_all(struct archive *); -extern int archive_read_support_filter_bzip2(struct archive *); -extern int archive_read_support_filter_compress(struct archive *); -extern int archive_read_support_filter_gzip(struct archive *); -extern int archive_read_support_filter_lzip(struct archive *); -extern int archive_read_support_filter_lzma(struct archive *); -extern int archive_read_support_filter_none(struct archive *); -extern int archive_read_support_filter_rpm(struct archive *); -extern int archive_read_support_filter_uu(struct archive *); -extern int archive_read_support_filter_xz(struct archive *); -extern int archive_read_support_filter_zstd(struct archive *); - -extern int archive_filter_count(struct archive *); -extern const char * archive_filter_name(struct archive *, int); - -/* FORMATS */ -extern int archive_read_support_format_all(struct archive *); -extern int archive_read_support_format_7zip(struct archive *); -extern int archive_read_support_format_ar(struct archive *); -extern int archive_read_support_format_cab(struct archive *); -extern int archive_read_support_format_cpio(struct archive *); -extern int archive_read_support_format_empty(struct archive *); -extern int archive_read_support_format_gnutar(struct archive *); -extern int archive_read_support_format_iso9660(struct archive *); -extern int archive_read_support_format_lha(struct archive *); -/*extern int archive_read_support_format_mtree(struct archive *);*/ -extern int archive_read_support_format_rar(struct archive *); -extern int archive_read_support_format_raw(struct archive *); -extern int archive_read_support_format_tar(struct archive *); -extern int archive_read_support_format_xar(struct archive *); -extern int archive_read_support_format_zip(struct archive *); -/*extern int archive_read_support_format_by_code(struct archive *, int);*/ - -/* OPTIONS */ -extern int archive_write_set_bytes_in_last_block(struct archive *_a, int bytes_in_last_block); -extern int archive_write_set_filter_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_write_zip_set_compression_deflate(struct archive *_a); -extern int archive_write_set_format_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_read_set_filter_option(struct archive *_a, const char *m, const char *o, const char *v); -extern int archive_read_set_format_option(struct archive *_a, const char *m, const char *o, const char *v); - -/* ARCHIVE WRITING */ -extern struct archive *archive_write_new(void); -extern int archive_write_free(struct archive *); - -/* opening */ -extern int archive_write_open(struct archive *, void *, - archive_open_callback *, archive_write_callback *, - archive_close_callback *); -extern int archive_write_open_fd(struct archive *, int _fd); -extern int archive_write_open_filename(struct archive *, const char *_file); -extern int archive_write_open_filename_w(struct archive *, - const wchar_t *_file); -extern int archive_write_open_memory(struct archive *, - void *_buffer, size_t _buffSize, size_t *_used); - -/* closing */ -extern int archive_write_close(struct archive *); - -/* headers */ -extern int archive_write_header(struct archive *, - struct archive_entry *); - -/* data */ - -/* commit */ -extern int archive_write_finish_entry(struct archive *); - -/* FILTERS */ -extern int archive_write_add_filter_bzip2(struct archive *); -extern int archive_write_add_filter_compress(struct archive *); -extern int archive_write_add_filter_gzip(struct archive *); -extern int archive_write_add_filter_lzip(struct archive *); -extern int archive_write_add_filter_lzma(struct archive *); -extern int archive_write_add_filter_none(struct archive *); -extern int archive_write_add_filter_xz(struct archive *); -extern int archive_write_add_filter_zstd(struct archive *); - - -/* FORMATS */ -/* A convenience function to set the format based on the code or name. */ -extern int archive_write_set_format(struct archive *, int format_code); -extern int archive_write_set_format_by_name(struct archive *, - const char *name); -/* To minimize link pollution, use one or more of the following. */ -extern int archive_write_set_format_ar_bsd(struct archive *); -extern int archive_write_set_format_ar_svr4(struct archive *); -extern int archive_write_set_format_cpio(struct archive *); -extern int archive_write_set_format_cpio_newc(struct archive *); -extern int archive_write_set_format_gnutar(struct archive *); -extern int archive_write_set_format_iso9660(struct archive *); -/*extern int archive_write_set_format_mtree(struct archive *);*/ -/* TODO: int archive_write_set_format_old_tar(struct archive *); */ -extern int archive_write_set_format_pax(struct archive *); -extern int archive_write_set_format_pax_restricted(struct archive *); -extern int archive_write_set_format_shar(struct archive *); -extern int archive_write_set_format_shar_dump(struct archive *); -extern int archive_write_set_format_ustar(struct archive *); -extern int archive_write_set_format_xar(struct archive *); -extern int archive_write_set_format_zip(struct archive *); - -/* ARCHIVE ENTRY */ -extern struct archive_entry *archive_entry_new(void); -extern void archive_entry_free(struct archive_entry *); -extern const char *archive_entry_symlink(struct archive_entry *); -extern void archive_entry_set_symlink(struct archive_entry *, const char *); -extern const char *archive_entry_hardlink(struct archive_entry *); -extern void archive_entry_set_hardlink(struct archive_entry *, const char *); - -/* ARCHIVE ENTRY PROPERTY ACCESS */ -/* reading */ -extern const char *archive_entry_pathname(struct archive_entry *); -extern const wchar_t *archive_entry_pathname_w(struct archive_entry *); -extern __LA_INT64_T archive_entry_size(struct archive_entry *); -extern time_t archive_entry_mtime(struct archive_entry *); -extern time_t archive_entry_mtime_nsec(struct archive_entry *); -extern __LA_MODE_T archive_entry_filetype(struct archive_entry *); -extern __LA_MODE_T archive_entry_perm(struct archive_entry *); - -/* writing */ -extern void archive_entry_set_pathname(struct archive_entry *, const char *); -extern void archive_entry_set_size(struct archive_entry *, __LA_INT64_T); -extern void archive_entry_set_mtime(struct archive_entry *, time_t, long); -extern void archive_entry_set_filetype(struct archive_entry *, unsigned int); -extern void archive_entry_set_perm(struct archive_entry *, __LA_MODE_T); - - -/* ERROR HANDLING */ -extern int archive_errno(struct archive *); -extern const char *archive_error_string(struct archive *); - - -/* CONSTANTS */ -#define ARCHIVE_VERSION_NUMBER 3000001 -#define ARCHIVE_VERSION_STRING "libarchive 3.0.1b" -#define ARCHIVE_EOF 1 /* Found end of archive. */ -#define ARCHIVE_OK 0 /* Operation was successful. */ -#define ARCHIVE_RETRY (-10) /* Retry might succeed. */ -#define ARCHIVE_WARN (-20) /* Partial success. */ -#define ARCHIVE_FAILED (-25) /* Current operation cannot complete. */ -#define ARCHIVE_FATAL (-30) /* No more operations are possible. */ - -#define ARCHIVE_FILTER_NONE 0 -#define ARCHIVE_FILTER_GZIP 1 -#define ARCHIVE_FILTER_BZIP2 2 -#define ARCHIVE_FILTER_COMPRESS 3 -#define ARCHIVE_FILTER_PROGRAM 4 -#define ARCHIVE_FILTER_LZMA 5 -#define ARCHIVE_FILTER_XZ 6 -#define ARCHIVE_FILTER_UU 7 -#define ARCHIVE_FILTER_RPM 8 -#define ARCHIVE_FILTER_LZIP 9 - -#define ARCHIVE_FORMAT_BASE_MASK 0xff0000 -#define ARCHIVE_FORMAT_CPIO 0x10000 -#define ARCHIVE_FORMAT_CPIO_POSIX (ARCHIVE_FORMAT_CPIO | 1) -#define ARCHIVE_FORMAT_CPIO_BIN_LE (ARCHIVE_FORMAT_CPIO | 2) -#define ARCHIVE_FORMAT_CPIO_BIN_BE (ARCHIVE_FORMAT_CPIO | 3) -#define ARCHIVE_FORMAT_CPIO_SVR4_NOCRC (ARCHIVE_FORMAT_CPIO | 4) -#define ARCHIVE_FORMAT_CPIO_SVR4_CRC (ARCHIVE_FORMAT_CPIO | 5) -#define ARCHIVE_FORMAT_CPIO_AFIO_LARGE (ARCHIVE_FORMAT_CPIO | 6) -#define ARCHIVE_FORMAT_SHAR 0x20000 -#define ARCHIVE_FORMAT_SHAR_BASE (ARCHIVE_FORMAT_SHAR | 1) -#define ARCHIVE_FORMAT_SHAR_DUMP (ARCHIVE_FORMAT_SHAR | 2) -#define ARCHIVE_FORMAT_TAR 0x30000 -#define ARCHIVE_FORMAT_TAR_USTAR (ARCHIVE_FORMAT_TAR | 1) -#define ARCHIVE_FORMAT_TAR_PAX_INTERCHANGE (ARCHIVE_FORMAT_TAR | 2) -#define ARCHIVE_FORMAT_TAR_PAX_RESTRICTED (ARCHIVE_FORMAT_TAR | 3) -#define ARCHIVE_FORMAT_TAR_GNUTAR (ARCHIVE_FORMAT_TAR | 4) -#define ARCHIVE_FORMAT_ISO9660 0x40000 -#define ARCHIVE_FORMAT_ISO9660_ROCKRIDGE (ARCHIVE_FORMAT_ISO9660 | 1) -#define ARCHIVE_FORMAT_ZIP 0x50000 -#define ARCHIVE_FORMAT_EMPTY 0x60000 -#define ARCHIVE_FORMAT_AR 0x70000 -#define ARCHIVE_FORMAT_AR_GNU (ARCHIVE_FORMAT_AR | 1) -#define ARCHIVE_FORMAT_AR_BSD (ARCHIVE_FORMAT_AR | 2) -#define ARCHIVE_FORMAT_MTREE 0x80000 -#define ARCHIVE_FORMAT_RAW 0x90000 -#define ARCHIVE_FORMAT_XAR 0xA0000 -#define ARCHIVE_FORMAT_LHA 0xB0000 -#define ARCHIVE_FORMAT_CAB 0xC0000 -#define ARCHIVE_FORMAT_RAR 0xD0000 -#define ARCHIVE_FORMAT_7ZIP 0xE0000 - -#define ARCHIVE_EXTRACT_OWNER (0x0001) -#define ARCHIVE_EXTRACT_PERM (0x0002) -#define ARCHIVE_EXTRACT_TIME (0x0004) -#define ARCHIVE_EXTRACT_NO_OVERWRITE (0x0008) -#define ARCHIVE_EXTRACT_UNLINK (0x0010) -#define ARCHIVE_EXTRACT_ACL (0x0020) -#define ARCHIVE_EXTRACT_FFLAGS (0x0040) -#define ARCHIVE_EXTRACT_XATTR (0x0080) -#define ARCHIVE_EXTRACT_SECURE_SYMLINKS (0x0100) -#define ARCHIVE_EXTRACT_SECURE_NODOTDOT (0x0200) -#define ARCHIVE_EXTRACT_NO_AUTODIR (0x0400) -#define ARCHIVE_EXTRACT_NO_OVERWRITE_NEWER (0x0800) -#define ARCHIVE_EXTRACT_SPARSE (0x1000) -#define ARCHIVE_EXTRACT_MAC_METADATA (0x2000) - -%inline %{ -PyObject *archive_read_data_into_str(struct archive *archive, int len) { - PyObject *str = NULL; - if (!(str = PyBytes_FromStringAndSize(NULL, len))) { - PyErr_SetString(PyExc_MemoryError, "could not allocate string."); - return NULL; - } - if (len != archive_read_data(archive, PyBytes_AS_STRING(str), len)) { - PyErr_SetString(PyExc_RuntimeError, "could not read requested data."); - return NULL; - } - return str; -} - -PyObject *archive_write_data_from_str(struct archive *archive, PyObject *str) { - int len = PyBytes_Size(str); - if (len == 0) - return PyInt_FromLong(len); - int ret = archive_write_data(archive, PyBytes_AS_STRING(str), len); - if (ret == ARCHIVE_FATAL) { - PyErr_Format(PyExc_RuntimeError, "Could not write requested data - most likely no space left on device (error code: %d)", ret); - return NULL; - } - else if (ret <= 0) { - PyErr_Format(PyExc_RuntimeError, "Could not write requested data (error code: %d)", ret); - return NULL; - } - return PyInt_FromLong(len); -} -%} diff --git a/contrib/python/python-libarchive/libarchive/tar.py b/contrib/python/python-libarchive/libarchive/tar.py deleted file mode 100644 index f14149804b9..00000000000 --- a/contrib/python/python-libarchive/libarchive/tar.py +++ /dev/null @@ -1,135 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import time -from libarchive import is_archive, Entry, SeekableArchive -from tarfile import DEFAULT_FORMAT, USTAR_FORMAT, GNU_FORMAT, PAX_FORMAT, ENCODING -from tarfile import REGTYPE, AREGTYPE, LNKTYPE, SYMTYPE, DIRTYPE, FIFOTYPE, CONTTYPE, CHRTYPE, BLKTYPE, GNUTYPE_SPARSE - -FORMAT_CONVERSION = { - USTAR_FORMAT: 'tar', - GNU_FORMAT: 'gnu', - PAX_FORMAT: 'pax', -} - - -def is_tarfile(filename): - return is_archive(filename, formats=('tar', 'gnu', 'pax')) - - -def open(**kwargs): - return TarFile(**kwargs) - - -class TarInfo(Entry): - def __init__(self, name): - super(TarInfo, self).__init__(pathname=name) - - fromtarfile = Entry.from_archive - - def get_name(self): - return self.pathname - - def set_name(self, value): - self.pathname = value - - name = property(get_name, set_name) - - @property - def get_type(self): - for attr, type in ( - ('isdir', DIRTYPE), ('isfile', REGTYPE), ('issym', SYMTYPE), - ('isfifo', FIFOTYPE), ('ischr', CHRTYPE), ('isblk', BLKTYPE), - ): - if getattr(self, attr)(): - return type - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - pax_headers = property(_get_missing, _set_missing) - - -class TarFile(SeekableArchive): - def __init__(self, name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, encoding=ENCODING): - if name: - f = name - elif fileobj: - f = fileobj - try: - format = FORMAT_CONVERSON.get(format) - except KeyError: - raise Exception('Invalid tar format: %s' % format) - super(TarFile, self).__init__(f, mode=mode, format=format, entry_class=tarinfo, encoding=encoding) - - getmember = SeekableArchive.getentry - list = SeekableArchive.printlist - extract = SeekableArchive.readpath - extractfile = SeekableArchive.readstream - - def getmembers(self): - return list(self) - - def getnames(self): - return list(self.iterpaths) - - def next(self): - pass # TODO: how to do this? - - def extract(self, member, path=None): - if path is None: - path = os.getcwd() - if isinstance(member, basestring): - f = os.path.join(path, member) - else: - f = os.path.join(path, member.pathname) - return self.readpath(member, f) - - def add(self, name, arcname, recursive=True, exclude=None, filter=None): - pass # TODO: implement this. - - def addfile(tarinfo, fileobj): - return self.writepath(fileobj, tarinfo) - - def gettarinfo(name=None, arcname=None, fileobj=None): - if name: - f = name - elif fileobj: - f = fileobj - entry = self.entry_class.from_file(f) - if arcname: - entry.pathname = arcname - return entry - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - pax_headers = property(_get_missing, _set_missing) diff --git a/contrib/python/python-libarchive/libarchive/zip.py b/contrib/python/python-libarchive/libarchive/zip.py deleted file mode 100644 index 539f6dbcc49..00000000000 --- a/contrib/python/python-libarchive/libarchive/zip.py +++ /dev/null @@ -1,151 +0,0 @@ -# Copyright (c) 2011, SmartFile <btimby@smartfile.com> -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions are met: -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in the -# documentation and/or other materials provided with the distribution. -# * Neither the name of the organization nor the -# names of its contributors may be used to endorse or promote products -# derived from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY -# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -import os, time -from libarchive import is_archive, Entry, SeekableArchive -from zipfile import ZIP_STORED, ZIP_DEFLATED - - -def is_zipfile(filename): - return is_archive(filename, formats=('zip', )) - - -class ZipEntry(Entry): - def __init__(self, *args, **kwargs): - super(ZipEntry, self).__init__(*args, **kwargs) - - def get_filename(self): - return self.pathname - - def set_filename(self, value): - self.pathname = value - - filename = property(get_filename, set_filename) - - def get_file_size(self): - return self.size - - def set_file_size(self, value): - assert isinstance(size, (int, long)), 'Please provide size as int or long.' - self.size = value - - file_size = property(get_file_size, set_file_size) - - def get_date_time(self): - return time.localtime(self.mtime)[0:6] - - def set_date_time(self, value): - assert isinstance(value, tuple), 'mtime should be tuple (year, month, day, hour, minute, second).' - assert len(value) == 6, 'mtime should be tuple (year, month, day, hour, minute, second).' - self.mtime = time.mktime(value + (0, 0, 0)) - - date_time = property(get_date_time, set_date_time) - - header_offset = Entry.header_position - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - compress_type = property(_get_missing, _set_missing) - comment = property(_get_missing, _set_missing) - extra = property(_get_missing, _set_missing) - create_system = property(_get_missing, _set_missing) - create_version = property(_get_missing, _set_missing) - extract_version = property(_get_missing, _set_missing) - reserved = property(_get_missing, _set_missing) - flag_bits = property(_get_missing, _set_missing) - volume = property(_get_missing, _set_missing) - internal_attr = property(_get_missing, _set_missing) - external_attr = property(_get_missing, _set_missing) - CRC = property(_get_missing, _set_missing) - compress_size = property(_get_missing, _set_missing) - - -class ZipFile(SeekableArchive): - def __init__(self, f, mode='r', compression=ZIP_DEFLATED, allowZip64=False): - super(ZipFile, self).__init__(f, mode=mode, format='zip', entry_class=ZipEntry, encoding='CP437') - if mode == 'w' and compression == ZIP_STORED: - # Disable compression for writing. - _libarchive.archive_write_set_format_option(self.archive._a, "zip", "compression", "store") - self.compression = compression - - getinfo = SeekableArchive.getentry - - def namelist(self): - return list(self.iterpaths) - - def infolist(self): - return list(self) - - def open(self, name, mode, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if mode == 'r': - return self.readstream(name) - else: - return self.writestream(name) - - def extract(self, name, path=None, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if not path: - path = os.getcwd() - return self.readpath(name, os.path.join(path, name)) - - def extractall(self, path, names=None, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - if not names: - names = self.namelist() - if names: - for name in names: - self.extract(name, path) - - def read(self, name, pwd=None): - if pwd: - raise NotImplemented('Encryption not supported.') - return self.read(name) - - def writestr(self, member, data, compress_type=None): - if compress_type != self.compression: - raise Exception('Cannot change compression type for individual entries.') - return self.write(member, data) - - def setpassword(self, pwd): - raise NotImplemented('Encryption not supported.') - - def testzip(self): - raise NotImplemented() - - def _get_missing(self): - raise NotImplemented() - - def _set_missing(self, value): - raise NotImplemented() - - comment = property(_get_missing, _set_missing) diff --git a/contrib/python/python-libarchive/ya.make b/contrib/python/python-libarchive/ya.make deleted file mode 100644 index db042fa6797..00000000000 --- a/contrib/python/python-libarchive/ya.make +++ /dev/null @@ -1,28 +0,0 @@ -PY23_LIBRARY() - -LICENSE(BSD-3-Clause) - -VERSION(3.1.2.post1) - -PEERDIR( - contrib/libs/libarchive - contrib/python/contextlib2 - contrib/python/six -) - -ADDINCL( - contrib/libs/libarchive/libarchive -) - -NO_LINT() - -PY_SRCS( - SWIG_C - TOP_LEVEL - libarchive/__init__.py - libarchive/tar.py - libarchive/zip.py - libarchive/_libarchive.swg -) - -END() diff --git a/contrib/python/simplejson/.dist-info/METADATA b/contrib/python/simplejson/.dist-info/METADATA deleted file mode 100644 index 030823c2a19..00000000000 --- a/contrib/python/simplejson/.dist-info/METADATA +++ /dev/null @@ -1,68 +0,0 @@ -Metadata-Version: 2.1 -Name: simplejson -Version: 3.18.4 -Summary: Simple, fast, extensible JSON encoder/decoder for Python -Home-page: https://github.com/simplejson/simplejson -Author: Bob Ippolito -Author-email: bob@redivi.com -License: MIT License -Platform: any -Classifier: Development Status :: 5 - Production/Stable -Classifier: Intended Audience :: Developers -Classifier: License :: OSI Approved :: MIT License -Classifier: License :: OSI Approved :: Academic Free License (AFL) -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.5 -Classifier: Programming Language :: Python :: 2.6 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Programming Language :: Python :: 3.9 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Programming Language :: Python :: 3.11 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Classifier: Topic :: Software Development :: Libraries :: Python Modules -Requires-Python: >=2.5, !=3.0.*, !=3.1.*, !=3.2.* -License-File: LICENSE.txt - -simplejson ----------- - -simplejson is a simple, fast, complete, correct and extensible -JSON <http://json.org> encoder and decoder for Python 3.3+ -with legacy support for Python 2.5+. It is pure Python code -with no dependencies, but includes an optional C extension -for a serious speed boost. - -The latest documentation for simplejson can be read online here: -https://simplejson.readthedocs.io/ - -simplejson is the externally maintained development version of the -json library included with Python (since 2.6). This version is tested -with the latest Python 3.8 and maintains backwards compatibility -with Python 3.3+ and the legacy Python 2.5 - Python 2.7 releases. - -The encoder can be specialized to provide serialization in any kind of -situation, without any special support by the objects to be serialized -(somewhat like pickle). This is best done with the ``default`` kwarg -to dumps. - -The decoder can handle incoming JSON strings of any specified encoding -(UTF-8 by default). It can also be specialized to post-process JSON -objects with the ``object_hook`` or ``object_pairs_hook`` kwargs. This -is particularly useful for implementing protocols such as JSON-RPC -that have a richer type system than JSON itself. - -For those of you that have legacy systems to maintain, there is a -very old fork of simplejson in the `python2.2`_ branch that supports -Python 2.2. This is based on a very old version of simplejson, -is not maintained, and should only be used as a last resort. - -.. _python2.2: https://github.com/simplejson/simplejson/tree/python2.2 diff --git a/contrib/python/simplejson/.dist-info/top_level.txt b/contrib/python/simplejson/.dist-info/top_level.txt deleted file mode 100644 index 322630ee75f..00000000000 --- a/contrib/python/simplejson/.dist-info/top_level.txt +++ /dev/null @@ -1 +0,0 @@ -simplejson diff --git a/contrib/python/simplejson/LICENSE.txt b/contrib/python/simplejson/LICENSE.txt deleted file mode 100644 index e05f49c3fd0..00000000000 --- a/contrib/python/simplejson/LICENSE.txt +++ /dev/null @@ -1,79 +0,0 @@ -simplejson is dual-licensed software. It is available under the terms -of the MIT license, or the Academic Free License version 2.1. The full -text of each license agreement is included below. This code is also -licensed to the Python Software Foundation (PSF) under a Contributor -Agreement. - -MIT License -=========== - -Copyright (c) 2006 Bob Ippolito - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in -the Software without restriction, including without limitation the rights to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies -of the Software, and to permit persons to whom the Software is furnished to do -so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. - -Academic Free License v. 2.1 -============================ - -Copyright (c) 2006 Bob Ippolito. All rights reserved. - -This Academic Free License (the "License") applies to any original work of authorship (the "Original Work") whose owner (the "Licensor") has placed the following notice immediately following the copyright notice for the Original Work: - -Licensed under the Academic Free License version 2.1 - -1) Grant of Copyright License. Licensor hereby grants You a world-wide, royalty-free, non-exclusive, perpetual, sublicenseable license to do the following: - -a) to reproduce the Original Work in copies; - -b) to prepare derivative works ("Derivative Works") based upon the Original Work; - -c) to distribute copies of the Original Work and Derivative Works to the public; - -d) to perform the Original Work publicly; and - -e) to display the Original Work publicly. - -2) Grant of Patent License. Licensor hereby grants You a world-wide, royalty-free, non-exclusive, perpetual, sublicenseable license, under patent claims owned or controlled by the Licensor that are embodied in the Original Work as furnished by the Licensor, to make, use, sell and offer for sale the Original Work and Derivative Works. - -3) Grant of Source Code License. The term "Source Code" means the preferred form of the Original Work for making modifications to it and all available documentation describing how to modify the Original Work. Licensor hereby agrees to provide a machine-readable copy of the Source Code of the Original Work along with each copy of the Original Work that Licensor distributes. Licensor reserves the right to satisfy this obligation by placing a machine-readable copy of the Source Code in an information repository reasonably calculated to permit inexpensive and convenient access by You for as long as Licensor continues to distribute the Original Work, and by publishing the address of that information repository in a notice immediately following the copyright notice that applies to the Original Work. - -4) Exclusions From License Grant. Neither the names of Licensor, nor the names of any contributors to the Original Work, nor any of their trademarks or service marks, may be used to endorse or promote products derived from this Original Work without express prior written permission of the Licensor. Nothing in this License shall be deemed to grant any rights to trademarks, copyrights, patents, trade secrets or any other intellectual property of Licensor except as expressly stated herein. No patent license is granted to make, use, sell or offer to sell embodiments of any patent claims other than the licensed claims defined in Section 2. No right is granted to the trademarks of Licensor even if such marks are included in the Original Work. Nothing in this License shall be interpreted to prohibit Licensor from licensing under different terms from this License any Original Work that Licensor otherwise would have a right to license. - -5) This section intentionally omitted. - -6) Attribution Rights. You must retain, in the Source Code of any Derivative Works that You create, all copyright, patent or trademark notices from the Source Code of the Original Work, as well as any notices of licensing and any descriptive text identified therein as an "Attribution Notice." You must cause the Source Code for any Derivative Works that You create to carry a prominent Attribution Notice reasonably calculated to inform recipients that You have modified the Original Work. - -7) Warranty of Provenance and Disclaimer of Warranty. Licensor warrants that the copyright in and to the Original Work and the patent rights granted herein by Licensor are owned by the Licensor or are sublicensed to You under the terms of this License with the permission of the contributor(s) of those copyrights and patent rights. Except as expressly stated in the immediately proceeding sentence, the Original Work is provided under this License on an "AS IS" BASIS and WITHOUT WARRANTY, either express or implied, including, without limitation, the warranties of NON-INFRINGEMENT, MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY OF THE ORIGINAL WORK IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an essential part of this License. No license to Original Work is granted hereunder except under this disclaimer. - -8) Limitation of Liability. Under no circumstances and under no legal theory, whether in tort (including negligence), contract, or otherwise, shall the Licensor be liable to any person for any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or the use of the Original Work including, without limitation, damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses. This limitation of liability shall not apply to liability for death or personal injury resulting from Licensor's negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. - -9) Acceptance and Termination. If You distribute copies of the Original Work or a Derivative Work, You must make a reasonable effort under the circumstances to obtain the express assent of recipients to the terms of this License. Nothing else but this License (or another written agreement between Licensor and You) grants You permission to create Derivative Works based upon the Original Work or to exercise any of the rights granted in Section 1 herein, and any attempt to do so except under the terms of this License (or another written agreement between Licensor and You) is expressly prohibited by U.S. copyright law, the equivalent laws of other countries, and by international treaty. Therefore, by exercising any of the rights granted to You in Section 1 herein, You indicate Your acceptance of this License and all of its terms and conditions. - -10) Termination for Patent Action. This License shall terminate automatically and You may no longer exercise any of the rights granted to You by this License as of the date You commence an action, including a cross-claim or counterclaim, against Licensor or any licensee alleging that the Original Work infringes a patent. This termination provision shall not apply for an action alleging patent infringement by combinations of the Original Work with other software or hardware. - -11) Jurisdiction, Venue and Governing Law. Any action or suit relating to this License may be brought only in the courts of a jurisdiction wherein the Licensor resides or in which Licensor conducts its primary business, and under the laws of that jurisdiction excluding its conflict-of-law provisions. The application of the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any use of the Original Work outside the scope of this License or after its termination shall be subject to the requirements and penalties of the U.S. Copyright Act, 17 U.S.C. § 101 et seq., the equivalent laws of other countries, and international treaty. This section shall survive the termination of this License. - -12) Attorneys Fees. In any action to enforce the terms of this License or seeking damages relating thereto, the prevailing party shall be entitled to recover its costs and expenses, including, without limitation, reasonable attorneys' fees and costs incurred in connection with such action, including any appeal of such action. This section shall survive the termination of this License. - -13) Miscellaneous. This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. - -14) Definition of "You" in This License. "You" throughout this License, whether in upper or lower case, means an individual or a legal entity exercising rights under, and complying with all of the terms of, this License. For legal entities, "You" includes any entity that controls, is controlled by, or is under common control with you. For purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. - -15) Right to Use. You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You. - -This license is Copyright (C) 2003-2004 Lawrence E. Rosen. All rights reserved. Permission is hereby granted to copy and distribute this license without modification. This license may not be modified without the express written permission of its copyright owner. diff --git a/contrib/python/simplejson/README.rst b/contrib/python/simplejson/README.rst deleted file mode 100644 index 6580ddac899..00000000000 --- a/contrib/python/simplejson/README.rst +++ /dev/null @@ -1,34 +0,0 @@ -simplejson ----------- - -simplejson is a simple, fast, complete, correct and extensible -JSON <http://json.org> encoder and decoder for Python 3.3+ -with legacy support for Python 2.5+. It is pure Python code -with no dependencies, but includes an optional C extension -for a serious speed boost. - -The latest documentation for simplejson can be read online here: -https://simplejson.readthedocs.io/ - -simplejson is the externally maintained development version of the -json library included with Python (since 2.6). This version is tested -with the latest Python 3.8 and maintains backwards compatibility -with Python 3.3+ and the legacy Python 2.5 - Python 2.7 releases. - -The encoder can be specialized to provide serialization in any kind of -situation, without any special support by the objects to be serialized -(somewhat like pickle). This is best done with the ``default`` kwarg -to dumps. - -The decoder can handle incoming JSON strings of any specified encoding -(UTF-8 by default). It can also be specialized to post-process JSON -objects with the ``object_hook`` or ``object_pairs_hook`` kwargs. This -is particularly useful for implementing protocols such as JSON-RPC -that have a richer type system than JSON itself. - -For those of you that have legacy systems to maintain, there is a -very old fork of simplejson in the `python2.2`_ branch that supports -Python 2.2. This is based on a very old version of simplejson, -is not maintained, and should only be used as a last resort. - -.. _python2.2: https://github.com/simplejson/simplejson/tree/python2.2 diff --git a/contrib/python/simplejson/simplejson/__init__.py b/contrib/python/simplejson/simplejson/__init__.py deleted file mode 100644 index 47e49a318d8..00000000000 --- a/contrib/python/simplejson/simplejson/__init__.py +++ /dev/null @@ -1,584 +0,0 @@ -r"""JSON (JavaScript Object Notation) <http://json.org> is a subset of -JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data -interchange format. - -:mod:`simplejson` exposes an API familiar to users of the standard library -:mod:`marshal` and :mod:`pickle` modules. It is the externally maintained -version of the :mod:`json` library contained in Python 2.6, but maintains -compatibility back to Python 2.5 and (currently) has significant performance -advantages, even without using the optional C extension for speedups. - -Encoding basic Python object hierarchies:: - - >>> import simplejson as json - >>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}]) - '["foo", {"bar": ["baz", null, 1.0, 2]}]' - >>> print(json.dumps("\"foo\bar")) - "\"foo\bar" - >>> print(json.dumps(u'\u1234')) - "\u1234" - >>> print(json.dumps('\\')) - "\\" - >>> print(json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True)) - {"a": 0, "b": 0, "c": 0} - >>> from simplejson.compat import StringIO - >>> io = StringIO() - >>> json.dump(['streaming API'], io) - >>> io.getvalue() - '["streaming API"]' - -Compact encoding:: - - >>> import simplejson as json - >>> obj = [1,2,3,{'4': 5, '6': 7}] - >>> json.dumps(obj, separators=(',',':'), sort_keys=True) - '[1,2,3,{"4":5,"6":7}]' - -Pretty printing:: - - >>> import simplejson as json - >>> print(json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=' ')) - { - "4": 5, - "6": 7 - } - -Decoding JSON:: - - >>> import simplejson as json - >>> obj = [u'foo', {u'bar': [u'baz', None, 1.0, 2]}] - >>> json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]') == obj - True - >>> json.loads('"\\"foo\\bar"') == u'"foo\x08ar' - True - >>> from simplejson.compat import StringIO - >>> io = StringIO('["streaming API"]') - >>> json.load(io)[0] == 'streaming API' - True - -Specializing JSON object decoding:: - - >>> import simplejson as json - >>> def as_complex(dct): - ... if '__complex__' in dct: - ... return complex(dct['real'], dct['imag']) - ... return dct - ... - >>> json.loads('{"__complex__": true, "real": 1, "imag": 2}', - ... object_hook=as_complex) - (1+2j) - >>> from decimal import Decimal - >>> json.loads('1.1', parse_float=Decimal) == Decimal('1.1') - True - -Specializing JSON object encoding:: - - >>> import simplejson as json - >>> def encode_complex(obj): - ... if isinstance(obj, complex): - ... return [obj.real, obj.imag] - ... raise TypeError('Object of type %s is not JSON serializable' % - ... obj.__class__.__name__) - ... - >>> json.dumps(2 + 1j, default=encode_complex) - '[2.0, 1.0]' - >>> json.JSONEncoder(default=encode_complex).encode(2 + 1j) - '[2.0, 1.0]' - >>> ''.join(json.JSONEncoder(default=encode_complex).iterencode(2 + 1j)) - '[2.0, 1.0]' - -Using simplejson.tool from the shell to validate and pretty-print:: - - $ echo '{"json":"obj"}' | python -m simplejson.tool - { - "json": "obj" - } - $ echo '{ 1.2:3.4}' | python -m simplejson.tool - Expecting property name: line 1 column 3 (char 2) - -Parsing multiple documents serialized as JSON lines (newline-delimited JSON):: - - >>> import simplejson as json - >>> def loads_lines(docs): - ... for doc in docs.splitlines(): - ... yield json.loads(doc) - ... - >>> sum(doc["count"] for doc in loads_lines('{"count":1}\n{"count":2}\n{"count":3}\n')) - 6 - -Serializing multiple objects to JSON lines (newline-delimited JSON):: - - >>> import simplejson as json - >>> def dumps_lines(objs): - ... for obj in objs: - ... yield json.dumps(obj, separators=(',',':')) + '\n' - ... - >>> ''.join(dumps_lines([{'count': 1}, {'count': 2}, {'count': 3}])) - '{"count":1}\n{"count":2}\n{"count":3}\n' - -""" -from __future__ import absolute_import -__version__ = '3.18.4' -__all__ = [ - 'dump', 'dumps', 'load', 'loads', - 'JSONDecoder', 'JSONDecodeError', 'JSONEncoder', - 'OrderedDict', 'simple_first', 'RawJSON' -] - -__author__ = 'Bob Ippolito <bob@redivi.com>' - -from decimal import Decimal - -from .errors import JSONDecodeError -from .raw_json import RawJSON -from .decoder import JSONDecoder -from .encoder import JSONEncoder, JSONEncoderForHTML -def _import_OrderedDict(): - import collections - try: - return collections.OrderedDict - except AttributeError: - from . import ordered_dict - return ordered_dict.OrderedDict -OrderedDict = _import_OrderedDict() - -def _import_c_make_encoder(): - try: - from ._speedups import make_encoder - return make_encoder - except ImportError: - return None - -_default_encoder = JSONEncoder( - skipkeys=False, - ensure_ascii=True, - check_circular=True, - allow_nan=True, - indent=None, - separators=None, - encoding='utf-8', - default=None, - use_decimal=True, - namedtuple_as_object=True, - tuple_as_array=True, - iterable_as_array=False, - bigint_as_string=False, - item_sort_key=None, - for_json=False, - ignore_nan=False, - int_as_string_bitcount=None, -) - -def dump(obj, fp, skipkeys=False, ensure_ascii=True, check_circular=True, - allow_nan=True, cls=None, indent=None, separators=None, - encoding='utf-8', default=None, use_decimal=True, - namedtuple_as_object=True, tuple_as_array=True, - bigint_as_string=False, sort_keys=False, item_sort_key=None, - for_json=False, ignore_nan=False, int_as_string_bitcount=None, - iterable_as_array=False, **kw): - """Serialize ``obj`` as a JSON formatted stream to ``fp`` (a - ``.write()``-supporting file-like object). - - If *skipkeys* is true then ``dict`` keys that are not basic types - (``str``, ``int``, ``long``, ``float``, ``bool``, ``None``) - will be skipped instead of raising a ``TypeError``. - - If *ensure_ascii* is false (default: ``True``), then the output may - contain non-ASCII characters, so long as they do not need to be escaped - by JSON. When it is true, all non-ASCII characters are escaped. - - If *allow_nan* is false, then it will be a ``ValueError`` to - serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``) - in strict compliance of the original JSON specification, instead of using - the JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``). See - *ignore_nan* for ECMA-262 compliant behavior. - - If *indent* is a string, then JSON array elements and object members - will be pretty-printed with a newline followed by that string repeated - for each level of nesting. ``None`` (the default) selects the most compact - representation without any newlines. - - If specified, *separators* should be an - ``(item_separator, key_separator)`` tuple. The default is ``(', ', ': ')`` - if *indent* is ``None`` and ``(',', ': ')`` otherwise. To get the most - compact JSON representation, you should specify ``(',', ':')`` to eliminate - whitespace. - - *encoding* is the character encoding for str instances, default is UTF-8. - - *default(obj)* is a function that should return a serializable version - of obj or raise ``TypeError``. The default simply raises ``TypeError``. - - If *use_decimal* is true (default: ``True``) then decimal.Decimal - will be natively serialized to JSON with full precision. - - If *namedtuple_as_object* is true (default: ``True``), - :class:`tuple` subclasses with ``_asdict()`` methods will be encoded - as JSON objects. - - If *tuple_as_array* is true (default: ``True``), - :class:`tuple` (and subclasses) will be encoded as JSON arrays. - - If *iterable_as_array* is true (default: ``False``), - any object not in the above table that implements ``__iter__()`` - will be encoded as a JSON array. - - If *bigint_as_string* is true (default: ``False``), ints 2**53 and higher - or lower than -2**53 will be encoded as strings. This is to avoid the - rounding that happens in Javascript otherwise. Note that this is still a - lossy operation that will not round-trip correctly and should be used - sparingly. - - If *int_as_string_bitcount* is a positive number (n), then int of size - greater than or equal to 2**n or lower than or equal to -2**n will be - encoded as strings. - - If specified, *item_sort_key* is a callable used to sort the items in - each dictionary. This is useful if you want to sort items other than - in alphabetical order by key. This option takes precedence over - *sort_keys*. - - If *sort_keys* is true (default: ``False``), the output of dictionaries - will be sorted by item. - - If *for_json* is true (default: ``False``), objects with a ``for_json()`` - method will use the return value of that method for encoding as JSON - instead of the object. - - If *ignore_nan* is true (default: ``False``), then out of range - :class:`float` values (``nan``, ``inf``, ``-inf``) will be serialized as - ``null`` in compliance with the ECMA-262 specification. If true, this will - override *allow_nan*. - - To use a custom ``JSONEncoder`` subclass (e.g. one that overrides the - ``.default()`` method to serialize additional types), specify it with - the ``cls`` kwarg. NOTE: You should use *default* or *for_json* instead - of subclassing whenever possible. - - """ - # cached encoder - if (not skipkeys and ensure_ascii and - check_circular and allow_nan and - cls is None and indent is None and separators is None and - encoding == 'utf-8' and default is None and use_decimal - and namedtuple_as_object and tuple_as_array and not iterable_as_array - and not bigint_as_string and not sort_keys - and not item_sort_key and not for_json - and not ignore_nan and int_as_string_bitcount is None - and not kw - ): - iterable = _default_encoder.iterencode(obj) - else: - if cls is None: - cls = JSONEncoder - iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii, - check_circular=check_circular, allow_nan=allow_nan, indent=indent, - separators=separators, encoding=encoding, - default=default, use_decimal=use_decimal, - namedtuple_as_object=namedtuple_as_object, - tuple_as_array=tuple_as_array, - iterable_as_array=iterable_as_array, - bigint_as_string=bigint_as_string, - sort_keys=sort_keys, - item_sort_key=item_sort_key, - for_json=for_json, - ignore_nan=ignore_nan, - int_as_string_bitcount=int_as_string_bitcount, - **kw).iterencode(obj) - # could accelerate with writelines in some versions of Python, at - # a debuggability cost - for chunk in iterable: - fp.write(chunk) - - -def dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True, - allow_nan=True, cls=None, indent=None, separators=None, - encoding='utf-8', default=None, use_decimal=True, - namedtuple_as_object=True, tuple_as_array=True, - bigint_as_string=False, sort_keys=False, item_sort_key=None, - for_json=False, ignore_nan=False, int_as_string_bitcount=None, - iterable_as_array=False, **kw): - """Serialize ``obj`` to a JSON formatted ``str``. - - If ``skipkeys`` is true then ``dict`` keys that are not basic types - (``str``, ``int``, ``long``, ``float``, ``bool``, ``None``) - will be skipped instead of raising a ``TypeError``. - - If *ensure_ascii* is false (default: ``True``), then the output may - contain non-ASCII characters, so long as they do not need to be escaped - by JSON. When it is true, all non-ASCII characters are escaped. - - If ``check_circular`` is false, then the circular reference check - for container types will be skipped and a circular reference will - result in an ``OverflowError`` (or worse). - - If ``allow_nan`` is false, then it will be a ``ValueError`` to - serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``) in - strict compliance of the JSON specification, instead of using the - JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``). - - If ``indent`` is a string, then JSON array elements and object members - will be pretty-printed with a newline followed by that string repeated - for each level of nesting. ``None`` (the default) selects the most compact - representation without any newlines. For backwards compatibility with - versions of simplejson earlier than 2.1.0, an integer is also accepted - and is converted to a string with that many spaces. - - If specified, ``separators`` should be an - ``(item_separator, key_separator)`` tuple. The default is ``(', ', ': ')`` - if *indent* is ``None`` and ``(',', ': ')`` otherwise. To get the most - compact JSON representation, you should specify ``(',', ':')`` to eliminate - whitespace. - - ``encoding`` is the character encoding for bytes instances, default is - UTF-8. - - ``default(obj)`` is a function that should return a serializable version - of obj or raise TypeError. The default simply raises TypeError. - - If *use_decimal* is true (default: ``True``) then decimal.Decimal - will be natively serialized to JSON with full precision. - - If *namedtuple_as_object* is true (default: ``True``), - :class:`tuple` subclasses with ``_asdict()`` methods will be encoded - as JSON objects. - - If *tuple_as_array* is true (default: ``True``), - :class:`tuple` (and subclasses) will be encoded as JSON arrays. - - If *iterable_as_array* is true (default: ``False``), - any object not in the above table that implements ``__iter__()`` - will be encoded as a JSON array. - - If *bigint_as_string* is true (not the default), ints 2**53 and higher - or lower than -2**53 will be encoded as strings. This is to avoid the - rounding that happens in Javascript otherwise. - - If *int_as_string_bitcount* is a positive number (n), then int of size - greater than or equal to 2**n or lower than or equal to -2**n will be - encoded as strings. - - If specified, *item_sort_key* is a callable used to sort the items in - each dictionary. This is useful if you want to sort items other than - in alphabetical order by key. This option takes precedence over - *sort_keys*. - - If *sort_keys* is true (default: ``False``), the output of dictionaries - will be sorted by item. - - If *for_json* is true (default: ``False``), objects with a ``for_json()`` - method will use the return value of that method for encoding as JSON - instead of the object. - - If *ignore_nan* is true (default: ``False``), then out of range - :class:`float` values (``nan``, ``inf``, ``-inf``) will be serialized as - ``null`` in compliance with the ECMA-262 specification. If true, this will - override *allow_nan*. - - To use a custom ``JSONEncoder`` subclass (e.g. one that overrides the - ``.default()`` method to serialize additional types), specify it with - the ``cls`` kwarg. NOTE: You should use *default* instead of subclassing - whenever possible. - - """ - # cached encoder - if (not skipkeys and ensure_ascii and - check_circular and allow_nan and - cls is None and indent is None and separators is None and - encoding == 'utf-8' and default is None and use_decimal - and namedtuple_as_object and tuple_as_array and not iterable_as_array - and not bigint_as_string and not sort_keys - and not item_sort_key and not for_json - and not ignore_nan and int_as_string_bitcount is None - and not kw - ): - return _default_encoder.encode(obj) - if cls is None: - cls = JSONEncoder - return cls( - skipkeys=skipkeys, ensure_ascii=ensure_ascii, - check_circular=check_circular, allow_nan=allow_nan, indent=indent, - separators=separators, encoding=encoding, default=default, - use_decimal=use_decimal, - namedtuple_as_object=namedtuple_as_object, - tuple_as_array=tuple_as_array, - iterable_as_array=iterable_as_array, - bigint_as_string=bigint_as_string, - sort_keys=sort_keys, - item_sort_key=item_sort_key, - for_json=for_json, - ignore_nan=ignore_nan, - int_as_string_bitcount=int_as_string_bitcount, - **kw).encode(obj) - - -_default_decoder = JSONDecoder(encoding=None, object_hook=None, - object_pairs_hook=None) - - -def load(fp, encoding=None, cls=None, object_hook=None, parse_float=None, - parse_int=None, parse_constant=None, object_pairs_hook=None, - use_decimal=False, namedtuple_as_object=True, tuple_as_array=True, - **kw): - """Deserialize ``fp`` (a ``.read()``-supporting file-like object containing - a JSON document as `str` or `bytes`) to a Python object. - - *encoding* determines the encoding used to interpret any - `bytes` objects decoded by this instance (``'utf-8'`` by - default). It has no effect when decoding `str` objects. - - *object_hook*, if specified, will be called with the result of every - JSON object decoded and its return value will be used in place of the - given :class:`dict`. This can be used to provide custom - deserializations (e.g. to support JSON-RPC class hinting). - - *object_pairs_hook* is an optional function that will be called with - the result of any object literal decode with an ordered list of pairs. - The return value of *object_pairs_hook* will be used instead of the - :class:`dict`. This feature can be used to implement custom decoders - that rely on the order that the key and value pairs are decoded (for - example, :func:`collections.OrderedDict` will remember the order of - insertion). If *object_hook* is also defined, the *object_pairs_hook* - takes priority. - - *parse_float*, if specified, will be called with the string of every - JSON float to be decoded. By default, this is equivalent to - ``float(num_str)``. This can be used to use another datatype or parser - for JSON floats (e.g. :class:`decimal.Decimal`). - - *parse_int*, if specified, will be called with the string of every - JSON int to be decoded. By default, this is equivalent to - ``int(num_str)``. This can be used to use another datatype or parser - for JSON integers (e.g. :class:`float`). - - *parse_constant*, if specified, will be called with one of the - following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This - can be used to raise an exception if invalid JSON numbers are - encountered. - - If *use_decimal* is true (default: ``False``) then it implies - parse_float=decimal.Decimal for parity with ``dump``. - - To use a custom ``JSONDecoder`` subclass, specify it with the ``cls`` - kwarg. NOTE: You should use *object_hook* or *object_pairs_hook* instead - of subclassing whenever possible. - - """ - return loads(fp.read(), - encoding=encoding, cls=cls, object_hook=object_hook, - parse_float=parse_float, parse_int=parse_int, - parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, - use_decimal=use_decimal, **kw) - - -def loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, - parse_int=None, parse_constant=None, object_pairs_hook=None, - use_decimal=False, **kw): - """Deserialize ``s`` (a ``str`` or ``unicode`` instance containing a JSON - document) to a Python object. - - *encoding* determines the encoding used to interpret any - :class:`bytes` objects decoded by this instance (``'utf-8'`` by - default). It has no effect when decoding :class:`unicode` objects. - - *object_hook*, if specified, will be called with the result of every - JSON object decoded and its return value will be used in place of the - given :class:`dict`. This can be used to provide custom - deserializations (e.g. to support JSON-RPC class hinting). - - *object_pairs_hook* is an optional function that will be called with - the result of any object literal decode with an ordered list of pairs. - The return value of *object_pairs_hook* will be used instead of the - :class:`dict`. This feature can be used to implement custom decoders - that rely on the order that the key and value pairs are decoded (for - example, :func:`collections.OrderedDict` will remember the order of - insertion). If *object_hook* is also defined, the *object_pairs_hook* - takes priority. - - *parse_float*, if specified, will be called with the string of every - JSON float to be decoded. By default, this is equivalent to - ``float(num_str)``. This can be used to use another datatype or parser - for JSON floats (e.g. :class:`decimal.Decimal`). - - *parse_int*, if specified, will be called with the string of every - JSON int to be decoded. By default, this is equivalent to - ``int(num_str)``. This can be used to use another datatype or parser - for JSON integers (e.g. :class:`float`). - - *parse_constant*, if specified, will be called with one of the - following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This - can be used to raise an exception if invalid JSON numbers are - encountered. - - If *use_decimal* is true (default: ``False``) then it implies - parse_float=decimal.Decimal for parity with ``dump``. - - To use a custom ``JSONDecoder`` subclass, specify it with the ``cls`` - kwarg. NOTE: You should use *object_hook* or *object_pairs_hook* instead - of subclassing whenever possible. - - """ - if (cls is None and encoding is None and object_hook is None and - parse_int is None and parse_float is None and - parse_constant is None and object_pairs_hook is None - and not use_decimal and not kw): - return _default_decoder.decode(s) - if cls is None: - cls = JSONDecoder - if object_hook is not None: - kw['object_hook'] = object_hook - if object_pairs_hook is not None: - kw['object_pairs_hook'] = object_pairs_hook - if parse_float is not None: - kw['parse_float'] = parse_float - if parse_int is not None: - kw['parse_int'] = parse_int - if parse_constant is not None: - kw['parse_constant'] = parse_constant - if use_decimal: - if parse_float is not None: - raise TypeError("use_decimal=True implies parse_float=Decimal") - kw['parse_float'] = Decimal - return cls(encoding=encoding, **kw).decode(s) - - -def _toggle_speedups(enabled): - from . import decoder as dec - from . import encoder as enc - from . import scanner as scan - c_make_encoder = _import_c_make_encoder() - if enabled: - dec.scanstring = dec.c_scanstring or dec.py_scanstring - enc.c_make_encoder = c_make_encoder - enc.encode_basestring_ascii = (enc.c_encode_basestring_ascii or - enc.py_encode_basestring_ascii) - scan.make_scanner = scan.c_make_scanner or scan.py_make_scanner - else: - dec.scanstring = dec.py_scanstring - enc.c_make_encoder = None - enc.encode_basestring_ascii = enc.py_encode_basestring_ascii - scan.make_scanner = scan.py_make_scanner - dec.make_scanner = scan.make_scanner - global _default_decoder - _default_decoder = JSONDecoder( - encoding=None, - object_hook=None, - object_pairs_hook=None, - ) - global _default_encoder - _default_encoder = JSONEncoder( - skipkeys=False, - ensure_ascii=True, - check_circular=True, - allow_nan=True, - indent=None, - separators=None, - encoding='utf-8', - default=None, - ) - -def simple_first(kv): - """Helper function to pass to item_sort_key to sort simple - elements to the top, then container elements. - """ - return (isinstance(kv[1], (list, dict, tuple)), kv[0]) diff --git a/contrib/python/simplejson/simplejson/_speedups.c b/contrib/python/simplejson/simplejson/_speedups.c deleted file mode 100644 index ec054c7293e..00000000000 --- a/contrib/python/simplejson/simplejson/_speedups.c +++ /dev/null @@ -1,3402 +0,0 @@ -/* -*- mode: C; c-file-style: "python"; c-basic-offset: 4 -*- */ -#include "Python.h" -#include "structmember.h" - -#if PY_MAJOR_VERSION >= 3 -#define PyInt_FromSsize_t PyLong_FromSsize_t -#define PyInt_AsSsize_t PyLong_AsSsize_t -#define PyInt_Check(obj) 0 -#define PyInt_CheckExact(obj) 0 -#define JSON_UNICHR Py_UCS4 -#define JSON_InternFromString PyUnicode_InternFromString -#define PyString_GET_SIZE PyUnicode_GET_LENGTH -#define PY2_UNUSED -#define PY3_UNUSED UNUSED -#else /* PY_MAJOR_VERSION >= 3 */ -#define PY2_UNUSED UNUSED -#define PY3_UNUSED -#define PyBytes_Check PyString_Check -#define PyUnicode_READY(obj) 0 -#define PyUnicode_KIND(obj) (sizeof(Py_UNICODE)) -#define PyUnicode_DATA(obj) ((void *)(PyUnicode_AS_UNICODE(obj))) -#define PyUnicode_READ(kind, data, index) ((JSON_UNICHR)((const Py_UNICODE *)(data))[(index)]) -#define PyUnicode_GET_LENGTH PyUnicode_GET_SIZE -#define JSON_UNICHR Py_UNICODE -#define JSON_InternFromString PyString_InternFromString -#endif /* PY_MAJOR_VERSION < 3 */ - -#if PY_VERSION_HEX < 0x03090000 -#if !defined(PyObject_CallNoArgs) -#define PyObject_CallNoArgs(callable) PyObject_CallFunctionObjArgs(callable, NULL); -#endif -#if !defined(PyObject_CallOneArg) -#define PyObject_CallOneArg(callable, arg) PyObject_CallFunctionObjArgs(callable, arg, NULL); -#endif -#endif /* PY_VERSION_HEX < 0x03090000 */ - -#if PY_VERSION_HEX < 0x02070000 -#if !defined(PyOS_string_to_double) -#define PyOS_string_to_double json_PyOS_string_to_double -static double -json_PyOS_string_to_double(const char *s, char **endptr, PyObject *overflow_exception); -static double -json_PyOS_string_to_double(const char *s, char **endptr, PyObject *overflow_exception) -{ - double x; - assert(endptr == NULL); - assert(overflow_exception == NULL); - PyFPE_START_PROTECT("json_PyOS_string_to_double", return -1.0;) - x = PyOS_ascii_atof(s); - PyFPE_END_PROTECT(x) - return x; -} -#endif -#endif /* PY_VERSION_HEX < 0x02070000 */ - -#if PY_VERSION_HEX < 0x02060000 -#if !defined(Py_TYPE) -#define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) -#endif -#if !defined(Py_SIZE) -#define Py_SIZE(ob) (((PyVarObject*)(ob))->ob_size) -#endif -#if !defined(PyVarObject_HEAD_INIT) -#define PyVarObject_HEAD_INIT(type, size) PyObject_HEAD_INIT(type) size, -#endif -#endif /* PY_VERSION_HEX < 0x02060000 */ - -#ifdef __GNUC__ -#define UNUSED __attribute__((__unused__)) -#else -#define UNUSED -#endif - -#define DEFAULT_ENCODING "utf-8" - -#define PyScanner_Check(op) PyObject_TypeCheck(op, &PyScannerType) -#define PyScanner_CheckExact(op) (Py_TYPE(op) == &PyScannerType) -#define PyEncoder_Check(op) PyObject_TypeCheck(op, &PyEncoderType) -#define PyEncoder_CheckExact(op) (Py_TYPE(op) == &PyEncoderType) - -#define JSON_ALLOW_NAN 1 -#define JSON_IGNORE_NAN 2 - -static PyObject *JSON_Infinity = NULL; -static PyObject *JSON_NegInfinity = NULL; -static PyObject *JSON_NaN = NULL; -static PyObject *JSON_EmptyUnicode = NULL; -#if PY_MAJOR_VERSION < 3 -static PyObject *JSON_EmptyStr = NULL; -#endif - -static PyTypeObject PyScannerType; -static PyTypeObject PyEncoderType; - -typedef struct { - PyObject *large_strings; /* A list of previously accumulated large strings */ - PyObject *small_strings; /* Pending small strings */ -} JSON_Accu; - -static int -JSON_Accu_Init(JSON_Accu *acc); -static int -JSON_Accu_Accumulate(JSON_Accu *acc, PyObject *unicode); -static PyObject * -JSON_Accu_FinishAsList(JSON_Accu *acc); -static void -JSON_Accu_Destroy(JSON_Accu *acc); - -#define ERR_EXPECTING_VALUE "Expecting value" -#define ERR_ARRAY_DELIMITER "Expecting ',' delimiter or ']'" -#define ERR_ARRAY_VALUE_FIRST "Expecting value or ']'" -#define ERR_OBJECT_DELIMITER "Expecting ',' delimiter or '}'" -#define ERR_OBJECT_PROPERTY "Expecting property name enclosed in double quotes" -#define ERR_OBJECT_PROPERTY_FIRST "Expecting property name enclosed in double quotes or '}'" -#define ERR_OBJECT_PROPERTY_DELIMITER "Expecting ':' delimiter" -#define ERR_STRING_UNTERMINATED "Unterminated string starting at" -#define ERR_STRING_CONTROL "Invalid control character %r at" -#define ERR_STRING_ESC1 "Invalid \\X escape sequence %r" -#define ERR_STRING_ESC4 "Invalid \\uXXXX escape sequence" -#define FOR_JSON_METHOD_NAME "for_json" -#define ASDICT_METHOD_NAME "_asdict" - - -typedef struct _PyScannerObject { - PyObject_HEAD - PyObject *encoding; - PyObject *strict_bool; - int strict; - PyObject *object_hook; - PyObject *pairs_hook; - PyObject *parse_float; - PyObject *parse_int; - PyObject *parse_constant; - PyObject *memo; -} PyScannerObject; - -static PyMemberDef scanner_members[] = { - {"encoding", T_OBJECT, offsetof(PyScannerObject, encoding), READONLY, "encoding"}, - {"strict", T_OBJECT, offsetof(PyScannerObject, strict_bool), READONLY, "strict"}, - {"object_hook", T_OBJECT, offsetof(PyScannerObject, object_hook), READONLY, "object_hook"}, - {"object_pairs_hook", T_OBJECT, offsetof(PyScannerObject, pairs_hook), READONLY, "object_pairs_hook"}, - {"parse_float", T_OBJECT, offsetof(PyScannerObject, parse_float), READONLY, "parse_float"}, - {"parse_int", T_OBJECT, offsetof(PyScannerObject, parse_int), READONLY, "parse_int"}, - {"parse_constant", T_OBJECT, offsetof(PyScannerObject, parse_constant), READONLY, "parse_constant"}, - {NULL} -}; - -typedef struct _PyEncoderObject { - PyObject_HEAD - PyObject *markers; - PyObject *defaultfn; - PyObject *encoder; - PyObject *indent; - PyObject *key_separator; - PyObject *item_separator; - PyObject *sort_keys; - PyObject *key_memo; - PyObject *encoding; - PyObject *Decimal; - PyObject *skipkeys_bool; - int skipkeys; - int fast_encode; - /* 0, JSON_ALLOW_NAN, JSON_IGNORE_NAN */ - int allow_or_ignore_nan; - int use_decimal; - int namedtuple_as_object; - int tuple_as_array; - int iterable_as_array; - PyObject *max_long_size; - PyObject *min_long_size; - PyObject *item_sort_key; - PyObject *item_sort_kw; - int for_json; -} PyEncoderObject; - -static PyMemberDef encoder_members[] = { - {"markers", T_OBJECT, offsetof(PyEncoderObject, markers), READONLY, "markers"}, - {"default", T_OBJECT, offsetof(PyEncoderObject, defaultfn), READONLY, "default"}, - {"encoder", T_OBJECT, offsetof(PyEncoderObject, encoder), READONLY, "encoder"}, - {"encoding", T_OBJECT, offsetof(PyEncoderObject, encoder), READONLY, "encoding"}, - {"indent", T_OBJECT, offsetof(PyEncoderObject, indent), READONLY, "indent"}, - {"key_separator", T_OBJECT, offsetof(PyEncoderObject, key_separator), READONLY, "key_separator"}, - {"item_separator", T_OBJECT, offsetof(PyEncoderObject, item_separator), READONLY, "item_separator"}, - {"sort_keys", T_OBJECT, offsetof(PyEncoderObject, sort_keys), READONLY, "sort_keys"}, - /* Python 2.5 does not support T_BOOl */ - {"skipkeys", T_OBJECT, offsetof(PyEncoderObject, skipkeys_bool), READONLY, "skipkeys"}, - {"key_memo", T_OBJECT, offsetof(PyEncoderObject, key_memo), READONLY, "key_memo"}, - {"item_sort_key", T_OBJECT, offsetof(PyEncoderObject, item_sort_key), READONLY, "item_sort_key"}, - {"max_long_size", T_OBJECT, offsetof(PyEncoderObject, max_long_size), READONLY, "max_long_size"}, - {"min_long_size", T_OBJECT, offsetof(PyEncoderObject, min_long_size), READONLY, "min_long_size"}, - {NULL} -}; - -static PyObject * -join_list_unicode(PyObject *lst); -static PyObject * -JSON_ParseEncoding(PyObject *encoding); -static PyObject * -maybe_quote_bigint(PyEncoderObject* s, PyObject *encoded, PyObject *obj); -static Py_ssize_t -ascii_char_size(JSON_UNICHR c); -static Py_ssize_t -ascii_escape_char(JSON_UNICHR c, char *output, Py_ssize_t chars); -static PyObject * -ascii_escape_unicode(PyObject *pystr); -static PyObject * -ascii_escape_str(PyObject *pystr); -static PyObject * -py_encode_basestring_ascii(PyObject* self UNUSED, PyObject *pystr); -#if PY_MAJOR_VERSION < 3 -static PyObject * -join_list_string(PyObject *lst); -static PyObject * -scan_once_str(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr); -static PyObject * -scanstring_str(PyObject *pystr, Py_ssize_t end, char *encoding, int strict, Py_ssize_t *next_end_ptr); -static PyObject * -_parse_object_str(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr); -#endif -static PyObject * -scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next_end_ptr); -static PyObject * -scan_once_unicode(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr); -static PyObject * -_build_rval_index_tuple(PyObject *rval, Py_ssize_t idx); -static PyObject * -scanner_new(PyTypeObject *type, PyObject *args, PyObject *kwds); -static void -scanner_dealloc(PyObject *self); -static int -scanner_clear(PyObject *self); -static PyObject * -encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwds); -static void -encoder_dealloc(PyObject *self); -static int -encoder_clear(PyObject *self); -static int -is_raw_json(PyObject *obj); -static PyObject * -encoder_stringify_key(PyEncoderObject *s, PyObject *key); -static int -encoder_listencode_list(PyEncoderObject *s, JSON_Accu *rval, PyObject *seq, Py_ssize_t indent_level); -static int -encoder_listencode_obj(PyEncoderObject *s, JSON_Accu *rval, PyObject *obj, Py_ssize_t indent_level); -static int -encoder_listencode_dict(PyEncoderObject *s, JSON_Accu *rval, PyObject *dct, Py_ssize_t indent_level); -static PyObject * -_encoded_const(PyObject *obj); -static void -raise_errmsg(char *msg, PyObject *s, Py_ssize_t end); -static PyObject * -encoder_encode_string(PyEncoderObject *s, PyObject *obj); -static int -_convertPyInt_AsSsize_t(PyObject *o, Py_ssize_t *size_ptr); -static PyObject * -_convertPyInt_FromSsize_t(Py_ssize_t *size_ptr); -static int -_call_json_method(PyObject *obj, const char *method_name, PyObject **result); -static PyObject * -encoder_encode_float(PyEncoderObject *s, PyObject *obj); -static PyObject * -moduleinit(void); - -#define S_CHAR(c) (c >= ' ' && c <= '~' && c != '\\' && c != '"') -#define IS_WHITESPACE(c) (((c) == ' ') || ((c) == '\t') || ((c) == '\n') || ((c) == '\r')) - -#define MIN_EXPANSION 6 - -static PyObject* RawJSONType = NULL; -static int -is_raw_json(PyObject *obj) -{ - return PyObject_IsInstance(obj, RawJSONType) ? 1 : 0; -} - -static int -JSON_Accu_Init(JSON_Accu *acc) -{ - /* Lazily allocated */ - acc->large_strings = NULL; - acc->small_strings = PyList_New(0); - if (acc->small_strings == NULL) - return -1; - return 0; -} - -static int -flush_accumulator(JSON_Accu *acc) -{ - Py_ssize_t nsmall = PyList_GET_SIZE(acc->small_strings); - if (nsmall) { - int ret; - PyObject *joined; - if (acc->large_strings == NULL) { - acc->large_strings = PyList_New(0); - if (acc->large_strings == NULL) - return -1; - } -#if PY_MAJOR_VERSION >= 3 - joined = join_list_unicode(acc->small_strings); -#else /* PY_MAJOR_VERSION >= 3 */ - joined = join_list_string(acc->small_strings); -#endif /* PY_MAJOR_VERSION < 3 */ - if (joined == NULL) - return -1; - if (PyList_SetSlice(acc->small_strings, 0, nsmall, NULL)) { - Py_DECREF(joined); - return -1; - } - ret = PyList_Append(acc->large_strings, joined); - Py_DECREF(joined); - return ret; - } - return 0; -} - -static int -JSON_Accu_Accumulate(JSON_Accu *acc, PyObject *unicode) -{ - Py_ssize_t nsmall; -#if PY_MAJOR_VERSION >= 3 - assert(PyUnicode_Check(unicode)); -#else /* PY_MAJOR_VERSION >= 3 */ - assert(PyString_Check(unicode) || PyUnicode_Check(unicode)); -#endif /* PY_MAJOR_VERSION < 3 */ - - if (PyList_Append(acc->small_strings, unicode)) - return -1; - nsmall = PyList_GET_SIZE(acc->small_strings); - /* Each item in a list of unicode objects has an overhead (in 64-bit - * builds) of: - * - 8 bytes for the list slot - * - 56 bytes for the header of the unicode object - * that is, 64 bytes. 100000 such objects waste more than 6MB - * compared to a single concatenated string. - */ - if (nsmall < 100000) - return 0; - return flush_accumulator(acc); -} - -static PyObject * -JSON_Accu_FinishAsList(JSON_Accu *acc) -{ - int ret; - PyObject *res; - - ret = flush_accumulator(acc); - Py_CLEAR(acc->small_strings); - if (ret) { - Py_CLEAR(acc->large_strings); - return NULL; - } - res = acc->large_strings; - acc->large_strings = NULL; - if (res == NULL) - return PyList_New(0); - return res; -} - -static void -JSON_Accu_Destroy(JSON_Accu *acc) -{ - Py_CLEAR(acc->small_strings); - Py_CLEAR(acc->large_strings); -} - -static int -IS_DIGIT(JSON_UNICHR c) -{ - return c >= '0' && c <= '9'; -} - -static PyObject * -maybe_quote_bigint(PyEncoderObject* s, PyObject *encoded, PyObject *obj) -{ - if (s->max_long_size != Py_None && s->min_long_size != Py_None) { - if (PyObject_RichCompareBool(obj, s->max_long_size, Py_GE) || - PyObject_RichCompareBool(obj, s->min_long_size, Py_LE)) { -#if PY_MAJOR_VERSION >= 3 - PyObject* quoted = PyUnicode_FromFormat("\"%U\"", encoded); -#else - PyObject* quoted = PyString_FromFormat("\"%s\"", - PyString_AsString(encoded)); -#endif - Py_DECREF(encoded); - encoded = quoted; - } - } - - return encoded; -} - -static int -_call_json_method(PyObject *obj, const char *method_name, PyObject **result) -{ - int rval = 0; - PyObject *method = PyObject_GetAttrString(obj, method_name); - if (method == NULL) { - PyErr_Clear(); - return 0; - } - if (PyCallable_Check(method)) { - PyObject *tmp = PyObject_CallNoArgs(method); - if (tmp == NULL && PyErr_ExceptionMatches(PyExc_TypeError)) { - PyErr_Clear(); - } else { - // This will set result to NULL if a TypeError occurred, - // which must be checked by the caller - *result = tmp; - rval = 1; - } - } - Py_DECREF(method); - return rval; -} - -static int -_convertPyInt_AsSsize_t(PyObject *o, Py_ssize_t *size_ptr) -{ - /* PyObject to Py_ssize_t converter */ - *size_ptr = PyInt_AsSsize_t(o); - if (*size_ptr == -1 && PyErr_Occurred()) - return 0; - return 1; -} - -static PyObject * -_convertPyInt_FromSsize_t(Py_ssize_t *size_ptr) -{ - /* Py_ssize_t to PyObject converter */ - return PyInt_FromSsize_t(*size_ptr); -} - -static Py_ssize_t -ascii_escape_char(JSON_UNICHR c, char *output, Py_ssize_t chars) -{ - /* Escape unicode code point c to ASCII escape sequences - in char *output. output must have at least 12 bytes unused to - accommodate an escaped surrogate pair "\uXXXX\uXXXX" */ - if (S_CHAR(c)) { - output[chars++] = (char)c; - } - else { - output[chars++] = '\\'; - switch (c) { - case '\\': output[chars++] = (char)c; break; - case '"': output[chars++] = (char)c; break; - case '\b': output[chars++] = 'b'; break; - case '\f': output[chars++] = 'f'; break; - case '\n': output[chars++] = 'n'; break; - case '\r': output[chars++] = 'r'; break; - case '\t': output[chars++] = 't'; break; - default: -#if PY_MAJOR_VERSION >= 3 || defined(Py_UNICODE_WIDE) - if (c >= 0x10000) { - /* UTF-16 surrogate pair */ - JSON_UNICHR v = c - 0x10000; - c = 0xd800 | ((v >> 10) & 0x3ff); - output[chars++] = 'u'; - output[chars++] = "0123456789abcdef"[(c >> 12) & 0xf]; - output[chars++] = "0123456789abcdef"[(c >> 8) & 0xf]; - output[chars++] = "0123456789abcdef"[(c >> 4) & 0xf]; - output[chars++] = "0123456789abcdef"[(c ) & 0xf]; - c = 0xdc00 | (v & 0x3ff); - output[chars++] = '\\'; - } -#endif - output[chars++] = 'u'; - output[chars++] = "0123456789abcdef"[(c >> 12) & 0xf]; - output[chars++] = "0123456789abcdef"[(c >> 8) & 0xf]; - output[chars++] = "0123456789abcdef"[(c >> 4) & 0xf]; - output[chars++] = "0123456789abcdef"[(c ) & 0xf]; - } - } - return chars; -} - -static Py_ssize_t -ascii_char_size(JSON_UNICHR c) -{ - if (S_CHAR(c)) { - return 1; - } - else if (c == '\\' || - c == '"' || - c == '\b' || - c == '\f' || - c == '\n' || - c == '\r' || - c == '\t') { - return 2; - } -#if PY_MAJOR_VERSION >= 3 || defined(Py_UNICODE_WIDE) - else if (c >= 0x10000U) { - return 2 * MIN_EXPANSION; - } -#endif - else { - return MIN_EXPANSION; - } -} - -static PyObject * -ascii_escape_unicode(PyObject *pystr) -{ - /* Take a PyUnicode pystr and return a new ASCII-only escaped PyString */ - Py_ssize_t i; - Py_ssize_t input_chars = PyUnicode_GET_LENGTH(pystr); - Py_ssize_t output_size = 2; - Py_ssize_t chars; - PY2_UNUSED int kind = PyUnicode_KIND(pystr); - void *data = PyUnicode_DATA(pystr); - PyObject *rval; - char *output; - - output_size = 2; - for (i = 0; i < input_chars; i++) { - output_size += ascii_char_size(PyUnicode_READ(kind, data, i)); - } -#if PY_MAJOR_VERSION >= 3 - rval = PyUnicode_New(output_size, 127); - if (rval == NULL) { - return NULL; - } - assert(PyUnicode_KIND(rval) == PyUnicode_1BYTE_KIND); - output = (char *)PyUnicode_DATA(rval); -#else - rval = PyString_FromStringAndSize(NULL, output_size); - if (rval == NULL) { - return NULL; - } - output = PyString_AS_STRING(rval); -#endif - chars = 0; - output[chars++] = '"'; - for (i = 0; i < input_chars; i++) { - chars = ascii_escape_char(PyUnicode_READ(kind, data, i), output, chars); - } - output[chars++] = '"'; - assert(chars == output_size); - return rval; -} - -#if PY_MAJOR_VERSION >= 3 - -static PyObject * -ascii_escape_str(PyObject *pystr) -{ - PyObject *rval; - PyObject *input = PyUnicode_DecodeUTF8(PyBytes_AS_STRING(pystr), PyBytes_GET_SIZE(pystr), NULL); - if (input == NULL) - return NULL; - rval = ascii_escape_unicode(input); - Py_DECREF(input); - return rval; -} - -#else /* PY_MAJOR_VERSION >= 3 */ - -static PyObject * -ascii_escape_str(PyObject *pystr) -{ - /* Take a PyString pystr and return a new ASCII-only escaped PyString */ - Py_ssize_t i; - Py_ssize_t input_chars; - Py_ssize_t output_size; - Py_ssize_t chars; - PyObject *rval; - char *output; - char *input_str; - - input_chars = PyString_GET_SIZE(pystr); - input_str = PyString_AS_STRING(pystr); - output_size = 2; - - /* Fast path for a string that's already ASCII */ - for (i = 0; i < input_chars; i++) { - JSON_UNICHR c = (JSON_UNICHR)input_str[i]; - if (c > 0x7f) { - /* We hit a non-ASCII character, bail to unicode mode */ - PyObject *uni; - uni = PyUnicode_DecodeUTF8(input_str, input_chars, "strict"); - if (uni == NULL) { - return NULL; - } - rval = ascii_escape_unicode(uni); - Py_DECREF(uni); - return rval; - } - output_size += ascii_char_size(c); - } - - rval = PyString_FromStringAndSize(NULL, output_size); - if (rval == NULL) { - return NULL; - } - chars = 0; - output = PyString_AS_STRING(rval); - output[chars++] = '"'; - for (i = 0; i < input_chars; i++) { - chars = ascii_escape_char((JSON_UNICHR)input_str[i], output, chars); - } - output[chars++] = '"'; - assert(chars == output_size); - return rval; -} -#endif /* PY_MAJOR_VERSION < 3 */ - -static PyObject * -encoder_stringify_key(PyEncoderObject *s, PyObject *key) -{ - if (PyUnicode_Check(key)) { - Py_INCREF(key); - return key; - } -#if PY_MAJOR_VERSION >= 3 - else if (PyBytes_Check(key) && s->encoding != NULL) { - const char *encoding = PyUnicode_AsUTF8(s->encoding); - if (encoding == NULL) - return NULL; - return PyUnicode_Decode( - PyBytes_AS_STRING(key), - PyBytes_GET_SIZE(key), - encoding, - NULL); - } -#else /* PY_MAJOR_VERSION >= 3 */ - else if (PyString_Check(key)) { - Py_INCREF(key); - return key; - } -#endif /* PY_MAJOR_VERSION < 3 */ - else if (PyFloat_Check(key)) { - return encoder_encode_float(s, key); - } - else if (key == Py_True || key == Py_False || key == Py_None) { - /* This must come before the PyInt_Check because - True and False are also 1 and 0.*/ - return _encoded_const(key); - } - else if (PyInt_Check(key) || PyLong_Check(key)) { - if (!(PyInt_CheckExact(key) || PyLong_CheckExact(key))) { - /* See #118, do not trust custom str/repr */ - PyObject *res; - PyObject *tmp = PyObject_CallOneArg((PyObject *)&PyLong_Type, key); - if (tmp == NULL) { - return NULL; - } - res = PyObject_Str(tmp); - Py_DECREF(tmp); - return res; - } - else { - return PyObject_Str(key); - } - } - else if (s->use_decimal && PyObject_TypeCheck(key, (PyTypeObject *)s->Decimal)) { - return PyObject_Str(key); - } - if (s->skipkeys) { - Py_INCREF(Py_None); - return Py_None; - } - PyErr_Format(PyExc_TypeError, - "keys must be str, int, float, bool or None, " - "not %.100s", key->ob_type->tp_name); - return NULL; -} - -static PyObject * -encoder_dict_iteritems(PyEncoderObject *s, PyObject *dct) -{ - PyObject *items; - PyObject *iter = NULL; - PyObject *lst = NULL; - PyObject *item = NULL; - PyObject *kstr = NULL; - PyObject *sortfun = NULL; - PyObject *sortres; - static PyObject *sortargs = NULL; - - if (sortargs == NULL) { - sortargs = PyTuple_New(0); - if (sortargs == NULL) - return NULL; - } - - if (PyDict_CheckExact(dct)) - items = PyDict_Items(dct); - else - items = PyMapping_Items(dct); - if (items == NULL) - return NULL; - iter = PyObject_GetIter(items); - Py_DECREF(items); - if (iter == NULL) - return NULL; - if (s->item_sort_kw == Py_None) - return iter; - lst = PyList_New(0); - if (lst == NULL) - goto bail; - while ((item = PyIter_Next(iter))) { - PyObject *key, *value; - if (!PyTuple_Check(item) || Py_SIZE(item) != 2) { - PyErr_SetString(PyExc_ValueError, "items must return 2-tuples"); - goto bail; - } - key = PyTuple_GET_ITEM(item, 0); - if (key == NULL) - goto bail; -#if PY_MAJOR_VERSION < 3 - else if (PyString_Check(key)) { - /* item can be added as-is */ - } -#endif /* PY_MAJOR_VERSION < 3 */ - else if (PyUnicode_Check(key)) { - /* item can be added as-is */ - } - else { - PyObject *tpl; - kstr = encoder_stringify_key(s, key); - if (kstr == NULL) - goto bail; - else if (kstr == Py_None) { - /* skipkeys */ - Py_DECREF(kstr); - continue; - } - value = PyTuple_GET_ITEM(item, 1); - if (value == NULL) - goto bail; - tpl = PyTuple_Pack(2, kstr, value); - if (tpl == NULL) - goto bail; - Py_CLEAR(kstr); - Py_DECREF(item); - item = tpl; - } - if (PyList_Append(lst, item)) - goto bail; - Py_DECREF(item); - } - Py_CLEAR(iter); - if (PyErr_Occurred()) - goto bail; - sortfun = PyObject_GetAttrString(lst, "sort"); - if (sortfun == NULL) - goto bail; - sortres = PyObject_Call(sortfun, sortargs, s->item_sort_kw); - if (!sortres) - goto bail; - Py_DECREF(sortres); - Py_CLEAR(sortfun); - iter = PyObject_GetIter(lst); - Py_CLEAR(lst); - return iter; -bail: - Py_XDECREF(sortfun); - Py_XDECREF(kstr); - Py_XDECREF(item); - Py_XDECREF(lst); - Py_XDECREF(iter); - return NULL; -} - -/* Use JSONDecodeError exception to raise a nice looking ValueError subclass */ -static PyObject *JSONDecodeError = NULL; -static void -raise_errmsg(char *msg, PyObject *s, Py_ssize_t end) -{ - PyObject *exc = PyObject_CallFunction(JSONDecodeError, "(zOO&)", msg, s, _convertPyInt_FromSsize_t, &end); - if (exc) { - PyErr_SetObject(JSONDecodeError, exc); - Py_DECREF(exc); - } -} - -static PyObject * -join_list_unicode(PyObject *lst) -{ - /* return u''.join(lst) */ - return PyUnicode_Join(JSON_EmptyUnicode, lst); -} - -#if PY_MAJOR_VERSION >= 3 -#define join_list_string join_list_unicode -#else /* PY_MAJOR_VERSION >= 3 */ -static PyObject * -join_list_string(PyObject *lst) -{ - /* return ''.join(lst) */ - static PyObject *joinfn = NULL; - if (joinfn == NULL) { - joinfn = PyObject_GetAttrString(JSON_EmptyStr, "join"); - if (joinfn == NULL) - return NULL; - } - return PyObject_CallOneArg(joinfn, lst); -} -#endif /* PY_MAJOR_VERSION < 3 */ - -static PyObject * -_build_rval_index_tuple(PyObject *rval, Py_ssize_t idx) -{ - /* return (rval, idx) tuple, stealing reference to rval */ - PyObject *tpl; - PyObject *pyidx; - /* - steal a reference to rval, returns (rval, idx) - */ - if (rval == NULL) { - assert(PyErr_Occurred()); - return NULL; - } - pyidx = PyInt_FromSsize_t(idx); - if (pyidx == NULL) { - Py_DECREF(rval); - return NULL; - } - tpl = PyTuple_New(2); - if (tpl == NULL) { - Py_DECREF(pyidx); - Py_DECREF(rval); - return NULL; - } - PyTuple_SET_ITEM(tpl, 0, rval); - PyTuple_SET_ITEM(tpl, 1, pyidx); - return tpl; -} - -#define APPEND_OLD_CHUNK \ - if (chunk != NULL) { \ - if (chunks == NULL) { \ - chunks = PyList_New(0); \ - if (chunks == NULL) { \ - goto bail; \ - } \ - } \ - if (PyList_Append(chunks, chunk)) { \ - goto bail; \ - } \ - Py_CLEAR(chunk); \ - } - -#if PY_MAJOR_VERSION < 3 -static PyObject * -scanstring_str(PyObject *pystr, Py_ssize_t end, char *encoding, int strict, Py_ssize_t *next_end_ptr) -{ - /* Read the JSON string from PyString pystr. - end is the index of the first character after the quote. - encoding is the encoding of pystr (must be an ASCII superset) - if strict is zero then literal control characters are allowed - *next_end_ptr is a return-by-reference index of the character - after the end quote - - Return value is a new PyString (if ASCII-only) or PyUnicode - */ - PyObject *rval; - Py_ssize_t len = PyString_GET_SIZE(pystr); - Py_ssize_t begin = end - 1; - Py_ssize_t next = begin; - int has_unicode = 0; - char *buf = PyString_AS_STRING(pystr); - PyObject *chunks = NULL; - PyObject *chunk = NULL; - PyObject *strchunk = NULL; - - if (len == end) { - raise_errmsg(ERR_STRING_UNTERMINATED, pystr, begin); - goto bail; - } - else if (end < 0 || len < end) { - PyErr_SetString(PyExc_ValueError, "end is out of bounds"); - goto bail; - } - while (1) { - /* Find the end of the string or the next escape */ - Py_UNICODE c = 0; - for (next = end; next < len; next++) { - c = (unsigned char)buf[next]; - if (c == '"' || c == '\\') { - break; - } - else if (strict && c <= 0x1f) { - raise_errmsg(ERR_STRING_CONTROL, pystr, next); - goto bail; - } - else if (c > 0x7f) { - has_unicode = 1; - } - } - if (!(c == '"' || c == '\\')) { - raise_errmsg(ERR_STRING_UNTERMINATED, pystr, begin); - goto bail; - } - /* Pick up this chunk if it's not zero length */ - if (next != end) { - APPEND_OLD_CHUNK - strchunk = PyString_FromStringAndSize(&buf[end], next - end); - if (strchunk == NULL) { - goto bail; - } - if (has_unicode) { - chunk = PyUnicode_FromEncodedObject(strchunk, encoding, NULL); - Py_DECREF(strchunk); - if (chunk == NULL) { - goto bail; - } - } - else { - chunk = strchunk; - } - } - next++; - if (c == '"') { - end = next; - break; - } - if (next == len) { - raise_errmsg(ERR_STRING_UNTERMINATED, pystr, begin); - goto bail; - } - c = buf[next]; - if (c != 'u') { - /* Non-unicode backslash escapes */ - end = next + 1; - switch (c) { - case '"': break; - case '\\': break; - case '/': break; - case 'b': c = '\b'; break; - case 'f': c = '\f'; break; - case 'n': c = '\n'; break; - case 'r': c = '\r'; break; - case 't': c = '\t'; break; - default: c = 0; - } - if (c == 0) { - raise_errmsg(ERR_STRING_ESC1, pystr, end - 2); - goto bail; - } - } - else { - c = 0; - next++; - end = next + 4; - if (end >= len) { - raise_errmsg(ERR_STRING_ESC4, pystr, next - 1); - goto bail; - } - /* Decode 4 hex digits */ - for (; next < end; next++) { - JSON_UNICHR digit = (JSON_UNICHR)buf[next]; - c <<= 4; - switch (digit) { - case '0': case '1': case '2': case '3': case '4': - case '5': case '6': case '7': case '8': case '9': - c |= (digit - '0'); break; - case 'a': case 'b': case 'c': case 'd': case 'e': - case 'f': - c |= (digit - 'a' + 10); break; - case 'A': case 'B': case 'C': case 'D': case 'E': - case 'F': - c |= (digit - 'A' + 10); break; - default: - raise_errmsg(ERR_STRING_ESC4, pystr, end - 5); - goto bail; - } - } -#if defined(Py_UNICODE_WIDE) - /* Surrogate pair */ - if ((c & 0xfc00) == 0xd800) { - if (end + 6 < len && buf[next] == '\\' && buf[next+1] == 'u') { - JSON_UNICHR c2 = 0; - end += 6; - /* Decode 4 hex digits */ - for (next += 2; next < end; next++) { - c2 <<= 4; - JSON_UNICHR digit = buf[next]; - switch (digit) { - case '0': case '1': case '2': case '3': case '4': - case '5': case '6': case '7': case '8': case '9': - c2 |= (digit - '0'); break; - case 'a': case 'b': case 'c': case 'd': case 'e': - case 'f': - c2 |= (digit - 'a' + 10); break; - case 'A': case 'B': case 'C': case 'D': case 'E': - case 'F': - c2 |= (digit - 'A' + 10); break; - default: - raise_errmsg(ERR_STRING_ESC4, pystr, end - 5); - goto bail; - } - } - if ((c2 & 0xfc00) != 0xdc00) { - /* not a low surrogate, rewind */ - end -= 6; - next = end; - } - else { - c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); - } - } - } -#endif /* Py_UNICODE_WIDE */ - } - if (c > 0x7f) { - has_unicode = 1; - } - APPEND_OLD_CHUNK - if (has_unicode) { - chunk = PyUnicode_FromOrdinal(c); - if (chunk == NULL) { - goto bail; - } - } - else { - char c_char = Py_CHARMASK(c); - chunk = PyString_FromStringAndSize(&c_char, 1); - if (chunk == NULL) { - goto bail; - } - } - } - - if (chunks == NULL) { - if (chunk != NULL) - rval = chunk; - else { - rval = JSON_EmptyStr; - Py_INCREF(rval); - } - } - else { - APPEND_OLD_CHUNK - rval = join_list_string(chunks); - if (rval == NULL) { - goto bail; - } - Py_CLEAR(chunks); - } - - *next_end_ptr = end; - return rval; -bail: - *next_end_ptr = -1; - Py_XDECREF(chunk); - Py_XDECREF(chunks); - return NULL; -} -#endif /* PY_MAJOR_VERSION < 3 */ - -static PyObject * -scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next_end_ptr) -{ - /* Read the JSON string from PyUnicode pystr. - end is the index of the first character after the quote. - if strict is zero then literal control characters are allowed - *next_end_ptr is a return-by-reference index of the character - after the end quote - - Return value is a new PyUnicode - */ - PyObject *rval; - Py_ssize_t begin = end - 1; - Py_ssize_t next = begin; - PY2_UNUSED int kind = PyUnicode_KIND(pystr); - Py_ssize_t len = PyUnicode_GET_LENGTH(pystr); - void *buf = PyUnicode_DATA(pystr); - PyObject *chunks = NULL; - PyObject *chunk = NULL; - - if (len == end) { - raise_errmsg(ERR_STRING_UNTERMINATED, pystr, begin); - goto bail; - } - else if (end < 0 || len < end) { - PyErr_SetString(PyExc_ValueError, "end is out of bounds"); - goto bail; - } - while (1) { - /* Find the end of the string or the next escape */ - JSON_UNICHR c = 0; - for (next = end; next < len; next++) { - c = PyUnicode_READ(kind, buf, next); - if (c == '"' || c == '\\') { - break; - } - else if (strict && c <= 0x1f) { - raise_errmsg(ERR_STRING_CONTROL, pystr, next); - goto bail; - } - } - if (!(c == '"' || c == '\\')) { - raise_errmsg(ERR_STRING_UNTERMINATED, pystr, begin); - goto bail; - } - /* Pick up this chunk if it's not zero length */ - if (next != end) { - APPEND_OLD_CHUNK -#if PY_MAJOR_VERSION < 3 - chunk = PyUnicode_FromUnicode(&((const Py_UNICODE *)buf)[end], next - end); -#else - chunk = PyUnicode_Substring(pystr, end, next); -#endif - if (chunk == NULL) { - goto bail; - } - } - next++; - if (c == '"') { - end = next; - break; - } - if (next == len) { - raise_errmsg(ERR_STRING_UNTERMINATED, pystr, begin); - goto bail; - } - c = PyUnicode_READ(kind, buf, next); - if (c != 'u') { - /* Non-unicode backslash escapes */ - end = next + 1; - switch (c) { - case '"': break; - case '\\': break; - case '/': break; - case 'b': c = '\b'; break; - case 'f': c = '\f'; break; - case 'n': c = '\n'; break; - case 'r': c = '\r'; break; - case 't': c = '\t'; break; - default: c = 0; - } - if (c == 0) { - raise_errmsg(ERR_STRING_ESC1, pystr, end - 2); - goto bail; - } - } - else { - c = 0; - next++; - end = next + 4; - if (end >= len) { - raise_errmsg(ERR_STRING_ESC4, pystr, next - 1); - goto bail; - } - /* Decode 4 hex digits */ - for (; next < end; next++) { - JSON_UNICHR digit = PyUnicode_READ(kind, buf, next); - c <<= 4; - switch (digit) { - case '0': case '1': case '2': case '3': case '4': - case '5': case '6': case '7': case '8': case '9': - c |= (digit - '0'); break; - case 'a': case 'b': case 'c': case 'd': case 'e': - case 'f': - c |= (digit - 'a' + 10); break; - case 'A': case 'B': case 'C': case 'D': case 'E': - case 'F': - c |= (digit - 'A' + 10); break; - default: - raise_errmsg(ERR_STRING_ESC4, pystr, end - 5); - goto bail; - } - } -#if PY_MAJOR_VERSION >= 3 || defined(Py_UNICODE_WIDE) - /* Surrogate pair */ - if ((c & 0xfc00) == 0xd800) { - JSON_UNICHR c2 = 0; - if (end + 6 < len && - PyUnicode_READ(kind, buf, next) == '\\' && - PyUnicode_READ(kind, buf, next + 1) == 'u') { - end += 6; - /* Decode 4 hex digits */ - for (next += 2; next < end; next++) { - JSON_UNICHR digit = PyUnicode_READ(kind, buf, next); - c2 <<= 4; - switch (digit) { - case '0': case '1': case '2': case '3': case '4': - case '5': case '6': case '7': case '8': case '9': - c2 |= (digit - '0'); break; - case 'a': case 'b': case 'c': case 'd': case 'e': - case 'f': - c2 |= (digit - 'a' + 10); break; - case 'A': case 'B': case 'C': case 'D': case 'E': - case 'F': - c2 |= (digit - 'A' + 10); break; - default: - raise_errmsg(ERR_STRING_ESC4, pystr, end - 5); - goto bail; - } - } - if ((c2 & 0xfc00) != 0xdc00) { - /* not a low surrogate, rewind */ - end -= 6; - next = end; - } - else { - c = 0x10000 + (((c - 0xd800) << 10) | (c2 - 0xdc00)); - } - } - } -#endif - } - APPEND_OLD_CHUNK - chunk = PyUnicode_FromOrdinal(c); - if (chunk == NULL) { - goto bail; - } - } - - if (chunks == NULL) { - if (chunk != NULL) - rval = chunk; - else { - rval = JSON_EmptyUnicode; - Py_INCREF(rval); - } - } - else { - APPEND_OLD_CHUNK - rval = join_list_unicode(chunks); - if (rval == NULL) { - goto bail; - } - Py_CLEAR(chunks); - } - *next_end_ptr = end; - return rval; -bail: - *next_end_ptr = -1; - Py_XDECREF(chunk); - Py_XDECREF(chunks); - return NULL; -} - -PyDoc_STRVAR(pydoc_scanstring, - "scanstring(basestring, end, encoding, strict=True) -> (str, end)\n" - "\n" - "Scan the string s for a JSON string. End is the index of the\n" - "character in s after the quote that started the JSON string.\n" - "Unescapes all valid JSON string escape sequences and raises ValueError\n" - "on attempt to decode an invalid string. If strict is False then literal\n" - "control characters are allowed in the string.\n" - "\n" - "Returns a tuple of the decoded string and the index of the character in s\n" - "after the end quote." -); - -static PyObject * -py_scanstring(PyObject* self UNUSED, PyObject *args) -{ - PyObject *pystr; - PyObject *rval; - Py_ssize_t end; - Py_ssize_t next_end = -1; - char *encoding = NULL; - int strict = 1; - if (!PyArg_ParseTuple(args, "OO&|zi:scanstring", &pystr, _convertPyInt_AsSsize_t, &end, &encoding, &strict)) { - return NULL; - } - if (encoding == NULL) { - encoding = DEFAULT_ENCODING; - } - if (PyUnicode_Check(pystr)) { - if (PyUnicode_READY(pystr)) - return NULL; - rval = scanstring_unicode(pystr, end, strict, &next_end); - } -#if PY_MAJOR_VERSION < 3 - /* Using a bytes input is unsupported for scanning in Python 3. - It is coerced to str in the decoder before it gets here. */ - else if (PyString_Check(pystr)) { - rval = scanstring_str(pystr, end, encoding, strict, &next_end); - } -#endif - else { - PyErr_Format(PyExc_TypeError, - "first argument must be a string, not %.80s", - Py_TYPE(pystr)->tp_name); - return NULL; - } - return _build_rval_index_tuple(rval, next_end); -} - -PyDoc_STRVAR(pydoc_encode_basestring_ascii, - "encode_basestring_ascii(basestring) -> str\n" - "\n" - "Return an ASCII-only JSON representation of a Python string" -); - -static PyObject * -py_encode_basestring_ascii(PyObject* self UNUSED, PyObject *pystr) -{ - /* Return an ASCII-only JSON representation of a Python string */ - /* METH_O */ - if (PyBytes_Check(pystr)) { - return ascii_escape_str(pystr); - } - else if (PyUnicode_Check(pystr)) { - if (PyUnicode_READY(pystr)) - return NULL; - return ascii_escape_unicode(pystr); - } - else { - PyErr_Format(PyExc_TypeError, - "first argument must be a string, not %.80s", - Py_TYPE(pystr)->tp_name); - return NULL; - } -} - -static void -scanner_dealloc(PyObject *self) -{ - /* bpo-31095: UnTrack is needed before calling any callbacks */ - PyObject_GC_UnTrack(self); - scanner_clear(self); - Py_TYPE(self)->tp_free(self); -} - -static int -scanner_traverse(PyObject *self, visitproc visit, void *arg) -{ - PyScannerObject *s; - assert(PyScanner_Check(self)); - s = (PyScannerObject *)self; - Py_VISIT(s->encoding); - Py_VISIT(s->strict_bool); - Py_VISIT(s->object_hook); - Py_VISIT(s->pairs_hook); - Py_VISIT(s->parse_float); - Py_VISIT(s->parse_int); - Py_VISIT(s->parse_constant); - Py_VISIT(s->memo); - return 0; -} - -static int -scanner_clear(PyObject *self) -{ - PyScannerObject *s; - assert(PyScanner_Check(self)); - s = (PyScannerObject *)self; - Py_CLEAR(s->encoding); - Py_CLEAR(s->strict_bool); - Py_CLEAR(s->object_hook); - Py_CLEAR(s->pairs_hook); - Py_CLEAR(s->parse_float); - Py_CLEAR(s->parse_int); - Py_CLEAR(s->parse_constant); - Py_CLEAR(s->memo); - return 0; -} - -#if PY_MAJOR_VERSION < 3 -static PyObject * -_parse_object_str(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON object from PyString pystr. - idx is the index of the first character after the opening curly brace. - *next_idx_ptr is a return-by-reference index to the first character after - the closing curly brace. - - Returns a new PyObject (usually a dict, but object_hook or - object_pairs_hook can change that) - */ - char *str = PyString_AS_STRING(pystr); - Py_ssize_t end_idx = PyString_GET_SIZE(pystr) - 1; - PyObject *rval = NULL; - PyObject *pairs = NULL; - PyObject *item; - PyObject *key = NULL; - PyObject *val = NULL; - char *encoding = PyString_AS_STRING(s->encoding); - int has_pairs_hook = (s->pairs_hook != Py_None); - int did_parse = 0; - Py_ssize_t next_idx; - if (has_pairs_hook) { - pairs = PyList_New(0); - if (pairs == NULL) - return NULL; - } - else { - rval = PyDict_New(); - if (rval == NULL) - return NULL; - } - - /* skip whitespace after { */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - - /* only loop if the object is non-empty */ - if (idx <= end_idx && str[idx] != '}') { - int trailing_delimiter = 0; - while (idx <= end_idx) { - PyObject *memokey; - trailing_delimiter = 0; - - /* read key */ - if (str[idx] != '"') { - raise_errmsg(ERR_OBJECT_PROPERTY, pystr, idx); - goto bail; - } - key = scanstring_str(pystr, idx + 1, encoding, s->strict, &next_idx); - if (key == NULL) - goto bail; - memokey = PyDict_GetItem(s->memo, key); - if (memokey != NULL) { - Py_INCREF(memokey); - Py_DECREF(key); - key = memokey; - } - else { - if (PyDict_SetItem(s->memo, key, key) < 0) - goto bail; - } - idx = next_idx; - - /* skip whitespace between key and : delimiter, read :, skip whitespace */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - if (idx > end_idx || str[idx] != ':') { - raise_errmsg(ERR_OBJECT_PROPERTY_DELIMITER, pystr, idx); - goto bail; - } - idx++; - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - - /* read any JSON data type */ - val = scan_once_str(s, pystr, idx, &next_idx); - if (val == NULL) - goto bail; - - if (has_pairs_hook) { - item = PyTuple_Pack(2, key, val); - if (item == NULL) - goto bail; - Py_CLEAR(key); - Py_CLEAR(val); - if (PyList_Append(pairs, item) == -1) { - Py_DECREF(item); - goto bail; - } - Py_DECREF(item); - } - else { - if (PyDict_SetItem(rval, key, val) < 0) - goto bail; - Py_CLEAR(key); - Py_CLEAR(val); - } - idx = next_idx; - - /* skip whitespace before } or , */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - - /* bail if the object is closed or we didn't get the , delimiter */ - did_parse = 1; - if (idx > end_idx) break; - if (str[idx] == '}') { - break; - } - else if (str[idx] != ',') { - raise_errmsg(ERR_OBJECT_DELIMITER, pystr, idx); - goto bail; - } - idx++; - - /* skip whitespace after , delimiter */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - trailing_delimiter = 1; - } - if (trailing_delimiter) { - raise_errmsg(ERR_OBJECT_PROPERTY, pystr, idx); - goto bail; - } - } - /* verify that idx < end_idx, str[idx] should be '}' */ - if (idx > end_idx || str[idx] != '}') { - if (did_parse) { - raise_errmsg(ERR_OBJECT_DELIMITER, pystr, idx); - } else { - raise_errmsg(ERR_OBJECT_PROPERTY_FIRST, pystr, idx); - } - goto bail; - } - - /* if pairs_hook is not None: rval = object_pairs_hook(pairs) */ - if (s->pairs_hook != Py_None) { - val = PyObject_CallOneArg(s->pairs_hook, pairs); - if (val == NULL) - goto bail; - Py_DECREF(pairs); - *next_idx_ptr = idx + 1; - return val; - } - - /* if object_hook is not None: rval = object_hook(rval) */ - if (s->object_hook != Py_None) { - val = PyObject_CallOneArg(s->object_hook, rval); - if (val == NULL) - goto bail; - Py_DECREF(rval); - rval = val; - val = NULL; - } - *next_idx_ptr = idx + 1; - return rval; -bail: - Py_XDECREF(rval); - Py_XDECREF(key); - Py_XDECREF(val); - Py_XDECREF(pairs); - return NULL; -} -#endif /* PY_MAJOR_VERSION < 3 */ - -static PyObject * -_parse_object_unicode(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON object from PyUnicode pystr. - idx is the index of the first character after the opening curly brace. - *next_idx_ptr is a return-by-reference index to the first character after - the closing curly brace. - - Returns a new PyObject (usually a dict, but object_hook can change that) - */ - void *str = PyUnicode_DATA(pystr); - Py_ssize_t end_idx = PyUnicode_GET_LENGTH(pystr) - 1; - PY2_UNUSED int kind = PyUnicode_KIND(pystr); - PyObject *rval = NULL; - PyObject *pairs = NULL; - PyObject *item; - PyObject *key = NULL; - PyObject *val = NULL; - int has_pairs_hook = (s->pairs_hook != Py_None); - int did_parse = 0; - Py_ssize_t next_idx; - - if (has_pairs_hook) { - pairs = PyList_New(0); - if (pairs == NULL) - return NULL; - } - else { - rval = PyDict_New(); - if (rval == NULL) - return NULL; - } - - /* skip whitespace after { */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - - /* only loop if the object is non-empty */ - if (idx <= end_idx && PyUnicode_READ(kind, str, idx) != '}') { - int trailing_delimiter = 0; - while (idx <= end_idx) { - PyObject *memokey; - trailing_delimiter = 0; - - /* read key */ - if (PyUnicode_READ(kind, str, idx) != '"') { - raise_errmsg(ERR_OBJECT_PROPERTY, pystr, idx); - goto bail; - } - key = scanstring_unicode(pystr, idx + 1, s->strict, &next_idx); - if (key == NULL) - goto bail; - memokey = PyDict_GetItem(s->memo, key); - if (memokey != NULL) { - Py_INCREF(memokey); - Py_DECREF(key); - key = memokey; - } - else { - if (PyDict_SetItem(s->memo, key, key) < 0) - goto bail; - } - idx = next_idx; - - /* skip whitespace between key and : delimiter, read :, skip - whitespace */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - if (idx > end_idx || PyUnicode_READ(kind, str, idx) != ':') { - raise_errmsg(ERR_OBJECT_PROPERTY_DELIMITER, pystr, idx); - goto bail; - } - idx++; - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - - /* read any JSON term */ - val = scan_once_unicode(s, pystr, idx, &next_idx); - if (val == NULL) - goto bail; - - if (has_pairs_hook) { - item = PyTuple_Pack(2, key, val); - if (item == NULL) - goto bail; - Py_CLEAR(key); - Py_CLEAR(val); - if (PyList_Append(pairs, item) == -1) { - Py_DECREF(item); - goto bail; - } - Py_DECREF(item); - } - else { - if (PyDict_SetItem(rval, key, val) < 0) - goto bail; - Py_CLEAR(key); - Py_CLEAR(val); - } - idx = next_idx; - - /* skip whitespace before } or , */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - - /* bail if the object is closed or we didn't get the , - delimiter */ - did_parse = 1; - if (idx > end_idx) break; - if (PyUnicode_READ(kind, str, idx) == '}') { - break; - } - else if (PyUnicode_READ(kind, str, idx) != ',') { - raise_errmsg(ERR_OBJECT_DELIMITER, pystr, idx); - goto bail; - } - idx++; - - /* skip whitespace after , delimiter */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - trailing_delimiter = 1; - } - if (trailing_delimiter) { - raise_errmsg(ERR_OBJECT_PROPERTY, pystr, idx); - goto bail; - } - } - - /* verify that idx < end_idx, str[idx] should be '}' */ - if (idx > end_idx || PyUnicode_READ(kind, str, idx) != '}') { - if (did_parse) { - raise_errmsg(ERR_OBJECT_DELIMITER, pystr, idx); - } else { - raise_errmsg(ERR_OBJECT_PROPERTY_FIRST, pystr, idx); - } - goto bail; - } - - /* if pairs_hook is not None: rval = object_pairs_hook(pairs) */ - if (s->pairs_hook != Py_None) { - val = PyObject_CallOneArg(s->pairs_hook, pairs); - if (val == NULL) - goto bail; - Py_DECREF(pairs); - *next_idx_ptr = idx + 1; - return val; - } - - /* if object_hook is not None: rval = object_hook(rval) */ - if (s->object_hook != Py_None) { - val = PyObject_CallOneArg(s->object_hook, rval); - if (val == NULL) - goto bail; - Py_DECREF(rval); - rval = val; - val = NULL; - } - *next_idx_ptr = idx + 1; - return rval; -bail: - Py_XDECREF(rval); - Py_XDECREF(key); - Py_XDECREF(val); - Py_XDECREF(pairs); - return NULL; -} - -#if PY_MAJOR_VERSION < 3 -static PyObject * -_parse_array_str(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON array from PyString pystr. - idx is the index of the first character after the opening brace. - *next_idx_ptr is a return-by-reference index to the first character after - the closing brace. - - Returns a new PyList - */ - char *str = PyString_AS_STRING(pystr); - Py_ssize_t end_idx = PyString_GET_SIZE(pystr) - 1; - PyObject *val = NULL; - PyObject *rval = PyList_New(0); - Py_ssize_t next_idx; - if (rval == NULL) - return NULL; - - /* skip whitespace after [ */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - - /* only loop if the array is non-empty */ - if (idx <= end_idx && str[idx] != ']') { - int trailing_delimiter = 0; - while (idx <= end_idx) { - trailing_delimiter = 0; - /* read any JSON term and de-tuplefy the (rval, idx) */ - val = scan_once_str(s, pystr, idx, &next_idx); - if (val == NULL) { - goto bail; - } - - if (PyList_Append(rval, val) == -1) - goto bail; - - Py_CLEAR(val); - idx = next_idx; - - /* skip whitespace between term and , */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - - /* bail if the array is closed or we didn't get the , delimiter */ - if (idx > end_idx) break; - if (str[idx] == ']') { - break; - } - else if (str[idx] != ',') { - raise_errmsg(ERR_ARRAY_DELIMITER, pystr, idx); - goto bail; - } - idx++; - - /* skip whitespace after , */ - while (idx <= end_idx && IS_WHITESPACE(str[idx])) idx++; - trailing_delimiter = 1; - } - if (trailing_delimiter) { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - goto bail; - } - } - - /* verify that idx < end_idx, str[idx] should be ']' */ - if (idx > end_idx || str[idx] != ']') { - if (PyList_GET_SIZE(rval)) { - raise_errmsg(ERR_ARRAY_DELIMITER, pystr, idx); - } else { - raise_errmsg(ERR_ARRAY_VALUE_FIRST, pystr, idx); - } - goto bail; - } - *next_idx_ptr = idx + 1; - return rval; -bail: - Py_XDECREF(val); - Py_DECREF(rval); - return NULL; -} -#endif /* PY_MAJOR_VERSION < 3 */ - -static PyObject * -_parse_array_unicode(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON array from PyString pystr. - idx is the index of the first character after the opening brace. - *next_idx_ptr is a return-by-reference index to the first character after - the closing brace. - - Returns a new PyList - */ - PY2_UNUSED int kind = PyUnicode_KIND(pystr); - void *str = PyUnicode_DATA(pystr); - Py_ssize_t end_idx = PyUnicode_GET_LENGTH(pystr) - 1; - PyObject *val = NULL; - PyObject *rval = PyList_New(0); - Py_ssize_t next_idx; - if (rval == NULL) - return NULL; - - /* skip whitespace after [ */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - - /* only loop if the array is non-empty */ - if (idx <= end_idx && PyUnicode_READ(kind, str, idx) != ']') { - int trailing_delimiter = 0; - while (idx <= end_idx) { - trailing_delimiter = 0; - /* read any JSON term */ - val = scan_once_unicode(s, pystr, idx, &next_idx); - if (val == NULL) { - goto bail; - } - - if (PyList_Append(rval, val) == -1) - goto bail; - - Py_CLEAR(val); - idx = next_idx; - - /* skip whitespace between term and , */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - - /* bail if the array is closed or we didn't get the , delimiter */ - if (idx > end_idx) break; - if (PyUnicode_READ(kind, str, idx) == ']') { - break; - } - else if (PyUnicode_READ(kind, str, idx) != ',') { - raise_errmsg(ERR_ARRAY_DELIMITER, pystr, idx); - goto bail; - } - idx++; - - /* skip whitespace after , */ - while (idx <= end_idx && IS_WHITESPACE(PyUnicode_READ(kind, str, idx))) idx++; - trailing_delimiter = 1; - } - if (trailing_delimiter) { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - goto bail; - } - } - - /* verify that idx < end_idx, str[idx] should be ']' */ - if (idx > end_idx || PyUnicode_READ(kind, str, idx) != ']') { - if (PyList_GET_SIZE(rval)) { - raise_errmsg(ERR_ARRAY_DELIMITER, pystr, idx); - } else { - raise_errmsg(ERR_ARRAY_VALUE_FIRST, pystr, idx); - } - goto bail; - } - *next_idx_ptr = idx + 1; - return rval; -bail: - Py_XDECREF(val); - Py_DECREF(rval); - return NULL; -} - -static PyObject * -_parse_constant(PyScannerObject *s, PyObject *constant, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON constant from PyString pystr. - constant is the Python string that was found - ("NaN", "Infinity", "-Infinity"). - idx is the index of the first character of the constant - *next_idx_ptr is a return-by-reference index to the first character after - the constant. - - Returns the result of parse_constant - */ - PyObject *rval; - - /* rval = parse_constant(constant) */ - rval = PyObject_CallOneArg(s->parse_constant, constant); - idx += PyString_GET_SIZE(constant); - *next_idx_ptr = idx; - return rval; -} - -#if PY_MAJOR_VERSION < 3 -static PyObject * -_match_number_str(PyScannerObject *s, PyObject *pystr, Py_ssize_t start, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON number from PyString pystr. - idx is the index of the first character of the number - *next_idx_ptr is a return-by-reference index to the first character after - the number. - - Returns a new PyObject representation of that number: - PyInt, PyLong, or PyFloat. - May return other types if parse_int or parse_float are set - */ - char *str = PyString_AS_STRING(pystr); - Py_ssize_t end_idx = PyString_GET_SIZE(pystr) - 1; - Py_ssize_t idx = start; - int is_float = 0; - PyObject *rval; - PyObject *numstr; - - /* read a sign if it's there, make sure it's not the end of the string */ - if (str[idx] == '-') { - if (idx >= end_idx) { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - return NULL; - } - idx++; - } - - /* read as many integer digits as we find as long as it doesn't start with 0 */ - if (str[idx] >= '1' && str[idx] <= '9') { - idx++; - while (idx <= end_idx && str[idx] >= '0' && str[idx] <= '9') idx++; - } - /* if it starts with 0 we only expect one integer digit */ - else if (str[idx] == '0') { - idx++; - } - /* no integer digits, error */ - else { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - return NULL; - } - - /* if the next char is '.' followed by a digit then read all float digits */ - if (idx < end_idx && str[idx] == '.' && str[idx + 1] >= '0' && str[idx + 1] <= '9') { - is_float = 1; - idx += 2; - while (idx <= end_idx && str[idx] >= '0' && str[idx] <= '9') idx++; - } - - /* if the next char is 'e' or 'E' then maybe read the exponent (or backtrack) */ - if (idx < end_idx && (str[idx] == 'e' || str[idx] == 'E')) { - - /* save the index of the 'e' or 'E' just in case we need to backtrack */ - Py_ssize_t e_start = idx; - idx++; - - /* read an exponent sign if present */ - if (idx < end_idx && (str[idx] == '-' || str[idx] == '+')) idx++; - - /* read all digits */ - while (idx <= end_idx && str[idx] >= '0' && str[idx] <= '9') idx++; - - /* if we got a digit, then parse as float. if not, backtrack */ - if (str[idx - 1] >= '0' && str[idx - 1] <= '9') { - is_float = 1; - } - else { - idx = e_start; - } - } - - /* copy the section we determined to be a number */ - numstr = PyString_FromStringAndSize(&str[start], idx - start); - if (numstr == NULL) - return NULL; - if (is_float) { - /* parse as a float using a fast path if available, otherwise call user defined method */ - if (s->parse_float != (PyObject *)&PyFloat_Type) { - rval = PyObject_CallOneArg(s->parse_float, numstr); - } - else { - /* rval = PyFloat_FromDouble(PyOS_ascii_atof(PyString_AS_STRING(numstr))); */ - double d = PyOS_string_to_double(PyString_AS_STRING(numstr), - NULL, NULL); - if (d == -1.0 && PyErr_Occurred()) - return NULL; - rval = PyFloat_FromDouble(d); - } - } - else { - /* parse as an int using a fast path if available, otherwise call user defined method */ - if (s->parse_int != (PyObject *)&PyInt_Type) { - rval = PyObject_CallOneArg(s->parse_int, numstr); - } - else { - rval = PyInt_FromString(PyString_AS_STRING(numstr), NULL, 10); - } - } - Py_DECREF(numstr); - *next_idx_ptr = idx; - return rval; -} -#endif /* PY_MAJOR_VERSION < 3 */ - -static PyObject * -_match_number_unicode(PyScannerObject *s, PyObject *pystr, Py_ssize_t start, Py_ssize_t *next_idx_ptr) -{ - /* Read a JSON number from PyUnicode pystr. - idx is the index of the first character of the number - *next_idx_ptr is a return-by-reference index to the first character after - the number. - - Returns a new PyObject representation of that number: - PyInt, PyLong, or PyFloat. - May return other types if parse_int or parse_float are set - */ - PY2_UNUSED int kind = PyUnicode_KIND(pystr); - void *str = PyUnicode_DATA(pystr); - Py_ssize_t end_idx = PyUnicode_GET_LENGTH(pystr) - 1; - Py_ssize_t idx = start; - int is_float = 0; - JSON_UNICHR c; - PyObject *rval; - PyObject *numstr; - - /* read a sign if it's there, make sure it's not the end of the string */ - if (PyUnicode_READ(kind, str, idx) == '-') { - if (idx >= end_idx) { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - return NULL; - } - idx++; - } - - /* read as many integer digits as we find as long as it doesn't start with 0 */ - c = PyUnicode_READ(kind, str, idx); - if (c == '0') { - /* if it starts with 0 we only expect one integer digit */ - idx++; - } - else if (IS_DIGIT(c)) { - idx++; - while (idx <= end_idx && IS_DIGIT(PyUnicode_READ(kind, str, idx))) { - idx++; - } - } - else { - /* no integer digits, error */ - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - return NULL; - } - - /* if the next char is '.' followed by a digit then read all float digits */ - if (idx < end_idx && - PyUnicode_READ(kind, str, idx) == '.' && - IS_DIGIT(PyUnicode_READ(kind, str, idx + 1))) { - is_float = 1; - idx += 2; - while (idx <= end_idx && IS_DIGIT(PyUnicode_READ(kind, str, idx))) idx++; - } - - /* if the next char is 'e' or 'E' then maybe read the exponent (or backtrack) */ - if (idx < end_idx && - (PyUnicode_READ(kind, str, idx) == 'e' || - PyUnicode_READ(kind, str, idx) == 'E')) { - Py_ssize_t e_start = idx; - idx++; - - /* read an exponent sign if present */ - if (idx < end_idx && - (PyUnicode_READ(kind, str, idx) == '-' || - PyUnicode_READ(kind, str, idx) == '+')) idx++; - - /* read all digits */ - while (idx <= end_idx && IS_DIGIT(PyUnicode_READ(kind, str, idx))) idx++; - - /* if we got a digit, then parse as float. if not, backtrack */ - if (IS_DIGIT(PyUnicode_READ(kind, str, idx - 1))) { - is_float = 1; - } - else { - idx = e_start; - } - } - - /* copy the section we determined to be a number */ -#if PY_MAJOR_VERSION >= 3 - numstr = PyUnicode_Substring(pystr, start, idx); -#else - numstr = PyUnicode_FromUnicode(&((Py_UNICODE *)str)[start], idx - start); -#endif - if (numstr == NULL) - return NULL; - if (is_float) { - /* parse as a float using a fast path if available, otherwise call user defined method */ - if (s->parse_float != (PyObject *)&PyFloat_Type) { - rval = PyObject_CallOneArg(s->parse_float, numstr); - } - else { -#if PY_MAJOR_VERSION >= 3 - rval = PyFloat_FromString(numstr); -#else - rval = PyFloat_FromString(numstr, NULL); -#endif - } - } - else { - /* no fast path for unicode -> int, just call */ - rval = PyObject_CallOneArg(s->parse_int, numstr); - } - Py_DECREF(numstr); - *next_idx_ptr = idx; - return rval; -} - -#if PY_MAJOR_VERSION < 3 -static PyObject * -scan_once_str(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read one JSON term (of any kind) from PyString pystr. - idx is the index of the first character of the term - *next_idx_ptr is a return-by-reference index to the first character after - the number. - - Returns a new PyObject representation of the term. - */ - char *str = PyString_AS_STRING(pystr); - Py_ssize_t length = PyString_GET_SIZE(pystr); - PyObject *rval = NULL; - int fallthrough = 0; - if (idx < 0 || idx >= length) { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - return NULL; - } - switch (str[idx]) { - case '"': - /* string */ - rval = scanstring_str(pystr, idx + 1, - PyString_AS_STRING(s->encoding), - s->strict, - next_idx_ptr); - break; - case '{': - /* object */ - if (Py_EnterRecursiveCall(" while decoding a JSON object " - "from a string")) - return NULL; - rval = _parse_object_str(s, pystr, idx + 1, next_idx_ptr); - Py_LeaveRecursiveCall(); - break; - case '[': - /* array */ - if (Py_EnterRecursiveCall(" while decoding a JSON array " - "from a string")) - return NULL; - rval = _parse_array_str(s, pystr, idx + 1, next_idx_ptr); - Py_LeaveRecursiveCall(); - break; - case 'n': - /* null */ - if ((idx + 3 < length) && str[idx + 1] == 'u' && str[idx + 2] == 'l' && str[idx + 3] == 'l') { - Py_INCREF(Py_None); - *next_idx_ptr = idx + 4; - rval = Py_None; - } - else - fallthrough = 1; - break; - case 't': - /* true */ - if ((idx + 3 < length) && str[idx + 1] == 'r' && str[idx + 2] == 'u' && str[idx + 3] == 'e') { - Py_INCREF(Py_True); - *next_idx_ptr = idx + 4; - rval = Py_True; - } - else - fallthrough = 1; - break; - case 'f': - /* false */ - if ((idx + 4 < length) && str[idx + 1] == 'a' && str[idx + 2] == 'l' && str[idx + 3] == 's' && str[idx + 4] == 'e') { - Py_INCREF(Py_False); - *next_idx_ptr = idx + 5; - rval = Py_False; - } - else - fallthrough = 1; - break; - case 'N': - /* NaN */ - if ((idx + 2 < length) && str[idx + 1] == 'a' && str[idx + 2] == 'N') { - rval = _parse_constant(s, JSON_NaN, idx, next_idx_ptr); - } - else - fallthrough = 1; - break; - case 'I': - /* Infinity */ - if ((idx + 7 < length) && str[idx + 1] == 'n' && str[idx + 2] == 'f' && str[idx + 3] == 'i' && str[idx + 4] == 'n' && str[idx + 5] == 'i' && str[idx + 6] == 't' && str[idx + 7] == 'y') { - rval = _parse_constant(s, JSON_Infinity, idx, next_idx_ptr); - } - else - fallthrough = 1; - break; - case '-': - /* -Infinity */ - if ((idx + 8 < length) && str[idx + 1] == 'I' && str[idx + 2] == 'n' && str[idx + 3] == 'f' && str[idx + 4] == 'i' && str[idx + 5] == 'n' && str[idx + 6] == 'i' && str[idx + 7] == 't' && str[idx + 8] == 'y') { - rval = _parse_constant(s, JSON_NegInfinity, idx, next_idx_ptr); - } - else - fallthrough = 1; - break; - default: - fallthrough = 1; - } - /* Didn't find a string, object, array, or named constant. Look for a number. */ - if (fallthrough) - rval = _match_number_str(s, pystr, idx, next_idx_ptr); - return rval; -} -#endif /* PY_MAJOR_VERSION < 3 */ - - -static PyObject * -scan_once_unicode(PyScannerObject *s, PyObject *pystr, Py_ssize_t idx, Py_ssize_t *next_idx_ptr) -{ - /* Read one JSON term (of any kind) from PyUnicode pystr. - idx is the index of the first character of the term - *next_idx_ptr is a return-by-reference index to the first character after - the number. - - Returns a new PyObject representation of the term. - */ - PY2_UNUSED int kind = PyUnicode_KIND(pystr); - void *str = PyUnicode_DATA(pystr); - Py_ssize_t length = PyUnicode_GET_LENGTH(pystr); - PyObject *rval = NULL; - int fallthrough = 0; - if (idx < 0 || idx >= length) { - raise_errmsg(ERR_EXPECTING_VALUE, pystr, idx); - return NULL; - } - switch (PyUnicode_READ(kind, str, idx)) { - case '"': - /* string */ - rval = scanstring_unicode(pystr, idx + 1, - s->strict, - next_idx_ptr); - break; - case '{': - /* object */ - if (Py_EnterRecursiveCall(" while decoding a JSON object " - "from a unicode string")) - return NULL; - rval = _parse_object_unicode(s, pystr, idx + 1, next_idx_ptr); - Py_LeaveRecursiveCall(); - break; - case '[': - /* array */ - if (Py_EnterRecursiveCall(" while decoding a JSON array " - "from a unicode string")) - return NULL; - rval = _parse_array_unicode(s, pystr, idx + 1, next_idx_ptr); - Py_LeaveRecursiveCall(); - break; - case 'n': - /* null */ - if ((idx + 3 < length) && - PyUnicode_READ(kind, str, idx + 1) == 'u' && - PyUnicode_READ(kind, str, idx + 2) == 'l' && - PyUnicode_READ(kind, str, idx + 3) == 'l') { - Py_INCREF(Py_None); - *next_idx_ptr = idx + 4; - rval = Py_None; - } - else - fallthrough = 1; - break; - case 't': - /* true */ - if ((idx + 3 < length) && - PyUnicode_READ(kind, str, idx + 1) == 'r' && - PyUnicode_READ(kind, str, idx + 2) == 'u' && - PyUnicode_READ(kind, str, idx + 3) == 'e') { - Py_INCREF(Py_True); - *next_idx_ptr = idx + 4; - rval = Py_True; - } - else - fallthrough = 1; - break; - case 'f': - /* false */ - if ((idx + 4 < length) && - PyUnicode_READ(kind, str, idx + 1) == 'a' && - PyUnicode_READ(kind, str, idx + 2) == 'l' && - PyUnicode_READ(kind, str, idx + 3) == 's' && - PyUnicode_READ(kind, str, idx + 4) == 'e') { - Py_INCREF(Py_False); - *next_idx_ptr = idx + 5; - rval = Py_False; - } - else - fallthrough = 1; - break; - case 'N': - /* NaN */ - if ((idx + 2 < length) && - PyUnicode_READ(kind, str, idx + 1) == 'a' && - PyUnicode_READ(kind, str, idx + 2) == 'N') { - rval = _parse_constant(s, JSON_NaN, idx, next_idx_ptr); - } - else - fallthrough = 1; - break; - case 'I': - /* Infinity */ - if ((idx + 7 < length) && - PyUnicode_READ(kind, str, idx + 1) == 'n' && - PyUnicode_READ(kind, str, idx + 2) == 'f' && - PyUnicode_READ(kind, str, idx + 3) == 'i' && - PyUnicode_READ(kind, str, idx + 4) == 'n' && - PyUnicode_READ(kind, str, idx + 5) == 'i' && - PyUnicode_READ(kind, str, idx + 6) == 't' && - PyUnicode_READ(kind, str, idx + 7) == 'y') { - rval = _parse_constant(s, JSON_Infinity, idx, next_idx_ptr); - } - else - fallthrough = 1; - break; - case '-': - /* -Infinity */ - if ((idx + 8 < length) && - PyUnicode_READ(kind, str, idx + 1) == 'I' && - PyUnicode_READ(kind, str, idx + 2) == 'n' && - PyUnicode_READ(kind, str, idx + 3) == 'f' && - PyUnicode_READ(kind, str, idx + 4) == 'i' && - PyUnicode_READ(kind, str, idx + 5) == 'n' && - PyUnicode_READ(kind, str, idx + 6) == 'i' && - PyUnicode_READ(kind, str, idx + 7) == 't' && - PyUnicode_READ(kind, str, idx + 8) == 'y') { - rval = _parse_constant(s, JSON_NegInfinity, idx, next_idx_ptr); - } - else - fallthrough = 1; - break; - default: - fallthrough = 1; - } - /* Didn't find a string, object, array, or named constant. Look for a number. */ - if (fallthrough) - rval = _match_number_unicode(s, pystr, idx, next_idx_ptr); - return rval; -} - -static PyObject * -scanner_call(PyObject *self, PyObject *args, PyObject *kwds) -{ - /* Python callable interface to scan_once_{str,unicode} */ - PyObject *pystr; - PyObject *rval; - Py_ssize_t idx; - Py_ssize_t next_idx = -1; - static char *kwlist[] = {"string", "idx", NULL}; - PyScannerObject *s; - assert(PyScanner_Check(self)); - s = (PyScannerObject *)self; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO&:scan_once", kwlist, &pystr, _convertPyInt_AsSsize_t, &idx)) - return NULL; - - if (PyUnicode_Check(pystr)) { - if (PyUnicode_READY(pystr)) - return NULL; - rval = scan_once_unicode(s, pystr, idx, &next_idx); - } -#if PY_MAJOR_VERSION < 3 - else if (PyString_Check(pystr)) { - rval = scan_once_str(s, pystr, idx, &next_idx); - } -#endif /* PY_MAJOR_VERSION < 3 */ - else { - PyErr_Format(PyExc_TypeError, - "first argument must be a string, not %.80s", - Py_TYPE(pystr)->tp_name); - return NULL; - } - PyDict_Clear(s->memo); - return _build_rval_index_tuple(rval, next_idx); -} - -static PyObject * -JSON_ParseEncoding(PyObject *encoding) -{ - if (encoding == Py_None) - return JSON_InternFromString(DEFAULT_ENCODING); -#if PY_MAJOR_VERSION >= 3 - if (PyUnicode_Check(encoding)) { - if (PyUnicode_AsUTF8(encoding) == NULL) { - return NULL; - } - Py_INCREF(encoding); - return encoding; - } -#else /* PY_MAJOR_VERSION >= 3 */ - if (PyString_Check(encoding)) { - Py_INCREF(encoding); - return encoding; - } - if (PyUnicode_Check(encoding)) - return PyUnicode_AsEncodedString(encoding, NULL, NULL); -#endif /* PY_MAJOR_VERSION >= 3 */ - PyErr_SetString(PyExc_TypeError, "encoding must be a string"); - return NULL; -} - -static PyObject * -scanner_new(PyTypeObject *type, PyObject *args, PyObject *kwds) -{ - /* Initialize Scanner object */ - PyObject *ctx; - static char *kwlist[] = {"context", NULL}; - PyScannerObject *s; - PyObject *encoding; - - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O:make_scanner", kwlist, &ctx)) - return NULL; - - s = (PyScannerObject *)type->tp_alloc(type, 0); - if (s == NULL) - return NULL; - - if (s->memo == NULL) { - s->memo = PyDict_New(); - if (s->memo == NULL) - goto bail; - } - - encoding = PyObject_GetAttrString(ctx, "encoding"); - if (encoding == NULL) - goto bail; - s->encoding = JSON_ParseEncoding(encoding); - Py_XDECREF(encoding); - if (s->encoding == NULL) - goto bail; - - /* All of these will fail "gracefully" so we don't need to verify them */ - s->strict_bool = PyObject_GetAttrString(ctx, "strict"); - if (s->strict_bool == NULL) - goto bail; - s->strict = PyObject_IsTrue(s->strict_bool); - if (s->strict < 0) - goto bail; - s->object_hook = PyObject_GetAttrString(ctx, "object_hook"); - if (s->object_hook == NULL) - goto bail; - s->pairs_hook = PyObject_GetAttrString(ctx, "object_pairs_hook"); - if (s->pairs_hook == NULL) - goto bail; - s->parse_float = PyObject_GetAttrString(ctx, "parse_float"); - if (s->parse_float == NULL) - goto bail; - s->parse_int = PyObject_GetAttrString(ctx, "parse_int"); - if (s->parse_int == NULL) - goto bail; - s->parse_constant = PyObject_GetAttrString(ctx, "parse_constant"); - if (s->parse_constant == NULL) - goto bail; - - return (PyObject *)s; - -bail: - Py_DECREF(s); - return NULL; -} - -PyDoc_STRVAR(scanner_doc, "JSON scanner object"); - -static -PyTypeObject PyScannerType = { - PyVarObject_HEAD_INIT(NULL, 0) - "simplejson._speedups.Scanner", /* tp_name */ - sizeof(PyScannerObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - scanner_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - scanner_call, /* tp_call */ - 0, /* tp_str */ - 0,/* PyObject_GenericGetAttr, */ /* tp_getattro */ - 0,/* PyObject_GenericSetAttr, */ /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - scanner_doc, /* tp_doc */ - scanner_traverse, /* tp_traverse */ - scanner_clear, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - scanner_members, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0,/* PyType_GenericAlloc, */ /* tp_alloc */ - scanner_new, /* tp_new */ - 0,/* PyObject_GC_Del, */ /* tp_free */ -}; - -static PyObject * -encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwds) -{ - static char *kwlist[] = { - "markers", - "default", - "encoder", - "indent", - "key_separator", - "item_separator", - "sort_keys", - "skipkeys", - "allow_nan", - "key_memo", - "use_decimal", - "namedtuple_as_object", - "tuple_as_array", - "int_as_string_bitcount", - "item_sort_key", - "encoding", - "for_json", - "ignore_nan", - "Decimal", - "iterable_as_array", - NULL}; - - PyEncoderObject *s; - PyObject *markers, *defaultfn, *encoder, *indent, *key_separator; - PyObject *item_separator, *sort_keys, *skipkeys, *allow_nan, *key_memo; - PyObject *use_decimal, *namedtuple_as_object, *tuple_as_array, *iterable_as_array; - PyObject *int_as_string_bitcount, *item_sort_key, *encoding, *for_json; - PyObject *ignore_nan, *Decimal; - int is_true; - - if (!PyArg_ParseTupleAndKeywords(args, kwds, "OOOOOOOOOOOOOOOOOOOO:make_encoder", kwlist, - &markers, &defaultfn, &encoder, &indent, &key_separator, &item_separator, - &sort_keys, &skipkeys, &allow_nan, &key_memo, &use_decimal, - &namedtuple_as_object, &tuple_as_array, - &int_as_string_bitcount, &item_sort_key, &encoding, &for_json, - &ignore_nan, &Decimal, &iterable_as_array)) - return NULL; - - s = (PyEncoderObject *)type->tp_alloc(type, 0); - if (s == NULL) - return NULL; - - Py_INCREF(markers); - s->markers = markers; - Py_INCREF(defaultfn); - s->defaultfn = defaultfn; - Py_INCREF(encoder); - s->encoder = encoder; -#if PY_MAJOR_VERSION >= 3 - if (encoding == Py_None) { - s->encoding = NULL; - } - else -#endif /* PY_MAJOR_VERSION >= 3 */ - { - s->encoding = JSON_ParseEncoding(encoding); - if (s->encoding == NULL) - goto bail; - } - Py_INCREF(indent); - s->indent = indent; - Py_INCREF(key_separator); - s->key_separator = key_separator; - Py_INCREF(item_separator); - s->item_separator = item_separator; - Py_INCREF(skipkeys); - s->skipkeys_bool = skipkeys; - s->skipkeys = PyObject_IsTrue(skipkeys); - if (s->skipkeys < 0) - goto bail; - Py_INCREF(key_memo); - s->key_memo = key_memo; - s->fast_encode = (PyCFunction_Check(s->encoder) && PyCFunction_GetFunction(s->encoder) == (PyCFunction)py_encode_basestring_ascii); - is_true = PyObject_IsTrue(ignore_nan); - if (is_true < 0) - goto bail; - s->allow_or_ignore_nan = is_true ? JSON_IGNORE_NAN : 0; - is_true = PyObject_IsTrue(allow_nan); - if (is_true < 0) - goto bail; - s->allow_or_ignore_nan |= is_true ? JSON_ALLOW_NAN : 0; - s->use_decimal = PyObject_IsTrue(use_decimal); - if (s->use_decimal < 0) - goto bail; - s->namedtuple_as_object = PyObject_IsTrue(namedtuple_as_object); - if (s->namedtuple_as_object < 0) - goto bail; - s->tuple_as_array = PyObject_IsTrue(tuple_as_array); - if (s->tuple_as_array < 0) - goto bail; - s->iterable_as_array = PyObject_IsTrue(iterable_as_array); - if (s->iterable_as_array < 0) - goto bail; - if (PyInt_Check(int_as_string_bitcount) || PyLong_Check(int_as_string_bitcount)) { - static const unsigned long long_long_bitsize = SIZEOF_LONG_LONG * 8; - long int_as_string_bitcount_val = PyLong_AsLong(int_as_string_bitcount); - if (int_as_string_bitcount_val > 0 && int_as_string_bitcount_val < (long)long_long_bitsize) { - s->max_long_size = PyLong_FromUnsignedLongLong(1ULL << (int)int_as_string_bitcount_val); - s->min_long_size = PyLong_FromLongLong(-1LL << (int)int_as_string_bitcount_val); - if (s->min_long_size == NULL || s->max_long_size == NULL) { - goto bail; - } - } - else { - PyErr_Format(PyExc_TypeError, - "int_as_string_bitcount (%ld) must be greater than 0 and less than the number of bits of a `long long` type (%lu bits)", - int_as_string_bitcount_val, long_long_bitsize); - goto bail; - } - } - else if (int_as_string_bitcount == Py_None) { - Py_INCREF(Py_None); - s->max_long_size = Py_None; - Py_INCREF(Py_None); - s->min_long_size = Py_None; - } - else { - PyErr_SetString(PyExc_TypeError, "int_as_string_bitcount must be None or an integer"); - goto bail; - } - if (item_sort_key != Py_None) { - if (!PyCallable_Check(item_sort_key)) { - PyErr_SetString(PyExc_TypeError, "item_sort_key must be None or callable"); - goto bail; - } - } - else { - is_true = PyObject_IsTrue(sort_keys); - if (is_true < 0) - goto bail; - if (is_true) { - static PyObject *itemgetter0 = NULL; - if (!itemgetter0) { - PyObject *operator = PyImport_ImportModule("operator"); - if (!operator) - goto bail; - itemgetter0 = PyObject_CallMethod(operator, "itemgetter", "i", 0); - Py_DECREF(operator); - } - item_sort_key = itemgetter0; - if (!item_sort_key) - goto bail; - } - } - if (item_sort_key == Py_None) { - Py_INCREF(Py_None); - s->item_sort_kw = Py_None; - } - else { - s->item_sort_kw = PyDict_New(); - if (s->item_sort_kw == NULL) - goto bail; - if (PyDict_SetItemString(s->item_sort_kw, "key", item_sort_key)) - goto bail; - } - Py_INCREF(sort_keys); - s->sort_keys = sort_keys; - Py_INCREF(item_sort_key); - s->item_sort_key = item_sort_key; - Py_INCREF(Decimal); - s->Decimal = Decimal; - s->for_json = PyObject_IsTrue(for_json); - if (s->for_json < 0) - goto bail; - - return (PyObject *)s; - -bail: - Py_DECREF(s); - return NULL; -} - -static PyObject * -encoder_call(PyObject *self, PyObject *args, PyObject *kwds) -{ - /* Python callable interface to encode_listencode_obj */ - static char *kwlist[] = {"obj", "_current_indent_level", NULL}; - PyObject *obj; - Py_ssize_t indent_level; - PyEncoderObject *s; - JSON_Accu rval; - assert(PyEncoder_Check(self)); - s = (PyEncoderObject *)self; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO&:_iterencode", kwlist, - &obj, _convertPyInt_AsSsize_t, &indent_level)) - return NULL; - if (JSON_Accu_Init(&rval)) - return NULL; - if (encoder_listencode_obj(s, &rval, obj, indent_level)) { - JSON_Accu_Destroy(&rval); - return NULL; - } - return JSON_Accu_FinishAsList(&rval); -} - -static PyObject * -_encoded_const(PyObject *obj) -{ - /* Return the JSON string representation of None, True, False */ - if (obj == Py_None) { - static PyObject *s_null = NULL; - if (s_null == NULL) { - s_null = JSON_InternFromString("null"); - } - Py_INCREF(s_null); - return s_null; - } - else if (obj == Py_True) { - static PyObject *s_true = NULL; - if (s_true == NULL) { - s_true = JSON_InternFromString("true"); - } - Py_INCREF(s_true); - return s_true; - } - else if (obj == Py_False) { - static PyObject *s_false = NULL; - if (s_false == NULL) { - s_false = JSON_InternFromString("false"); - } - Py_INCREF(s_false); - return s_false; - } - else { - PyErr_SetString(PyExc_ValueError, "not a const"); - return NULL; - } -} - -static PyObject * -encoder_encode_float(PyEncoderObject *s, PyObject *obj) -{ - /* Return the JSON representation of a PyFloat */ - double i = PyFloat_AS_DOUBLE(obj); - if (!Py_IS_FINITE(i)) { - if (!s->allow_or_ignore_nan) { - PyErr_SetString(PyExc_ValueError, "Out of range float values are not JSON compliant"); - return NULL; - } - if (s->allow_or_ignore_nan & JSON_IGNORE_NAN) { - return _encoded_const(Py_None); - } - /* JSON_ALLOW_NAN is set */ - else if (i > 0) { - Py_INCREF(JSON_Infinity); - return JSON_Infinity; - } - else if (i < 0) { - Py_INCREF(JSON_NegInfinity); - return JSON_NegInfinity; - } - else { - Py_INCREF(JSON_NaN); - return JSON_NaN; - } - } - /* Use a better float format here? */ - if (PyFloat_CheckExact(obj)) { - return PyObject_Repr(obj); - } - else { - /* See #118, do not trust custom str/repr */ - PyObject *res; - PyObject *tmp = PyObject_CallOneArg((PyObject *)&PyFloat_Type, obj); - if (tmp == NULL) { - return NULL; - } - res = PyObject_Repr(tmp); - Py_DECREF(tmp); - return res; - } -} - -static PyObject * -encoder_encode_string(PyEncoderObject *s, PyObject *obj) -{ - /* Return the JSON representation of a string */ - PyObject *encoded; - - if (s->fast_encode) { - return py_encode_basestring_ascii(NULL, obj); - } - encoded = PyObject_CallOneArg(s->encoder, obj); - if (encoded != NULL && -#if PY_MAJOR_VERSION < 3 - !PyString_Check(encoded) && -#endif /* PY_MAJOR_VERSION < 3 */ - !PyUnicode_Check(encoded)) - { - PyErr_Format(PyExc_TypeError, - "encoder() must return a string, not %.80s", - Py_TYPE(encoded)->tp_name); - Py_DECREF(encoded); - return NULL; - } - return encoded; -} - -static int -_steal_accumulate(JSON_Accu *accu, PyObject *stolen) -{ - /* Append stolen and then decrement its reference count */ - int rval = JSON_Accu_Accumulate(accu, stolen); - Py_DECREF(stolen); - return rval; -} - -static int -encoder_listencode_obj(PyEncoderObject *s, JSON_Accu *rval, PyObject *obj, Py_ssize_t indent_level) -{ - /* Encode Python object obj to a JSON term, rval is a PyList */ - int rv = -1; - do { - PyObject *newobj; - if (obj == Py_None || obj == Py_True || obj == Py_False) { - PyObject *cstr = _encoded_const(obj); - if (cstr != NULL) - rv = _steal_accumulate(rval, cstr); - } - else if ((PyBytes_Check(obj) && s->encoding != NULL) || - PyUnicode_Check(obj)) - { - PyObject *encoded = encoder_encode_string(s, obj); - if (encoded != NULL) - rv = _steal_accumulate(rval, encoded); - } - else if (PyInt_Check(obj) || PyLong_Check(obj)) { - PyObject *encoded; - if (PyInt_CheckExact(obj) || PyLong_CheckExact(obj)) { - encoded = PyObject_Str(obj); - } - else { - /* See #118, do not trust custom str/repr */ - PyObject *tmp = PyObject_CallOneArg((PyObject *)&PyLong_Type, obj); - if (tmp == NULL) { - encoded = NULL; - } - else { - encoded = PyObject_Str(tmp); - Py_DECREF(tmp); - } - } - if (encoded != NULL) { - encoded = maybe_quote_bigint(s, encoded, obj); - if (encoded == NULL) - break; - rv = _steal_accumulate(rval, encoded); - } - } - else if (PyFloat_Check(obj)) { - PyObject *encoded = encoder_encode_float(s, obj); - if (encoded != NULL) - rv = _steal_accumulate(rval, encoded); - } - else if (s->for_json && _call_json_method(obj, FOR_JSON_METHOD_NAME, &newobj)) { - if (newobj == NULL) { - return -1; - } - if (Py_EnterRecursiveCall(" while encoding a JSON object")) { - Py_DECREF(newobj); - return rv; - } - rv = encoder_listencode_obj(s, rval, newobj, indent_level); - Py_DECREF(newobj); - Py_LeaveRecursiveCall(); - } - else if (s->namedtuple_as_object && _call_json_method(obj, ASDICT_METHOD_NAME, &newobj)) { - if (newobj == NULL) { - return -1; - } - if (Py_EnterRecursiveCall(" while encoding a JSON object")) { - Py_DECREF(newobj); - return rv; - } - if (PyDict_Check(newobj)) { - rv = encoder_listencode_dict(s, rval, newobj, indent_level); - } else { - PyErr_Format( - PyExc_TypeError, - "_asdict() must return a dict, not %.80s", - Py_TYPE(newobj)->tp_name - ); - rv = -1; - } - Py_DECREF(newobj); - Py_LeaveRecursiveCall(); - } - else if (PyList_Check(obj) || (s->tuple_as_array && PyTuple_Check(obj))) { - if (Py_EnterRecursiveCall(" while encoding a JSON object")) - return rv; - rv = encoder_listencode_list(s, rval, obj, indent_level); - Py_LeaveRecursiveCall(); - } - else if (PyDict_Check(obj)) { - if (Py_EnterRecursiveCall(" while encoding a JSON object")) - return rv; - rv = encoder_listencode_dict(s, rval, obj, indent_level); - Py_LeaveRecursiveCall(); - } - else if (s->use_decimal && PyObject_TypeCheck(obj, (PyTypeObject *)s->Decimal)) { - PyObject *encoded = PyObject_Str(obj); - if (encoded != NULL) - rv = _steal_accumulate(rval, encoded); - } - else if (is_raw_json(obj)) - { - PyObject *encoded = PyObject_GetAttrString(obj, "encoded_json"); - if (encoded != NULL) - rv = _steal_accumulate(rval, encoded); - } - else { - PyObject *ident = NULL; - PyObject *newobj; - if (s->iterable_as_array) { - newobj = PyObject_GetIter(obj); - if (newobj == NULL) - PyErr_Clear(); - else { - rv = encoder_listencode_list(s, rval, newobj, indent_level); - Py_DECREF(newobj); - break; - } - } - if (s->markers != Py_None) { - int has_key; - ident = PyLong_FromVoidPtr(obj); - if (ident == NULL) - break; - has_key = PyDict_Contains(s->markers, ident); - if (has_key) { - if (has_key != -1) - PyErr_SetString(PyExc_ValueError, "Circular reference detected"); - Py_DECREF(ident); - break; - } - if (PyDict_SetItem(s->markers, ident, obj)) { - Py_DECREF(ident); - break; - } - } - if (Py_EnterRecursiveCall(" while encoding a JSON object")) - return rv; - newobj = PyObject_CallOneArg(s->defaultfn, obj); - if (newobj == NULL) { - Py_XDECREF(ident); - Py_LeaveRecursiveCall(); - break; - } - rv = encoder_listencode_obj(s, rval, newobj, indent_level); - Py_LeaveRecursiveCall(); - Py_DECREF(newobj); - if (rv) { - Py_XDECREF(ident); - rv = -1; - } - else if (ident != NULL) { - if (PyDict_DelItem(s->markers, ident)) { - Py_XDECREF(ident); - rv = -1; - } - Py_XDECREF(ident); - } - } - } while (0); - return rv; -} - -static int -encoder_listencode_dict(PyEncoderObject *s, JSON_Accu *rval, PyObject *dct, Py_ssize_t indent_level) -{ - /* Encode Python dict dct a JSON term */ - static PyObject *open_dict = NULL; - static PyObject *close_dict = NULL; - static PyObject *empty_dict = NULL; - PyObject *kstr = NULL; - PyObject *ident = NULL; - PyObject *iter = NULL; - PyObject *item = NULL; - PyObject *items = NULL; - PyObject *encoded = NULL; - Py_ssize_t idx; - - if (open_dict == NULL || close_dict == NULL || empty_dict == NULL) { - open_dict = JSON_InternFromString("{"); - close_dict = JSON_InternFromString("}"); - empty_dict = JSON_InternFromString("{}"); - if (open_dict == NULL || close_dict == NULL || empty_dict == NULL) - return -1; - } - if (PyDict_Size(dct) == 0) - return JSON_Accu_Accumulate(rval, empty_dict); - - if (s->markers != Py_None) { - int has_key; - ident = PyLong_FromVoidPtr(dct); - if (ident == NULL) - goto bail; - has_key = PyDict_Contains(s->markers, ident); - if (has_key) { - if (has_key != -1) - PyErr_SetString(PyExc_ValueError, "Circular reference detected"); - goto bail; - } - if (PyDict_SetItem(s->markers, ident, dct)) { - goto bail; - } - } - - if (JSON_Accu_Accumulate(rval, open_dict)) - goto bail; - - if (s->indent != Py_None) { - /* TODO: DOES NOT RUN */ - indent_level += 1; - /* - newline_indent = '\n' + (_indent * _current_indent_level) - separator = _item_separator + newline_indent - buf += newline_indent - */ - } - - iter = encoder_dict_iteritems(s, dct); - if (iter == NULL) - goto bail; - - idx = 0; - while ((item = PyIter_Next(iter))) { - PyObject *encoded, *key, *value; - if (!PyTuple_Check(item) || Py_SIZE(item) != 2) { - PyErr_SetString(PyExc_ValueError, "items must return 2-tuples"); - goto bail; - } - key = PyTuple_GET_ITEM(item, 0); - if (key == NULL) - goto bail; - value = PyTuple_GET_ITEM(item, 1); - if (value == NULL) - goto bail; - - encoded = PyDict_GetItem(s->key_memo, key); - if (encoded != NULL) { - Py_INCREF(encoded); - } else { - kstr = encoder_stringify_key(s, key); - if (kstr == NULL) - goto bail; - else if (kstr == Py_None) { - /* skipkeys */ - Py_DECREF(item); - Py_DECREF(kstr); - continue; - } - } - if (idx) { - if (JSON_Accu_Accumulate(rval, s->item_separator)) - goto bail; - } - if (encoded == NULL) { - encoded = encoder_encode_string(s, kstr); - Py_CLEAR(kstr); - if (encoded == NULL) - goto bail; - if (PyDict_SetItem(s->key_memo, key, encoded)) - goto bail; - } - if (JSON_Accu_Accumulate(rval, encoded)) { - goto bail; - } - Py_CLEAR(encoded); - if (JSON_Accu_Accumulate(rval, s->key_separator)) - goto bail; - if (encoder_listencode_obj(s, rval, value, indent_level)) - goto bail; - Py_CLEAR(item); - idx += 1; - } - Py_CLEAR(iter); - if (PyErr_Occurred()) - goto bail; - if (ident != NULL) { - if (PyDict_DelItem(s->markers, ident)) - goto bail; - Py_CLEAR(ident); - } - if (s->indent != Py_None) { - /* TODO: DOES NOT RUN */ - indent_level -= 1; - /* - yield '\n' + (_indent * _current_indent_level) - */ - } - if (JSON_Accu_Accumulate(rval, close_dict)) - goto bail; - return 0; - -bail: - Py_XDECREF(encoded); - Py_XDECREF(items); - Py_XDECREF(item); - Py_XDECREF(iter); - Py_XDECREF(kstr); - Py_XDECREF(ident); - return -1; -} - - -static int -encoder_listencode_list(PyEncoderObject *s, JSON_Accu *rval, PyObject *seq, Py_ssize_t indent_level) -{ - /* Encode Python list seq to a JSON term */ - static PyObject *open_array = NULL; - static PyObject *close_array = NULL; - static PyObject *empty_array = NULL; - PyObject *ident = NULL; - PyObject *iter = NULL; - PyObject *obj = NULL; - int is_true; - int i = 0; - - if (open_array == NULL || close_array == NULL || empty_array == NULL) { - open_array = JSON_InternFromString("["); - close_array = JSON_InternFromString("]"); - empty_array = JSON_InternFromString("[]"); - if (open_array == NULL || close_array == NULL || empty_array == NULL) - return -1; - } - ident = NULL; - is_true = PyObject_IsTrue(seq); - if (is_true == -1) - return -1; - else if (is_true == 0) - return JSON_Accu_Accumulate(rval, empty_array); - - if (s->markers != Py_None) { - int has_key; - ident = PyLong_FromVoidPtr(seq); - if (ident == NULL) - goto bail; - has_key = PyDict_Contains(s->markers, ident); - if (has_key) { - if (has_key != -1) - PyErr_SetString(PyExc_ValueError, "Circular reference detected"); - goto bail; - } - if (PyDict_SetItem(s->markers, ident, seq)) { - goto bail; - } - } - - iter = PyObject_GetIter(seq); - if (iter == NULL) - goto bail; - - if (JSON_Accu_Accumulate(rval, open_array)) - goto bail; - if (s->indent != Py_None) { - /* TODO: DOES NOT RUN */ - indent_level += 1; - /* - newline_indent = '\n' + (_indent * _current_indent_level) - separator = _item_separator + newline_indent - buf += newline_indent - */ - } - while ((obj = PyIter_Next(iter))) { - if (i) { - if (JSON_Accu_Accumulate(rval, s->item_separator)) - goto bail; - } - if (encoder_listencode_obj(s, rval, obj, indent_level)) - goto bail; - i++; - Py_CLEAR(obj); - } - Py_CLEAR(iter); - if (PyErr_Occurred()) - goto bail; - if (ident != NULL) { - if (PyDict_DelItem(s->markers, ident)) - goto bail; - Py_CLEAR(ident); - } - if (s->indent != Py_None) { - /* TODO: DOES NOT RUN */ - indent_level -= 1; - /* - yield '\n' + (_indent * _current_indent_level) - */ - } - if (JSON_Accu_Accumulate(rval, close_array)) - goto bail; - return 0; - -bail: - Py_XDECREF(obj); - Py_XDECREF(iter); - Py_XDECREF(ident); - return -1; -} - -static void -encoder_dealloc(PyObject *self) -{ - /* bpo-31095: UnTrack is needed before calling any callbacks */ - PyObject_GC_UnTrack(self); - encoder_clear(self); - Py_TYPE(self)->tp_free(self); -} - -static int -encoder_traverse(PyObject *self, visitproc visit, void *arg) -{ - PyEncoderObject *s; - assert(PyEncoder_Check(self)); - s = (PyEncoderObject *)self; - Py_VISIT(s->markers); - Py_VISIT(s->defaultfn); - Py_VISIT(s->encoder); - Py_VISIT(s->encoding); - Py_VISIT(s->indent); - Py_VISIT(s->key_separator); - Py_VISIT(s->item_separator); - Py_VISIT(s->key_memo); - Py_VISIT(s->sort_keys); - Py_VISIT(s->item_sort_kw); - Py_VISIT(s->item_sort_key); - Py_VISIT(s->max_long_size); - Py_VISIT(s->min_long_size); - Py_VISIT(s->Decimal); - return 0; -} - -static int -encoder_clear(PyObject *self) -{ - /* Deallocate Encoder */ - PyEncoderObject *s; - assert(PyEncoder_Check(self)); - s = (PyEncoderObject *)self; - Py_CLEAR(s->markers); - Py_CLEAR(s->defaultfn); - Py_CLEAR(s->encoder); - Py_CLEAR(s->encoding); - Py_CLEAR(s->indent); - Py_CLEAR(s->key_separator); - Py_CLEAR(s->item_separator); - Py_CLEAR(s->key_memo); - Py_CLEAR(s->skipkeys_bool); - Py_CLEAR(s->sort_keys); - Py_CLEAR(s->item_sort_kw); - Py_CLEAR(s->item_sort_key); - Py_CLEAR(s->max_long_size); - Py_CLEAR(s->min_long_size); - Py_CLEAR(s->Decimal); - return 0; -} - -PyDoc_STRVAR(encoder_doc, "_iterencode(obj, _current_indent_level) -> iterable"); - -static -PyTypeObject PyEncoderType = { - PyVarObject_HEAD_INIT(NULL, 0) - "simplejson._speedups.Encoder", /* tp_name */ - sizeof(PyEncoderObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - encoder_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - encoder_call, /* tp_call */ - 0, /* tp_str */ - 0, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - encoder_doc, /* tp_doc */ - encoder_traverse, /* tp_traverse */ - encoder_clear, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - encoder_members, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - encoder_new, /* tp_new */ - 0, /* tp_free */ -}; - -static PyMethodDef speedups_methods[] = { - {"encode_basestring_ascii", - (PyCFunction)py_encode_basestring_ascii, - METH_O, - pydoc_encode_basestring_ascii}, - {"scanstring", - (PyCFunction)py_scanstring, - METH_VARARGS, - pydoc_scanstring}, - {NULL, NULL, 0, NULL} -}; - -PyDoc_STRVAR(module_doc, -"simplejson speedups\n"); - -#if PY_MAJOR_VERSION >= 3 -static struct PyModuleDef moduledef = { - PyModuleDef_HEAD_INIT, - "_speedups", /* m_name */ - module_doc, /* m_doc */ - -1, /* m_size */ - speedups_methods, /* m_methods */ - NULL, /* m_reload */ - NULL, /* m_traverse */ - NULL, /* m_clear*/ - NULL, /* m_free */ -}; -#endif - -PyObject * -import_dependency(char *module_name, char *attr_name) -{ - PyObject *rval; - PyObject *module = PyImport_ImportModule(module_name); - if (module == NULL) - return NULL; - rval = PyObject_GetAttrString(module, attr_name); - Py_DECREF(module); - return rval; -} - -static int -init_constants(void) -{ - JSON_NaN = JSON_InternFromString("NaN"); - if (JSON_NaN == NULL) - return 0; - JSON_Infinity = JSON_InternFromString("Infinity"); - if (JSON_Infinity == NULL) - return 0; - JSON_NegInfinity = JSON_InternFromString("-Infinity"); - if (JSON_NegInfinity == NULL) - return 0; -#if PY_MAJOR_VERSION >= 3 - JSON_EmptyUnicode = PyUnicode_New(0, 127); -#else /* PY_MAJOR_VERSION >= 3 */ - JSON_EmptyStr = PyString_FromString(""); - if (JSON_EmptyStr == NULL) - return 0; - JSON_EmptyUnicode = PyUnicode_FromUnicode(NULL, 0); -#endif /* PY_MAJOR_VERSION >= 3 */ - if (JSON_EmptyUnicode == NULL) - return 0; - - return 1; -} - -static PyObject * -moduleinit(void) -{ - PyObject *m; - if (PyType_Ready(&PyScannerType) < 0) - return NULL; - if (PyType_Ready(&PyEncoderType) < 0) - return NULL; - if (!init_constants()) - return NULL; - -#if PY_MAJOR_VERSION >= 3 - m = PyModule_Create(&moduledef); -#else - m = Py_InitModule3("_speedups", speedups_methods, module_doc); -#endif - Py_INCREF((PyObject*)&PyScannerType); - PyModule_AddObject(m, "make_scanner", (PyObject*)&PyScannerType); - Py_INCREF((PyObject*)&PyEncoderType); - PyModule_AddObject(m, "make_encoder", (PyObject*)&PyEncoderType); - RawJSONType = import_dependency("simplejson.raw_json", "RawJSON"); - if (RawJSONType == NULL) - return NULL; - JSONDecodeError = import_dependency("simplejson.errors", "JSONDecodeError"); - if (JSONDecodeError == NULL) - return NULL; - return m; -} - -#if PY_MAJOR_VERSION >= 3 -PyMODINIT_FUNC -PyInit__speedups(void) -{ - return moduleinit(); -} -#else -void -init_speedups(void) -{ - moduleinit(); -} -#endif diff --git a/contrib/python/simplejson/simplejson/compat.py b/contrib/python/simplejson/simplejson/compat.py deleted file mode 100644 index 5fc14128443..00000000000 --- a/contrib/python/simplejson/simplejson/compat.py +++ /dev/null @@ -1,34 +0,0 @@ -"""Python 3 compatibility shims -""" -import sys -if sys.version_info[0] < 3: - PY3 = False - def b(s): - return s - try: - from cStringIO import StringIO - except ImportError: - from StringIO import StringIO - BytesIO = StringIO - text_type = unicode - binary_type = str - string_types = (basestring,) - integer_types = (int, long) - unichr = unichr - reload_module = reload -else: - PY3 = True - if sys.version_info[:2] >= (3, 4): - from importlib import reload as reload_module - else: - from imp import reload as reload_module - def b(s): - return bytes(s, 'latin1') - from io import StringIO, BytesIO - text_type = str - binary_type = bytes - string_types = (str,) - integer_types = (int,) - unichr = chr - -long_type = integer_types[-1] diff --git a/contrib/python/simplejson/simplejson/decoder.py b/contrib/python/simplejson/simplejson/decoder.py deleted file mode 100644 index 1a8f772f13c..00000000000 --- a/contrib/python/simplejson/simplejson/decoder.py +++ /dev/null @@ -1,402 +0,0 @@ -"""Implementation of JSONDecoder -""" -from __future__ import absolute_import -import re -import sys -import struct -from .compat import PY3, unichr -from .scanner import make_scanner, JSONDecodeError - -def _import_c_scanstring(): - try: - from ._speedups import scanstring - return scanstring - except ImportError: - return None -c_scanstring = _import_c_scanstring() - -# NOTE (3.1.0): JSONDecodeError may still be imported from this module for -# compatibility, but it was never in the __all__ -__all__ = ['JSONDecoder'] - -FLAGS = re.VERBOSE | re.MULTILINE | re.DOTALL - -def _floatconstants(): - if sys.version_info < (2, 6): - _BYTES = '7FF80000000000007FF0000000000000'.decode('hex') - nan, inf = struct.unpack('>dd', _BYTES) - else: - nan = float('nan') - inf = float('inf') - return nan, inf, -inf - -NaN, PosInf, NegInf = _floatconstants() - -_CONSTANTS = { - '-Infinity': NegInf, - 'Infinity': PosInf, - 'NaN': NaN, -} - -STRINGCHUNK = re.compile(r'(.*?)(["\\\x00-\x1f])', FLAGS) -BACKSLASH = { - '"': u'"', '\\': u'\\', '/': u'/', - 'b': u'\b', 'f': u'\f', 'n': u'\n', 'r': u'\r', 't': u'\t', -} - -DEFAULT_ENCODING = "utf-8" - -def py_scanstring(s, end, encoding=None, strict=True, - _b=BACKSLASH, _m=STRINGCHUNK.match, _join=u''.join, - _PY3=PY3, _maxunicode=sys.maxunicode): - """Scan the string s for a JSON string. End is the index of the - character in s after the quote that started the JSON string. - Unescapes all valid JSON string escape sequences and raises ValueError - on attempt to decode an invalid string. If strict is False then literal - control characters are allowed in the string. - - Returns a tuple of the decoded string and the index of the character in s - after the end quote.""" - if encoding is None: - encoding = DEFAULT_ENCODING - chunks = [] - _append = chunks.append - begin = end - 1 - while 1: - chunk = _m(s, end) - if chunk is None: - raise JSONDecodeError( - "Unterminated string starting at", s, begin) - end = chunk.end() - content, terminator = chunk.groups() - # Content is contains zero or more unescaped string characters - if content: - if not _PY3 and not isinstance(content, unicode): - content = unicode(content, encoding) - _append(content) - # Terminator is the end of string, a literal control character, - # or a backslash denoting that an escape sequence follows - if terminator == '"': - break - elif terminator != '\\': - if strict: - msg = "Invalid control character %r at" - raise JSONDecodeError(msg, s, end) - else: - _append(terminator) - continue - try: - esc = s[end] - except IndexError: - raise JSONDecodeError( - "Unterminated string starting at", s, begin) - # If not a unicode escape sequence, must be in the lookup table - if esc != 'u': - try: - char = _b[esc] - except KeyError: - msg = "Invalid \\X escape sequence %r" - raise JSONDecodeError(msg, s, end) - end += 1 - else: - # Unicode escape sequence - msg = "Invalid \\uXXXX escape sequence" - esc = s[end + 1:end + 5] - escX = esc[1:2] - if len(esc) != 4 or escX == 'x' or escX == 'X': - raise JSONDecodeError(msg, s, end - 1) - try: - uni = int(esc, 16) - except ValueError: - raise JSONDecodeError(msg, s, end - 1) - if uni < 0 or uni > _maxunicode: - raise JSONDecodeError(msg, s, end - 1) - end += 5 - # Check for surrogate pair on UCS-4 systems - # Note that this will join high/low surrogate pairs - # but will also pass unpaired surrogates through - if (_maxunicode > 65535 and - uni & 0xfc00 == 0xd800 and - s[end:end + 2] == '\\u'): - esc2 = s[end + 2:end + 6] - escX = esc2[1:2] - if len(esc2) == 4 and not (escX == 'x' or escX == 'X'): - try: - uni2 = int(esc2, 16) - except ValueError: - raise JSONDecodeError(msg, s, end) - if uni2 & 0xfc00 == 0xdc00: - uni = 0x10000 + (((uni - 0xd800) << 10) | - (uni2 - 0xdc00)) - end += 6 - char = unichr(uni) - # Append the unescaped character - _append(char) - return _join(chunks), end - - -# Use speedup if available -scanstring = c_scanstring or py_scanstring - -WHITESPACE = re.compile(r'[ \t\n\r]*', FLAGS) -WHITESPACE_STR = ' \t\n\r' - -def JSONObject(state, encoding, strict, scan_once, object_hook, - object_pairs_hook, memo=None, - _w=WHITESPACE.match, _ws=WHITESPACE_STR): - (s, end) = state - # Backwards compatibility - if memo is None: - memo = {} - memo_get = memo.setdefault - pairs = [] - # Use a slice to prevent IndexError from being raised, the following - # check will raise a more specific ValueError if the string is empty - nextchar = s[end:end + 1] - # Normally we expect nextchar == '"' - if nextchar != '"': - if nextchar in _ws: - end = _w(s, end).end() - nextchar = s[end:end + 1] - # Trivial empty object - if nextchar == '}': - if object_pairs_hook is not None: - result = object_pairs_hook(pairs) - return result, end + 1 - pairs = {} - if object_hook is not None: - pairs = object_hook(pairs) - return pairs, end + 1 - elif nextchar != '"': - raise JSONDecodeError( - "Expecting property name enclosed in double quotes", - s, end) - end += 1 - while True: - key, end = scanstring(s, end, encoding, strict) - key = memo_get(key, key) - - # To skip some function call overhead we optimize the fast paths where - # the JSON key separator is ": " or just ":". - if s[end:end + 1] != ':': - end = _w(s, end).end() - if s[end:end + 1] != ':': - raise JSONDecodeError("Expecting ':' delimiter", s, end) - - end += 1 - - try: - if s[end] in _ws: - end += 1 - if s[end] in _ws: - end = _w(s, end + 1).end() - except IndexError: - pass - - value, end = scan_once(s, end) - pairs.append((key, value)) - - try: - nextchar = s[end] - if nextchar in _ws: - end = _w(s, end + 1).end() - nextchar = s[end] - except IndexError: - nextchar = '' - end += 1 - - if nextchar == '}': - break - elif nextchar != ',': - raise JSONDecodeError("Expecting ',' delimiter or '}'", s, end - 1) - - try: - nextchar = s[end] - if nextchar in _ws: - end += 1 - nextchar = s[end] - if nextchar in _ws: - end = _w(s, end + 1).end() - nextchar = s[end] - except IndexError: - nextchar = '' - - end += 1 - if nextchar != '"': - raise JSONDecodeError( - "Expecting property name enclosed in double quotes", - s, end - 1) - - if object_pairs_hook is not None: - result = object_pairs_hook(pairs) - return result, end - pairs = dict(pairs) - if object_hook is not None: - pairs = object_hook(pairs) - return pairs, end - -def JSONArray(state, scan_once, _w=WHITESPACE.match, _ws=WHITESPACE_STR): - (s, end) = state - values = [] - nextchar = s[end:end + 1] - if nextchar in _ws: - end = _w(s, end + 1).end() - nextchar = s[end:end + 1] - # Look-ahead for trivial empty array - if nextchar == ']': - return values, end + 1 - elif nextchar == '': - raise JSONDecodeError("Expecting value or ']'", s, end) - _append = values.append - while True: - value, end = scan_once(s, end) - _append(value) - nextchar = s[end:end + 1] - if nextchar in _ws: - end = _w(s, end + 1).end() - nextchar = s[end:end + 1] - end += 1 - if nextchar == ']': - break - elif nextchar != ',': - raise JSONDecodeError("Expecting ',' delimiter or ']'", s, end - 1) - - try: - if s[end] in _ws: - end += 1 - if s[end] in _ws: - end = _w(s, end + 1).end() - except IndexError: - pass - - return values, end - -class JSONDecoder(object): - """Simple JSON <http://json.org> decoder - - Performs the following translations in decoding by default: - - +---------------+-------------------+ - | JSON | Python | - +===============+===================+ - | object | dict | - +---------------+-------------------+ - | array | list | - +---------------+-------------------+ - | string | str, unicode | - +---------------+-------------------+ - | number (int) | int, long | - +---------------+-------------------+ - | number (real) | float | - +---------------+-------------------+ - | true | True | - +---------------+-------------------+ - | false | False | - +---------------+-------------------+ - | null | None | - +---------------+-------------------+ - - It also understands ``NaN``, ``Infinity``, and ``-Infinity`` as - their corresponding ``float`` values, which is outside the JSON spec. - - """ - - def __init__(self, encoding=None, object_hook=None, parse_float=None, - parse_int=None, parse_constant=None, strict=True, - object_pairs_hook=None): - """ - *encoding* determines the encoding used to interpret any - :class:`str` objects decoded by this instance (``'utf-8'`` by - default). It has no effect when decoding :class:`unicode` objects. - - Note that currently only encodings that are a superset of ASCII work, - strings of other encodings should be passed in as :class:`unicode`. - - *object_hook*, if specified, will be called with the result of every - JSON object decoded and its return value will be used in place of the - given :class:`dict`. This can be used to provide custom - deserializations (e.g. to support JSON-RPC class hinting). - - *object_pairs_hook* is an optional function that will be called with - the result of any object literal decode with an ordered list of pairs. - The return value of *object_pairs_hook* will be used instead of the - :class:`dict`. This feature can be used to implement custom decoders - that rely on the order that the key and value pairs are decoded (for - example, :func:`collections.OrderedDict` will remember the order of - insertion). If *object_hook* is also defined, the *object_pairs_hook* - takes priority. - - *parse_float*, if specified, will be called with the string of every - JSON float to be decoded. By default, this is equivalent to - ``float(num_str)``. This can be used to use another datatype or parser - for JSON floats (e.g. :class:`decimal.Decimal`). - - *parse_int*, if specified, will be called with the string of every - JSON int to be decoded. By default, this is equivalent to - ``int(num_str)``. This can be used to use another datatype or parser - for JSON integers (e.g. :class:`float`). - - *parse_constant*, if specified, will be called with one of the - following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This - can be used to raise an exception if invalid JSON numbers are - encountered. - - *strict* controls the parser's behavior when it encounters an - invalid control character in a string. The default setting of - ``True`` means that unescaped control characters are parse errors, if - ``False`` then control characters will be allowed in strings. - - """ - if encoding is None: - encoding = DEFAULT_ENCODING - self.encoding = encoding - self.object_hook = object_hook - self.object_pairs_hook = object_pairs_hook - self.parse_float = parse_float or float - self.parse_int = parse_int or int - self.parse_constant = parse_constant or _CONSTANTS.__getitem__ - self.strict = strict - self.parse_object = JSONObject - self.parse_array = JSONArray - self.parse_string = scanstring - self.memo = {} - self.scan_once = make_scanner(self) - - def decode(self, s, _w=WHITESPACE.match, _PY3=PY3): - """Return the Python representation of ``s`` (a ``str`` or ``unicode`` - instance containing a JSON document) - - """ - if _PY3 and isinstance(s, bytes): - s = str(s, self.encoding) - obj, end = self.raw_decode(s) - end = _w(s, end).end() - if end != len(s): - raise JSONDecodeError("Extra data", s, end, len(s)) - return obj - - def raw_decode(self, s, idx=0, _w=WHITESPACE.match, _PY3=PY3): - """Decode a JSON document from ``s`` (a ``str`` or ``unicode`` - beginning with a JSON document) and return a 2-tuple of the Python - representation and the index in ``s`` where the document ended. - Optionally, ``idx`` can be used to specify an offset in ``s`` where - the JSON document begins. - - This can be used to decode a JSON document from a string that may - have extraneous data at the end. - - """ - if idx < 0: - # Ensure that raw_decode bails on negative indexes, the regex - # would otherwise mask this behavior. #98 - raise JSONDecodeError('Expecting value', s, idx) - if _PY3 and not isinstance(s, str): - raise TypeError("Input string must be text, not bytes") - # strip UTF-8 bom - if len(s) > idx: - ord0 = ord(s[idx]) - if ord0 == 0xfeff: - idx += 1 - elif ord0 == 0xef and s[idx:idx + 3] == '\xef\xbb\xbf': - idx += 3 - return self.scan_once(s, idx=_w(s, idx).end()) diff --git a/contrib/python/simplejson/simplejson/encoder.py b/contrib/python/simplejson/simplejson/encoder.py deleted file mode 100644 index e93fe43f423..00000000000 --- a/contrib/python/simplejson/simplejson/encoder.py +++ /dev/null @@ -1,740 +0,0 @@ -"""Implementation of JSONEncoder -""" -from __future__ import absolute_import -import re -from operator import itemgetter -# Do not import Decimal directly to avoid reload issues -import decimal -from .compat import unichr, binary_type, text_type, string_types, integer_types, PY3 -def _import_speedups(): - try: - from . import _speedups - return _speedups.encode_basestring_ascii, _speedups.make_encoder - except ImportError: - return None, None -c_encode_basestring_ascii, c_make_encoder = _import_speedups() - -from .decoder import PosInf -from .raw_json import RawJSON - -ESCAPE = re.compile(r'[\x00-\x1f\\"]') -ESCAPE_ASCII = re.compile(r'([\\"]|[^\ -~])') -HAS_UTF8 = re.compile(r'[\x80-\xff]') -ESCAPE_DCT = { - '\\': '\\\\', - '"': '\\"', - '\b': '\\b', - '\f': '\\f', - '\n': '\\n', - '\r': '\\r', - '\t': '\\t', -} -for i in range(0x20): - #ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i)) - ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,)) -del i - -FLOAT_REPR = repr - -def encode_basestring(s, _PY3=PY3, _q=u'"'): - """Return a JSON representation of a Python string - - """ - if _PY3: - if isinstance(s, bytes): - s = str(s, 'utf-8') - elif type(s) is not str: - # convert an str subclass instance to exact str - # raise a TypeError otherwise - s = str.__str__(s) - else: - if isinstance(s, str) and HAS_UTF8.search(s) is not None: - s = unicode(s, 'utf-8') - elif type(s) not in (str, unicode): - # convert an str subclass instance to exact str - # convert a unicode subclass instance to exact unicode - # raise a TypeError otherwise - if isinstance(s, str): - s = str.__str__(s) - else: - s = unicode.__getnewargs__(s)[0] - def replace(match): - return ESCAPE_DCT[match.group(0)] - return _q + ESCAPE.sub(replace, s) + _q - - -def py_encode_basestring_ascii(s, _PY3=PY3): - """Return an ASCII-only JSON representation of a Python string - - """ - if _PY3: - if isinstance(s, bytes): - s = str(s, 'utf-8') - elif type(s) is not str: - # convert an str subclass instance to exact str - # raise a TypeError otherwise - s = str.__str__(s) - else: - if isinstance(s, str) and HAS_UTF8.search(s) is not None: - s = unicode(s, 'utf-8') - elif type(s) not in (str, unicode): - # convert an str subclass instance to exact str - # convert a unicode subclass instance to exact unicode - # raise a TypeError otherwise - if isinstance(s, str): - s = str.__str__(s) - else: - s = unicode.__getnewargs__(s)[0] - def replace(match): - s = match.group(0) - try: - return ESCAPE_DCT[s] - except KeyError: - n = ord(s) - if n < 0x10000: - #return '\\u{0:04x}'.format(n) - return '\\u%04x' % (n,) - else: - # surrogate pair - n -= 0x10000 - s1 = 0xd800 | ((n >> 10) & 0x3ff) - s2 = 0xdc00 | (n & 0x3ff) - #return '\\u{0:04x}\\u{1:04x}'.format(s1, s2) - return '\\u%04x\\u%04x' % (s1, s2) - return '"' + str(ESCAPE_ASCII.sub(replace, s)) + '"' - - -encode_basestring_ascii = ( - c_encode_basestring_ascii or py_encode_basestring_ascii) - -class JSONEncoder(object): - """Extensible JSON <http://json.org> encoder for Python data structures. - - Supports the following objects and types by default: - - +-------------------+---------------+ - | Python | JSON | - +===================+===============+ - | dict, namedtuple | object | - +-------------------+---------------+ - | list, tuple | array | - +-------------------+---------------+ - | str, unicode | string | - +-------------------+---------------+ - | int, long, float | number | - +-------------------+---------------+ - | True | true | - +-------------------+---------------+ - | False | false | - +-------------------+---------------+ - | None | null | - +-------------------+---------------+ - - To extend this to recognize other objects, subclass and implement a - ``.default()`` method with another method that returns a serializable - object for ``o`` if possible, otherwise it should call the superclass - implementation (to raise ``TypeError``). - - """ - item_separator = ', ' - key_separator = ': ' - - def __init__(self, skipkeys=False, ensure_ascii=True, - check_circular=True, allow_nan=True, sort_keys=False, - indent=None, separators=None, encoding='utf-8', default=None, - use_decimal=True, namedtuple_as_object=True, - tuple_as_array=True, bigint_as_string=False, - item_sort_key=None, for_json=False, ignore_nan=False, - int_as_string_bitcount=None, iterable_as_array=False): - """Constructor for JSONEncoder, with sensible defaults. - - If skipkeys is false, then it is a TypeError to attempt - encoding of keys that are not str, int, long, float or None. If - skipkeys is True, such items are simply skipped. - - If ensure_ascii is true, the output is guaranteed to be str - objects with all incoming unicode characters escaped. If - ensure_ascii is false, the output will be unicode object. - - If check_circular is true, then lists, dicts, and custom encoded - objects will be checked for circular references during encoding to - prevent an infinite recursion (which would cause an OverflowError). - Otherwise, no such check takes place. - - If allow_nan is true, then NaN, Infinity, and -Infinity will be - encoded as such. This behavior is not JSON specification compliant, - but is consistent with most JavaScript based encoders and decoders. - Otherwise, it will be a ValueError to encode such floats. - - If sort_keys is true, then the output of dictionaries will be - sorted by key; this is useful for regression tests to ensure - that JSON serializations can be compared on a day-to-day basis. - - If indent is a string, then JSON array elements and object members - will be pretty-printed with a newline followed by that string repeated - for each level of nesting. ``None`` (the default) selects the most compact - representation without any newlines. For backwards compatibility with - versions of simplejson earlier than 2.1.0, an integer is also accepted - and is converted to a string with that many spaces. - - If specified, separators should be an (item_separator, key_separator) - tuple. The default is (', ', ': ') if *indent* is ``None`` and - (',', ': ') otherwise. To get the most compact JSON representation, - you should specify (',', ':') to eliminate whitespace. - - If specified, default is a function that gets called for objects - that can't otherwise be serialized. It should return a JSON encodable - version of the object or raise a ``TypeError``. - - If encoding is not None, then all input strings will be - transformed into unicode using that encoding prior to JSON-encoding. - The default is UTF-8. - - If use_decimal is true (default: ``True``), ``decimal.Decimal`` will - be supported directly by the encoder. For the inverse, decode JSON - with ``parse_float=decimal.Decimal``. - - If namedtuple_as_object is true (the default), objects with - ``_asdict()`` methods will be encoded as JSON objects. - - If tuple_as_array is true (the default), tuple (and subclasses) will - be encoded as JSON arrays. - - If *iterable_as_array* is true (default: ``False``), - any object not in the above table that implements ``__iter__()`` - will be encoded as a JSON array. - - If bigint_as_string is true (not the default), ints 2**53 and higher - or lower than -2**53 will be encoded as strings. This is to avoid the - rounding that happens in Javascript otherwise. - - If int_as_string_bitcount is a positive number (n), then int of size - greater than or equal to 2**n or lower than or equal to -2**n will be - encoded as strings. - - If specified, item_sort_key is a callable used to sort the items in - each dictionary. This is useful if you want to sort items other than - in alphabetical order by key. - - If for_json is true (not the default), objects with a ``for_json()`` - method will use the return value of that method for encoding as JSON - instead of the object. - - If *ignore_nan* is true (default: ``False``), then out of range - :class:`float` values (``nan``, ``inf``, ``-inf``) will be serialized - as ``null`` in compliance with the ECMA-262 specification. If true, - this will override *allow_nan*. - - """ - - self.skipkeys = skipkeys - self.ensure_ascii = ensure_ascii - self.check_circular = check_circular - self.allow_nan = allow_nan - self.sort_keys = sort_keys - self.use_decimal = use_decimal - self.namedtuple_as_object = namedtuple_as_object - self.tuple_as_array = tuple_as_array - self.iterable_as_array = iterable_as_array - self.bigint_as_string = bigint_as_string - self.item_sort_key = item_sort_key - self.for_json = for_json - self.ignore_nan = ignore_nan - self.int_as_string_bitcount = int_as_string_bitcount - if indent is not None and not isinstance(indent, string_types): - indent = indent * ' ' - self.indent = indent - if separators is not None: - self.item_separator, self.key_separator = separators - elif indent is not None: - self.item_separator = ',' - if default is not None: - self.default = default - self.encoding = encoding - - def default(self, o): - """Implement this method in a subclass such that it returns - a serializable object for ``o``, or calls the base implementation - (to raise a ``TypeError``). - - For example, to support arbitrary iterators, you could - implement default like this:: - - def default(self, o): - try: - iterable = iter(o) - except TypeError: - pass - else: - return list(iterable) - return JSONEncoder.default(self, o) - - """ - raise TypeError('Object of type %s is not JSON serializable' % - o.__class__.__name__) - - def encode(self, o): - """Return a JSON string representation of a Python data structure. - - >>> from simplejson import JSONEncoder - >>> JSONEncoder().encode({"foo": ["bar", "baz"]}) - '{"foo": ["bar", "baz"]}' - - """ - # This is for extremely simple cases and benchmarks. - if isinstance(o, binary_type): - _encoding = self.encoding - if (_encoding is not None and not (_encoding == 'utf-8')): - o = text_type(o, _encoding) - if isinstance(o, string_types): - if self.ensure_ascii: - return encode_basestring_ascii(o) - else: - return encode_basestring(o) - # This doesn't pass the iterator directly to ''.join() because the - # exceptions aren't as detailed. The list call should be roughly - # equivalent to the PySequence_Fast that ''.join() would do. - chunks = self.iterencode(o, _one_shot=True) - if not isinstance(chunks, (list, tuple)): - chunks = list(chunks) - if self.ensure_ascii: - return ''.join(chunks) - else: - return u''.join(chunks) - - def iterencode(self, o, _one_shot=False): - """Encode the given object and yield each string - representation as available. - - For example:: - - for chunk in JSONEncoder().iterencode(bigobject): - mysocket.write(chunk) - - """ - if self.check_circular: - markers = {} - else: - markers = None - if self.ensure_ascii: - _encoder = encode_basestring_ascii - else: - _encoder = encode_basestring - if self.encoding != 'utf-8' and self.encoding is not None: - def _encoder(o, _orig_encoder=_encoder, _encoding=self.encoding): - if isinstance(o, binary_type): - o = text_type(o, _encoding) - return _orig_encoder(o) - - def floatstr(o, allow_nan=self.allow_nan, ignore_nan=self.ignore_nan, - _repr=FLOAT_REPR, _inf=PosInf, _neginf=-PosInf): - # Check for specials. Note that this type of test is processor - # and/or platform-specific, so do tests which don't depend on - # the internals. - - if o != o: - text = 'NaN' - elif o == _inf: - text = 'Infinity' - elif o == _neginf: - text = '-Infinity' - else: - if type(o) != float: - # See #118, do not trust custom str/repr - o = float(o) - return _repr(o) - - if ignore_nan: - text = 'null' - elif not allow_nan: - raise ValueError( - "Out of range float values are not JSON compliant: " + - repr(o)) - - return text - - key_memo = {} - int_as_string_bitcount = ( - 53 if self.bigint_as_string else self.int_as_string_bitcount) - if (_one_shot and c_make_encoder is not None - and self.indent is None): - _iterencode = c_make_encoder( - markers, self.default, _encoder, self.indent, - self.key_separator, self.item_separator, self.sort_keys, - self.skipkeys, self.allow_nan, key_memo, self.use_decimal, - self.namedtuple_as_object, self.tuple_as_array, - int_as_string_bitcount, - self.item_sort_key, self.encoding, self.for_json, - self.ignore_nan, decimal.Decimal, self.iterable_as_array) - else: - _iterencode = _make_iterencode( - markers, self.default, _encoder, self.indent, floatstr, - self.key_separator, self.item_separator, self.sort_keys, - self.skipkeys, _one_shot, self.use_decimal, - self.namedtuple_as_object, self.tuple_as_array, - int_as_string_bitcount, - self.item_sort_key, self.encoding, self.for_json, - self.iterable_as_array, Decimal=decimal.Decimal) - try: - return _iterencode(o, 0) - finally: - key_memo.clear() - - -class JSONEncoderForHTML(JSONEncoder): - """An encoder that produces JSON safe to embed in HTML. - - To embed JSON content in, say, a script tag on a web page, the - characters &, < and > should be escaped. They cannot be escaped - with the usual entities (e.g. &) because they are not expanded - within <script> tags. - - This class also escapes the line separator and paragraph separator - characters U+2028 and U+2029, irrespective of the ensure_ascii setting, - as these characters are not valid in JavaScript strings (see - http://timelessrepo.com/json-isnt-a-javascript-subset). - """ - - def encode(self, o): - # Override JSONEncoder.encode because it has hacks for - # performance that make things more complicated. - chunks = self.iterencode(o, True) - if self.ensure_ascii: - return ''.join(chunks) - else: - return u''.join(chunks) - - def iterencode(self, o, _one_shot=False): - chunks = super(JSONEncoderForHTML, self).iterencode(o, _one_shot) - for chunk in chunks: - chunk = chunk.replace('&', '\\u0026') - chunk = chunk.replace('<', '\\u003c') - chunk = chunk.replace('>', '\\u003e') - - if not self.ensure_ascii: - chunk = chunk.replace(u'\u2028', '\\u2028') - chunk = chunk.replace(u'\u2029', '\\u2029') - - yield chunk - - -def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, - _key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot, - _use_decimal, _namedtuple_as_object, _tuple_as_array, - _int_as_string_bitcount, _item_sort_key, - _encoding,_for_json, - _iterable_as_array, - ## HACK: hand-optimized bytecode; turn globals into locals - _PY3=PY3, - ValueError=ValueError, - string_types=string_types, - Decimal=None, - dict=dict, - float=float, - id=id, - integer_types=integer_types, - isinstance=isinstance, - list=list, - str=str, - tuple=tuple, - iter=iter, - ): - if _use_decimal and Decimal is None: - Decimal = decimal.Decimal - if _item_sort_key and not callable(_item_sort_key): - raise TypeError("item_sort_key must be None or callable") - elif _sort_keys and not _item_sort_key: - _item_sort_key = itemgetter(0) - - if (_int_as_string_bitcount is not None and - (_int_as_string_bitcount <= 0 or - not isinstance(_int_as_string_bitcount, integer_types))): - raise TypeError("int_as_string_bitcount must be a positive integer") - - def call_method(obj, method_name): - method = getattr(obj, method_name, None) - if callable(method): - try: - return (method(),) - except TypeError: - pass - return None - - def _encode_int(value): - skip_quoting = ( - _int_as_string_bitcount is None - or - _int_as_string_bitcount < 1 - ) - if type(value) not in integer_types: - # See #118, do not trust custom str/repr - value = int(value) - if ( - skip_quoting or - (-1 << _int_as_string_bitcount) - < value < - (1 << _int_as_string_bitcount) - ): - return str(value) - return '"' + str(value) + '"' - - def _iterencode_list(lst, _current_indent_level): - if not lst: - yield '[]' - return - if markers is not None: - markerid = id(lst) - if markerid in markers: - raise ValueError("Circular reference detected") - markers[markerid] = lst - buf = '[' - if _indent is not None: - _current_indent_level += 1 - newline_indent = '\n' + (_indent * _current_indent_level) - separator = _item_separator + newline_indent - buf += newline_indent - else: - newline_indent = None - separator = _item_separator - first = True - for value in lst: - if first: - first = False - else: - buf = separator - if isinstance(value, string_types): - yield buf + _encoder(value) - elif _PY3 and isinstance(value, bytes) and _encoding is not None: - yield buf + _encoder(value) - elif isinstance(value, RawJSON): - yield buf + value.encoded_json - elif value is None: - yield buf + 'null' - elif value is True: - yield buf + 'true' - elif value is False: - yield buf + 'false' - elif isinstance(value, integer_types): - yield buf + _encode_int(value) - elif isinstance(value, float): - yield buf + _floatstr(value) - elif _use_decimal and isinstance(value, Decimal): - yield buf + str(value) - else: - yield buf - for_json = _for_json and call_method(value, 'for_json') - if for_json: - chunks = _iterencode(for_json[0], _current_indent_level) - elif isinstance(value, list): - chunks = _iterencode_list(value, _current_indent_level) - else: - _asdict = _namedtuple_as_object and call_method(value, '_asdict') - if _asdict: - dct = _asdict[0] - if not isinstance(dct, dict): - raise TypeError("_asdict() must return a dict, not %s" % (type(dct).__name__,)) - chunks = _iterencode_dict(dct, - _current_indent_level) - elif _tuple_as_array and isinstance(value, tuple): - chunks = _iterencode_list(value, _current_indent_level) - elif isinstance(value, dict): - chunks = _iterencode_dict(value, _current_indent_level) - else: - chunks = _iterencode(value, _current_indent_level) - for chunk in chunks: - yield chunk - if first: - # iterable_as_array misses the fast path at the top - yield '[]' - else: - if newline_indent is not None: - _current_indent_level -= 1 - yield '\n' + (_indent * _current_indent_level) - yield ']' - if markers is not None: - del markers[markerid] - - def _stringify_key(key): - if isinstance(key, string_types): # pragma: no cover - pass - elif _PY3 and isinstance(key, bytes) and _encoding is not None: - key = str(key, _encoding) - elif isinstance(key, float): - key = _floatstr(key) - elif key is True: - key = 'true' - elif key is False: - key = 'false' - elif key is None: - key = 'null' - elif isinstance(key, integer_types): - if type(key) not in integer_types: - # See #118, do not trust custom str/repr - key = int(key) - key = str(key) - elif _use_decimal and isinstance(key, Decimal): - key = str(key) - elif _skipkeys: - key = None - else: - raise TypeError('keys must be str, int, float, bool or None, ' - 'not %s' % key.__class__.__name__) - return key - - def _iterencode_dict(dct, _current_indent_level): - if not dct: - yield '{}' - return - if markers is not None: - markerid = id(dct) - if markerid in markers: - raise ValueError("Circular reference detected") - markers[markerid] = dct - yield '{' - if _indent is not None: - _current_indent_level += 1 - newline_indent = '\n' + (_indent * _current_indent_level) - item_separator = _item_separator + newline_indent - yield newline_indent - else: - newline_indent = None - item_separator = _item_separator - first = True - if _PY3: - iteritems = dct.items() - else: - iteritems = dct.iteritems() - if _item_sort_key: - items = [] - for k, v in dct.items(): - if not isinstance(k, string_types): - k = _stringify_key(k) - if k is None: - continue - items.append((k, v)) - items.sort(key=_item_sort_key) - else: - items = iteritems - for key, value in items: - if not (_item_sort_key or isinstance(key, string_types)): - key = _stringify_key(key) - if key is None: - # _skipkeys must be True - continue - if first: - first = False - else: - yield item_separator - yield _encoder(key) - yield _key_separator - if isinstance(value, string_types): - yield _encoder(value) - elif _PY3 and isinstance(value, bytes) and _encoding is not None: - yield _encoder(value) - elif isinstance(value, RawJSON): - yield value.encoded_json - elif value is None: - yield 'null' - elif value is True: - yield 'true' - elif value is False: - yield 'false' - elif isinstance(value, integer_types): - yield _encode_int(value) - elif isinstance(value, float): - yield _floatstr(value) - elif _use_decimal and isinstance(value, Decimal): - yield str(value) - else: - for_json = _for_json and call_method(value, 'for_json') - if for_json: - chunks = _iterencode(for_json[0], _current_indent_level) - elif isinstance(value, list): - chunks = _iterencode_list(value, _current_indent_level) - else: - _asdict = _namedtuple_as_object and call_method(value, '_asdict') - if _asdict: - dct = _asdict[0] - if not isinstance(dct, dict): - raise TypeError("_asdict() must return a dict, not %s" % (type(dct).__name__,)) - chunks = _iterencode_dict(dct, - _current_indent_level) - elif _tuple_as_array and isinstance(value, tuple): - chunks = _iterencode_list(value, _current_indent_level) - elif isinstance(value, dict): - chunks = _iterencode_dict(value, _current_indent_level) - else: - chunks = _iterencode(value, _current_indent_level) - for chunk in chunks: - yield chunk - if newline_indent is not None: - _current_indent_level -= 1 - yield '\n' + (_indent * _current_indent_level) - yield '}' - if markers is not None: - del markers[markerid] - - def _iterencode(o, _current_indent_level): - if isinstance(o, string_types): - yield _encoder(o) - elif _PY3 and isinstance(o, bytes) and _encoding is not None: - yield _encoder(o) - elif isinstance(o, RawJSON): - yield o.encoded_json - elif o is None: - yield 'null' - elif o is True: - yield 'true' - elif o is False: - yield 'false' - elif isinstance(o, integer_types): - yield _encode_int(o) - elif isinstance(o, float): - yield _floatstr(o) - else: - for_json = _for_json and call_method(o, 'for_json') - if for_json: - for chunk in _iterencode(for_json[0], _current_indent_level): - yield chunk - elif isinstance(o, list): - for chunk in _iterencode_list(o, _current_indent_level): - yield chunk - else: - _asdict = _namedtuple_as_object and call_method(o, '_asdict') - if _asdict: - dct = _asdict[0] - if not isinstance(dct, dict): - raise TypeError("_asdict() must return a dict, not %s" % (type(dct).__name__,)) - for chunk in _iterencode_dict(dct, _current_indent_level): - yield chunk - elif (_tuple_as_array and isinstance(o, tuple)): - for chunk in _iterencode_list(o, _current_indent_level): - yield chunk - elif isinstance(o, dict): - for chunk in _iterencode_dict(o, _current_indent_level): - yield chunk - elif _use_decimal and isinstance(o, Decimal): - yield str(o) - else: - while _iterable_as_array: - # Markers are not checked here because it is valid for - # an iterable to return self. - try: - o = iter(o) - except TypeError: - break - for chunk in _iterencode_list(o, _current_indent_level): - yield chunk - return - if markers is not None: - markerid = id(o) - if markerid in markers: - raise ValueError("Circular reference detected") - markers[markerid] = o - o = _default(o) - for chunk in _iterencode(o, _current_indent_level): - yield chunk - if markers is not None: - del markers[markerid] - - return _iterencode diff --git a/contrib/python/simplejson/simplejson/errors.py b/contrib/python/simplejson/simplejson/errors.py deleted file mode 100644 index b97ab1e913f..00000000000 --- a/contrib/python/simplejson/simplejson/errors.py +++ /dev/null @@ -1,53 +0,0 @@ -"""Error classes used by simplejson -""" -__all__ = ['JSONDecodeError'] - - -def linecol(doc, pos): - lineno = doc.count('\n', 0, pos) + 1 - if lineno == 1: - colno = pos + 1 - else: - colno = pos - doc.rindex('\n', 0, pos) - return lineno, colno - - -def errmsg(msg, doc, pos, end=None): - lineno, colno = linecol(doc, pos) - msg = msg.replace('%r', repr(doc[pos:pos + 1])) - if end is None: - fmt = '%s: line %d column %d (char %d)' - return fmt % (msg, lineno, colno, pos) - endlineno, endcolno = linecol(doc, end) - fmt = '%s: line %d column %d - line %d column %d (char %d - %d)' - return fmt % (msg, lineno, colno, endlineno, endcolno, pos, end) - - -class JSONDecodeError(ValueError): - """Subclass of ValueError with the following additional properties: - - msg: The unformatted error message - doc: The JSON document being parsed - pos: The start index of doc where parsing failed - end: The end index of doc where parsing failed (may be None) - lineno: The line corresponding to pos - colno: The column corresponding to pos - endlineno: The line corresponding to end (may be None) - endcolno: The column corresponding to end (may be None) - - """ - # Note that this exception is used from _speedups - def __init__(self, msg, doc, pos, end=None): - ValueError.__init__(self, errmsg(msg, doc, pos, end=end)) - self.msg = msg - self.doc = doc - self.pos = pos - self.end = end - self.lineno, self.colno = linecol(doc, pos) - if end is not None: - self.endlineno, self.endcolno = linecol(doc, end) - else: - self.endlineno, self.endcolno = None, None - - def __reduce__(self): - return self.__class__, (self.msg, self.doc, self.pos, self.end) diff --git a/contrib/python/simplejson/simplejson/ordered_dict.py b/contrib/python/simplejson/simplejson/ordered_dict.py deleted file mode 100644 index d5a55ebd0f7..00000000000 --- a/contrib/python/simplejson/simplejson/ordered_dict.py +++ /dev/null @@ -1,103 +0,0 @@ -"""Drop-in replacement for collections.OrderedDict by Raymond Hettinger - -http://code.activestate.com/recipes/576693/ - -""" -from UserDict import DictMixin - -class OrderedDict(dict, DictMixin): - - def __init__(self, *args, **kwds): - if len(args) > 1: - raise TypeError('expected at most 1 arguments, got %d' % len(args)) - try: - self.__end - except AttributeError: - self.clear() - self.update(*args, **kwds) - - def clear(self): - self.__end = end = [] - end += [None, end, end] # sentinel node for doubly linked list - self.__map = {} # key --> [key, prev, next] - dict.clear(self) - - def __setitem__(self, key, value): - if key not in self: - end = self.__end - curr = end[1] - curr[2] = end[1] = self.__map[key] = [key, curr, end] - dict.__setitem__(self, key, value) - - def __delitem__(self, key): - dict.__delitem__(self, key) - key, prev, next = self.__map.pop(key) - prev[2] = next - next[1] = prev - - def __iter__(self): - end = self.__end - curr = end[2] - while curr is not end: - yield curr[0] - curr = curr[2] - - def __reversed__(self): - end = self.__end - curr = end[1] - while curr is not end: - yield curr[0] - curr = curr[1] - - def popitem(self, last=True): - if not self: - raise KeyError('dictionary is empty') - key = reversed(self).next() if last else iter(self).next() - value = self.pop(key) - return key, value - - def __reduce__(self): - items = [[k, self[k]] for k in self] - tmp = self.__map, self.__end - del self.__map, self.__end - inst_dict = vars(self).copy() - self.__map, self.__end = tmp - if inst_dict: - return (self.__class__, (items,), inst_dict) - return self.__class__, (items,) - - def keys(self): - return list(self) - - setdefault = DictMixin.setdefault - update = DictMixin.update - pop = DictMixin.pop - values = DictMixin.values - items = DictMixin.items - iterkeys = DictMixin.iterkeys - itervalues = DictMixin.itervalues - iteritems = DictMixin.iteritems - - def __repr__(self): - if not self: - return '%s()' % (self.__class__.__name__,) - return '%s(%r)' % (self.__class__.__name__, self.items()) - - def copy(self): - return self.__class__(self) - - @classmethod - def fromkeys(cls, iterable, value=None): - d = cls() - for key in iterable: - d[key] = value - return d - - def __eq__(self, other): - if isinstance(other, OrderedDict): - return len(self)==len(other) and \ - all(p==q for p, q in zip(self.items(), other.items())) - return dict.__eq__(self, other) - - def __ne__(self, other): - return not self == other diff --git a/contrib/python/simplejson/simplejson/raw_json.py b/contrib/python/simplejson/simplejson/raw_json.py deleted file mode 100644 index 2071a70206b..00000000000 --- a/contrib/python/simplejson/simplejson/raw_json.py +++ /dev/null @@ -1,9 +0,0 @@ -"""Implementation of RawJSON -""" - -class RawJSON(object): - """Wrap an encoded JSON document for direct embedding in the output - - """ - def __init__(self, encoded_json): - self.encoded_json = encoded_json diff --git a/contrib/python/simplejson/simplejson/scanner.py b/contrib/python/simplejson/simplejson/scanner.py deleted file mode 100644 index 85e385e1470..00000000000 --- a/contrib/python/simplejson/simplejson/scanner.py +++ /dev/null @@ -1,85 +0,0 @@ -"""JSON token scanner -""" -import re -from .errors import JSONDecodeError -def _import_c_make_scanner(): - try: - from ._speedups import make_scanner - return make_scanner - except ImportError: - return None -c_make_scanner = _import_c_make_scanner() - -__all__ = ['make_scanner', 'JSONDecodeError'] - -NUMBER_RE = re.compile( - r'(-?(?:0|[1-9]\d*))(\.\d+)?([eE][-+]?\d+)?', - (re.VERBOSE | re.MULTILINE | re.DOTALL)) - - -def py_make_scanner(context): - parse_object = context.parse_object - parse_array = context.parse_array - parse_string = context.parse_string - match_number = NUMBER_RE.match - encoding = context.encoding - strict = context.strict - parse_float = context.parse_float - parse_int = context.parse_int - parse_constant = context.parse_constant - object_hook = context.object_hook - object_pairs_hook = context.object_pairs_hook - memo = context.memo - - def _scan_once(string, idx): - errmsg = 'Expecting value' - try: - nextchar = string[idx] - except IndexError: - raise JSONDecodeError(errmsg, string, idx) - - if nextchar == '"': - return parse_string(string, idx + 1, encoding, strict) - elif nextchar == '{': - return parse_object((string, idx + 1), encoding, strict, - _scan_once, object_hook, object_pairs_hook, memo) - elif nextchar == '[': - return parse_array((string, idx + 1), _scan_once) - elif nextchar == 'n' and string[idx:idx + 4] == 'null': - return None, idx + 4 - elif nextchar == 't' and string[idx:idx + 4] == 'true': - return True, idx + 4 - elif nextchar == 'f' and string[idx:idx + 5] == 'false': - return False, idx + 5 - - m = match_number(string, idx) - if m is not None: - integer, frac, exp = m.groups() - if frac or exp: - res = parse_float(integer + (frac or '') + (exp or '')) - else: - res = parse_int(integer) - return res, m.end() - elif nextchar == 'N' and string[idx:idx + 3] == 'NaN': - return parse_constant('NaN'), idx + 3 - elif nextchar == 'I' and string[idx:idx + 8] == 'Infinity': - return parse_constant('Infinity'), idx + 8 - elif nextchar == '-' and string[idx:idx + 9] == '-Infinity': - return parse_constant('-Infinity'), idx + 9 - else: - raise JSONDecodeError(errmsg, string, idx) - - def scan_once(string, idx): - if idx < 0: - # Ensure the same behavior as the C speedup, otherwise - # this would work for *some* negative string indices due - # to the behavior of __getitem__ for strings. #98 - raise JSONDecodeError('Expecting value', string, idx) - try: - return _scan_once(string, idx) - finally: - memo.clear() - - return scan_once - -make_scanner = c_make_scanner or py_make_scanner diff --git a/contrib/python/simplejson/simplejson/tool.py b/contrib/python/simplejson/simplejson/tool.py deleted file mode 100644 index 062e8e2c181..00000000000 --- a/contrib/python/simplejson/simplejson/tool.py +++ /dev/null @@ -1,42 +0,0 @@ -r"""Command-line tool to validate and pretty-print JSON - -Usage:: - - $ echo '{"json":"obj"}' | python -m simplejson.tool - { - "json": "obj" - } - $ echo '{ 1.2:3.4}' | python -m simplejson.tool - Expecting property name: line 1 column 2 (char 2) - -""" -from __future__ import with_statement -import sys -import simplejson as json - -def main(): - if len(sys.argv) == 1: - infile = sys.stdin - outfile = sys.stdout - elif len(sys.argv) == 2: - infile = open(sys.argv[1], 'r') - outfile = sys.stdout - elif len(sys.argv) == 3: - infile = open(sys.argv[1], 'r') - outfile = open(sys.argv[2], 'w') - else: - raise SystemExit(sys.argv[0] + " [infile [outfile]]") - with infile: - try: - obj = json.load(infile, - object_pairs_hook=json.OrderedDict, - use_decimal=True) - except ValueError: - raise SystemExit(sys.exc_info()[1]) - with outfile: - json.dump(obj, outfile, sort_keys=True, indent=' ', use_decimal=True) - outfile.write('\n') - - -if __name__ == '__main__': - main() diff --git a/contrib/python/simplejson/ya.make b/contrib/python/simplejson/ya.make deleted file mode 100644 index b0890214d7b..00000000000 --- a/contrib/python/simplejson/ya.make +++ /dev/null @@ -1,48 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY23_LIBRARY() - -VERSION(3.18.4) - -LICENSE(MIT) - -NO_COMPILER_WARNINGS() - -NO_LINT() - -NO_CHECK_IMPORTS( - simplejson.ordered_dict -) - -SRCS( - simplejson/_speedups.c -) - -PY_REGISTER( - simplejson._speedups -) - -PY_SRCS( - TOP_LEVEL - simplejson/__init__.py - simplejson/compat.py - simplejson/decoder.py - simplejson/encoder.py - simplejson/errors.py - simplejson/ordered_dict.py - simplejson/raw_json.py - simplejson/scanner.py - simplejson/tool.py -) - -RESOURCE_FILES( - PREFIX contrib/python/simplejson/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/toolz/py2/.dist-info/METADATA b/contrib/python/toolz/py2/.dist-info/METADATA deleted file mode 100644 index c43bc308d4f..00000000000 --- a/contrib/python/toolz/py2/.dist-info/METADATA +++ /dev/null @@ -1,159 +0,0 @@ -Metadata-Version: 2.1 -Name: toolz -Version: 0.10.0 -Summary: List processing tools and functional utilities -Home-page: https://github.com/pytoolz/toolz/ -Author: https://raw.github.com/pytoolz/toolz/master/AUTHORS.md -Maintainer: Matthew Rocklin -Maintainer-email: mrocklin@gmail.com -License: BSD -Keywords: functional utility itertools functools -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: License :: OSI Approved :: BSD License -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* - -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies, partial - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 2.7 and Python 3.4+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://travis-ci.org/pytoolz/toolz.svg?branch=master - :target: https://travis-ci.org/pytoolz/toolz -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz - - diff --git a/contrib/python/toolz/py2/.dist-info/top_level.txt b/contrib/python/toolz/py2/.dist-info/top_level.txt deleted file mode 100644 index e58ef014ac8..00000000000 --- a/contrib/python/toolz/py2/.dist-info/top_level.txt +++ /dev/null @@ -1,2 +0,0 @@ -tlz -toolz diff --git a/contrib/python/toolz/py2/LICENSE.txt b/contrib/python/toolz/py2/LICENSE.txt deleted file mode 100644 index eeb91b202ca..00000000000 --- a/contrib/python/toolz/py2/LICENSE.txt +++ /dev/null @@ -1,28 +0,0 @@ -Copyright (c) 2013 Matthew Rocklin - -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - - a. Redistributions of source code must retain the above copyright notice, - this list of conditions and the following disclaimer. - b. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - c. Neither the name of toolz nor the names of its contributors - may be used to endorse or promote products derived from this software - without specific prior written permission. - - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR -ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL -DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR -SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER -CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY -OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH -DAMAGE. diff --git a/contrib/python/toolz/py2/README.rst b/contrib/python/toolz/py2/README.rst deleted file mode 100644 index 099c3ff8071..00000000000 --- a/contrib/python/toolz/py2/README.rst +++ /dev/null @@ -1,132 +0,0 @@ -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies, partial - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 2.7 and Python 3.4+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://travis-ci.org/pytoolz/toolz.svg?branch=master - :target: https://travis-ci.org/pytoolz/toolz -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz diff --git a/contrib/python/toolz/py2/tlz/__init__.py b/contrib/python/toolz/py2/tlz/__init__.py deleted file mode 100644 index 9c9c84afe1b..00000000000 --- a/contrib/python/toolz/py2/tlz/__init__.py +++ /dev/null @@ -1,9 +0,0 @@ -"""``tlz`` mirrors the ``toolz`` API and uses ``cytoolz`` if possible. - -The ``tlz`` package is installed when ``toolz`` is installed. It provides -a convenient way to use functions from ``cytoolz``--a faster Cython -implementation of ``toolz``--if it is installed, otherwise it uses -functions from ``toolz``. -""" - -from . import _build_tlz diff --git a/contrib/python/toolz/py2/tlz/_build_tlz.py b/contrib/python/toolz/py2/tlz/_build_tlz.py deleted file mode 100644 index 3c017a542c1..00000000000 --- a/contrib/python/toolz/py2/tlz/_build_tlz.py +++ /dev/null @@ -1,100 +0,0 @@ -import sys -import types -import toolz -from importlib import import_module - - -class TlzLoader(object): - """ Finds and loads ``tlz`` modules when added to sys.meta_path""" - def __init__(self): - self.always_from_toolz = { - toolz.pipe, - } - - def _load_toolz(self, fullname): - rv = {} - package, dot, submodules = fullname.partition('.') - try: - module_name = ''.join(['cytoolz', dot, submodules]) - rv['cytoolz'] = import_module(module_name) - except ImportError: - pass - try: - module_name = ''.join(['toolz', dot, submodules]) - rv['toolz'] = import_module(module_name) - except ImportError: - pass - if not rv: - raise ImportError(fullname) - return rv - - def find_module(self, fullname, path=None): # pragma: py3 no cover - package, dot, submodules = fullname.partition('.') - if package == 'tlz': - return self - - def load_module(self, fullname): # pragma: py3 no cover - if fullname in sys.modules: # pragma: no cover - return sys.modules[fullname] - spec = TlzSpec(fullname, self) - module = self.create_module(spec) - sys.modules[fullname] = module - self.exec_module(module) - return module - - def find_spec(self, fullname, path, target=None): # pragma: no cover - package, dot, submodules = fullname.partition('.') - if package == 'tlz': - return TlzSpec(fullname, self) - - def create_module(self, spec): - return types.ModuleType(spec.name) - - def exec_module(self, module): - toolz_mods = self._load_toolz(module.__name__) - fast_mod = toolz_mods.get('cytoolz') or toolz_mods['toolz'] - slow_mod = toolz_mods.get('toolz') or toolz_mods['cytoolz'] - module.__dict__.update(toolz.merge(fast_mod.__dict__, module.__dict__)) - package = fast_mod.__package__ - if package is not None: - package, dot, submodules = package.partition('.') - module.__package__ = ''.join(['tlz', dot, submodules]) - if not module.__doc__: - module.__doc__ = fast_mod.__doc__ - - # show file from toolz during introspection - module.__file__ = slow_mod.__file__ - - for k, v in fast_mod.__dict__.items(): - tv = slow_mod.__dict__.get(k) - try: - hash(tv) - except TypeError: - tv = None - if tv in self.always_from_toolz: - module.__dict__[k] = tv - elif ( - isinstance(v, types.ModuleType) - and v.__package__ == fast_mod.__name__ - ): - package, dot, submodules = v.__name__.partition('.') - module_name = ''.join(['tlz', dot, submodules]) - submodule = import_module(module_name) - module.__dict__[k] = submodule - - -class TlzSpec(object): - def __init__(self, name, loader): - self.name = name - self.loader = loader - self.origin = None - self.submodule_search_locations = [] - self.loader_state = None - self.cached = None - self.parent = None - self.has_location = False - - -tlz_loader = TlzLoader() -sys.meta_path.append(tlz_loader) -tlz_loader.exec_module(sys.modules['tlz']) diff --git a/contrib/python/toolz/py2/toolz/__init__.py b/contrib/python/toolz/py2/toolz/__init__.py deleted file mode 100644 index 7fa86ab4739..00000000000 --- a/contrib/python/toolz/py2/toolz/__init__.py +++ /dev/null @@ -1,22 +0,0 @@ -from .itertoolz import * - -from .functoolz import * - -from .dicttoolz import * - -from .recipes import * - -from .compatibility import map, filter - -from functools import partial, reduce - -sorted = sorted - -# Aliases -comp = compose - -from . import curried, sandbox - -functoolz._sigs.create_signature_registry() - -__version__ = '0.10.0' diff --git a/contrib/python/toolz/py2/toolz/_signatures.py b/contrib/python/toolz/py2/toolz/_signatures.py deleted file mode 100644 index c55a778b3bc..00000000000 --- a/contrib/python/toolz/py2/toolz/_signatures.py +++ /dev/null @@ -1,832 +0,0 @@ -"""Internal module for better introspection of builtins. - -The main functions are ``is_builtin_valid_args``, ``is_builtin_partial_args``, -and ``has_unknown_args``. Other functions in this module support these three. - -Notably, we create a ``signatures`` registry to enable introspection of -builtin functions in any Python version. This includes builtins that -have more than one valid signature. Currently, the registry includes -builtins from ``builtins``, ``functools``, ``itertools``, and ``operator`` -modules. More can be added as requested. We don't guarantee full coverage. - -Everything in this module should be regarded as implementation details. -Users should try to not use this module directly. -""" -import functools -import inspect -import itertools -import operator -from importlib import import_module - -from .compatibility import PY3 -from .functoolz import (is_partial_args, is_arity, has_varargs, - has_keywords, num_required_args) - -if PY3: # pragma: py2 no cover - import builtins -else: # pragma: py3 no cover - import __builtin__ as builtins - -# We mock builtin callables using lists of tuples with lambda functions. -# -# The tuple spec is (num_position_args, lambda_func, keyword_only_args). -# -# num_position_args: -# - The number of positional-only arguments. If not specified, -# all positional arguments are considered positional-only. -# -# lambda_func: -# - lambda function that matches a signature of a builtin, but does -# not include keyword-only arguments. -# -# keyword_only_args: (optional) -# - Tuple of keyword-only argumemts. - -module_info = {} - -module_info[builtins] = dict( - abs=[ - lambda x: None], - all=[ - lambda iterable: None], - any=[ - lambda iterable: None], - apply=[ - lambda object: None, - lambda object, args: None, - lambda object, args, kwargs: None], - ascii=[ - lambda obj: None], - bin=[ - lambda number: None], - bool=[ - lambda x=False: None], - buffer=[ - lambda object: None, - lambda object, offset: None, - lambda object, offset, size: None], - bytearray=[ - lambda: None, - lambda int: None, - lambda string, encoding='utf8', errors='strict': None], - callable=[ - lambda obj: None], - chr=[ - lambda i: None], - classmethod=[ - lambda function: None], - cmp=[ - lambda x, y: None], - coerce=[ - lambda x, y: None], - complex=[ - lambda real=0, imag=0: None], - delattr=[ - lambda obj, name: None], - dict=[ - lambda **kwargs: None, - lambda mapping, **kwargs: None], - dir=[ - lambda: None, - lambda object: None], - divmod=[ - lambda x, y: None], - enumerate=[ - (0, lambda iterable, start=0: None)], - eval=[ - lambda source: None, - lambda source, globals: None, - lambda source, globals, locals: None], - execfile=[ - lambda filename: None, - lambda filename, globals: None, - lambda filename, globals, locals: None], - file=[ - (0, lambda name, mode='r', buffering=-1: None)], - filter=[ - lambda function, iterable: None], - float=[ - lambda x=0.0: None], - format=[ - lambda value: None, - lambda value, format_spec: None], - frozenset=[ - lambda: None, - lambda iterable: None], - getattr=[ - lambda object, name: None, - lambda object, name, default: None], - globals=[ - lambda: None], - hasattr=[ - lambda obj, name: None], - hash=[ - lambda obj: None], - hex=[ - lambda number: None], - id=[ - lambda obj: None], - input=[ - lambda: None, - lambda prompt: None], - int=[ - lambda x=0: None, - (0, lambda x, base=10: None)], - intern=[ - lambda string: None], - isinstance=[ - lambda obj, class_or_tuple: None], - issubclass=[ - lambda cls, class_or_tuple: None], - iter=[ - lambda iterable: None, - lambda callable, sentinel: None], - len=[ - lambda obj: None], - list=[ - lambda: None, - lambda iterable: None], - locals=[ - lambda: None], - long=[ - lambda x=0: None, - (0, lambda x, base=10: None)], - map=[ - lambda func, sequence, *iterables: None], - memoryview=[ - (0, lambda object: None)], - next=[ - lambda iterator: None, - lambda iterator, default: None], - object=[ - lambda: None], - oct=[ - lambda number: None], - ord=[ - lambda c: None], - pow=[ - lambda x, y: None, - lambda x, y, z: None], - property=[ - lambda fget=None, fset=None, fdel=None, doc=None: None], - range=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - raw_input=[ - lambda: None, - lambda prompt: None], - reduce=[ - lambda function, sequence: None, - lambda function, sequence, initial: None], - reload=[ - lambda module: None], - repr=[ - lambda obj: None], - reversed=[ - lambda sequence: None], - round=[ - (0, lambda number, ndigits=0: None)], - set=[ - lambda: None, - lambda iterable: None], - setattr=[ - lambda obj, name, value: None], - slice=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - staticmethod=[ - lambda function: None], - sum=[ - lambda iterable: None, - lambda iterable, start: None], - super=[ - lambda type: None, - lambda type, obj: None], - tuple=[ - lambda: None, - lambda iterable: None], - type=[ - lambda object: None, - lambda name, bases, dict: None], - unichr=[ - lambda i: None], - unicode=[ - lambda object: None, - lambda string='', encoding='utf8', errors='strict': None], - vars=[ - lambda: None, - lambda object: None], - xrange=[ - lambda stop: None, - lambda start, stop: None, - lambda start, stop, step: None], - zip=[ - lambda *iterables: None], - __build_class__=[ - (2, lambda func, name, *bases, **kwds: None, ('metaclass',))], - __import__=[ - (0, lambda name, globals=None, locals=None, fromlist=None, - level=None: None)], -) -module_info[builtins]['exec'] = [ - lambda source: None, - lambda source, globals: None, - lambda source, globals, locals: None] - -if PY3: # pragma: py2 no cover - module_info[builtins].update( - breakpoint=[ - lambda *args, **kws: None], - bytes=[ - lambda: None, - lambda int: None, - lambda string, encoding='utf8', errors='strict': None], - compile=[ - (0, lambda source, filename, mode, flags=0, - dont_inherit=False, optimize=-1: None)], - max=[ - (1, lambda iterable: None, ('default', 'key',)), - (1, lambda arg1, arg2, *args: None, ('key',))], - min=[ - (1, lambda iterable: None, ('default', 'key',)), - (1, lambda arg1, arg2, *args: None, ('key',))], - open=[ - (0, lambda file, mode='r', buffering=-1, encoding=None, - errors=None, newline=None, closefd=True, opener=None: None)], - sorted=[ - (1, lambda iterable: None, ('key', 'reverse'))], - str=[ - lambda object='', encoding='utf', errors='strict': None], - ) - module_info[builtins]['print'] = [ - (0, lambda *args: None, ('sep', 'end', 'file', 'flush',))] - -else: # pragma: py3 no cover - module_info[builtins].update( - bytes=[ - lambda object='': None], - compile=[ - (0, lambda source, filename, mode, flags=0, - dont_inherit=False: None)], - max=[ - (1, lambda iterable, *args: None, ('key',))], - min=[ - (1, lambda iterable, *args: None, ('key',))], - open=[ - (0, lambda file, mode='r', buffering=-1: None)], - sorted=[ - lambda iterable, cmp=None, key=None, reverse=False: None], - str=[ - lambda object='': None], - ) - module_info[builtins]['print'] = [ - (0, lambda *args: None, ('sep', 'end', 'file',))] - -module_info[functools] = dict( - cmp_to_key=[ - (0, lambda mycmp: None)], - partial=[ - lambda func, *args, **kwargs: None], - partialmethod=[ - lambda func, *args, **kwargs: None], - reduce=[ - lambda function, sequence: None, - lambda function, sequence, initial: None], -) - -module_info[itertools] = dict( - accumulate=[ - (0, lambda iterable, func=None: None)], - chain=[ - lambda *iterables: None], - combinations=[ - (0, lambda iterable, r: None)], - combinations_with_replacement=[ - (0, lambda iterable, r: None)], - compress=[ - (0, lambda data, selectors: None)], - count=[ - lambda start=0, step=1: None], - cycle=[ - lambda iterable: None], - dropwhile=[ - lambda predicate, iterable: None], - filterfalse=[ - lambda function, sequence: None], - groupby=[ - (0, lambda iterable, key=None: None)], - ifilter=[ - lambda function, sequence: None], - ifilterfalse=[ - lambda function, sequence: None], - imap=[ - lambda func, sequence, *iterables: None], - islice=[ - lambda iterable, stop: None, - lambda iterable, start, stop: None, - lambda iterable, start, stop, step: None], - izip=[ - lambda *iterables: None], - izip_longest=[ - (0, lambda *iterables: None, ('fillvalue',))], - permutations=[ - (0, lambda iterable, r=0: None)], - repeat=[ - (0, lambda object, times=0: None)], - starmap=[ - lambda function, sequence: None], - takewhile=[ - lambda predicate, iterable: None], - tee=[ - lambda iterable: None, - lambda iterable, n: None], - zip_longest=[ - (0, lambda *iterables: None, ('fillvalue',))], -) - -if PY3: # pragma: py2 no cover - module_info[itertools].update( - product=[ - (0, lambda *iterables: None, ('repeat',))], - ) -else: # pragma: py3 no cover - module_info[itertools].update( - product=[ - lambda *iterables: None], - ) - -module_info[operator] = dict( - __abs__=[ - lambda a: None], - __add__=[ - lambda a, b: None], - __and__=[ - lambda a, b: None], - __concat__=[ - lambda a, b: None], - __contains__=[ - lambda a, b: None], - __delitem__=[ - lambda a, b: None], - __delslice__=[ - lambda a, b, c: None], - __div__=[ - lambda a, b: None], - __eq__=[ - lambda a, b: None], - __floordiv__=[ - lambda a, b: None], - __ge__=[ - lambda a, b: None], - __getitem__=[ - lambda a, b: None], - __getslice__=[ - lambda a, b, c: None], - __gt__=[ - lambda a, b: None], - __iadd__=[ - lambda a, b: None], - __iand__=[ - lambda a, b: None], - __iconcat__=[ - lambda a, b: None], - __idiv__=[ - lambda a, b: None], - __ifloordiv__=[ - lambda a, b: None], - __ilshift__=[ - lambda a, b: None], - __imatmul__=[ - lambda a, b: None], - __imod__=[ - lambda a, b: None], - __imul__=[ - lambda a, b: None], - __index__=[ - lambda a: None], - __inv__=[ - lambda a: None], - __invert__=[ - lambda a: None], - __ior__=[ - lambda a, b: None], - __ipow__=[ - lambda a, b: None], - __irepeat__=[ - lambda a, b: None], - __irshift__=[ - lambda a, b: None], - __isub__=[ - lambda a, b: None], - __itruediv__=[ - lambda a, b: None], - __ixor__=[ - lambda a, b: None], - __le__=[ - lambda a, b: None], - __lshift__=[ - lambda a, b: None], - __lt__=[ - lambda a, b: None], - __matmul__=[ - lambda a, b: None], - __mod__=[ - lambda a, b: None], - __mul__=[ - lambda a, b: None], - __ne__=[ - lambda a, b: None], - __neg__=[ - lambda a: None], - __not__=[ - lambda a: None], - __or__=[ - lambda a, b: None], - __pos__=[ - lambda a: None], - __pow__=[ - lambda a, b: None], - __repeat__=[ - lambda a, b: None], - __rshift__=[ - lambda a, b: None], - __setitem__=[ - lambda a, b, c: None], - __setslice__=[ - lambda a, b, c, d: None], - __sub__=[ - lambda a, b: None], - __truediv__=[ - lambda a, b: None], - __xor__=[ - lambda a, b: None], - _abs=[ - lambda x: None], - _compare_digest=[ - lambda a, b: None], - abs=[ - lambda a: None], - add=[ - lambda a, b: None], - and_=[ - lambda a, b: None], - attrgetter=[ - lambda attr, *args: None], - concat=[ - lambda a, b: None], - contains=[ - lambda a, b: None], - countOf=[ - lambda a, b: None], - delitem=[ - lambda a, b: None], - delslice=[ - lambda a, b, c: None], - div=[ - lambda a, b: None], - eq=[ - lambda a, b: None], - floordiv=[ - lambda a, b: None], - ge=[ - lambda a, b: None], - getitem=[ - lambda a, b: None], - getslice=[ - lambda a, b, c: None], - gt=[ - lambda a, b: None], - iadd=[ - lambda a, b: None], - iand=[ - lambda a, b: None], - iconcat=[ - lambda a, b: None], - idiv=[ - lambda a, b: None], - ifloordiv=[ - lambda a, b: None], - ilshift=[ - lambda a, b: None], - imatmul=[ - lambda a, b: None], - imod=[ - lambda a, b: None], - imul=[ - lambda a, b: None], - index=[ - lambda a: None], - indexOf=[ - lambda a, b: None], - inv=[ - lambda a: None], - invert=[ - lambda a: None], - ior=[ - lambda a, b: None], - ipow=[ - lambda a, b: None], - irepeat=[ - lambda a, b: None], - irshift=[ - lambda a, b: None], - is_=[ - lambda a, b: None], - is_not=[ - lambda a, b: None], - isCallable=[ - lambda a: None], - isMappingType=[ - lambda a: None], - isNumberType=[ - lambda a: None], - isSequenceType=[ - lambda a: None], - isub=[ - lambda a, b: None], - itemgetter=[ - lambda item, *args: None], - itruediv=[ - lambda a, b: None], - ixor=[ - lambda a, b: None], - le=[ - lambda a, b: None], - length_hint=[ - lambda obj: None, - lambda obj, default: None], - lshift=[ - lambda a, b: None], - lt=[ - lambda a, b: None], - matmul=[ - lambda a, b: None], - methodcaller=[ - lambda name, *args, **kwargs: None], - mod=[ - lambda a, b: None], - mul=[ - lambda a, b: None], - ne=[ - lambda a, b: None], - neg=[ - lambda a: None], - not_=[ - lambda a: None], - or_=[ - lambda a, b: None], - pos=[ - lambda a: None], - pow=[ - lambda a, b: None], - repeat=[ - lambda a, b: None], - rshift=[ - lambda a, b: None], - sequenceIncludes=[ - lambda a, b: None], - setitem=[ - lambda a, b, c: None], - setslice=[ - lambda a, b, c, d: None], - sub=[ - lambda a, b: None], - truediv=[ - lambda a, b: None], - truth=[ - lambda a: None], - xor=[ - lambda a, b: None], -) - -module_info['toolz'] = dict( - curry=[ - (0, lambda *args, **kwargs: None)], - excepts=[ - (0, lambda exc, func, handler=None: None)], - flip=[ - (0, lambda func=None, a=None, b=None: None)], - juxt=[ - (0, lambda *funcs: None)], - memoize=[ - (0, lambda func=None, cache=None, key=None: None)], -) - -module_info['toolz.functoolz'] = dict( - Compose=[ - (0, lambda funcs: None)], - InstanceProperty=[ - (0, lambda fget=None, fset=None, fdel=None, doc=None, - classval=None: None)], -) - -if PY3: # pragma: py2 no cover - def num_pos_args(sigspec): - """ Return the number of positional arguments. ``f(x, y=1)`` has 1""" - return sum(1 for x in sigspec.parameters.values() - if x.kind == x.POSITIONAL_OR_KEYWORD - and x.default is x.empty) - - def get_exclude_keywords(num_pos_only, sigspec): - """ Return the names of position-only arguments if func has **kwargs""" - if num_pos_only == 0: - return () - has_kwargs = any(x.kind == x.VAR_KEYWORD - for x in sigspec.parameters.values()) - if not has_kwargs: - return () - pos_args = list(sigspec.parameters.values())[:num_pos_only] - return tuple(x.name for x in pos_args) - - def signature_or_spec(func): - try: - return inspect.signature(func) - except (ValueError, TypeError): - return None - -else: # pragma: py3 no cover - def num_pos_args(sigspec): - """ Return the number of positional arguments. ``f(x, y=1)`` has 1""" - if sigspec.defaults: - return len(sigspec.args) - len(sigspec.defaults) - return len(sigspec.args) - - def get_exclude_keywords(num_pos_only, sigspec): - """ Return the names of position-only arguments if func has **kwargs""" - if num_pos_only == 0: - return () - has_kwargs = sigspec.keywords is not None - if not has_kwargs: - return () - return tuple(sigspec.args[:num_pos_only]) - - def signature_or_spec(func): - try: - return inspect.getargspec(func) - except TypeError: - return None - - -def expand_sig(sig): - """ Convert the signature spec in ``module_info`` to add to ``signatures`` - - The input signature spec is one of: - - ``lambda_func`` - - ``(num_position_args, lambda_func)`` - - ``(num_position_args, lambda_func, keyword_only_args)`` - - The output signature spec is: - ``(num_position_args, lambda_func, keyword_exclude, sigspec)`` - - where ``keyword_exclude`` includes keyword only arguments and, if variadic - keywords is present, the names of position-only argument. The latter is - included to support builtins such as ``partial(func, *args, **kwargs)``, - which allows ``func=`` to be used as a keyword even though it's the name - of a positional argument. - """ - if isinstance(sig, tuple): - if len(sig) == 3: - num_pos_only, func, keyword_only = sig - assert isinstance(sig[-1], tuple) - else: - num_pos_only, func = sig - keyword_only = () - sigspec = signature_or_spec(func) - else: - func = sig - sigspec = signature_or_spec(func) - num_pos_only = num_pos_args(sigspec) - keyword_only = () - keyword_exclude = get_exclude_keywords(num_pos_only, sigspec) - return num_pos_only, func, keyword_only + keyword_exclude, sigspec - - -signatures = {} - - -def create_signature_registry(module_info=module_info, signatures=signatures): - for module, info in module_info.items(): - if isinstance(module, str): - module = import_module(module) - for name, sigs in info.items(): - if hasattr(module, name): - new_sigs = tuple(expand_sig(sig) for sig in sigs) - signatures[getattr(module, name)] = new_sigs - - -def check_valid(sig, args, kwargs): - """ Like ``is_valid_args`` for the given signature spec""" - num_pos_only, func, keyword_exclude, sigspec = sig - if len(args) < num_pos_only: - return False - if keyword_exclude: - kwargs = dict(kwargs) - for item in keyword_exclude: - kwargs.pop(item, None) - try: - func(*args, **kwargs) - return True - except TypeError: - return False - - -def _is_valid_args(func, args, kwargs): - """ Like ``is_valid_args`` for builtins in our ``signatures`` registry""" - if func not in signatures: - return None - sigs = signatures[func] - return any(check_valid(sig, args, kwargs) for sig in sigs) - - -def check_partial(sig, args, kwargs): - """ Like ``is_partial_args`` for the given signature spec""" - num_pos_only, func, keyword_exclude, sigspec = sig - if len(args) < num_pos_only: - pad = (None,) * (num_pos_only - len(args)) - args = args + pad - if keyword_exclude: - kwargs = dict(kwargs) - for item in keyword_exclude: - kwargs.pop(item, None) - return is_partial_args(func, args, kwargs, sigspec) - - -def _is_partial_args(func, args, kwargs): - """ Like ``is_partial_args`` for builtins in our ``signatures`` registry""" - if func not in signatures: - return None - sigs = signatures[func] - return any(check_partial(sig, args, kwargs) for sig in sigs) - - -def check_arity(n, sig): - num_pos_only, func, keyword_exclude, sigspec = sig - if keyword_exclude or num_pos_only > n: - return False - return is_arity(n, func, sigspec) - - -def _is_arity(n, func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_arity(n, sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_varargs(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - return has_varargs(func, sigspec) - - -def _has_varargs(func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_varargs(sig) for sig in sigs] - if all(checks): - return True - elif any(checks): # pragma: py2 no cover - return None - return False - - -def check_keywords(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - if keyword_exclude: - return True - return has_keywords(func, sigspec) - - -def _has_keywords(func): - if func not in signatures: - return None - sigs = signatures[func] - checks = [check_keywords(sig) for sig in sigs] - if all(checks): - return True - elif any(checks): - return None - return False - - -def check_required_args(sig): - num_pos_only, func, keyword_exclude, sigspec = sig - return num_required_args(func, sigspec) - - -def _num_required_args(func): - if func not in signatures: - return None - sigs = signatures[func] - vals = [check_required_args(sig) for sig in sigs] - val = vals[0] - if all(x == val for x in vals): - return val - return None diff --git a/contrib/python/toolz/py2/toolz/compatibility.py b/contrib/python/toolz/py2/toolz/compatibility.py deleted file mode 100644 index 51e3673fadf..00000000000 --- a/contrib/python/toolz/py2/toolz/compatibility.py +++ /dev/null @@ -1,34 +0,0 @@ -import operator -import sys -PY3 = sys.version_info[0] > 2 -PY34 = sys.version_info[0] == 3 and sys.version_info[1] == 4 -PYPY = hasattr(sys, 'pypy_version_info') - -__all__ = ('map', 'filter', 'range', 'zip', 'reduce', 'zip_longest', - 'iteritems', 'iterkeys', 'itervalues', 'filterfalse', - 'PY3', 'PY34', 'PYPY') - -if PY3: - map = map - filter = filter - range = range - zip = zip - from functools import reduce - from itertools import zip_longest - from itertools import filterfalse - iteritems = operator.methodcaller('items') - iterkeys = operator.methodcaller('keys') - itervalues = operator.methodcaller('values') - from collections.abc import Sequence -else: - range = xrange - reduce = reduce - from itertools import imap as map - from itertools import ifilter as filter - from itertools import ifilterfalse as filterfalse - from itertools import izip as zip - from itertools import izip_longest as zip_longest - iteritems = operator.methodcaller('iteritems') - iterkeys = operator.methodcaller('iterkeys') - itervalues = operator.methodcaller('itervalues') - from collections import Sequence diff --git a/contrib/python/toolz/py2/toolz/curried/__init__.py b/contrib/python/toolz/py2/toolz/curried/__init__.py deleted file mode 100644 index 356eddbd3ba..00000000000 --- a/contrib/python/toolz/py2/toolz/curried/__init__.py +++ /dev/null @@ -1,103 +0,0 @@ -""" -Alternate namespace for toolz such that all functions are curried - -Currying provides implicit partial evaluation of all functions - -Example: - - Get usually requires two arguments, an index and a collection - >>> from toolz.curried import get - >>> get(0, ('a', 'b')) - 'a' - - When we use it in higher order functions we often want to pass a partially - evaluated form - >>> data = [(1, 2), (11, 22), (111, 222)] - >>> list(map(lambda seq: get(0, seq), data)) - [1, 11, 111] - - The curried version allows simple expression of partial evaluation - >>> list(map(get(0), data)) - [1, 11, 111] - -See Also: - toolz.functoolz.curry -""" -import toolz -from . import operator -from toolz import ( - apply, - comp, - complement, - compose, - compose_left, - concat, - concatv, - count, - curry, - diff, - first, - flip, - frequencies, - identity, - interleave, - isdistinct, - isiterable, - juxt, - last, - memoize, - merge_sorted, - peek, - pipe, - second, - thread_first, - thread_last, -) -from .exceptions import merge, merge_with - -accumulate = toolz.curry(toolz.accumulate) -assoc = toolz.curry(toolz.assoc) -assoc_in = toolz.curry(toolz.assoc_in) -cons = toolz.curry(toolz.cons) -countby = toolz.curry(toolz.countby) -dissoc = toolz.curry(toolz.dissoc) -do = toolz.curry(toolz.do) -drop = toolz.curry(toolz.drop) -excepts = toolz.curry(toolz.excepts) -filter = toolz.curry(toolz.filter) -get = toolz.curry(toolz.get) -get_in = toolz.curry(toolz.get_in) -groupby = toolz.curry(toolz.groupby) -interpose = toolz.curry(toolz.interpose) -itemfilter = toolz.curry(toolz.itemfilter) -itemmap = toolz.curry(toolz.itemmap) -iterate = toolz.curry(toolz.iterate) -join = toolz.curry(toolz.join) -keyfilter = toolz.curry(toolz.keyfilter) -keymap = toolz.curry(toolz.keymap) -map = toolz.curry(toolz.map) -mapcat = toolz.curry(toolz.mapcat) -nth = toolz.curry(toolz.nth) -partial = toolz.curry(toolz.partial) -partition = toolz.curry(toolz.partition) -partition_all = toolz.curry(toolz.partition_all) -partitionby = toolz.curry(toolz.partitionby) -peekn = toolz.curry(toolz.peekn) -pluck = toolz.curry(toolz.pluck) -random_sample = toolz.curry(toolz.random_sample) -reduce = toolz.curry(toolz.reduce) -reduceby = toolz.curry(toolz.reduceby) -remove = toolz.curry(toolz.remove) -sliding_window = toolz.curry(toolz.sliding_window) -sorted = toolz.curry(toolz.sorted) -tail = toolz.curry(toolz.tail) -take = toolz.curry(toolz.take) -take_nth = toolz.curry(toolz.take_nth) -topk = toolz.curry(toolz.topk) -unique = toolz.curry(toolz.unique) -update_in = toolz.curry(toolz.update_in) -valfilter = toolz.curry(toolz.valfilter) -valmap = toolz.curry(toolz.valmap) - -del exceptions -del toolz diff --git a/contrib/python/toolz/py2/toolz/curried/exceptions.py b/contrib/python/toolz/py2/toolz/curried/exceptions.py deleted file mode 100644 index 75a52bbbf27..00000000000 --- a/contrib/python/toolz/py2/toolz/curried/exceptions.py +++ /dev/null @@ -1,18 +0,0 @@ -import toolz - - -__all__ = ['merge_with', 'merge'] - - -@toolz.curry -def merge_with(func, d, *dicts, **kwargs): - return toolz.merge_with(func, d, *dicts, **kwargs) - - -@toolz.curry -def merge(d, *dicts, **kwargs): - return toolz.merge(d, *dicts, **kwargs) - - -merge_with.__doc__ = toolz.merge_with.__doc__ -merge.__doc__ = toolz.merge.__doc__ diff --git a/contrib/python/toolz/py2/toolz/curried/operator.py b/contrib/python/toolz/py2/toolz/curried/operator.py deleted file mode 100644 index 8bc9e52317a..00000000000 --- a/contrib/python/toolz/py2/toolz/curried/operator.py +++ /dev/null @@ -1,23 +0,0 @@ -from __future__ import absolute_import - -import operator - -from toolz.functoolz import curry, num_required_args, has_keywords - - -def should_curry(f): - num = num_required_args(f) - return num is None or num > 1 or num == 1 and has_keywords(f) is not False - - -locals().update( - {name: curry(f) if should_curry(f) else f - for name, f in vars(operator).items() if callable(f)}, -) - -# Clean up the namespace. -del curry -del num_required_args -del has_keywords -del operator -del should_curry diff --git a/contrib/python/toolz/py2/toolz/dicttoolz.py b/contrib/python/toolz/py2/toolz/dicttoolz.py deleted file mode 100644 index 91bff23cef7..00000000000 --- a/contrib/python/toolz/py2/toolz/dicttoolz.py +++ /dev/null @@ -1,337 +0,0 @@ -import operator -from toolz.compatibility import (map, zip, iteritems, iterkeys, itervalues, - reduce) - -__all__ = ('merge', 'merge_with', 'valmap', 'keymap', 'itemmap', - 'valfilter', 'keyfilter', 'itemfilter', - 'assoc', 'dissoc', 'assoc_in', 'update_in', 'get_in') - - -def _get_factory(f, kwargs): - factory = kwargs.pop('factory', dict) - if kwargs: - raise TypeError("{}() got an unexpected keyword argument " - "'{}'".format(f.__name__, kwargs.popitem()[0])) - return factory - - -def merge(*dicts, **kwargs): - """ Merge a collection of dictionaries - - >>> merge({1: 'one'}, {2: 'two'}) - {1: 'one', 2: 'two'} - - Later dictionaries have precedence - - >>> merge({1: 2, 3: 4}, {3: 3, 4: 4}) - {1: 2, 3: 3, 4: 4} - - See Also: - merge_with - """ - if len(dicts) == 1 and not isinstance(dicts[0], dict): - dicts = dicts[0] - factory = _get_factory(merge, kwargs) - - rv = factory() - for d in dicts: - rv.update(d) - return rv - - -def merge_with(func, *dicts, **kwargs): - """ Merge dictionaries and apply function to combined values - - A key may occur in more than one dict, and all values mapped from the key - will be passed to the function as a list, such as func([val1, val2, ...]). - - >>> merge_with(sum, {1: 1, 2: 2}, {1: 10, 2: 20}) - {1: 11, 2: 22} - - >>> merge_with(first, {1: 1, 2: 2}, {2: 20, 3: 30}) # doctest: +SKIP - {1: 1, 2: 2, 3: 30} - - See Also: - merge - """ - if len(dicts) == 1 and not isinstance(dicts[0], dict): - dicts = dicts[0] - factory = _get_factory(merge_with, kwargs) - - result = factory() - for d in dicts: - for k, v in iteritems(d): - if k not in result: - result[k] = [v] - else: - result[k].append(v) - return valmap(func, result, factory) - - -def valmap(func, d, factory=dict): - """ Apply function to values of dictionary - - >>> bills = {"Alice": [20, 15, 30], "Bob": [10, 35]} - >>> valmap(sum, bills) # doctest: +SKIP - {'Alice': 65, 'Bob': 45} - - See Also: - keymap - itemmap - """ - rv = factory() - rv.update(zip(iterkeys(d), map(func, itervalues(d)))) - return rv - - -def keymap(func, d, factory=dict): - """ Apply function to keys of dictionary - - >>> bills = {"Alice": [20, 15, 30], "Bob": [10, 35]} - >>> keymap(str.lower, bills) # doctest: +SKIP - {'alice': [20, 15, 30], 'bob': [10, 35]} - - See Also: - valmap - itemmap - """ - rv = factory() - rv.update(zip(map(func, iterkeys(d)), itervalues(d))) - return rv - - -def itemmap(func, d, factory=dict): - """ Apply function to items of dictionary - - >>> accountids = {"Alice": 10, "Bob": 20} - >>> itemmap(reversed, accountids) # doctest: +SKIP - {10: "Alice", 20: "Bob"} - - See Also: - keymap - valmap - """ - rv = factory() - rv.update(map(func, iteritems(d))) - return rv - - -def valfilter(predicate, d, factory=dict): - """ Filter items in dictionary by value - - >>> iseven = lambda x: x % 2 == 0 - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> valfilter(iseven, d) - {1: 2, 3: 4} - - See Also: - keyfilter - itemfilter - valmap - """ - rv = factory() - for k, v in iteritems(d): - if predicate(v): - rv[k] = v - return rv - - -def keyfilter(predicate, d, factory=dict): - """ Filter items in dictionary by key - - >>> iseven = lambda x: x % 2 == 0 - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> keyfilter(iseven, d) - {2: 3, 4: 5} - - See Also: - valfilter - itemfilter - keymap - """ - rv = factory() - for k, v in iteritems(d): - if predicate(k): - rv[k] = v - return rv - - -def itemfilter(predicate, d, factory=dict): - """ Filter items in dictionary by item - - >>> def isvalid(item): - ... k, v = item - ... return k % 2 == 0 and v < 4 - - >>> d = {1: 2, 2: 3, 3: 4, 4: 5} - >>> itemfilter(isvalid, d) - {2: 3} - - See Also: - keyfilter - valfilter - itemmap - """ - rv = factory() - for item in iteritems(d): - if predicate(item): - k, v = item - rv[k] = v - return rv - - -def assoc(d, key, value, factory=dict): - """ Return a new dict with new key value pair - - New dict has d[key] set to value. Does not modify the initial dictionary. - - >>> assoc({'x': 1}, 'x', 2) - {'x': 2} - >>> assoc({'x': 1}, 'y', 3) # doctest: +SKIP - {'x': 1, 'y': 3} - """ - d2 = factory() - d2.update(d) - d2[key] = value - return d2 - - -def dissoc(d, *keys, **kwargs): - """ Return a new dict with the given key(s) removed. - - New dict has d[key] deleted for each supplied key. - Does not modify the initial dictionary. - - >>> dissoc({'x': 1, 'y': 2}, 'y') - {'x': 1} - >>> dissoc({'x': 1, 'y': 2}, 'y', 'x') - {} - >>> dissoc({'x': 1}, 'y') # Ignores missing keys - {'x': 1} - """ - factory = _get_factory(dissoc, kwargs) - d2 = factory() - - if len(keys) < len(d) * .6: - d2.update(d) - for key in keys: - if key in d2: - del d2[key] - else: - remaining = set(d) - remaining.difference_update(keys) - for k in remaining: - d2[k] = d[k] - return d2 - - -def assoc_in(d, keys, value, factory=dict): - """ Return a new dict with new, potentially nested, key value pair - - >>> purchase = {'name': 'Alice', - ... 'order': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> assoc_in(purchase, ['order', 'costs'], [0.25, 1.00]) # doctest: +SKIP - {'credit card': '5555-1234-1234-1234', - 'name': 'Alice', - 'order': {'costs': [0.25, 1.00], 'items': ['Apple', 'Orange']}} - """ - return update_in(d, keys, lambda x: value, value, factory) - - -def update_in(d, keys, func, default=None, factory=dict): - """ Update value in a (potentially) nested dictionary - - inputs: - d - dictionary on which to operate - keys - list or tuple giving the location of the value to be changed in d - func - function to operate on that value - - If keys == [k0,..,kX] and d[k0]..[kX] == v, update_in returns a copy of the - original dictionary with v replaced by func(v), but does not mutate the - original dictionary. - - If k0 is not a key in d, update_in creates nested dictionaries to the depth - specified by the keys, with the innermost value set to func(default). - - >>> inc = lambda x: x + 1 - >>> update_in({'a': 0}, ['a'], inc) - {'a': 1} - - >>> transaction = {'name': 'Alice', - ... 'purchase': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> update_in(transaction, ['purchase', 'costs'], sum) # doctest: +SKIP - {'credit card': '5555-1234-1234-1234', - 'name': 'Alice', - 'purchase': {'costs': 1.75, 'items': ['Apple', 'Orange']}} - - >>> # updating a value when k0 is not in d - >>> update_in({}, [1, 2, 3], str, default="bar") - {1: {2: {3: 'bar'}}} - >>> update_in({1: 'foo'}, [2, 3, 4], inc, 0) - {1: 'foo', 2: {3: {4: 1}}} - """ - ks = iter(keys) - k = next(ks) - - rv = inner = factory() - rv.update(d) - - for key in ks: - if k in d: - d = d[k] - dtemp = factory() - dtemp.update(d) - else: - d = dtemp = factory() - - inner[k] = inner = dtemp - k = key - - if k in d: - inner[k] = func(d[k]) - else: - inner[k] = func(default) - return rv - - -def get_in(keys, coll, default=None, no_default=False): - """ Returns coll[i0][i1]...[iX] where [i0, i1, ..., iX]==keys. - - If coll[i0][i1]...[iX] cannot be found, returns ``default``, unless - ``no_default`` is specified, then it raises KeyError or IndexError. - - ``get_in`` is a generalization of ``operator.getitem`` for nested data - structures such as dictionaries and lists. - - >>> transaction = {'name': 'Alice', - ... 'purchase': {'items': ['Apple', 'Orange'], - ... 'costs': [0.50, 1.25]}, - ... 'credit card': '5555-1234-1234-1234'} - >>> get_in(['purchase', 'items', 0], transaction) - 'Apple' - >>> get_in(['name'], transaction) - 'Alice' - >>> get_in(['purchase', 'total'], transaction) - >>> get_in(['purchase', 'items', 'apple'], transaction) - >>> get_in(['purchase', 'items', 10], transaction) - >>> get_in(['purchase', 'total'], transaction, 0) - 0 - >>> get_in(['y'], {}, no_default=True) - Traceback (most recent call last): - ... - KeyError: 'y' - - See Also: - itertoolz.get - operator.getitem - """ - try: - return reduce(operator.getitem, keys, coll) - except (KeyError, IndexError, TypeError): - if no_default: - raise - return default diff --git a/contrib/python/toolz/py2/toolz/functoolz.py b/contrib/python/toolz/py2/toolz/functoolz.py deleted file mode 100644 index 01d3857a199..00000000000 --- a/contrib/python/toolz/py2/toolz/functoolz.py +++ /dev/null @@ -1,1152 +0,0 @@ -from functools import reduce, partial -import inspect -import operator -from operator import attrgetter -from importlib import import_module -from textwrap import dedent -from types import MethodType - -from .compatibility import PY3, PY34, PYPY -from .utils import no_default - - -__all__ = ('identity', 'apply', 'thread_first', 'thread_last', 'memoize', - 'compose', 'compose_left', 'pipe', 'complement', 'juxt', 'do', - 'curry', 'flip', 'excepts') - - -def identity(x): - """ Identity function. Return x - - >>> identity(3) - 3 - """ - return x - - -def apply(*func_and_args, **kwargs): - """ Applies a function and returns the results - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> apply(double, 5) - 10 - - >>> tuple(map(apply, [double, inc, double], [10, 500, 8000])) - (20, 501, 16000) - """ - if not func_and_args: - raise TypeError('func argument is required') - func, args = func_and_args[0], func_and_args[1:] - return func(*args, **kwargs) - - -def thread_first(val, *forms): - """ Thread value through a sequence of functions/forms - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> thread_first(1, inc, double) - 4 - - If the function expects more than one input you can specify those inputs - in a tuple. The value is used as the first input. - - >>> def add(x, y): return x + y - >>> def pow(x, y): return x**y - >>> thread_first(1, (add, 4), (pow, 2)) # pow(add(1, 4), 2) - 25 - - So in general - thread_first(x, f, (g, y, z)) - expands to - g(f(x), y, z) - - See Also: - thread_last - """ - def evalform_front(val, form): - if callable(form): - return form(val) - if isinstance(form, tuple): - func, args = form[0], form[1:] - args = (val,) + args - return func(*args) - return reduce(evalform_front, forms, val) - - -def thread_last(val, *forms): - """ Thread value through a sequence of functions/forms - - >>> def double(x): return 2*x - >>> def inc(x): return x + 1 - >>> thread_last(1, inc, double) - 4 - - If the function expects more than one input you can specify those inputs - in a tuple. The value is used as the last input. - - >>> def add(x, y): return x + y - >>> def pow(x, y): return x**y - >>> thread_last(1, (add, 4), (pow, 2)) # pow(2, add(4, 1)) - 32 - - So in general - thread_last(x, f, (g, y, z)) - expands to - g(y, z, f(x)) - - >>> def iseven(x): - ... return x % 2 == 0 - >>> list(thread_last([1, 2, 3], (map, inc), (filter, iseven))) - [2, 4] - - See Also: - thread_first - """ - def evalform_back(val, form): - if callable(form): - return form(val) - if isinstance(form, tuple): - func, args = form[0], form[1:] - args = args + (val,) - return func(*args) - return reduce(evalform_back, forms, val) - - -def instanceproperty(fget=None, fset=None, fdel=None, doc=None, classval=None): - """ Like @property, but returns ``classval`` when used as a class attribute - - >>> class MyClass(object): - ... '''The class docstring''' - ... @instanceproperty(classval=__doc__) - ... def __doc__(self): - ... return 'An object docstring' - ... @instanceproperty - ... def val(self): - ... return 42 - ... - >>> MyClass.__doc__ - 'The class docstring' - >>> MyClass.val is None - True - >>> obj = MyClass() - >>> obj.__doc__ - 'An object docstring' - >>> obj.val - 42 - """ - if fget is None: - return partial(instanceproperty, fset=fset, fdel=fdel, doc=doc, - classval=classval) - return InstanceProperty(fget=fget, fset=fset, fdel=fdel, doc=doc, - classval=classval) - - -class InstanceProperty(property): - """ Like @property, but returns ``classval`` when used as a class attribute - - Should not be used directly. Use ``instanceproperty`` instead. - """ - def __init__(self, fget=None, fset=None, fdel=None, doc=None, - classval=None): - self.classval = classval - property.__init__(self, fget=fget, fset=fset, fdel=fdel, doc=doc) - - def __get__(self, obj, type=None): - if obj is None: - return self.classval - return property.__get__(self, obj, type) - - def __reduce__(self): - state = (self.fget, self.fset, self.fdel, self.__doc__, self.classval) - return InstanceProperty, state - - -class curry(object): - """ Curry a callable function - - Enables partial application of arguments through calling a function with an - incomplete set of arguments. - - >>> def mul(x, y): - ... return x * y - >>> mul = curry(mul) - - >>> double = mul(2) - >>> double(10) - 20 - - Also supports keyword arguments - - >>> @curry # Can use curry as a decorator - ... def f(x, y, a=10): - ... return a * (x + y) - - >>> add = f(a=1) - >>> add(2, 3) - 5 - - See Also: - toolz.curried - namespace of curried functions - https://toolz.readthedocs.io/en/latest/curry.html - """ - def __init__(self, *args, **kwargs): - if not args: - raise TypeError('__init__() takes at least 2 arguments (1 given)') - func, args = args[0], args[1:] - if not callable(func): - raise TypeError("Input must be callable") - - # curry- or functools.partial-like object? Unpack and merge arguments - if ( - hasattr(func, 'func') - and hasattr(func, 'args') - and hasattr(func, 'keywords') - and isinstance(func.args, tuple) - ): - _kwargs = {} - if func.keywords: - _kwargs.update(func.keywords) - _kwargs.update(kwargs) - kwargs = _kwargs - args = func.args + args - func = func.func - - if kwargs: - self._partial = partial(func, *args, **kwargs) - else: - self._partial = partial(func, *args) - - self.__doc__ = getattr(func, '__doc__', None) - self.__name__ = getattr(func, '__name__', '<curry>') - self.__module__ = getattr(func, '__module__', None) - self.__qualname__ = getattr(func, '__qualname__', None) - self._sigspec = None - self._has_unknown_args = None - - @instanceproperty - def func(self): - return self._partial.func - - if PY3: # pragma: py2 no cover - @instanceproperty - def __signature__(self): - sig = inspect.signature(self.func) - args = self.args or () - keywords = self.keywords or {} - if is_partial_args(self.func, args, keywords, sig) is False: - raise TypeError('curry object has incorrect arguments') - - params = list(sig.parameters.values()) - skip = 0 - for param in params[:len(args)]: - if param.kind == param.VAR_POSITIONAL: - break - skip += 1 - - kwonly = False - newparams = [] - for param in params[skip:]: - kind = param.kind - default = param.default - if kind == param.VAR_KEYWORD: - pass - elif kind == param.VAR_POSITIONAL: - if kwonly: - continue - elif param.name in keywords: - default = keywords[param.name] - kind = param.KEYWORD_ONLY - kwonly = True - else: - if kwonly: - kind = param.KEYWORD_ONLY - if default is param.empty: - default = no_default - newparams.append(param.replace(default=default, kind=kind)) - - return sig.replace(parameters=newparams) - - @instanceproperty - def args(self): - return self._partial.args - - @instanceproperty - def keywords(self): - return self._partial.keywords - - @instanceproperty - def func_name(self): - return self.__name__ - - def __str__(self): - return str(self.func) - - def __repr__(self): - return repr(self.func) - - def __hash__(self): - return hash((self.func, self.args, - frozenset(self.keywords.items()) if self.keywords - else None)) - - def __eq__(self, other): - return (isinstance(other, curry) and self.func == other.func and - self.args == other.args and self.keywords == other.keywords) - - def __ne__(self, other): - return not self.__eq__(other) - - def __call__(self, *args, **kwargs): - try: - return self._partial(*args, **kwargs) - except TypeError as exc: - if self._should_curry(args, kwargs, exc): - return self.bind(*args, **kwargs) - raise - - def _should_curry(self, args, kwargs, exc=None): - func = self.func - args = self.args + args - if self.keywords: - kwargs = dict(self.keywords, **kwargs) - if self._sigspec is None: - sigspec = self._sigspec = _sigs.signature_or_spec(func) - self._has_unknown_args = has_varargs(func, sigspec) is not False - else: - sigspec = self._sigspec - - if is_partial_args(func, args, kwargs, sigspec) is False: - # Nothing can make the call valid - return False - elif self._has_unknown_args: - # The call may be valid and raised a TypeError, but we curry - # anyway because the function may have `*args`. This is useful - # for decorators with signature `func(*args, **kwargs)`. - return True - elif not is_valid_args(func, args, kwargs, sigspec): - # Adding more arguments may make the call valid - return True - else: - # There was a genuine TypeError - return False - - def bind(self, *args, **kwargs): - return type(self)(self, *args, **kwargs) - - def call(self, *args, **kwargs): - return self._partial(*args, **kwargs) - - def __get__(self, instance, owner): - if instance is None: - return self - return curry(self, instance) - - def __reduce__(self): - func = self.func - modname = getattr(func, '__module__', None) - qualname = getattr(func, '__qualname__', None) - if qualname is None: # pragma: py3 no cover - qualname = getattr(func, '__name__', None) - is_decorated = None - if modname and qualname: - attrs = [] - obj = import_module(modname) - for attr in qualname.split('.'): - if isinstance(obj, curry): # pragma: py2 no cover - attrs.append('func') - obj = obj.func - obj = getattr(obj, attr, None) - if obj is None: - break - attrs.append(attr) - if isinstance(obj, curry) and obj.func is func: - is_decorated = obj is self - qualname = '.'.join(attrs) - func = '%s:%s' % (modname, qualname) - - # functools.partial objects can't be pickled - userdict = tuple((k, v) for k, v in self.__dict__.items() - if k not in ('_partial', '_sigspec')) - state = (type(self), func, self.args, self.keywords, userdict, - is_decorated) - return _restore_curry, state - - -def _restore_curry(cls, func, args, kwargs, userdict, is_decorated): - if isinstance(func, str): - modname, qualname = func.rsplit(':', 1) - obj = import_module(modname) - for attr in qualname.split('.'): - obj = getattr(obj, attr) - if is_decorated: - return obj - func = obj.func - obj = cls(func, *args, **(kwargs or {})) - obj.__dict__.update(userdict) - return obj - - -@curry -def memoize(func, cache=None, key=None): - """ Cache a function's result for speedy future evaluation - - Considerations: - Trades memory for speed. - Only use on pure functions. - - >>> def add(x, y): return x + y - >>> add = memoize(add) - - Or use as a decorator - - >>> @memoize - ... def add(x, y): - ... return x + y - - Use the ``cache`` keyword to provide a dict-like object as an initial cache - - >>> @memoize(cache={(1, 2): 3}) - ... def add(x, y): - ... return x + y - - Note that the above works as a decorator because ``memoize`` is curried. - - It is also possible to provide a ``key(args, kwargs)`` function that - calculates keys used for the cache, which receives an ``args`` tuple and - ``kwargs`` dict as input, and must return a hashable value. However, - the default key function should be sufficient most of the time. - - >>> # Use key function that ignores extraneous keyword arguments - >>> @memoize(key=lambda args, kwargs: args) - ... def add(x, y, verbose=False): - ... if verbose: - ... print('Calculating %s + %s' % (x, y)) - ... return x + y - """ - if cache is None: - cache = {} - - try: - may_have_kwargs = has_keywords(func) is not False - # Is unary function (single arg, no variadic argument or keywords)? - is_unary = is_arity(1, func) - except TypeError: # pragma: no cover - may_have_kwargs = True - is_unary = False - - if key is None: - if is_unary: - def key(args, kwargs): - return args[0] - elif may_have_kwargs: - def key(args, kwargs): - return ( - args or None, - frozenset(kwargs.items()) if kwargs else None, - ) - else: - def key(args, kwargs): - return args - - def memof(*args, **kwargs): - k = key(args, kwargs) - try: - return cache[k] - except TypeError: - raise TypeError("Arguments to memoized function must be hashable") - except KeyError: - cache[k] = result = func(*args, **kwargs) - return result - - try: - memof.__name__ = func.__name__ - except AttributeError: - pass - memof.__doc__ = func.__doc__ - memof.__wrapped__ = func - return memof - - -class Compose(object): - """ A composition of functions - - See Also: - compose - """ - __slots__ = 'first', 'funcs' - - def __init__(self, funcs): - funcs = tuple(reversed(funcs)) - self.first = funcs[0] - self.funcs = funcs[1:] - - def __call__(self, *args, **kwargs): - ret = self.first(*args, **kwargs) - for f in self.funcs: - ret = f(ret) - return ret - - def __getstate__(self): - return self.first, self.funcs - - def __setstate__(self, state): - self.first, self.funcs = state - - @instanceproperty(classval=__doc__) - def __doc__(self): - def composed_doc(*fs): - """Generate a docstring for the composition of fs. - """ - if not fs: - # Argument name for the docstring. - return '*args, **kwargs' - - return '{f}({g})'.format(f=fs[0].__name__, g=composed_doc(*fs[1:])) - - try: - return ( - 'lambda *args, **kwargs: ' + - composed_doc(*reversed((self.first,) + self.funcs)) - ) - except AttributeError: - # One of our callables does not have a `__name__`, whatever. - return 'A composition of functions' - - @property - def __name__(self): - try: - return '_of_'.join( - (f.__name__ for f in reversed((self.first,) + self.funcs)) - ) - except AttributeError: - return type(self).__name__ - - def __repr__(self): - return '{.__class__.__name__}{!r}'.format( - self, tuple(reversed((self.first, ) + self.funcs))) - - def __eq__(self, other): - if isinstance(other, Compose): - return other.first == self.first and other.funcs == self.funcs - return NotImplemented - - def __ne__(self, other): - equality = self.__eq__(other) - return NotImplemented if equality is NotImplemented else not equality - - def __hash__(self): - return hash(self.first) ^ hash(self.funcs) - - # Mimic the descriptor behavior of python functions. - # i.e. let Compose be called as a method when bound to a class. - if PY3: # pragma: py2 no cover - # adapted from - # docs.python.org/3/howto/descriptor.html#functions-and-methods - def __get__(self, obj, objtype=None): - return self if obj is None else MethodType(self, obj) - else: # pragma: py3 no cover - # adapted from - # docs.python.org/2/howto/descriptor.html#functions-and-methods - def __get__(self, obj, objtype=None): - return self if obj is None else MethodType(self, obj, objtype) - - # introspection with Signature is only possible from py3.3+ - if PY3: # pragma: py2 no cover - @instanceproperty - def __signature__(self): - base = inspect.signature(self.first) - last = inspect.signature(self.funcs[-1]) - return base.replace(return_annotation=last.return_annotation) - - __wrapped__ = instanceproperty(attrgetter('first')) - - -def compose(*funcs): - """ Compose functions to operate in series. - - Returns a function that applies other functions in sequence. - - Functions are applied from right to left so that - ``compose(f, g, h)(x, y)`` is the same as ``f(g(h(x, y)))``. - - If no arguments are provided, the identity function (f(x) = x) is returned. - - >>> inc = lambda i: i + 1 - >>> compose(str, inc)(3) - '4' - - See Also: - compose_left - pipe - """ - if not funcs: - return identity - if len(funcs) == 1: - return funcs[0] - else: - return Compose(funcs) - - -def compose_left(*funcs): - """ Compose functions to operate in series. - - Returns a function that applies other functions in sequence. - - Functions are applied from left to right so that - ``compose_left(f, g, h)(x, y)`` is the same as ``h(g(f(x, y)))``. - - If no arguments are provided, the identity function (f(x) = x) is returned. - - >>> inc = lambda i: i + 1 - >>> compose_left(inc, str)(3) - '4' - - See Also: - compose - pipe - """ - return compose(*reversed(funcs)) - - -def pipe(data, *funcs): - """ Pipe a value through a sequence of functions - - I.e. ``pipe(data, f, g, h)`` is equivalent to ``h(g(f(data)))`` - - We think of the value as progressing through a pipe of several - transformations, much like pipes in UNIX - - ``$ cat data | f | g | h`` - - >>> double = lambda i: 2 * i - >>> pipe(3, double, str) - '6' - - See Also: - compose - compose_left - thread_first - thread_last - """ - for func in funcs: - data = func(data) - return data - - -def complement(func): - """ Convert a predicate function to its logical complement. - - In other words, return a function that, for inputs that normally - yield True, yields False, and vice-versa. - - >>> def iseven(n): return n % 2 == 0 - >>> isodd = complement(iseven) - >>> iseven(2) - True - >>> isodd(2) - False - """ - return compose(operator.not_, func) - - -class juxt(object): - """ Creates a function that calls several functions with the same arguments - - Takes several functions and returns a function that applies its arguments - to each of those functions then returns a tuple of the results. - - Name comes from juxtaposition: the fact of two things being seen or placed - close together with contrasting effect. - - >>> inc = lambda x: x + 1 - >>> double = lambda x: x * 2 - >>> juxt(inc, double)(10) - (11, 20) - >>> juxt([inc, double])(10) - (11, 20) - """ - __slots__ = ['funcs'] - - def __init__(self, *funcs): - if len(funcs) == 1 and not callable(funcs[0]): - funcs = funcs[0] - self.funcs = tuple(funcs) - - def __call__(self, *args, **kwargs): - return tuple(func(*args, **kwargs) for func in self.funcs) - - def __getstate__(self): - return self.funcs - - def __setstate__(self, state): - self.funcs = state - - -def do(func, x): - """ Runs ``func`` on ``x``, returns ``x`` - - Because the results of ``func`` are not returned, only the side - effects of ``func`` are relevant. - - Logging functions can be made by composing ``do`` with a storage function - like ``list.append`` or ``file.write`` - - >>> from toolz import compose - >>> from toolz.curried import do - - >>> log = [] - >>> inc = lambda x: x + 1 - >>> inc = compose(inc, do(log.append)) - >>> inc(1) - 2 - >>> inc(11) - 12 - >>> log - [1, 11] - """ - func(x) - return x - - -@curry -def flip(func, a, b): - """ Call the function call with the arguments flipped - - This function is curried. - - >>> def div(a, b): - ... return a // b - ... - >>> flip(div, 2, 6) - 3 - >>> div_by_two = flip(div, 2) - >>> div_by_two(4) - 2 - - This is particularly useful for built in functions and functions defined - in C extensions that accept positional only arguments. For example: - isinstance, issubclass. - - >>> data = [1, 'a', 'b', 2, 1.5, object(), 3] - >>> only_ints = list(filter(flip(isinstance, int), data)) - >>> only_ints - [1, 2, 3] - """ - return func(b, a) - - -def return_none(exc): - """ Returns None. - """ - return None - - -class excepts(object): - """A wrapper around a function to catch exceptions and - dispatch to a handler. - - This is like a functional try/except block, in the same way that - ifexprs are functional if/else blocks. - - Examples - -------- - >>> excepting = excepts( - ... ValueError, - ... lambda a: [1, 2].index(a), - ... lambda _: -1, - ... ) - >>> excepting(1) - 0 - >>> excepting(3) - -1 - - Multiple exceptions and default except clause. - >>> excepting = excepts((IndexError, KeyError), lambda a: a[0]) - >>> excepting([]) - >>> excepting([1]) - 1 - >>> excepting({}) - >>> excepting({0: 1}) - 1 - """ - def __init__(self, exc, func, handler=return_none): - self.exc = exc - self.func = func - self.handler = handler - - def __call__(self, *args, **kwargs): - try: - return self.func(*args, **kwargs) - except self.exc as e: - return self.handler(e) - - @instanceproperty(classval=__doc__) - def __doc__(self): - exc = self.exc - try: - if isinstance(exc, tuple): - exc_name = '(%s)' % ', '.join( - map(attrgetter('__name__'), exc), - ) - else: - exc_name = exc.__name__ - - return dedent( - """\ - A wrapper around {inst.func.__name__!r} that will except: - {exc} - and handle any exceptions with {inst.handler.__name__!r}. - - Docs for {inst.func.__name__!r}: - {inst.func.__doc__} - - Docs for {inst.handler.__name__!r}: - {inst.handler.__doc__} - """ - ).format( - inst=self, - exc=exc_name, - ) - except AttributeError: - return type(self).__doc__ - - @property - def __name__(self): - exc = self.exc - try: - if isinstance(exc, tuple): - exc_name = '_or_'.join(map(attrgetter('__name__'), exc)) - else: - exc_name = exc.__name__ - return '%s_excepting_%s' % (self.func.__name__, exc_name) - except AttributeError: - return 'excepting' - - -if PY3: # pragma: py2 no cover - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - if sigspec is None: - try: - sigspec = inspect.signature(func) - except (ValueError, TypeError) as e: - sigspec = e - if isinstance(sigspec, ValueError): - return None, builtin_func(*builtin_args) - elif not isinstance(sigspec, inspect.Signature): - if ( - func in _sigs.signatures - and (( - hasattr(func, '__signature__') - and hasattr(func.__signature__, '__get__') - )) - ): # pragma: no cover (not covered in Python 3.4) - val = builtin_func(*builtin_args) - return None, val - return None, False - return sigspec, None - -else: # pragma: py3 no cover - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - if sigspec is None: - try: - sigspec = inspect.getargspec(func) - except TypeError as e: - sigspec = e - if isinstance(sigspec, TypeError): - if not callable(func): - return None, False - return None, builtin_func(*builtin_args) - return sigspec, None - - -if PY34 or PYPY: # pragma: no cover - _check_sigspec_orig = _check_sigspec - - def _check_sigspec(sigspec, func, builtin_func, *builtin_args): - # Python 3.4 and PyPy may lie, so use our registry for builtins instead - if func in _sigs.signatures: - val = builtin_func(*builtin_args) - return None, val - return _check_sigspec_orig(sigspec, func, builtin_func, *builtin_args) - -_check_sigspec.__doc__ = """ \ -Private function to aid in introspection compatibly across Python versions. - -If a callable doesn't have a signature (Python 3) or an argspec (Python 2), -the signature registry in toolz._signatures is used. -""" - -if PY3: # pragma: py2 no cover - def num_required_args(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._num_required_args, - func) - if sigspec is None: - return rv - return sum(1 for p in sigspec.parameters.values() - if p.default is p.empty - and p.kind in (p.POSITIONAL_OR_KEYWORD, p.POSITIONAL_ONLY)) - - def has_varargs(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_varargs, func) - if sigspec is None: - return rv - return any(p.kind == p.VAR_POSITIONAL - for p in sigspec.parameters.values()) - - def has_keywords(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_keywords, func) - if sigspec is None: - return rv - return any(p.default is not p.empty - or p.kind in (p.KEYWORD_ONLY, p.VAR_KEYWORD) - for p in sigspec.parameters.values()) - - def is_valid_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_valid_args, - func, args, kwargs) - if sigspec is None: - return rv - try: - sigspec.bind(*args, **kwargs) - except TypeError: - return False - return True - - def is_partial_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_partial_args, - func, args, kwargs) - if sigspec is None: - return rv - try: - sigspec.bind_partial(*args, **kwargs) - except TypeError: - return False - return True - -else: # pragma: py3 no cover - def num_required_args(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._num_required_args, - func) - if sigspec is None: - return rv - num_defaults = len(sigspec.defaults) if sigspec.defaults else 0 - return len(sigspec.args) - num_defaults - - def has_varargs(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_varargs, func) - if sigspec is None: - return rv - return sigspec.varargs is not None - - def has_keywords(func, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._has_keywords, func) - if sigspec is None: - return rv - return sigspec.defaults is not None or sigspec.keywords is not None - - def is_valid_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_valid_args, - func, args, kwargs) - if sigspec is None: - return rv - spec = sigspec - defaults = spec.defaults or () - num_pos = len(spec.args) - len(defaults) - missing_pos = spec.args[len(args):num_pos] - if any(arg not in kwargs for arg in missing_pos): - return False - - if spec.varargs is None: - num_extra_pos = max(0, len(args) - num_pos) - else: - num_extra_pos = 0 - - kwargs = dict(kwargs) - - # Add missing keyword arguments (unless already included in `args`) - missing_kwargs = spec.args[num_pos + num_extra_pos:] - kwargs.update(zip(missing_kwargs, defaults[num_extra_pos:])) - - # Convert call to use positional arguments - args = args + tuple(kwargs.pop(key) for key in spec.args[len(args):]) - - if ( - not spec.keywords and kwargs - or not spec.varargs and len(args) > len(spec.args) - or set(spec.args[:len(args)]) & set(kwargs) - ): - return False - else: - return True - - def is_partial_args(func, args, kwargs, sigspec=None): - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_partial_args, - func, args, kwargs) - if sigspec is None: - return rv - spec = sigspec - defaults = spec.defaults or () - num_pos = len(spec.args) - len(defaults) - if spec.varargs is None: - num_extra_pos = max(0, len(args) - num_pos) - else: - num_extra_pos = 0 - - kwargs = dict(kwargs) - - # Add missing keyword arguments (unless already included in `args`) - missing_kwargs = spec.args[num_pos + num_extra_pos:] - kwargs.update(zip(missing_kwargs, defaults[num_extra_pos:])) - - # Add missing position arguments as keywords (may already be in kwargs) - missing_args = spec.args[len(args):num_pos + num_extra_pos] - kwargs.update((x, None) for x in missing_args) - - # Convert call to use positional arguments - args = args + tuple(kwargs.pop(key) for key in spec.args[len(args):]) - - if ( - not spec.keywords and kwargs - or not spec.varargs and len(args) > len(spec.args) - or set(spec.args[:len(args)]) & set(kwargs) - ): - return False - else: - return True - - -def is_arity(n, func, sigspec=None): - """ Does a function have only n positional arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x): - ... return x - >>> is_arity(1, f) - True - >>> def g(x, y=1): - ... return x + y - >>> is_arity(1, g) - False - """ - sigspec, rv = _check_sigspec(sigspec, func, _sigs._is_arity, n, func) - if sigspec is None: - return rv - num = num_required_args(func, sigspec) - if num is not None: - num = num == n - if not num: - return False - varargs = has_varargs(func, sigspec) - if varargs: - return False - keywords = has_keywords(func, sigspec) - if keywords: - return False - if num is None or varargs is None or keywords is None: # pragma: no cover - return None - return True - - -num_required_args.__doc__ = """ \ -Number of required positional arguments - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x, y, z=3): - ... return x + y + z - >>> num_required_args(f) - 2 - >>> def g(*args, **kwargs): - ... pass - >>> num_required_args(g) - 0 - """ - -has_varargs.__doc__ = """ \ -Does a function have variadic positional arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(*args): - ... return args - >>> has_varargs(f) - True - >>> def g(**kwargs): - ... return kwargs - >>> has_varargs(g) - False - """ - -has_keywords.__doc__ = """ \ -Does a function have keyword arguments? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def f(x, y=0): - ... return x + y - - >>> has_keywords(f) - True - """ - -is_valid_args.__doc__ = """ \ -Is ``func(*args, **kwargs)`` a valid function call? - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def add(x, y): - ... return x + y - - >>> is_valid_args(add, (1,), {}) - False - >>> is_valid_args(add, (1, 2), {}) - True - >>> is_valid_args(map, (), {}) - False - - **Implementation notes** - Python 2 relies on ``inspect.getargspec``, which only works for - user-defined functions. Python 3 uses ``inspect.signature``, which - works for many more types of callables. - - Many builtins in the standard library are also supported. - """ - -is_partial_args.__doc__ = """ \ -Can partial(func, *args, **kwargs)(*args2, **kwargs2) be a valid call? - - Returns True *only* if the call is valid or if it is possible for the - call to become valid by adding more positional or keyword arguments. - - This function relies on introspection and does not call the function. - Returns None if validity can't be determined. - - >>> def add(x, y): - ... return x + y - - >>> is_partial_args(add, (1,), {}) - True - >>> is_partial_args(add, (1, 2), {}) - True - >>> is_partial_args(add, (1, 2, 3), {}) - False - >>> is_partial_args(map, (), {}) - True - - **Implementation notes** - Python 2 relies on ``inspect.getargspec``, which only works for - user-defined functions. Python 3 uses ``inspect.signature``, which - works for many more types of callables. - - Many builtins in the standard library are also supported. - """ - -from . import _signatures as _sigs diff --git a/contrib/python/toolz/py2/toolz/itertoolz.py b/contrib/python/toolz/py2/toolz/itertoolz.py deleted file mode 100644 index e71f1eeef0a..00000000000 --- a/contrib/python/toolz/py2/toolz/itertoolz.py +++ /dev/null @@ -1,1056 +0,0 @@ -import itertools -import heapq -import collections -import operator -from functools import partial -from random import Random -from toolz.compatibility import (map, filterfalse, zip, zip_longest, iteritems, - filter, Sequence) -from toolz.utils import no_default - - -__all__ = ('remove', 'accumulate', 'groupby', 'merge_sorted', 'interleave', - 'unique', 'isiterable', 'isdistinct', 'take', 'drop', 'take_nth', - 'first', 'second', 'nth', 'last', 'get', 'concat', 'concatv', - 'mapcat', 'cons', 'interpose', 'frequencies', 'reduceby', 'iterate', - 'sliding_window', 'partition', 'partition_all', 'count', 'pluck', - 'join', 'tail', 'diff', 'topk', 'peek', 'peekn', 'random_sample') - - -def remove(predicate, seq): - """ Return those items of sequence for which predicate(item) is False - - >>> def iseven(x): - ... return x % 2 == 0 - >>> list(remove(iseven, [1, 2, 3, 4])) - [1, 3] - """ - return filterfalse(predicate, seq) - - -def accumulate(binop, seq, initial=no_default): - """ Repeatedly apply binary function to a sequence, accumulating results - - >>> from operator import add, mul - >>> list(accumulate(add, [1, 2, 3, 4, 5])) - [1, 3, 6, 10, 15] - >>> list(accumulate(mul, [1, 2, 3, 4, 5])) - [1, 2, 6, 24, 120] - - Accumulate is similar to ``reduce`` and is good for making functions like - cumulative sum: - - >>> from functools import partial, reduce - >>> sum = partial(reduce, add) - >>> cumsum = partial(accumulate, add) - - Accumulate also takes an optional argument that will be used as the first - value. This is similar to reduce. - - >>> list(accumulate(add, [1, 2, 3], -1)) - [-1, 0, 2, 5] - >>> list(accumulate(add, [], 1)) - [1] - - See Also: - itertools.accumulate : In standard itertools for Python 3.2+ - """ - seq = iter(seq) - if initial == no_default: - try: - result = next(seq) - except StopIteration: - return - else: - result = initial - yield result - for elem in seq: - result = binop(result, elem) - yield result - - -def groupby(key, seq): - """ Group a collection by a key function - - >>> names = ['Alice', 'Bob', 'Charlie', 'Dan', 'Edith', 'Frank'] - >>> groupby(len, names) # doctest: +SKIP - {3: ['Bob', 'Dan'], 5: ['Alice', 'Edith', 'Frank'], 7: ['Charlie']} - - >>> iseven = lambda x: x % 2 == 0 - >>> groupby(iseven, [1, 2, 3, 4, 5, 6, 7, 8]) # doctest: +SKIP - {False: [1, 3, 5, 7], True: [2, 4, 6, 8]} - - Non-callable keys imply grouping on a member. - - >>> groupby('gender', [{'name': 'Alice', 'gender': 'F'}, - ... {'name': 'Bob', 'gender': 'M'}, - ... {'name': 'Charlie', 'gender': 'M'}]) # doctest:+SKIP - {'F': [{'gender': 'F', 'name': 'Alice'}], - 'M': [{'gender': 'M', 'name': 'Bob'}, - {'gender': 'M', 'name': 'Charlie'}]} - - Not to be confused with ``itertools.groupby`` - - See Also: - countby - """ - if not callable(key): - key = getter(key) - d = collections.defaultdict(lambda: [].append) - for item in seq: - d[key(item)](item) - rv = {} - for k, v in iteritems(d): - rv[k] = v.__self__ - return rv - - -def merge_sorted(*seqs, **kwargs): - """ Merge and sort a collection of sorted collections - - This works lazily and only keeps one value from each iterable in memory. - - >>> list(merge_sorted([1, 3, 5], [2, 4, 6])) - [1, 2, 3, 4, 5, 6] - - >>> ''.join(merge_sorted('abc', 'abc', 'abc')) - 'aaabbbccc' - - The "key" function used to sort the input may be passed as a keyword. - - >>> list(merge_sorted([2, 3], [1, 3], key=lambda x: x // 3)) - [2, 1, 3, 3] - """ - if len(seqs) == 0: - return iter([]) - elif len(seqs) == 1: - return iter(seqs[0]) - - key = kwargs.get('key', None) - if key is None: - return _merge_sorted_binary(seqs) - else: - return _merge_sorted_binary_key(seqs, key) - - -def _merge_sorted_binary(seqs): - mid = len(seqs) // 2 - L1 = seqs[:mid] - if len(L1) == 1: - seq1 = iter(L1[0]) - else: - seq1 = _merge_sorted_binary(L1) - L2 = seqs[mid:] - if len(L2) == 1: - seq2 = iter(L2[0]) - else: - seq2 = _merge_sorted_binary(L2) - - try: - val2 = next(seq2) - except StopIteration: - for val1 in seq1: - yield val1 - return - - for val1 in seq1: - if val2 < val1: - yield val2 - for val2 in seq2: - if val2 < val1: - yield val2 - else: - yield val1 - break - else: - break - else: - yield val1 - else: - yield val2 - for val2 in seq2: - yield val2 - return - yield val1 - for val1 in seq1: - yield val1 - - -def _merge_sorted_binary_key(seqs, key): - mid = len(seqs) // 2 - L1 = seqs[:mid] - if len(L1) == 1: - seq1 = iter(L1[0]) - else: - seq1 = _merge_sorted_binary_key(L1, key) - L2 = seqs[mid:] - if len(L2) == 1: - seq2 = iter(L2[0]) - else: - seq2 = _merge_sorted_binary_key(L2, key) - - try: - val2 = next(seq2) - except StopIteration: - for val1 in seq1: - yield val1 - return - key2 = key(val2) - - for val1 in seq1: - key1 = key(val1) - if key2 < key1: - yield val2 - for val2 in seq2: - key2 = key(val2) - if key2 < key1: - yield val2 - else: - yield val1 - break - else: - break - else: - yield val1 - else: - yield val2 - for val2 in seq2: - yield val2 - return - yield val1 - for val1 in seq1: - yield val1 - - -def interleave(seqs): - """ Interleave a sequence of sequences - - >>> list(interleave([[1, 2], [3, 4]])) - [1, 3, 2, 4] - - >>> ''.join(interleave(('ABC', 'XY'))) - 'AXBYC' - - Both the individual sequences and the sequence of sequences may be infinite - - Returns a lazy iterator - """ - iters = itertools.cycle(map(iter, seqs)) - while True: - try: - for itr in iters: - yield next(itr) - return - except StopIteration: - predicate = partial(operator.is_not, itr) - iters = itertools.cycle(itertools.takewhile(predicate, iters)) - - -def unique(seq, key=None): - """ Return only unique elements of a sequence - - >>> tuple(unique((1, 2, 3))) - (1, 2, 3) - >>> tuple(unique((1, 2, 1, 3))) - (1, 2, 3) - - Uniqueness can be defined by key keyword - - >>> tuple(unique(['cat', 'mouse', 'dog', 'hen'], key=len)) - ('cat', 'mouse') - """ - seen = set() - seen_add = seen.add - if key is None: - for item in seq: - if item not in seen: - seen_add(item) - yield item - else: # calculate key - for item in seq: - val = key(item) - if val not in seen: - seen_add(val) - yield item - - -def isiterable(x): - """ Is x iterable? - - >>> isiterable([1, 2, 3]) - True - >>> isiterable('abc') - True - >>> isiterable(5) - False - """ - try: - iter(x) - return True - except TypeError: - return False - - -def isdistinct(seq): - """ All values in sequence are distinct - - >>> isdistinct([1, 2, 3]) - True - >>> isdistinct([1, 2, 1]) - False - - >>> isdistinct("Hello") - False - >>> isdistinct("World") - True - """ - if iter(seq) is seq: - seen = set() - seen_add = seen.add - for item in seq: - if item in seen: - return False - seen_add(item) - return True - else: - return len(seq) == len(set(seq)) - - -def take(n, seq): - """ The first n elements of a sequence - - >>> list(take(2, [10, 20, 30, 40, 50])) - [10, 20] - - See Also: - drop - tail - """ - return itertools.islice(seq, n) - - -def tail(n, seq): - """ The last n elements of a sequence - - >>> tail(2, [10, 20, 30, 40, 50]) - [40, 50] - - See Also: - drop - take - """ - try: - return seq[-n:] - except (TypeError, KeyError): - return tuple(collections.deque(seq, n)) - - -def drop(n, seq): - """ The sequence following the first n elements - - >>> list(drop(2, [10, 20, 30, 40, 50])) - [30, 40, 50] - - See Also: - take - tail - """ - return itertools.islice(seq, n, None) - - -def take_nth(n, seq): - """ Every nth item in seq - - >>> list(take_nth(2, [10, 20, 30, 40, 50])) - [10, 30, 50] - """ - return itertools.islice(seq, 0, None, n) - - -def first(seq): - """ The first element in a sequence - - >>> first('ABC') - 'A' - """ - return next(iter(seq)) - - -def second(seq): - """ The second element in a sequence - - >>> second('ABC') - 'B' - """ - seq = iter(seq) - next(seq) - return next(seq) - - -def nth(n, seq): - """ The nth element in a sequence - - >>> nth(1, 'ABC') - 'B' - """ - if isinstance(seq, (tuple, list, Sequence)): - return seq[n] - else: - return next(itertools.islice(seq, n, None)) - - -def last(seq): - """ The last element in a sequence - - >>> last('ABC') - 'C' - """ - return tail(1, seq)[0] - - -rest = partial(drop, 1) - - -def _get(ind, seq, default): - try: - return seq[ind] - except (KeyError, IndexError): - return default - - -def get(ind, seq, default=no_default): - """ Get element in a sequence or dict - - Provides standard indexing - - >>> get(1, 'ABC') # Same as 'ABC'[1] - 'B' - - Pass a list to get multiple values - - >>> get([1, 2], 'ABC') # ('ABC'[1], 'ABC'[2]) - ('B', 'C') - - Works on any value that supports indexing/getitem - For example here we see that it works with dictionaries - - >>> phonebook = {'Alice': '555-1234', - ... 'Bob': '555-5678', - ... 'Charlie':'555-9999'} - >>> get('Alice', phonebook) - '555-1234' - - >>> get(['Alice', 'Bob'], phonebook) - ('555-1234', '555-5678') - - Provide a default for missing values - - >>> get(['Alice', 'Dennis'], phonebook, None) - ('555-1234', None) - - See Also: - pluck - """ - try: - return seq[ind] - except TypeError: # `ind` may be a list - if isinstance(ind, list): - if default == no_default: - if len(ind) > 1: - return operator.itemgetter(*ind)(seq) - elif ind: - return seq[ind[0]], - else: - return () - else: - return tuple(_get(i, seq, default) for i in ind) - elif default != no_default: - return default - else: - raise - except (KeyError, IndexError): # we know `ind` is not a list - if default == no_default: - raise - else: - return default - - -def concat(seqs): - """ Concatenate zero or more iterables, any of which may be infinite. - - An infinite sequence will prevent the rest of the arguments from - being included. - - We use chain.from_iterable rather than ``chain(*seqs)`` so that seqs - can be a generator. - - >>> list(concat([[], [1], [2, 3]])) - [1, 2, 3] - - See also: - itertools.chain.from_iterable equivalent - """ - return itertools.chain.from_iterable(seqs) - - -def concatv(*seqs): - """ Variadic version of concat - - >>> list(concatv([], ["a"], ["b", "c"])) - ['a', 'b', 'c'] - - See also: - itertools.chain - """ - return concat(seqs) - - -def mapcat(func, seqs): - """ Apply func to each sequence in seqs, concatenating results. - - >>> list(mapcat(lambda s: [c.upper() for c in s], - ... [["a", "b"], ["c", "d", "e"]])) - ['A', 'B', 'C', 'D', 'E'] - """ - return concat(map(func, seqs)) - - -def cons(el, seq): - """ Add el to beginning of (possibly infinite) sequence seq. - - >>> list(cons(1, [2, 3])) - [1, 2, 3] - """ - return itertools.chain([el], seq) - - -def interpose(el, seq): - """ Introduce element between each pair of elements in seq - - >>> list(interpose("a", [1, 2, 3])) - [1, 'a', 2, 'a', 3] - """ - inposed = concat(zip(itertools.repeat(el), seq)) - next(inposed) - return inposed - - -def frequencies(seq): - """ Find number of occurrences of each value in seq - - >>> frequencies(['cat', 'cat', 'ox', 'pig', 'pig', 'cat']) #doctest: +SKIP - {'cat': 3, 'ox': 1, 'pig': 2} - - See Also: - countby - groupby - """ - d = collections.defaultdict(int) - for item in seq: - d[item] += 1 - return dict(d) - - -def reduceby(key, binop, seq, init=no_default): - """ Perform a simultaneous groupby and reduction - - The computation: - - >>> result = reduceby(key, binop, seq, init) # doctest: +SKIP - - is equivalent to the following: - - >>> def reduction(group): # doctest: +SKIP - ... return reduce(binop, group, init) # doctest: +SKIP - - >>> groups = groupby(key, seq) # doctest: +SKIP - >>> result = valmap(reduction, groups) # doctest: +SKIP - - But the former does not build the intermediate groups, allowing it to - operate in much less space. This makes it suitable for larger datasets - that do not fit comfortably in memory - - The ``init`` keyword argument is the default initialization of the - reduction. This can be either a constant value like ``0`` or a callable - like ``lambda : 0`` as might be used in ``defaultdict``. - - Simple Examples - --------------- - - >>> from operator import add, mul - >>> iseven = lambda x: x % 2 == 0 - - >>> data = [1, 2, 3, 4, 5] - - >>> reduceby(iseven, add, data) # doctest: +SKIP - {False: 9, True: 6} - - >>> reduceby(iseven, mul, data) # doctest: +SKIP - {False: 15, True: 8} - - Complex Example - --------------- - - >>> projects = [{'name': 'build roads', 'state': 'CA', 'cost': 1000000}, - ... {'name': 'fight crime', 'state': 'IL', 'cost': 100000}, - ... {'name': 'help farmers', 'state': 'IL', 'cost': 2000000}, - ... {'name': 'help farmers', 'state': 'CA', 'cost': 200000}] - - >>> reduceby('state', # doctest: +SKIP - ... lambda acc, x: acc + x['cost'], - ... projects, 0) - {'CA': 1200000, 'IL': 2100000} - - Example Using ``init`` - ---------------------- - - >>> def set_add(s, i): - ... s.add(i) - ... return s - - >>> reduceby(iseven, set_add, [1, 2, 3, 4, 1, 2, 3], set) # doctest: +SKIP - {True: set([2, 4]), - False: set([1, 3])} - """ - is_no_default = init == no_default - if not is_no_default and not callable(init): - _init = init - init = lambda: _init - if not callable(key): - key = getter(key) - d = {} - for item in seq: - k = key(item) - if k not in d: - if is_no_default: - d[k] = item - continue - else: - d[k] = init() - d[k] = binop(d[k], item) - return d - - -def iterate(func, x): - """ Repeatedly apply a function func onto an original input - - Yields x, then func(x), then func(func(x)), then func(func(func(x))), etc.. - - >>> def inc(x): return x + 1 - >>> counter = iterate(inc, 0) - >>> next(counter) - 0 - >>> next(counter) - 1 - >>> next(counter) - 2 - - >>> double = lambda x: x * 2 - >>> powers_of_two = iterate(double, 1) - >>> next(powers_of_two) - 1 - >>> next(powers_of_two) - 2 - >>> next(powers_of_two) - 4 - >>> next(powers_of_two) - 8 - """ - while True: - yield x - x = func(x) - - -def sliding_window(n, seq): - """ A sequence of overlapping subsequences - - >>> list(sliding_window(2, [1, 2, 3, 4])) - [(1, 2), (2, 3), (3, 4)] - - This function creates a sliding window suitable for transformations like - sliding means / smoothing - - >>> mean = lambda seq: float(sum(seq)) / len(seq) - >>> list(map(mean, sliding_window(2, [1, 2, 3, 4]))) - [1.5, 2.5, 3.5] - """ - return zip(*(collections.deque(itertools.islice(it, i), 0) or it - for i, it in enumerate(itertools.tee(seq, n)))) - - -no_pad = '__no__pad__' - - -def partition(n, seq, pad=no_pad): - """ Partition sequence into tuples of length n - - >>> list(partition(2, [1, 2, 3, 4])) - [(1, 2), (3, 4)] - - If the length of ``seq`` is not evenly divisible by ``n``, the final tuple - is dropped if ``pad`` is not specified, or filled to length ``n`` by pad: - - >>> list(partition(2, [1, 2, 3, 4, 5])) - [(1, 2), (3, 4)] - - >>> list(partition(2, [1, 2, 3, 4, 5], pad=None)) - [(1, 2), (3, 4), (5, None)] - - See Also: - partition_all - """ - args = [iter(seq)] * n - if pad is no_pad: - return zip(*args) - else: - return zip_longest(*args, fillvalue=pad) - - -def partition_all(n, seq): - """ Partition all elements of sequence into tuples of length at most n - - The final tuple may be shorter to accommodate extra elements. - - >>> list(partition_all(2, [1, 2, 3, 4])) - [(1, 2), (3, 4)] - - >>> list(partition_all(2, [1, 2, 3, 4, 5])) - [(1, 2), (3, 4), (5,)] - - See Also: - partition - """ - args = [iter(seq)] * n - it = zip_longest(*args, fillvalue=no_pad) - try: - prev = next(it) - except StopIteration: - return - for item in it: - yield prev - prev = item - if prev[-1] is no_pad: - try: - # If seq defines __len__, then - # we can quickly calculate where no_pad starts - yield prev[:len(seq) % n] - except TypeError: - # Get first index of no_pad without using .index() - # https://github.com/pytoolz/toolz/issues/387 - # Binary search from CPython's bisect module, - # modified for identity testing. - lo, hi = 0, n - while lo < hi: - mid = (lo + hi) // 2 - if prev[mid] is no_pad: - hi = mid - else: - lo = mid + 1 - yield prev[:lo] - else: - yield prev - - -def count(seq): - """ Count the number of items in seq - - Like the builtin ``len`` but works on lazy sequencies. - - Not to be confused with ``itertools.count`` - - See also: - len - """ - if hasattr(seq, '__len__'): - return len(seq) - return sum(1 for i in seq) - - -def pluck(ind, seqs, default=no_default): - """ plucks an element or several elements from each item in a sequence. - - ``pluck`` maps ``itertoolz.get`` over a sequence and returns one or more - elements of each item in the sequence. - - This is equivalent to running `map(curried.get(ind), seqs)` - - ``ind`` can be either a single string/index or a list of strings/indices. - ``seqs`` should be sequence containing sequences or dicts. - - e.g. - - >>> data = [{'id': 1, 'name': 'Cheese'}, {'id': 2, 'name': 'Pies'}] - >>> list(pluck('name', data)) - ['Cheese', 'Pies'] - >>> list(pluck([0, 1], [[1, 2, 3], [4, 5, 7]])) - [(1, 2), (4, 5)] - - See Also: - get - map - """ - if default == no_default: - get = getter(ind) - return map(get, seqs) - elif isinstance(ind, list): - return (tuple(_get(item, seq, default) for item in ind) - for seq in seqs) - return (_get(ind, seq, default) for seq in seqs) - - -def getter(index): - if isinstance(index, list): - if len(index) == 1: - index = index[0] - return lambda x: (x[index],) - elif index: - return operator.itemgetter(*index) - else: - return lambda x: () - else: - return operator.itemgetter(index) - - -def join(leftkey, leftseq, rightkey, rightseq, - left_default=no_default, right_default=no_default): - """ Join two sequences on common attributes - - This is a semi-streaming operation. The LEFT sequence is fully evaluated - and placed into memory. The RIGHT sequence is evaluated lazily and so can - be arbitrarily large. - (Note: If right_default is defined, then unique keys of rightseq - will also be stored in memory.) - - >>> friends = [('Alice', 'Edith'), - ... ('Alice', 'Zhao'), - ... ('Edith', 'Alice'), - ... ('Zhao', 'Alice'), - ... ('Zhao', 'Edith')] - - >>> cities = [('Alice', 'NYC'), - ... ('Alice', 'Chicago'), - ... ('Dan', 'Syndey'), - ... ('Edith', 'Paris'), - ... ('Edith', 'Berlin'), - ... ('Zhao', 'Shanghai')] - - >>> # Vacation opportunities - >>> # In what cities do people have friends? - >>> result = join(second, friends, - ... first, cities) - >>> for ((a, b), (c, d)) in sorted(unique(result)): - ... print((a, d)) - ('Alice', 'Berlin') - ('Alice', 'Paris') - ('Alice', 'Shanghai') - ('Edith', 'Chicago') - ('Edith', 'NYC') - ('Zhao', 'Chicago') - ('Zhao', 'NYC') - ('Zhao', 'Berlin') - ('Zhao', 'Paris') - - Specify outer joins with keyword arguments ``left_default`` and/or - ``right_default``. Here is a full outer join in which unmatched elements - are paired with None. - - >>> identity = lambda x: x - >>> list(join(identity, [1, 2, 3], - ... identity, [2, 3, 4], - ... left_default=None, right_default=None)) - [(2, 2), (3, 3), (None, 4), (1, None)] - - Usually the key arguments are callables to be applied to the sequences. If - the keys are not obviously callable then it is assumed that indexing was - intended, e.g. the following is a legal change. - The join is implemented as a hash join and the keys of leftseq must be - hashable. Additionally, if right_default is defined, then keys of rightseq - must also be hashable. - - >>> # result = join(second, friends, first, cities) - >>> result = join(1, friends, 0, cities) # doctest: +SKIP - """ - if not callable(leftkey): - leftkey = getter(leftkey) - if not callable(rightkey): - rightkey = getter(rightkey) - - d = groupby(leftkey, leftseq) - - if left_default == no_default and right_default == no_default: - # Inner Join - for item in rightseq: - key = rightkey(item) - if key in d: - for left_match in d[key]: - yield (left_match, item) - elif left_default != no_default and right_default == no_default: - # Right Join - for item in rightseq: - key = rightkey(item) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - yield (left_default, item) - elif right_default != no_default: - seen_keys = set() - seen = seen_keys.add - - if left_default == no_default: - # Left Join - for item in rightseq: - key = rightkey(item) - seen(key) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - # Full Join - for item in rightseq: - key = rightkey(item) - seen(key) - if key in d: - for left_match in d[key]: - yield (left_match, item) - else: - yield (left_default, item) - - for key, matches in iteritems(d): - if key not in seen_keys: - for match in matches: - yield (match, right_default) - - -def diff(*seqs, **kwargs): - """ Return those items that differ between sequences - - >>> list(diff([1, 2, 3], [1, 2, 10, 100])) - [(3, 10)] - - Shorter sequences may be padded with a ``default`` value: - - >>> list(diff([1, 2, 3], [1, 2, 10, 100], default=None)) - [(3, 10), (None, 100)] - - A ``key`` function may also be applied to each item to use during - comparisons: - - >>> list(diff(['apples', 'bananas'], ['Apples', 'Oranges'], key=str.lower)) - [('bananas', 'Oranges')] - """ - N = len(seqs) - if N == 1 and isinstance(seqs[0], list): - seqs = seqs[0] - N = len(seqs) - if N < 2: - raise TypeError('Too few sequences given (min 2 required)') - default = kwargs.get('default', no_default) - if default == no_default: - iters = zip(*seqs) - else: - iters = zip_longest(*seqs, fillvalue=default) - key = kwargs.get('key', None) - if key is None: - for items in iters: - if items.count(items[0]) != N: - yield items - else: - for items in iters: - vals = tuple(map(key, items)) - if vals.count(vals[0]) != N: - yield items - - -def topk(k, seq, key=None): - """ Find the k largest elements of a sequence - - Operates lazily in ``n*log(k)`` time - - >>> topk(2, [1, 100, 10, 1000]) - (1000, 100) - - Use a key function to change sorted order - - >>> topk(2, ['Alice', 'Bob', 'Charlie', 'Dan'], key=len) - ('Charlie', 'Alice') - - See also: - heapq.nlargest - """ - if key is not None and not callable(key): - key = getter(key) - return tuple(heapq.nlargest(k, seq, key=key)) - - -def peek(seq): - """ Retrieve the next element of a sequence - - Returns the first element and an iterable equivalent to the original - sequence, still having the element retrieved. - - >>> seq = [0, 1, 2, 3, 4] - >>> first, seq = peek(seq) - >>> first - 0 - >>> list(seq) - [0, 1, 2, 3, 4] - """ - iterator = iter(seq) - item = next(iterator) - return item, itertools.chain((item,), iterator) - - -def peekn(n, seq): - """ Retrieve the next n elements of a sequence - - Returns a tuple of the first n elements and an iterable equivalent - to the original, still having the elements retrieved. - - >>> seq = [0, 1, 2, 3, 4] - >>> first_two, seq = peekn(2, seq) - >>> first_two - (0, 1) - >>> list(seq) - [0, 1, 2, 3, 4] - """ - iterator = iter(seq) - peeked = tuple(take(n, iterator)) - return peeked, itertools.chain(iter(peeked), iterator) - - -def random_sample(prob, seq, random_state=None): - """ Return elements from a sequence with probability of prob - - Returns a lazy iterator of random items from seq. - - ``random_sample`` considers each item independently and without - replacement. See below how the first time it returned 13 items and the - next time it returned 6 items. - - >>> seq = list(range(100)) - >>> list(random_sample(0.1, seq)) # doctest: +SKIP - [6, 9, 19, 35, 45, 50, 58, 62, 68, 72, 78, 86, 95] - >>> list(random_sample(0.1, seq)) # doctest: +SKIP - [6, 44, 54, 61, 69, 94] - - Providing an integer seed for ``random_state`` will result in - deterministic sampling. Given the same seed it will return the same sample - every time. - - >>> list(random_sample(0.1, seq, random_state=2016)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - >>> list(random_sample(0.1, seq, random_state=2016)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - - ``random_state`` can also be any object with a method ``random`` that - returns floats between 0.0 and 1.0 (exclusive). - - >>> from random import Random - >>> randobj = Random(2016) - >>> list(random_sample(0.1, seq, random_state=randobj)) - [7, 9, 19, 25, 30, 32, 34, 48, 59, 60, 81, 98] - """ - if not hasattr(random_state, 'random'): - random_state = Random(random_state) - return filter(lambda _: random_state.random() < prob, seq) diff --git a/contrib/python/toolz/py2/toolz/recipes.py b/contrib/python/toolz/py2/toolz/recipes.py deleted file mode 100644 index 08c6c8c1e2a..00000000000 --- a/contrib/python/toolz/py2/toolz/recipes.py +++ /dev/null @@ -1,47 +0,0 @@ -import itertools -from .itertoolz import frequencies, pluck, getter -from .compatibility import map - - -__all__ = ('countby', 'partitionby') - - -def countby(key, seq): - """ Count elements of a collection by a key function - - >>> countby(len, ['cat', 'mouse', 'dog']) - {3: 2, 5: 1} - - >>> def iseven(x): return x % 2 == 0 - >>> countby(iseven, [1, 2, 3]) # doctest:+SKIP - {True: 1, False: 2} - - See Also: - groupby - """ - if not callable(key): - key = getter(key) - return frequencies(map(key, seq)) - - -def partitionby(func, seq): - """ Partition a sequence according to a function - - Partition `s` into a sequence of lists such that, when traversing - `s`, every time the output of `func` changes a new list is started - and that and subsequent items are collected into that list. - - >>> is_space = lambda c: c == " " - >>> list(partitionby(is_space, "I have space")) - [('I',), (' ',), ('h', 'a', 'v', 'e'), (' ',), ('s', 'p', 'a', 'c', 'e')] - - >>> is_large = lambda x: x > 10 - >>> list(partitionby(is_large, [1, 2, 1, 99, 88, 33, 99, -1, 5])) - [(1, 2, 1), (99, 88, 33, 99), (-1, 5)] - - See also: - partition - groupby - itertools.groupby - """ - return map(tuple, pluck(1, itertools.groupby(seq, key=func))) diff --git a/contrib/python/toolz/py2/toolz/sandbox/__init__.py b/contrib/python/toolz/py2/toolz/sandbox/__init__.py deleted file mode 100644 index 0abda1cb427..00000000000 --- a/contrib/python/toolz/py2/toolz/sandbox/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -from .core import EqualityHashKey, unzip -from .parallel import fold diff --git a/contrib/python/toolz/py2/toolz/sandbox/core.py b/contrib/python/toolz/py2/toolz/sandbox/core.py deleted file mode 100644 index 915f06c213b..00000000000 --- a/contrib/python/toolz/py2/toolz/sandbox/core.py +++ /dev/null @@ -1,133 +0,0 @@ -from toolz.itertoolz import getter, cons, pluck -from itertools import tee, starmap - - -# See #166: https://github.com/pytoolz/toolz/issues/166 -# See #173: https://github.com/pytoolz/toolz/pull/173 -class EqualityHashKey(object): - """ Create a hash key that uses equality comparisons between items. - - This may be used to create hash keys for otherwise unhashable types: - - >>> from toolz import curry - >>> EqualityHashDefault = curry(EqualityHashKey, None) - >>> set(map(EqualityHashDefault, [[], (), [1], [1]])) # doctest: +SKIP - {=[]=, =()=, =[1]=} - - **Caution:** adding N ``EqualityHashKey`` items to a hash container - may require O(N**2) operations, not O(N) as for typical hashable types. - Therefore, a suitable key function such as ``tuple`` or ``frozenset`` - is usually preferred over using ``EqualityHashKey`` if possible. - - The ``key`` argument to ``EqualityHashKey`` should be a function or - index that returns a hashable object that effectively distinguishes - unequal items. This helps avoid the poor scaling that occurs when - using the default key. For example, the above example can be improved - by using a key function that distinguishes items by length or type: - - >>> EqualityHashLen = curry(EqualityHashKey, len) - >>> EqualityHashType = curry(EqualityHashKey, type) # this works too - >>> set(map(EqualityHashLen, [[], (), [1], [1]])) # doctest: +SKIP - {=[]=, =()=, =[1]=} - - ``EqualityHashKey`` is convenient to use when a suitable key function - is complicated or unavailable. For example, the following returns all - unique values based on equality: - - >>> from toolz import unique - >>> vals = [[], [], (), [1], [1], [2], {}, {}, {}] - >>> list(unique(vals, key=EqualityHashDefault)) - [[], (), [1], [2], {}] - - **Warning:** don't change the equality value of an item already in a hash - containter. Unhashable types are unhashable for a reason. For example: - - >>> L1 = [1] ; L2 = [2] - >>> s = set(map(EqualityHashDefault, [L1, L2])) - >>> s # doctest: +SKIP - {=[1]=, =[2]=} - - >>> L1[0] = 2 # Don't do this! ``s`` now has duplicate items! - >>> s # doctest: +SKIP - {=[2]=, =[2]=} - - Although this may appear problematic, immutable data types is a common - idiom in functional programming, and``EqualityHashKey`` easily allows - the same idiom to be used by convention rather than strict requirement. - - See Also: - identity - """ - __slots__ = ['item', 'key'] - _default_hashkey = '__default__hashkey__' - - def __init__(self, key, item): - if key is None: - self.key = self._default_hashkey - elif not callable(key): - self.key = getter(key) - else: - self.key = key - self.item = item - - def __hash__(self): - if self.key == self._default_hashkey: - val = self.key - else: - val = self.key(self.item) - return hash(val) - - def __eq__(self, other): - try: - return (self._default_hashkey == other._default_hashkey and - self.item == other.item) - except AttributeError: - return False - - def __ne__(self, other): - return not self.__eq__(other) - - def __str__(self): - return '=%s=' % str(self.item) - - def __repr__(self): - return '=%s=' % repr(self.item) - - -# See issue #293: https://github.com/pytoolz/toolz/issues/239 -def unzip(seq): - """Inverse of ``zip`` - - >>> a, b = unzip([('a', 1), ('b', 2)]) - >>> list(a) - ['a', 'b'] - >>> list(b) - [1, 2] - - Unlike the naive implementation ``def unzip(seq): zip(*seq)`` this - implementation can handle an infinite sequence ``seq``. - - Caveats: - - * The implementation uses ``tee``, and so can use a significant amount - of auxiliary storage if the resulting iterators are consumed at - different times. - - * The inner sequence cannot be infinite. In Python 3 ``zip(*seq)`` can be - used if ``seq`` is a finite sequence of infinite sequences. - - """ - - seq = iter(seq) - - # Check how many iterators we need - try: - first = tuple(next(seq)) - except StopIteration: - return tuple() - - # and create them - niters = len(first) - seqs = tee(cons(first, seq), niters) - - return tuple(starmap(pluck, enumerate(seqs))) diff --git a/contrib/python/toolz/py2/toolz/sandbox/parallel.py b/contrib/python/toolz/py2/toolz/sandbox/parallel.py deleted file mode 100644 index ef8ed39dbd7..00000000000 --- a/contrib/python/toolz/py2/toolz/sandbox/parallel.py +++ /dev/null @@ -1,76 +0,0 @@ -import functools -from toolz.itertoolz import partition_all -from toolz.compatibility import reduce, map -from toolz.utils import no_default - - -def _reduce(func, seq, initial=None): - if initial is None: - return functools.reduce(func, seq) - else: - return functools.reduce(func, seq, initial) - - -def fold(binop, seq, default=no_default, map=map, chunksize=128, combine=None): - """ - Reduce without guarantee of ordered reduction. - - inputs: - - ``binop`` - associative operator. The associative property allows us to - leverage a parallel map to perform reductions in parallel. - ``seq`` - a sequence to be aggregated - ``default`` - an identity element like 0 for ``add`` or 1 for mul - - ``map`` - an implementation of ``map``. This may be parallel and - determines how work is distributed. - ``chunksize`` - Number of elements of ``seq`` that should be handled - within a single function call - ``combine`` - Binary operator to combine two intermediate results. - If ``binop`` is of type (total, item) -> total - then ``combine`` is of type (total, total) -> total - Defaults to ``binop`` for common case of operators like add - - Fold chunks up the collection into blocks of size ``chunksize`` and then - feeds each of these to calls to ``reduce``. This work is distributed - with a call to ``map``, gathered back and then refolded to finish the - computation. In this way ``fold`` specifies only how to chunk up data but - leaves the distribution of this work to an externally provided ``map`` - function. This function can be sequential or rely on multithreading, - multiprocessing, or even distributed solutions. - - If ``map`` intends to serialize functions it should be prepared to accept - and serialize lambdas. Note that the standard ``pickle`` module fails - here. - - Example - ------- - - >>> # Provide a parallel map to accomplish a parallel sum - >>> from operator import add - >>> fold(add, [1, 2, 3, 4], chunksize=2, map=map) - 10 - """ - assert chunksize > 1 - - if combine is None: - combine = binop - - chunks = partition_all(chunksize, seq) - - # Evaluate sequence in chunks via map - if default == no_default: - results = map( - functools.partial(_reduce, binop), - chunks) - else: - results = map( - functools.partial(_reduce, binop, initial=default), - chunks) - - results = list(results) # TODO: Support complete laziness - - if len(results) == 1: # Return completed result - return results[0] - else: # Recurse to reaggregate intermediate results - return fold(combine, results, map=map, chunksize=chunksize) diff --git a/contrib/python/toolz/py2/toolz/utils.py b/contrib/python/toolz/py2/toolz/utils.py deleted file mode 100644 index 1002c4649f8..00000000000 --- a/contrib/python/toolz/py2/toolz/utils.py +++ /dev/null @@ -1,9 +0,0 @@ -def raises(err, lamda): - try: - lamda() - return False - except err: - return True - - -no_default = '__no__default__' diff --git a/contrib/python/toolz/py2/ya.make b/contrib/python/toolz/py2/ya.make deleted file mode 100644 index f64a13da67c..00000000000 --- a/contrib/python/toolz/py2/ya.make +++ /dev/null @@ -1,41 +0,0 @@ -# Generated by devtools/yamaker (pypi). - -PY2_LIBRARY() - -VERSION(0.10.0) - -LICENSE(BSD-3-Clause) - -NO_LINT() - -PY_SRCS( - TOP_LEVEL - tlz/__init__.py - tlz/_build_tlz.py - toolz/__init__.py - toolz/_signatures.py - toolz/compatibility.py - toolz/curried/__init__.py - toolz/curried/exceptions.py - toolz/curried/operator.py - toolz/dicttoolz.py - toolz/functoolz.py - toolz/itertoolz.py - toolz/recipes.py - toolz/sandbox/__init__.py - toolz/sandbox/core.py - toolz/sandbox/parallel.py - toolz/utils.py -) - -RESOURCE_FILES( - PREFIX contrib/python/toolz/py2/ - .dist-info/METADATA - .dist-info/top_level.txt -) - -END() - -RECURSE_FOR_TESTS( - tests -) diff --git a/contrib/python/toolz/py3/AUTHORS.md b/contrib/python/toolz/py3/AUTHORS.md deleted file mode 100644 index bd4a563d9b3..00000000000 --- a/contrib/python/toolz/py3/AUTHORS.md +++ /dev/null @@ -1,33 +0,0 @@ -[Matthew Rocklin](http://matthewrocklin.com) [@mrocklin](http://github.com/mrocklin/) - -[John Jacobsen](http://eigenhombre.com) [@eigenhombre](http://github.com/eigenhombre/) - -Erik Welch [@eriknw](https://github.com/eriknw/) - -John Crichton [@jcrichton](https://github.com/jcrichton/) - -Han Semaj [@microamp](https://github.com/microamp/) - -[Graeme Coupar](https://twitter.com/obmarg) [@obmarg](https://github.com/obmarg/) - -[Leonid Shvechikov](http://brainstorage.me/shvechikov) [@shvechikov](https://github.com/shvechikov) - -Lars Buitinck [@larsmans](http://github.com/larsmans) - -José Ricardo [@josericardo](https://github.com/josericardo) - -Tom Prince [@tomprince](https://github.com/tomprince) - -Bart van Merriënboer [@bartvm](https://github.com/bartvm) - -Nikolaos-Digenis Karagiannis [@digenis](https://github.com/digenis/) - -[Antonio Lima](https://twitter.com/themiurgo) [@themiurgo](https://github.com/themiurgo/) - -Joe Jevnik [@llllllllll](https://github.com/llllllllll) - -Rory Kirchner [@roryk](https://github.com/roryk) - -[Steven Cutting](http://steven-cutting.github.io) [@steven_cutting](https://github.com/steven-cutting) - -Aric Coady [@coady](https://github.com/coady) diff --git a/contrib/python/toolz/py3/LICENSE.txt b/contrib/python/toolz/py3/LICENSE.txt deleted file mode 100644 index eeb91b202ca..00000000000 --- a/contrib/python/toolz/py3/LICENSE.txt +++ /dev/null @@ -1,28 +0,0 @@ -Copyright (c) 2013 Matthew Rocklin - -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: - - a. Redistributions of source code must retain the above copyright notice, - this list of conditions and the following disclaimer. - b. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - c. Neither the name of toolz nor the names of its contributors - may be used to endorse or promote products derived from this software - without specific prior written permission. - - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR -ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL -DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR -SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER -CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY -OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH -DAMAGE. diff --git a/contrib/python/toolz/py3/README.rst b/contrib/python/toolz/py3/README.rst deleted file mode 100644 index e62ac7917e3..00000000000 --- a/contrib/python/toolz/py3/README.rst +++ /dev/null @@ -1,132 +0,0 @@ -Toolz -===== - -|Build Status| |Coverage Status| |Version Status| - -A set of utility functions for iterators, functions, and dictionaries. - -See the PyToolz documentation at https://toolz.readthedocs.io - -LICENSE -------- - -New BSD. See `License File <https://github.com/pytoolz/toolz/blob/master/LICENSE.txt>`__. - -Install -------- - -``toolz`` is on the Python Package Index (PyPI): - -:: - - pip install toolz - -Structure and Heritage ----------------------- - -``toolz`` is implemented in three parts: - -|literal itertoolz|_, for operations on iterables. Examples: ``groupby``, -``unique``, ``interpose``, - -|literal functoolz|_, for higher-order functions. Examples: ``memoize``, -``curry``, ``compose``, - -|literal dicttoolz|_, for operations on dictionaries. Examples: ``assoc``, -``update-in``, ``merge``. - -.. |literal itertoolz| replace:: ``itertoolz`` -.. _literal itertoolz: https://github.com/pytoolz/toolz/blob/master/toolz/itertoolz.py - -.. |literal functoolz| replace:: ``functoolz`` -.. _literal functoolz: https://github.com/pytoolz/toolz/blob/master/toolz/functoolz.py - -.. |literal dicttoolz| replace:: ``dicttoolz`` -.. _literal dicttoolz: https://github.com/pytoolz/toolz/blob/master/toolz/dicttoolz.py - -These functions come from the legacy of functional languages for list -processing. They interoperate well to accomplish common complex tasks. - -Read our `API -Documentation <https://toolz.readthedocs.io/en/latest/api.html>`__ for -more details. - -Example -------- - -This builds a standard wordcount function from pieces within ``toolz``: - -.. code:: python - - >>> def stem(word): - ... """ Stem word to primitive form """ - ... return word.lower().rstrip(",.!:;'-\"").lstrip("'\"") - - >>> from toolz import compose, frequencies - >>> from toolz.curried import map - >>> wordcount = compose(frequencies, map(stem), str.split) - - >>> sentence = "This cat jumped over this other cat!" - >>> wordcount(sentence) - {'this': 2, 'cat': 2, 'jumped': 1, 'over': 1, 'other': 1} - -Dependencies ------------- - -``toolz`` supports Python 3.5+ with a common codebase. -It is pure Python and requires no dependencies beyond the standard -library. - -It is, in short, a lightweight dependency. - - -CyToolz -------- - -The ``toolz`` project has been reimplemented in `Cython <http://cython.org>`__. -The ``cytoolz`` project is a drop-in replacement for the Pure Python -implementation. -See `CyToolz GitHub Page <https://github.com/pytoolz/cytoolz/>`__ for more -details. - -See Also --------- - -- `Underscore.js <https://underscorejs.org/>`__: A similar library for - JavaScript -- `Enumerable <https://ruby-doc.org/core-2.0.0/Enumerable.html>`__: A - similar library for Ruby -- `Clojure <https://clojure.org/>`__: A functional language whose - standard library has several counterparts in ``toolz`` -- `itertools <https://docs.python.org/2/library/itertools.html>`__: The - Python standard library for iterator tools -- `functools <https://docs.python.org/2/library/functools.html>`__: The - Python standard library for function tools - -Contributions Welcome ---------------------- - -``toolz`` aims to be a repository for utility functions, particularly -those that come from the functional programming and list processing -traditions. We welcome contributions that fall within this scope. - -We also try to keep the API small to keep ``toolz`` manageable. The ideal -contribution is significantly different from existing functions and has -precedent in a few other functional systems. - -Please take a look at our -`issue page <https://github.com/pytoolz/toolz/issues>`__ -for contribution ideas. - -Community ---------- - -See our `mailing list <https://groups.google.com/forum/#!forum/pytoolz>`__. -We're friendly. - -.. |Build Status| image:: https://github.com/pytoolz/toolz/workflows/Test/badge.svg - :target: https://github.com/pytoolz/toolz/actions -.. |Coverage Status| image:: https://coveralls.io/repos/pytoolz/toolz/badge.svg?branch=master - :target: https://coveralls.io/r/pytoolz/toolz -.. |Version Status| image:: https://badge.fury.io/py/toolz.svg - :target: https://badge.fury.io/py/toolz diff --git a/contrib/python/toolz/ya.make b/contrib/python/toolz/ya.make deleted file mode 100644 index 0c05eedeac7..00000000000 --- a/contrib/python/toolz/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/toolz/py2) -ELSE() - PEERDIR(contrib/python/toolz/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) diff --git a/contrib/python/ujson/py2/lib/ultrajson.h b/contrib/python/ujson/py2/lib/ultrajson.h deleted file mode 100644 index 92902b4910c..00000000000 --- a/contrib/python/ujson/py2/lib/ultrajson.h +++ /dev/null @@ -1,333 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms - * Copyright (c) 1988-1993 The Regents of the University of California. - * Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -/* -Ultra fast JSON encoder and decoder -Developed by Jonas Tarnstrom (jonas@esn.me). - -Encoder notes: ------------------- - -:: Cyclic references :: -Cyclic referenced objects are not detected. -Set JSONObjectEncoder.recursionMax to suitable value or make sure input object -tree doesn't have cyclic references. - -*/ - -#ifndef __ULTRAJSON_H__ -#define __ULTRAJSON_H__ - -#include <stdio.h> -#include <wchar.h> - -// Don't output any extra whitespaces when encoding -#define JSON_NO_EXTRA_WHITESPACE - -// Max decimals to encode double floating point numbers with -#ifndef JSON_DOUBLE_MAX_DECIMALS -#define JSON_DOUBLE_MAX_DECIMALS 15 -#endif - -// Max recursion depth, default for encoder -#ifndef JSON_MAX_RECURSION_DEPTH -#define JSON_MAX_RECURSION_DEPTH 1024 -#endif - -// Max recursion depth, default for decoder -#ifndef JSON_MAX_OBJECT_DEPTH -#define JSON_MAX_OBJECT_DEPTH 1024 -#endif - -/* -Dictates and limits how much stack space for buffers UltraJSON will use before resorting to provided heap functions */ -#ifndef JSON_MAX_STACK_BUFFER_SIZE -#define JSON_MAX_STACK_BUFFER_SIZE 131072 -#endif - -#ifdef _WIN32 - -typedef __int64 JSINT64; -typedef unsigned __int64 JSUINT64; - -typedef __int32 JSINT32; -typedef unsigned __int32 JSUINT32; -typedef unsigned __int8 JSUINT8; -typedef unsigned __int16 JSUTF16; -typedef unsigned __int32 JSUTF32; -typedef __int64 JSLONG; - -#define EXPORTFUNCTION __declspec(dllexport) - -#define FASTCALL_MSVC __fastcall -#define FASTCALL_ATTR -#define INLINE_PREFIX __inline - -#else - -#include <stdint.h> -typedef int64_t JSINT64; -typedef uint64_t JSUINT64; - -typedef int32_t JSINT32; -typedef uint32_t JSUINT32; - -#define FASTCALL_MSVC - -#if !defined __x86_64__ -#define FASTCALL_ATTR __attribute__((fastcall)) -#else -#define FASTCALL_ATTR -#endif - - -#if defined(__clang__) -#define INLINE_PREFIX inline __attribute__((always_inline)) -#else -#define INLINE_PREFIX static inline -#endif - -typedef uint8_t JSUINT8; -typedef uint16_t JSUTF16; -typedef uint32_t JSUTF32; - -typedef int64_t JSLONG; - -#define EXPORTFUNCTION -#endif - -#if !(defined(__LITTLE_ENDIAN__) || defined(__BIG_ENDIAN__)) - -#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ -#define __LITTLE_ENDIAN__ -#else - -#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ -#define __BIG_ENDIAN__ -#endif - -#endif - -#endif - -#if !defined(__LITTLE_ENDIAN__) && !defined(__BIG_ENDIAN__) -#error "Endianess not supported" -#endif - -enum JSTYPES -{ - JT_NULL, // NULL - JT_TRUE, // boolean true - JT_FALSE, // boolean false - JT_INT, // (JSINT32 (signed 32-bit)) - JT_LONG, // (JSINT64 (signed 64-bit)) - JT_ULONG, // (JSUINT64 (unsigned 64-bit)) - JT_DOUBLE, // (double) - JT_UTF8, // (char 8-bit) - JT_RAW, // (raw char 8-bit) - JT_ARRAY, // Array structure - JT_OBJECT, // Key/Value structure - JT_INVALID, // Internal, do not return nor expect -}; - -typedef void * JSOBJ; -typedef void * JSITER; - -typedef struct __JSONTypeContext -{ - int type; - void *prv; - void *encoder_prv; -} JSONTypeContext; - -/* -Function pointer declarations, suitable for implementing UltraJSON */ -typedef int (*JSPFN_ITERNEXT)(JSOBJ obj, JSONTypeContext *tc); -typedef void (*JSPFN_ITEREND)(JSOBJ obj, JSONTypeContext *tc); -typedef JSOBJ (*JSPFN_ITERGETVALUE)(JSOBJ obj, JSONTypeContext *tc); -typedef char *(*JSPFN_ITERGETNAME)(JSOBJ obj, JSONTypeContext *tc, size_t *outLen); -typedef void *(*JSPFN_MALLOC)(size_t size); -typedef void (*JSPFN_FREE)(void *pptr); -typedef void *(*JSPFN_REALLOC)(void *base, size_t size); - - -struct __JSONObjectEncoder; - -typedef struct __JSONObjectEncoder -{ - void (*beginTypeContext)(JSOBJ obj, JSONTypeContext *tc, struct __JSONObjectEncoder *enc); - void (*endTypeContext)(JSOBJ obj, JSONTypeContext *tc); - const char *(*getStringValue)(JSOBJ obj, JSONTypeContext *tc, size_t *_outLen); - JSINT64 (*getLongValue)(JSOBJ obj, JSONTypeContext *tc); - JSUINT64 (*getUnsignedLongValue)(JSOBJ obj, JSONTypeContext *tc); - JSINT32 (*getIntValue)(JSOBJ obj, JSONTypeContext *tc); - double (*getDoubleValue)(JSOBJ obj, JSONTypeContext *tc); - - /* - Retrieve next object in an iteration. Should return 0 to indicate iteration has reached end or 1 if there are more items. - Implementor is responsible for keeping state of the iteration. Use ti->prv fields for this - */ - JSPFN_ITERNEXT iterNext; - - /* - Ends the iteration of an iteratable object. - Any iteration state stored in ti->prv can be freed here - */ - JSPFN_ITEREND iterEnd; - - /* - Returns a reference to the value object of an iterator - The is responsible for the life-cycle of the returned string. Use iterNext/iterEnd and ti->prv to keep track of current object - */ - JSPFN_ITERGETVALUE iterGetValue; - - /* - Return name of iterator. - The is responsible for the life-cycle of the returned string. Use iterNext/iterEnd and ti->prv to keep track of current object - */ - JSPFN_ITERGETNAME iterGetName; - - /* - Release a value as indicated by setting ti->release = 1 in the previous getValue call. - The ti->prv array should contain the necessary context to release the value - */ - void (*releaseObject)(JSOBJ obj); - - /* Library functions - Set to NULL to use STDLIB malloc,realloc,free */ - JSPFN_MALLOC malloc; - JSPFN_REALLOC realloc; - JSPFN_FREE free; - - /* - Configuration for max recursion, set to 0 to use default (see JSON_MAX_RECURSION_DEPTH)*/ - int recursionMax; - - /* - Configuration for max decimals of double floating point numbers to encode (0-9) */ - int doublePrecision; - - /* - If true output will be ASCII with all characters above 127 encoded as \uXXXX. If false output will be UTF-8 or what ever charset strings are brought as */ - int forceASCII; - - /* - If true, '<', '>', and '&' characters will be encoded as \u003c, \u003e, and \u0026, respectively. If false, no special encoding will be used. */ - int encodeHTMLChars; - - /* - If true, '/' will be encoded as \/. If false, no escaping. */ - int escapeForwardSlashes; - - /* - If true, dictionaries are iterated through in sorted key order. */ - int sortKeys; - - /* - Configuration for spaces of indent */ - int indent; - - /* - Private pointer to be used by the caller. Passed as encoder_prv in JSONTypeContext */ - void *prv; - - /* - Set to an error message if error occured */ - const char *errorMsg; - JSOBJ errorObj; - - /* Buffer stuff */ - char *start; - char *offset; - char *end; - int heap; - int level; - -} JSONObjectEncoder; - - -/* -Encode an object structure into JSON. - -Arguments: -obj - An anonymous type representing the object -enc - Function definitions for querying JSOBJ type -buffer - Preallocated buffer to store result in. If NULL function allocates own buffer -cbBuffer - Length of buffer (ignored if buffer is NULL) - -Returns: -Encoded JSON object as a null terminated char string. - -NOTE: -If the supplied buffer wasn't enough to hold the result the function will allocate a new buffer. -Life cycle of the provided buffer must still be handled by caller. - -If the return value doesn't equal the specified buffer caller must release the memory using -JSONObjectEncoder.free or free() as specified when calling this function. -*/ -EXPORTFUNCTION char *JSON_EncodeObject(JSOBJ obj, JSONObjectEncoder *enc, char *buffer, size_t cbBuffer); - - - -typedef struct __JSONObjectDecoder -{ - JSOBJ (*newString)(void *prv, wchar_t *start, wchar_t *end); - void (*objectAddKey)(void *prv, JSOBJ obj, JSOBJ name, JSOBJ value); - void (*arrayAddItem)(void *prv, JSOBJ obj, JSOBJ value); - JSOBJ (*newTrue)(void *prv); - JSOBJ (*newFalse)(void *prv); - JSOBJ (*newNull)(void *prv); - JSOBJ (*newObject)(void *prv); - JSOBJ (*newArray)(void *prv); - JSOBJ (*newInt)(void *prv, JSINT32 value); - JSOBJ (*newLong)(void *prv, JSINT64 value); - JSOBJ (*newUnsignedLong)(void *prv, JSUINT64 value); - JSOBJ (*newDouble)(void *prv, double value); - void (*releaseObject)(void *prv, JSOBJ obj); - JSPFN_MALLOC malloc; - JSPFN_FREE free; - JSPFN_REALLOC realloc; - char *errorStr; - char *errorOffset; - int preciseFloat; - void *prv; -} JSONObjectDecoder; - -EXPORTFUNCTION JSOBJ JSON_DecodeObject(JSONObjectDecoder *dec, const char *buffer, size_t cbBuffer); - -#endif diff --git a/contrib/python/ujson/py2/lib/ultrajsondec.c b/contrib/python/ujson/py2/lib/ultrajsondec.c deleted file mode 100644 index 21a732eceb8..00000000000 --- a/contrib/python/ujson/py2/lib/ultrajsondec.c +++ /dev/null @@ -1,907 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms -* Copyright (c) 1988-1993 The Regents of the University of California. -* Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#include "ultrajson.h" -#include <math.h> -#include <assert.h> -#include <string.h> -#include <limits.h> -#include <wchar.h> -#include <stdlib.h> -#include <errno.h> - -#ifndef TRUE -#define TRUE 1 -#define FALSE 0 -#endif -#ifndef NULL -#define NULL 0 -#endif - -struct DecoderState -{ - char *start; - char *end; - wchar_t *escStart; - wchar_t *escEnd; - int escHeap; - int lastType; - JSUINT32 objDepth; - void *prv; - JSONObjectDecoder *dec; -}; - -JSOBJ FASTCALL_MSVC decode_any( struct DecoderState *ds) FASTCALL_ATTR; -typedef JSOBJ (*PFN_DECODER)( struct DecoderState *ds); - -static JSOBJ SetError( struct DecoderState *ds, int offset, const char *message) -{ - ds->dec->errorOffset = ds->start + offset; - ds->dec->errorStr = (char *) message; - return NULL; -} - -double createDouble(double intNeg, double intValue, double frcValue, int frcDecimalCount) -{ - static const double g_pow10[] = {1.0, 0.1, 0.01, 0.001, 0.0001, 0.00001, 0.000001,0.0000001, 0.00000001, 0.000000001, 0.0000000001, 0.00000000001, 0.000000000001, 0.0000000000001, 0.00000000000001, 0.000000000000001}; - return (intValue + (frcValue * g_pow10[frcDecimalCount])) * intNeg; -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decodePreciseFloat(struct DecoderState *ds) -{ - char *end; - double value; - errno = 0; - - value = strtod(ds->start, &end); - - if (errno == ERANGE) - { - return SetError(ds, -1, "Range error when decoding numeric as double"); - } - - ds->start = end; - return ds->dec->newDouble(ds->prv, value); -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_numeric (struct DecoderState *ds) -{ - int intNeg = 1; - int mantSize = 0; - JSUINT64 intValue; - JSUINT64 prevIntValue; - int chr; - int decimalCount = 0; - double frcValue = 0.0; - double expNeg; - double expValue; - char *offset = ds->start; - - JSUINT64 overflowLimit = LLONG_MAX; - - if (*(offset) == '-') - { - offset ++; - intNeg = -1; - overflowLimit = LLONG_MIN; - } - - // Scan integer part - intValue = 0; - - while (1) - { - chr = (int) (unsigned char) *(offset); - - switch (chr) - { - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': - { - //PERF: Don't do 64-bit arithmetic here unless we know we have to - prevIntValue = intValue; - intValue = intValue * 10ULL + (JSLONG) (chr - 48); - - if (intNeg == 1 && prevIntValue > intValue) - { - return SetError(ds, -1, "Value is too big!"); - } - else if (intNeg == -1 && intValue > overflowLimit) - { - return SetError(ds, -1, overflowLimit == LLONG_MAX ? "Value is too big!" : "Value is too small"); - } - - offset ++; - mantSize ++; - break; - } - case '.': - { - offset ++; - goto DECODE_FRACTION; - break; - } - case 'e': - case 'E': - { - offset ++; - goto DECODE_EXPONENT; - break; - } - - default: - { - goto BREAK_INT_LOOP; - break; - } - } - } - -BREAK_INT_LOOP: - - ds->lastType = JT_INT; - ds->start = offset; - - if (intNeg == 1 && (intValue & 0x8000000000000000ULL) != 0) - { - return ds->dec->newUnsignedLong(ds->prv, intValue); - } - else if ((intValue >> 31)) - { - return ds->dec->newLong(ds->prv, (JSINT64) (intValue * (JSINT64) intNeg)); - } - else - { - return ds->dec->newInt(ds->prv, (JSINT32) (intValue * intNeg)); - } - -DECODE_FRACTION: - - if (ds->dec->preciseFloat) - { - return decodePreciseFloat(ds); - } - - // Scan fraction part - frcValue = 0.0; - for (;;) - { - chr = (int) (unsigned char) *(offset); - - switch (chr) - { - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': - { - if (decimalCount < JSON_DOUBLE_MAX_DECIMALS) - { - frcValue = frcValue * 10.0 + (double) (chr - 48); - decimalCount ++; - } - offset ++; - break; - } - case 'e': - case 'E': - { - offset ++; - goto DECODE_EXPONENT; - break; - } - default: - { - goto BREAK_FRC_LOOP; - } - } - } - -BREAK_FRC_LOOP: - //FIXME: Check for arithemtic overflow here - ds->lastType = JT_DOUBLE; - ds->start = offset; - return ds->dec->newDouble (ds->prv, createDouble( (double) intNeg, (double) intValue, frcValue, decimalCount)); - -DECODE_EXPONENT: - if (ds->dec->preciseFloat) - { - return decodePreciseFloat(ds); - } - - expNeg = 1.0; - - if (*(offset) == '-') - { - expNeg = -1.0; - offset ++; - } - else - if (*(offset) == '+') - { - expNeg = +1.0; - offset ++; - } - - expValue = 0.0; - - for (;;) - { - chr = (int) (unsigned char) *(offset); - - switch (chr) - { - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': - { - expValue = expValue * 10.0 + (double) (chr - 48); - offset ++; - break; - } - default: - { - goto BREAK_EXP_LOOP; - } - } - } - -BREAK_EXP_LOOP: - //FIXME: Check for arithemtic overflow here - ds->lastType = JT_DOUBLE; - ds->start = offset; - return ds->dec->newDouble (ds->prv, createDouble( (double) intNeg, (double) intValue , frcValue, decimalCount) * pow(10.0, expValue * expNeg)); -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_true ( struct DecoderState *ds) -{ - char *offset = ds->start; - offset ++; - - if (*(offset++) != 'r') - goto SETERROR; - if (*(offset++) != 'u') - goto SETERROR; - if (*(offset++) != 'e') - goto SETERROR; - - ds->lastType = JT_TRUE; - ds->start = offset; - return ds->dec->newTrue(ds->prv); - -SETERROR: - return SetError(ds, -1, "Unexpected character found when decoding 'true'"); -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_false ( struct DecoderState *ds) -{ - char *offset = ds->start; - offset ++; - - if (*(offset++) != 'a') - goto SETERROR; - if (*(offset++) != 'l') - goto SETERROR; - if (*(offset++) != 's') - goto SETERROR; - if (*(offset++) != 'e') - goto SETERROR; - - ds->lastType = JT_FALSE; - ds->start = offset; - return ds->dec->newFalse(ds->prv); - -SETERROR: - return SetError(ds, -1, "Unexpected character found when decoding 'false'"); -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_null ( struct DecoderState *ds) -{ - char *offset = ds->start; - offset ++; - - if (*(offset++) != 'u') - goto SETERROR; - if (*(offset++) != 'l') - goto SETERROR; - if (*(offset++) != 'l') - goto SETERROR; - - ds->lastType = JT_NULL; - ds->start = offset; - return ds->dec->newNull(ds->prv); - -SETERROR: - return SetError(ds, -1, "Unexpected character found when decoding 'null'"); -} - -FASTCALL_ATTR void FASTCALL_MSVC SkipWhitespace(struct DecoderState *ds) -{ - char *offset = ds->start; - - for (;;) - { - switch (*offset) - { - case ' ': - case '\t': - case '\r': - case '\n': - offset ++; - break; - - default: - ds->start = offset; - return; - } - } -} - -enum DECODESTRINGSTATE -{ - DS_ISNULL = 0x32, - DS_ISQUOTE, - DS_ISESCAPE, - DS_UTFLENERROR, - -}; - -static const JSUINT8 g_decoderLookup[256] = -{ - /* 0x00 */ DS_ISNULL, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x10 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x20 */ 1, 1, DS_ISQUOTE, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x30 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x40 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x50 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, DS_ISESCAPE, 1, 1, 1, - /* 0x60 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x70 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x80 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0x90 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0xa0 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0xb0 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - /* 0xc0 */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, - /* 0xd0 */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, - /* 0xe0 */ 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, - /* 0xf0 */ 4, 4, 4, 4, 4, 4, 4, 4, DS_UTFLENERROR, DS_UTFLENERROR, DS_UTFLENERROR, DS_UTFLENERROR, DS_UTFLENERROR, DS_UTFLENERROR, DS_UTFLENERROR, DS_UTFLENERROR, -}; - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_string ( struct DecoderState *ds) -{ - JSUTF16 sur[2] = { 0 }; - int iSur = 0; - int index; - wchar_t *escOffset; - wchar_t *escStart; - size_t escLen = (ds->escEnd - ds->escStart); - JSUINT8 *inputOffset; - JSUINT8 oct; - JSUTF32 ucs; - ds->lastType = JT_INVALID; - ds->start ++; - - if ( (size_t) (ds->end - ds->start) > escLen) - { - size_t newSize = (ds->end - ds->start); - - if (ds->escHeap) - { - if (newSize > (SIZE_MAX / sizeof(wchar_t))) - { - return SetError(ds, -1, "Could not reserve memory block"); - } - escStart = (wchar_t *)ds->dec->realloc(ds->escStart, newSize * sizeof(wchar_t)); - if (!escStart) - { - ds->dec->free(ds->escStart); - return SetError(ds, -1, "Could not reserve memory block"); - } - ds->escStart = escStart; - } - else - { - wchar_t *oldStart = ds->escStart; - if (newSize > (SIZE_MAX / sizeof(wchar_t))) - { - return SetError(ds, -1, "Could not reserve memory block"); - } - ds->escStart = (wchar_t *) ds->dec->malloc(newSize * sizeof(wchar_t)); - if (!ds->escStart) - { - return SetError(ds, -1, "Could not reserve memory block"); - } - ds->escHeap = 1; - memcpy(ds->escStart, oldStart, escLen * sizeof(wchar_t)); - } - - ds->escEnd = ds->escStart + newSize; - } - - escOffset = ds->escStart; - inputOffset = (JSUINT8 *) ds->start; - - for (;;) - { - switch (g_decoderLookup[(JSUINT8)(*inputOffset)]) - { - case DS_ISNULL: - { - return SetError(ds, -1, "Unmatched ''\"' when when decoding 'string'"); - } - case DS_ISQUOTE: - { - ds->lastType = JT_UTF8; - inputOffset ++; - ds->start += ( (char *) inputOffset - (ds->start)); - return ds->dec->newString(ds->prv, ds->escStart, escOffset); - } - case DS_UTFLENERROR: - { - return SetError (ds, -1, "Invalid UTF-8 sequence length when decoding 'string'"); - } - case DS_ISESCAPE: - inputOffset ++; - switch (*inputOffset) - { - case '\\': *(escOffset++) = L'\\'; inputOffset++; continue; - case '\"': *(escOffset++) = L'\"'; inputOffset++; continue; - case '/': *(escOffset++) = L'/'; inputOffset++; continue; - case 'b': *(escOffset++) = L'\b'; inputOffset++; continue; - case 'f': *(escOffset++) = L'\f'; inputOffset++; continue; - case 'n': *(escOffset++) = L'\n'; inputOffset++; continue; - case 'r': *(escOffset++) = L'\r'; inputOffset++; continue; - case 't': *(escOffset++) = L'\t'; inputOffset++; continue; - - case 'u': - { - int index; - inputOffset ++; - - for (index = 0; index < 4; index ++) - { - switch (*inputOffset) - { - case '\0': return SetError (ds, -1, "Unterminated unicode escape sequence when decoding 'string'"); - default: return SetError (ds, -1, "Unexpected character in unicode escape sequence when decoding 'string'"); - - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': - sur[iSur] = (sur[iSur] << 4) + (JSUTF16) (*inputOffset - '0'); - break; - - case 'a': - case 'b': - case 'c': - case 'd': - case 'e': - case 'f': - sur[iSur] = (sur[iSur] << 4) + 10 + (JSUTF16) (*inputOffset - 'a'); - break; - - case 'A': - case 'B': - case 'C': - case 'D': - case 'E': - case 'F': - sur[iSur] = (sur[iSur] << 4) + 10 + (JSUTF16) (*inputOffset - 'A'); - break; - } - - inputOffset ++; - } - - if (iSur == 0) - { - if((sur[iSur] & 0xfc00) == 0xd800) - { - // First of a surrogate pair, continue parsing - iSur ++; - break; - } - (*escOffset++) = (wchar_t) sur[iSur]; - iSur = 0; - } - else - { - // Decode pair - if ((sur[1] & 0xfc00) != 0xdc00) - { - return SetError (ds, -1, "Unpaired high surrogate when decoding 'string'"); - } -#if WCHAR_MAX == 0xffff - (*escOffset++) = (wchar_t) sur[0]; - (*escOffset++) = (wchar_t) sur[1]; -#else - (*escOffset++) = (wchar_t) 0x10000 + (((sur[0] - 0xd800) << 10) | (sur[1] - 0xdc00)); -#endif - iSur = 0; - } - break; - } - - case '\0': return SetError(ds, -1, "Unterminated escape sequence when decoding 'string'"); - default: return SetError(ds, -1, "Unrecognized escape sequence when decoding 'string'"); - } - break; - - case 1: - { - *(escOffset++) = (wchar_t) (*inputOffset++); - break; - } - - case 2: - { - ucs = (*inputOffset++) & 0x1f; - ucs <<= 6; - if (((*inputOffset) & 0x80) != 0x80) - { - return SetError(ds, -1, "Invalid octet in UTF-8 sequence when decoding 'string'"); - } - ucs |= (*inputOffset++) & 0x3f; - if (ucs < 0x80) return SetError (ds, -1, "Overlong 2 byte UTF-8 sequence detected when decoding 'string'"); - *(escOffset++) = (wchar_t) ucs; - break; - } - - case 3: - { - JSUTF32 ucs = 0; - ucs |= (*inputOffset++) & 0x0f; - - for (index = 0; index < 2; index ++) - { - ucs <<= 6; - oct = (*inputOffset++); - - if ((oct & 0x80) != 0x80) - { - return SetError(ds, -1, "Invalid octet in UTF-8 sequence when decoding 'string'"); - } - - ucs |= oct & 0x3f; - } - - if (ucs < 0x800) return SetError (ds, -1, "Overlong 3 byte UTF-8 sequence detected when encoding string"); - *(escOffset++) = (wchar_t) ucs; - break; - } - - case 4: - { - JSUTF32 ucs = 0; - ucs |= (*inputOffset++) & 0x07; - - for (index = 0; index < 3; index ++) - { - ucs <<= 6; - oct = (*inputOffset++); - - if ((oct & 0x80) != 0x80) - { - return SetError(ds, -1, "Invalid octet in UTF-8 sequence when decoding 'string'"); - } - - ucs |= oct & 0x3f; - } - - if (ucs < 0x10000) return SetError (ds, -1, "Overlong 4 byte UTF-8 sequence detected when decoding 'string'"); - -#if WCHAR_MAX == 0xffff - if (ucs >= 0x10000) - { - ucs -= 0x10000; - *(escOffset++) = (wchar_t) (ucs >> 10) + 0xd800; - *(escOffset++) = (wchar_t) (ucs & 0x3ff) + 0xdc00; - } - else - { - *(escOffset++) = (wchar_t) ucs; - } -#else - *(escOffset++) = (wchar_t) ucs; -#endif - break; - } - } - } -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_array(struct DecoderState *ds) -{ - JSOBJ itemValue; - JSOBJ newObj; - int len; - ds->objDepth++; - if (ds->objDepth > JSON_MAX_OBJECT_DEPTH) { - return SetError(ds, -1, "Reached object decoding depth limit"); - } - - newObj = ds->dec->newArray(ds->prv); - len = 0; - - ds->lastType = JT_INVALID; - ds->start ++; - - for (;;) - { - SkipWhitespace(ds); - - if ((*ds->start) == ']') - { - ds->objDepth--; - if (len == 0) - { - ds->start ++; - return newObj; - } - - ds->dec->releaseObject(ds->prv, newObj); - return SetError(ds, -1, "Unexpected character found when decoding array value (1)"); - } - - itemValue = decode_any(ds); - - if (itemValue == NULL) - { - ds->dec->releaseObject(ds->prv, newObj); - return NULL; - } - - ds->dec->arrayAddItem (ds->prv, newObj, itemValue); - - SkipWhitespace(ds); - - switch (*(ds->start++)) - { - case ']': - { - ds->objDepth--; - return newObj; - } - case ',': - break; - - default: - ds->dec->releaseObject(ds->prv, newObj); - return SetError(ds, -1, "Unexpected character found when decoding array value (2)"); - } - - len ++; - } -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_object( struct DecoderState *ds) -{ - JSOBJ itemName; - JSOBJ itemValue; - JSOBJ newObj; - - ds->objDepth++; - if (ds->objDepth > JSON_MAX_OBJECT_DEPTH) { - return SetError(ds, -1, "Reached object decoding depth limit"); - } - - newObj = ds->dec->newObject(ds->prv); - - ds->start ++; - - for (;;) - { - SkipWhitespace(ds); - - if ((*ds->start) == '}') - { - ds->objDepth--; - ds->start ++; - return newObj; - } - - ds->lastType = JT_INVALID; - itemName = decode_any(ds); - - if (itemName == NULL) - { - ds->dec->releaseObject(ds->prv, newObj); - return NULL; - } - - if (ds->lastType != JT_UTF8) - { - ds->dec->releaseObject(ds->prv, newObj); - ds->dec->releaseObject(ds->prv, itemName); - return SetError(ds, -1, "Key name of object must be 'string' when decoding 'object'"); - } - - SkipWhitespace(ds); - - if (*(ds->start++) != ':') - { - ds->dec->releaseObject(ds->prv, newObj); - ds->dec->releaseObject(ds->prv, itemName); - return SetError(ds, -1, "No ':' found when decoding object value"); - } - - SkipWhitespace(ds); - - itemValue = decode_any(ds); - - if (itemValue == NULL) - { - ds->dec->releaseObject(ds->prv, newObj); - ds->dec->releaseObject(ds->prv, itemName); - return NULL; - } - - ds->dec->objectAddKey (ds->prv, newObj, itemName, itemValue); - - SkipWhitespace(ds); - - switch (*(ds->start++)) - { - case '}': - { - ds->objDepth--; - return newObj; - } - case ',': - break; - - default: - ds->dec->releaseObject(ds->prv, newObj); - return SetError(ds, -1, "Unexpected character in found when decoding object value"); - } - } -} - -FASTCALL_ATTR JSOBJ FASTCALL_MSVC decode_any(struct DecoderState *ds) -{ - for (;;) - { - switch (*ds->start) - { - case '\"': - return decode_string (ds); - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': - case '-': - return decode_numeric (ds); - - case '[': return decode_array (ds); - case '{': return decode_object (ds); - case 't': return decode_true (ds); - case 'f': return decode_false (ds); - case 'n': return decode_null (ds); - - case ' ': - case '\t': - case '\r': - case '\n': - // White space - ds->start ++; - break; - - default: - return SetError(ds, -1, "Expected object or value"); - } - } -} - -JSOBJ JSON_DecodeObject(JSONObjectDecoder *dec, const char *buffer, size_t cbBuffer) -{ - /* - FIXME: Base the size of escBuffer of that of cbBuffer so that the unicode escaping doesn't run into the wall each time */ - struct DecoderState ds; - wchar_t escBuffer[(JSON_MAX_STACK_BUFFER_SIZE / sizeof(wchar_t))]; - JSOBJ ret; - - ds.start = (char *) buffer; - ds.end = ds.start + cbBuffer; - - ds.escStart = escBuffer; - ds.escEnd = ds.escStart + (JSON_MAX_STACK_BUFFER_SIZE / sizeof(wchar_t)); - ds.escHeap = 0; - ds.prv = dec->prv; - ds.dec = dec; - ds.dec->errorStr = NULL; - ds.dec->errorOffset = NULL; - ds.objDepth = 0; - - ds.dec = dec; - - ret = decode_any (&ds); - - if (ds.escHeap) - { - dec->free(ds.escStart); - } - - if (!(dec->errorStr)) - { - if ((ds.end - ds.start) > 0) - { - SkipWhitespace(&ds); - } - - if (ds.start != ds.end && ret) - { - dec->releaseObject(ds.prv, ret); - return SetError(&ds, -1, "Trailing data"); - } - } - - return ret; -} diff --git a/contrib/python/ujson/py2/lib/ultrajsonenc.c b/contrib/python/ujson/py2/lib/ultrajsonenc.c deleted file mode 100644 index ed6645a760a..00000000000 --- a/contrib/python/ujson/py2/lib/ultrajsonenc.c +++ /dev/null @@ -1,1029 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms - * Copyright (c) 1988-1993 The Regents of the University of California. - * Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#include "ultrajson.h" -#include <stdio.h> -#include <assert.h> -#include <string.h> -#include <stdlib.h> -#include <math.h> - -#include <float.h> - -#ifndef TRUE -#define TRUE 1 -#endif -#ifndef FALSE -#define FALSE 0 -#endif - -#if ( (defined(_WIN32) || defined(WIN32) ) && ( defined(_MSC_VER) ) ) -#define snprintf sprintf_s -#endif - -/* -Worst cases being: - -Control characters (ASCII < 32) -0x00 (1 byte) input => \u0000 output (6 bytes) -1 * 6 => 6 (6 bytes required) - -or UTF-16 surrogate pairs -4 bytes input in UTF-8 => \uXXXX\uYYYY (12 bytes). - -4 * 6 => 24 bytes (12 bytes required) - -The extra 2 bytes are for the quotes around the string - -*/ -#define RESERVE_STRING(_len) (2 + ((_len) * 6)) - -static const double g_pow10[] = {1, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000}; -static const char g_hexChars[] = "0123456789abcdef"; -static const char g_escapeChars[] = "0123456789\\b\\t\\n\\f\\r\\\"\\\\\\/"; - -/* -FIXME: While this is fine dandy and working it's a magic value mess which probably only the author understands. -Needs a cleanup and more documentation */ - -/* -Table for pure ascii output escaping all characters above 127 to \uXXXX */ -static const JSUINT8 g_asciiOutputTable[256] = -{ -/* 0x00 */ 0, 30, 30, 30, 30, 30, 30, 30, 10, 12, 14, 30, 16, 18, 30, 30, -/* 0x10 */ 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, -/* 0x20 */ 1, 1, 20, 1, 1, 1, 29, 1, 1, 1, 1, 1, 1, 1, 1, 24, -/* 0x30 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 29, 1, 29, 1, -/* 0x40 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0x50 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 22, 1, 1, 1, -/* 0x60 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0x70 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0x80 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0x90 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0xa0 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0xb0 */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -/* 0xc0 */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, -/* 0xd0 */ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, -/* 0xe0 */ 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, -/* 0xf0 */ 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 1, 1 -}; - -static void SetError (JSOBJ obj, JSONObjectEncoder *enc, const char *message) -{ - enc->errorMsg = message; - enc->errorObj = obj; -} - -/* -FIXME: Keep track of how big these get across several encoder calls and try to make an estimate -That way we won't run our head into the wall each call */ -void Buffer_Realloc (JSONObjectEncoder *enc, size_t cbNeeded) -{ - size_t curSize = enc->end - enc->start; - size_t newSize = curSize * 2; - size_t offset = enc->offset - enc->start; - - while (newSize < curSize + cbNeeded) - { - newSize *= 2; - } - - if (enc->heap) - { - enc->start = (char *) enc->realloc (enc->start, newSize); - if (!enc->start) - { - SetError (NULL, enc, "Could not reserve memory block"); - return; - } - } - else - { - char *oldStart = enc->start; - enc->heap = 1; - enc->start = (char *) enc->malloc (newSize); - if (!enc->start) - { - SetError (NULL, enc, "Could not reserve memory block"); - return; - } - memcpy (enc->start, oldStart, offset); - } - enc->offset = enc->start + offset; - enc->end = enc->start + newSize; -} - -FASTCALL_ATTR INLINE_PREFIX void FASTCALL_MSVC Buffer_AppendShortHexUnchecked (char *outputOffset, unsigned short value) -{ - *(outputOffset++) = g_hexChars[(value & 0xf000) >> 12]; - *(outputOffset++) = g_hexChars[(value & 0x0f00) >> 8]; - *(outputOffset++) = g_hexChars[(value & 0x00f0) >> 4]; - *(outputOffset++) = g_hexChars[(value & 0x000f) >> 0]; -} - -int Buffer_EscapeStringUnvalidated (JSONObjectEncoder *enc, const char *io, const char *end) -{ - char *of = (char *) enc->offset; - - for (;;) - { - switch (*io) - { - case 0x00: - { - if (io < end) - { - *(of++) = '\\'; - *(of++) = 'u'; - *(of++) = '0'; - *(of++) = '0'; - *(of++) = '0'; - *(of++) = '0'; - break; - } - else - { - enc->offset += (of - enc->offset); - return TRUE; - } - } - case '\"': (*of++) = '\\'; (*of++) = '\"'; break; - case '\\': (*of++) = '\\'; (*of++) = '\\'; break; - case '\b': (*of++) = '\\'; (*of++) = 'b'; break; - case '\f': (*of++) = '\\'; (*of++) = 'f'; break; - case '\n': (*of++) = '\\'; (*of++) = 'n'; break; - case '\r': (*of++) = '\\'; (*of++) = 'r'; break; - case '\t': (*of++) = '\\'; (*of++) = 't'; break; - - case '/': - { - if (enc->escapeForwardSlashes) - { - (*of++) = '\\'; - (*of++) = '/'; - } - else - { - // Same as default case below. - (*of++) = (*io); - } - break; - } - case 0x26: // '&' - case 0x3c: // '<' - case 0x3e: // '>' - { - if (enc->encodeHTMLChars) - { - // Fall through to \u00XX case below. - } - else - { - // Same as default case below. - (*of++) = (*io); - break; - } - } - case 0x01: - case 0x02: - case 0x03: - case 0x04: - case 0x05: - case 0x06: - case 0x07: - case 0x0b: - case 0x0e: - case 0x0f: - case 0x10: - case 0x11: - case 0x12: - case 0x13: - case 0x14: - case 0x15: - case 0x16: - case 0x17: - case 0x18: - case 0x19: - case 0x1a: - case 0x1b: - case 0x1c: - case 0x1d: - case 0x1e: - case 0x1f: - { - *(of++) = '\\'; - *(of++) = 'u'; - *(of++) = '0'; - *(of++) = '0'; - *(of++) = g_hexChars[ (unsigned char) (((*io) & 0xf0) >> 4)]; - *(of++) = g_hexChars[ (unsigned char) ((*io) & 0x0f)]; - break; - } - default: (*of++) = (*io); break; - } - io++; - } -} - -int Buffer_EscapeStringValidated (JSOBJ obj, JSONObjectEncoder *enc, const char *io, const char *end) -{ - JSUTF32 ucs; - char *of = (char *) enc->offset; - - for (;;) - { - JSUINT8 utflen = g_asciiOutputTable[(unsigned char) *io]; - - switch (utflen) - { - case 0: - { - if (io < end) - { - *(of++) = '\\'; - *(of++) = 'u'; - *(of++) = '0'; - *(of++) = '0'; - *(of++) = '0'; - *(of++) = '0'; - io ++; - continue; - } - else - { - enc->offset += (of - enc->offset); - return TRUE; - } - } - - case 1: - { - *(of++)= (*io++); - continue; - } - - case 2: - { - JSUTF32 in; - JSUTF16 in16; - - if (end - io < 1) - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Unterminated UTF-8 sequence when encoding string"); - return FALSE; - } - - memcpy(&in16, io, sizeof(JSUTF16)); - in = (JSUTF32) in16; - -#ifdef __LITTLE_ENDIAN__ - ucs = ((in & 0x1f) << 6) | ((in >> 8) & 0x3f); -#else - ucs = ((in & 0x1f00) >> 2) | (in & 0x3f); -#endif - - if (ucs < 0x80) - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Overlong 2 byte UTF-8 sequence detected when encoding string"); - return FALSE; - } - - io += 2; - break; - } - - case 3: - { - JSUTF32 in; - JSUTF16 in16; - JSUINT8 in8; - - if (end - io < 2) - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Unterminated UTF-8 sequence when encoding string"); - return FALSE; - } - - memcpy(&in16, io, sizeof(JSUTF16)); - memcpy(&in8, io + 2, sizeof(JSUINT8)); -#ifdef __LITTLE_ENDIAN__ - in = (JSUTF32) in16; - in |= in8 << 16; - ucs = ((in & 0x0f) << 12) | ((in & 0x3f00) >> 2) | ((in & 0x3f0000) >> 16); -#else - in = in16 << 8; - in |= in8; - ucs = ((in & 0x0f0000) >> 4) | ((in & 0x3f00) >> 2) | (in & 0x3f); -#endif - - if (ucs < 0x800) - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Overlong 3 byte UTF-8 sequence detected when encoding string"); - return FALSE; - } - - io += 3; - break; - } - case 4: - { - JSUTF32 in; - - if (end - io < 3) - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Unterminated UTF-8 sequence when encoding string"); - return FALSE; - } - - memcpy(&in, io, sizeof(JSUTF32)); -#ifdef __LITTLE_ENDIAN__ - ucs = ((in & 0x07) << 18) | ((in & 0x3f00) << 4) | ((in & 0x3f0000) >> 10) | ((in & 0x3f000000) >> 24); -#else - ucs = ((in & 0x07000000) >> 6) | ((in & 0x3f0000) >> 4) | ((in & 0x3f00) >> 2) | (in & 0x3f); -#endif - if (ucs < 0x10000) - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Overlong 4 byte UTF-8 sequence detected when encoding string"); - return FALSE; - } - - io += 4; - break; - } - - - case 5: - case 6: - { - enc->offset += (of - enc->offset); - SetError (obj, enc, "Unsupported UTF-8 sequence length when encoding string"); - return FALSE; - } - - case 29: - { - if (enc->encodeHTMLChars) - { - // Fall through to \u00XX case 30 below. - } - else - { - // Same as case 1 above. - *(of++) = (*io++); - continue; - } - } - - case 30: - { - // \uXXXX encode - *(of++) = '\\'; - *(of++) = 'u'; - *(of++) = '0'; - *(of++) = '0'; - *(of++) = g_hexChars[ (unsigned char) (((*io) & 0xf0) >> 4)]; - *(of++) = g_hexChars[ (unsigned char) ((*io) & 0x0f)]; - io ++; - continue; - } - case 10: - case 12: - case 14: - case 16: - case 18: - case 20: - case 22: - { - *(of++) = *( (char *) (g_escapeChars + utflen + 0)); - *(of++) = *( (char *) (g_escapeChars + utflen + 1)); - io ++; - continue; - } - case 24: - { - if (enc->escapeForwardSlashes) - { - *(of++) = *( (char *) (g_escapeChars + utflen + 0)); - *(of++) = *( (char *) (g_escapeChars + utflen + 1)); - io ++; - } - else - { - // Same as case 1 above. - *(of++) = (*io++); - } - continue; - } - // This can never happen, it's here to make L4 VC++ happy - default: - { - ucs = 0; - break; - } - } - - /* - If the character is a UTF8 sequence of length > 1 we end up here */ - if (ucs >= 0x10000) - { - ucs -= 0x10000; - *(of++) = '\\'; - *(of++) = 'u'; - Buffer_AppendShortHexUnchecked(of, (unsigned short) (ucs >> 10) + 0xd800); - of += 4; - - *(of++) = '\\'; - *(of++) = 'u'; - Buffer_AppendShortHexUnchecked(of, (unsigned short) (ucs & 0x3ff) + 0xdc00); - of += 4; - } - else - { - *(of++) = '\\'; - *(of++) = 'u'; - Buffer_AppendShortHexUnchecked(of, (unsigned short) ucs); - of += 4; - } - } -} - -#define Buffer_Reserve(__enc, __len) \ - if ( (size_t) ((__enc)->end - (__enc)->offset) < (size_t) (__len)) \ - { \ - Buffer_Realloc((__enc), (__len));\ - } \ - - -#define Buffer_AppendCharUnchecked(__enc, __chr) \ - *((__enc)->offset++) = __chr; \ - -FASTCALL_ATTR INLINE_PREFIX void FASTCALL_MSVC strreverse(char* begin, char* end) -{ - char aux; - while (end > begin) - aux = *end, *end-- = *begin, *begin++ = aux; -} - -void Buffer_AppendIndentNewlineUnchecked(JSONObjectEncoder *enc) -{ - if (enc->indent > 0) Buffer_AppendCharUnchecked(enc, '\n'); -} - -void Buffer_AppendIndentUnchecked(JSONObjectEncoder *enc, JSINT32 value) -{ - int i; - if (enc->indent > 0) - while (value-- > 0) - for (i = 0; i < enc->indent; i++) - Buffer_AppendCharUnchecked(enc, ' '); -} - -void Buffer_AppendIntUnchecked(JSONObjectEncoder *enc, JSINT32 value) -{ - char* wstr; - JSUINT32 uvalue = (value < 0) ? -value : value; - - wstr = enc->offset; - // Conversion. Number is reversed. - - do *wstr++ = (char)(48 + (uvalue % 10)); while(uvalue /= 10); - if (value < 0) *wstr++ = '-'; - - // Reverse string - strreverse(enc->offset,wstr - 1); - enc->offset += (wstr - (enc->offset)); -} - -void Buffer_AppendLongUnchecked(JSONObjectEncoder *enc, JSINT64 value) -{ - char* wstr; - JSUINT64 uvalue = (value < 0) ? -value : value; - - wstr = enc->offset; - // Conversion. Number is reversed. - - do *wstr++ = (char)(48 + (uvalue % 10ULL)); while(uvalue /= 10ULL); - if (value < 0) *wstr++ = '-'; - - // Reverse string - strreverse(enc->offset,wstr - 1); - enc->offset += (wstr - (enc->offset)); -} - -void Buffer_AppendUnsignedLongUnchecked(JSONObjectEncoder *enc, JSUINT64 value) -{ - char* wstr; - JSUINT64 uvalue = value; - - wstr = enc->offset; - // Conversion. Number is reversed. - - do *wstr++ = (char)(48 + (uvalue % 10ULL)); while(uvalue /= 10ULL); - - // Reverse string - strreverse(enc->offset,wstr - 1); - enc->offset += (wstr - (enc->offset)); -} - -int Buffer_AppendDoubleUnchecked(JSOBJ obj, JSONObjectEncoder *enc, double value) -{ - /* if input is larger than thres_max, revert to exponential */ - const double thres_max = (double) 1e16 - 1; - int count; - double diff = 0.0; - char* str = enc->offset; - char* wstr = str; - unsigned long long whole; - double tmp; - unsigned long long frac; - int neg; - double pow10; - - if (value == HUGE_VAL || value == -HUGE_VAL) - { - SetError (obj, enc, "Invalid Inf value when encoding double"); - return FALSE; - } - - if (!(value == value)) - { - SetError (obj, enc, "Invalid Nan value when encoding double"); - return FALSE; - } - - /* we'll work in positive values and deal with the - negative sign issue later */ - neg = 0; - if (value < 0) - { - neg = 1; - value = -value; - } - - pow10 = g_pow10[enc->doublePrecision]; - - whole = (unsigned long long) value; - tmp = (value - whole) * pow10; - frac = (unsigned long long)(tmp); - diff = tmp - frac; - - if (diff > 0.5) - { - ++frac; - /* handle rollover, e.g. case 0.99 with prec 1 is 1.0 */ - if (frac >= pow10) - { - frac = 0; - ++whole; - } - } - else - if (diff == 0.5 && ((frac == 0) || (frac & 1))) - { - /* if halfway, round up if odd, OR - if last digit is 0. That last part is strange */ - ++frac; - } - - /* for very large numbers switch back to native sprintf for exponentials. - anyone want to write code to replace this? */ - /* - normal printf behavior is to print EVERY whole number digit - which can be 100s of characters overflowing your buffers == bad - */ - if (value > thres_max) - { - enc->offset += snprintf(str, enc->end - enc->offset, "%.15e", neg ? -value : value); - return TRUE; - } - - if (enc->doublePrecision == 0) - { - diff = value - whole; - - if (diff > 0.5) - { - /* greater than 0.5, round up, e.g. 1.6 -> 2 */ - ++whole; - } - else - if (diff == 0.5 && (whole & 1)) - { - /* exactly 0.5 and ODD, then round up */ - /* 1.5 -> 2, but 2.5 -> 2 */ - ++whole; - } - - //vvvvvvvvvvvvvvvvvvv Diff from modp_dto2 - } - else - if (frac) - { - count = enc->doublePrecision; - // now do fractional part, as an unsigned number - // we know it is not 0 but we can have leading zeros, these - // should be removed - while (!(frac % 10)) - { - --count; - frac /= 10; - } - //^^^^^^^^^^^^^^^^^^^ Diff from modp_dto2 - - // now do fractional part, as an unsigned number - do - { - --count; - *wstr++ = (char)(48 + (frac % 10)); - } while (frac /= 10); - // add extra 0s - while (count-- > 0) - { - *wstr++ = '0'; - } - // add decimal - *wstr++ = '.'; - } - else - { - *wstr++ = '0'; - *wstr++ = '.'; - } - - // do whole part - // Take care of sign - // Conversion. Number is reversed. - do *wstr++ = (char)(48 + (whole % 10)); while (whole /= 10); - - if (neg) - { - *wstr++ = '-'; - } - strreverse(str, wstr-1); - enc->offset += (wstr - (enc->offset)); - - return TRUE; -} - -/* -FIXME: -Handle integration functions returning NULL here */ - -/* -FIXME: -Perhaps implement recursion detection */ - -void encode(JSOBJ obj, JSONObjectEncoder *enc, const char *name, size_t cbName) -{ - const char *value; - char *objName; - int count; - JSOBJ iterObj; - size_t szlen; - JSONTypeContext tc; - - if (enc->level > enc->recursionMax) - { - SetError (obj, enc, "Maximum recursion level reached"); - return; - } - - /* - This reservation must hold - - length of _name as encoded worst case + - maxLength of double to string OR maxLength of JSLONG to string - */ - - Buffer_Reserve(enc, 256 + RESERVE_STRING(cbName)); - if (enc->errorMsg) - { - return; - } - - if (name) - { - Buffer_AppendCharUnchecked(enc, '\"'); - - if (enc->forceASCII) - { - if (!Buffer_EscapeStringValidated(obj, enc, name, name + cbName)) - { - return; - } - } - else - { - if (!Buffer_EscapeStringUnvalidated(enc, name, name + cbName)) - { - return; - } - } - - Buffer_AppendCharUnchecked(enc, '\"'); - - Buffer_AppendCharUnchecked (enc, ':'); -#ifdef JSON_NO_EXTRA_WHITESPACE - if (enc->indent) - { - Buffer_AppendCharUnchecked (enc, ' '); - } -#else - Buffer_AppendCharUnchecked (enc, ' '); -#endif - } - - tc.encoder_prv = enc->prv; - enc->beginTypeContext(obj, &tc, enc); - - switch (tc.type) - { - case JT_INVALID: - { - return; - } - - case JT_ARRAY: - { - count = 0; - - Buffer_AppendCharUnchecked (enc, '['); - Buffer_AppendIndentNewlineUnchecked (enc); - - while (enc->iterNext(obj, &tc)) - { - if (count > 0) - { - Buffer_AppendCharUnchecked (enc, ','); -#ifndef JSON_NO_EXTRA_WHITESPACE - Buffer_AppendCharUnchecked (buffer, ' '); -#endif - Buffer_AppendIndentNewlineUnchecked (enc); - } - - iterObj = enc->iterGetValue(obj, &tc); - - enc->level ++; - Buffer_AppendIndentUnchecked (enc, enc->level); - encode (iterObj, enc, NULL, 0); - count ++; - } - - enc->iterEnd(obj, &tc); - Buffer_AppendIndentNewlineUnchecked (enc); - Buffer_AppendIndentUnchecked (enc, enc->level); - Buffer_AppendCharUnchecked (enc, ']'); - break; - } - - case JT_OBJECT: - { - count = 0; - - Buffer_AppendCharUnchecked (enc, '{'); - Buffer_AppendIndentNewlineUnchecked (enc); - - while (enc->iterNext(obj, &tc)) - { - if (count > 0) - { - Buffer_AppendCharUnchecked (enc, ','); -#ifndef JSON_NO_EXTRA_WHITESPACE - Buffer_AppendCharUnchecked (enc, ' '); -#endif - Buffer_AppendIndentNewlineUnchecked (enc); - } - - iterObj = enc->iterGetValue(obj, &tc); - objName = enc->iterGetName(obj, &tc, &szlen); - - enc->level ++; - Buffer_AppendIndentUnchecked (enc, enc->level); - encode (iterObj, enc, objName, szlen); - count ++; - } - - enc->iterEnd(obj, &tc); - Buffer_AppendIndentNewlineUnchecked (enc); - Buffer_AppendIndentUnchecked (enc, enc->level); - Buffer_AppendCharUnchecked (enc, '}'); - break; - } - - case JT_LONG: - { - Buffer_AppendLongUnchecked (enc, enc->getLongValue(obj, &tc)); - break; - } - - case JT_ULONG: - { - Buffer_AppendUnsignedLongUnchecked (enc, enc->getUnsignedLongValue(obj, &tc)); - break; - } - - case JT_INT: - { - Buffer_AppendIntUnchecked (enc, enc->getIntValue(obj, &tc)); - break; - } - - case JT_TRUE: - { - Buffer_AppendCharUnchecked (enc, 't'); - Buffer_AppendCharUnchecked (enc, 'r'); - Buffer_AppendCharUnchecked (enc, 'u'); - Buffer_AppendCharUnchecked (enc, 'e'); - break; - } - - case JT_FALSE: - { - Buffer_AppendCharUnchecked (enc, 'f'); - Buffer_AppendCharUnchecked (enc, 'a'); - Buffer_AppendCharUnchecked (enc, 'l'); - Buffer_AppendCharUnchecked (enc, 's'); - Buffer_AppendCharUnchecked (enc, 'e'); - break; - } - - - case JT_NULL: - { - Buffer_AppendCharUnchecked (enc, 'n'); - Buffer_AppendCharUnchecked (enc, 'u'); - Buffer_AppendCharUnchecked (enc, 'l'); - Buffer_AppendCharUnchecked (enc, 'l'); - break; - } - - case JT_DOUBLE: - { - if (!Buffer_AppendDoubleUnchecked (obj, enc, enc->getDoubleValue(obj, &tc))) - { - enc->endTypeContext(obj, &tc); - enc->level --; - return; - } - break; - } - - case JT_UTF8: - { - value = enc->getStringValue(obj, &tc, &szlen); - if(!value) - { - SetError(obj, enc, "utf-8 encoding error"); - return; - } - - Buffer_Reserve(enc, RESERVE_STRING(szlen)); - if (enc->errorMsg) - { - enc->endTypeContext(obj, &tc); - return; - } - Buffer_AppendCharUnchecked (enc, '\"'); - - if (enc->forceASCII) - { - if (!Buffer_EscapeStringValidated(obj, enc, value, value + szlen)) - { - enc->endTypeContext(obj, &tc); - enc->level --; - return; - } - } - else - { - if (!Buffer_EscapeStringUnvalidated(enc, value, value + szlen)) - { - enc->endTypeContext(obj, &tc); - enc->level --; - return; - } - } - - Buffer_AppendCharUnchecked (enc, '\"'); - break; - } - - case JT_RAW: - { - value = enc->getStringValue(obj, &tc, &szlen); - if(!value) - { - SetError(obj, enc, "utf-8 encoding error"); - return; - } - - Buffer_Reserve(enc, RESERVE_STRING(szlen)); - if (enc->errorMsg) - { - enc->endTypeContext(obj, &tc); - return; - } - - memcpy(enc->offset, value, szlen); - enc->offset += szlen; - - break; - } - } - - enc->endTypeContext(obj, &tc); - enc->level --; -} - -char *JSON_EncodeObject(JSOBJ obj, JSONObjectEncoder *enc, char *_buffer, size_t _cbBuffer) -{ - enc->malloc = enc->malloc ? enc->malloc : malloc; - enc->free = enc->free ? enc->free : free; - enc->realloc = enc->realloc ? enc->realloc : realloc; - enc->errorMsg = NULL; - enc->errorObj = NULL; - enc->level = 0; - - if (enc->recursionMax < 1) - { - enc->recursionMax = JSON_MAX_RECURSION_DEPTH; - } - - if (enc->doublePrecision < 0 || - enc->doublePrecision > JSON_DOUBLE_MAX_DECIMALS) - { - enc->doublePrecision = JSON_DOUBLE_MAX_DECIMALS; - } - - if (_buffer == NULL) - { - _cbBuffer = 32768; - enc->start = (char *) enc->malloc (_cbBuffer); - if (!enc->start) - { - SetError(obj, enc, "Could not reserve memory block"); - return NULL; - } - enc->heap = 1; - } - else - { - enc->start = _buffer; - enc->heap = 0; - } - - enc->end = enc->start + _cbBuffer; - enc->offset = enc->start; - - encode (obj, enc, NULL, 0); - - Buffer_Reserve(enc, 1); - if (enc->errorMsg) - { - return NULL; - } - Buffer_AppendCharUnchecked(enc, '\0'); - - return enc->start; -} diff --git a/contrib/python/ujson/py2/python/JSONtoObj.c b/contrib/python/ujson/py2/python/JSONtoObj.c deleted file mode 100644 index 79d9f1af6e7..00000000000 --- a/contrib/python/ujson/py2/python/JSONtoObj.c +++ /dev/null @@ -1,252 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms - * Copyright (c) 1988-1993 The Regents of the University of California. - * Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#include "py_defines.h" -#include <ultrajson.h> - - -//#define PRINTMARK() fprintf(stderr, "%s: MARK(%d)\n", __FILE__, __LINE__) -#define PRINTMARK() - -void Object_objectAddKey(void *prv, JSOBJ obj, JSOBJ name, JSOBJ value) -{ - PyDict_SetItem (obj, name, value); - Py_DECREF( (PyObject *) name); - Py_DECREF( (PyObject *) value); - return; -} - -void Object_arrayAddItem(void *prv, JSOBJ obj, JSOBJ value) -{ - PyList_Append(obj, value); - Py_DECREF( (PyObject *) value); - return; -} - -JSOBJ Object_newString(void *prv, wchar_t *start, wchar_t *end) -{ - return PyUnicode_FromWideChar (start, (end - start)); -} - -JSOBJ Object_newTrue(void *prv) -{ - Py_RETURN_TRUE; -} - -JSOBJ Object_newFalse(void *prv) -{ - Py_RETURN_FALSE; -} - -JSOBJ Object_newNull(void *prv) -{ - Py_RETURN_NONE; -} - -JSOBJ Object_newObject(void *prv) -{ - return PyDict_New(); -} - -JSOBJ Object_newArray(void *prv) -{ - return PyList_New(0); -} - -JSOBJ Object_newInteger(void *prv, JSINT32 value) -{ - return PyInt_FromLong( (long) value); -} - -JSOBJ Object_newLong(void *prv, JSINT64 value) -{ - return PyLong_FromLongLong (value); -} - -JSOBJ Object_newUnsignedLong(void *prv, JSUINT64 value) -{ - return PyLong_FromUnsignedLongLong (value); -} - -JSOBJ Object_newDouble(void *prv, double value) -{ - return PyFloat_FromDouble(value); -} - -static void Object_releaseObject(void *prv, JSOBJ obj) -{ - Py_DECREF( ((PyObject *)obj)); -} - -static char *g_kwlist[] = {"obj", "precise_float", NULL}; - -PyObject* JSONToObj(PyObject* self, PyObject *args, PyObject *kwargs) -{ - PyObject *ret; - PyObject *sarg; - PyObject *arg; - PyObject *opreciseFloat = NULL; - JSONObjectDecoder decoder = - { - Object_newString, - Object_objectAddKey, - Object_arrayAddItem, - Object_newTrue, - Object_newFalse, - Object_newNull, - Object_newObject, - Object_newArray, - Object_newInteger, - Object_newLong, - Object_newUnsignedLong, - Object_newDouble, - Object_releaseObject, - PyObject_Malloc, - PyObject_Free, - PyObject_Realloc - }; - - decoder.preciseFloat = 0; - decoder.prv = NULL; - - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|O", g_kwlist, &arg, &opreciseFloat)) - { - return NULL; - } - - if (opreciseFloat && PyObject_IsTrue(opreciseFloat)) - { - decoder.preciseFloat = 1; - } - - if (PyString_Check(arg)) - { - sarg = arg; - } - else - if (PyUnicode_Check(arg)) - { - sarg = PyUnicode_AsUTF8String(arg); - if (sarg == NULL) - { - //Exception raised above us by codec according to docs - return NULL; - } - } - else - { - PyErr_Format(PyExc_TypeError, "Expected String or Unicode"); - return NULL; - } - - decoder.errorStr = NULL; - decoder.errorOffset = NULL; - - ret = JSON_DecodeObject(&decoder, PyString_AS_STRING(sarg), PyString_GET_SIZE(sarg)); - - if (sarg != arg) - { - Py_DECREF(sarg); - } - - if (decoder.errorStr) - { - /* - FIXME: It's possible to give a much nicer error message here with actual failing element in input etc*/ - - PyErr_Format (PyExc_ValueError, "%s", decoder.errorStr); - - if (ret) - { - Py_DECREF( (PyObject *) ret); - } - - return NULL; - } - - return ret; -} - -PyObject* JSONFileToObj(PyObject* self, PyObject *args, PyObject *kwargs) -{ - PyObject *read; - PyObject *string; - PyObject *result; - PyObject *file = NULL; - PyObject *argtuple; - - if (!PyArg_ParseTuple (args, "O", &file)) - { - return NULL; - } - - if (!PyObject_HasAttrString (file, "read")) - { - PyErr_Format (PyExc_TypeError, "expected file"); - return NULL; - } - - read = PyObject_GetAttrString (file, "read"); - - if (!PyCallable_Check (read)) { - Py_XDECREF(read); - PyErr_Format (PyExc_TypeError, "expected file"); - return NULL; - } - - string = PyObject_CallObject (read, NULL); - Py_XDECREF(read); - - if (string == NULL) - { - return NULL; - } - - argtuple = PyTuple_Pack(1, string); - - result = JSONToObj (self, argtuple, kwargs); - - Py_XDECREF(argtuple); - Py_XDECREF(string); - - if (result == NULL) { - return NULL; - } - - return result; -} diff --git a/contrib/python/ujson/py2/python/objToJSON.c b/contrib/python/ujson/py2/python/objToJSON.c deleted file mode 100644 index b3a821e7aeb..00000000000 --- a/contrib/python/ujson/py2/python/objToJSON.c +++ /dev/null @@ -1,1168 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms -* Copyright (c) 1988-1993 The Regents of the University of California. -* Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#include "py_defines.h" -#include <stdio.h> -#include <datetime.h> -#include <ultrajson.h> - -#define EPOCH_ORD 719163 -static PyObject* type_decimal = NULL; - -typedef void *(*PFN_PyTypeToJSON)(JSOBJ obj, JSONTypeContext *ti, void *outValue, size_t *_outLen); - -#if (PY_VERSION_HEX < 0x02050000) -typedef ssize_t Py_ssize_t; -#endif - -typedef struct __TypeContext -{ - JSPFN_ITEREND iterEnd; - JSPFN_ITERNEXT iterNext; - JSPFN_ITERGETNAME iterGetName; - JSPFN_ITERGETVALUE iterGetValue; - PFN_PyTypeToJSON PyTypeToJSON; - PyObject *newObj; - PyObject *dictObj; - Py_ssize_t index; - Py_ssize_t size; - PyObject *itemValue; - PyObject *itemName; - PyObject *attrList; - PyObject *iterator; - - union - { - PyObject *rawJSONValue; - JSINT64 longValue; - JSUINT64 unsignedLongValue; - }; -} TypeContext; - -#define GET_TC(__ptrtc) ((TypeContext *)((__ptrtc)->prv)) - -struct PyDictIterState -{ - PyObject *keys; - size_t i; - size_t sz; -}; - -//#define PRINTMARK() fprintf(stderr, "%s: MARK(%d)\n", __FILE__, __LINE__) -#define PRINTMARK() - -void initObjToJSON(void) -{ - PyObject* mod_decimal = PyImport_ImportModule("decimal"); - if (mod_decimal) - { - type_decimal = PyObject_GetAttrString(mod_decimal, "Decimal"); - Py_INCREF(type_decimal); - Py_DECREF(mod_decimal); - } - else - PyErr_Clear(); - - PyDateTime_IMPORT; -} - -#ifdef _LP64 -static void *PyIntToINT64(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - *((JSINT64 *) outValue) = PyInt_AS_LONG (obj); - return NULL; -} -#else -static void *PyIntToINT32(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - *((JSINT32 *) outValue) = PyInt_AS_LONG (obj); - return NULL; -} -#endif - -static void *PyLongToINT64(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - *((JSINT64 *) outValue) = GET_TC(tc)->longValue; - return NULL; -} - -static void *PyLongToUINT64(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - *((JSUINT64 *) outValue) = GET_TC(tc)->unsignedLongValue; - return NULL; -} - -static void *PyFloatToDOUBLE(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - *((double *) outValue) = PyFloat_AsDouble (obj); - return NULL; -} - -static void *PyStringToUTF8(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - *_outLen = PyString_GET_SIZE(obj); - return PyString_AS_STRING(obj); -} - -static void *PyUnicodeToUTF8(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - PyObject *newObj; -#if (PY_VERSION_HEX >= 0x03030000) - if(PyUnicode_IS_COMPACT_ASCII(obj)) - { - Py_ssize_t len; - char *data = PyUnicode_AsUTF8AndSize(obj, &len); - *_outLen = len; - return data; - } -#endif - newObj = PyUnicode_AsUTF8String(obj); - if(!newObj) - { - return NULL; - } - - GET_TC(tc)->newObj = newObj; - - *_outLen = PyString_GET_SIZE(newObj); - return PyString_AS_STRING(newObj); -} - -static void *PyRawJSONToUTF8(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = GET_TC(tc)->rawJSONValue; - if (PyUnicode_Check(obj)) { - return PyUnicodeToUTF8(obj, tc, outValue, _outLen); - } - else { - return PyStringToUTF8(obj, tc, outValue, _outLen); - } -} - -static void *PyDateTimeToINT64(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - PyObject *date, *ord, *utcoffset; - int y, m, d, h, mn, s, days; - - utcoffset = PyObject_CallMethod(obj, "utcoffset", NULL); - if(utcoffset != Py_None){ - obj = PyNumber_Subtract(obj, utcoffset); - } - - y = PyDateTime_GET_YEAR(obj); - m = PyDateTime_GET_MONTH(obj); - d = PyDateTime_GET_DAY(obj); - h = PyDateTime_DATE_GET_HOUR(obj); - mn = PyDateTime_DATE_GET_MINUTE(obj); - s = PyDateTime_DATE_GET_SECOND(obj); - - date = PyDate_FromDate(y, m, 1); - ord = PyObject_CallMethod(date, "toordinal", NULL); - days = PyInt_AS_LONG(ord) - EPOCH_ORD + d - 1; - Py_DECREF(date); - Py_DECREF(ord); - *( (JSINT64 *) outValue) = (((JSINT64) ((days * 24 + h) * 60 + mn)) * 60 + s); - return NULL; -} - -static void *PyDateToINT64(JSOBJ _obj, JSONTypeContext *tc, void *outValue, size_t *_outLen) -{ - PyObject *obj = (PyObject *) _obj; - PyObject *date, *ord; - int y, m, d, days; - - y = PyDateTime_GET_YEAR(obj); - m = PyDateTime_GET_MONTH(obj); - d = PyDateTime_GET_DAY(obj); - - date = PyDate_FromDate(y, m, 1); - ord = PyObject_CallMethod(date, "toordinal", NULL); - days = PyInt_AS_LONG(ord) - EPOCH_ORD + d - 1; - Py_DECREF(date); - Py_DECREF(ord); - *( (JSINT64 *) outValue) = ((JSINT64) days * 86400); - - return NULL; -} - -int Tuple_iterNext(JSOBJ obj, JSONTypeContext *tc) -{ - PyObject *item; - - if (GET_TC(tc)->index >= GET_TC(tc)->size) - { - return 0; - } - - item = PyTuple_GET_ITEM (obj, GET_TC(tc)->index); - - GET_TC(tc)->itemValue = item; - GET_TC(tc)->index ++; - return 1; -} - -void Tuple_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ -} - -JSOBJ Tuple_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->itemValue; -} - -char *Tuple_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - return NULL; -} - -int Iter_iterNext(JSOBJ obj, JSONTypeContext *tc) -{ - PyObject *item; - - if (GET_TC(tc)->itemValue) - { - Py_DECREF(GET_TC(tc)->itemValue); - GET_TC(tc)->itemValue = NULL; - } - - if (GET_TC(tc)->iterator == NULL) - { - return 0; - } - - item = PyIter_Next(GET_TC(tc)->iterator); - - if (item == NULL) - { - return 0; - } - - GET_TC(tc)->itemValue = item; - return 1; -} - -void Iter_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ - if (GET_TC(tc)->itemValue) - { - Py_DECREF(GET_TC(tc)->itemValue); - GET_TC(tc)->itemValue = NULL; - } - - if (GET_TC(tc)->iterator) - { - Py_DECREF(GET_TC(tc)->iterator); - GET_TC(tc)->iterator = NULL; - } -} - -JSOBJ Iter_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->itemValue; -} - -char *Iter_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - return NULL; -} - -void Dir_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ - if (GET_TC(tc)->itemValue) - { - Py_DECREF(GET_TC(tc)->itemValue); - GET_TC(tc)->itemValue = NULL; - } - - if (GET_TC(tc)->itemName) - { - Py_DECREF(GET_TC(tc)->itemName); - GET_TC(tc)->itemName = NULL; - } - - Py_DECREF( (PyObject *) GET_TC(tc)->attrList); - PRINTMARK(); -} - -int Dir_iterNext(JSOBJ _obj, JSONTypeContext *tc) -{ - PyObject *obj = (PyObject *) _obj; - PyObject *itemValue = GET_TC(tc)->itemValue; - PyObject *itemName = GET_TC(tc)->itemName; - PyObject* attr; - PyObject* attrName; - char* attrStr; - - if (itemValue) - { - Py_DECREF(GET_TC(tc)->itemValue); - GET_TC(tc)->itemValue = itemValue = NULL; - } - - if (itemName) - { - Py_DECREF(GET_TC(tc)->itemName); - GET_TC(tc)->itemName = itemName = NULL; - } - - for (; GET_TC(tc)->index < GET_TC(tc)->size; GET_TC(tc)->index ++) - { - attrName = PyList_GET_ITEM(GET_TC(tc)->attrList, GET_TC(tc)->index); -#if PY_MAJOR_VERSION >= 3 - attr = PyUnicode_AsUTF8String(attrName); -#else - attr = attrName; - Py_INCREF(attr); -#endif - attrStr = PyString_AS_STRING(attr); - - if (attrStr[0] == '_') - { - PRINTMARK(); - Py_DECREF(attr); - continue; - } - - itemValue = PyObject_GetAttr(obj, attrName); - if (itemValue == NULL) - { - PyErr_Clear(); - Py_DECREF(attr); - PRINTMARK(); - continue; - } - - if (PyCallable_Check(itemValue)) - { - Py_DECREF(itemValue); - Py_DECREF(attr); - PRINTMARK(); - continue; - } - - PRINTMARK(); - itemName = attr; - break; - } - - if (itemName == NULL) - { - GET_TC(tc)->index = GET_TC(tc)->size; - GET_TC(tc)->itemValue = NULL; - return 0; - } - - GET_TC(tc)->itemName = itemName; - GET_TC(tc)->itemValue = itemValue; - GET_TC(tc)->index ++; - - PRINTMARK(); - return 1; -} - -JSOBJ Dir_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - PRINTMARK(); - return GET_TC(tc)->itemValue; -} - -char *Dir_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - PRINTMARK(); - *outLen = PyString_GET_SIZE(GET_TC(tc)->itemName); - return PyString_AS_STRING(GET_TC(tc)->itemName); -} - -int List_iterNext(JSOBJ obj, JSONTypeContext *tc) -{ - if (GET_TC(tc)->index >= GET_TC(tc)->size) - { - PRINTMARK(); - return 0; - } - - GET_TC(tc)->itemValue = PyList_GET_ITEM (obj, GET_TC(tc)->index); - GET_TC(tc)->index ++; - return 1; -} - -void List_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ -} - -JSOBJ List_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->itemValue; -} - -char *List_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - return NULL; -} - -//============================================================================= -// Dict iteration functions -// itemName might converted to string (Python_Str). Do refCounting -// itemValue is borrowed from object (which is dict). No refCounting -//============================================================================= - -int Dict_iterNext(JSOBJ obj, JSONTypeContext *tc) -{ -#if PY_MAJOR_VERSION >= 3 - PyObject* itemNameTmp; -#endif - - if (GET_TC(tc)->itemName) - { - Py_DECREF(GET_TC(tc)->itemName); - GET_TC(tc)->itemName = NULL; - } - - - if (!PyDict_Next ( (PyObject *)GET_TC(tc)->dictObj, &GET_TC(tc)->index, &GET_TC(tc)->itemName, &GET_TC(tc)->itemValue)) - { - PRINTMARK(); - return 0; - } - - if (PyUnicode_Check(GET_TC(tc)->itemName)) - { - GET_TC(tc)->itemName = PyUnicode_AsUTF8String (GET_TC(tc)->itemName); - } - else - if (!PyString_Check(GET_TC(tc)->itemName)) - { - GET_TC(tc)->itemName = PyObject_Str(GET_TC(tc)->itemName); -#if PY_MAJOR_VERSION >= 3 - itemNameTmp = GET_TC(tc)->itemName; - GET_TC(tc)->itemName = PyUnicode_AsUTF8String (GET_TC(tc)->itemName); - Py_DECREF(itemNameTmp); -#endif - } - else - { - Py_INCREF(GET_TC(tc)->itemName); - } - PRINTMARK(); - return 1; -} - -void Dict_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ - if (GET_TC(tc)->itemName) - { - Py_DECREF(GET_TC(tc)->itemName); - GET_TC(tc)->itemName = NULL; - } - Py_DECREF(GET_TC(tc)->dictObj); - PRINTMARK(); -} - -JSOBJ Dict_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->itemValue; -} - -char *Dict_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - *outLen = PyString_GET_SIZE(GET_TC(tc)->itemName); - return PyString_AS_STRING(GET_TC(tc)->itemName); -} - -int SortedDict_iterNext(JSOBJ obj, JSONTypeContext *tc) -{ - PyObject *items = NULL, *item = NULL, *key = NULL, *value = NULL; - Py_ssize_t i, nitems; -#if PY_MAJOR_VERSION >= 3 - PyObject* keyTmp; -#endif - - // Upon first call, obtain a list of the keys and sort them. This follows the same logic as the - // stanard library's _json.c sort_keys handler. - if (GET_TC(tc)->newObj == NULL) - { - // Obtain the list of keys from the dictionary. - items = PyMapping_Keys(GET_TC(tc)->dictObj); - if (items == NULL) - { - goto error; - } - else if (!PyList_Check(items)) - { - PyErr_SetString(PyExc_ValueError, "keys must return list"); - goto error; - } - - // Sort the list. - if (PyList_Sort(items) < 0) - { - goto error; - } - - // Obtain the value for each key, and pack a list of (key, value) 2-tuples. - nitems = PyList_GET_SIZE(items); - for (i = 0; i < nitems; i++) - { - key = PyList_GET_ITEM(items, i); - value = PyDict_GetItem(GET_TC(tc)->dictObj, key); - - // Subject the key to the same type restrictions and conversions as in Dict_iterGetValue. - if (PyUnicode_Check(key)) - { - key = PyUnicode_AsUTF8String(key); - } - else if (!PyString_Check(key)) - { - key = PyObject_Str(key); -#if PY_MAJOR_VERSION >= 3 - keyTmp = key; - key = PyUnicode_AsUTF8String(key); - Py_DECREF(keyTmp); -#endif - } - else - { - Py_INCREF(key); - } - - item = PyTuple_Pack(2, key, value); - if (item == NULL) - { - goto error; - } - if (PyList_SetItem(items, i, item)) - { - goto error; - } - Py_DECREF(key); - } - - // Store the sorted list of tuples in the newObj slot. - GET_TC(tc)->newObj = items; - GET_TC(tc)->size = nitems; - } - - if (GET_TC(tc)->index >= GET_TC(tc)->size) - { - PRINTMARK(); - return 0; - } - - item = PyList_GET_ITEM(GET_TC(tc)->newObj, GET_TC(tc)->index); - GET_TC(tc)->itemName = PyTuple_GET_ITEM(item, 0); - GET_TC(tc)->itemValue = PyTuple_GET_ITEM(item, 1); - GET_TC(tc)->index++; - return 1; - -error: - Py_XDECREF(item); - Py_XDECREF(key); - Py_XDECREF(value); - Py_XDECREF(items); - return -1; -} - -void SortedDict_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ - GET_TC(tc)->itemName = NULL; - GET_TC(tc)->itemValue = NULL; - Py_DECREF(GET_TC(tc)->dictObj); - PRINTMARK(); -} - -JSOBJ SortedDict_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->itemValue; -} - -char *SortedDict_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - *outLen = PyString_GET_SIZE(GET_TC(tc)->itemName); - return PyString_AS_STRING(GET_TC(tc)->itemName); -} - - -void SetupDictIter(PyObject *dictObj, TypeContext *pc, JSONObjectEncoder *enc) -{ - if (enc->sortKeys) { - pc->iterEnd = SortedDict_iterEnd; - pc->iterNext = SortedDict_iterNext; - pc->iterGetValue = SortedDict_iterGetValue; - pc->iterGetName = SortedDict_iterGetName; - } - else { - pc->iterEnd = Dict_iterEnd; - pc->iterNext = Dict_iterNext; - pc->iterGetValue = Dict_iterGetValue; - pc->iterGetName = Dict_iterGetName; - } - pc->dictObj = dictObj; - pc->index = 0; -} - -void Object_beginTypeContext (JSOBJ _obj, JSONTypeContext *tc, JSONObjectEncoder *enc) -{ - PyObject *obj, *exc, *iter; - TypeContext *pc; - PRINTMARK(); - if (!_obj) { - tc->type = JT_INVALID; - return; - } - - obj = (PyObject*) _obj; - - tc->prv = PyObject_Malloc(sizeof(TypeContext)); - pc = (TypeContext *) tc->prv; - if (!pc) - { - tc->type = JT_INVALID; - PyErr_NoMemory(); - return; - } - pc->newObj = NULL; - pc->dictObj = NULL; - pc->itemValue = NULL; - pc->itemName = NULL; - pc->iterator = NULL; - pc->attrList = NULL; - pc->index = 0; - pc->size = 0; - pc->longValue = 0; - pc->rawJSONValue = NULL; - - if (PyIter_Check(obj)) - { - PRINTMARK(); - goto ISITERABLE; - } - - if (PyBool_Check(obj)) - { - PRINTMARK(); - tc->type = (obj == Py_True) ? JT_TRUE : JT_FALSE; - return; - } - else - if (PyLong_Check(obj)) - { - PRINTMARK(); - pc->PyTypeToJSON = PyLongToINT64; - tc->type = JT_LONG; - GET_TC(tc)->longValue = PyLong_AsLongLong(obj); - - exc = PyErr_Occurred(); - if (!exc) - { - return; - } - - if (exc && PyErr_ExceptionMatches(PyExc_OverflowError)) - { - PyErr_Clear(); - pc->PyTypeToJSON = PyLongToUINT64; - tc->type = JT_ULONG; - GET_TC(tc)->unsignedLongValue = PyLong_AsUnsignedLongLong(obj); - - exc = PyErr_Occurred(); - if (exc && PyErr_ExceptionMatches(PyExc_OverflowError)) - { - PRINTMARK(); - goto INVALID; - } - } - - return; - } - else - if (PyInt_Check(obj)) - { - PRINTMARK(); -#ifdef _LP64 - pc->PyTypeToJSON = PyIntToINT64; tc->type = JT_LONG; -#else - pc->PyTypeToJSON = PyIntToINT32; tc->type = JT_INT; -#endif - return; - } - else - if (PyString_Check(obj)) - { - PRINTMARK(); - pc->PyTypeToJSON = PyStringToUTF8; tc->type = JT_UTF8; - return; - } - else - if (PyUnicode_Check(obj)) - { - PRINTMARK(); - pc->PyTypeToJSON = PyUnicodeToUTF8; tc->type = JT_UTF8; - return; - } - else - if (PyFloat_Check(obj) || (type_decimal && PyObject_IsInstance(obj, type_decimal))) - { - PRINTMARK(); - pc->PyTypeToJSON = PyFloatToDOUBLE; tc->type = JT_DOUBLE; - return; - } - else - if (PyDateTime_Check(obj)) - { - PRINTMARK(); - pc->PyTypeToJSON = PyDateTimeToINT64; tc->type = JT_LONG; - return; - } - else - if (PyDate_Check(obj)) - { - PRINTMARK(); - pc->PyTypeToJSON = PyDateToINT64; tc->type = JT_LONG; - return; - } - else - if (obj == Py_None) - { - PRINTMARK(); - tc->type = JT_NULL; - return; - } - -ISITERABLE: - if (PyDict_Check(obj)) - { - PRINTMARK(); - tc->type = JT_OBJECT; - SetupDictIter(obj, pc, enc); - Py_INCREF(obj); - return; - } - else - if (PyList_Check(obj)) - { - PRINTMARK(); - tc->type = JT_ARRAY; - pc->iterEnd = List_iterEnd; - pc->iterNext = List_iterNext; - pc->iterGetValue = List_iterGetValue; - pc->iterGetName = List_iterGetName; - GET_TC(tc)->index = 0; - GET_TC(tc)->size = PyList_GET_SIZE( (PyObject *) obj); - return; - } - else - if (PyTuple_Check(obj)) - { - PRINTMARK(); - tc->type = JT_ARRAY; - pc->iterEnd = Tuple_iterEnd; - pc->iterNext = Tuple_iterNext; - pc->iterGetValue = Tuple_iterGetValue; - pc->iterGetName = Tuple_iterGetName; - GET_TC(tc)->index = 0; - GET_TC(tc)->size = PyTuple_GET_SIZE( (PyObject *) obj); - GET_TC(tc)->itemValue = NULL; - - return; - } - /* - else - if (PyAnySet_Check(obj)) - { - PRINTMARK(); - tc->type = JT_ARRAY; - pc->iterBegin = NULL; - pc->iterEnd = Iter_iterEnd; - pc->iterNext = Iter_iterNext; - pc->iterGetValue = Iter_iterGetValue; - pc->iterGetName = Iter_iterGetName; - return; - } - */ - - if (PyObject_HasAttrString(obj, "toDict")) - { - PyObject* toDictFunc = PyObject_GetAttrString(obj, "toDict"); - PyObject* tuple = PyTuple_New(0); - PyObject* toDictResult = PyObject_Call(toDictFunc, tuple, NULL); - Py_DECREF(tuple); - Py_DECREF(toDictFunc); - - if (toDictResult == NULL) - { - goto INVALID; - } - - if (!PyDict_Check(toDictResult)) - { - Py_DECREF(toDictResult); - tc->type = JT_NULL; - return; - } - - PRINTMARK(); - tc->type = JT_OBJECT; - SetupDictIter(toDictResult, pc, enc); - return; - } - else - if (PyObject_HasAttrString(obj, "__json__")) - { - PyObject* toJSONFunc = PyObject_GetAttrString(obj, "__json__"); - PyObject* tuple = PyTuple_New(0); - PyObject* toJSONResult = PyObject_Call(toJSONFunc, tuple, NULL); - Py_DECREF(tuple); - Py_DECREF(toJSONFunc); - - if (toJSONResult == NULL) - { - goto INVALID; - } - - if (PyErr_Occurred()) - { - Py_DECREF(toJSONResult); - goto INVALID; - } - - if (!PyString_Check(toJSONResult) && !PyUnicode_Check(toJSONResult)) - { - Py_DECREF(toJSONResult); - PyErr_Format (PyExc_TypeError, "expected string"); - goto INVALID; - } - - PRINTMARK(); - pc->PyTypeToJSON = PyRawJSONToUTF8; - tc->type = JT_RAW; - GET_TC(tc)->rawJSONValue = toJSONResult; - return; - } - - PRINTMARK(); - PyErr_Clear(); - - iter = PyObject_GetIter(obj); - - if (iter != NULL) - { - PRINTMARK(); - tc->type = JT_ARRAY; - pc->iterator = iter; - pc->iterEnd = Iter_iterEnd; - pc->iterNext = Iter_iterNext; - pc->iterGetValue = Iter_iterGetValue; - pc->iterGetName = Iter_iterGetName; - return; - } - - PRINTMARK(); - PyErr_Clear(); - - PRINTMARK(); - tc->type = JT_OBJECT; - GET_TC(tc)->attrList = PyObject_Dir(obj); - - if (GET_TC(tc)->attrList == NULL) - { - PyErr_Clear(); - goto INVALID; - } - - GET_TC(tc)->index = 0; - GET_TC(tc)->size = PyList_GET_SIZE(GET_TC(tc)->attrList); - PRINTMARK(); - - pc->iterEnd = Dir_iterEnd; - pc->iterNext = Dir_iterNext; - pc->iterGetValue = Dir_iterGetValue; - pc->iterGetName = Dir_iterGetName; - return; - -INVALID: - PRINTMARK(); - tc->type = JT_INVALID; - PyObject_Free(tc->prv); - tc->prv = NULL; - return; -} - -void Object_endTypeContext(JSOBJ obj, JSONTypeContext *tc) -{ - Py_XDECREF(GET_TC(tc)->newObj); - - if (tc->type == JT_RAW) - { - Py_XDECREF(GET_TC(tc)->rawJSONValue); - } - PyObject_Free(tc->prv); - tc->prv = NULL; -} - -const char *Object_getStringValue(JSOBJ obj, JSONTypeContext *tc, size_t *_outLen) -{ - return GET_TC(tc)->PyTypeToJSON (obj, tc, NULL, _outLen); -} - -JSINT64 Object_getLongValue(JSOBJ obj, JSONTypeContext *tc) -{ - JSINT64 ret; - GET_TC(tc)->PyTypeToJSON (obj, tc, &ret, NULL); - return ret; -} - -JSUINT64 Object_getUnsignedLongValue(JSOBJ obj, JSONTypeContext *tc) -{ - JSUINT64 ret; - GET_TC(tc)->PyTypeToJSON (obj, tc, &ret, NULL); - return ret; -} - -JSINT32 Object_getIntValue(JSOBJ obj, JSONTypeContext *tc) -{ - JSINT32 ret; - GET_TC(tc)->PyTypeToJSON (obj, tc, &ret, NULL); - return ret; -} - -double Object_getDoubleValue(JSOBJ obj, JSONTypeContext *tc) -{ - double ret; - GET_TC(tc)->PyTypeToJSON (obj, tc, &ret, NULL); - return ret; -} - -static void Object_releaseObject(JSOBJ _obj) -{ - Py_DECREF( (PyObject *) _obj); -} - -int Object_iterNext(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->iterNext(obj, tc); -} - -void Object_iterEnd(JSOBJ obj, JSONTypeContext *tc) -{ - GET_TC(tc)->iterEnd(obj, tc); -} - -JSOBJ Object_iterGetValue(JSOBJ obj, JSONTypeContext *tc) -{ - return GET_TC(tc)->iterGetValue(obj, tc); -} - -char *Object_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) -{ - return GET_TC(tc)->iterGetName(obj, tc, outLen); -} - -PyObject* objToJSON(PyObject* self, PyObject *args, PyObject *kwargs) -{ - static char *kwlist[] = { "obj", "ensure_ascii", "double_precision", "encode_html_chars", "escape_forward_slashes", "sort_keys", "indent", NULL }; - - char buffer[65536]; - char *ret; - PyObject *newobj; - PyObject *oinput = NULL; - PyObject *oensureAscii = NULL; - PyObject *oencodeHTMLChars = NULL; - PyObject *oescapeForwardSlashes = NULL; - PyObject *osortKeys = NULL; - - JSONObjectEncoder encoder = - { - Object_beginTypeContext, - Object_endTypeContext, - Object_getStringValue, - Object_getLongValue, - Object_getUnsignedLongValue, - Object_getIntValue, - Object_getDoubleValue, - Object_iterNext, - Object_iterEnd, - Object_iterGetValue, - Object_iterGetName, - Object_releaseObject, - PyObject_Malloc, - PyObject_Realloc, - PyObject_Free, - -1, //recursionMax - 10, // default double precision setting - 1, //forceAscii - 0, //encodeHTMLChars - 1, //escapeForwardSlashes - 0, //sortKeys - 0, //indent - NULL, //prv - }; - - - PRINTMARK(); - - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|OiOOOi", kwlist, &oinput, &oensureAscii, &encoder.doublePrecision, &oencodeHTMLChars, &oescapeForwardSlashes, &osortKeys, &encoder.indent)) - { - return NULL; - } - - if (oensureAscii != NULL && !PyObject_IsTrue(oensureAscii)) - { - encoder.forceASCII = 0; - } - - if (oencodeHTMLChars != NULL && PyObject_IsTrue(oencodeHTMLChars)) - { - encoder.encodeHTMLChars = 1; - } - - if (oescapeForwardSlashes != NULL && !PyObject_IsTrue(oescapeForwardSlashes)) - { - encoder.escapeForwardSlashes = 0; - } - - if (osortKeys != NULL && PyObject_IsTrue(osortKeys)) - { - encoder.sortKeys = 1; - } - - PRINTMARK(); - ret = JSON_EncodeObject (oinput, &encoder, buffer, sizeof (buffer)); - PRINTMARK(); - - if (PyErr_Occurred()) - { - return NULL; - } - - if (encoder.errorMsg) - { - if (ret != buffer) - { - encoder.free (ret); - } - - PyErr_Format (PyExc_OverflowError, "%s", encoder.errorMsg); - return NULL; - } - - newobj = PyString_FromString (ret); - - if (ret != buffer) - { - encoder.free (ret); - } - - PRINTMARK(); - - return newobj; -} - -PyObject* objToJSONFile(PyObject* self, PyObject *args, PyObject *kwargs) -{ - PyObject *data; - PyObject *file; - PyObject *string; - PyObject *write; - PyObject *argtuple; - PyObject *write_result; - - PRINTMARK(); - - if (!PyArg_ParseTuple (args, "OO", &data, &file)) - { - return NULL; - } - - if (!PyObject_HasAttrString (file, "write")) - { - PyErr_Format (PyExc_TypeError, "expected file"); - return NULL; - } - - write = PyObject_GetAttrString (file, "write"); - - if (!PyCallable_Check (write)) - { - Py_XDECREF(write); - PyErr_Format (PyExc_TypeError, "expected file"); - return NULL; - } - - argtuple = PyTuple_Pack(1, data); - - string = objToJSON (self, argtuple, kwargs); - - if (string == NULL) - { - Py_XDECREF(write); - Py_XDECREF(argtuple); - return NULL; - } - - Py_XDECREF(argtuple); - - argtuple = PyTuple_Pack (1, string); - if (argtuple == NULL) - { - Py_XDECREF(write); - return NULL; - } - - write_result = PyObject_CallObject (write, argtuple); - if (write_result == NULL) - { - Py_XDECREF(write); - Py_XDECREF(argtuple); - return NULL; - } - - Py_DECREF(write_result); - Py_XDECREF(write); - Py_DECREF(argtuple); - Py_XDECREF(string); - - PRINTMARK(); - - Py_RETURN_NONE; -} diff --git a/contrib/python/ujson/py2/python/py_defines.h b/contrib/python/ujson/py2/python/py_defines.h deleted file mode 100644 index 2b38b41bdba..00000000000 --- a/contrib/python/ujson/py2/python/py_defines.h +++ /dev/null @@ -1,53 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms - * Copyright (c) 1988-1993 The Regents of the University of California. - * Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#include <Python.h> - -#if PY_MAJOR_VERSION >= 3 - -#define PyInt_Check PyLong_Check -#define PyInt_AS_LONG PyLong_AsLong -#define PyInt_FromLong PyLong_FromLong - -#define PyString_Check PyBytes_Check -#define PyString_GET_SIZE PyBytes_GET_SIZE -#define PyString_AS_STRING PyBytes_AS_STRING - -#define PyString_FromString PyUnicode_FromString - -#endif diff --git a/contrib/python/ujson/py2/python/ujson.c b/contrib/python/ujson/py2/python/ujson.c deleted file mode 100644 index d0b15c65cf1..00000000000 --- a/contrib/python/ujson/py2/python/ujson.c +++ /dev/null @@ -1,113 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms -* Copyright (c) 1988-1993 The Regents of the University of California. -* Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#include "py_defines.h" -#include "version.h" - -/* objToJSON */ -PyObject* objToJSON(PyObject* self, PyObject *args, PyObject *kwargs); -void initObjToJSON(void); - -/* JSONToObj */ -PyObject* JSONToObj(PyObject* self, PyObject *args, PyObject *kwargs); - -/* objToJSONFile */ -PyObject* objToJSONFile(PyObject* self, PyObject *args, PyObject *kwargs); - -/* JSONFileToObj */ -PyObject* JSONFileToObj(PyObject* self, PyObject *args, PyObject *kwargs); - - -#define ENCODER_HELP_TEXT "Use ensure_ascii=false to output UTF-8. Pass in double_precision to alter the maximum digit precision of doubles. Set encode_html_chars=True to encode < > & as unicode escape sequences. Set escape_forward_slashes=False to prevent escaping / characters." - -static PyMethodDef ujsonMethods[] = { - {"encode", (PyCFunction) objToJSON, METH_VARARGS | METH_KEYWORDS, "Converts arbitrary object recursively into JSON. " ENCODER_HELP_TEXT}, - {"decode", (PyCFunction) JSONToObj, METH_VARARGS | METH_KEYWORDS, "Converts JSON as string to dict object structure. Use precise_float=True to use high precision float decoder."}, - {"dumps", (PyCFunction) objToJSON, METH_VARARGS | METH_KEYWORDS, "Converts arbitrary object recursively into JSON. " ENCODER_HELP_TEXT}, - {"loads", (PyCFunction) JSONToObj, METH_VARARGS | METH_KEYWORDS, "Converts JSON as string to dict object structure. Use precise_float=True to use high precision float decoder."}, - {"dump", (PyCFunction) objToJSONFile, METH_VARARGS | METH_KEYWORDS, "Converts arbitrary object recursively into JSON file. " ENCODER_HELP_TEXT}, - {"load", (PyCFunction) JSONFileToObj, METH_VARARGS | METH_KEYWORDS, "Converts JSON as file to dict object structure. Use precise_float=True to use high precision float decoder."}, - {NULL, NULL, 0, NULL} /* Sentinel */ -}; - -#if PY_MAJOR_VERSION >= 3 - -static struct PyModuleDef moduledef = { - PyModuleDef_HEAD_INIT, - "ujson", - 0, /* m_doc */ - -1, /* m_size */ - ujsonMethods, /* m_methods */ - NULL, /* m_reload */ - NULL, /* m_traverse */ - NULL, /* m_clear */ - NULL /* m_free */ -}; - -#define PYMODINITFUNC PyObject *PyInit_ujson(void) -#define PYMODULE_CREATE() PyModule_Create(&moduledef) -#define MODINITERROR return NULL - -#else - -#define PYMODINITFUNC PyMODINIT_FUNC initujson(void) -#define PYMODULE_CREATE() Py_InitModule("ujson", ujsonMethods) -#define MODINITERROR return - -#endif - -PYMODINITFUNC -{ - PyObject *module; - PyObject *version_string; - - initObjToJSON(); - module = PYMODULE_CREATE(); - - if (module == NULL) - { - MODINITERROR; - } - - version_string = PyString_FromString (UJSON_VERSION); - PyModule_AddObject (module, "__version__", version_string); - -#if PY_MAJOR_VERSION >= 3 - return module; -#endif -} diff --git a/contrib/python/ujson/py2/python/version.h b/contrib/python/ujson/py2/python/version.h deleted file mode 100644 index f0ce6bb7338..00000000000 --- a/contrib/python/ujson/py2/python/version.h +++ /dev/null @@ -1,39 +0,0 @@ -/* -Developed by ESN, an Electronic Arts Inc. studio. -Copyright (c) 2014, Electronic Arts Inc. -All rights reserved. - -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions are met: -* Redistributions of source code must retain the above copyright -notice, this list of conditions and the following disclaimer. -* Redistributions in binary form must reproduce the above copyright -notice, this list of conditions and the following disclaimer in the -documentation and/or other materials provided with the distribution. -* Neither the name of ESN, Electronic Arts Inc. nor the -names of its contributors may be used to endorse or promote products -derived from this software without specific prior written permission. - -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND -ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -DISCLAIMED. IN NO EVENT SHALL ELECTRONIC ARTS INC. BE LIABLE -FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND -ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS -SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - -Portions of code from MODP_ASCII - Ascii transformations (upper/lower, etc) -http://code.google.com/p/stringencoders/ -Copyright (c) 2007 Nick Galbreath -- nickg [at] modp [dot] com. All rights reserved. - -Numeric decoder derived from from TCL library -http://www.opensource.apple.com/source/tcl/tcl-14/tcl/license.terms - * Copyright (c) 1988-1993 The Regents of the University of California. - * Copyright (c) 1994 Sun Microsystems, Inc. -*/ - -#define UJSON_VERSION "1.35" diff --git a/contrib/python/ujson/py2/ya.make b/contrib/python/ujson/py2/ya.make deleted file mode 100644 index 65971c8e9bd..00000000000 --- a/contrib/python/ujson/py2/ya.make +++ /dev/null @@ -1,30 +0,0 @@ -PY2_LIBRARY() - -LICENSE(BSD-3-Clause) - -VERSION(1.35+dev) - -NO_COMPILER_WARNINGS() -NO_UTIL() - -PY_REGISTER(ujson) - -ADDINCL( - contrib/python/ujson/py2/lib - contrib/python/ujson/py2/python -) - -SRCS( - lib/ultrajsondec.c - lib/ultrajsonenc.c - python/JSONtoObj.c - python/objToJSON.c - python/ujson.c -) - -PY_SRCS( - TOP_LEVEL - ujson.pyi -) - -END() diff --git a/contrib/python/ujson/ya.make b/contrib/python/ujson/ya.make deleted file mode 100644 index 0503b6b21b6..00000000000 --- a/contrib/python/ujson/ya.make +++ /dev/null @@ -1,18 +0,0 @@ -PY23_LIBRARY() - -LICENSE(Service-Py23-Proxy) - -IF (PYTHON2) - PEERDIR(contrib/python/ujson/py2) -ELSE() - PEERDIR(contrib/python/ujson/py3) -ENDIF() - -NO_LINT() - -END() - -RECURSE( - py2 - py3 -) |