diff options
author | nkozlovskiy <nmk@ydb.tech> | 2023-09-29 12:24:06 +0300 |
---|---|---|
committer | nkozlovskiy <nmk@ydb.tech> | 2023-09-29 12:41:34 +0300 |
commit | e0e3e1717e3d33762ce61950504f9637a6e669ed (patch) | |
tree | bca3ff6939b10ed60c3d5c12439963a1146b9711 /contrib/python/atomicwrites | |
parent | 38f2c5852db84c7b4d83adfcb009eb61541d1ccd (diff) | |
download | ydb-e0e3e1717e3d33762ce61950504f9637a6e669ed.tar.gz |
add ydb deps
Diffstat (limited to 'contrib/python/atomicwrites')
17 files changed, 1326 insertions, 0 deletions
diff --git a/contrib/python/atomicwrites/py2/.dist-info/METADATA b/contrib/python/atomicwrites/py2/.dist-info/METADATA new file mode 100644 index 0000000000..880f350789 --- /dev/null +++ b/contrib/python/atomicwrites/py2/.dist-info/METADATA @@ -0,0 +1,149 @@ +Metadata-Version: 2.1 +Name: atomicwrites +Version: 1.4.1 +Summary: Atomic file writes. +Home-page: https://github.com/untitaker/python-atomicwrites +Author: Markus Unterwaditzer +Author-email: markus@unterwaditzer.net +License: MIT +Platform: UNKNOWN +Classifier: License :: OSI Approved :: MIT License +Classifier: Programming Language :: Python :: 2 +Classifier: Programming Language :: Python :: 2.7 +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.4 +Classifier: Programming Language :: Python :: 3.5 +Classifier: Programming Language :: Python :: 3.6 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Programming Language :: Python :: Implementation :: CPython +Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* + +=================== +python-atomicwrites +=================== + +.. image:: https://travis-ci.com/untitaker/python-atomicwrites.svg?branch=master + :target: https://travis-ci.com/untitaker/python-atomicwrites +.. image:: https://ci.appveyor.com/api/projects/status/vadc4le3c27to59x/branch/master?svg=true + :target: https://ci.appveyor.com/project/untitaker/python-atomicwrites/branch/master +.. image:: https://readthedocs.org/projects/python-atomicwrites/badge/?version=latest + :target: https://python-atomicwrites.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + +**Atomic file writes.** + +.. code-block:: python + + from atomicwrites import atomic_write + + with atomic_write('foo.txt', overwrite=True) as f: + f.write('Hello world.') + # "foo.txt" doesn't exist yet. + + # Now it does. + +See `API documentation <https://python-atomicwrites.readthedocs.io/en/latest/#api>`_ for more +low-level interfaces. + +Features that distinguish it from other similar libraries (see `Alternatives and Credit`_): + +- Race-free assertion that the target file doesn't yet exist. This can be + controlled with the ``overwrite`` parameter. + +- Windows support, although not well-tested. The MSDN resources are not very + explicit about which operations are atomic. I'm basing my assumptions off `a + comment + <https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/449bb49d-8acc-48dc-a46f-0760ceddbfc3/movefileexmovefilereplaceexisting-ntfs-same-volume-atomic?forum=windowssdk#a239bc26-eaf0-4920-9f21-440bd2be9cc8>`_ + by `Doug Cook + <https://social.msdn.microsoft.com/Profile/doug%20e.%20cook>`_, who appears + to be a Microsoft employee: + + Question: Is MoveFileEx atomic if the existing and new + files are both on the same drive? + + The simple answer is "usually, but in some cases it will silently fall-back + to a non-atomic method, so don't count on it". + + The implementation of MoveFileEx looks something like this: [...] + + The problem is if the rename fails, you might end up with a CopyFile, which + is definitely not atomic. + + If you really need atomic-or-nothing, you can try calling + NtSetInformationFile, which is unsupported but is much more likely to be + atomic. + +- Simple high-level API that wraps a very flexible class-based API. + +- Consistent error handling across platforms. + + +How it works +============ + +It uses a temporary file in the same directory as the given path. This ensures +that the temporary file resides on the same filesystem. + +The temporary file will then be atomically moved to the target location: On +POSIX, it will use ``rename`` if files should be overwritten, otherwise a +combination of ``link`` and ``unlink``. On Windows, it uses MoveFileEx_ through +stdlib's ``ctypes`` with the appropriate flags. + +Note that with ``link`` and ``unlink``, there's a timewindow where the file +might be available under two entries in the filesystem: The name of the +temporary file, and the name of the target file. + +Also note that the permissions of the target file may change this way. In some +situations a ``chmod`` can be issued without any concurrency problems, but +since that is not always the case, this library doesn't do it by itself. + +.. _MoveFileEx: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx + +fsync +----- + +On POSIX, ``fsync`` is invoked on the temporary file after it is written (to +flush file content and metadata), and on the parent directory after the file is +moved (to flush filename). + +``fsync`` does not take care of disks' internal buffers, but there don't seem +to be any standard POSIX APIs for that. On OS X, ``fcntl`` is used with +``F_FULLFSYNC`` instead of ``fsync`` for that reason. + +On Windows, `_commit <https://msdn.microsoft.com/en-us/library/17618685.aspx>`_ +is used, but there are no guarantees about disk internal buffers. + +Alternatives and Credit +======================= + +Atomicwrites is directly inspired by the following libraries (and shares a +minimal amount of code): + +- The Trac project's `utility functions + <http://www.edgewall.org/docs/tags-trac-0.11.7/epydoc/trac.util-pysrc.html>`_, + also used in `Werkzeug <http://werkzeug.pocoo.org/>`_ and + `mitsuhiko/python-atomicfile + <https://github.com/mitsuhiko/python-atomicfile>`_. The idea to use + ``ctypes`` instead of ``PyWin32`` originated there. + +- `abarnert/fatomic <https://github.com/abarnert/fatomic>`_. Windows support + (based on ``PyWin32``) was originally taken from there. + +Other alternatives to atomicwrites include: + +- `sashka/atomicfile <https://github.com/sashka/atomicfile>`_. Originally I + considered using that, but at the time it was lacking a lot of features I + needed (Windows support, overwrite-parameter, overriding behavior through + subclassing). + +- The `Boltons library collection <https://github.com/mahmoud/boltons>`_ + features a class for atomic file writes, which seems to have a very similar + ``overwrite`` parameter. It is lacking Windows support though. + +License +======= + +Licensed under the MIT, see ``LICENSE``. + + diff --git a/contrib/python/atomicwrites/py2/.dist-info/top_level.txt b/contrib/python/atomicwrites/py2/.dist-info/top_level.txt new file mode 100644 index 0000000000..5fa5a87d8b --- /dev/null +++ b/contrib/python/atomicwrites/py2/.dist-info/top_level.txt @@ -0,0 +1 @@ +atomicwrites diff --git a/contrib/python/atomicwrites/py2/LICENSE b/contrib/python/atomicwrites/py2/LICENSE new file mode 100644 index 0000000000..3bbadc3af2 --- /dev/null +++ b/contrib/python/atomicwrites/py2/LICENSE @@ -0,0 +1,19 @@ +Copyright (c) 2015-2016 Markus Unterwaditzer + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/contrib/python/atomicwrites/py2/README.rst b/contrib/python/atomicwrites/py2/README.rst new file mode 100644 index 0000000000..8b297543ae --- /dev/null +++ b/contrib/python/atomicwrites/py2/README.rst @@ -0,0 +1,126 @@ +=================== +python-atomicwrites +=================== + +.. image:: https://travis-ci.com/untitaker/python-atomicwrites.svg?branch=master + :target: https://travis-ci.com/untitaker/python-atomicwrites +.. image:: https://ci.appveyor.com/api/projects/status/vadc4le3c27to59x/branch/master?svg=true + :target: https://ci.appveyor.com/project/untitaker/python-atomicwrites/branch/master +.. image:: https://readthedocs.org/projects/python-atomicwrites/badge/?version=latest + :target: https://python-atomicwrites.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + +**Atomic file writes.** + +.. code-block:: python + + from atomicwrites import atomic_write + + with atomic_write('foo.txt', overwrite=True) as f: + f.write('Hello world.') + # "foo.txt" doesn't exist yet. + + # Now it does. + +See `API documentation <https://python-atomicwrites.readthedocs.io/en/latest/#api>`_ for more +low-level interfaces. + +Features that distinguish it from other similar libraries (see `Alternatives and Credit`_): + +- Race-free assertion that the target file doesn't yet exist. This can be + controlled with the ``overwrite`` parameter. + +- Windows support, although not well-tested. The MSDN resources are not very + explicit about which operations are atomic. I'm basing my assumptions off `a + comment + <https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/449bb49d-8acc-48dc-a46f-0760ceddbfc3/movefileexmovefilereplaceexisting-ntfs-same-volume-atomic?forum=windowssdk#a239bc26-eaf0-4920-9f21-440bd2be9cc8>`_ + by `Doug Cook + <https://social.msdn.microsoft.com/Profile/doug%20e.%20cook>`_, who appears + to be a Microsoft employee: + + Question: Is MoveFileEx atomic if the existing and new + files are both on the same drive? + + The simple answer is "usually, but in some cases it will silently fall-back + to a non-atomic method, so don't count on it". + + The implementation of MoveFileEx looks something like this: [...] + + The problem is if the rename fails, you might end up with a CopyFile, which + is definitely not atomic. + + If you really need atomic-or-nothing, you can try calling + NtSetInformationFile, which is unsupported but is much more likely to be + atomic. + +- Simple high-level API that wraps a very flexible class-based API. + +- Consistent error handling across platforms. + + +How it works +============ + +It uses a temporary file in the same directory as the given path. This ensures +that the temporary file resides on the same filesystem. + +The temporary file will then be atomically moved to the target location: On +POSIX, it will use ``rename`` if files should be overwritten, otherwise a +combination of ``link`` and ``unlink``. On Windows, it uses MoveFileEx_ through +stdlib's ``ctypes`` with the appropriate flags. + +Note that with ``link`` and ``unlink``, there's a timewindow where the file +might be available under two entries in the filesystem: The name of the +temporary file, and the name of the target file. + +Also note that the permissions of the target file may change this way. In some +situations a ``chmod`` can be issued without any concurrency problems, but +since that is not always the case, this library doesn't do it by itself. + +.. _MoveFileEx: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx + +fsync +----- + +On POSIX, ``fsync`` is invoked on the temporary file after it is written (to +flush file content and metadata), and on the parent directory after the file is +moved (to flush filename). + +``fsync`` does not take care of disks' internal buffers, but there don't seem +to be any standard POSIX APIs for that. On OS X, ``fcntl`` is used with +``F_FULLFSYNC`` instead of ``fsync`` for that reason. + +On Windows, `_commit <https://msdn.microsoft.com/en-us/library/17618685.aspx>`_ +is used, but there are no guarantees about disk internal buffers. + +Alternatives and Credit +======================= + +Atomicwrites is directly inspired by the following libraries (and shares a +minimal amount of code): + +- The Trac project's `utility functions + <http://www.edgewall.org/docs/tags-trac-0.11.7/epydoc/trac.util-pysrc.html>`_, + also used in `Werkzeug <http://werkzeug.pocoo.org/>`_ and + `mitsuhiko/python-atomicfile + <https://github.com/mitsuhiko/python-atomicfile>`_. The idea to use + ``ctypes`` instead of ``PyWin32`` originated there. + +- `abarnert/fatomic <https://github.com/abarnert/fatomic>`_. Windows support + (based on ``PyWin32``) was originally taken from there. + +Other alternatives to atomicwrites include: + +- `sashka/atomicfile <https://github.com/sashka/atomicfile>`_. Originally I + considered using that, but at the time it was lacking a lot of features I + needed (Windows support, overwrite-parameter, overriding behavior through + subclassing). + +- The `Boltons library collection <https://github.com/mahmoud/boltons>`_ + features a class for atomic file writes, which seems to have a very similar + ``overwrite`` parameter. It is lacking Windows support though. + +License +======= + +Licensed under the MIT, see ``LICENSE``. diff --git a/contrib/python/atomicwrites/py2/atomicwrites/__init__.py b/contrib/python/atomicwrites/py2/atomicwrites/__init__.py new file mode 100644 index 0000000000..669191bb5f --- /dev/null +++ b/contrib/python/atomicwrites/py2/atomicwrites/__init__.py @@ -0,0 +1,229 @@ +import contextlib +import io +import os +import sys +import tempfile + +try: + import fcntl +except ImportError: + fcntl = None + +# `fspath` was added in Python 3.6 +try: + from os import fspath +except ImportError: + fspath = None + +__version__ = '1.4.1' + + +PY2 = sys.version_info[0] == 2 + +text_type = unicode if PY2 else str # noqa + + +def _path_to_unicode(x): + if not isinstance(x, text_type): + return x.decode(sys.getfilesystemencoding()) + return x + + +DEFAULT_MODE = "wb" if PY2 else "w" + + +_proper_fsync = os.fsync + + +if sys.platform != 'win32': + if hasattr(fcntl, 'F_FULLFSYNC'): + def _proper_fsync(fd): + # https://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html + # https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html + # https://github.com/untitaker/python-atomicwrites/issues/6 + fcntl.fcntl(fd, fcntl.F_FULLFSYNC) + + def _sync_directory(directory): + # Ensure that filenames are written to disk + fd = os.open(directory, 0) + try: + _proper_fsync(fd) + finally: + os.close(fd) + + def _replace_atomic(src, dst): + os.rename(src, dst) + _sync_directory(os.path.normpath(os.path.dirname(dst))) + + def _move_atomic(src, dst): + os.link(src, dst) + os.unlink(src) + + src_dir = os.path.normpath(os.path.dirname(src)) + dst_dir = os.path.normpath(os.path.dirname(dst)) + _sync_directory(dst_dir) + if src_dir != dst_dir: + _sync_directory(src_dir) +else: + from ctypes import windll, WinError + + _MOVEFILE_REPLACE_EXISTING = 0x1 + _MOVEFILE_WRITE_THROUGH = 0x8 + _windows_default_flags = _MOVEFILE_WRITE_THROUGH + + def _handle_errors(rv): + if not rv: + raise WinError() + + def _replace_atomic(src, dst): + _handle_errors(windll.kernel32.MoveFileExW( + _path_to_unicode(src), _path_to_unicode(dst), + _windows_default_flags | _MOVEFILE_REPLACE_EXISTING + )) + + def _move_atomic(src, dst): + _handle_errors(windll.kernel32.MoveFileExW( + _path_to_unicode(src), _path_to_unicode(dst), + _windows_default_flags + )) + + +def replace_atomic(src, dst): + ''' + Move ``src`` to ``dst``. If ``dst`` exists, it will be silently + overwritten. + + Both paths must reside on the same filesystem for the operation to be + atomic. + ''' + return _replace_atomic(src, dst) + + +def move_atomic(src, dst): + ''' + Move ``src`` to ``dst``. There might a timewindow where both filesystem + entries exist. If ``dst`` already exists, :py:exc:`FileExistsError` will be + raised. + + Both paths must reside on the same filesystem for the operation to be + atomic. + ''' + return _move_atomic(src, dst) + + +class AtomicWriter(object): + ''' + A helper class for performing atomic writes. Usage:: + + with AtomicWriter(path).open() as f: + f.write(...) + + :param path: The destination filepath. May or may not exist. + :param mode: The filemode for the temporary file. This defaults to `wb` in + Python 2 and `w` in Python 3. + :param overwrite: If set to false, an error is raised if ``path`` exists. + Errors are only raised after the file has been written to. Either way, + the operation is atomic. + :param open_kwargs: Keyword-arguments to pass to the underlying + :py:func:`open` call. This can be used to set the encoding when opening + files in text-mode. + + If you need further control over the exact behavior, you are encouraged to + subclass. + ''' + + def __init__(self, path, mode=DEFAULT_MODE, overwrite=False, + **open_kwargs): + if 'a' in mode: + raise ValueError( + 'Appending to an existing file is not supported, because that ' + 'would involve an expensive `copy`-operation to a temporary ' + 'file. Open the file in normal `w`-mode and copy explicitly ' + 'if that\'s what you\'re after.' + ) + if 'x' in mode: + raise ValueError('Use the `overwrite`-parameter instead.') + if 'w' not in mode: + raise ValueError('AtomicWriters can only be written to.') + + # Attempt to convert `path` to `str` or `bytes` + if fspath is not None: + path = fspath(path) + + self._path = path + self._mode = mode + self._overwrite = overwrite + self._open_kwargs = open_kwargs + + def open(self): + ''' + Open the temporary file. + ''' + return self._open(self.get_fileobject) + + @contextlib.contextmanager + def _open(self, get_fileobject): + f = None # make sure f exists even if get_fileobject() fails + try: + success = False + with get_fileobject(**self._open_kwargs) as f: + yield f + self.sync(f) + self.commit(f) + success = True + finally: + if not success: + try: + self.rollback(f) + except Exception: + pass + + def get_fileobject(self, suffix="", prefix=tempfile.gettempprefix(), + dir=None, **kwargs): + '''Return the temporary file to use.''' + if dir is None: + dir = os.path.normpath(os.path.dirname(self._path)) + descriptor, name = tempfile.mkstemp(suffix=suffix, prefix=prefix, + dir=dir) + # io.open() will take either the descriptor or the name, but we need + # the name later for commit()/replace_atomic() and couldn't find a way + # to get the filename from the descriptor. + os.close(descriptor) + kwargs['mode'] = self._mode + kwargs['file'] = name + return io.open(**kwargs) + + def sync(self, f): + '''responsible for clearing as many file caches as possible before + commit''' + f.flush() + _proper_fsync(f.fileno()) + + def commit(self, f): + '''Move the temporary file to the target location.''' + if self._overwrite: + replace_atomic(f.name, self._path) + else: + move_atomic(f.name, self._path) + + def rollback(self, f): + '''Clean up all temporary resources.''' + os.unlink(f.name) + + +def atomic_write(path, writer_cls=AtomicWriter, **cls_kwargs): + ''' + Simple atomic writes. This wraps :py:class:`AtomicWriter`:: + + with atomic_write(path) as f: + f.write(...) + + :param path: The target path to write to. + :param writer_cls: The writer class to use. This parameter is useful if you + subclassed :py:class:`AtomicWriter` to change some behavior and want to + use that new subclass. + + Additional keyword arguments are passed to the writer class. See + :py:class:`AtomicWriter`. + ''' + return writer_cls(path, **cls_kwargs).open() diff --git a/contrib/python/atomicwrites/py2/tests/test_atomicwrites.py b/contrib/python/atomicwrites/py2/tests/test_atomicwrites.py new file mode 100644 index 0000000000..b3128c5b4f --- /dev/null +++ b/contrib/python/atomicwrites/py2/tests/test_atomicwrites.py @@ -0,0 +1,91 @@ +import errno +import os + +from atomicwrites import atomic_write + +import pytest + + +def test_atomic_write(tmpdir): + fname = tmpdir.join('ha') + for i in range(2): + with atomic_write(str(fname), overwrite=True) as f: + f.write('hoho') + + with pytest.raises(OSError) as excinfo: + with atomic_write(str(fname), overwrite=False) as f: + f.write('haha') + + assert excinfo.value.errno == errno.EEXIST + + assert fname.read() == 'hoho' + assert len(tmpdir.listdir()) == 1 + + +def test_teardown(tmpdir): + fname = tmpdir.join('ha') + with pytest.raises(AssertionError): + with atomic_write(str(fname), overwrite=True): + assert False + + assert not tmpdir.listdir() + + +def test_replace_simultaneously_created_file(tmpdir): + fname = tmpdir.join('ha') + with atomic_write(str(fname), overwrite=True) as f: + f.write('hoho') + fname.write('harhar') + assert fname.read() == 'harhar' + assert fname.read() == 'hoho' + assert len(tmpdir.listdir()) == 1 + + +def test_dont_remove_simultaneously_created_file(tmpdir): + fname = tmpdir.join('ha') + with pytest.raises(OSError) as excinfo: + with atomic_write(str(fname), overwrite=False) as f: + f.write('hoho') + fname.write('harhar') + assert fname.read() == 'harhar' + + assert excinfo.value.errno == errno.EEXIST + assert fname.read() == 'harhar' + assert len(tmpdir.listdir()) == 1 + + +# Verify that nested exceptions during rollback do not overwrite the initial +# exception that triggered a rollback. +def test_open_reraise(tmpdir): + fname = tmpdir.join('ha') + with pytest.raises(AssertionError): + aw = atomic_write(str(fname), overwrite=False) + with aw: + # Mess with internals, so commit will trigger a ValueError. We're + # testing that the initial AssertionError triggered below is + # propagated up the stack, not the second exception triggered + # during commit. + aw.rollback = lambda: 1 / 0 + # Now trigger our own exception. + assert False, "Intentional failure for testing purposes" + + +def test_atomic_write_in_pwd(tmpdir): + orig_curdir = os.getcwd() + try: + os.chdir(str(tmpdir)) + fname = 'ha' + for i in range(2): + with atomic_write(str(fname), overwrite=True) as f: + f.write('hoho') + + with pytest.raises(OSError) as excinfo: + with atomic_write(str(fname), overwrite=False) as f: + f.write('haha') + + assert excinfo.value.errno == errno.EEXIST + + assert open(fname).read() == 'hoho' + assert len(tmpdir.listdir()) == 1 + finally: + os.chdir(orig_curdir) diff --git a/contrib/python/atomicwrites/py2/tests/ya.make b/contrib/python/atomicwrites/py2/tests/ya.make new file mode 100644 index 0000000000..c193d0b3ba --- /dev/null +++ b/contrib/python/atomicwrites/py2/tests/ya.make @@ -0,0 +1,13 @@ +PY2TEST() + +PEERDIR( + contrib/python/atomicwrites +) + +TEST_SRCS( + test_atomicwrites.py +) + +NO_LINT() + +END() diff --git a/contrib/python/atomicwrites/py2/ya.make b/contrib/python/atomicwrites/py2/ya.make new file mode 100644 index 0000000000..6a353b6821 --- /dev/null +++ b/contrib/python/atomicwrites/py2/ya.make @@ -0,0 +1,26 @@ +# Generated by devtools/yamaker (pypi). + +PY2_LIBRARY() + +VERSION(1.4.1) + +LICENSE(MIT) + +NO_LINT() + +PY_SRCS( + TOP_LEVEL + atomicwrites/__init__.py +) + +RESOURCE_FILES( + PREFIX contrib/python/atomicwrites/py2/ + .dist-info/METADATA + .dist-info/top_level.txt +) + +END() + +RECURSE_FOR_TESTS( + tests +) diff --git a/contrib/python/atomicwrites/py3/.dist-info/METADATA b/contrib/python/atomicwrites/py3/.dist-info/METADATA new file mode 100644 index 0000000000..880f350789 --- /dev/null +++ b/contrib/python/atomicwrites/py3/.dist-info/METADATA @@ -0,0 +1,149 @@ +Metadata-Version: 2.1 +Name: atomicwrites +Version: 1.4.1 +Summary: Atomic file writes. +Home-page: https://github.com/untitaker/python-atomicwrites +Author: Markus Unterwaditzer +Author-email: markus@unterwaditzer.net +License: MIT +Platform: UNKNOWN +Classifier: License :: OSI Approved :: MIT License +Classifier: Programming Language :: Python :: 2 +Classifier: Programming Language :: Python :: 2.7 +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.4 +Classifier: Programming Language :: Python :: 3.5 +Classifier: Programming Language :: Python :: 3.6 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Programming Language :: Python :: Implementation :: CPython +Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* + +=================== +python-atomicwrites +=================== + +.. image:: https://travis-ci.com/untitaker/python-atomicwrites.svg?branch=master + :target: https://travis-ci.com/untitaker/python-atomicwrites +.. image:: https://ci.appveyor.com/api/projects/status/vadc4le3c27to59x/branch/master?svg=true + :target: https://ci.appveyor.com/project/untitaker/python-atomicwrites/branch/master +.. image:: https://readthedocs.org/projects/python-atomicwrites/badge/?version=latest + :target: https://python-atomicwrites.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + +**Atomic file writes.** + +.. code-block:: python + + from atomicwrites import atomic_write + + with atomic_write('foo.txt', overwrite=True) as f: + f.write('Hello world.') + # "foo.txt" doesn't exist yet. + + # Now it does. + +See `API documentation <https://python-atomicwrites.readthedocs.io/en/latest/#api>`_ for more +low-level interfaces. + +Features that distinguish it from other similar libraries (see `Alternatives and Credit`_): + +- Race-free assertion that the target file doesn't yet exist. This can be + controlled with the ``overwrite`` parameter. + +- Windows support, although not well-tested. The MSDN resources are not very + explicit about which operations are atomic. I'm basing my assumptions off `a + comment + <https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/449bb49d-8acc-48dc-a46f-0760ceddbfc3/movefileexmovefilereplaceexisting-ntfs-same-volume-atomic?forum=windowssdk#a239bc26-eaf0-4920-9f21-440bd2be9cc8>`_ + by `Doug Cook + <https://social.msdn.microsoft.com/Profile/doug%20e.%20cook>`_, who appears + to be a Microsoft employee: + + Question: Is MoveFileEx atomic if the existing and new + files are both on the same drive? + + The simple answer is "usually, but in some cases it will silently fall-back + to a non-atomic method, so don't count on it". + + The implementation of MoveFileEx looks something like this: [...] + + The problem is if the rename fails, you might end up with a CopyFile, which + is definitely not atomic. + + If you really need atomic-or-nothing, you can try calling + NtSetInformationFile, which is unsupported but is much more likely to be + atomic. + +- Simple high-level API that wraps a very flexible class-based API. + +- Consistent error handling across platforms. + + +How it works +============ + +It uses a temporary file in the same directory as the given path. This ensures +that the temporary file resides on the same filesystem. + +The temporary file will then be atomically moved to the target location: On +POSIX, it will use ``rename`` if files should be overwritten, otherwise a +combination of ``link`` and ``unlink``. On Windows, it uses MoveFileEx_ through +stdlib's ``ctypes`` with the appropriate flags. + +Note that with ``link`` and ``unlink``, there's a timewindow where the file +might be available under two entries in the filesystem: The name of the +temporary file, and the name of the target file. + +Also note that the permissions of the target file may change this way. In some +situations a ``chmod`` can be issued without any concurrency problems, but +since that is not always the case, this library doesn't do it by itself. + +.. _MoveFileEx: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx + +fsync +----- + +On POSIX, ``fsync`` is invoked on the temporary file after it is written (to +flush file content and metadata), and on the parent directory after the file is +moved (to flush filename). + +``fsync`` does not take care of disks' internal buffers, but there don't seem +to be any standard POSIX APIs for that. On OS X, ``fcntl`` is used with +``F_FULLFSYNC`` instead of ``fsync`` for that reason. + +On Windows, `_commit <https://msdn.microsoft.com/en-us/library/17618685.aspx>`_ +is used, but there are no guarantees about disk internal buffers. + +Alternatives and Credit +======================= + +Atomicwrites is directly inspired by the following libraries (and shares a +minimal amount of code): + +- The Trac project's `utility functions + <http://www.edgewall.org/docs/tags-trac-0.11.7/epydoc/trac.util-pysrc.html>`_, + also used in `Werkzeug <http://werkzeug.pocoo.org/>`_ and + `mitsuhiko/python-atomicfile + <https://github.com/mitsuhiko/python-atomicfile>`_. The idea to use + ``ctypes`` instead of ``PyWin32`` originated there. + +- `abarnert/fatomic <https://github.com/abarnert/fatomic>`_. Windows support + (based on ``PyWin32``) was originally taken from there. + +Other alternatives to atomicwrites include: + +- `sashka/atomicfile <https://github.com/sashka/atomicfile>`_. Originally I + considered using that, but at the time it was lacking a lot of features I + needed (Windows support, overwrite-parameter, overriding behavior through + subclassing). + +- The `Boltons library collection <https://github.com/mahmoud/boltons>`_ + features a class for atomic file writes, which seems to have a very similar + ``overwrite`` parameter. It is lacking Windows support though. + +License +======= + +Licensed under the MIT, see ``LICENSE``. + + diff --git a/contrib/python/atomicwrites/py3/.dist-info/top_level.txt b/contrib/python/atomicwrites/py3/.dist-info/top_level.txt new file mode 100644 index 0000000000..5fa5a87d8b --- /dev/null +++ b/contrib/python/atomicwrites/py3/.dist-info/top_level.txt @@ -0,0 +1 @@ +atomicwrites diff --git a/contrib/python/atomicwrites/py3/LICENSE b/contrib/python/atomicwrites/py3/LICENSE new file mode 100644 index 0000000000..3bbadc3af2 --- /dev/null +++ b/contrib/python/atomicwrites/py3/LICENSE @@ -0,0 +1,19 @@ +Copyright (c) 2015-2016 Markus Unterwaditzer + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/contrib/python/atomicwrites/py3/README.rst b/contrib/python/atomicwrites/py3/README.rst new file mode 100644 index 0000000000..8b297543ae --- /dev/null +++ b/contrib/python/atomicwrites/py3/README.rst @@ -0,0 +1,126 @@ +=================== +python-atomicwrites +=================== + +.. image:: https://travis-ci.com/untitaker/python-atomicwrites.svg?branch=master + :target: https://travis-ci.com/untitaker/python-atomicwrites +.. image:: https://ci.appveyor.com/api/projects/status/vadc4le3c27to59x/branch/master?svg=true + :target: https://ci.appveyor.com/project/untitaker/python-atomicwrites/branch/master +.. image:: https://readthedocs.org/projects/python-atomicwrites/badge/?version=latest + :target: https://python-atomicwrites.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + +**Atomic file writes.** + +.. code-block:: python + + from atomicwrites import atomic_write + + with atomic_write('foo.txt', overwrite=True) as f: + f.write('Hello world.') + # "foo.txt" doesn't exist yet. + + # Now it does. + +See `API documentation <https://python-atomicwrites.readthedocs.io/en/latest/#api>`_ for more +low-level interfaces. + +Features that distinguish it from other similar libraries (see `Alternatives and Credit`_): + +- Race-free assertion that the target file doesn't yet exist. This can be + controlled with the ``overwrite`` parameter. + +- Windows support, although not well-tested. The MSDN resources are not very + explicit about which operations are atomic. I'm basing my assumptions off `a + comment + <https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/449bb49d-8acc-48dc-a46f-0760ceddbfc3/movefileexmovefilereplaceexisting-ntfs-same-volume-atomic?forum=windowssdk#a239bc26-eaf0-4920-9f21-440bd2be9cc8>`_ + by `Doug Cook + <https://social.msdn.microsoft.com/Profile/doug%20e.%20cook>`_, who appears + to be a Microsoft employee: + + Question: Is MoveFileEx atomic if the existing and new + files are both on the same drive? + + The simple answer is "usually, but in some cases it will silently fall-back + to a non-atomic method, so don't count on it". + + The implementation of MoveFileEx looks something like this: [...] + + The problem is if the rename fails, you might end up with a CopyFile, which + is definitely not atomic. + + If you really need atomic-or-nothing, you can try calling + NtSetInformationFile, which is unsupported but is much more likely to be + atomic. + +- Simple high-level API that wraps a very flexible class-based API. + +- Consistent error handling across platforms. + + +How it works +============ + +It uses a temporary file in the same directory as the given path. This ensures +that the temporary file resides on the same filesystem. + +The temporary file will then be atomically moved to the target location: On +POSIX, it will use ``rename`` if files should be overwritten, otherwise a +combination of ``link`` and ``unlink``. On Windows, it uses MoveFileEx_ through +stdlib's ``ctypes`` with the appropriate flags. + +Note that with ``link`` and ``unlink``, there's a timewindow where the file +might be available under two entries in the filesystem: The name of the +temporary file, and the name of the target file. + +Also note that the permissions of the target file may change this way. In some +situations a ``chmod`` can be issued without any concurrency problems, but +since that is not always the case, this library doesn't do it by itself. + +.. _MoveFileEx: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx + +fsync +----- + +On POSIX, ``fsync`` is invoked on the temporary file after it is written (to +flush file content and metadata), and on the parent directory after the file is +moved (to flush filename). + +``fsync`` does not take care of disks' internal buffers, but there don't seem +to be any standard POSIX APIs for that. On OS X, ``fcntl`` is used with +``F_FULLFSYNC`` instead of ``fsync`` for that reason. + +On Windows, `_commit <https://msdn.microsoft.com/en-us/library/17618685.aspx>`_ +is used, but there are no guarantees about disk internal buffers. + +Alternatives and Credit +======================= + +Atomicwrites is directly inspired by the following libraries (and shares a +minimal amount of code): + +- The Trac project's `utility functions + <http://www.edgewall.org/docs/tags-trac-0.11.7/epydoc/trac.util-pysrc.html>`_, + also used in `Werkzeug <http://werkzeug.pocoo.org/>`_ and + `mitsuhiko/python-atomicfile + <https://github.com/mitsuhiko/python-atomicfile>`_. The idea to use + ``ctypes`` instead of ``PyWin32`` originated there. + +- `abarnert/fatomic <https://github.com/abarnert/fatomic>`_. Windows support + (based on ``PyWin32``) was originally taken from there. + +Other alternatives to atomicwrites include: + +- `sashka/atomicfile <https://github.com/sashka/atomicfile>`_. Originally I + considered using that, but at the time it was lacking a lot of features I + needed (Windows support, overwrite-parameter, overriding behavior through + subclassing). + +- The `Boltons library collection <https://github.com/mahmoud/boltons>`_ + features a class for atomic file writes, which seems to have a very similar + ``overwrite`` parameter. It is lacking Windows support though. + +License +======= + +Licensed under the MIT, see ``LICENSE``. diff --git a/contrib/python/atomicwrites/py3/atomicwrites/__init__.py b/contrib/python/atomicwrites/py3/atomicwrites/__init__.py new file mode 100644 index 0000000000..669191bb5f --- /dev/null +++ b/contrib/python/atomicwrites/py3/atomicwrites/__init__.py @@ -0,0 +1,229 @@ +import contextlib +import io +import os +import sys +import tempfile + +try: + import fcntl +except ImportError: + fcntl = None + +# `fspath` was added in Python 3.6 +try: + from os import fspath +except ImportError: + fspath = None + +__version__ = '1.4.1' + + +PY2 = sys.version_info[0] == 2 + +text_type = unicode if PY2 else str # noqa + + +def _path_to_unicode(x): + if not isinstance(x, text_type): + return x.decode(sys.getfilesystemencoding()) + return x + + +DEFAULT_MODE = "wb" if PY2 else "w" + + +_proper_fsync = os.fsync + + +if sys.platform != 'win32': + if hasattr(fcntl, 'F_FULLFSYNC'): + def _proper_fsync(fd): + # https://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html + # https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html + # https://github.com/untitaker/python-atomicwrites/issues/6 + fcntl.fcntl(fd, fcntl.F_FULLFSYNC) + + def _sync_directory(directory): + # Ensure that filenames are written to disk + fd = os.open(directory, 0) + try: + _proper_fsync(fd) + finally: + os.close(fd) + + def _replace_atomic(src, dst): + os.rename(src, dst) + _sync_directory(os.path.normpath(os.path.dirname(dst))) + + def _move_atomic(src, dst): + os.link(src, dst) + os.unlink(src) + + src_dir = os.path.normpath(os.path.dirname(src)) + dst_dir = os.path.normpath(os.path.dirname(dst)) + _sync_directory(dst_dir) + if src_dir != dst_dir: + _sync_directory(src_dir) +else: + from ctypes import windll, WinError + + _MOVEFILE_REPLACE_EXISTING = 0x1 + _MOVEFILE_WRITE_THROUGH = 0x8 + _windows_default_flags = _MOVEFILE_WRITE_THROUGH + + def _handle_errors(rv): + if not rv: + raise WinError() + + def _replace_atomic(src, dst): + _handle_errors(windll.kernel32.MoveFileExW( + _path_to_unicode(src), _path_to_unicode(dst), + _windows_default_flags | _MOVEFILE_REPLACE_EXISTING + )) + + def _move_atomic(src, dst): + _handle_errors(windll.kernel32.MoveFileExW( + _path_to_unicode(src), _path_to_unicode(dst), + _windows_default_flags + )) + + +def replace_atomic(src, dst): + ''' + Move ``src`` to ``dst``. If ``dst`` exists, it will be silently + overwritten. + + Both paths must reside on the same filesystem for the operation to be + atomic. + ''' + return _replace_atomic(src, dst) + + +def move_atomic(src, dst): + ''' + Move ``src`` to ``dst``. There might a timewindow where both filesystem + entries exist. If ``dst`` already exists, :py:exc:`FileExistsError` will be + raised. + + Both paths must reside on the same filesystem for the operation to be + atomic. + ''' + return _move_atomic(src, dst) + + +class AtomicWriter(object): + ''' + A helper class for performing atomic writes. Usage:: + + with AtomicWriter(path).open() as f: + f.write(...) + + :param path: The destination filepath. May or may not exist. + :param mode: The filemode for the temporary file. This defaults to `wb` in + Python 2 and `w` in Python 3. + :param overwrite: If set to false, an error is raised if ``path`` exists. + Errors are only raised after the file has been written to. Either way, + the operation is atomic. + :param open_kwargs: Keyword-arguments to pass to the underlying + :py:func:`open` call. This can be used to set the encoding when opening + files in text-mode. + + If you need further control over the exact behavior, you are encouraged to + subclass. + ''' + + def __init__(self, path, mode=DEFAULT_MODE, overwrite=False, + **open_kwargs): + if 'a' in mode: + raise ValueError( + 'Appending to an existing file is not supported, because that ' + 'would involve an expensive `copy`-operation to a temporary ' + 'file. Open the file in normal `w`-mode and copy explicitly ' + 'if that\'s what you\'re after.' + ) + if 'x' in mode: + raise ValueError('Use the `overwrite`-parameter instead.') + if 'w' not in mode: + raise ValueError('AtomicWriters can only be written to.') + + # Attempt to convert `path` to `str` or `bytes` + if fspath is not None: + path = fspath(path) + + self._path = path + self._mode = mode + self._overwrite = overwrite + self._open_kwargs = open_kwargs + + def open(self): + ''' + Open the temporary file. + ''' + return self._open(self.get_fileobject) + + @contextlib.contextmanager + def _open(self, get_fileobject): + f = None # make sure f exists even if get_fileobject() fails + try: + success = False + with get_fileobject(**self._open_kwargs) as f: + yield f + self.sync(f) + self.commit(f) + success = True + finally: + if not success: + try: + self.rollback(f) + except Exception: + pass + + def get_fileobject(self, suffix="", prefix=tempfile.gettempprefix(), + dir=None, **kwargs): + '''Return the temporary file to use.''' + if dir is None: + dir = os.path.normpath(os.path.dirname(self._path)) + descriptor, name = tempfile.mkstemp(suffix=suffix, prefix=prefix, + dir=dir) + # io.open() will take either the descriptor or the name, but we need + # the name later for commit()/replace_atomic() and couldn't find a way + # to get the filename from the descriptor. + os.close(descriptor) + kwargs['mode'] = self._mode + kwargs['file'] = name + return io.open(**kwargs) + + def sync(self, f): + '''responsible for clearing as many file caches as possible before + commit''' + f.flush() + _proper_fsync(f.fileno()) + + def commit(self, f): + '''Move the temporary file to the target location.''' + if self._overwrite: + replace_atomic(f.name, self._path) + else: + move_atomic(f.name, self._path) + + def rollback(self, f): + '''Clean up all temporary resources.''' + os.unlink(f.name) + + +def atomic_write(path, writer_cls=AtomicWriter, **cls_kwargs): + ''' + Simple atomic writes. This wraps :py:class:`AtomicWriter`:: + + with atomic_write(path) as f: + f.write(...) + + :param path: The target path to write to. + :param writer_cls: The writer class to use. This parameter is useful if you + subclassed :py:class:`AtomicWriter` to change some behavior and want to + use that new subclass. + + Additional keyword arguments are passed to the writer class. See + :py:class:`AtomicWriter`. + ''' + return writer_cls(path, **cls_kwargs).open() diff --git a/contrib/python/atomicwrites/py3/tests/test_atomicwrites.py b/contrib/python/atomicwrites/py3/tests/test_atomicwrites.py new file mode 100644 index 0000000000..b3128c5b4f --- /dev/null +++ b/contrib/python/atomicwrites/py3/tests/test_atomicwrites.py @@ -0,0 +1,91 @@ +import errno +import os + +from atomicwrites import atomic_write + +import pytest + + +def test_atomic_write(tmpdir): + fname = tmpdir.join('ha') + for i in range(2): + with atomic_write(str(fname), overwrite=True) as f: + f.write('hoho') + + with pytest.raises(OSError) as excinfo: + with atomic_write(str(fname), overwrite=False) as f: + f.write('haha') + + assert excinfo.value.errno == errno.EEXIST + + assert fname.read() == 'hoho' + assert len(tmpdir.listdir()) == 1 + + +def test_teardown(tmpdir): + fname = tmpdir.join('ha') + with pytest.raises(AssertionError): + with atomic_write(str(fname), overwrite=True): + assert False + + assert not tmpdir.listdir() + + +def test_replace_simultaneously_created_file(tmpdir): + fname = tmpdir.join('ha') + with atomic_write(str(fname), overwrite=True) as f: + f.write('hoho') + fname.write('harhar') + assert fname.read() == 'harhar' + assert fname.read() == 'hoho' + assert len(tmpdir.listdir()) == 1 + + +def test_dont_remove_simultaneously_created_file(tmpdir): + fname = tmpdir.join('ha') + with pytest.raises(OSError) as excinfo: + with atomic_write(str(fname), overwrite=False) as f: + f.write('hoho') + fname.write('harhar') + assert fname.read() == 'harhar' + + assert excinfo.value.errno == errno.EEXIST + assert fname.read() == 'harhar' + assert len(tmpdir.listdir()) == 1 + + +# Verify that nested exceptions during rollback do not overwrite the initial +# exception that triggered a rollback. +def test_open_reraise(tmpdir): + fname = tmpdir.join('ha') + with pytest.raises(AssertionError): + aw = atomic_write(str(fname), overwrite=False) + with aw: + # Mess with internals, so commit will trigger a ValueError. We're + # testing that the initial AssertionError triggered below is + # propagated up the stack, not the second exception triggered + # during commit. + aw.rollback = lambda: 1 / 0 + # Now trigger our own exception. + assert False, "Intentional failure for testing purposes" + + +def test_atomic_write_in_pwd(tmpdir): + orig_curdir = os.getcwd() + try: + os.chdir(str(tmpdir)) + fname = 'ha' + for i in range(2): + with atomic_write(str(fname), overwrite=True) as f: + f.write('hoho') + + with pytest.raises(OSError) as excinfo: + with atomic_write(str(fname), overwrite=False) as f: + f.write('haha') + + assert excinfo.value.errno == errno.EEXIST + + assert open(fname).read() == 'hoho' + assert len(tmpdir.listdir()) == 1 + finally: + os.chdir(orig_curdir) diff --git a/contrib/python/atomicwrites/py3/tests/ya.make b/contrib/python/atomicwrites/py3/tests/ya.make new file mode 100644 index 0000000000..7c9ba234df --- /dev/null +++ b/contrib/python/atomicwrites/py3/tests/ya.make @@ -0,0 +1,13 @@ +PY3TEST() + +PEERDIR( + contrib/python/atomicwrites +) + +TEST_SRCS( + test_atomicwrites.py +) + +NO_LINT() + +END() diff --git a/contrib/python/atomicwrites/py3/ya.make b/contrib/python/atomicwrites/py3/ya.make new file mode 100644 index 0000000000..94a9e4f519 --- /dev/null +++ b/contrib/python/atomicwrites/py3/ya.make @@ -0,0 +1,26 @@ +# Generated by devtools/yamaker (pypi). + +PY3_LIBRARY() + +VERSION(1.4.1) + +LICENSE(MIT) + +NO_LINT() + +PY_SRCS( + TOP_LEVEL + atomicwrites/__init__.py +) + +RESOURCE_FILES( + PREFIX contrib/python/atomicwrites/py3/ + .dist-info/METADATA + .dist-info/top_level.txt +) + +END() + +RECURSE_FOR_TESTS( + tests +) diff --git a/contrib/python/atomicwrites/ya.make b/contrib/python/atomicwrites/ya.make new file mode 100644 index 0000000000..212e0edaa8 --- /dev/null +++ b/contrib/python/atomicwrites/ya.make @@ -0,0 +1,18 @@ +PY23_LIBRARY() + +LICENSE(Service-Py23-Proxy) + +IF (PYTHON2) + PEERDIR(contrib/python/atomicwrites/py2) +ELSE() + PEERDIR(contrib/python/atomicwrites/py3) +ENDIF() + +NO_LINT() + +END() + +RECURSE( + py2 + py3 +) |