aboutsummaryrefslogtreecommitdiffstats
path: root/contrib/python/tenacity/py3/README.rst
blob: bdf7ff21174bd1175c3e4a35288d1ff2a5d590bc (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
Tenacity
========
.. image:: https://img.shields.io/pypi/v/tenacity.svg
    :target: https://pypi.python.org/pypi/tenacity

.. image:: https://circleci.com/gh/jd/tenacity.svg?style=svg
    :target: https://circleci.com/gh/jd/tenacity

.. image:: https://img.shields.io/endpoint.svg?url=https://api.mergify.com/badges/jd/tenacity&style=flat
   :target: https://mergify.io
   :alt: Mergify Status

**Please refer to the** `tenacity documentation <https://tenacity.readthedocs.io/en/latest/>`_ **for a better experience.**

Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in
Python, to simplify the task of adding retry behavior to just about anything.
It originates from `a fork of retrying
<https://github.com/rholder/retrying/issues/65>`_ which is sadly no longer
`maintained <https://julien.danjou.info/python-tenacity/>`_. Tenacity isn't
api compatible with retrying but adds significant new functionality and
fixes a number of longstanding bugs.

The simplest use case is retrying a flaky function whenever an `Exception`
occurs until a value is returned.

.. testcode::

    import random
    from tenacity import retry

    @retry
    def do_something_unreliable():
        if random.randint(0, 10) > 1:
            raise IOError("Broken sauce, everything is hosed!!!111one")
        else:
            return "Awesome sauce!"

    print(do_something_unreliable())

.. testoutput::
   :hide:

   Awesome sauce!


.. toctree::
    :hidden:
    :maxdepth: 2

    changelog
    api


Features
--------

- Generic Decorator API
- Specify stop condition (i.e. limit by number of attempts)
- Specify wait condition (i.e. exponential backoff sleeping between attempts)
- Customize retrying on Exceptions
- Customize retrying on expected returned result
- Retry on coroutines
- Retry code block with context manager


Installation
------------

To install *tenacity*, simply:

.. code-block:: bash

    $ pip install tenacity


Examples
----------

Basic Retry
~~~~~~~~~~~

.. testsetup:: *

    import logging
    #
    # Note the following import is used for demonstration convenience only.
    # Production code should always explicitly import the names it needs.
    #
    from tenacity import *

    class MyException(Exception):
        pass

As you saw above, the default behavior is to retry forever without waiting when
an exception is raised.

.. testcode::

    @retry
    def never_gonna_give_you_up():
        print("Retry forever ignoring Exceptions, don't wait between retries")
        raise Exception

Stopping
~~~~~~~~

Let's be a little less persistent and set some boundaries, such as the number
of attempts before giving up.

.. testcode::

    @retry(stop=stop_after_attempt(7))
    def stop_after_7_attempts():
        print("Stopping after 7 attempts")
        raise Exception

We don't have all day, so let's set a boundary for how long we should be
retrying stuff.

.. testcode::

    @retry(stop=stop_after_delay(10))
    def stop_after_10_s():
        print("Stopping after 10 seconds")
        raise Exception

If you're on a tight deadline, and exceeding your delay time isn't ok, 
then you can give up on retries one attempt before you would exceed the delay. 

.. testcode::

    @retry(stop=stop_before_delay(10))
    def stop_before_10_s():
        print("Stopping 1 attempt before 10 seconds")
        raise Exception

You can combine several stop conditions by using the `|` operator:

.. testcode::

    @retry(stop=(stop_after_delay(10) | stop_after_attempt(5)))
    def stop_after_10_s_or_5_retries():
        print("Stopping after 10 seconds or 5 retries")
        raise Exception

Waiting before retrying
~~~~~~~~~~~~~~~~~~~~~~~

Most things don't like to be polled as fast as possible, so let's just wait 2
seconds between retries.

.. testcode::

    @retry(wait=wait_fixed(2))
    def wait_2_s():
        print("Wait 2 second between retries")
        raise Exception

Some things perform best with a bit of randomness injected.

.. testcode::

    @retry(wait=wait_random(min=1, max=2))
    def wait_random_1_to_2_s():
        print("Randomly wait 1 to 2 seconds between retries")
        raise Exception

Then again, it's hard to beat exponential backoff when retrying distributed
services and other remote endpoints.

.. testcode::

    @retry(wait=wait_exponential(multiplier=1, min=4, max=10))
    def wait_exponential_1():
        print("Wait 2^x * 1 second between each retry starting with 4 seconds, then up to 10 seconds, then 10 seconds afterwards")
        raise Exception


Then again, it's also hard to beat combining fixed waits and jitter (to
help avoid thundering herds) when retrying distributed services and other
remote endpoints.

.. testcode::

    @retry(wait=wait_fixed(3) + wait_random(0, 2))
    def wait_fixed_jitter():
        print("Wait at least 3 seconds, and add up to 2 seconds of random delay")
        raise Exception

When multiple processes are in contention for a shared resource, exponentially
increasing jitter helps minimise collisions.

.. testcode::

    @retry(wait=wait_random_exponential(multiplier=1, max=60))
    def wait_exponential_jitter():
        print("Randomly wait up to 2^x * 1 seconds between each retry until the range reaches 60 seconds, then randomly up to 60 seconds afterwards")
        raise Exception


Sometimes it's necessary to build a chain of backoffs.

.. testcode::

    @retry(wait=wait_chain(*[wait_fixed(3) for i in range(3)] +
                           [wait_fixed(7) for i in range(2)] +
                           [wait_fixed(9)]))
    def wait_fixed_chained():
        print("Wait 3s for 3 attempts, 7s for the next 2 attempts and 9s for all attempts thereafter")
        raise Exception

Whether to retry
~~~~~~~~~~~~~~~~

We have a few options for dealing with retries that raise specific or general
exceptions, as in the cases here.

.. testcode::

    class ClientError(Exception):
        """Some type of client error."""

    @retry(retry=retry_if_exception_type(IOError))
    def might_io_error():
        print("Retry forever with no wait if an IOError occurs, raise any other errors")
        raise Exception

    @retry(retry=retry_if_not_exception_type(ClientError))
    def might_client_error():
        print("Retry forever with no wait if any error other than ClientError occurs. Immediately raise ClientError.")
        raise Exception

We can also use the result of the function to alter the behavior of retrying.

.. testcode::

    def is_none_p(value):
        """Return True if value is None"""
        return value is None

    @retry(retry=retry_if_result(is_none_p))
    def might_return_none():
        print("Retry with no wait if return value is None")

See also these methods:

.. testcode::

    retry_if_exception
    retry_if_exception_type
    retry_if_not_exception_type
    retry_unless_exception_type
    retry_if_result
    retry_if_not_result
    retry_if_exception_message
    retry_if_not_exception_message
    retry_any
    retry_all

We can also combine several conditions:

.. testcode::

    def is_none_p(value):
        """Return True if value is None"""
        return value is None

    @retry(retry=(retry_if_result(is_none_p) | retry_if_exception_type()))
    def might_return_none():
        print("Retry forever ignoring Exceptions with no wait if return value is None")

Any combination of stop, wait, etc. is also supported to give you the freedom
to mix and match.

It's also possible to retry explicitly at any time by raising the `TryAgain`
exception:

.. testcode::

   @retry
   def do_something():
       result = something_else()
       if result == 23:
          raise TryAgain

Error Handling
~~~~~~~~~~~~~~

Normally when your function fails its final time (and will not be retried again based on your settings),
a `RetryError` is raised. The exception your code encountered will be shown somewhere in the *middle*
of the stack trace.

If you would rather see the exception your code encountered at the *end* of the stack trace (where it
is most visible), you can set `reraise=True`.

.. testcode::

    @retry(reraise=True, stop=stop_after_attempt(3))
    def raise_my_exception():
        raise MyException("Fail")

    try:
        raise_my_exception()
    except MyException:
        # timed out retrying
        pass

Before and After Retry, and Logging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It's possible to execute an action before any attempt of calling the function
by using the before callback function:

.. testcode::

    import logging
    import sys

    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

    logger = logging.getLogger(__name__)

    @retry(stop=stop_after_attempt(3), before=before_log(logger, logging.DEBUG))
    def raise_my_exception():
        raise MyException("Fail")

In the same spirit, It's possible to execute after a call that failed:

.. testcode::

    import logging
    import sys

    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

    logger = logging.getLogger(__name__)

    @retry(stop=stop_after_attempt(3), after=after_log(logger, logging.DEBUG))
    def raise_my_exception():
        raise MyException("Fail")

It's also possible to only log failures that are going to be retried. Normally
retries happen after a wait interval, so the keyword argument is called
``before_sleep``:

.. testcode::

    import logging
    import sys

    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

    logger = logging.getLogger(__name__)

    @retry(stop=stop_after_attempt(3),
           before_sleep=before_sleep_log(logger, logging.DEBUG))
    def raise_my_exception():
        raise MyException("Fail")


Statistics
~~~~~~~~~~

You can access the statistics about the retry made over a function by using the
`retry` attribute attached to the function and its `statistics` attribute:

.. testcode::

    @retry(stop=stop_after_attempt(3))
    def raise_my_exception():
        raise MyException("Fail")

    try:
        raise_my_exception()
    except Exception:
        pass

    print(raise_my_exception.retry.statistics)

.. testoutput::
   :hide:

   ...

Custom Callbacks
~~~~~~~~~~~~~~~~

You can also define your own callbacks. The callback should accept one
parameter called ``retry_state`` that contains all information about current
retry invocation.

For example, you can call a custom callback function after all retries failed,
without raising an exception (or you can re-raise or do anything really)

.. testcode::

    def return_last_value(retry_state):
        """return the result of the last call attempt"""
        return retry_state.outcome.result()

    def is_false(value):
        """Return True if value is False"""
        return value is False

    # will return False after trying 3 times to get a different result
    @retry(stop=stop_after_attempt(3),
           retry_error_callback=return_last_value,
           retry=retry_if_result(is_false))
    def eventually_return_false():
        return False

RetryCallState
~~~~~~~~~~~~~~

``retry_state`` argument is an object of :class:`~tenacity.RetryCallState` class.

Other Custom Callbacks
~~~~~~~~~~~~~~~~~~~~~~

It's also possible to define custom callbacks for other keyword arguments.

.. function:: my_stop(retry_state)

   :param RetryCallState retry_state: info about current retry invocation
   :return: whether or not retrying should stop
   :rtype: bool

.. function:: my_wait(retry_state)

   :param RetryCallState retry_state: info about current retry invocation
   :return: number of seconds to wait before next retry
   :rtype: float

.. function:: my_retry(retry_state)

   :param RetryCallState retry_state: info about current retry invocation
   :return: whether or not retrying should continue
   :rtype: bool

.. function:: my_before(retry_state)

   :param RetryCallState retry_state: info about current retry invocation

.. function:: my_after(retry_state)

   :param RetryCallState retry_state: info about current retry invocation

.. function:: my_before_sleep(retry_state)

   :param RetryCallState retry_state: info about current retry invocation

Here's an example with a custom ``before_sleep`` function:

.. testcode::

    import logging

    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

    logger = logging.getLogger(__name__)

    def my_before_sleep(retry_state):
        if retry_state.attempt_number < 1:
            loglevel = logging.INFO
        else:
            loglevel = logging.WARNING
        logger.log(
            loglevel, 'Retrying %s: attempt %s ended with: %s',
            retry_state.fn, retry_state.attempt_number, retry_state.outcome)

    @retry(stop=stop_after_attempt(3), before_sleep=my_before_sleep)
    def raise_my_exception():
        raise MyException("Fail")

    try:
        raise_my_exception()
    except RetryError:
        pass


Changing Arguments at Run Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can change the arguments of a retry decorator as needed when calling it by
using the `retry_with` function attached to the wrapped function:

.. testcode::

    @retry(stop=stop_after_attempt(3))
    def raise_my_exception():
        raise MyException("Fail")

    try:
        raise_my_exception.retry_with(stop=stop_after_attempt(4))()
    except Exception:
        pass

    print(raise_my_exception.retry.statistics)

.. testoutput::
   :hide:

   ...

If you want to use variables to set up the retry parameters, you don't have
to use the `retry` decorator - you can instead use `Retrying` directly:

.. testcode::

    def never_good_enough(arg1):
        raise Exception('Invalid argument: {}'.format(arg1))

    def try_never_good_enough(max_attempts=3):
        retryer = Retrying(stop=stop_after_attempt(max_attempts), reraise=True)
        retryer(never_good_enough, 'I really do try')

Retrying code block
~~~~~~~~~~~~~~~~~~~

Tenacity allows you to retry a code block without the need to wraps it in an
isolated function. This makes it easy to isolate failing block while sharing
context. The trick is to combine a for loop and a context manager.

.. testcode::

   from tenacity import Retrying, RetryError, stop_after_attempt

   try:
       for attempt in Retrying(stop=stop_after_attempt(3)):
           with attempt:
               raise Exception('My code is failing!')
   except RetryError:
       pass

You can configure every details of retry policy by configuring the Retrying
object.

With async code you can use AsyncRetrying.

.. testcode::

   from tenacity import AsyncRetrying, RetryError, stop_after_attempt

   async def function():
      try:
          async for attempt in AsyncRetrying(stop=stop_after_attempt(3)):
              with attempt:
                  raise Exception('My code is failing!')
      except RetryError:
          pass

In both cases, you may want to set the result to the attempt so it's available
in retry strategies like ``retry_if_result``. This can be done accessing the
``retry_state`` property:

.. testcode::

    from tenacity import AsyncRetrying, retry_if_result

    async def function():
       async for attempt in AsyncRetrying(retry=retry_if_result(lambda x: x < 3)):
           with attempt:
               result = 1  # Some complex calculation, function call, etc.
           if not attempt.retry_state.outcome.failed:
               attempt.retry_state.set_result(result)
       return result

Async and retry
~~~~~~~~~~~~~~~

Finally, ``retry`` works also on asyncio and Tornado (>= 4.5) coroutines.
Sleeps are done asynchronously too.

.. code-block:: python

    @retry
    async def my_async_function(loop):
        await loop.getaddrinfo('8.8.8.8', 53)

.. code-block:: python

    @retry
    @tornado.gen.coroutine
    def my_async_function(http_client, url):
        yield http_client.fetch(url)

You can even use alternative event loops such as `curio` or `Trio` by passing the correct sleep function:

.. code-block:: python

    @retry(sleep=trio.sleep)
    async def my_async_function(loop):
        await asks.get('https://example.org')

Contribute
----------

#. Check for open issues or open a fresh issue to start a discussion around a
   feature idea or a bug.
#. Fork `the repository`_ on GitHub to start making your changes to the
   **main** branch (or branch off of it).
#. Write a test which shows that the bug was fixed or that the feature works as
   expected.
#. Add a `changelog <#Changelogs>`_
#. Make the docs better (or more detailed, or more easier to read, or ...)

.. _`the repository`: https://github.com/jd/tenacity

Changelogs
~~~~~~~~~~

`reno`_ is used for managing changelogs. Take a look at their usage docs.

The doc generation will automatically compile the changelogs. You just need to add them.

.. code-block:: sh

    # Opens a template file in an editor
    tox -e reno -- new some-slug-for-my-change --edit

.. _`reno`: https://docs.openstack.org/reno/latest/user/usage.html