1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
|
Internationalized Domain Names in Applications (IDNA)
=====================================================
Support for the Internationalised Domain Names in Applications
(IDNA) protocol as specified in `RFC 5891 <https://tools.ietf.org/html/rfc5891>`_.
This is the latest version of the protocol and is sometimes referred to as
“IDNA 2008”.
This library also provides support for Unicode Technical Standard 46,
`Unicode IDNA Compatibility Processing <https://unicode.org/reports/tr46/>`_.
This acts as a suitable replacement for the “encodings.idna” module that
comes with the Python standard library, but which only supports the
older superseded IDNA specification (`RFC 3490 <https://tools.ietf.org/html/rfc3490>`_).
Basic functions are simply executed:
.. code-block:: pycon
>>> import idna
>>> idna.encode('ドメイン.テスト')
b'xn--eckwd4c7c.xn--zckzah'
>>> print(idna.decode('xn--eckwd4c7c.xn--zckzah'))
ドメイン.テスト
Installation
------------
To install this library, you can use pip:
.. code-block:: bash
$ pip install idna
Alternatively, you can install the package using the bundled setup script:
.. code-block:: bash
$ python setup.py install
Usage
-----
For typical usage, the ``encode`` and ``decode`` functions will take a domain
name argument and perform a conversion to A-labels or U-labels respectively.
.. code-block:: pycon
>>> import idna
>>> idna.encode('ドメイン.テスト')
b'xn--eckwd4c7c.xn--zckzah'
>>> print(idna.decode('xn--eckwd4c7c.xn--zckzah'))
ドメイン.テスト
You may use the codec encoding and decoding methods using the
``idna.codec`` module:
.. code-block:: pycon
>>> import idna.codec
>>> print('домен.испытание'.encode('idna'))
b'xn--d1acufc.xn--80akhbyknj4f'
>>> print(b'xn--d1acufc.xn--80akhbyknj4f'.decode('idna'))
домен.испытание
Conversions can be applied at a per-label basis using the ``ulabel`` or ``alabel``
functions if necessary:
.. code-block:: pycon
>>> idna.alabel('测试')
b'xn--0zwm56d'
Compatibility Mapping (UTS #46)
+++++++++++++++++++++++++++++++
As described in `RFC 5895 <https://tools.ietf.org/html/rfc5895>`_, the IDNA
specification does not normalize input from different potential ways a user
may input a domain name. This functionality, known as a “mapping”, is
considered by the specification to be a local user-interface issue distinct
from IDNA conversion functionality.
This library provides one such mapping, that was developed by the Unicode
Consortium. Known as `Unicode IDNA Compatibility Processing <https://unicode.org/reports/tr46/>`_,
it provides for both a regular mapping for typical applications, as well as
a transitional mapping to help migrate from older IDNA 2003 applications.
For example, “Königsgäßchen” is not a permissible label as *LATIN CAPITAL
LETTER K* is not allowed (nor are capital letters in general). UTS 46 will
convert this into lower case prior to applying the IDNA conversion.
.. code-block:: pycon
>>> import idna
>>> idna.encode('Königsgäßchen')
...
idna.core.InvalidCodepoint: Codepoint U+004B at position 1 of 'Königsgäßchen' not allowed
>>> idna.encode('Königsgäßchen', uts46=True)
b'xn--knigsgchen-b4a3dun'
>>> print(idna.decode('xn--knigsgchen-b4a3dun'))
königsgäßchen
Transitional processing provides conversions to help transition from the older
2003 standard to the current standard. For example, in the original IDNA
specification, the *LATIN SMALL LETTER SHARP S* (ß) was converted into two
*LATIN SMALL LETTER S* (ss), whereas in the current IDNA specification this
conversion is not performed.
.. code-block:: pycon
>>> idna.encode('Königsgäßchen', uts46=True, transitional=True)
'xn--knigsgsschen-lcb0w'
Implementors should use transitional processing with caution, only in rare
cases where conversion from legacy labels to current labels must be performed
(i.e. IDNA implementations that pre-date 2008). For typical applications
that just need to convert labels, transitional processing is unlikely to be
beneficial and could produce unexpected incompatible results.
``encodings.idna`` Compatibility
++++++++++++++++++++++++++++++++
Function calls from the Python built-in ``encodings.idna`` module are
mapped to their IDNA 2008 equivalents using the ``idna.compat`` module.
Simply substitute the ``import`` clause in your code to refer to the
new module name.
Exceptions
----------
All errors raised during the conversion following the specification should
raise an exception derived from the ``idna.IDNAError`` base class.
More specific exceptions that may be generated as ``idna.IDNABidiError``
when the error reflects an illegal combination of left-to-right and
right-to-left characters in a label; ``idna.InvalidCodepoint`` when
a specific codepoint is an illegal character in an IDN label (i.e.
INVALID); and ``idna.InvalidCodepointContext`` when the codepoint is
illegal based on its positional context (i.e. it is CONTEXTO or CONTEXTJ
but the contextual requirements are not satisfied.)
Building and Diagnostics
------------------------
The IDNA and UTS 46 functionality relies upon pre-calculated lookup
tables for performance. These tables are derived from computing against
eligibility criteria in the respective standards. These tables are
computed using the command-line script ``tools/idna-data``.
This tool will fetch relevant codepoint data from the Unicode repository
and perform the required calculations to identify eligibility. There are
three main modes:
* ``idna-data make-libdata``. Generates ``idnadata.py`` and ``uts46data.py``,
the pre-calculated lookup tables using for IDNA and UTS 46 conversions. Implementors
who wish to track this library against a different Unicode version may use this tool
to manually generate a different version of the ``idnadata.py`` and ``uts46data.py``
files.
* ``idna-data make-table``. Generate a table of the IDNA disposition
(e.g. PVALID, CONTEXTJ, CONTEXTO) in the format found in Appendix B.1 of RFC
5892 and the pre-computed tables published by `IANA <https://www.iana.org/>`_.
* ``idna-data U+0061``. Prints debugging output on the various properties
associated with an individual Unicode codepoint (in this case, U+0061), that are
used to assess the IDNA and UTS 46 status of a codepoint. This is helpful in debugging
or analysis.
The tool accepts a number of arguments, described using ``idna-data -h``. Most notably,
the ``--version`` argument allows the specification of the version of Unicode to use
in computing the table data. For example, ``idna-data --version 9.0.0 make-libdata``
will generate library data against Unicode 9.0.0.
Additional Notes
----------------
* **Packages**. The latest tagged release version is published in the
`Python Package Index <https://pypi.org/project/idna/>`_.
* **Version support**. This library supports Python 3.5 and higher. As this library
serves as a low-level toolkit for a variety of applications, many of which strive
for broad compatibility with older Python versions, there is no rush to remove
older intepreter support. Removing support for older versions should be well
justified in that the maintenance burden has become too high.
* **Python 2**. Python 2 is supported by version 2.x of this library. While active
development of the version 2.x series has ended, notable issues being corrected
may be backported to 2.x. Use "idna<3" in your requirements file if you need this
library for a Python 2 application.
* **Testing**. The library has a test suite based on each rule of the IDNA specification, as
well as tests that are provided as part of the Unicode Technical Standard 46,
`Unicode IDNA Compatibility Processing <https://unicode.org/reports/tr46/>`_.
* **Emoji**. It is an occasional request to support emoji domains in this library. Encoding
of symbols like emoji is expressly prohibited by the technical standard IDNA 2008 and
emoji domains are broadly phased out across the domain industry due to associated security
risks. For now, applications that wish need to support these non-compliant labels may
wish to consider trying the encode/decode operation in this library first, and then falling
back to using `encodings.idna`. See `the Github project <https://github.com/kjd/idna/issues/18>`_
for more discussion.
|
