aboutsummaryrefslogtreecommitdiffstats
path: root/contrib/python/mypy-protobuf/README.md
blob: 19ecd5824aa5cd94c57a847c417b868c75e2f3b5 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
mypy-protobuf: Generate mypy stub files from protobuf specs

[![CI](https://github.com/dropbox/mypy-protobuf/workflows/CI/badge.svg)](https://github.com/dropbox/mypy-protobuf/actions?query=branch%3Amain)
[![pypi](https://img.shields.io/pypi/v/mypy-protobuf?logo=Pypi)](https://pypi.org/project/mypy-protobuf/)
[![license](https://img.shields.io/github/license/dropbox/mypy-protobuf)](https://github.com/dropbox/mypy-protobuf/blob/main/LICENSE)
===========================================================

Starting in mypy-protobuf 3.0.0, only python3 targeting is supported
2.10 is the last version of mypy-protobuf which supports targeting python 2.7.

See [Changelog](CHANGELOG.md) for recent changes.

## Requirements to run mypy-protobuf
Earlier releases might work, but aren't tested
- [protoc >= 3.19.3](https://github.com/protocolbuffers/protobuf/releases)
- [python-protobuf >= 3.19.3](https://pypi.org/project/protobuf/) - matching protoc release
- [python >= 3.7](https://www.python.org/downloads/source/) - for running mypy-protobuf plugin.

## Requirements to run typecheckers on stubs generated by mypy-protobuf
Earlier releases might work, but aren't tested
- [mypy >= v0.931](https://pypi.org/project/mypy) or [pyright >= 1.1.206](https://github.com/microsoft/pyright)
- [python-protobuf >= 3.19.3](https://pypi.org/project/protobuf/) - matching protoc release
- [types-protobuf >= 3.19.5](https://pypi.org/project/types-protobuf/) - for stubs from the google.protobuf library

### To run typecheckers on code generated with grpc plugin - you'll additionally need
Earlier releases might work, but aren't tested
- [grpcio>=1.40.0](https://pypi.org/project/grpcio/)
- [grpcio-tools>=1.40.0](https://pypi.org/project/grpcio-tools/)
- [grpc-stubs>=1.24.7](https://pypi.org/project/grpc-stubs/)

Other configurations may work, but are not continuously tested currently.
We would be open to expanding this list - file an issue on the issue tracker.

## Installation

The plugin can be installed with
```
pip3 install mypy-protobuf
```
To install unreleased
```
REV=main  # or whichever unreleased git rev you'd like
pip3 install git+https://github.com/dropbox/mypy-protobuf.git@$REV

# For older (1.x) versions of mypy protobuf - you may need
pip3 install git+https://github.com/dropbox/mypy-protobuf.git@$REV#subdirectory=python
```

In order to run mypy on the generated code, you'll need to install
```
pip3 install mypy>=0.910 types-protobuf>=0.1.14
```

# Usage

On posix, protoc-gen-mypy is installed to python's executable bin. Assuming that's
on your $PATH, you can run
```
protoc --python_out=output/location --mypy_out=output/location
```
Alternately, you can explicitly provide the path:
```
protoc --plugin=protoc-gen-mypy=path/to/protoc-gen-mypy --python_out=output/location --mypy_out=output/location
```
Check the version number with
```
> protoc-gen-mypy --version
```

## Implementation

The implementation of the plugin is in `mypy_protobuf/main.py`, which installs to
an executable protoc-gen-mypy. On windows it installs to `protoc-gen-mypy.exe`

## Features

See [Changelog](CHANGELOG.md) for full listing

### Bring comments from .proto files to docstrings in .pyi files

Comments in the .proto files on messages, fields, enums, enum variants, extensions, services, and methods
will appear as docstrings in .pyi files. Useful in IDEs for showing completions with comments.

### Types enum int values more strongly

Enum int values produce stubs which wrap the int values in NewType
```proto
enum MyEnum {
  HELLO = 0;
  WORLD = 1;
}
```
Will yield an [enum type wrapper](https://github.com/python/typeshed/blob/16ae4c61201cd8b96b8b22cdfb2ab9e89ba5bcf2/stubs/protobuf/google/protobuf/internal/enum_type_wrapper.pyi) whose methods type to `MyEnum.ValueType` (a `NewType(int)` rather than `int`.
This allows mypy to catch bugs where the wrong enum value is being used.

Calling code may be typed as follows.

In python >= 3.7
```python
# May need [PEP 563](https://www.python.org/dev/peps/pep-0563/) to postpone evaluation of annotations
# from __future__ import annotations  # Not needed with python>=3.10 or protobuf>=3.20.0
def f(x: MyEnum.ValueType):
    print(x)
f(MyEnum.Value("HELLO"))
```

With protobuf <= 3.20.0, for usages of cast, the type of `x` must be quoted
After protobuf >= 3.20.0 - `ValueType` exists in the python code and quotes aren't needed
until [upstream protobuf](https://github.com/protocolbuffers/protobuf/pull/8182) includes `ValueType`
```python
cast('MyEnum.ValueType', x)
```

Similarly, for type aliases with protobuf < 3.20.0, you must either quote the type or hide it behind `TYPE_CHECKING`
```python
from typing import Tuple, TYPE_CHECKING
HELLO = Tuple['MyEnum.ValueType', 'MyEnum.ValueType']
if TYPE_CHECKING:
    HELLO = Tuple[MyEnum.ValueType, MyEnum.ValueType]
```

#### Enum int impl details

mypy-protobuf  autogenerates an instance of the EnumTypeWrapper as follows.

```python
class _MyEnum:
    ValueType = typing.NewType('ValueType', builtins.int)
    V: typing_extensions.TypeAlias = ValueType
class _MyEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_MyEnum.ValueType], builtins.type):
    DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
    HELLO: _MyEnum.ValueType  # 0
    WORLD: _MyEnum.ValueType  # 1
class MyEnum(_MyEnum, metaclass=_MyEnumEnumTypeWrapper):
    pass

HELLO: MyEnum.ValueType  # 0
WORLD: MyEnum.ValueType  # 1
```

`_MyEnumEnumTypeWrapper` extends the EnumTypeWrapper to take/return MyEnum.ValueType rather than int
`MyEnum` is an instance of the `EnumTypeWrapper`.
- Use `_MyEnum` and of metaclass is an implementation detail to make MyEnum.ValueType a valid type w/o a circular dependency
- `V` is supported as an alias of `ValueType` for backward compatibility



### Supports generating type wrappers for fields and maps

M.proto
```proto
message M {
  uint32 user_id = 1 [(mypy_protobuf.casttype)="mymod.UserId"];
  map<uint32, string> email_by_uid = 2 [
    (mypy_protobuf.keytype)="path/to/mymod.UserId",
    (mypy_protobuf.valuetype)="path/to/mymod.Email"
  ];
}
```
mymod.py
```python
UserId = NewType("UserId", int)
Email = NewType("Email", Text)
```

### `py_generic_services`
If `py_generic_services` is set in your proto file, then mypy-protobuf will
generate service stubs. If you want GRPC stubs instead - use the GRPC instructions.

### `readable_stubs`
If `readable_stubs` is set, mypy-protobuf will generate easier-to-read stubs. The downside
to this approach - is that it's possible to generate stubs which do not pass mypy - particularly
in the case of name collisions. mypy-protobuf defaults to generating stubs with fully qualified
imports and mangled global-level identifiers to defend against name collisions between global
identifiers and field names.

If you're ok with this risk, try it out!
```
protoc --python_out=output/location --mypy_out=readable_stubs:output/location
```

### `relax_strict_optional_primitives`

If you are using proto3, then primitives cannot be represented as NULL on the wire -
only as their zero value. By default mypy-protobuf types message constructors to have
non-nullable primitives (eg `int` instead of `Optional[int]`). python-protobuf itself will
internally convert None -> zero value. If you intentionally want to use this behavior,
set this flag! We recommend avoiding this, as it can lead to developer error - confusing
NULL and 0 as distinct on the wire.
However, it may be helpful when migrating existing proto2 code, where the distinction is meaningful

```
protoc --python_out=output/location --mypy_out=relax_strict_optional_primitives:output/location
```

### Output suppression
To suppress output, you can run
```
protoc --python_out=output/location --mypy_out=quiet:output/location
```

### GRPC

This plugin provides stubs generation for grpcio generated code.
```
protoc \
    --python_out=output/location \
    --mypy_out=output/location \
    --grpc_out=output/location \
    --mypy_grpc_out=output/location
```

Note that generated code for grpc will work only together with code for python and locations should be the same.
If you need stubs for grpc internal code we suggest using this package https://github.com/shabbyrobe/grpc-stubs 

### Targeting python2 support

mypy-protobuf's drops support for targeting python2 with version 3.0. If you still need python2 support -
```
python3 -m pip install mypy_protobuf==2.10
protoc --python_out=output/location --mypy_out=output/location
mypy --target-version=2.7 {files}
```


## Contributing
Contributions to the implementation are welcome. Please run tests using `./run_test.sh`.
Ensure code is formatted using black.
```
pip3 install black
black .
```

## Contributors

### Dropboxers
- [@nipunn1313](https://github.com/nipunn1313)
- [@dzbarsky](https://github.com/dzbarsky)
- [@gvanrossum](https://github.com/gvanrossum)
- [@peterlvilim](https://github.com/peterlvilim)
- [@msullivan](https://github.com/msullivan)
- [@bradenaw](https://github.com/bradenaw)
- [@ilevkivskyi](https://github.com/ilevkivskyi)

### Others
- [@Ketouem](https://github.com/Ketouem)
- [@nmiculinic](https://github.com/nmiculinic)
- [@onto](https://github.com/onto)
- [@jcppkkk](https://github.com/jcppkkk)
- [@drather19](https://github.com/drather19)
- [@smessmer](https://github.com/smessmer)
- [@pcorpet](https://github.com/pcorpet)
- [@zozoens31](https://github.com/zozoens31)
- [@abhishekrb19](https://github.com/abhishekrb19)
- [@jaens](https://github.com/jaens)
- [@arussellsaw](https://github.com/arussellsaw)
- [@shabbyrobe](https://github.com/shabbyrobe)
- [@reorx](https://github.com/reorx)
- [@zifter](https://github.com/zifter)
- [@juzna](https://github.com/juzna)
- [@mikolajz](https://github.com/mikolajz)
- [@chadrik](https://github.com/chadrik)
- [@EPronovost](https://github.com/EPronovost)
- [@chrislawlor](https://github.com/chrislawlor)
- [@henribru](https://github.com/henribru)
- [@Evgenus](https://github.com/Evgenus)
- [@MHDante](https://github.com/MHDante)
- [@nelfin](https://github.com/nelfin)
- [@alkasm](https://github.com/alkasm)
- [@tarmath](https://github.com/tarmath)
- [@jaredkhan](https://github.com/jaredkhan)
- [@sodul](https://github.com/sodul)

Licence etc.
------------

1. License: Apache 2.0.
2. Copyright attribution: Copyright (c) 2017 Dropbox, Inc.
3. External contributions to the project should be subject to
   Dropbox's Contributor License Agreement (CLA):
   https://opensource.dropbox.com/cla/