| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rename `percpu.{cpp,h}` to `per_cpu.{cpp,h}` and update all includers.
Add an rseq-backed implementation of the hot per-CPU counter, time counter and
gauge (`rseq_sensor_impl.{h,cpp}`): each update commits to the calling CPU's shard
lock-free via an rseq critical section (`library/cpp/yt/rseq`
`AddPerCpu`/`StorePerCpu`) — no atomic and no lock on the fast path. The shard
array is sized to `NRseq::GetCpuCount()` and folded into the sensor allocation
(`NewWithExtraSpace`); reads aggregate with `LoadPerCpu`. The gauge keeps
last-writer-wins semantics. Linux-only.
The existing `TPerCpu{Counter,TimeCounter,Gauge}` stay (`per_cpu_sensor_impl.{h,cpp}`)
as the atomic sharded fallback. The two are interchangeable and chosen per sensor
at construction in `TSolomonRegistry`, so the hot Increment/Update path carries no
dispatch:
* The rseq fast path is **off by default**; opt in via
`singletons/solomon_registry/enable_rseq`. Even when on, a hot sensor uses it
only in a process where the kernel rseq area sits at a fixed thread-pointer
offset (tcmalloc/glibc-owned), per the rseq library's runtime safety probe
(`NRseq::IsPerCpuFastPathSafe`). Everything else — notably a `dlopen`'d YQL UDF
whose `__rseq_abi` lands in dynamically allocated TLS — uses the atomic sharded
sensors.
* `TSolomonRegistry` is a reconfigurable singleton (`solomon_registry`) with an
`enable_rseq` knob (default false), settable in static config and updatable via
dynamic config.
Off Linux the rseq sensors do not exist and the registry uses the atomic sharded
sensors for hot requests. The per-CPU summary is unchanged (TTscp + spinlock).
Unit tests cover the atomic sensors, the rseq sensors (Linux-only), and the simple
sensors. The controller-agent memory-watchdog integration tests are made resilient
to the (core-count-dependent) per-sensor footprint.
Benchmark (hot per-CPU path, 64-core host), atomic sharded vs rseq:
| Benchmark | atomic | rseq | speedup |
| --- | --- | --- | --- |
| BM_PerCpuCounter, threads:1 | 30.7 ns | 3.6 ns | ~8.5x |
| BM_PerCpuCounter, threads:16 | 30.9 ns | 4.4 ns | ~7x |
| BM_PerCpuGauge, threads:1 | 32.8 ns | 12.4 ns | ~2.6x |
| BM_PerCpuGauge, threads:16 | 32.7 ns | 12.5 ns | ~2.5x |
commit_hash:8c633d31f03b2cc862ed2217ae08342bf42adc52
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Hardens `library/cpp/yt/rseq` for the case where it is linked into a dlopen'd,
position-independent module (e.g. a YQL UDF `.so`). Extracted from the profiling
work that enables the rseq fast path by default.
**TLS model.** The weak `__rseq_abi` gets `global-dynamic` linkage under
`__PIC__/__PIE__` (`initial-exec` otherwise), mirroring `contrib/libs/tcmalloc`.
`initial-exec` needs a slot in the static TLS block reserved at startup, which
the loader cannot grant a module dlopen'd later — the module would fail to load
with "cannot allocate memory in static TLS block". This only changes the cold
`&__rseq_abi` accesses; the hot path still reads `*(thread_pointer + CpuIdFieldOffset)`.
**Runtime safety probe `IsPerCpuFastPathSafe()`.** The cached thread-pointer
offset is valid only when `__rseq_abi` sits at a fixed offset from the thread
pointer — a glibc-owned area or the static TLS block (incl. tcmalloc), the common
case. When our `__rseq_abi` instead lands in a dlopen'd module's *dynamically
allocated* TLS, the offset is valid only on the thread that computed it; on other
threads the hot path's first store (`area->rseq_cs`) would corrupt unrelated
memory. The probe spawns one thread and checks — by pointer comparison, never
dereferencing the suspect offset — that the offset names that thread's rseq area;
if not, callers use the atomic fallback. Decided once and cached (one thread spawn
at first use).=
commit_hash:633f58f500d9d097800da81f526c56283445ffc7
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce AddPerCpu and StorePerCpu over an rseq-sharded per-CPU array.
On the x86-64 Linux fast path the update is committed by a hand-rolled
rseq critical section (non-atomic, migration-safe): addq for the 8-byte
accumulate, movq / movdqu for the 8- or 16-byte store. The kernel
restarts the sequence on preemption or migration, and only one thread
runs on a CPU at a time, so no atomic or lock is needed. Off the fast
path (other arches, no kernel rseq) the operation falls back to an
atomic on the slot indexed by sched_getcpu().
A naturally-aligned 8-byte store is single-copy atomic on x86-64, so it
is never observed torn; the 16-byte store may be, which is acceptable for
a last-writer-wins gauge.
commit_hash:6250f6e9e35cf3895ebafe0b534ec12cca50b03b
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The own-area approach did not deliver the fast path on glibc 2.31 (YT's current
runtime). There tcmalloc registers the conventional `__rseq_abi` area for every
thread; our attempt to register a separate area was rejected by the kernel with
EINVAL (a thread may have only one rseq area), so `cpu_id` stayed -1 and every
`GetCurrentCpuId()` fell back to `sched_getcpu()` (~17-20 ns, slower than the
rdtscp it replaced).
Read the shared `__rseq_abi` symbol instead -- the area tcmalloc, librseq and
pre-2.35 glibc all register. Our definition is weak, so it coalesces with theirs
when present (the common case -- tcmalloc owns it) and stands alone otherwise
(e.g. musl), with us registering it. We register with the conventional signature
`0x53053053` and size 32, so re-registering an already-registered area returns
EBUSY (treated as success) rather than EINVAL -- coexisting cleanly with tcmalloc.
glibc >= 2.35 still takes the `__rseq_offset` path unchanged.
Measured on sas2-2769 (glibc 2.31 + tcmalloc): `GetCurrentCpuId()` 20.0 ns -> 0.60 ns,
verified via strace that our registration now returns EBUSY against tcmalloc's
`__rseq_abi` (was EINVAL against a separate area).
commit_hash:509809deeb5f7c671817fcd9ebcc8499eabf096e
|
|
|
Self-contained current-CPU-id reader backed by Linux **rseq** (restartable
sequences), with **no third-party dependency** (no librseq):
* The rseq ABI is hand-defined; the calling thread is registered lazily via the
rseq syscall.
* Fast path is a single inlined, **branch-free** thread-local read. The offset
always points at a readable `cpu_id` -- the glibc-owned area when glibc registers
rseq (>= 2.35, via the weak `__rseq_offset`/`__rseq_size`), otherwise our own
area -- so an unregistered thread reads `-1` and routes to the slow path.
* Falls back to `sched_getcpu()` (Linux) or `0` (darwin/windows). Works on glibc
**and musl** alike (librseq does not build on musl).
Fiber-TLS contract: the inlined read must be reached only via a non-inlinable,
fiber-switch-free frame (a virtual call or `YT_PREVENT_TLS_CACHING`).
#### Benchmark -- cost of one cpu-id read
| source | time / call |
|---|---|
| `GetCurrentCpuId()` (rseq) | **0.34 ns** |
| `sched_getcpu()` (vDSO) | 3.5 ns |
| `rdtscp` (what `TTscp::Get()` does) | 23 ns |
This is an alternative to the librseq-based review/13886037 -- same speed, but no
contrib dependency and it also covers musl. The unit test pins to each allowed CPU
and asserts the reported id matches.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
commit_hash:09d282c2f48755836b1cd68cedbffc3c6a662eed
|
