Security

bcrypt Stops Reading at Byte 72: The 18-Subkey Ceiling That Let a 52-Character Username Stand In for an Okta Password

bcrypt silently discards every input byte past 72 because Blowfish's P-array holds exactly 18 32-bit subkeys, and when Okta fed it the unseparated string userId + username + password as a cache key, any username of 52+ characters pushed the password entirely out of the hashed region - this walks the mechanism from the 1999 paper and the OpenBSD source (including the separate mod-256 length wraparound that forced the $2b$ version bump) to the 2024 advisory and the fixes that actually hold.

7 min read

All articles

LLM Infra & Ops

1,000 Prompts at Temperature 0, 80 Different Answers: LLM Nondeterminism Is Batch Variance, Not Floating-Point Chaos

The standard "floating point + concurrency" explanation for nondeterministic LLM outputs doesn't hold up - every kernel in the forward pass is run-to-run deterministic, and the real culprit is batch variance: your output is a function of everyone else's requests, which is why Thinking Machines' batch-invariant RMSNorm, matmul, and attention kernels turned 80 unique temperature-0 completions into 1 at roughly 1.6x the runtime.

6 min read

Open Source

590 Lines of Test Code per Line of Source: How SQLite Hit 100% MC/DC — and Why That Made It Easier to Fuzz

SQLite backs its 155.8 KSLOC core with ~590x as much test code across three independent harnesses - the avionics-grade TH3 delivers 100% branch and MC/DC coverage via a three-definition ALWAYS()/NEVER() macro trick and assembly-level mutation testing - yet the project documents that full MC/DC discourages defensive code and made fuzzing more effective, which is why most testing CPU cycles now go to the dbsqlfuzz fuzzer that mutates SQL and database-file bytes simultaneously.

7 min read

ML Engineering

13% More FLOPs, 5.7x Faster: How FlashAttention's Online Softmax Keeps the N×N Matrix Out of HBM

FlashAttention does more arithmetic than standard attention yet runs 5.7x faster on GPT-2 because attention is bound by HBM traffic, not FLOPs - the online-softmax running max and normalizer let the kernel rescale its output accumulator tile by tile, cutting HBM accesses from Θ(N²) to Θ(N²d²/M) and extra memory to O(N), with FlashAttention-2 then doubling throughput by fixing what I/O-optimality left on the table.

6 min read

DevOps & Cloud

The Kernel Discarded Your SIGTERM: Why PID 1 Has No Default Signal Handlers, and the 10-Second docker stop Hang

Inside a PID namespace the kernel only delivers signals to PID 1 if a handler is installed (SIGKILL/SIGSTOP from the ancestor namespace excepted), so an app relying on default dispositions silently ignores docker stop's SIGTERM until the 10-second SIGKILL - and shell-form ENTRYPOINTs, orphan reaping, and tini's signal-forwarding single-child design all follow from that one rule.

6 min read

Engineering

Two write() Calls, One 40 ms Stall: How Nagle's Algorithm and Delayed ACKs Deadlock Until a Kernel Timer Fires

Nagle's algorithm holds small segments until the last one is ACKed, delayed ACKs hold that ACK up to 500 ms hoping to piggyback it on response data, and a write-write-read application wedges both heuristics against each other until Linux's 40 ms TCP_DELACK_MIN timer fires - which is why TCP_NODELAY is a MUST-exist opt-out and the first thing to check in a latency investigation.

7 min read

Systems Programming

The Kernel Has Never Heard of Your Mutex: Futexes, the Three-State Lock Word, and the Livelock That Hid in NPTL for Months

An uncontended pthread mutex is one atomic instruction to lock and one to unlock, with the kernel keeping no record it exists - getting there requires FUTEX_WAIT's compare-under-lock, a third lock-word state so unlock knows when to skip the syscall, and FUTEX_CMP_REQUEUE, which moves broadcast waiters to the mutex queue instead of waking N threads so N-1 can go straight back to sleep.

7 min read

LLM Infra & Ops

Only 20.4% of Your KV Cache Held Actual Tokens: How PagedAttention Put a Page Table Inside the Attention Kernel

Pre-vLLM serving systems stored each request's KV cache as one contiguous max-length tensor, leaving only 20.4%-38.2% of that memory holding real token states - PagedAttention maps logical 16-token blocks to arbitrary physical blocks through a per-request block table, capping waste below one block per sequence (96.3% utilization), enabling refcounted copy-on-write sharing across samples and beams, and buying 2-4x throughput despite a 20-26% slower attention kernel.

7 min read

ML Engineering

Your 7.5B Model's Weights Fit in 15 GB. Training State Was 120 GB per GPU: The 16-Byte Parameter and ZeRO's Three-Stage Split

Mixed-precision Adam carries 16 bytes of state per parameter - 12 of them fp32 optimizer state that can't be cut because fp16 weight updates underflow or round away - so ZeRO (and PyTorch FSDP) partitions optimizer states, gradients, and parameters across data-parallel ranks, taking a 7.5B model from 120 GB to 1.88 GB per GPU while stages 1-2 add zero communication and stage 3 costs 1.5x.

6 min read

Open Source

The 9-Byte Blob That Points Backward: How Git Packfiles Turn Full Snapshots Into Reverse Deltas

Git stores every commit as full zlib-compressed snapshots until repacking, when a sliding window of 10 size-sorted objects finds delta pairs and stores the newest version whole with older versions as reverse deltas - tiny copy/insert programs capped at depth 50 - which is why an appended line costs 9 bytes and why big or pre-compressed files never shrink.

6 min read

Engineering

Count 2^64 Uniques in 12 KB: HyperLogLog's Harmonic-Mean Trick, and the Bias Table Google Ships Because 99% of Counts Are Under 100

HyperLogLog estimates distinct counts up to 2^64 within 0.81% using 16,384 six-bit registers (12 KB) by tracking leading-zero runs and taming outliers with a harmonic mean - but both Google's HLL++ and Redis had to patch the small-count regime, where 99% of PowerDrill's 5 million daily count-distincts live, with empirical bias tables, LinearCounting switches, and sparse encodings.

7 min read

Systems Programming

mmap Matched Direct I/O for 27 Seconds, Then the Page Cache Filled: Why Databases Keep Hiring and Firing the Kernel as Their Buffer Pool

The CIDR 2022 mmap paper shows memory-mapped file I/O tracking an O_DIRECT baseline right up until page eviction begins — then TLB shootdowns, a CPU-bound single-threaded kswapd, and page-table contention cut throughput 2-20x — while the correctness story is worse: the OS may flush dirty pages mid-transaction even under mlock, I/O errors arrive as SIGBUS, and every system that kept mmap ended up rebuilding the buffer pool it was meant to replace.

7 min read

AI Agents

Three Reasonable Permissions, One Exfiltration Machine: The Lethal Trifecta, and Why Google's Fix Is an Interpreter, Not a Classifier

Every documented agent data-leak follows Simon Willison's lethal trifecta — private data access, untrusted content, and an outbound channel — and because '95% of attacks caught' is a failing grade against free retries, Google DeepMind's CaMeL abandons detection for structure: a privileged LLM compiles the plan to restricted Python before any untrusted byte is read, a toolless quarantined LLM extracts schema-typed values, and a capability-tracking interpreter enforces policies on tainted data, buying provable security on AgentDojo at a 7-point utility cost (77% vs 84%).

7 min read

Databases

Two $900 Withdrawals, Zero Row Conflicts: Write Skew, and the Two-Edge Pattern Postgres Hunts Instead of Cycles

Snapshot isolation lets two withdrawals with disjoint write sets overdraw a shared balance because first-committer-wins never fires, so PostgreSQL 9.1's SSI instead watches for the dangerous structure Fekete et al. proved sits inside every cycle — two adjacent rw-antidependencies with the pivot's successor committing first — tracked via non-blocking SIReadLocks that outlive commit, at a measured cost under 7% versus snapshot isolation while two-phase locking lost half its throughput.

7 min read

ML Engineering

Your Loss Gets Multiplied by 65,536 Before Every Backward Pass: Dynamic Loss Scaling, and the Bit Layout That Lets bf16 Skip It

fp16's entire dynamic range spans 40 powers of 2 with a floor at 2^-24 - below which 67% of SSD's activation gradients already sat at zero - so PyTorch's GradScaler multiplies the loss by 65,536, halves the scale on every inf and doubles it after 2,000 clean steps, skipping optimizer updates along the way, while bf16's 8 fp32-matched exponent bits make the whole apparatus unnecessary at the cost of an 8x coarser mantissa.

6 min read

LLM Infra & Ops

It's Not “Accept If They Match”: The min(1, p/q) Rule That Makes Speculative Decoding Provably Lossless

Speculative decoding accepts a draft token with probability min(1, p/q) and reroutes rejections to the residual distribution norm(max(0, p−q)), which provably reproduces the target model's exact output distribution for any draft — the acceptance-rate algebra (α = E[min(p,q)], speedup requires α > c) and Chinchilla-70B measurements (2.46× on code vs 1.92× on summarization) then decide whether the trick pays, and vLLM's docs show the gain shrinking once high-QPS batching eats the idle compute it depends on.

6 min read

Systems Programming

You Freed 200 MB. RSS Dropped 1.5 MB: Where glibc malloc Keeps Freed Memory and Who Can Take It Back

free() returns memory to glibc, not the kernel — automatic trimming only fires when contiguous free space sits at the top of the sbrk heap, the mmap threshold silently slides from 128 KiB up to 32 MiB based on your free() history, and 8-arenas-per-core multiplies the retention in threaded programs — which is why a demo on glibc 2.39 frees 200 MB and recovers 1.5 MB of RSS until a single malloc_trim(0) call releases it all.

7 min read

DevOps & Cloud

One curl, Eight DNS Queries: The Four-Dot Name That Forces Kubernetes to Ship ndots:5

Kubernetes writes ndots:5 into every pod's resolv.conf, so a two-dot name like api.stripe.com walks the whole search list first — 8 queries (10 with a cloud node's extra search domain, since glibc fires A and AAAA in parallel), most of them guaranteed NXDOMAINs — yet the value is no accident: the DNS spec's SRV form _port._proto.service.ns.svc carries four dots, making 5 the smallest threshold that keeps relative service names resolving.

6 min read

Engineering

Karger Needed 1,000 Points per Server. Google Needed Five Lines: Consistent Hashing From the Ring to Jump Hash

The classic consistent hashing ring needs about 1,000 virtual nodes per server just to hold bucket sizes within 8% (costing 46 MB of routing state per client at 1,000 buckets), Dynamo's production numbers showed 10-20% of nodes out of load balance before it retreated to fixed equal partitions, and Google's jump consistent hash replaces the whole structure with five lines and near-perfect balance at 65 ns - as long as you never need to remove a bucket from the middle.

7 min read

Open Source

Bit-for-Bit Identical, Bit-for-Bit Backdoored: What Reproducible Builds Prove — and Why xz Passed Anyway

Reproducible builds pin a binary to its source by eliminating timestamps, build paths, and environment leakage — SOURCE_DATE_EPOCH, Go 1.21's perfectly reproducible toolchain, and Debian's 94-95% adversarial double-build rate show it works — yet the xz backdoor lived in a tarball that differed from git, so every rebuild reproduced the backdoor faithfully, exposing the second, missing link: source-to-reviewed-history verification.

6 min read

Security

The Token Dies One Hop From Home: How IMDSv2 Stacks a PUT, a Banned Header, and TTL=1 Against SSRF

IMDSv2 defends EC2 instance credentials not with cryptography but with three cheap asymmetries — a session that must open with an HTTP PUT most proxies and WAFs won't forward, outright rejection of any token request carrying X-Forwarded-For, and a response packet whose IP TTL of 1 physically cannot survive a network hop — yet Datadog measured ~93% of instances still not enforcing it three years after launch.

6 min read

AI Agents

The Logit Mask Is Free. Your Key Order Isn't: How Constrained Decoding Works and the Schema Mistake That Cost 63 Points on GSM8K

Structured outputs work by compiling a JSON Schema into a finite-state or pushdown automaton and masking invalid tokens at every step for O(1) average per-token cost — a mechanism so cheap that the 86.51%-to-23.44% GSM8K collapse measured under JSON mode traces not to the mask but to schemas that place the answer key before the reasoning key, converting chain-of-thought into zero-shot direct answering.

7 min read

Databases

Power Failed at Sector 9 of 16: Why InnoDB Writes Every Page Twice and Postgres Photographs Pages Into the WAL

A multi-sector page write can tear mid-crash into a mix of old and new bytes that no redo log can repair, so InnoDB stages every flushed page in a doublewrite buffer (~5-10% steady cost) while PostgreSQL logs a full page image on the first change after each checkpoint (a WAL sawtooth tuned by checkpoint distance and wal_compression) — and copy-on-write filesystems like ZFS or atomic 16 KiB writes on RDS Optimized Writes make both mechanisms droppable, worth up to 2x write throughput.

7 min read

ML Engineering

Gradient Accumulation Was Never Equivalent: The Mean-of-Means Bug That Overweighted Short Sequences 9x

Gradient accumulation is only equivalent to large-batch training when every micro-batch averages over the same number of items — a precondition variable-length causal LM batches violate, which is why transformers silently mis-weighted tokens (9x in a simple two-micro-batch example) until the October 2024 num_items_in_batch fix in v4.46.0 made losses sum first and divide by the whole window's real-token count, with an accelerator.gather of that count fixing the same mean-of-means mismatch across DDP ranks.

6 min read

LLM Infra & Ops

Your Batch Size Changes Every Token: Continuous Batching's 23x Win and the Long Prompt That Stalls Every Stream

Continuous batching schedules LLM inference per iteration instead of per request — worth 23x throughput in Anyscale's OPT-13B benchmark and 36.9x in the Orca paper — but mixing compute-saturating prefills with one-token decode steps lets a single long prompt stall every active stream, which is why SARATHI's chunked prefill and vLLM V1's decode-first scheduler bound iteration time and turn max_num_batched_tokens into the TTFT-versus-ITL dial.

6 min read

Systems Programming

fork() Took 6 ms. The Latency Spike Was 300 ms: Copy-on-Write's Deferred Bill

fork() on Linux copies only page tables — 48 MB for a 24 GB process — so the real cost of fork-based snapshotting arrives afterward as copy-on-write page faults, which transparent huge pages amplify 512x per write and overcommit accounting can veto entirely, as Redis's production guidance and antirez's 339 ms-vs-21 ms allocator measurements show.

7 min read

DevOps & Cloud

Your Pod Used Half a CPU of Its Four-CPU Limit and Got Throttled Anyway: Inside the 100 ms Window of CFS Bandwidth Control

A Kubernetes CPU limit is not a cap on average usage but a budget that refills every 100 ms and is burned by all threads in parallel — which is why multithreaded services get hard-parked at low utilization, why a years-old slice-expiration kernel bug made it far worse until Dave Chiluk's fix, and why nr_throttled/nr_periods is the metric to watch instead of utilization.

7 min read

Engineering

Subtracting Two Timestamps Gave −990 ms: The Leap Second That Panicked Cloudflare DNS and Put a Second Clock Inside time.Time

Wall clocks answer "what time is it?" and can step backwards — which turned time.Now().Sub(start) into a −990 ms round-trip time during the 2016 leap second, panicked Cloudflare's RRDNS via rand.Int63n, and led Go 1.9 to pack a monotonic clock reading inside every time.Time, with documented seams (serialization, Round(0), ==) where that protection silently disappears.

7 min read

Security

bcrypt Stops Reading Your Password at Byte 72 — and It Let People Into Okta Without One

bcrypt's key schedule XORs input into just 18 32-bit subkeys — 72 bytes of capacity, everything after silently ignored — which is structural in the 1999 eksblowfish design, is handled differently by PHP, Go, and pyca/bcrypt today, and turned into a real authentication bypass when Okta hashed userId + username + password as a cache key and usernames of 52+ characters pushed the password past the cliff.

6 min read

Open Source

SQLite Ships 590 Lines of Test Code per Line of Library — and the Suite That Gates Releases Isn't Open Source

SQLite pairs 155.8 KSLOC of public-domain C with 92 million lines of tests across four independent harnesses — TCL, the DO-178B-grade proprietary TH3 with 100% MC/DC coverage, SQL Logic Test, and a billion-mutations-a-day fuzzer — and that proprietary test machinery, not licensing philosophy, is the structural reason SQLite is open-source but not open-contribution.

6 min read

Databases

Your Primary Key Chooses Which Pages Every Insert Touches: UUIDv4, Page Splits, and the UUIDv7 Fix

UUIDv4 primary keys turn every B-tree insert into a random page touch — half-empty pages, a working set the size of the whole index, and a collapse to ~2 IOPs per insert once the tree outgrows the buffer pool — which is exactly what RFC 9562's timestamp-prefixed UUIDv7 (now native in PostgreSQL 18) fixes, at the documented cost of leaking creation time and losing entropy.

7 min read

AI Agents

Your Agent Resends the Whole Conversation Every Turn. One Edited Byte Makes It Pay Full Price

Agent loops re-send their entire history on every step, and prefix caching's exact-match, byte-for-byte contract — enforced from vLLM's parent-chained block hashes and SGLang's radix tree up to Anthropic's and OpenAI's 0.1x cached-token pricing — is why append-only transcripts, frozen tool lists, and volatile-data-last are now first-order agent design rules.

6 min read

ML Engineering

You Seeded Everything. 1,000 Training Runs Still Gave 1,000 Different Models

Fixed seeds don't make GPU training reproducible: atomicAdd commits floating-point sums in scheduler-dependent order, cuDNN and cuBLAS can swap algorithms between runs, and an Oak Ridge study that isolated a single index_add op still got 1,000 unique weight sets from 1,000 runs — here's what torch.use_deterministic_algorithms actually buys, and what it costs.

6 min read

LLM Infra & Ops

One Token, 14 GB of Reads: Why LLM Decode Speed Is a Bandwidth Spec, Not a TFLOPS Spec

Single-stream LLM decode runs matrix-vector math at ~1-2 ops per byte on GPUs that need ~208 to stay busy, so time-per-token is just model bytes divided by memory bandwidth — one formula that explains quantization speedups, speculative decoding, the ~208/240 batch-size crossover, and why the KV cache keeps you memory-bound anyway.

6 min read

Systems Programming

Locked, But Is Anyone Waiting? The Three-State Futex Trick Behind Every Linux Mutex

Linux mutexes stay out of the kernel entirely until threads collide, thanks to the futex compare-and-block syscall and a deceptively subtle three-state protocol (0/1/2) from Drepper's 'Futexes Are Tricky' — whose naive version livelocked NPTL for months and cost one four-CPU application an 8-10x slowdown.

7 min read

DevOps & Cloud

One Click, 64 Queries: Why Backoff Can't Stop a Retry Storm but a Token Bucket Can

Layered retries turn one failed request into 64 database attempts, and even perfectly jittered exponential backoff only defers that load — which is why Google caps retries with per-process budgets and AWS SDKs price every retry against a 500-token bucket that fails fast during sustained outages.

7 min read

Engineering

Two Correct Algorithms, One 40ms Stall: How Nagle and Delayed ACK Deadlock Each Other

Nagle's algorithm (RFC 896) and delayed ACKs (RFC 1122) are each sensible bets on their own, but on opposite ends of a write-write-read exchange they wait on each other until Linux's 40-200ms delayed-ACK timer fires — which is why TCP_NODELAY fixes so many latency mysteries and why John Nagle himself says to set TCP_QUICKACK instead.

7 min read

Security

SameSite=Lax Did Not Kill CSRF: The Cross-Site vs Cross-Origin Gap

SameSite=Lax blocks the naive cross-site form POST but not CSRF itself — GET-based state changes, sibling-domain positions, client-side redirect gadgets, and Chrome's 2-minute Lax+POST window all survive, which is why tokens and Fetch Metadata still matter.

5 min read

Databases

The Second fsync() Always Succeeds: How fsyncgate Made PostgreSQL Choose to Crash

In 2018 PostgreSQL discovered that Linux marks pages clean and forgets the error after a failed fsync(), so retrying returns a false success and truncates the WAL over lost data — which is why Postgres now PANICs on fsync failure, and why a 2020 USENIX study found every major database mishandled the same case.

7 min read