Cohere's LayerNorm gives x̂ = (h − mean(h)) / std(h), so x̂ ⟂ 1 and ‖x̂‖ = √d. Folding the gain γ into lm_head and keeping only the part of each row that x̂ can see:
w̃ᵢ = P⊥1(γ ⊙ wᵢ) − c, sᵢ = ‖w̃ᵢ‖, uᵢ = w̃ᵢ / sᵢ, giving
logitᵢ = logit_scale·√d · sᵢ · cos θᵢ (+ a term common to all tokens).
With d = 4 the unit sphere in the 3-dim subspace ⟂ 1 is an ordinary S², which is this picture.
Softmax's invariance to adding a constant to all logits is a freedom in parameters (the shift c), not in hidden space, so it does not cost a dimension of the sphere. It does mean "row norm" is only meaningful after fixing c (e.g. subtract the mean row).
The hard argmax fields are a multiplicatively weighted spherical Voronoi diagram: equal sᵢ gives ordinary Voronoi cells, and larger sᵢ inflates a cell. A token is reachable by greedy decoding iff its point sᵢuᵢ is a vertex of the convex hull of all {sⱼuⱼ}, which is gauge-invariant. Try σ ≈ 1 to see tokens with zero area.
β sets how soft the boundaries are; for Command-R 35B, 0.0625·√8192 ≈ 5.66 multiplies sᵢ.
Caveat: areas here assume hidden states uniformly distributed on the sphere. Real hidden states are concentrated, so for actual pruning, weight catchment by argmax counts or softmax mass over real calibration activations.