Mixture-of-Experts Token Routing50 layers · 384 experts + 1 shared per layer · top-6 router · 78B total / ~3.46B active

Layer

1 / 50

dim expert heatmap (accumulated usage) top-6 routed expert shared expert

Active experts (this layer)

7 of 385

Active parameters (per token)

~3.46B of 78B

4.4% of weights compute per token

Expert activation heatmap

0 expert activations so far

Brighter purple cells = used more often across all tokens & layers.

Router: deterministic pseudo-random selection seeded by token text + layer number (FNV-1a + LCG), so the same token always takes the same 6 experts in a given layer. The shared expert fires for every token. Each FFN layer holds 385 experts; only 7 are active → 6 routed + 1 shared ≈ 3.46B / 78B ≈ 4.4% active parameters.