Embedding
Attention
MLP
Output
Pulse Atom — 27,840 params
43
Vocab
32
n_embd
2
Layers
Hover or click a parameter block. Drag to orbit, scroll to zoom.
core/atom.py — forward() is 34 linesdef forward(self, token_id, pos_id, keys, values):
sd = self.sd
x = [t + p for t, p in zip(
sd['wte'][token_id],
sd['wpe'][pos_id])]
x = self._rmsnorm(x)
for li in range(self.n_layer):
... # attention + MLP
return self._linear(x, sd['lm_head'])