Mixture of Experts
Definition
MoE. A transformer architecture routing inputs dynamically to specialized sub-networks (experts) during inference.
Example Case
Mixtral 8x7B routing tokens to 2 of 8 experts per forward pass.
MoE. A transformer architecture routing inputs dynamically to specialized sub-networks (experts) during inference.