Kawrakow ac1d259b93 Fused MoE ffn_up and ffn_gate (#229) ... * Fusing MoE up * unary(gate) * Fusing MoE up * unary(gate): CUDA We get ~13% speedup for PP-512 and ~2% for TG-128 for DeepSeek-Lite * On CUDA also fuse MoE down * (up * unary(gate)) in case the MUL_MAT_ID op for the down experts is the next op in the graph. * Command line option to enable fused MoE up*unary(gate) * Add fmoe option to llama-bench * Adding forgotten gelu, relu, silu on ARM --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>		2025-02-23 14:31:11 +02:00
..
llama.h	Fused MoE ffn_up and ffn_gate (#229)	2025-02-23 14:31:11 +02:00