Base Released Apache-2.0
BananaMind-2-MoE
A sparse mixture-of-experts model: 25.1M total parameters, about 2.0M active per token.
BananaMind-2-MoE is a sparse decoder-only causal language model trained from scratch on a 30B-token curriculum. It routes each token to one of 48 experts per layer, so only about 1,985,152 parameters are active per token.
- Total parameters
- 25,086,592
- Active per token
- ~1,985,152
- Context length
- 4,096 tokens
Specification
Architecture
- Total parameters
- 25,086,592
- Active parameters per token
- approximately 1,985,152
- Architecture
- BananaMind2MoE decoder-only Transformer
- Routing
- Sparse top-1, no capacity drops
- Experts per layer
- 48
- Active experts per token
- 1
- Layers
- 8
- Hidden size
- 128
- Expert intermediate size
- 160
- Attention heads
- 4
- KV heads
- 2
- Head dimension
- 32
- Attention style
- Grouped-query attention with QK norm
- Expert MLP
- SwiGLU
- Position embeddings
- RoPE, theta 100,000
- Normalization
- RMSNorm
- Vocabulary size
- 8,192
- Context length
- 4,096
- Embeddings
- Tied input/output
- HF architecture
BananaMind2MoEForCausalLM
Active parameters count the tied embedding and output matrix, every attention and router parameter, normalization parameters, and one selected expert per layer.
Evaluation
Benchmark results
| Benchmark | Result |
|---|---|
| Average | 34.90 |
| ARC Easy | 34.64 |
| PIQA | 56.37 |
| ARC Challenge | 21.16 |
| HellaSwag | 27.45 |
Scores produced with lm_eval, all using acc_norm,none. The average is the mean of ARC Easy, PIQA, ARC Challenge, and HellaSwag.
Training
Tokenizer and training run
The same custom 8,192-token byte-level BPE tokenizer as BananaMind-2-Mini, with digit-aware pre-tokenization so numbers stay as separate digit tokens.
The 30B-token run used a progressive curriculum, with router balance and router z-losses added to the objective to keep expert usage spread out.
- Tokens seen
- 29,999,726,592
- Final optimizer step
- 55,485
- Sequence length
- 4,096
- Effective batch
- 132 sequences
- Tokens per optimizer step
- 540,672
- Optimizer
- AdamW, betas 0.9 / 0.95
- Peak learning rate
- 2.3e-3
- Router balance loss
- 0.01
- Router z-loss
- 1e-3
Getting started
Load BananaMind-2-MoE
This repository ships custom architecture files, so both the tokenizer and the model need
trust_remote_code=True. This is a base model: give it a continuation-style
prompt rather than an instruction.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-MoE"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))Licensing
License and use
BananaMind-2-MoE is released under Apache-2.0. Licenses differ between BananaMind models, so check the license on each Hugging Face repository before use. The model has not received dedicated safety alignment and can produce incorrect, biased, or otherwise undesirable text — do not rely on it for high-stakes decisions.