Base Released Apache-2.0
BananaMind-2-Mini
The first model in the BananaMind 2 series, at 25.2M parameters.
BananaMind-2-Mini is a small decoder-only causal language model trained from scratch on a 30B-token curriculum, with a 4,096-token context window and a custom 8,192-token digit-aware byte-level BPE tokenizer.
- Parameters
- 25,178,752
- Context length
- 4,096 tokens
- Training tokens
- 30.0B completed
Specification
Architecture
- Parameters
- 25,178,752
- Architecture
- BananaMind2Mini decoder-only Transformer
- Layers
- 14
- Hidden size
- 384
- Intermediate size
- 1,024
- Attention heads
- 6
- KV heads
- 2
- Head dimension
- 64
- Attention style
- Grouped-query attention with QK norm
- MLP
- SwiGLU
- Position embeddings
- RoPE, theta 100,000
- Normalization
- RMSNorm, epsilon 1e-6
- Vocabulary size
- 8,192
- Context length
- 4,096
- Embeddings
- Tied input/output
- Weight format
- safetensors
- HF architecture
BananaMind2MiniForCausalLM
Evaluation
Benchmark results
| Benchmark | Result |
|---|---|
| Average | 38.72 |
| ARC Easy | 39.86 |
| PIQA | 59.63 |
| ARC Challenge | 25.68 |
| HellaSwag | 29.72 |
Scores produced with lm_eval, all using acc_norm,none. The average is the mean of ARC Easy, PIQA, ARC Challenge, and HellaSwag.
Training
Tokenizer and training run
A custom 8,192-token byte-level BPE tokenizer trained from FineWeb-Edu text with digit-aware pre-tokenization, so numbers do not collapse into large number tokens.
The 30B-token run used a progressive curriculum, starting web-heavy and gradually increasing synthetic textbook and mathematics data.
- Tokens seen
- 29,999,726,592
- Final optimizer step
- 55,485
- Sequence length
- 4,096
- Effective batch
- 132 sequences
- Tokens per optimizer step
- 540,672
- Optimizer
- AdamW, betas 0.9 / 0.95
- Peak learning rate
- 2.3e-3
- LR schedule
- Warmup-stable-decay with cosine decay
Getting started
Load BananaMind-2-Mini
This repository ships custom architecture files, so both the tokenizer and the model need
trust_remote_code=True. This is a base model: give it a continuation-style
prompt rather than an instruction.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-Mini"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))Instruction-tuned variant
BananaMind-2-Mini-Chat
The same 25,178,752-parameter architecture and 4,096-token context, instruction-tuned on SmolTalk. It scores 654 Elo overall on BananaMind Instruct Bench 1.1.
Instruct Bench and Base Bench are separate benchmarks with different tasks and scoring. Their Elo values are not directly comparable.
Licensing
License and use
BananaMind-2-Mini is released under Apache-2.0. Licenses differ between BananaMind models, so check the license on each Hugging Face repository before use. The model has not received dedicated safety alignment and can produce incorrect, biased, or otherwise undesirable text — do not rely on it for high-stakes decisions.