Base Released Apache-2.0
BananaMind-2-Nano
A 10M-parameter base model, the smallest released BananaMind 2.
BananaMind-2-Nano is a compact decoder-only causal language model trained from scratch on a 30B-token curriculum, with a 4,096-token context window and the same 8,192-token digit-aware tokenizer as BananaMind-2-Mini.
- Parameters
- 9,968,128
- Context length
- 4,096 tokens
- Training tokens
- 30.0B completed
Specification
Architecture
- Parameters
- 9,968,128
- Architecture
- BananaMind2Nano decoder-only Transformer
- Layers
- 10
- Hidden size
- 256
- Intermediate size
- 768
- Attention heads
- 4
- KV heads
- 2
- Head dimension
- 64
- Attention style
- Grouped-query attention with QK norm
- MLP
- SwiGLU
- Position embeddings
- RoPE, theta 100,000
- Normalization
- RMSNorm, epsilon 1e-6
- Vocabulary size
- 8,192
- Context length
- 4,096
- Embeddings
- Tied input/output
- Weight format
- safetensors
- HF architecture
BananaMind2NanoForCausalLM
Evaluation
Benchmark results
| Benchmark | Result |
|---|---|
| Average | 35.77 |
| ARC Easy | 36.20 |
| PIQA | 55.98 |
| ARC Challenge | 23.38 |
| HellaSwag | 27.50 |
Scores produced with lm_eval, all using acc_norm,none. The average is the mean of ARC Easy, PIQA, ARC Challenge, and HellaSwag.
Training
Tokenizer and training run
The same custom 8,192-token byte-level BPE tokenizer as BananaMind-2-Mini, with digit-aware pre-tokenization so numbers stay as separate digit tokens.
The 30B-token run used a progressive curriculum, starting web-heavy and gradually increasing synthetic textbook and mathematics data.
- Tokens seen
- 29,999,726,592
- Final optimizer step
- 55,485
- Sequence length
- 4,096
- Effective batch
- 132 sequences
- Tokens per optimizer step
- 540,672
- Optimizer
- AdamW, betas 0.9 / 0.95
- Peak learning rate
- 3e-3
- LR schedule
- Warmup-stable-decay with cosine decay
Getting started
Load BananaMind-2-Nano
This repository ships custom architecture files, so both the tokenizer and the model need
trust_remote_code=True. This is a base model: give it a continuation-style
prompt rather than an instruction.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-Nano"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))Instruction-tuned variant
BananaMind-2-Nano-Chat
The same 9,968,128-parameter architecture and 4,096-token context, instruction-tuned on SmolTalk. It scores 643 Elo overall on BananaMind Instruct Bench 1.1.
Instruct Bench and Base Bench are separate benchmarks with different tasks and scoring. Their Elo values are not directly comparable.
Licensing
License and use
BananaMind-2-Nano is released under Apache-2.0. Licenses differ between BananaMind models, so check the license on each Hugging Face repository before use. The model has not received dedicated safety alignment and can produce incorrect, biased, or otherwise undesirable text — do not rely on it for high-stakes decisions.