Base Released Apache-2.0
BananaMind-2-Medium
A 49.6M-parameter base model trained on a 50B-token curriculum.
BananaMind-2-Medium is a decoder-only causal language model trained from scratch on a 50B-token curriculum, with a 3,072-token context window and a custom 12,288-token digit-aware byte-level BPE tokenizer.
- Parameters
- 49,559,552
- Context length
- 3,072 tokens
- Training tokens
- 50.0B completed
Specification
Architecture
- Parameters
- 49,559,552
- Architecture
- BananaMind2Medium decoder-only Transformer
- Layers
- 12
- Hidden size
- 512
- Intermediate size
- 1,920
- Attention heads
- 8
- KV heads
- 2
- Head dimension
- 64
- Attention style
- Grouped-query attention with QK norm
- MLP
- SwiGLU
- Position embeddings
- RoPE, theta 100,000
- Normalization
- RMSNorm, epsilon 1e-6
- Vocabulary size
- 12,288
- Context length
- 3,072
- Embeddings
- Tied input/output
- Weight format
- safetensors
- HF architecture
BananaMind2MediumForCausalLM
Evaluation
Benchmark results
| Benchmark | Result |
|---|---|
| Average | 39.27 |
| ARC Easy | 43.81 |
| ARC Challenge | 25.34 |
| PIQA | 61.86 |
| HellaSwag | 32.43 |
| ArithMark 2.0 | 28.20 |
| BananaMind Base Bench 1.1 | 1034 Elo |
Scores produced with lm_eval and the official ArithMark 2.0 script. ARC Easy, ARC Challenge, PIQA, and HellaSwag use acc_norm,none. The Base Bench 1.1 value is taken from the BananaMind Base Bench leaderboard.
Training
Tokenizer and training run
A custom 12,288-token byte-level BPE tokenizer with NFKC normalization and digit-aware pre-tokenization, trained on representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data.
The 50B-token run used a capacity-aware curriculum across five phases rather than sampling the aggregate mixture from the first token.
- Tokens seen
- 49,999,749,120
- Final optimizer step
- 90,421
- Sequence length
- 3,072
- Effective batch
- 180 sequences
- Tokens per optimizer step
- 552,960
- Optimizer
- AdamW, betas 0.9 / 0.95
- Peak learning rate
- 1.8e-3
- LR schedule
- Warmup-stable-decay with cosine decay
Getting started
Load BananaMind-2-Medium
This repository ships custom architecture files, so both the tokenizer and the model need
trust_remote_code=True. This is a base model: give it a continuation-style
prompt rather than an instruction.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-Medium"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))Instruction-tuned variant
BananaMind-2-Medium-Chat
The same 49,559,552-parameter architecture and 3,072-token context, instruction-tuned on SmolTalk. It scores 787 Elo overall on BananaMind Instruct Bench 1.1.
Instruct Bench and Base Bench are separate benchmarks with different tasks and scoring. Their Elo values are not directly comparable.
Licensing
License and use
BananaMind-2-Medium is released under Apache-2.0. Licenses differ between BananaMind models, so check the license on each Hugging Face repository before use. The model has not received dedicated safety alignment and can produce incorrect, biased, or otherwise undesirable text — do not rely on it for high-stakes decisions.