Base Preview checkpoint BananaMind Community License 1.0
BananaMind-2-Pro-Preview
The first public checkpoint of BananaMind 2 Pro, and the largest released model.
BananaMind-2-Pro-Preview is a decoder-only base causal language model trained from scratch. This checkpoint was exported after 96,000 optimizer steps and 51,904,512,000 training tokens of an ongoing 100B-token pretraining run. Its benchmark scores describe this exact checkpoint and are not final BananaMind 2 Pro results.
- Parameters
- 138,971,520
- Context length
- 3,072 tokens
- Training tokens
- 51.9B at 96K steps
Specification
Architecture
- Parameters
- 138,971,520
- Architecture
- BananaMind2Pro decoder-only Transformer
- Layers
- 24
- Hidden size
- 640
- Intermediate size
- 1,920
- Attention heads
- 8
- KV heads
- 4
- Head dimension
- 80
- Attention style
- Grouped-query attention with QK norm
- MLP
- SwiGLU
- Position embeddings
- RoPE, theta 100,000
- Normalization
- RMSNorm, epsilon 1e-6
- Vocabulary size
- 32,768
- Context length
- 3,072
- Embeddings
- Tied input/output
- Weight format
- safetensors
- HF architecture
BananaMind2ProForCausalLM
Evaluation
Benchmark results
| Benchmark | Result |
|---|---|
| ARC Easy | 51.01% |
| ARC Challenge | 27.13% |
| PIQA | 66.76% |
| HellaSwag | 39.83% |
| ArithMark 3 | 38.90% |
| ArithMark 2 | 28.60% |
| INT Index | 23.04 |
| BananaMind Base Bench 1.1 | 1106 Elo |
Self-reported evaluations of the exported 96K checkpoint. ARC Easy, ARC Challenge, PIQA, and HellaSwag use acc_norm,none; ArithMark 3 uses length-normalized continuation accuracy; ArithMark 2 uses raw continuation accuracy. Results vary with harness version, dtype, and scoring configuration.
Training
Tokenizer and training run
A custom 32,768-token byte-level BPE tokenizer with NFKC normalization and digit-aware pre-tokenization, trained on representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. Digits are isolated before byte-level BPE, so complete numbers are never merged into large single number tokens.
The ongoing 100B-token run uses a capacity-aware curriculum. This preview was exported during the reasoning core phase.
- Tokens seen at this checkpoint
- 51,904,512,000
- Full-run target
- 100B tokens
- Optimizer steps completed
- 96,000
- Sequence length
- 3,072
- Effective batch
- 176 sequences
- Tokens per optimizer step
- 540,672
- Optimizer
- AdamW, betas 0.9 / 0.95
- Peak learning rate
- 1.5e-3
- LR schedule
- Warmup-stable-decay with cosine decay
- Training phase at export
- Reasoning core
Getting started
Load BananaMind-2-Pro-Preview
This repository ships custom architecture files, so both the tokenizer and the model need
trust_remote_code=True. This is a base model: give it a continuation-style
prompt rather than an instruction.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-Pro-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))Instruction-tuned variant
BananaMind-2-Pro-Preview-Chat
The same 138,971,520-parameter architecture and 3,072-token context, instruction-tuned on SmolTalk. It scores 888 Elo overall on BananaMind Instruct Bench 1.1, passing 98 of 300 tasks, with 1194 Elo on Context Recall and 1337 Elo on Code.
Instruct Bench and Base Bench are separate benchmarks with different tasks and scoring. Their Elo values are not directly comparable.
Licensing
License and use
BananaMind-2-Pro-Preview is released under BananaMind Community License 1.0. Licenses differ between BananaMind models, so check the license on each Hugging Face repository before use. The model has not received dedicated safety alignment and can produce incorrect, biased, or otherwise undesirable text — do not rely on it for high-stakes decisions.