Open small language models by Banaxi-Tech
Small models,
trained from scratch.
BananaMind builds compact decoder-only language models on consumer hardware and releases the
weights, tokenizers, benchmarks, and training details openly on Hugging Face. Nine public
models from 10M to 139M parameters, in base and chat
variants.
- Public models
- 5 base 4 chat
- Largest model
- 138,971,520 params
- Pro Preview tokens
- 51.9B pretraining
One architecture family
Every BananaMind 2 model is a decoder-only causal language model built on the same set of
components, scaled down or up. Each repository ships custom architecture files, so the models
load natively through Transformers with trust_remote_code=True.
- Grouped-query attention
- QK normalization
- RoPE
- SwiGLU
- RMSNorm
- Tied embeddings
- Digit-aware byte-level BPE
- Native Transformers support
Model lineup
Every public BananaMind 2 model
Base models continue text. Chat models are instruction-tuned on SmolTalk and use a chat template. Pick a variant to filter the list.
Showing all 11 models
| Model | Parameters | Context | Variant | Status | Link |
|---|---|---|---|---|---|
| BananaMind-2-Ultra Announced, not yet released | 400M | 16,384 tokens | Base | Coming soon | |
| BananaMind-2-Pro-Preview Public 96K checkpoint of an ongoing run | 138,971,520 | 3,072 tokens | Base | Preview | Open |
| BananaMind-2-Pro-Preview-Chat Instruction-tuned on SmolTalk | 138,971,520 | 3,072 tokens | Chat | Preview | Open |
| BananaMind-2-Medium 49,999,749,120 training tokens | 49,559,552 | 3,072 tokens | Base | Released | Open |
| BananaMind-2-Medium-Chat Instruction-tuned on SmolTalk | 49,559,552 | 3,072 tokens | Chat | Released | Open |
| BananaMind-2-Mini 29,999,726,592 training tokens | 25,178,752 | 4,096 tokens | Base | Released | Open |
| BananaMind-2-Mini-Chat Instruction-tuned on SmolTalk | 25,178,752 | 4,096 tokens | Chat | Released | Open |
| BananaMind-2-MoE Sparse: about 1,985,152 active parameters per token | 25,086,592 | 4,096 tokens | Base | Released | Open |
| BananaMind-2-Nano 29,999,726,592 training tokens | 9,968,128 | 4,096 tokens | Base | Released | Open |
| BananaMind-2-Nano-Chat Instruction-tuned on SmolTalk | 9,968,128 | 4,096 tokens | Chat | Released | Open |
| BananaMind-2-Micro Announced, not yet released | 2.9M | — | Base | Coming soon |
Model names open a detail page on this site; Open goes to the Hugging Face repository. Parameter counts for released models are exact totals from each model card. BananaMind-2-MoE is sparse: it holds 25,086,592 total parameters but activates roughly 1,985,152 per token. BananaMind-2-Ultra and BananaMind-2-Micro are announced but not yet released, and their figures are planned targets rather than measured results.
Featured
BananaMind-2-Pro-Preview
Preview checkpoint Base model Community License 1.0
The first public checkpoint of BananaMind 2 Pro, exported after 96,000 optimizer steps and 51,904,512,000 training tokens of an ongoing 100B-token pretraining run. Its scores describe this exact checkpoint and are not final BananaMind 2 Pro results.
96K checkpoint results
- ARC Easy
- 51.01%
- ARC Challenge
- 27.13%
- PIQA
- 66.76%
- HellaSwag
- 39.83%
- ArithMark 3
- 38.90%
- ArithMark 2
- 28.60%
- INT Index
- 23.04
- Base Bench 1.1
- 1106Elo
Self-reported evaluations of the exported 96K checkpoint. ARC Easy, ARC Challenge, PIQA, and
HellaSwag use acc_norm,none. Results vary with harness version, dtype, and
scoring configuration.
Architecture
- Parameters
- 138,971,520
- Context length
- 3,072 tokens
- Layers
- 24
- Hidden size
- 640
- Attention
- 8 heads, 4 KV heads
- Head dimension
- 80
- MLP
- SwiGLU, 1,920 intermediate
- Normalization
- RMSNorm with QK norm
- Positions
- RoPE, theta 100,000
- Vocabulary
- 32,768 digit-aware BPE
- Embeddings
- Tied input/output
- Tokens seen
- 51,904,512,000
BananaMind-2-Pro-Preview-Chat
The same 138,971,520-parameter architecture and 3,072-token context, instruction-tuned on SmolTalk. On BananaMind Instruct Bench 1.1 it scores 888 Elo overall, passing 98 of 300 tasks, with 1194 Elo on Context Recall and 1337 Elo on Code.
Instruct Bench and Base Bench are separate benchmarks with different tasks and scoring. Their Elo values are not directly comparable.
Evaluation
Published benchmark results
All figures below are self-reported evaluations of released checkpoints. Charts are the same ones published on the model cards.
| Benchmark | Result | Metric |
|---|---|---|
| ARC Easy | 51.01% | acc_norm,none |
| ARC Challenge | 27.13% | acc_norm,none |
| PIQA | 66.76% | acc_norm,none |
| HellaSwag | 39.83% | acc_norm,none |
| ArithMark 3 | 38.90% | Length-normalized continuation accuracy |
| ArithMark 2 | 28.60% | Raw continuation accuracy |
| INT Index | 23.04 | Open SLM Leaderboard-style aggregate |
| BananaMind Base Bench 1.1 | 1106 Elo | Fixed-item Overall Elo |
| Scope | Elo | Tasks passed |
|---|---|---|
| Overall | 888 | 98 / 300 |
| Context Recall | 1194 | 19 / 30 |
| Code | 1337 | 11 / 15 |
| Multi-turn | 911 | 26 / 75 |
| System Prompts | 793 | 9 / 60 |
| General | 720 | 33 / 120 |
Benchmark
BananaMind Base Bench 1.1
An English text-completion benchmark for base causal language models. Models receive plain text followed by four continuations; the runner selects the continuation with the highest mean conditional log-likelihood. No chat template, no sampling, no free-form generation.
- Tasks
- 350 authored continuation items
- Categories
- Seven
- Continuations per task
- Four
- Scoring
- Mean conditional log-likelihood
- Headline metric
- Fixed-item Overall Elo
- Random-choice baseline
- 25%, approximately 805 Elo
- Maximum submission size
- 250M parameters
Getting started
Load a model in a few lines
BananaMind repositories ship custom architecture files, so both the tokenizer and the model
need trust_remote_code=True. Base models take continuation-style prompts; chat
models use their own chat template.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BananaMind/BananaMind-2-Pro-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()
inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))Base models
Base models continue text. They are not instruction-tuned and have no chat template. Give them a partial sentence, document, or code fragment and they carry it on. They suit continuation research, likelihood scoring, and tokenizer experiments.
Chat models
Chat models are fine-tuned on SmolTalk and support system prompts and multi-turn
conversations. Render messages with tokenizer.apply_chat_template rather than
concatenating text yourself, so the role markers match training.
Training
Curriculum training on consumer hardware
BananaMind models are pretrained from scratch on a staged data curriculum rather than sampling the final mixture from the first token. Runs are single-machine, and the model cards publish the token counts, optimizer settings, and schedules used.
Staged curriculum
The Pro run moves through five phases across its 100B-token target: a web-heavy breadth foundation, a knowledge ramp, a reasoning core, a synthesis ramp, and a quality finish. The public preview was exported during the reasoning core phase.
Digit-aware tokenizers
Each tokenizer is a custom byte-level BPE with NFKC normalization and digit-aware pre-tokenization. Digits are isolated before merging, so numbers never collapse into large single number tokens.
Documented cost
The Pro Preview card publishes an engineering estimate of the energy used to reach the 96K checkpoint, measuring GPU and CPU package power and estimating the rest. It is an estimate, not a wall-meter measurement.
| Model | Parameters | Training tokens | Context |
|---|---|---|---|
| BananaMind-2-Pro-Preview | 138,971,520 | 51,904,512,000 | 3,072 |
| BananaMind-2-Medium | 49,559,552 | 49,999,749,120 | 3,072 |
| BananaMind-2-Mini | 25,178,752 | 29,999,726,592 | 4,096 |
| BananaMind-2-MoE | 25,086,592 | 29,999,726,592 | 4,096 |
| BananaMind-2-Nano | 9,968,128 | 29,999,726,592 | 4,096 |
The Pro Preview figure is the token count at the public 96K checkpoint of an ongoing run, not a completed total.
Community
Open weights, per-model licensing
Licensing
BananaMind-2-Pro-Preview and BananaMind-2-Pro-Preview-Chat are released under the BananaMind Community License 1.0. Other models carry their own licenses — check the license on each Hugging Face repository before use, since they are not all the same.
Under the Community License, commercial products or services exceeding the thresholds in Section 1 require a separate commercial license from Banaxi-Tech.
Get in touch
Questions, benchmark submissions, evaluation results, and commercial licensing enquiries are welcome. Model discussions are best opened directly on the relevant Hugging Face repository.