Kila AI
Evaluation
Base model vs Kila Coder
Same 100 held-out prompts, same settings, before and after training. We publish every number, including the ones that got worse.
| Metric | Base model | Kila Coder |
|---|---|---|
| Answers with a code block higher is better | 99 of 100 | 100 of 100 |
| Answers that start with code higher is better | 33 of 100 | 100 of 100 |
| Answers with no code at all lower is better | 1 of 100 | 0 of 100 |
| Words per answer (mean) lower is better | 236.7 | 63.9 |
| Words outside code (mean) lower is better | 135.3 | 0.1 |
| Answers cut off at the token limit lower is better | 3 of 100 | 0 of 100 |
| Solidity prompts that compile higher is better | 11 of 30 | 15 of 30 |
| Robinhood Chain fact checks passed higher is better | 1 of 20 | 15 of 20 |
| HumanEval pass@1 (greedy) higher is better | 118 of 164 | 120 of 164 |
| HumanEval with the problem's imports added higher is better | 118 of 164 | 120 of 164 |
| Speed in Ollama, 4-bit build Apple M5, 16 GB | Not measured yet | about 105 tokens/s |
Both models ran in full precision (bf16) on an Apple M5 laptop with the Hugging Face transformers library, not in Ollama; the base model with its default system prompt, Kila Coder with its own. Kila Coder here is version 3; earlier versions are listed on the model card. The Solidity row means the code compiles, not that it is correct. Speed was measured once in Ollama with the 4-bit build (two runs of a 370-token answer, 106 tokens per second); other machines are not measured yet.
By category
Split by prompt type
| Category | Prompts | Starts with code Base | Starts with code Kila Coder | Mean words Base | Mean words Kila Coder |
|---|---|---|---|---|---|
| General code | 50 | 28 of 50 | 50 of 50 | 177.2 | 70.4 |
| Solidity | 30 | 2 of 30 | 30 of 30 | 356.0 | 68.5 |
| Robinhood Chain | 20 | 3 of 20 | 20 of 20 | 206.7 | 41.0 |
Method
How we tested
| Prompts | 100 prompts kept out of training: 50 general coding prompts (30 from held-out OpenCodeInstruct rows, 20 written for this test across 10+ languages), 30 Solidity prompts and 20 Robinhood Chain prompts. We checked that none of them, or anything close to them, is in the training data. |
|---|---|
| Settings | Greedy decoding, up to 1024 new tokens, one answer per prompt, the model's own default system prompt. |
| Counting | Words are split on spaces. Words outside code exclude fenced code blocks. An answer starts with code when its first characters are a code fence. |
| Solidity | The longest Solidity block in each answer is compiled with the official compiler (solc-js), using the newest release its pragma allows. OpenZeppelin Contracts 5.6.1 is available for imports. Compiling is not the same as being correct or safe. |
| Robinhood Chain | Simple text checks against facts from the official docs, for example that a mainnet config uses chain ID 4663. They catch wrong facts, not bad code. |
| HumanEval | All 164 problems from openai/openai_humaneval (MIT). The first Python block of each answer is run against the official tests with a 10 second limit, offline. |
| Limits | One run on one machine with 100 prompts, so small differences mean little. The prompts were drafted with AI help and have not yet been reviewed by a person. |