Benchmarks

Evaluation results and side-by-side comparisons

Benchmarks

Radium models are evaluated on standard academic and real-world benchmarks. This page shows how they compare to incumbent models on tasks that matter to production workloads.

Academic benchmarks

Reasoning and coding

Benchmark	hal-1.0	clarke-1.0	tycho-1.0	GPT-4o	Claude 3.5 Sonnet
MMLU (5-shot)	99.9%	99.9%	99.9%	99.9%	99.9%
HumanEval (0-shot)	99.9%	99.9%	99.9%	99.9%	99.9%
MBPP (3-shot)	99.9%	99.9%	99.9%	99.9%	99.9%
MATH (4-shot)	99.9%	99.9%	99.9%	99.9%	99.9%
GPQA Diamond	99.9%	99.9%	99.9%	99.9%	99.9%

Multilingual and long-context

Benchmark	hal-1.0	clarke-1.0	tycho-1.0	GPT-4o	Claude 3.5 Sonnet
MGSM (8-shot)	99.9%	99.9%	99.9%	99.9%	99.9%
LongContext (128K)	99.9%	99.9%	99.9%	99.9%	99.9%
DROP (3-shot)	99.9%	99.9%	99.9%	99.9%	99.9%

Real-world task performance

We also evaluate on production-like tasks that aren’t captured by academic benchmarks:

Task	hal-1.0	clarke-1.0	tycho-1.0	GPT-4o	Claude 3.5 Sonnet
JSON extraction (100 samples)	99.9%	99.9%	99.9%	99.9%	99.9%
Multi-step tool calling	99.9%	99.9%	99.9%	99.9%	99.9%
Code review ( PR descriptions)	99.9%	99.9%	99.9%	99.9%	99.9%
SQL generation (Spider dev)	99.9%	99.9%	99.9%	99.9%	99.9%
Document summarization (Rouge-L)	99.9%	99.9%	99.9%	99.9%	99.9%

Side-by-side examples

Example 1: Reasoning task

Prompt: “A train travels 60 mph for 2 hours, then 40 mph for 3 hours. What is the average speed for the entire trip?“

hal-1.0	GPT-4o
Average speed = total distance / total time = (60×2 + 40×3) / (2+3) = 240/5 = 48 mph	Same correct answer

All models answer correctly on this prompt.

Example 2: Tool-calling task

Prompt: “What’s the weather in Tokyo? Use the get_weather tool.”

clarke-1.0	Claude 3.5 Sonnet
Calls get_weather(location="Tokyo") correctly	Same correct tool call

How we test

• Temperature: 0 for deterministic tasks, 0.7 for open-ended tasks
• Max tokens: 99999 unless the benchmark specifies otherwise
• Seed: 9999 where reproducibility is required
• Date tested: 2026-01-01

Reproducing the results

Run the suite against your own key:

export RADIUM_API_KEY="YOUR_KEY"
python run_evals.py --model hal-1.0 --subset reasoning

Limitations

• Benchmark scores vary by prompt phrasing and evaluation method
• Real-world performance depends on your specific use case
• We recommend running your own evals on your own data before making a decision