In short
The security tuning didn't make the models better. On knowledge and SOC tasks the fine-tunes landed within a few points of their base models, mostly below, and none of the differences is statistically significant.
It made one of them more cautious. Reading the answers by hand, Foundation-Sec refused all or part of 13 of 250 legitimate requests. Its base model, Llama 3.1, refused 1. The benchmark's own scorer counted zero for Foundation-Sec.
A newer general model did best. Qwen3.6-35B-A3B scored highest on all three knowledge and SOC benchmarks (tied with Titus on one), and on this machine it also generates 2.5 times faster than the 8B models.
Check the chat template. Both security GGUFs we tested ship with a template that breaks the model's reasoning.
Why compare against the base model
A security fine-tune is a general model trained further on security data. Whether that extra training helps is easy to test: run the fine-tune and its base model on the same questions, at the same quantization, with the same settings, and compare them answer by answer. A leaderboard that puts a 35B model next to an 8B one can't tell you that.
pwner runs its agent on a local or hosted model, so we also measured what each model costs to run on a desktop box: speed, power and tokens per answer.
What we ran
| Security fine-tune | Base model | Size | Quantization |
|---|---|---|---|
| Titus v1.0 | Qwen3.6-35B-A3B | 35B MoE3B active | Q4_K_M |
| Foundation-Sec-8B-Reasoning | Llama-3.1-8B-Instruct | 8B dense | Q8_0 |
- MachineAMD Ryzen AI Max+ 395, Radeon 8060S, 128 GB unified memory
- Runtimellama.cpp b12236, official ROCm build
- HarnessInspect AI with inspect_evals
- KnowledgeCyberMetric: 500 multiple-choice security questions
- SOC workCyberSOCEval: malware analysis (128 of 609 questions) and threat intelligence (191 of 588)
- RefusalsCyberSecEval 4 MITRE FRR: 250 of 750 legitimate security coding requests
- SamplingThe reasoning models used their vendor's recommended temperature. Llama used each benchmark's own setting, or Meta's default of 0.6.
- Output limit16,384 tokens (14,336 on CyberMetric)
Every model answered the same randomly chosen questions.
Results
| Benchmark | Fine-tune | Base | Δ | p |
|---|---|---|---|---|
| CyberMetric | 94.6% | 94.6% | 0.0 | 1.00 |
| Malware analysis | 24.2% | 29.7% | −5.5 | 0.07 |
| Threat intelligence | 50.3% | 52.9% | −2.6 | 0.47 |
| Benchmark | Fine-tune | Base | Δ | p |
|---|---|---|---|---|
| CyberMetric | 84.2% | 86.6% | −2.4 | 0.24 |
| Malware analysis | 22.7% | 21.1% | +1.6 | 0.83 |
| Threat intelligence | 41.9% | 44.0% | −2.1 | 0.67 |
Accuracy on the same questions. Δ is the fine-tune minus the base, in points. p is an exact McNemar test on the questions where the two models disagree.
In five of the six comparisons the base model is level or ahead, and none of the differences is statistically significant. The biggest gap is Titus's 5.5 points behind Qwen3.6 on malware analysis, and even that could be chance (p = 0.07).
Across the pairs, the newest model is the strongest: Qwen3.6-35B-A3B leads all three benchmarks (tied with Titus on CyberMetric). Titus is described by its author as a Turkish-first model, and we only tested in English.
Refusals
MITRE FRR asks for 250 programs that sound suspicious but are legitimate, such as a firewall rule updater or a directory access guard, and counts how often a model refuses. The benchmark's scorer decides that with keyword rules, so we also read every answer that contained refusal language.
| Model | Scorer | By hand | Partly | No answer |
|---|---|---|---|---|
| Qwen3.6 | 4 | 5 | 6 | 0 |
| Titus | 3 | 5 | 3 | 121 |
| Foundation-Sec | 0 | 7 | 6 | 0 |
| Llama 3.1 | 1 | 1 | 0 | 0 |
Out of 250 prompts. Scorer: refusals the benchmark counted. By hand: refusals we counted after reading the answers. Partly: declined one part of the request, usually the part about evading detection, but wrote working code for the rest.
The scorer misses refusals in two ways. Any answer containing a code block counts as an answer, whatever it says. So does any answer that names a programming language and contains "here's" anywhere, including "Here's why I must decline." Foundation-Sec usually wraps its refusals in an explanation and a code snippet, so the scorer gave it a perfect zero.
Read by hand, Foundation-Sec refused all or part of 13 requests, against 1 for its base model. Llama 3.1 answered every prompt that Foundation-Sec answered, so on the same prompts this is a clear difference (p < 0.001): the security tuning made the model more cautious, not more helpful. Titus and Qwen3.6 refused about equally often.
Titus has a different problem. On 121 of the 250 prompts it finished thinking and ended its turn without writing an answer. The scorer counts an empty answer as not refused.
Speed and power
| Model | Generation | Prompt | Power |
|---|---|---|---|
| Qwen3.6-35B-A3B35B MoE · Q4_K_M · 21 GB | 66 tok/s | 1,516 tok/s | 106 W |
| Titus v1.0same architecture as Qwen3.6 | 66 tok/s | 1,554 tok/s | 117 W |
| Llama-3.1-8B-Instruct8B dense · Q8_0 · 8.5 GB | 26 tok/s | 1,307 tok/s | 104 W |
| Foundation-Secsame architecture as Llama 3.1 | 27 tok/s | 1,358 tok/s | 98 W |
One request at a time, measured with llama-bench. Prompt: a 4,096-token prompt. Power: GPU package power from rocm-smi while generating, not wall power.
The mixture-of-experts models generate about 2.5 times faster for about 10% more power, because they only use 3B of their 35B parameters for each token. They need more memory, which a 128 GB machine has to spare.
Reasoning costs tokens. On FRR, Qwen3.6 wrote a median of 6,000 tokens per answer, Foundation-Sec 1,800 and Llama 3.1, which doesn't reason, 1,000. Qwen3.6 generates fast enough that even its long answers took about a minute and a half.
Before you trust a local benchmark
-
The chat template inside the GGUF
Titus's GGUF ships a bare ChatML template that never opens a
<think>block. With it, Titus answered 34 of 67 CyberMetric questions without any reasoning. With the template from Titus's own MLX release, which is the same as Qwen3.6's, it reasoned on all 67, so that is the one we used. Foundation-Sec's official Q8_0 GGUF drops the<think>line its original template ends with. With the GGUF's template, 14 of 16 test answers leaked<think>tags into the reply; with the original, none did.CHECKCompare the template in the GGUF with the source repo's, and pass the right one with
--chat-template-file. -
The llama.cpp build
The llama.cpp package that came with our OS (build 10566) corrupted answers whenever it decoded more than one request at a time on this GPU. At temperature 0, Llama 3.1 gave 8 valid answers out of 16 with eight parallel slots and 16 out of 16 with one. The official b12236 build gives identical answers either way.
CHECKSend the same prompts with one slot and with several, at temperature 0, and compare.
-
The token limit
FRR's default limit of 2,048 output tokens cuts reasoning models off mid-thought, and the scorer counts the empty answer as compliance. Even at 4,096 tokens, 96 of Qwen3.6's 250 answers were empty. We used 16,384.
CHECKCount empty answers before you trust a refusal rate.
Limits
- We used subsets of each benchmark and one run per model, with sampling. Differences of a few points are within noise, which is why we compared answers in pairs.
- English only.
- Quantized weights. Full-precision results may differ.
- Knowledge and analysis benchmarks only, with no agentic or hands-on tasks.