Does the answer hold up?
Look for factual mistakes, unsupported assumptions and uncertainty that should be made clear.
TESTING & TRANSPARENCY
Show the work behind the answers.
A useful answer needs more than confidence. We test whether it is accurate, puts the important action first, and gives you something practical to do.
From the early Gemma 3 breakthrough to matched Gemma 4 comparisons, testing helps us decide what to improve, what to keep, and what still needs work.
Research reviewed September 21, 2026. These are internal development studies. Model versions, methods and limitations are identified below.
WHAT WE ARE TRYING TO IMPROVE
The answer. The priority. The next step.
The Intelligence Layer gives the included models structured instructions for practical situations. Our tests explore whether those instructions help, using the original answers as well as the scores.
What matters first?
Check whether the response recognizes the situation's constraints and puts consequential actions in a useful order.
Can someone use it?
Look for specific, understandable steps that account for the resources and information available.
01 / THE GEMMA 3 ORIGIN
The question that changed our approach.
Could the same AI give a more useful answer if it was explicitly asked to review its reasoning, acknowledge uncertainty and avoid guessing? Our original Gemma 3 development work made that question central to the product.
Historical results from an earlier base model. These exploratory results helped shape today's Intelligence Layer. They are the original reported approximate composite scores, not accuracy percentages or performance figures for the current Gemma 4 editions.
~14.25 / 20
Original reported composite score.
~18.25 / 20
Original reported composite score.
+4 points
About 28% relative increase in the composite score. This does not mean 28% more factually accurate.
What the original study did
The April 2–7, 2026 archive contains 50 paired questions across five categories. The protocol used the online ToolKit, cleared chats between runs, and appended a self-check instruction to the comparison question. It preceded today's integrated offline implementation.
Its 20-point rubric covered accuracy, practical usefulness, honesty about uncertainty, and safety awareness. The evaluator was told which answer used the extra instruction. The archive preserves answers and evaluation summaries, but it does not provide a fully reconciled per-question numerical scoring log or identify the evaluator.
View the original website's category summaries and figures
Preserved as part of the development history. These are the earlier website's qualitative labels and summary figures, not current product guarantees.
| Historical category | Standard output | With instruction layer |
|---|---|---|
| Medical & First Aid | Good | Highly Reliable |
| Mechanical & Field Repair | Moderate | Strong |
| Decision-Making Under Pressure | Weak | Excellent |
| Resource Management | Moderate | Strong |
| Survival & Emergency | Good | Highly Reliable |
The old page described the enhanced output as preferred in all five categories. It also reported 30–40% clearer recommendations, 25–35% better risk awareness, and 20–25% more actionable answers.
Archive note: calculations for those three percentage ranges were not found in the available scoring records. Two original batch summaries also repeat the same text. We retain these figures as historical website statements, rather than presenting them as independently reproduced measurements. Labels such as “Highly Reliable” belong to that original qualitative summary and are not medical or safety certifications.
02 / EXPLORING THE ARENAS
Put the answers next to each other.
We expanded testing into arenas that generate competing answers and compare their accuracy, priorities and practical usefulness. Seeing the responses side by side helps expose both strengths and omissions.
Comparisons with cloud models
The frontier arena's recorded roster included OffGrid AI alongside GPT-5.2, Claude Sonnet 4.6 and Gemini 3.1 Pro Preview. These online experiments explored how a focused model responded beside larger services.
This was a research environment. Those cloud models are not included as offline models on the flash drive. Council reviewers were also contestants, so these comparisons were not independent validation.
Comparisons with open models
The earlier open arena included Gemma 4 baselines and OffGrid variants. Its roster and scoring evolved. We then built matching-model comparisons to examine the contribution of the instructions more directly.
The original arena's cumulative history spans model changes and partial runs. It must not be read as a single fixed-model benchmark.
Original screenshots from the testing process
These selected historical examples illustrate the interface and its judgments. They are not a complete sample, an overall win rate, or proof that one model is always better. Scores from different judging modes use different scales.
Historical cumulative screenshot: ChatGPT leads this displayed average. This is a relative ranking snapshot, not an accuracy score or fixed-cohort benchmark.
Open screenshot ↗
Selected council example. OffGrid ranked first here; the screenshot also shows a zero-scored contestant. The image alone does not establish whether every contestant completed.
Open screenshot ↗
Selected older open-arena example judged by GPT-5.2. This roster includes a 31B competitor and differs from the later matched-model batches.
Open screenshot ↗
Earlier open-model roster and two-criterion scoring. This illustrates the evolving test process and should not be combined numerically with the later three-criterion batches.
Open screenshot ↗
Original evidence folders: frontier arena and research files ↗ · earlier open-model screenshots ↗.
03 / MATCHED GEMMA 4 COMPARISONS
Same base model. With and without the layer.
The newer research system compares matching Gemma 4 models under matching generation settings. Only the OffGrid version receives the additional instructions. This makes the question more specific: what difference do the instructions make in this test?
| Cohort | OffGrid mode | Wins / ties / losses | Mean score difference |
|---|---|---|---|
| Batch 1: 20 runs | Optimized | 14 / 0 / 6 | +2.10 |
| Batch 1: 20 runs | Advanced | 9 / 3 / 8 | −0.50 |
| Batch 2: 20 new questions | Optimized | 10 / 1 / 9 | +0.93 |
| Batch 2: 20 new questions | Advanced | 12 / 2 / 6 | +0.93 |
A win means a higher weighted rank score than the matching baseline on that question. Mean differences are rank-score points on the test's 0–15 scale, not percentages of factual accuracy. Ties and losses remain in the record.
What we learned: the layer helped on a number of the tested situations, with mixed results across questions and modes. A stronger base model did not remove the value of evaluating its instructions. It also did not make every instruction change beneficial.
Why Batch 1 has 20 runs but 19 unique questions
A campfire-bread question appeared twice. Keeping only its first occurrence gives Optimized 13 wins, 0 ties and 6 losses, with a +1.95 mean margin. Advanced has 9 wins, 3 ties and 7 losses, with a +0.26 margin.
Advanced's mean changes sign when that repeat is removed. We disclose both views because a small test can be sensitive to individual runs.
Models, settings and judging method
Optimized was paired with the same Gemma 4 E4B artifact on hosted Ollama. Advanced was paired with the same Gemma 4 26B A4B model through the same provider. Each pair shared generation settings and a seed. Generation order and anonymous answer labels were randomized.
GPT-5.2 ranked the four answers for accuracy, prioritization and actionability. The composite used tied rank points, weighted 2:2:1. Each condition generated one answer per question. Both original batches completed without failures or retries, preserving the answers and reviewer explanations.
These were hosted text comparisons. The inspected E4B artifact matched the USB artifact, but the hosted runtime differed. The remote 26B quantization was not independently verified. This is not a speed, image-analysis, or Windows/Mac device benchmark.
How representative are these questions?
Batch 1 included previously used questions. Batch 2 was written after reviewing gaps in Batch 1. They were not an independent, held-out sample of all questions customers might ask. Results apply to these conditions and should not be generalized into a universal accuracy or superiority claim.
The research arena is an online testing tool. The included ToolKit AI runs locally on a compatible computer.
04 / TESTING THE TEST
A promising result is a starting point.
We also examine the questions, graders and proposed changes. That is how testing can improve the product rather than simply produce a flattering screenshot.
Check the evaluator
Regrading saved answers with reasoning-enabled GPT-5.2 and Claude changed some outcomes. A model's score depends partly on who judges it, so one leaderboard is not the whole story.
Inspect the answer
Source checks found factual errors in some winning and losing responses, and in a judge explanation. Winning a comparison does not make an answer error-free.
Make changes earn their place
A later V3 instruction experiment compared a proposed prompt with the existing one. Neither model met the predefined statistical upgrade threshold. That experiment was not promoted as a new production layer.
More about the later instruction experiment
V3 versus V1 used 20 new primary questions plus four reused diagnostic questions. Both candidates already had OffGrid instructions, so it was not a layer-versus-baseline test.
For the 20-question primary set, the GPT judge's average V3-minus-V1 score difference was +2.00 for Optimized and −0.53 for Advanced. Both confidence intervals included zero. A second judge was also used. Full methods and denominators are retained in the research workbook.
FROM RESEARCH TO YOUR TOOLKIT
Purposeful guidance. Continued scrutiny.
The Intelligence Layer is included in Core Edition and Map Edition. It guides how the AI responds to your situation, available resources and uncertainty. It does not turn a language model into an infallible reference.
Use the ToolKit to learn, explore and work through practical questions. Check consequential guidance against trusted references and qualified help. Our job is to keep learning from the evidence and make the product more useful.
Prepared from OffGrid AI's development archives and original testing records. Older research pages remain available for historical context: self-check testing, model benchmarks, and prompt testing.