Medieval Fantasy Benchmark Tests Local AI Models

The benchmark suite consisted of various tasks such as quest completion, scene endings, item and time tracking, character detection, storytelling, and drafting. An external LLM grader was used to judge the models' performance.

The results showed that gemma-4-31B topped the list with an overall pass rate of 87%, closely followed by Qwen3.6-27B at 82%. The pass rates dropped significantly after gemma-4-12B.

This benchmark provides valuable insights into the capabilities of local AI models in roleplay and agentic tasks, highlighting the strengths and weaknesses of each model.

The test also demonstrates the potential of using external LLM graders to evaluate the performance of AI models in complex tasks.

The results of this benchmark can be useful for developers and researchers looking to improve the performance of their AI models in roleplay and agentic tasks.

The benchmark suite and results can be found on the Reddit thread, along with a chart showing the pass rates for each model.

This benchmark is a significant development in the field of AI research, as it provides a comprehensive evaluation of local AI models in a complex task like medieval fantasy roleplay.

The results of this benchmark can have implications for the development of more advanced AI models, and can inform the design of future benchmarks and evaluations.

The use of an external LLM grader to judge the models' performance adds an extra layer of objectivity to the results, and demonstrates the potential of using LLMs as evaluators in AI research.

The benchmark also highlights the importance of considering the size and complexity of AI models when evaluating their performance, as smaller models like Qwen3.6-27B can still achieve impressive results.

The results of this benchmark can be used to inform the development of more efficient and effective AI models, and can provide valuable insights for researchers and developers working in the field of AI.