llama.cpp PR boosts inference speed by 50% with Top-N-Sigma optimization
Reported by the original publisher: Top-N-Sigma: Remove unconditional softmax+sort by TimNN · Pull Request #22645 · ggml-org/llama.cpp. Analysis and context written by TickrWire.
A PR in llama.cpp removes redundant softmax+sort in Top-N-Sigma sampler, boosting inference speed by 50% on Gemma-4-E4B-Q8_0.
- PR #22645 in llama.cpp removes redundant softmax+sort in Top-N-Sigma sampler when followed by Dist.
- Benchmarks show a 50% speedup (30t/s → 45t/s) on Gemma-4-E4B-Q8_0 on an M3 Max MacBook Pro.
- Token latency reduced by ~10ms per token due to the optimization.
- Targets local inference efficiency for quantized models.
- No change to model output; purely a performance optimization.
The pull request #22645 in the ggml-org/llama.cpp repository optimizes the Top-N-Sigma sampler by eliminating an unconditional softmax followed by sorting. This operation is unnecessary when the sampler is followed by a distribution sampler (Dist), as the computed values are discarded. Benchmarks on an M3 Max MacBook Pro show a 50% increase in tokens per second (from ~30t/s to ~45t/s) for the google_gemma-4-E4B-it-Q8_0 model, reducing token latency by 10ms. The change targets efficiency in local inference scenarios, particularly for quantized models.
Developers using llama.cpp for local inference can leverage this optimization for faster token generation without altering model behavior.
Businesses deploying local AI models benefit from reduced latency and improved throughput, enhancing user experience.
Indirectly supports the efficiency narrative in local AI inference, which may influence hardware and software investment trends.
Demonstrates practical optimization techniques in AI inference pipelines, useful for learning.
Highlights ongoing improvements in open-source AI tooling, making local AI more accessible and performant.
- Top-N-Sigma
- A sampling method in inference that selects tokens based on a probabilistic distribution derived from the top N values.
- softmax
- A function that converts a vector of values into a probability distribution.
- Dist
- A distribution sampler in inference pipelines that selects tokens based on a probability distribution.
- tokens per second (t/s)
- A metric measuring the speed of token generation during inference.
- quantized model (Q8_0)
- A model compressed to 8-bit precision to reduce memory usage and improve inference speed.
AI bias estimate: Neutral; focuses on technical improvements with clear benchmarks. (Automated estimate, not a definitive judgement.)
AI ToolsMeta AI’s new Mac app wants you to talk to your apps
How one Philadelphia school is using AI to strengthen student learning, not replace teachers - CBS News
Domain and publish date filters for Web Search on AgentCore - Amazon Web Services (AWS)
KnowledgeForge: mining gold from the ITSM ticket graveyard - Amazon Web Services (AWS)
Google launches new study tools for Students across Search and Gemini
New White House strategy clarifies military tech priorities: undersea, outer space and AI - Breaking Defense
The White House released a new strategy prioritizing military investments in artificial intelligence, space systems and undersea technologies to counter emerging threats.
AI in an iron grip: How dictatorships use artificial intelligence to strengthen their rule - theins.press
A new report examines how authoritarian governments deploy AI for surveillance, censorship, and propaganda to reinforce their power.
Stripe, OpenRouter finally strike a deal - Banking Dive
Stripe and OpenRouter have partnered to integrate Stripe's payment processing with OpenRouter's AI model aggregation platform.
Exclusive-How a Texas student blew the whistle on a rogue AI hacking attempt - The Mighty 790 KFGO
A Texas student uncovered an AI-powered hacking attempt targeting local systems, prompting a swift law enforcement response.
Student Journalists: AI Is Changing Our Work — And Not For the Better - The 74
A student journalism outlet argues that AI tools are degrading the quality and authenticity of their reporting.
Don’t mistake chatbot intelligence for consciousness - The Economist
The Economist argues that advanced chatbots lack true consciousness despite their impressive intelligence, urging caution against anthropomorphizing AI.