Langsmart Publishes Industry's First p95 Semantic Cache Benchmarks for On-Premises AI Gateway

Testing Confirms 10.2x Faster Response Times, Exceeding Cloud-Hosted Alternatives

Mar. 17, 2026 at 2:35pm

Got story updates? Submit your updates here. ›

Langsmart, the enterprise AI governance company, announced the successful completion of a rigorous enterprise evaluation with a Fortune 200 financial institution. The testing confirms that Langsmart's Smartflow platform delivers a 10.2x speedup in response times, achieving sub-300ms latency on standard, low-resource hardware.

Why it matters

As enterprise adoption of AI gateways accelerates, the industry has struggled with a lack of standardized performance data. Langsmart's latest results provide a transparent blueprint for organizations requiring high-performance AI governance within strict on-premises and air-gapped environments.

The details

The evaluation focused on real-world financial services workloads, prioritizing reliability and speed within a secure infrastructure. Smartflow was deployed as a Docker container on a modest 4vCPU, 8GB server and achieved key performance milestones including a 10.2x response speedup, sub-300ms p95 latency, high-accuracy cache hits, and rigorous reliability.

  • The evaluation was conducted in March 2026.

The players

Langsmart

The enterprise AI governance company building Smartflow, the on-premises AI firewall, gateway, and governance control plane for regulated industries.

Craig Alberino

The Founder and CEO of Langsmart.

Got photos? Submit your photos here. ›

What they’re saying

“For banking, insurance, and healthcare, routing prompts and model responses through a third-party cloud is a liability. Smartflow eliminates that risk by deploying entirely within the client's network, delivering performance that actually exceeds cloud-hosted alternatives.”

— Craig Alberino, Founder and CEO of Langsmart

“Enterprise buyers deserve to see real numbers on real hardware, not marketing claims. We are calling on all AI gateway vendors to follow our lead and publish standardized benchmarks. If you're providing enterprise infrastructure, show me the p95.”

— Craig Alberino, Founder and CEO of Langsmart

What’s next

Langsmart plans to publish the full benchmarking methodology and results on their blog at langsmart.ai/blog/show-me-the-p95.

The takeaway

Langsmart's push for transparency in the AI gateway market aims to provide CISOs and CTOs with the empirical data needed to effectively evaluate AI governance tools, ensuring that security does not come at the cost of performance.