It also provides traffic routing and rate limiting for local and third-party large language models (LLM) to maintain service availability and performance and control costs. F5 stated. Semantic caching drives faster response time and reduces operational costs by removing duplicate tasks from LLMs, according to the vendor.
The AI Gateway can inspect, identify, and block inbound attacks such as prompt injection, insecure output handling, model denial-of-service, sensitive information disclosure, and model theft. “For outbound responses, AI Gateway identifies and scrubs PII data and prevents hallucinations. Software development kits (SDKs) enable additional enforcement of operational rules and compliance requirements for both prompts and responses to further align to operational needs,” F5 stated.
“Additional capabilities such as reporting of a wide array of metrics via OpenTelemetry, careful attention to audit log requirements, semantic caching, rate-limiting, and content-based model routing ensure support for all three AI delivery and security requirements: observe, protect, and accelerate,” MacVittie wrote.
The AI Gateway can be integrated with F5’s NGINX application security suite and BIG-IP application delivery platforms offering customers legacy integration and access.
