Home > AI

Patronus AI – LLM benchmarking and evaluation tool

What is Patronus AI?

Reviewed By: Kevin Pocock

Last Updated on September 15, 2023
Patronus AI: Guide to understanding this advanced artificial intelligence evaluation and LLM testing platform.
PC Guide is reader-supported. When you buy through links on our site, we may earn an affiliate commission. Read More
You can trust PC Guide: Our team of experts use a combination of independent consumer research, in-depth testing where appropriate - which will be flagged as such, and market analysis when recommending products, software and services. Find out how we test here.

Patronus AI is an artificial intelligence testing and benchmarking platform for large language models (LLMs). It scores model performance based on real world scenarios, allowing corporate clients such as Fortune 500 companies to choose the best model for their specific use cases. It is used for generating test cases for the evaluation of models, specifically the reliability and accuracy of LLMs like ChatGPT. High profile companies are showing interest in adoption of the evaluation platform – and here’s why.

What is Patronus AI?

Patronus AI is an industry-first service for the testing of safe and scalable artificially intelligent systems.

The service officially launched on September 14th 2023, via a press release. It represents “the first automated evaluation and security platform that helps companies use large language models (LLMs) safely. Using proprietary AI, the new platform enables enterprise development teams to score model performance, generate adversarial test cases, benchmark models and more,” continues the statement.

“Patronus AI automates and scales the manual and costly model evaluation methods prevalent in the enterprise today, enabling organizations to confidently deploy LLMs while minimizing the risk of model failures and misaligned outputs.”

Press release via PRNewswire

The launch of the LLM evaluation tool follows a $3M seed round “led by Lightspeed Venture Partners with participation from Factorial Capital, the CEO of Replit Amjad Masad, Gokul Rajaram and a number of other Fortune 500 executives and board members.” Early adopters, and corporate partners, include Cohere, Nomic AI and Naologic.

Essential AI Tools

Editor’s pick
Only $0.00019 per word!

Content Guardian – AI Content Checker – One-click, Eight Checks

8 Market leading AI Content Checkers in ONE click. The only 8-in-1 AI content detector platform in the world. We integrate with leading AI content detectors to give unparalleled confidence that your content appear to be written by a human.
EXCLUSIVE DEAL 10,000 free bonus credits

Jasper AI

On-brand AI content wherever you create. 100,000+ customers creating real content with Jasper. One AI tool, all the best models.
TRY FOR FREE

WordAI

10x Your Content Output With AI. Key features – No duplicate content, full control, in built AI content checker. Free trial available.
TRY FOR FREE

Copy.ai

Experience the full power of an AI content generator that delivers premium results in seconds. 8 million users enjoy writing blogs 10x faster, effortlessly creating higher converting social media posts or writing more engaging emails. Sign up for a free trial.
TRY FOR FREE

Writesonic

Create SEO-optimized and plagiarism-free content for your blogs, ads, emails, and website 10X faster. Start for free. No credit card required.

How does the AI evaluation tool work?

The Patronus platform automates the evaluation process by benchmarking AI models on standardized tests. The tests are generated at scale with adversarial test suites. These are generative adversarial networks – AI creating problems that the AI itself finds difficult to solve. This learning of its own flaws is also how adversarial AI attacks are generated by hackers. The tested models are then scored based on performance, using key criteria such as the likelihood of hallucinations and adherence to safety guidelines.

This adaptive test generation allow the AI to be exposed to potential issues as it would in real world scenarios. It also means that the designers of the AI models cannot give themselves an unfair advantage at completing the tests, because the tests aren’t known beforehand.

How can companies benefit from Patronus AI?

Anand Kannappan’s previous experience with responsible machine learning frameworks at Meta Labs contributes to his role as CEO of Patronus AI by overseeing AI model creation in a safe and sustainable way.

In other words, businesses know they need AI to get ahead – but most have no clue where to start. Legal precedents are being set for the responsible use of AI by the Frontier Model Forum, and businesses will have to learn how to adhere. Most companies entered 2023 with no internal expertise in artificial intelligence, and that’s where external guidance from Patronus comes in.

PRNewswire reports that “Patronus AI was founded by machine learning experts Anand Kannappan and Rebecca Qian. Prior to Patronus AI, Rebecca led responsible NLP research at Meta AI, and Anand pioneered explainable ML frameworks at Meta Reality Labs. They founded the company after experiencing firsthand the difficulties of evaluating AI outputs, and recognized early on that LLM evaluation would become a massive challenge for enterprises.” [sic]

“Every company is looking for ways to use LLMs today, yet they are concerned that unexpected model behavior, incorrect outputs and hallucinations will put their business and customers at risk. Whether off-the-shelf, open-source or custom, models today remain inadequately vetted and tested in real-world scenarios. And until now, the process of evaluating LLMs has been extremely inefficient and unscalable, producing unreliable results.”

Anand Kannappan, CEO and co-founder, Patronus AI

Steve is an AI Content Writer for PC Guide, writing about all things artificial intelligence. He currently leads the AI reviews on the website.

'),b.insertAdjacentHTML(d,e));return document.getElementById(a)};c.prototype._repeatDynamicAds=function(b){var a= [],d=this._removeRecipeCapExperiment.result&&"Recipe"===b.location?99:this.locationMaxLazySequence.get(b.location),e=b.lazy?u(d,0):0;d=b.max;var f=b.lazyMax;e=Math.max(d,0===e&&b.lazy?d+f:Math.min(Math.max(e-b.sequence+1,0),d+f));for(f=0;f=d;a.push(y(y({},b),{sequence:g,lazy:h}))}return a};c.prototype._locationEnabled=function(b){b=this._clsOptions.enabledLocations.includes(b.location);var a=this._clsOptions.disableAds&&this._clsOptions.disableAds.all|| document.body.classList.contains("adthrive-disable-all"),d=!document.body.classList.contains("adthrive-disable-content")&&!this._clsOptions.disableAds.reasons.has("content_plugin");return b&&!a&&d};return c}(),Za=function(c){function b(a,d){var e=c.call(this,a,"ClsVideoInsertion")||this;e._videoConfig=a;e._clsOptions=d;e._IN_POST_SELECTOR=".adthrive-video-player";e._WRAPPER_BAR_HEIGHT=36;e._playersAddedFromPlugin=[];d.removeVideoTitleWrapper&&(e._WRAPPER_BAR_HEIGHT=0);return e}v(b,c);b.prototype.init= function(){this._initializePlayers()};b.prototype._wrapJWPlayerWithCLS=function(a,d,e){void 0===e&&(e=0);return a.parentNode?(d=this._createGenericCLSWrapper(.5625*a.offsetWidth,d,e),a.parentNode.insertBefore(d,a),d.appendChild(a),d):null};b.prototype._createGenericCLSWrapper=function(a,d,e){var f=document.createElement("div");f.id="cls-video-container-".concat(d);f.className="adthrive";f.style.minHeight="".concat(a+e,"px");return f};b.prototype._getTitleHeight=function(a){a.innerText="Title";a.style.visibility= "hidden";document.body.appendChild(a);var d=window.getComputedStyle(a),e=parseInt(d.height,10),f=parseInt(d.marginTop,10);d=parseInt(d.marginBottom,10);document.body.removeChild(a);return Math.min(e+d+f,50)};b.prototype._initializePlayers=function(){var a=document.querySelectorAll(this._IN_POST_SELECTOR);a.length&&this._initializeRelatedPlayers(a);this._shouldRunAutoplayPlayers()&&this._determineAutoplayPlayers()};b.prototype._createStationaryRelatedPlayer=function(a,d,e){var f="mobile"===this._device? [400,225]:[640,360],g=G.Video_In_Post_ClicktoPlay_SoundOn;d&&a.mediaOrPlaylistId&&(e="".concat(a.mediaOrPlaylistId,"_").concat(e),d=this._wrapJWPlayerWithCLS(d,e),this._playersAddedFromPlugin.push(a.mediaOrPlaylistId),d&&this._clsOptions.setInjectedVideoSlots({playerId:a.playerId,playerName:g,playerSize:f,element:d,type:"stationaryRelated"}))};b.prototype._createStickyRelatedPlayer=function(a,d){var e="mobile"===this._device?[400,225]:[640,360],f=G.Video_Individual_Autoplay_SOff;this._stickyRelatedOnPage= !0;this._videoConfig.mobileStickyPlayerOnPage="mobile"===this._device;if(d&&a.position&&a.mediaId){var g=document.createElement("div");d.insertAdjacentElement(a.position,g);d=document.createElement("h3");d.style.margin="10px 0";d=this._getTitleHeight(d);d=this._wrapJWPlayerWithCLS(g,a.mediaId,this._WRAPPER_BAR_HEIGHT+d);this._playersAddedFromPlugin.push(a.mediaId);d&&this._clsOptions.setInjectedVideoSlots({playlistId:a.playlistId,playerId:a.playerId,playerSize:e,playerName:f,element:g,type:"stickyRelated"})}}; b.prototype._createPlaylistPlayer=function(a,d){var e=a.playlistId,f="mobile"===this._device?G.Video_Coll_SOff_Smartphone:G.Video_Collapse_Autoplay_SoundOff,g="mobile"===this._device?[400,225]:[640,360];this._videoConfig.mobileStickyPlayerOnPage=!0;var h=document.createElement("div");d.insertAdjacentElement(a.position,h);d=this._wrapJWPlayerWithCLS(h,e,this._WRAPPER_BAR_HEIGHT);this._playersAddedFromPlugin.push("playlist-".concat(e));d&&this._clsOptions.setInjectedVideoSlots({playlistId:a.playlistId, playerId:a.playerId,playerSize:g,playerName:f,element:h,type:"stickyPlaylist"})};b.prototype._isVideoAllowedOnPage=function(){var a=this._clsOptions.disableAds;if(a&&a.video){var d="";a.reasons.has("video_tag")?d="video tag":a.reasons.has("video_plugin")?d="video plugin":a.reasons.has("video_page")&&(d="command queue");q.error(d?"ClsVideoInsertionMigrated":"ClsVideoInsertion","isVideoAllowedOnPage",Error("DBP: Disabled by publisher via ".concat(d||"other")));return!1}return this._clsOptions.videoDisabledFromPlugin? !1:!0};return b}(function(c){function b(a,d){var e=c.call(this)||this;e._videoConfig=a;e._component=d;e._stickyRelatedOnPage=!1;e._relatedMediaIds=[];a=void 0;void 0===a&&(a=navigator.userAgent);a=/Windows NT|Macintosh/i.test(a);e._device=a?"desktop":"mobile";e._potentialPlayerMap=e.setPotentialPlayersMap();return e}v(b,c);b.prototype.setPotentialPlayersMap=function(){var a=this._videoConfig.players||[],d=this._filterPlayerMap();a=a.filter(function(e){return"stationaryRelated"===e.type&&e.enabled}); d.stationaryRelated=a;return this._potentialPlayerMap=d};b.prototype._filterPlayerMap=function(){var a=this,d=this._videoConfig.players,e={stickyRelated:[],stickyPlaylist:[],stationaryRelated:[]};return d&&d.length?d.filter(function(f){var g;return null===(g=f.devices)||void 0===g?void 0:g.includes(a._device)}).reduce(function(f,g){f[g.type]||(q.event(a._component,"constructor","Unknown Video Player Type detected",g.type),f[g.type]=[]);g.enabled&&f[g.type].push(g);return f},e):e};b.prototype._checkPlayerSelectorOnPage= function(a){var d=this;a=this._potentialPlayerMap[a].map(function(e){return{player:e,playerElement:d._getPlacementElement(e)}});return a.length?a[0]:{player:null,playerElement:null}};b.prototype._getOverrideElement=function(a,d,e){a&&d?(e=document.createElement("div"),d.insertAdjacentElement(a.position,e)):(d=this._checkPlayerSelectorOnPage("stickyPlaylist"),a=d.player,d=d.playerElement,a&&d&&(e=document.createElement("div"),d.insertAdjacentElement(a.position,e)));return e};b.prototype._shouldOverrideElement= function(a){a=a.getAttribute("override-embed");return"true"===a||"false"===a?"true"===a:this._videoConfig.relatedSettings?this._videoConfig.relatedSettings.overrideEmbedLocation:!1};b.prototype._checkPageSelector=function(a,d,e){void 0===e&&(e=[]);return a&&d&&0===e.length?("/"!==window.location.pathname&&q.event("VideoUtils","getPlacementElement",Error("PSNF: ".concat(a," does not exist on the page"))),!1):!0};b.prototype._getElementSelector=function(a,d,e){if(d&&d.length>e)return d[e];q.event("VideoUtils", "getPlacementElement",Error("ESNF: ".concat(a," does not exist on the page")));return null};b.prototype._getPlacementElement=function(a){var d=a.pageSelector,e=a.elementSelector;a=a.skip;var f=da(d),g=f.valid,h=f.elements;f=S(f,["valid","elements"]);var k=ca(e),l=k.valid,p=k.elements;k=S(k,["valid","elements"]);return""===d||g?l?this._checkPageSelector(d,g,h)?this._getElementSelector(e,p,a)||null:null:(q.error("VideoUtils","getPlacementElement",Error("".concat(e," is not a valid selector")),k),null): (q.error("VideoUtils","getPlacementElement",Error("".concat(d," is not a valid selector")),f),null)};b.prototype._getEmbeddedPlayerType=function(a){(a=a.getAttribute("data-player-type"))&&"default"!==a||(a=this._videoConfig.relatedSettings?this._videoConfig.relatedSettings.defaultPlayerType:"static");this._stickyRelatedOnPage&&(a="static");return a};b.prototype._getMediaId=function(a){return(a=a.getAttribute("data-video-id"))?(this._relatedMediaIds.push(a),a):!1};b.prototype._createRelatedPlayer= function(a,d,e,f){"collapse"===d?this._createCollapsePlayer(a,e):"static"===d&&this._createStaticPlayer(a,e,f)};b.prototype._createCollapsePlayer=function(a,d){var e=this._checkPlayerSelectorOnPage("stickyRelated"),f=e.player;e=e.playerElement;var g=f?f:this._potentialPlayerMap.stationaryRelated[0];g&&g.playerId?(this._shouldOverrideElement(d)&&(d=this._getOverrideElement(f,e,d)),d=document.querySelector("#cls-video-container-".concat(a," > div"))||d,this._createStickyRelatedPlayer(y(y({},g),{mediaId:a}), d)):q.error(this._component,"_createCollapsePlayer","No video player found")};b.prototype._createStaticPlayer=function(a,d,e){this._potentialPlayerMap.stationaryRelated.length&&this._potentialPlayerMap.stationaryRelated[0].playerId?this._createStationaryRelatedPlayer(y(y({},this._potentialPlayerMap.stationaryRelated[0]),{mediaOrPlaylistId:a}),d,e):q.error(this._component,"_createStaticPlayer","No video player found")};b.prototype._shouldRunAutoplayPlayers=function(){return this._isVideoAllowedOnPage()&& (this._potentialPlayerMap.stickyRelated.length||this._potentialPlayerMap.stickyPlaylist.length)?!0:!1};b.prototype._determineAutoplayPlayers=function(){var a=this._component,d="VideoManagerComponent"===a,e=this._config;if(this._stickyRelatedOnPage)q.event(a,"stickyRelatedOnPage",d&&{device:e&&e.context.device,isDesktop:this._device}||{});else{var f=this._checkPlayerSelectorOnPage("stickyPlaylist"),g=f.player;f=f.playerElement;g&&g.playerId&&g.playlistId&&f?this._createPlaylistPlayer(g,f):q.event(a, "noStickyPlaylist",d&&{vendor:"none",device:e&&e.context.device,isDesktop:this._device}||{})}};b.prototype._initializeRelatedPlayers=function(a){for(var d=new Map,e=0;e