logo

Is GPT-4 worth the extra cost?

Pulse Labs • July 25, 2023
Is GPT-4 worth the extra cost?

Open AI has a freemium model whereby GPT-3.5 is free and GPT-4 (“Plus”) is not. Understandably, there are questions about whether GPT-4 is worth the cost. Is it better? How much better?

One simple answer is: GPT-4 is 8% better.

Why? The perceived quality of the response. Putting other features and benefits aside, responses are the main product of AI platforms, and how users perceive differences among them is a key factor in evaluating their value and price.  

  GPT-3.5 GPT-4
Preferred Response in Blind Side by Side Test
N=[437]		 46%
[202]		 54% *
 [235]



AIQ™ blind side-by-side (SBS) comparisons of the same prompts across several categories reveal:

  • GPT-4 responses are preferred more often than GPT-3.5, but only 8% more
  • GPT-4 does well on ambiguous queries, whereas 3.5 does better on those related to logic

Implications

There are many reasons why consumers consider paying for AI platform services, including the quality of response and whether it is worth the cost. For a business, response is a critical component to evaluate, but also coupled with other factors: threshold and throttle amounts, data training requirements, API capabilities, data ownership, privacy policies, and of course cost.

And key for businesses is response quality for their consumers, particularly in the first few moments. As we have seen in the search realm, products or businesses NOT on the first page of search results capture a much smaller audience. Businesses need to ensure the AI they choose strikes the right balance between the cost and the information presented, tone, and length. Not easy, but as we see here, testable with the right methodology.

Three Ways the Auto Industry Can Leverage Generative AI

By Pulse Labs • August 15, 2023
For most consumers, buying a vehicle is one of the largest purchasing decisions they will make, second only to homeownership. This kind of expensive, long-lasting decision is often emotional, and, at least in the consumer's mind, not easy to undo. With vehicle technology advancing faster than ever, consumers

AI Tube: Watch where platforms win, or lose

By Pulse Labs • August 8, 2023
As part of AIQ's effort to compile a comprehensive overview of users' real-time experiences across various AI platforms, this selection of videos captures real prompts and user reactions. Our AIQ database of thousands of videos is growing monthly. We'll periodically pull a few to compare

AI: A Useful Travel Agent

By Pulse Labs • August 1, 2023
Travel and trip planning is seen as a natural use case for AI platforms. In the planning stage, consumers are making large considered purchases and need to compare and weigh different options with a number of variables. Consumers see high potential in AI’s ability to alleviate trip planning

Top topics: travel and trip planning videos

By Pulse Labs • July 13, 2023
In AIQ, one component of the research design that we call “self-directed” is to let users decide what they want to do on an assigned AI platform. While the topics and interests they chose vary, overall we see certain topics and use cases emerge as the most popular.

Younger users want more human-like AI

By Pulse Labs • June 26, 2023
Across all users, the top three reasons for choosing a preferred AI response in our blinded tests are:Feels accurateFeels relevantFeels completeIn addition, among younger users, the attribute “feels human like” increases in importance. This video of a young user asking Bard for a breakup

AITube: Watch what happens on each platform

By Pulse Labs • June 26, 2023
Our study captures hundreds of videos of actual user experiences with the four AI platforms. A unique and powerful aspect of AIQ™️ is the ability to see these interactions in the moment.Two videos were chosen for each AI platform that demonstrate meaningful user insights. See what users

AIQ Stories

By Pulse Labs • June 1, 2023
Where AI becomes first choice over searchSearch is so dominant that it’s stunning to think it could be replaced by something else. But if you can only start in one place, where would you start? We asked that question of 227 US consumers familiar with AI to
Share by: