Welcome to Eye on AI. On this version…AI is outperforming some professionals…Google plans to deliver adverts to Gemini…main AI labs group up on AI agent requirements…a brand new effort to provide AI fashions an extended reminiscence…and the temper activates LLMsand AGI.
Greetings from San Francisco, the place we’re simply wrapping up Fortune Brainstorm AI. On Thursday, we’ll deliver you a roundup of insights from the convention. However in the present day, I need to discuss some notable research from the previous few weeks with doubtlessly massive implications for the enterprise impression AI could have.
First, there was a examine from the AI evaluations firm Vals AI that pitted a number of authorized AI functions in addition to ChatGPT in opposition to human attorneys on authorized analysis duties. All the AI functions beat the typical human attorneys (who have been allowed to make use of digital authorized search instruments) in drafting authorized analysis stories throughout three standards: accuracy, authoritativeness, and appropriateness. The attorneys’ mixture median rating was 69%, whereas ChatGPT scored 74%, Midpage 76%, Alexi 77%, and Counsel Stack, which had the very best general rating, 78%.
One of many extra intriguing findings is that for a lot of query varieties, it was the generalist ChatGPT that was probably the most correct, beating out the extra specialised functions. And whereas ChatGPT misplaced factors for authoritativeness and appropriateness, it nonetheless topped the human attorneys throughout these dimensions.
The examine has been faulted for not testing a number of the better-known and most generally adopted authorized AI analysis instruments, equivalent to Harvey, Legora, CoCounsel from Thompson Reuters, or LexisNexis Protégé, and for under testing ChatGPT among the many frontier general-purpose fashions. Nonetheless, the findings are notable and comport with what I’ve heard anecdotally from attorneys.
A short time in the past I had a dialog with Chris Kercher, a litigator at Quinn Emanuel who based that agency’s knowledge and analytics group. Quinn Emanuel has been utilizing Anthropic’s normal function AI mannequin Claude for lots of duties. (This was earlier than Anthropic’s newest mannequin, Claude Opus 4.5, debuted.) “Claude Opus 3 writes better than most of my associates,” Kercher instructed me. “It just does. It is clear and organized. It’s a great model.” He mentioned he’s “constantly amazed” by what LLMs can do, discovering new points, methods, and techniques that he can use to argue circumstances.
Kercher mentioned that AI fashions have allowed Quinn Emanuel to “invert” its prior work processes. Prior to now, junior attorneys—who’re often known as associates—used to spend days researching and writing up authorized memos, discovering citations for each sentence, earlier than presenting these memos to extra senior attorneys who would incorporate a few of that materials into briefs or arguments that may really be offered in courtroom. Right this moment, he says, AI is used to generate drafts that Kercher mentioned are by and huge higher, in a fraction of the time, after which these drafts are given to associates to vet. The associates are nonetheless chargeable for the accuracy of the memos and citations—simply as they all the time have been—however now they’re fact-checking the AI and enhancing what it produces, not performing the preliminary analysis and drafting, he mentioned.
He mentioned that probably the most skilled, senior attorneys typically get probably the most worth out of working with AI, as a result of they’ve the experience to know the right way to craft the right immediate, together with the skilled judgment and discernment to rapidly assess the standard of the AI’s response. Is the argument the mannequin has give you sound? Is it more likely to work in entrance of a selected choose or be convincing to a jury? These types of questions nonetheless require judgment that comes from expertise, Kercher mentioned.
Okay, in order that’s regulation, nevertheless it probably factors to methods by which AI is starting to upend work inside different “knowledge industries” too. Right here at Brainstorm AI yesterday, I interviewed Michael Truell, the cofounder and CEO of sizzling AI coding instrument Cursor. He famous that in a College of Chicago examine trying on the results of builders utilizing Cursor, it was typically probably the most skilled software program engineers who noticed probably the most profit from utilizing Cursor, maybe for a number of the identical causes Kercher says skilled attorneys get probably the most out of Claude—they’ve the skilled expertise to craft one of the best prompts and the judgment to raised assess the instruments’ outputs.
Then there was a examine out on the usage of generative AI to create visuals for commercials. Enterprise professors at New York College and Emory College examined whether or not commercials for magnificence merchandise created by human consultants alone, created by human consultants after which edited by AI fashions, or created completely by AI fashions have been most interesting to potential customers. They discovered the adverts that have been completely AI generated have been chosen as the best—growing clickthrough charges in a trial they performed on-line by 19%. In the meantime, these created by people and edited by AI have been really much less efficient than these merely created by human consultants with no AI intervention. However, critically, if folks have been instructed the adverts have been AI-generated, their probability of shopping for the product declined by nearly a 3rd.
These findings current a giant moral problem to manufacturers. Most AI ethicists assume folks ought to typically be instructed when they’re consuming content material generated by AI. And advertisers do want to barter numerous Federal Commerce Fee rulings round “truth in advertising.” However many adverts already use actors posing in numerous roles without having to essentially inform those who they’re actors—or the adverts achieve this solely in very positive print. How completely different is AI-generated promoting? The examine appears to level to a world the place increasingly more promoting can be AI-generated and the place disclosures can be minimal.
The examine additionally appears to problem the standard knowledge that “centaur” options (which mix the strengths of people and people of AI in complementary methods) will all the time carry out higher than both people or AI alone. (Typically that is condensed to the aphorism “AI won’t take your job. A human using AI will take your job.”) A rising physique of analysis appears to recommend that in lots of areas, this merely isn’t true. Typically, the AI by itself really produces one of the best outcomes.
However it’s also the case that whether or not centaur options work effectively relies upon tremendously on the precise design of the human-AI interplay. A examine on human medical doctors utilizing ChatGPT to assist prognosis, for instance, discovered that people working with AI might certainly produce higher diagnoses than both medical doctors or ChatGPT alone—however provided that ChatGPT was used to render an preliminary prognosis and human medical doctors, with entry to the ChatGPT prognosis, then gave a second opinion. If that course of was reversed, and ChatGPT was requested to render the second opinion on the physician’s prognosis, the outcomes have been worse—and in reality, the second-best outcomes have been simply having ChatGPT present the prognosis. Within the promoting examine, it could have been good if the researchers had checked out what occurs if AI generates the adverts after which human consultants edit them.
However in any case, momentum in the direction of automation—typically and not using a human within the loop—is constructing throughout many fields.
FORTUNE ON AI
Unique: Glean hits $200 million ARR, up from $100 million 9 months again—by Allie Garfinkle
Cursor developed an inside AI assist desk that handles 80% of its staff’ help tickets, says the $29 billion startup’s CEO —by Beatrice Nolan
HP’s chief business officer predicts the long run will embrace AI-powered PCs that don’t share knowledge within the cloud —by Nicholas Gordon
How Intuit’s chief AI officer supercharged the corporate’s rising applied sciences groups—and why not each firm ought to observe his lead —by John Kell
Google Cloud CEO lays out 3-part technique to fulfill AI’s vitality calls for, after figuring out it as ‘the most problematic thing’ —by Jason Ma
OpenAI COO Brad Lightcap says code purple will ‘force’ the corporate to focus, because the ChatGPT maker ramps up enterprise push —by Beatrice Nolan
AI IN THE NEWS
EYE ON AI RESEARCH
Google has created a brand new structure to provide AI fashions longer-term reminiscence. The structure, referred to as Titans—which Google first debuted in the beginning of 2025 and which Eye on AI lined on the time—is paired with a framework named MIRAS that’s designed to provide AI one thing nearer to long-term reminiscence. As a substitute of forgetting older particulars when its quick reminiscence window fills up, the system makes use of a separate reminiscence module that frequently updates itself. The system assesses how stunning any new piece of knowledge is in comparison with what it has saved in its long-term reminiscence, updating the reminiscence module solely when it encounters excessive shock. In testing, Titans with MIRAS carried out higher than older fashions on duties that require reasoning over lengthy stretches of knowledge, suggesting it might finally assist with issues like analyzing advanced paperwork, doing in-depth analysis, or studying constantly over time. You possibly can learn Google’s analysis weblog right here.
AI CALENDAR
Jan. 6: Fortune Brainstorm Tech CES Dinner. Apply to attend right here.
Jan. 19-23: World Financial Discussion board, Davos, Switzerland.
Feb. 10-11: AI Motion Summit, New Delhi, India.
BRAIN FOOD
At NeurIPS, the temper shifts in opposition to LLMs as a path to AGI. The Data reported {that a} rising variety of researchers attending NeurIPS, the AI analysis area’s most essential convention—which befell final week in San Diego (with satellite tv for pc occasions in different cities)—are more and more skeptical of the concept massive language fashions (LLMs) will ever result in synthetic normal intelligence (AGI). As a substitute, they really feel the sector might have a wholly new sort of AI structure to advance to extra human-like AI that may frequently be taught, can be taught effectively from fewer examples, and may extrapolate and analogize ideas to beforehand unseen issues.
Figures equivalent to Amazon’s David Luan and OpenAI co-founder Ilya Sutskever contend that present approaches, together with large-scale pre-training and reinforcement studying, fail to supply fashions that actually generalize, whereas new analysis offered on the convention explores self-adapting fashions that may purchase new information on the fly. Their skepticism contrasts with the view of leaders like Anthropic CEO Dario Amodei and OpenAI’s Sam Altman, who consider scaling present strategies can nonetheless obtain AGI. If critics are appropriate, it might undermine billions of {dollars} in deliberate funding in present coaching pipelines.
