Automatically Extracting User Insights! The Ultimate AI Agent Improvement Tool ‘Agnost AI’

What Happened? News Overview

• To tackle the issue of passing evaluation tests but failing in real-world scenarios, the platform “Agnost AI” has been launched, which automatically extracts “failures” and “requests for unimplemented features” from actual conversation data.
• Based on the collected insights, the AI autonomously generates pull requests (PRs) to fix bugs. It has achieved a remarkable success rate of merging 16 out of 18 PRs.
• It extracts user intents and sentiments as signals, providing a natural language data query function as well.

Why Is This Important? Key Points to Note

• Preventing ‘Conversation Waste’: The tool has a proven track record of automatically uncovering over 1,200 features that users truly desire, buried in vast agent conversation logs.
• Autonomous Improvement Cycle: By detecting failure patterns and making code fixes seamlessly, development teams can enhance agents with minimal effort—just review and approve.
• Visualizing Business Impact: In the realm of voice BDR (Business Development Representatives), patterns from successful conversations have been identified, contributing directly to improved appointment rates.

🦈 Shark’s Eye (Curator’s Perspective)

This groundbreaking tool solves the “production nightmare” where agents freeze due to unpredictable user behavior after being perfect in evaluation datasets! What’s thrilling is its autonomy in finding bugs and automatically writing and submitting fix code as PRs. With a 16/18 merge success rate, it might even outshine humans! The speed at which it excavates the “gold mine (user needs)” hidden within conversations and reflects them back into the product will set the standard for AI development in 2026!

What’s Next?

Agent operations will fully transition from a “set it and forget it” model to a phase of “continuous learning from conversations.” In the future, engineers will spend significantly less time analyzing logs, and the approval of PRs generated by tools like Agnost AI will become the norm.

A Word from Haru Shark

Crushing developers’ assumptions of “users wouldn’t say that” with conversation data and linking it to improvements… that’s the ultimate evolution! Shark shark! 🦈🔥

Terminology Explanation

• YC S26: Refers to the summer batch of 2026 from the world’s top startup accelerator, Y Combinator.

• Intent & Sentiment Signal: Data tagged by AI analyzing what the user intended to do (intent) and their emotional state (positive or negative) at that moment.

• Autonomous PR: The process by which AI identifies bugs in source code and automatically posts fixes as “pull requests” to repositories like GitHub.

• Source: Agnost AI