AI-driven issue triage: AWS’s 90% faster response time strategy

How to manage thousands of issues without scaling your team.

Managing thousands of open issues is every project maintainer’s nightmare, but what if AI could handle the heavy lifting? In this lightning talk at All Things Open, Arundeep Nagaraj and Dan Kiuna from Amazon Web Services share how the team transformed their issue triage workflow using domain-specific LLMs, cutting response times by 90% and catching critical regressions in under two minutes.

Read more: Deep dive into the Model Context Protocol

From overwhelmed to optimized

Popular open source projects face a scaling crisis. VS Code alone has over 8,800 open issues and counting, a number that keeps trending upward despite having more than 3,000 contributors. Arun and his colleague Dan experienced this firsthand while maintaining AWS’s Amplify and CDK projects. 

Their traditional triage workflow consumed 15 to 60 minutes per issue, requiring maintainers to manually read issue descriptions, reproduce problems locally, categorize bugs versus feature requests, and wait days for customer clarifications. 

The team’s solution started with automatic sentiment analysis. By feeding incoming issues through an AI-powered system, they could instantly identify highly negative sentiment and escalate critical problems to the front of the queue. This shift eliminated the need for dedicated staff to manually prioritize issues and helped engineers focus on high-impact problems first. 

But the real breakthrough came when they integrated Claude with Model Context Protocol (MCP) capabilities. Now, as engineers work on GitHub issues, Claude summarizes the problem, highlights critical details, analyzes the relevant source code in about one minute, and even suggests potential fixes, all before a human reviewer approves the response.

Beyond faster responses

The results speak for themselves. Total triage time dropped from 15 to 60 minutes per issue down to just 5 to 10 minutes. The team caught one regression in under two minutes, responded to the customer almost immediately, and shipped a fix within 24 hours. But the benefits extend beyond speed. 

The sentiment analysis system provides historical context, tracking whether major feature releases correlate with spikes in negative feedback. This data helps product and engineering teams make informed decisions about prioritization and feature development. Engineers report less frustration too, no longer spending excessive time on lower-priority issues while critical problems wait in the queue.

Key takeaways

1. Sentiment analysis transforms prioritization: Automatically identifying negative sentiment in incoming issues allows teams to escalate critical problems immediately and reduce customer frustration without requiring dedicated triage staff.
2. AI agents dramatically reduce triage time: Claude with Model Context Protocol cuts issue analysis from 15-60 minutes to 5-10 minutes by automating summarization, source code analysis, and fix suggestions.
3. Historical sentiment tracking informs product decisions: Tracking sentiment patterns over time reveals how feature releases impact user satisfaction, providing actionable data for planning and prioritization.

The human-in-the-loop advantage

Arun emphasizes that their system isn’t about letting AI run wild. Human reviewers validate every response before it reaches customers, ensuring accuracy while still achieving massive efficiency gains. For teams drowning in issues across popular open source projects like Flutter, React Native, and PyTorch, this approach offers a practical path forward. The combination of domain-specific LLMs, sentiment analysis, and thoughtful automation proves that scaling open source communities doesn’t require an ever-growing army of maintainers, just smarter tools that amplify human expertise.

More from We Love Open Source

• Want to get into AI? Start with this.
• Deep dive into the Model Context Protocol
• The secret skill every developer needs to succeed with AI today
• How I use AI agents to automate my workflow and save hours
• Skip the crowded job hunt: Find your tribe instead