Is your LLM lying to you?

Watch how GraphRAG eliminates hallucinations vector-only RAG can't fix.

Why vector similarity doesn’t equal relevance and what to do about it.

LLMs hallucinate. Vector-only RAG (Retrieval-Augmented Generation) helps, but it’s not enough when you need answers that depend on relationships between data sources. In this presentation at All Things Open, Nyah Macklin, Senior Developer Advocate at Neo4j, shares why adding knowledge graphs to your retrieval path solves the fundamental problem: Vector databases measure semantic similarity, which doesn’t always equal relevance, and they can’t traverse the connections that make complex answers possible.

Imagine running IT infrastructure at a utility company with energy grid data in one system, customer records in another, and maintenance logs in unstructured text. You want to answer questions like “which equipment has vibration anomalies” or “which customers are impacted if we take this transformer offline.” Vector-based RAG struggles because the data is fragmented and the logic spans multiple systems. The term “high-risk” requires understanding how relationships impact information, which assets power which customers, maintenance history, and customer dependency. GraphRAG solves this by mapping key entities into a knowledge graph that AI can traverse in a fast, verifiable, explainable manner.

The core limitation is that vector similarity doesn’t equal relevance. Vector databases measure semantic closeness in embedding space, not actual relationships. Nyah demonstrates with a concrete example: If asked “who is on the product management team,” a vector database might incorrectly identify someone who frequently accessed product documents, inferring team membership from document access patterns. A knowledge graph uses explicit nodes and relationships, returning only people with actual “MEMBER_OF” relationships to the product team. The structure is simple: Apples and oranges are both nodes with “IS_A” relationships to fruit, making queries precise instead of approximate.

Read more: How to secure agentic AI with Agent Identity Protocol (AIP)

The accuracy improvements are proven with independent research. A Data.world white paper tested 43 questions and found average response quality was three times higher when using knowledge graph plus vector search compared to vector-only. LinkedIn’s research on customer service question answering showed a 28.6 percent decrease in per-issue resolution time. Watch Nyah demonstrate live comparisons in the video, showing how GraphRAG transforms vague care plans into specific treatments and generic summaries into complete biomarker lists.

Key takeaways

• Vector databases measure semantic similarity, not relevance. They miss critical domain-specific context because they only use unstructured data and metadata, not the relationships in structured data.
• GraphRAG combines vector retrieval with knowledge graph traversal. This enables answering complex questions requiring relationship aggregation, like supply chain analysis or source code dependencies.
• Independent research proves 3x higher response quality and 28.6% faster issue resolution. Knowledge graphs provide explainability, traceability, and access controls that vector-only architectures can’t deliver.

GraphRAG isn’t replacing vector search, it’s adding a layer that makes AI applications production-ready by solving data management challenges including organization, standardization, and query diversity at scale.

💡Learn more about knowledge graphs and AI on the Neo4j YouTube channel.

More from We Love Open Source

• Zero trust Kubernetes: Context over credentials
• OpenClaw: Anatomy of a viral open source AI agent
• How to secure agentic AI with Agent Identity Protocol (AIP)
• The AI slop problem threatening open source maintainers
• Stop opening firewall ports and start using identity

📺 Watch the All Things Open “Extra’s” playlist