init.news●A FIELD JOURNAL
FOR HUMAN OPERATORS
Agent systems / FIELD GUIDE3 MIN READ

Before you add a graph, find the missing answer.

A decision worksheet for teams choosing between a simpler retrieval baseline and a graph-based memory system.

Get the useful answer ↓

THE USEFUL ANSWER

Evaluate GraphRAG when your important questions need relationships across documents or themes across a collection. Keep a retrieval baseline and compare both approaches on the same questions, corpus, model, and scoring rubric.

Measure answer support and operating cost as well as answer quality. A graph is useful only if it resolves a failure you can demonstrate.

A crimson thread connects an ivory source sheet through a row of smoked-glass archive cards.
A connected knowledge system still needs an evaluation against its source material.AI-GENERATED EDITORIAL ART

WHAT YOU'LL LEAVE WITH

A reusable evaluation worksheet, a graph-portability test and an adoption rule, with no unrun benchmark presented as a result.

  • Start with failed questions, not a graph vendor or construction pipeline.
  • Run the same retriever against more than one graph representation before committing to a migration.
  • Measure support, refresh cost and abstention behavior alongside answer quality.
A missing answer should justify the migrationA proposed retrieval experiment starts with a missing answer, establishes a baseline, and compares the candidate approach. No retrieval score or graph benefit is claimed./ AGENT SYSTEMSA WORKING MODELILLUSTRATION / NOT MEASURED RESULTSMISSING ANSWERBASELINECOMPAREADD STRUCTURE FOR A REASON
A missing answer should justify the migrationILLUSTRATIVE WORKFLOW

01Match the method to the failure

Microsoft's GraphRAG documentation describes a pipeline that extracts entities and relationships, forms communities, and summarizes them. Global search uses community summaries for collection-wide questions; local search focuses on entities and nearby context. The documentation identifies connecting information across sources and summarizing broad themes as weaknesses of a plain similarity-search baseline. These are the project's stated motivations, not evidence that every workload improves. [1]

02Make a question set before a migration plan

Our proposed evaluation begins with a small, versioned set of real work questions. Separate exact lookup, relationship traversal, collection-wide synthesis, and questions the system should decline to answer. Write expected evidence before comparing outputs. If you choose questions after seeing the graph's answers, it becomes too easy to select a flattering test.

Swipe or scroll to compare

Question classExampleWhat to inspect
Exact lookupWhich document records the approved release date?Correct source and date, without extra inference
RelationshipsWhich decisions depended on the same assumption?Explicit supporting links across source documents
SynthesisWhich recurring constraints explain delayed releases?Coverage and support across the collection
AbstentionWhat did the customer approve when no approval is recorded?An explicit evidence gap instead of a confident invention

03Keep the comparison honest

  • Freeze the document snapshot and record its identifier. Keep prompts, model versions, and retrieval settings with the run.
  • Start with a credible baseline: appropriate chunking, useful metadata filters, and a retrieval method that suits the corpus. Do not use a deliberately weak baseline.
  • Score supported claims, missing evidence, unsupported assertions, and useful abstentions separately. Have a reviewer inspect the source passages without knowing which system produced the answer when practical.
  • Record indexing time, refresh cost, query latency, token use, and failure recovery. A quality gain can still be the wrong operational choice.
  • Repeat with one update to the corpus. Test whether changed facts, deleted sources, and contradictory records are handled in a way your application can explain.

04Test whether the retriever survives a new graph shape

ArcRAG inspects the labels, relations and properties reachable from named entities at query time, plans a multi-hop traversal, generates Cypher and repairs failed queries. The same configuration is evaluated against schema-guided and open-information-extraction graphs on MuSiQue, HotpotQA and TriviaQA. The paper supports a portability experiment, but absolute accuracy still changes with graph construction, and the results were not reproduced here. [2]

Add one representation swap to the worksheet. Keep the source corpus, questions, retriever configuration and scoring fixed. Compare supported answers, failed queries, repair attempts, latency and refresh cost. A portable interface is useful only if it survives your ontology, access rules and operating budget.

05Define the adoption rule in advance

Pick the question classes that matter to the product, then set acceptable quality and cost thresholds before running the comparison. Keep the simpler system if the graph does not clear those thresholds. If only synthesis improves, consider routing that class of question to a different retrieval path instead of migrating every request.

That routing recommendation is an implementation hypothesis. This article has not run a GraphRAG benchmark, measured a speedup, or established a universal threshold. The useful artifact is the test design: a way to reject an unnecessary migration as confidently as you accept a justified one.

FOLLOW THE EVIDENCE

Sources & checks

  1. Microsoft GraphRAG: overview and query modes ↗

    [1] Graph construction, community summaries, global/local query modes, and the stated limitations of baseline retrieval.

    CHECKED 2026-10-07
  2. ArcRAG: Agentic Compositional Retrieval over Knowledge Graphs ↗

    [2] Query-local interface induction, Cypher planning and repair, graph-construction comparison and reported results. Results not reproduced here.

    CHECKED 2026-11-06

Sources stay attached so you can check the argument. Meaningful revisions keep this address.

FROM A GOOD READ TO A USEFUL NEXT STEP

Put one idea
to work.

Pick the problem on your desk. Try the method on the page, or take the whole kit.

The method is public. The kit needs one email address.

FIELD KIT 02 / EDITABLE WORKBOOK / HTML + EXPORTS

The agent decision kit

You are about to add another agent, tool, or memory layer and cannot say what it will buy you.

Decide whether to add it, and write down the check that would show it earned its place.

  • A handoff receipt template
  • A local inference test matrix
  • Memory and MCP decision checks

One decision, one sitting. Keep the receipt for the next one.

Get the kitNot available yet

Private downloads are not available yet. No email is collected while this form is unavailable.

The download appears on this page. Nothing is emailed to you, and the optional box stays off unless you tick it.