Retrieval-augmented generation is the standard way to ground a language model in domain knowledge. In IT operations the interesting question is not whether to use RAG, but which corpus you retrieve from — and what you are allowed to do with what comes back.
Closed RAG: the default path
Closed RAG retrieves from a corpus that never leaves the customer estate: their own incident history, their runbooks, their configuration data, their change records, and their asset inventory. Every retrieval is grounded in something that happened on their infrastructure.
The strengths are exactly what operations needs:
- High precision on recurring incidents, because the corpus contains prior resolutions of the same signature on the same asset class.
- Provenance — every suggestion traces to a specific prior incident with a known outcome.
- No data egress, which removes the entire class of confidentiality concerns.
- Low latency, since retrieval is local.
The weakness is equally clear: a closed corpus cannot help with something that has never happened before. Cold-start estates and genuinely novel failures fall outside it.
Open RAG: the fallback path
Open RAG augments with an external model that has broad general knowledge — vendor documentation, error-code semantics, known software defects, protocol behaviour. It is genuinely useful for the 10 to 13% of incidents that are novel, and it is the highest-risk component in the system.
Two risks dominate. The first is data leakage: incident context sent to an external model contains hostnames, topology, and software versions. The second is confident fabrication: an external model will produce a plausible remediation for an error code it has never actually seen, phrased with the same confidence as one it has.
The six-stage filter
Every external call passes through six stages, in order. Failure at any stage halts the call.
- Eligibility — is closed RAG genuinely insufficient? If a closed-corpus match exists above the confidence floor, no external call is made at all.
- Classification — is the incident context in a category permitted to leave the estate? Assets tagged as restricted are never eligible, regardless of incident severity.
- Redaction — hostnames, IPs, account names, schema names, and any matched secret patterns are replaced with stable pseudonyms before the payload is constructed.
- Minimisation — the payload is reduced to the error semantics and the minimum context needed. Full logs are never sent; extracted signatures are.
- Response validation — the returned suggestion is parsed against the platform action schema. Free-text advice that does not map to a known action type is not converted into anything executable.
- Human gate — any remediation derived from an external call is routed to a human as advisory, with the external origin explicitly labelled.
Routing between them
The router is deliberately biased toward closed. An incident goes to closed RAG first, always. The external path is considered only when the closed corpus returns no match above the confidence floor, or returns matches whose historical resolution outcomes were unsuccessful.
In steady-state deployments, between 6% and 11% of incidents reach the external path. That share falls over time as the closed corpus accumulates resolutions — which is the intended behaviour: every novel incident resolved by a human becomes closed-corpus knowledge for the next occurrence.
Handling the cold start
A new deployment has no closed corpus, which would make the external path dominant in month one — exactly backwards. We address this three ways: seeding with reviewed policy templates and signature libraries that carry no customer data; ingesting the customer's existing runbooks and historical ticket exports during implementation; and running observe mode, which builds resolution history against real incidents before any autonomy is enabled.
By the end of a typical observe period the closed corpus is dense enough that the external path settles into its normal single-digit share.
Evaluating a vendor's claims
If you are assessing any AI operations product, these are the questions that separate architecture from marketing:
- What exactly is sent to an external model, and can I see a sample payload?
- Can an external model response ever result in an executed action without human approval?
- What is redacted before egress, and how is redaction verified?
- Can I disable the external path entirely and still operate?
- Is the external-path share of incidents visible to me as a metric?
The fourth is the most diagnostic. If disabling external calls breaks the product, the closed corpus was never doing the work.
“A model that has never seen your estate should never be trusted to change it.”
Arjun Kumar, Chief AI Officer