Your knowledge base freshness is set by the trigger, not the pipeline. Teams optimise ingestion down to 90 seconds, then fire it from a nightly cron and ship a p99 staleness of 24 hours. There are six trigger mechanisms and they are not interchangeable: webhooks (seconds, no delivery guarantee, no delete events), change data capture (sub-second, ordered, source must be a database you control), object-store events (seconds, at-least-once, no rename semantics), delta-token polling (minutes, the only option for most SaaS sources, and the only one that reports deletions reliably), TTL refresh-on-read (cheap, bounded only for documents someone actually queries, blind to deletes), and scheduled full scan with content hashing (hours, the only one that is actually complete). The correct architecture is not one of them. It is a fast push path for latency plus a slow reconciliation sweep for correctness, because every push mechanism drops events and none of them reliably tell you a document was deleted or unshared. Budget your staleness as trigger latency plus detection latency plus embed latency plus index visibility, measure index lag as source-modified-time to first-retrievable, and coalesce bulk edits or one template change will re-embed 40,000 documents at once.
The pipeline is not the problem. Most teams that come to us with a stale retrieval system have already built the hard part: a metadata registry, content hashing, delta indexing, a worker pool that turns 100 changed documents into new vectors in under two minutes. Then they run it from a cron entry at 02:00 and wonder why the assistant is quoting a pricing page that was corrected yesterday morning.
Ingestion speed and knowledge freshness are different properties. One is throughput and the other is latency, and the thing that determines the second is the trigger. A 90 second pipeline behind a nightly schedule has a p99 staleness of 24 hours and a mean of 12. The same pipeline behind a webhook has a p99 of about 90 seconds and a failure mode nobody is watching. Choosing between those, and knowing which one silently loses documents, is the actual design decision.
This post is the trigger layer: the six mechanisms that can fire a RAG update, what each one costs in latency and completeness, why every push mechanism needs a reconciler behind it, and how to measure whether any of it is working. If you have not built the ingestion side yet, start with updating a RAG knowledge base without rebuilding everything and come back here for the trigger topology.
Your staleness budget is four numbers, not one
Before comparing mechanisms, decompose what you are actually buying. The gap between a document changing at the source and the new content being retrievable is the sum of four independent stages:
staleness = trigger_latency # source event -> your system knows
+ detection_latency # your system knows -> job enqueued for THIS doc
+ processing_latency # fetch, parse, chunk, embed
+ visibility_latency # upsert -> returned by a queryTeams optimise processing_latency because it is the part with a dashboard. It is almost never the dominant term. On a nightly cron, trigger_latency averages 12 hours and everything else rounds to zero. On a webhook, trigger_latency is seconds and visibility_latency becomes interesting, because some vector indexes do not make an upsert immediately searchable and a few make it searchable before the index structure has caught up with it.
Write the budget down as a number per content class. Pricing and policy content that a regulated business quotes to customers is not the same class as an archived engineering wiki. In practice we see three tiers hold up well: a few minutes for content with legal or commercial consequences, under an hour for operational content, and daily for reference and archive. Each tier picks a different trigger, and mixing them in one pipeline is how the urgent document ends up queued behind a 40,000 document backfill.
The six RAG update triggers compared
Read that table as a layered stack rather than a menu. The bottom row is not a fallback for teams without webhooks; it is the correctness backstop that belongs under every one of the rows above it.
Change data capture: the fastest, and the most constrained
Tailing the write-ahead log gives you an ordered stream of every row that changed, including deletes, with no polling cost and no dropped events as long as the replication slot survives. If your knowledge base is a table in a database you run, this is the correct trigger and there is not a close second. The constraints are real. A retained replication slot that nobody consumes will grow the source database's WAL until the disk fills, which turns your indexing outage into a production database outage. Schema changes need handling. And the stream gives you rows, not documents, so a document assembled from four joined tables needs a materialisation step that decides which row changes actually warrant a re-embed.
Webhooks: seconds of latency, no guarantee of anything
Webhooks are the default choice for content management systems and SaaS sources, and they work well right up until they do not. Three failure modes matter and all three are invisible by default. Delivery is best-effort. Your endpoint returns a 502 during a rolling deploy, the source retries a handful of times over a few minutes, then stops. That document is now stale forever, with no error in your logs, because from your side nothing happened. The interesting events often have no hook. Hard deletes, permission revocations, and administrative bulk operations are frequently absent from the event catalogue. A document can leave the source entirely and your index will keep serving it. The payload tells you almost nothing. Most hooks carry an object ID and an event type, not a diff. A metadata-only touch and a full rewrite arrive looking identical, which is why the content hash gate in the next section is not optional. Treat a webhook as a latency optimisation over the reconciler, never as the mechanism of record. Always accept the hook, always enqueue by document ID, never trust that the set of hooks you received is the set of changes that happened.
Object-store event notifications: good, with rename as the trap
Bucket events are the cleanest push trigger available for file-shaped corpora. They are at-least-once, so your handler must be idempotent, which the content hash gate gives you for free. The trap is that object stores have no rename. A move is a delete plus a create with a new key, and a handler that ignores delete events (common, because deletes look scary) will index the new key and leave the old one in the index forever. Every folder reorganisation quietly doubles a slice of your corpus. Handle the delete leg on day one, and let the reconciler catch what you miss.
Delta-token polling: the only option for most SaaS sources, and the most honest
For document sources you do not operate, the change-token pattern is usually the best available mechanism: you call a changes endpoint with a cursor, you get back everything that changed since that cursor, including tombstones for deleted and unshared items, plus a new cursor to store. It costs one API call per interval per scope and returns nothing when nothing changed, so a one minute interval is often affordable. Two rules keep it correct. Persist the cursor transactionally with the work it represents, or a crash between "processed" and "cursor saved" replays or skips a window. And handle cursor expiry explicitly: most delta APIs invalidate a token that is older than some retention window and expect you to fall back to a full enumeration, which is exactly the reconciler you already have.
TTL and refresh-on-read: cheap, and quietly wrong
The lazy pattern stamps every indexed chunk with an ingestion timestamp and a freshness window. Retrieval returns the stored vector immediately, and if the record is older than its TTL the system schedules a background revalidation so the next query gets the corrected version. Some teams go further and block on revalidation for a small set of high-stakes documents. The appeal is cost. On a corpus with a long tail where a small fraction of documents serve most queries, you only pay to refresh what people actually read, and you get a bounded staleness guarantee on exactly the content that matters to users. The failure is structural and worth being blunt about: a TTL is a statement about your index, not about the source. It cannot detect a deletion, it cannot detect a permission change, and it will happily serve a stale answer for the first query after expiry and only fix it for the second. Use TTL as a cost control layered on top of a real trigger, never as the trigger. The one place it stands alone is a corpus where the source has no change API of any kind and a full sweep is prohibitive.
Scheduled scan plus content hash: slow, and the only complete one
List every document in the source with its ID and version or hash. Diff against your ingestion registry. Enqueue the difference in both directions: present in source but missing or outdated in index means re-ingest, present in index but absent from source means tombstone and delete. This is the only mechanism that converges. It needs no cooperation from the source beyond an enumeration API, it detects the deletions that every push path missed, and it repairs the documents that got lost during your 40 second deploy. It is also too slow and too expensive to be your only trigger on a large corpus, which is the whole reason the other five rows exist.
| Trigger | Typical latency | Delete events | Ordering | Completeness | Works when |
|---|---|---|---|---|---|
| Change data capture (WAL / logical replication) | sub-second to seconds | Yes, explicit | Total order guaranteed | Gap-free while the slot is retained | Content lives in a database you operate |
| Webhook / event subscription | seconds | Rarely, source-dependent | None | Best-effort, drops on your downtime | Source offers hooks and you can expose an endpoint |
| Object-store event notification | seconds | Yes, as delete markers | Per-key only | At-least-once, duplicates expected | Documents are files in a bucket |
| Delta-token polling | 1 to 15 minutes | Yes, usually as tombstones | Per-cursor | Complete within the cursor's retention | SaaS source exposes a change token API |
| TTL / refresh-on-read | unbounded until read | No | None | Only for documents that get retrieved | Long-tail corpus, cost-constrained |
| Scheduled scan plus content hash | hours | Yes, by absence | None needed | Total, by construction | Always. This is the floor |
The architecture: fast path for latency, sweep for correctness
The design that holds up is not a choice between these mechanisms. It is a push trigger for the p50 and a reconciler for the p99 and for everything the push trigger structurally cannot see.
# Fast path: any push trigger converges to the same enqueue call
def on_change_event(source: str, doc_id: str, priority: str = "normal") -> None:
queue.enqueue(
IngestJob(source=source, doc_id=doc_id, priority=priority),
dedupe_key=f"{source}:{doc_id}", # coalesce a burst of edits into one job
delay_seconds=30, # debounce: let a save-storm settle
)
# Every job re-derives truth from the source. Events are hints, never payloads.
def process(job: IngestJob) -> None:
doc = source_client(job.source).fetch(job.doc_id)
if doc is None: # deleted or access revoked upstream
index.delete_by_doc(job.doc_id)
registry.tombstone(job.doc_id)
return
body_hash = sha256(normalize(doc.body)) # strip volatile headers, timestamps, tracking params
record = registry.get(job.doc_id)
if record and record.body_hash == body_hash:
registry.touch(job.doc_id, doc.acl, doc.metadata) # ACL and metadata only, no embedding call
return
chunks = chunk(doc)
changed = [c for c in chunks if registry.chunk_hash(job.doc_id, c.ord) != sha256(c.text)]
index.upsert(embed([c.text for c in changed]), doc=doc)
registry.commit(job.doc_id, body_hash, chunks)
# Correctness path: runs on a schedule, repairs whatever the fast path dropped
def reconcile(source: str) -> Drift:
live = {d.id: d.version for d in source_client(source).enumerate()}
known = registry.versions(source)
stale = [i for i, v in live.items() if known.get(i) != v]
orphans = [i for i in known if i not in live] # deleted upstream, still in your index
for doc_id in stale:
on_change_event(source, doc_id, priority="bulk")
for doc_id in orphans:
index.delete_by_doc(doc_id); registry.tombstone(doc_id)
return Drift(stale=len(stale), orphans=len(orphans))Four properties of that shape are worth stating explicitly, because they are the ones teams skip.
Events are hints, not payloads. The job re-fetches from the source rather than trusting the event body. This makes every trigger mechanism interchangeable, makes duplicate delivery harmless, and means a replayed six-hour-old event still produces the correct current state.
The hash gate sits before the embedding call, not after. Push sources generate a large volume of notifications that carry no semantic change: a metadata touch, a workflow state transition, a re-publish of identical content. Normalising and hashing before you spend an embedding request is the single highest-leverage line in the pipeline.
Deletion is a first-class path. A fetch returning nothing is a delete, not an error to retry. Combine that with the reconciler's orphan sweep and you close the gap that webhooks leave open. Access control revocation deserves its own loop entirely, because no content changed and no content event will fire, which is the failure described in stale permissions in RAG document ACL sync.
The reconciler enqueues at bulk priority. A sweep that finds 8,000 drifted documents must not starve the pricing page that a webhook enqueued four seconds ago.
Coalescing: what one template change does to your embedding bill
The event volume from a content source is not smoothly distributed. It is quiet, then one administrator does a bulk metadata migration, a taxonomy rename, or a folder move, and 40,000 change events arrive in 90 seconds.
Three controls contain it, applied in this order:
If a single bulk operation legitimately invalidated the whole corpus, that is not an update problem, that is a rebuild, and the decision of when to cross that line is covered in when to re-embed documents in your vector database.
Measure three things, and none of them is queue depth
Queue depth tells you the workers are behind. It does not tell you the index is wrong, and the failures in this layer are almost all silent.
Index lag. For every completed job, record the difference between the source's last-modified timestamp and the moment the new version became retrievable. Chart p50 and p99, and split by source system. One connector whose token expired three weeks ago is invisible in a blended average and obvious the moment you facet it.
Reconciliation drift. Every sweep emits two counts: documents in the source that your index has wrong or missing, and documents in your index that no longer exist upstream. Both should trend toward zero between sweeps. Drift that climbs monotonically is the signature of a push path that is dropping events, and it is the only signal that will tell you before a user does.
Re-embed amplification. Embeddings generated divided by documents that actually changed semantically. Near 1 is healthy. A number several times higher means your normalisation is leaking a volatile field into the hash, and you are paying for it on every notification.
Add one alert that is worth more than the three charts: reconciliation drift above zero for two consecutive sweeps. A single non-zero sweep is normal timing. Two in a row means the fast path is broken and nobody noticed.
Picking a trigger by source system
The pattern across all six rows is the same: the push mechanism sets your latency, the sweep sets your correctness, and the sweep cadence should track how expensive a wrong answer is rather than how large the corpus is.
| Source shape | Recommended trigger | Reconciler cadence |
|---|---|---|
| Your own Postgres or MySQL | Logical replication / CDC | Daily, as a slot-health check |
| Files in S3, Azure Blob, GCS | Bucket event notifications, deletes included | Daily full listing diff |
| Enterprise document platform (Microsoft 365, Google Workspace) | Delta-token polling, 1 to 5 minutes | Hourly on recently active scopes, weekly full |
| Content management system with hooks | Webhook plus hash gate | Hourly, because delete hooks are unreliable |
| Public web or crawled sources | Scheduled crawl with conditional requests | The crawl is the reconciler |
| Air-gapped file shares | Filesystem watch plus scan | Hourly, since watchers miss events under load |
Where this sits in the wider retrieval design
Freshness is one of four failure classes that make a working RAG system stop working: stale content, wrong chunks, wrong permissions, and wrong evaluation. They interact. A tight trigger topology on top of a chunking strategy that splits a policy across two chunks buys you a fast path to a wrong answer, which is why document chunking that preserves context is worth settling before you tune triggers. And a freshness SLA you cannot measure is a freshness SLA you do not have, which is the argument for the retrieval testing described in how to tell if your RAG system actually works. The full map of these decisions lives on our RAG and vector search pillar.
At Particula Tech we build these pipelines on-premise for teams in healthcare, legal, and financial services, where the reconciler is not an engineering nicety but the artifact that lets someone answer "how do you know the model is not quoting a withdrawn policy?" with a number instead of a shrug. If your assistant is confidently citing last quarter's content, the fix is almost never a better embedding model. It is a trigger that fires and a sweep that proves it did.
FAQ
Quick answers to the questions this post tends to raise.
