Fire-and-forget click ingestion means the redirect server records a click by handing an event to a background producer and returning the redirect (a 302 unless the link is set to 301 or 307) straight away, without waiting to hear that the event was stored. A separate worker reads events off a bus later, writes them to the analytics store in batches, and handles anything else that wants a copy. The visitor gets their redirect fast. The price is that a small window exists where an event can be lost.
That's the whole idea. The rest of this post covers how Elido's click event pipeline works, from the redirect handler to the batch insert, where events can go missing and what we'd change. The latency side of the story lives in the cornerstone, how we keep redirect p95 under 15ms.
What Fire-and-Forget Means on the Redirect Path
The redirect handler has one job. Resolve the slug, pick the destination, send the Location header. Once the destination is known (usually from the two-tier cache), the handler builds a click event: slug, workspace and link IDs, user agent, referrer, IP, country, region, city and coordinates from a local GeoIP lookup, device and browser, plus any suspicion flags from bot detection.
Then it starts the emit on its own goroutine and carries on writing the response. Here is the producer, trimmed only for width:
func (p *Producer) Emit(ctx context.Context, e Event) {
if p == nil {
return
}
b, err := json.Marshal(e)
if err != nil {
p.log.Warn("click marshal", zap.Error(err))
return
}
rec := &kgo.Record{Topic: p.topic, Value: b}
p.client.Produce(ctx, rec, func(_ *kgo.Record, err error) {
if err != nil && p.log != nil {
p.log.Warn("click produce", zap.Error(err))
}
})
}
Produce is asynchronous. The client library buffers the record and sends it in the background, and the callback only logs. No retry queue in the redirect process, no disk spool, no waiting for an ack.
The nil check is there for local development. With no broker configured the constructor returns a nil producer and every emit becomes a no-op, so you can run the redirect server on a laptop with nothing else installed.
Why Not Write Clicks Synchronously
Writing each click straight into the analytics database from the redirect server looks simpler. That's not what Elido does, for three reasons that stack.
Latency is the first. Any synchronous network call on the redirect path adds its full round trip to what the visitor waits for, and the tail of that round trip is what hurts p95. Our budget has no room for it.
Batch shape is the second. Columnar stores are built for large inserts. The usual vendor guidance is to insert in large batches rather than row by row, because every insert creates a part that background merges then have to fold together. Single-row writes from many redirect servers would be the worst possible pattern.
Failure isolation is the third. If the database restarts or slows down, a synchronous design passes that straight on to redirects: either they time out or clicks vanish. With a bus in the middle, each side fails on its own. Redirects keep working while analytics is degraded, and the backlog waits on the bus until the database is back, for as long as the topic's retention holds.
How the Click Ingestion Worker Batches Events
The worker is a Go service in a consumer group on the click topic. Auto-commit is off. Its loop is short:
- Poll a batch of records from the topic.
- Decode each record into a typed event.
- Push the event, together with its record, into the writer's in-memory buffer.
- Hand the click to any configured pixels and event forwarders, and publish a small event for the live click counter.
The writer flushes at 1,000 events or every 5 seconds, whichever comes first. Both are defaults in the service config. A flush prepares one batch insert into the click_events table with a 30-second timeout and sends it as a single statement.
The order of operations is what makes this at-least-once. Only after the insert succeeds does the writer mark those records and commit their offsets. If the insert fails, the batch goes back to the front of the buffer and the next flush tries again. If the process dies before the commit, the group rebalances and the next owner reads from the last committed offset, so the events are written again.
That has a cost I'd rather state plainly. The table is a plain MergeTree with no dedup key, so a crash between insert and commit leaves duplicate rows. The code comment calls it rare and says analytics tolerates it, which matches what we see. It's still a known gap, not a guarantee of exactly-once.
Poison messages are handled the other way. A record that fails to decode is logged and its offset marked, so one malformed record can't block the partition forever. That event is gone, and today nothing keeps the raw bytes.
Where Click Events Can Be Lost
Fire-and-forget gives up some events on purpose. It helps to be exact about where.
- Between the redirect and the bus. Records sit in the producer's buffer for a short time before they're sent. On shutdown the redirect server closes the client, but there's no dedicated drain step that waits for the buffer to empty, so a deploy can lose whatever was in flight.
- When the bus is unreachable. The client retries, but not forever. If the bus stays down past that, the produce callback logs a warning and the record is dropped.
- Malformed records. As above, skipped and logged.
After an event reaches the bus, the consumer side doesn't drop it. It can duplicate it.
There's no published loss rate. I haven't measured one I'd stand behind, and inventing a decimal would be worse than saying so. What we can say is why the design is acceptable for this workload: clicks are a statistical signal. Geo split, referrer mix and campaign comparisons hold up fine with a small miss rate. Anything billed per click, or anything an auditor reconciles row by row, needs a durable write path that this pipeline doesn't offer. If that's you, look at webhooks vs polling for click tracking and pull counts from the analytics API instead of relying on a stream.
What Happens When the Analytics Store Is Down
When inserts fail, the writer keeps the batch and retries on every flush. Offsets aren't committed, so the bus still holds every unprocessed event.
The weak spot is memory. The consumer keeps polling while inserts fail, and the writer's buffer keeps growing with nothing to cap it. A short outage is fine. A long one under real traffic could push the worker toward its memory limit, and if it restarts, it simply rereads from the last committed offset. Nothing is lost that way, but it's not elegant either. Pausing the fetch loop while the writer is failing is the obvious fix.
How long you can survive an outage comes down to the topic retention you configure on the bus, not to anything in the worker. Set it longer than the worst rebuild you can imagine.
Fan-Out: Pixels, Forwarders and Live Clicks
The worker is also where every other consumer of a click gets its copy.
- Server-side event forwarders. Per workspace, the worker can forward clicks to GA4 (Measurement Protocol), Klaviyo, Mixpanel, Brevo or Datadog, using a cached config. Each HTTP call has a 5-second client timeout. More on the GA4 side in GA4 server-side tracking via redirects.
- Server-side ad events. When a workspace has saved a Meta Conversions API or TikTok Events API access token, the worker sends each click as a Meta
PageViewor TikTokViewContent, with the same 5-second timeout, at most 64 requests in flight per instance and no retries. These are clicks, not purchases; server-side conversion tracking covers the purchase side. - Click webhooks. Workspaces with an endpoint subscribed to
click.createdget a compact, privacy-reduced event produced onto a separate topic, which api-core fans out to the endpoints with a cap of 50 events per second each. - Live click stream. A minimal event goes to a pub/sub channel that feeds the live counter in the dashboard.
None of this runs on the redirect. It runs at decode time, before the batch is written, so a click can reach Mixpanel a few seconds before it shows up in your own reports. A click.created webhook arrives a few seconds after the click, because it goes through the topic and a delivery worker that polls every 5 seconds; webhooks for link events has the payload. If you're curious how this ties into plans, the analytics feature page lists what each tier gets.
Why a Log, Not a Queue
The bus had to be a replayable log, not a work queue. A queue deletes a message once it's acknowledged. A log keeps events for the retention window, so a new consumer, or one that has to rewind, can read history again. It also had to run on our own infrastructure in the same EU region as everything else, and speak a Kafka-compatible protocol, so an Enterprise customer who self-hosts by contract can point it at a broker they already operate. The redirect tier configuration is documented in the edge redirect architecture page.
If you'd like to see the output of this pipeline on your own links, create a free workspace and watch clicks land in the dashboard a few seconds after a redirect.
What We Would Change Next
The pattern is right for this workload. The implementation has edges worth naming if you're copying it.
A drain on shutdown. Flush the producer explicitly, with a timeout, before closing the client. Today a deploy can drop buffered records.
A dead-letter store for decode failures. Skipping poison records keeps the partition moving but leaves no trace beyond a log line. Writing the raw bytes and the error to a side table would make them auditable.
Dedup on retry. An event ID on each click and a dedup-aware table would turn at-least-once into effectively-once for reporting.
Backpressure while the store is down. Pause polling when the writer can't flush, so the buffer can't grow without bound.
Per-workspace limits on forwarders. All workspaces share one worker, so one slow third-party endpoint slows clicks for everyone.
None of these has caused an incident we know of. They're on our list, and they'd be on yours too.
Read the cornerstone: how we keep redirect p95 under 15ms.
Related on the blog
Frequently asked questions
What does fire-and-forget mean for click tracking?
The redirect hands the click event to a background producer and returns the redirect without waiting for any acknowledgement. If the event fails to reach the bus, it is logged and dropped. The visitor never pays for analytics in latency.
Can a fire-and-forget pipeline lose clicks?
Yes, between the edge and the event bus. A process killed with records still buffered, or a bus that stays unreachable past the client's retries, means those events are gone. After the event reaches the bus, the consumer side is at-least-once.
Why not write each click straight into the analytics database?
Two reasons. A network round trip per redirect eats the latency budget, and columnar analytics stores want large batches, not single-row inserts. A bus in between also means a database outage doesn't turn into failed redirects.
What is at-least-once delivery in a click pipeline?
The consumer only commits its position in the stream after a batch has been written. A crash before the commit means the same events get read and written again. Nothing is lost on that side, but a crash at the wrong moment can produce duplicate rows.
How often does Elido flush click batches?
The worker flushes at 1,000 events or every 5 seconds, whichever comes first. Both values are configuration defaults in the worker, not hard limits, so they can be tuned without a code change.
Do server-side forwarders and pixels slow down redirects?
No. They run in the click worker, well after the redirect has been served. A slow third-party endpoint can slow the worker down, which delays analytics a little, but it never touches the redirect response.
Try Elido
Paste a URL, get a working short link
No signup. Link lives for 30 days. Sign up to keep it forever.
Free account required. The free plan includes 25 links in total. Already have an account? Log in
