Three different kinds of automated agent hit your links, and they want different things. One is collecting text to train a model. One is fetching a page right now because a person asked a question. One is building a search index. They arrive with different user agents, honour different rules, and deserve different answers.
A redirect domain is an unusual thing to point them at, because it has no content. Every request is a lookup and a Location header, so a training crawler gets nothing useful and a live fetcher gets one extra hop on the way to the page it wanted. That asymmetry is what makes the policy decision straightforward once you separate the classes. If you are looking at the marketing side of this rather than the plumbing, answer engine optimization covers what actually gets a page cited.
Three Classes, Not One
Training crawlers. These collect content for model training. OpenAI documents its bots with a separate user agent for each job, and Anthropic documents its crawler and the rules it follows in the same spirit. Google handles the equivalent through a robots token rather than a distinct crawler, which its crawler overview sets out alongside the search agents.
Live fetchers. These run when a person asks an assistant something and the assistant decides to read a page. Perplexity documents the split explicitly, with one agent for indexing and another for the user-triggered request. The distinction matters more than any other point in this post: a live fetch is a person, one step removed.
Search indexers. The oldest class, and the one whose behaviour around redirects is best understood. They follow a permanent redirect, attribute the content to the destination, and treat the short domain as a signpost.
Same infrastructure, three purposes. A blanket rule for all of them is nearly always the wrong call.
What Happens When One Hits a Redirect
The mechanics are unremarkable. The agent requests the short URL, the redirect answers with a 301 or 302 and a Location header, and the agent follows it if it follows redirects at all. The HTTP semantics for redirection have not changed for this, and there is no special handling for AI agents anywhere in the standard.
Two consequences follow. The short domain sees a request that is not a person, so it lands in your click data unless something filters it. And the destination sees whatever the agent does next, which for a live fetcher is a normal read of the page.
The part people get wrong is which host the rules live on. RFC 9309 defines robots.txt as scoped to an authority: scheme, host and port. Rules at example.com/robots.txt govern example.com and nothing else. Your redirect domain is a different host, so it needs its own file, served by whatever answers on that host. Plenty of short domains serve no robots.txt at all, which agents read as no restrictions.
What It Does to Your Click Counts
Automated traffic is a measurement problem before it is a policy problem.
Any link that is publicly reachable will accumulate crawler hits. Links posted in public forums, documentation and social profiles attract far more than links sent to a private list, and the effect compounds over months while a campaign link is only alive for two weeks.
Elido drops known automated agents before the click is recorded rather than after, so the dashboard shows people. That is the behaviour to want, and it is worth verifying in any tool you use, because a raw counter treats a crawler sweep as a traffic spike. What is worth measuring in short-link analytics covers the filtered-versus-raw distinction, and clicks versus GA4 sessions covers what the leftover gap consists of.
One practical note: filtering is signature-based, so a new agent is invisible until its signature is known. If a link shows a sudden flat plateau of identical requests, look at the user agent before you look at the campaign.
What to Put in robots.txt on a Redirect Domain
A short, defensible position for a redirect-only host.
Allow the live fetchers. Those requests exist because somebody asked a question that your page might answer, and blocking them removes you from the answer without removing you from the training data you were worried about. This is the single most consequential line in the file.
Decide about training crawlers on principle rather than on effect, because a redirect domain contains nothing to train on. If your organisation has a position on model training, apply it here for consistency and expect no measurable change either way.
Leave search indexers alone unless you have a reason. They follow the redirect and credit the destination, which is what you want.
And remember what robots.txt does not do. It is a request, not an access control, and it has no effect on whether the redirect works for a person. A Disallow line does not break your links; it asks a compliant agent not to fetch them. Anything that ignores robots.txt is a different problem with different tools. A separate proposal, llms.txt, gets mentioned in the same breath, but it carries no enforcement behind it at all, which llms.txt explained covers on its own.
Running links across a domain you own and want to control? Elido's custom domains let you serve that host, which is the prerequisite for having any of these rules apply at all.
Does a Short Link Cost You a Citation?
Not by itself. An agent that follows the redirect reads the destination and cites what it found there, exactly as a search crawler does.
What does cost you is friction in the hop. A chain of two or three redirects before the content, a resolution that depends on JavaScript, a slow response, or a destination that answers with an interstitial. Each one is a chance for the fetch to be abandoned, and an abandoned fetch is a page that did not make it into the answer.
The rule I would give a content team: publish canonical URLs inside the content, where a model reads them as the address of the thing, and use short links for distribution, where a human reads them. That way the citation points at your page and the tracking still works on the channels where you need it. Agentic commerce links covers the case where the agent is transacting rather than reading.
Measuring the Traffic That Comes Back
Some of it is visible. Assistants that send a referrer show up as a normal referral row, and one of them appends a campaign parameter of its own, which is why chatgpt.com sometimes appears in reports as a source. The remainder arrives as direct traffic, because a chat interface has no obligation to identify itself.
Treat the visible share as a floor, not a total. If AI referrals matter enough to report on, the honest version is a directional trend from the sources you can see plus a note that the real figure is higher. Anyone presenting a precise AI-referral percentage is presenting the part they could measure.
Read the Cornerstone Series
This post sits in the engineering cluster. Short links as Terraform is the cornerstone for managing link infrastructure as code, and answer engine optimization covers the content side of being cited.
Related on the Blog
Frequently asked questions
Do AI crawlers follow short links?
Generally yes. A redirect is an ordinary HTTP response and well-behaved agents follow it to the destination, which is where the content they came for lives. The hop shows up in your logs as a request on the short domain, and the destination sees the visit that follows.
Does robots.txt on my main site cover my short domain?
No. Robots rules apply per host, so a file served at example.com says nothing about go.example.com. A redirect domain needs its own robots.txt, served by whatever answers requests on that host, or agents will treat it as unrestricted.
Should I block AI crawlers from my short links?
There is nothing on a redirect domain worth training on, so blocking training crawlers costs you little. Blocking the user-triggered fetchers is a different decision: those requests happen because a person asked an assistant about something, and a blocked fetch means your page does not make it into the answer.
Do AI bots inflate my click counts?
They inflate raw counts. Any tool that publishes unfiltered request totals will show automated traffic mixed in with people, and public links attract more of it than private ones. Filtering known agents before counting is the fix, and it is worth checking whether your shortener does that by default.
Can I see traffic that came from an AI assistant?
Partly. Some assistants send a referrer you can read, and some append a campaign parameter of their own, so a share of the traffic is identifiable in analytics. The rest arrives as direct, because a chat interface is not obliged to tell your site where the visitor came from.
Does using a short link hurt my chances of being cited?
Not materially, as long as the redirect is fast and resolves to a page the agent can read. What does hurt is a chain of hops, a link that requires JavaScript to resolve, or a destination that answers slowly. Publish canonical URLs in the content itself and keep short links for distribution.
Try Elido
Paste a URL, get a working short link
No signup. Link lives for 30 days. Sign up to keep it forever.
Free, no signup required · 2 per day