“Let’s start with a confession: I love bots. Or at least, I love the idea of them.”
They’re efficient, tireless, and, if designed well, can be downright helpful. (They can also be downright unhelpful, but that’s a topic for a different blog post.) But the incentives around bot traffic are completely out of balance, and that makes things messy.
Not all bots are bad, but they all cost someone something. Until we fix the incentives for identifying and managing automated traffic, we’ll keep having the same tired fight: block all bots and break useful functionality, or get overrun by them and save our content and services.
You can Subscribe and Listen to the Podcast on Apple Podcasts, or wherever you listen to Podcasts.
And be sure to leave me a Rating and Review!
What do we mean by “bot”?
Let’s clarify the terminology. “Bot” is a term that covers everything from benign automation to outright criminal activity. For the purposes of this post, we’re talking about non-human actors who interact with web services, some with permission and some without.
That includes:
- Helpful bots: search crawlers, accessibility checkers, uptime monitors
- AI agents: tools that retrieve or generate content, often hitting your site through a third-party interface
- Enterprise automation: scripts and services performing integrations across APIs
- Malicious bots: scrapers, credential-stuffers, spam engines, DDoS zombies
Some are essential, some annoying, and some outright hostile. According to the 2025 Imperva Bad Bot Report, automated traffic now makes up 51% of all web traffic, with 37% of that classified as malicious. Cloudflare Radar data has its own stats that indicate bots account for approximately 30% of global web traffic.
Regardless of the type of bot, they all generate load at a rate faster than humans can manage on their own. And that’s where things get tricky.
Identity is only the first step
There’s been a lot of work recently on figuring out how bots can identify themselves in a standardized, trustworthy way. The Web Bot Authentication discussion at the IETF is a good example. More and more content and service providers are demanding the ability to identify and/or differentiate bot traffic from human. Fewer (but not zero) bot developers are eager to support that goal. A handful want to be good actors, to say clearly, “Hey, I’m not a human, but I’m not here to cause trouble either.”
It probably goes without saying (but I’m going to say it anyway): If you’re building a polite, well-behaved bot, the last thing you want is to be lumped in with attackers. But the other side of the equation is the cost to the site your bot is connecting to. Knowing a bot’s identity doesn’t change the fact that other organizations’ infrastructures are paying the price; they may want to block you to protect themselves.
Even a verified, well-meaning AI agent scraping a site to summarize its content for someone’s personalized feed still hits that site’s CDN, database, and cloud compute budget.
And if they’re not charging for that access—if there’s no business model that connects bot traffic to revenue—then the only thing that providing some form of identity to a bot does is to give that polite visitor a name tag before they raid the pantry.
Why incentives matter
The developers building these bots often say, “We just want access. Don’t block us.” And the site operators reply, “We just want you not to break our infrastructure.”
That’s not a disagreement. That’s a misaligned incentive.
From the bot developer’s perspective:
- Self-identifying should reduce the risk of being blocked (though, for the moment, it doesn’t)
- A clear spec helps them integrate in good faith (if they can figure out where the spec is being developed)
- They’d rather focus on product, not evasion tactics (who wouldn’t rather make real progress than jump through hoops?)
From the service provider’s perspective:
- Every request has a cost
- Authentication doesn’t offset bandwidth
- Good behavior still eats resources
Even well-behaved bots can DDoS you by accident. You can’t fix that with certificates or signatures.
Emerging ideas from the Web Bot Auth conversation
The Web Bot Auth mailing list had some smart commentary recently on what incentives actually look like:
- Reputation and differentiation: Bot operators don’t want impostors ruining their good name. Self-identification helps create reputational trust.
- Better treatment through transparency: Authenticated bots could be treated as “allowed by default” rather than punished by default, which would flip the current anti-abuse script.
- Load management: Many sites are being overloaded, not just attacked. Infrastructure strain is forcing even friendly sites to take defensive measures. This opens the door to load-based incentives: service operators could offer higher rate limits or more reliable access to bots that self-identify and follow documented behavior guidelines. Rather than treating all automation as abusive, a tiered system could encourage cooperative bots to behave responsibly in exchange for stable access.
All of that leads to an observation: identity is useful, but it doesn’t answer the real question. Who decides if the bot is worth the load it brings? That’s a value judgment that falls outside the scope of identity systems. What it does highlight is that services can’t ignore that automated traffic is hitting their infrastructure, and they need tools, not just blind faith, to manage it.
What could a better system look like?
Imagine a world where bots:
- Register and authenticate using open standards
- Earn a reputation score over time
- Get tiered access based on usage patterns and benefit to the service
- Pay—or pass value back—in proportion to their impact
This isn’t a fantasy. We already do this for humans via OAuth scopes, rate limiting, and usage tiers. The challenge is applying it to non-human actors in a way that scales.
(As an aside here, there are two people I recommend you follow if you’d like to dig into the gory, gory details of NHI taxonomy and the practical realities of NHI: Erik Wahlstöm and Pieter Kasselman.)
What you can do today
If you’re a product manager or DevOps lead, this doesn’t have to wait on a new IETF spec. You can start with:
- Separate metrics for bot vs. human traffic – Understand where your resources are going and whether that automation is helping or hurting. This won’t be perfect. If it were, we wouldn’t need to figure out how to differentiate the traffic in the first place. But you can get a gross approximation to start, using things like user-agent parsing, request behavior patterns, or identity-aware proxies. That can help you make smarter decisions about rate limits, caching strategies, or whether to even allow certain types of traffic at all.
- Bot policy transparency – If you expect bots to authenticate, say so. If you want them to throttle, document it, ideally in machine-readable formats. That could include published API docs, robots.txt extensions, or structured metadata in your OpenAPI spec. You could also express bot policies via HTTP headers, usage dashboards, or identity-aware gateways. Don’t hide the rules in your EULA; bots don’t read fine print, but their developers might parse structured access guidance.
- Selective encouragement – Are there bots that drive value? Give them the green light, but with boundaries. Tools like API gateways (e.g., Kong, AWS API Gateway, Apigee) already support rate limiting and tiered access policies that can help enforce those boundaries. Standards such as OAuth 2.0 and mutual TLS (mTLS) can be used to verify identity and scope access. Emerging efforts like the Web Bot Authentication discussions and SPIFFE/SPIRE for workload identity also offer structured ways to manage and audit bot and automation access without resorting to total denial or blanket approval.
And if you’re building a bot:
- Respect the load you place on services.
- Identify yourself if you want a long-term relationship.
- Assume you’re not entitled to the same treatment as a human user unless you bring similar value.
Final thought
This isn’t about punishing bots or yelling at them to get off your lawn. Automation is here to stay. But if we want to coexist, we have to stop pretending that identification alone is the solution.
Identity without incentives is just surveillance.
Incentives without constraints are just spam.
Let’s aim for something better than either.
📩 If you’d rather track the blog than the podcast, I have an option for you! Subscribe to get a notification when new blog posts go live. No spam, just announcements of new posts. [Subscribe here]
Transcript
Bots, Incentives, and Identity
Hi, welcome back to A Digital Identity Digest. I’m Heather Flanagan, and today we’re talking about bots.
[00:00:36] Speaker A: Not the horror stories, not the buzzwords, but the real, practical tension that comes up when your system starts to feel the weight of automated traffic.
If you’re a product manager, DevOps lead, or identity architect managing automated requests—or even a bot developer or AI agent creator—this episode has insights for you.
Because here’s the challenge:
- Not all bots are bad.
- But all bots cost someone something.
What We Mean When We Say Bot
Bots cover a wide spectrum of activity. They can mean:
- Helpful automation: search engine crawlers, uptime checkers, accessibility tools
- AI agents: fetching, generating, or summarizing content
- Enterprise scripts: internal automation and integrations
- Malicious actors: scrapers, spam bots, credential stuffers
Some are essential, some are annoying, and some are harmful. But regardless of intent, they all place a burden on infrastructure—and that cost usually lands on the target system.
Identity Is Only the First Step
Identity is one of my favorite topics, but identity alone doesn’t solve the bot challenge.
There is growing interest in creating standardized, trustworthy ways for bots to identify themselves. For example:
- Efforts like the Internet Engineering Task Force (IETF) are exploring bot authentication.
- Bots could say, I’m not human, but I’m not here to cause trouble either.
- Some providers welcome this—they want visibility into what’s hitting their sites.
However, many bot developers aren’t eager to adopt these practices because:
- Self-identification doesn’t currently bring them benefits.
- Even when they do identify, site operators still pay the cost in bandwidth, compute, and storage.
So the tension remains: without incentives for both sides, we’re stuck in the cycle of block everything or get overrun.
Why Incentives Matter
Bot developers want access without being mistaken for abusers.
Site operators want reliable service for their human users.
This isn’t pure conflict—it’s misalignment.
- Developers want recognition as ecosystem contributors.
- Operators need to judge whether a bot’s value is worth its load.
Identity management can’t make that judgment. It requires a value framework.
Emerging Ideas for Bot Incentives
Public discussions around bot authentication highlight some promising concepts:
- Legitimate bots want protection from imposters who could damage their reputation.
- Self-identification and adherence to published guidelines could be rewarded.
- Smaller site operators are overwhelmed and often default to blocking—even when bots aren’t malicious.
This points toward load-aware incentives. Imagine:
- Anonymous bots: limited access
- Known bots: more access
- Trusted, valuable bots: highest access
In other words, an API-style approach with access tiers.
Designing a Smarter System
We already use systems like OAuth scopes, rate limits, and usage quotas for humans. Why not for bots?
A smarter system could include:
- Open standards for bot registration and authentication
- Tiered access based on reputation, utility, and resource use
- Machine-readable policies that clearly define what’s allowed
- Monetization or resource-sharing models for high-impact bots
Companies like Cloudflare are already experimenting in this space, and it’s worth tracking their efforts.
What You Can Do Today
You don’t have to wait for global standards. There are steps you can implement right now:
- Measure traffic: Separate bot and human activity with user agent parsing, behavioral analysis, or identity-aware proxies.
- Be transparent: Publish expectations in API specifications, robots.txt, or OpenAPI metadata. Avoid hiding them in user agreements.
- Encourage good bots: Support bots that drive value (like discoverability or user support) through API gateways, OAuth scopes, or mTLS.
By doing this, you’re not punishing automation—you’re designing for it, while keeping your infrastructure sustainable.
Closing Thoughts
At the end of the day:
- Identity without incentives is just surveillance.
- Incentives without constraints are just spam.
The goal is something more useful—balanced systems where automation and infrastructure coexist productively.
Thank you for listening. You can find the full blog post with links and further reading at sphericalcowconsulting.com. Please share this with colleagues, encourage them to subscribe, and stay tuned for next week’s episode.
[00:08:59] Speaker B: And that’s it for this week’s Digital Identity Digest.
[00:09:03] Speaker A: If it made things a little clearer—or at least more interesting—share it with a friend or colleague.
[00:09:04] Speaker B: Connect with me on LinkedIn @hlflanagan. If you enjoy the show, subscribe and leave a rating wherever you listen.
Stay curious, stay engaged, and let’s keep these conversations going.