01 · Operator
Who operates SMTBot
- Company
- SimilarTech
- Type
- Other bot
- Official docs
- similartech.com
02 · Behavior
What SMTBot does
SMTBot fetches pages to identify technologies on websites. SimilarTech uses this data for market research and lead generation. It is selective and fetches about 200 pages per site.
03 · Impact
Why SMTBot matters for your site
If you allow it
- It follows robots.txt
- It fetches only a small number of pages
If you block it
- It brings no visitors
- Your technology data is sold for sales prospecting
04 · Allow
How to allow SMTBot
robots.txt
User-agent: SMTBot
Allow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "SMTBot")
Action: Skip › All Super Bot Fight Mode rules
# Also check Security › Bots: "Block AI bots" can block it regardless of robots.txt.WordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: SMTBot
Allow: /nginx
# nginx serves every user agent by default.
# Make sure no rule like this blocks it:
# if ($http_user_agent ~* "SMTBot") { return 403; }Apache
# Apache serves every user agent by default.
# Make sure .htaccess has no rule like this:
# RewriteCond %{HTTP_USER_AGENT} SMTBot [NC]
# RewriteRule .* - [F,L]05 · Block
How to block SMTBot
SMTBot follows robots.txt, so one rule is enough. Use a server or CDN rule only to stop spoofed copies.
robots.txt
User-agent: SMTBot
Disallow: /Cloudflare
# Security › WAF › Custom rules › Create rule
Expression: (http.user_agent contains "SMTBot")
Action: BlockWordPress
# WordPress serves a virtual robots.txt. Edit it with your SEO plugin:
# Yoast: SEO › Tools › File editor · Rank Math: General Settings › Edit robots.txt
User-agent: SMTBot
Disallow: /nginx
# In your server { } block:
if ($http_user_agent ~* "SMTBot") {
return 403;
}Apache
# .htaccess
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} SMTBot [NC]
RewriteRule .* - [F,L]
</IfModule>06 · User agents
User agents we see for SMTBot
The user agent published by the operator:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)07 · Verification
Is it really SMTBot?
SMTBot’s operator publishes no IP ranges or hostnames, so requests cannot be verified. Treat the user agent as a claim, and watch your logs for unusual request rates.
FAQ
Questions about SMTBot
What is SMTBot?
SMTBot is SimilarTech's crawler. It gathers data to identify the technologies websites use.
Does SMTBot respect robots.txt?
Yes. SimilarTech states that SMTBot follows robots.txt rules for the SMTBot user agent.
How many pages does SMTBot crawl?
SimilarTech says the bot is selective and crawls about 200 pages per site.
Last reviewed Oct 8, 2026