An online jewellery retailer’s host kept flagging CPU spikes on their WooCommerce store. The usual fixes were already in place: cart fragments disabled, WordPress Heartbeat tuned, robots.txt tightened and AI crawlers blocked at Cloudflare. The spikes kept coming.
The access logs showed the real cause. A large volume of traffic was coming from cloud datacentre IP addresses while pretending to be ordinary Chrome browsers. With no bot signature, robots.txt and named-bot blocks never applied. Much of it was hitting wishlist URLs, and every request made the server do real work. The answer is to filter this traffic at the network edge with Cloudflare’s bot management, before it reaches the server.
I’m seeing this more and more on e-commerce sites this year. Robots.txt only works on bots that choose to obey it, and the aggressive scrapers don’t. If your host keeps warning you about resource usage and you’ve done the standard optimisation, look at who is actually visiting before you pay for a bigger server.