616,000 add-to-cart requests in eight days: the cart trap

It started with a detail. While cleaning up the redirects in the online shop of Bauerntüte, an organic delivery service from Cologne, we noticed an odd pattern in the 404 log: hundreds of URLs ending in /null, such as /produkt-schlagwort/marmelade/null. Together they added up to 17,665 requests. The background is in our case study on 7,740 pages with redirect.

To find out who requests these URLs, we analysed eight days of the server's access logs. The /null URLs turned out to be the smaller problem.

What the access log showed

During these eight days the shop received between 100,000 and 270,000 requests a day. A large share of them carried the parameter add-to-cart:

WooCommerce evaluates ?add-to-cart= on every normal page request, before the page is built. It is meant for visitors without JavaScript. Requesting such a URL puts the product into a cart, and WooCommerce creates a guest session for it in the database. Then the server builds the full page, around 540 KB with 72 products.

Real customers hardly use this route on the shop's listing pages. There they add to cart via AJAX, and add-to-cart never appears in the URL. So the hundreds of thousands came from somewhere else.

The trap: sorting links that carry add-to-cart along

On shop categories the theme Enfold adds a form that lets visitors choose sort order and page size. It builds the links in that form from the current URL. If add-to-cart is already in there, it ends up in every sorting link as well:

<a href='?paged=1&avia_extended_shop_select=yes&product_order=default
    &product_count=24&add-to-cart=57851'>Standard</a>

For a crawler this is a trap. It finds an add-to-cart link, requests it, finds ten sorting links with the same parameter on the result page, requests each of them, and so on. Multiply the sort orders, the page sizes and the product IDs, and you get millions of URLs. Each of them puts something into a cart again.

The shop's robots.txt had long disallowed add-to-cart and the sorting parameters. For the two crawlers we found, that made no difference.

Who was caught in it

Every line of the log contains the user agent the client sends. For the add-to-cart requests of those eight days the picture was clear:

• 396,599 requests from a disguised scraper. It always sends the same user agent of a Chrome browser on a Mac. Within three hours it came from 3,625 different IP addresses, almost each of them only once. No human behaves like that, and it cannot be blocked by a single address either.
• 219,159 requests from the crawler meta-externalagent. Meta uses it to collect content for its AI models. On some days it accounted for more than half of all page requests to the shop, 53 percent on 27 October.
• Around 500 requests from everyone else combined. A few real browsers among them.

And the URLs ending in /null?

In our sample they came exclusively from Meta's crawler, always right after a sorting URL with add-to-cart. Nowhere in the delivered HTML is there a link to “null”; we checked with a normal user agent and with the crawler's. A normal browser does not request the URL either. It probably appears when Meta's environment runs a script on the page and uses an empty value as a URL. We did not find which script it is. For the fix we did not need to know.

The fix

We closed the trap where it is created, without taking anything away from customers. The rule sits in a must-use plugin and runs before WooCommerce touches the cart:

// URLs ending in /null: immediate 410, before the theme builds a 404 page
if ( preg_match( '#/null/?$#', (string) strtok( $_SERVER['REQUEST_URI'] ?? '', '?' ) ) ) {
	status_header( 410 );
	header( 'Cache-Control: public, max-age=86400' );
	exit;
}

add_action( 'wp_loaded', function () {
	if ( empty( $_GET['add-to-cart'] ) || wp_doing_ajax() || is_admin() ) {
		return;
	}
	$sorting = isset( $_GET['avia_extended_shop_select'] ) || isset( $_GET['product_order'] )
		|| isset( $_GET['product_count'] ) || isset( $_GET['product_sort'] );
	$bot = (bool) preg_match( '/bot|crawl|spider|externalagent|facebookexternalhit|headless/i',
		$_SERVER['HTTP_USER_AGENT'] ?? '' );
	if ( $sorting || $bot ) {
		wp_safe_redirect( remove_query_arg( 'add-to-cart' ), 301 );
		exit;
	}
}, 0 ); // before WooCommerce (wp_loaded, priority 20)

What the rule does:

• Sorting plus add-to-cart is redirected to the same URL without add-to-cart, without touching the cart. This removes the factor “number of products” from the equation, and the number of possible URLs shrinks by exactly that factor.
• add-to-cart from recognisable bots no longer puts anything into a cart either.
• URLs ending in /null get a 410 before the theme is loaded. It tells the crawler that the URL does not exist and will not exist, and it costs the server almost nothing.

What the rule deliberately leaves alone: the AJAX cart, product pages and simple add-to-cart links without sorting, such as those in newsletters. We checked all three cases live right after deploying: /null answers with 410, sorting with add-to-cart with a redirect that leaves the cart alone, normal pages with 200.

What the fix does not solve

The disguised scraper keeps requesting sorting URLs, now without the cart. The server still builds each of them in full. Against a crawler that poses as a browser and uses a new IP address for almost every request, no rule in a plugin helps reliably. That needs rate limiting or bot protection in front of the website, at the host or in an upstream service. That is the next step.

We will add how the numbers develop after the fix as soon as we have a few days of logs.

What you can take away for your own shop

• Count add-to-cart in your access log. A few dozen requests a day are normal. If they run into the thousands, there is almost always a crawler behind it.
• Check whether your sorting and filter links carry parameters along. Open a category with ?add-to-cart= and see whether the parameter shows up again in the links of the result page.
• Use the AJAX cart on listing pages. Then no customer needs the GET URL, and you can restrict it tightly for bots.
• Do not rely on robots.txt. It is a request. Whatever must not happen has to be prevented by the server.
• Read your 404 log. Odd patterns like /null are often the first visible sign of a crawler that has lost its way.

How to make a slow WordPress site fast again overall is covered in our guide to speeding up a slow WordPress site.

Frequently asked questions