It’s not about relying on it, it’s about changing the behaviour of web crawlers that respect 'em, which, as someone who has adminned a couple scarily popular sites over the years, is a surprisingly high percentage of them.

If someone wants to get around it, they obviously can, but this is true of basically all protective measures ever. Doesn’t make them pointless.