> The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge.

That's a CAN-SPAM act violation.

FTC: "Tell recipients how to opt out of receiving future marketing email from you. Your message must include a clear and conspicuous explanation of how the recipient can opt out of getting marketing email from you in the future. Craft the notice in a way that’s easy for an ordinary person to recognize, read, and understand. Creative use of type size, color, and location can improve clarity. Give a return email address or another easy Internet-based way to allow people to communicate their choice to you. You may create a menu to allow a recipient to opt out of certain types of messages, but you must include the option to stop all marketing messages from you. Make sure your spam filter doesn’t block these opt-out requests."[1]

Experian was recently fined for making it hard to opt out of their marketing emails.

The actual regulation text:

§ 316.5 Prohibition on charging a fee or imposing other requirements on recipients who wish to opt out.

Neither a sender nor any person acting on behalf of a sender may require that any recipient pay any fee, provide any information other than the recipient's electronic mail address and opt-out preferences, or take any other steps except sending a reply electronic mail message or visiting a single Internet Web page, in order to:

(a) Use a return electronic mail address or other Internet-based mechanism, required by 15 U.S.C. 7704(a)(3), to submit a request not to receive future commercial electronic mail messages from a sender; or

(b) Have such a request honored as required by 15 U.S.C. 7704(a)(3)(B) and (a)(4).

That seems to cover it. File a CAN-SPAM act complaint (spam@uce.gov). Send a copy to the legal department of the sender.

[1] https://www.ftc.gov/business-guidance/resources/can-spam-act...

You're collateral damage in the web's war against bots :(

Unfortunately, I think the Cloudflare challenges are designed to filter out users similar to your profile... once you stray far enough from the norm, it just looks like a bot / suspicious traffic to them. Statistically there's not enough users like you (privacy-conscious Linux users on nonstandard browsers) for them to really care enough to do anything about it. Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you... it's sad, but I don't think there is really anything you can do about it except conform. It's an ongoing arms race and you're caught in the middle.

The problem is that any solution so far proposed for this is very privacy-unfriendly.

For example, Google proposed https://github.com/explainers-by-googlers/Web-Environment-In... and this was shot down by privacy advocates (for very good reasons).

So basically the choice for website operators is either to fight the bots and accept that their service will be unusable for some subset of their users or not fight the bots, which will lead to their service becoming unusable for everyone.

More and more, you see services pushing you very hard towards using their app and the reason is that with the app, they are able to actually verify that you are likely not a bot (or rather, in reality, that at least the app is running on an actual physical device, mobile phone bot farms are unfortunately also a thing).

As for Cloudflare - they offer it as a service, so when the website operator has a choice between using them or allocating several engineers for bot-fighting, why would they not just go with Cloudflare? Doing it yourself can be slightly higher fidelity, as you know your customers better, but it is also a lot of effort which could be better spent elsewhere.

I deal with this fairly commonly, presumably because I use linux, and we all know only botnets use linux. Occasionally with cloudflare I'll just get summary rejection and supposed blocking of my IP, but either it's summary rejection or a pass without challenge.

Recently I had to deal with this for alibaba just to look at something, which I usually just use torbrowser with, and finally gave up as I couldn't pass the challenge. I suppose I shouldn't be surprised at that though, they trust me as much as I trust them.

The worst is usually adobe and cookielaw with all their related tracking crap, where I can't even get the captcha to render as it's so many layers buried in scripting I can't enable enough sites between ublock, noscript, privacy badger, and firefox strict modes. I treat adobe like malware, but unfortunately things like albertsons.com for groceries and other mega companies love to use it, and their sites literally do not work without allowing their heavy scripting/tracking.

There are other usually smaller captcha players that I haven't been human enough to pass with, I forget the names of the stupid to shame, but a few when I see them I recognize to just close the window and forget about whatever it was I was looking for there (like twitter/x).

Hooray commerce!

Yes. I wrote about this on my blog six months ago [1].

CloudFlare has positioned itself as the doorman of the Internet, deciding who gets to visit shitty websites written by AIs and who doesn't. Every time I try to visit a website and get blocked by this company and its unnecessary services, I congratulate myself for avoiding yet another terrible website and move on with my life.

[1] https://ido50.net/content/what-chafes-my-groin-9.html

It's ironic but I was having terrible problems accessing archive.today when I was using Cloudflare DNS (1.1.1.1) that cleared up when I switched to either my ISP's provider or Google's 8.8.8.8. I was not the only one

https://news.ycombinator.com/item?id=38063548

What's funny about it is that as a human I get tormented by those things all the time but I have been writing bots since 1999 and have yet to have had CAPTCHAs affect a webcrawling project in a big way: for instance I have a bot that collected 800,000 images from 4 web sites since last April, at times I thought they had anti-bot countermeasures but I realized that when they were having problems it was because the wheels were coming off their web site (don't blame me, that is 0.03 requests/second and are not parallelized and pipelined like the requests from a web browser.) I'm also prototyping one that can look at an article like

https://phys.org/news/2025-01-diversifying-dna-origami-gener...

see if there are links to journal articles in there, determine if the articles are Open Access and pick out an image for social... so far no problems. But if I want to pay my electric bill there's a CAPTCHA -- I mean, what kind of bot wants to pay my electric bill? (Kinda seems like it is asking for a lawsuit in this day and age if it prevents anyone 'differently abled' from accessing essential services...)

Cloudflare is so embedded into so many important services (like some other companies, including Google), that they need to be thinking of their role as having some government-like responsibilities.

For example, for starters, Cloudflare and Google need to find ways so that individual people who're wrongly being locked out of services by the company, have some way to get that unlocked. Not "sux2bu we dont do support bro".

(Then they can start thinking about the next step, which is due process, and what it means to wrongly lock out someone in the first place.)

That said, as an immediate pragmatic matter, one debugging tip with your Firefox is to go to the `about:profiles` URL, and temporarily create a new profile, and without using any Firefox sync feature, and see if Cloudflare lets you through, and then incrementally add back in your extensions and preference customizations, and see if/when CF stops letting you in. (Not that it will necessarily identify the sole and exact trigger, since they might be using scores of multiple factors, but it will be evidence of one thing that pushes it over the edge. And maybe get you to a compromise setup that lets you do your work for now.) Also helpful is to have alternate browsers installed; personally, I keep Chromium installed, as my "violate me every possible way, if you'll just let me access this one page/site I really need right now".

I had similar issues as an (also heavily customized) Firefox user, but was able to fix it by installing Cloudflare's Privacy Pass browser extension.

It seems ironic that as a human I can't seem to reliably prove I am a human with a realistic amount of effort via these systems, but having installed a specific automated browser extension does?

I am not a fan of Cloudflare and don't like the idea of running their software on my computer, but it seemed like the only options to continue using the internet at all.

I can't use any of the kerbalspaceprogram.com domains because of improper discrimination against IPv6 clients triggered by CloudFlare.

    Error 1015 Ray ID: .... • xxxx-xx-xx xx:xx:xx UTC
    You are being rate limited
    What happened?
    The owner of this website (wiki.kerbalspaceprogram.com) has banned you temporarily from accessing this website.

Exact same situation here. Linux, fairly funky firefox setup, eventually couldn't use half of the internet without hitting CF prompts, often wasn't able to get around them.

I wound up removing / reinstalling firefox...same exact setup otherwise. No more cloudflare (or vastly fewer) prompts. The internet is usable again.

Hope that helps.

People are focusing on your very non-standard setup, but I've experienced this - less than you to be sure - on a standard MacOS setup with Firefox and only uBlock Origin installed. If I switched to Chrome without uBlock Origin it worked. This was on the English National Ballet's ticketing website.

> - The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge.

Maybe indeed could be held liable here? From the can spam act (if you're from the US):

> You can’t charge a fee, require the recipient to give you any personally identifying information beyond an email address, or make the recipient take any step other than sending a reply email or visiting a single page on an Internet website as a condition for honoring an opt-out request.

https://www.ftc.gov/business-guidance/resources/can-spam-act...

I do NOT like it at all but I just want to show a way how it works with Cloudflare and to make it painless with them. Basically fully assimilating to them because Resistance is Futile ;)

1) Privacy Pass Extension

Install Privacy Pass Client Extension in your browser, here for Chrome https://chromewebstore.google.com/detail/silk-privacy-pass-c...

2) Use Cloudflare Warp (which is a VPN by Cloudflare basically, it's free):

https://one.one.one.one/

I'm really afraid of what kind of internet we'll have when these kinds of un-diagnosable un-appealable false-positives are not just transient blips, but become metadata companies use to blindly and permanently kill off accounts on other services.

I think it may have been what happened my since-2010 Reddit account was mysteriously killed a couple years ago, and literally the only cause I can think of is that I might've used the wrong public wifi for an evening.

I absolutely hate cloudflare for the same reason you have. Besides traveling and using a VPN, I like in Hong Kong, a country that many sites have decided to block completely. It's very frustrating that cloudflare easily enables those kind of blanket bans for no reasons.

Cloudflare is the enemy of open web.

Cloudflare works much, much better than Google - Google captchas for me, on Tor, are flatly impossible, always. They never let get through, no matter whether you get them right or wrong. You always get "try again".

The problem I do have with CF is their captchas seem to require human interaction on the page, and this makes getting through them problematic when you open half a dozen tabs, and each loads a CF captcha, and you have to move the mouse around for ten seconds just to get the captcha to load, and loading is not reliable. Often you need to reload the page. It's this type of performance, and poor performance, which is breaking web-pages for me.

Unfortunately, your setup makes you look like a scraper: no history for Cloudflare to identify, the sort of browser / OS config someone would use to homebrew an automated "I sure am not a bot, look at how authentic my user-agent is!" bot, and so on. If you also have JavaScript disabled and clear your cookies frequently, Cloudflare can't fingerprint your machine to know you passed a trust-check in the past.

Maybe keeping a heavily-sandboxed Chrome in a VM for situations where Cloudflare is getting in your way might help?

(In the large: this has been an issue a long time coming. Quite a bit of cyberpunk predicts the future where the web bifurcates into the "regular" web that is sanitized, corporate, controlled, and used by most people... And the "everyone else" web that is not, with all the pros and cons that entails. The tech has evolved to the point that companies that want a service provider "keeping the bad guys away" for them can pay to have that done, at the cost of false-positives... But at their scale, the false-positives may not matter to them).

My workaround for this as a person who travels a lot was to buy 2 raspberry Pi’s and put them at my family houses in different countries and use Tailscale on them as exit nodes, behaving like my own VPN. The residencial IP address makes things a lot easier when connecting from random places.

I appreciate you bringing up this issue about the Cloudflare challenges making it hard to browse. I had a similar experience where I couldn't access jsfiddle even without using a VPN. As a result, I switched to a different platform for my coding experiments.

JsFiddle used to be my favorite for quickly testing out code snippets. It's a shame that due to Cloudflare hurdles, I've stopped using it and don't plan on going back.

It may not be much but as more websites and businesses lose genuine web traffic like this, Cloudflare might eventually listen and fix this mess.

I have experience bypassing these.

The primary cause of this is most likely any kind of 'optimizations' you have in your browser (or missing fingerprints).

If you want to 'bypass' these I recommend removing any use of Proxy[1] (via extensions). You should also look into disabling any kind of forced backgrounding. Make sure service workers are working.

1: They catch Proxy usage by using exceptions and analyzing the stacktrace. I assume you know what a javascript proxy is, but incase you don't: It's something that allows you to override any kind of object function such as navigator.hardwareConcurrecy.

I'm experiencing the same issue which is definitely exacerbated by straying from a 'default' configuration e.g. using a custom browser screen reader, browsing from Brazil, using a VPN, using Firefox. I think eventually I'll be completely locked out of the 'mainstream' web

Yes, I run into it from time to time. I just move on. If someone is going to make their website inaccessible to me, I'm not going to bend over backwards to try to work around that.

Incidentally, since I configured DNS over HTTPS in Firefox, using Cloudflare's DNS, it seems I see this much less often.

AWS WAF is even worse. I recently moved from Australia to India, and quite a few high-profile websites are now completely inaccessible to me because WAF seems to be legitimately broken. Two such sites: https://officeworks.com.au/ and https://centrecom.com.au/. You successfully complete their annoying thingummy, and it redirects you… to the same Human Verification CAPTCHA. This has been the case for at least half a year, so it’s not a recent breakage.

If I tunnel via my VPS which is still in Australia, then I can access it.

But complete blocks via Cloudflare have also been a problem: I had to do something with VicRoads as part of selling my car, and was blocked outright when I got to the actual form page. Had I not had my VPS in Australia, I don’t know what I would have done.

My IP address is massively shared (CGNAT) with plenty of botnet around, so I’m frequently troubled by Cloudflare, but not often outright blocked, and if challenged rather than blocked, I’ve never had any problem with it. Linux, Firefox.

I've had to give up obfuscating my user agent because Cloudflare becomes nearly impassable as a result, and Cloudflare seems to own most web traffic now.

If you can't pass the captcha you have to ask yourself, are you really a human being or have you just been programmed to believe that you are?

i recommend everyone test the web with TOR to see how dead the public internet is. Reddit won't respond. Many sites have a 10-minute captcha challenge (e.g. substack).

So many sites have deployed countermeasures like Cloudflare, but they aren't actively monitoring the failure mode on those countermeasures.

The web is on it's knees and these countermeasures are another nail in the coffin if we don't act fast.

>I use a heavily customized Firefox config on Linux.

This is probably the cause, especially if you're doing stuff like spoofing user agent. It's not cloudflare "cracking down on privacy" or whatever either. Unmodified tor browser passes turnstile challenges just fine.

It used to be just for profit companies web dev's ignorantly putting themselves behind default cloudflare deploys and blocking everyone. But now big academic players like science.org/aaas elsevier and other publishers and individual journals are and I can't even read scientific papers anymore. Even sillier is the RSS/Atom feeds science.org ran have the same cloudflare rules so all actual feed readers were blocked (support told me only real feed readers as a service like Feedly corporation were allowed). It took me months of email back and forth to get them to realize their error and get to someone who could fix it. And that is what I consider a good response. Most just ignore the email.

Everyone reading this should start to contact websites/companies who use cloudflare and tell them in simple and few words that it's a problem and link them to a video or article that explains more, maybe even to this HN topic. We are not many, maybe 1-2% of their users/customers I keep reading people saying but I have in the past been able to get big tech companies to change to a friendlier tech. You would be surprised how effective it is to contact them about it. Maybe they have a tech support who already has same opinion as you but they can't make any change until a customer makes a complaint about it, then they happily see it as their opporunity to finally make a change.

> Cloudflare challenges have made large portions of the web unusable for me.

I guess the best web experience is when one filters Cloudfare, Google and Microsoft at the firewall.

Cloudflare puts challenges on their abuse contact page and rate limits it to much slower than human speed. It's also still broken after years in that you can't report abusers who register domains through Cloudflare and/or host their DNS using Cloudflare.

They really don't want feedback from people who don't pay them.

Amen. Another fun one is logging into bank and government sites while roaming... with sms delivered intermittently and with a 5 minute delay.

If it is triggered by the customizations you did in Firefox, then running a fresh Firefox in a container might help:

    docker run -it --rm -e DISPLAY --net=host -v $XAUTHORITY:/root/.Xauthority -v /tmp/.X11-unix:/tmp/.X11-unix debian:12-slim

    apt update
    apt install firefox-esr
    firefox

I get this all the time with firefox on linux + ublock origin extension. Often ending up with that blocked ip page.

I mostly shrug off and just avoid visiting that kind of sites again. For an unsubscribe challenge I just copy paste the url and visit it using firefox focus on my smartphone on my mobile connection.

Only the expensive bots with residential IP's and mechanical turks can survive, humans be damned.

I wish we could popularize some extension that pays a penny per page load or something using some shitcoin both as a means to support our favorite sites but also to validate that I'm not a bot, or at least if I am, I am willing to spend a lot of money in a DDOS that goes directly in your pocket

I'd expect this to increase with the proliferation of AI Crawlers and scraping becoming easier with AI.

Cloudflare challenges seem to be becoming more and more frequent on my general internet use. Yep, "Cloudflare loop" is a thing. No, I'm not going to download and install a different web browser, dump all my cookies, or whatever other nonsensical "solution" they recommend.

I've become to hate Cloudflare with a seething passion.

CrimeFlare is not interested in these problems for the users. If you have access to the hosting side, you can adjust the bot score for specific connections/clients. But consumers don't matter to CF so apart from jumping through their hoops, there's nothing better you can do.

Unless you accept the racket of course, start paying them and proxy your traffic through the CF workers https://github.com/pellaeon/cloudflare-worker-proxy and magically most barriers will disappear.

Can't you have a normal firefox profile for such cases? Do you have any javascript filters? I bet the issue must be related to configs messing with the JS runtime.

I ran into this, or something similar recently when our main connection went down (solar powered) and we switched to Starlink. Due to Starlink NAT issues I had tunneled our traffic to to a box colocated in a data center. This broke a number of web sites in weird ways. Became so annoying that I ended up bringing up a tunnel to our office in town to get back to the regular IP we used. Weird problems went away.

Vaguely related, youtube is lately doing a lot of unnecessary forced logouts & reconfirm password. I'm literally on a static IP. On the same computer & browser. With the same cookies. Not accessing anything particularly sensitive. There is no way in hell they don't know precisely who I am & that its me.

Slightly off topic, but Microsoft ones are even worse - when I tried to sign up to OpenAI/get a new Microsoft account, the captcha were so difficult that it took me 5 minutes to solve (unsuccessfully). As a libre wolf user with very strict settings, I think privacy-aware users bear the externalities of this bot vs server arms race.

I spent a few days agonizing over this same problem, and the culprit turned out to by my user-agent modifier extension.

What I don't understand is why you have to protect areas that require login so harshly?

If I can log in, especially with 2-factor, you can safely assume I am not a bot, or you have a larger problem.

If I have entered bad credentials 5+ times, okay, you can start backing me off or challenging me.

What am I missing? Fail2ban has been around a long time.

I use Whonix quite a lot, Most of the internet is unusable since i get into the "check the box" loop.

The challenges are configurable by cloudflare’s customers. The challenge can either be from turnstile, which is a captcha replacement service that websites use on their own pages, or a cloudflare CDN security setting that will block access to pages until a challenge is passed. It is not clear which one the original poster means.

Cloudflare’s customers can largely disable these and rely on other means of detecting bots.

In the case of turnstile, it has three modes, two of which are entirely automatic and work by interrogating the web browser, with the other requiring a client:

https://developers.cloudflare.com/turnstile/concepts/widget/

Cloudflare CDN’s security setting on its free tier also has an essentially off setting that will basically eliminate challenges when browsers accessing pages protected by cloudflare unless there are exceptional circumstances. I believe it can be fully turned off for the enterprise tier.

Whenever I configure cloudflare for a website, I always turn off challenges since they are annoying to users. There is an interesting write up about how cloudflare’s bot detection works here:

https://blog.capmonster.cloud/en/blog/web-scraping1/how-clou...

Note that I have yet to use turnstile, so I am speaking from documentation I read rather than from actual experience with it. I have used cloudflare’s CDN and I am speaking from experience with it.

Anyway, the website author is the one that should be blamed here.

you're not welcome. is their message.

not a single mention of advertising on all these comments.

those captcha are not against bots. bots are only one item in the broader category they block. you, an unmonetizable user, is another.

cloudflare et all have the "marketplace conundrum". they need to provide value to both sides, and for the site they do this by blocking hard to monetize traffic. that means traffic that won't generate high yield on ad networks those sites care about.

Just tried to disable User-Agent in Firefox by setting general.useragent.override to an empty string and Cloudflare captcha become impenetrable. Cloudflare actively blocks attempts to improve privacy :(

Try creating a cloudflare.com account and stay logged into it. I.e. every few days go into the cloudflare dashboard.

Don't know if it will help but they use lots of methods to see if you are hostile, and being logged in and authenticated with them can't harm

Mess? I got a 8 try guess Ilol to try in 5 tries... in an indistinguishable font. Ooo... Im gonna fail that one...

I am good at this stuff, and "Cloudflare challenges have made large portions of the web unusable for me" too.

Same here, but Cloudflare's captchas in particular are actually the easiest to pass in my experience. Google's ones are the killers. But yeah everything has a captcha if you're using a VPN or Firefox.

Try some popular user agent strings first before concluding that something else like TLS fingerprinting is the problem. Sometimes an acceptable UA string is all it takes.

My local TV station's website refuses to allow my to view their page and instead presents an a modal that cannot be blocked accusing me of using an ad blocker. The funny thing is that only happens on a mobile device using the default browser with no extensions. When I visit the same site on my laptop with uBO, the site is viewable with no blocking modals.

Sometimes you miss what you were aiming for I guess

Just in time: https://doom-captcha.vercel.app

I would recommend using FlareSolverr as a proxy in your browser to bypass the clownflare's captcha

Set up your own vpn on AWS ec2. It will bypass the vpn blocks they have. Problem solved.

It seems that if you use Firefox with an adblocker then cloudflare spam is all you see. Though I have experienced this in plain Firefox too.

Cloudflare are a scummy company trying to force you to use one browser and view all ads.

well, looks like a business opportunity - a service using AI to automatically pass the challenges like this so the people like the original poster could, for a small service subscription fee, use the Internet hassle-free again.

From the other perspective, I use Cloudflare for DNS and HTTPS certificates. Having an alternative that would cover these two use cases without the need for manually running letsencrypt would be enough for me to switch.

I don't want to think about HTTPS, my websites are low risk, mostly static pages (and there are tens of them).

I've honestly only experienced the opposite; their captcha is reasonably easy to bypass, and I've successfully automated access to a few sites "protected" by the Cloudflare captcha (behind a VPN, no less).

> I use a heavily customized Firefox config on Linux.

If you really care about privacy, you should blend in to look like everyone else. Avoiding being tracked raises alarm bells. You have to let them track something; but no one ever said it had to be you.

> I use a heavily customized Firefox config on Linux.

I also use a (not-so-heavily) customized Firefox config on Linux. I also see repeated abuse of my network activity by Cloudflare.

CF issue, or site programming issue.?

yes yes yes yes yes yes yes. I nearly wrote a borderline hit piece of cloudflare challenges because of this bullshit, but instead I gave into their games and repealed my privacy (only for niche cases mostly), likewise there's no solution for me either and it's just, like you said, some other variant of "too bad, so sad".

You’re using a dirty IP and not using Apple Safari, which has solved this via Private Access Tokens. Move out of the sticks.

Now every couple of minutes when scrolling through Reddit, red "Network issue" tab appears. Some comments don't load at all, some are labeled "deleted" even though they aren't. Refreshing the page usually does the trick, but I hate this new experience.

I guess they're just protecting themselves from bots, and I look like a bot in their eyes.

I found a GitHub captcha to be unsolvable. That captcha properly stressed me out.

Stop acting shady.

Cloudflare's —and most similar services'— stance here comes from these VPN funnelling not just people like you, but also attackers. It's untrustworthy traffic from their perspective.

Use a VPN but use a normal network. VPN back to your home, your office. Your traffic will probably take a throughput and latency hit but it looks like real residential traffic, and that's a lot less sus.

What do you mean impassable challenge...? Why isn't it passable? Are you a robot?