(Reference: On the Dangers of Stochastic Parrots | Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency)

The people/companies behind these models will keep trying to "fix" them by throwing ever-increasing amounts of computing power at them (with all the lovely real-world effects on everyone and everything) and by using ever-more-complex models. And yes, they'll perform better. But they're still just large exercises in statistics and linear algebra, they're still just stochastic parrots, and thus there's an upper bound that they may approach asymptotically -- but can't surpass.

That's not because they're broken -- which is why I put "fix" in quotes in the previous paragraph. It's because that's how they work: it's an intrinsic property of all such models and no amount of computing power and/or model tweaking can change that: all it can do is obfuscate it. And obfuscated problems are far worse than obvious problems.

Twice, in fact -- once in an academic research lab and once at a company that designed and built medical imaging equipment.

In both cases we worked on image classification using digital image processing and statistical pattern recognition. (In one of the two cases we also used syntactic pattern recognition and machine learning.) It's very, very, very hard to make this accurate enough for clinical use even if you pour effort and time and money into it. There's no way this technology should be deployed without humans backing it up.

As to the human mistakes: everyone can cite a case where a professional radiologist committed a false positive or false negative error. But did you stop to consider why they made a mistake? Were they 13 hours into a 14-hour shift, their third one in a row -- because the hospital CEO felt that money should go into his pocket instead of into hiring another radiologist to share the load? Was it an imaging anomaly (they happen) that was ambiguous? Was it because the study that was done wasn't the best choice? (I.e., imaging modality or location) There are all kinds of ways for this to go wrong that will result in blame being assigned to the radiologist, and only some of those assignments are fair.

AI isn't a magic fix for this. And I certainly wouldn't even try to use any of the general-purpose models -- as Zathras would say: "This is wrong tool." If I were to do this again today, I would return to the approach we used before with modest success, I'd take advantage of some of the improved algorithms that have come along, and obviously I'd use bigger/faster hardware, because that opens up approaches that were computationally infeasible. But I wouldn't even consider removing humans: these are, or can be, life-and-death decisions, and a human being needs to make them.

Captchas were thoroughly defeated years -- MANY years -- ago. The only reason that some people mistakenly think they're still working is that some targets aren't worth the time and trouble to attack.

A few of the numerous references that can easily be found to support this:

unCAPTCHA Breaks 450 ReCAPTCHAs in Under 6 Seconds

Bots are better at CAPTCHA than humans, researchers find

AI researchers demonstrate 100% success rate in bypassing online CAPTCHAs

Troy Hunt: Breaking CAPTCHA with automated humans

Stanford researchers outsmart captcha codes

"At some point, the right answer is to buy NICs and compute boards and built your own router like we used to do."

I'm still doing it for a lot of applications. Same for firewalls. The cost is a fraction of commercial offerings, the performance is more than adequate, maintenance is in-house and easy (because I keep a stash of spare parts), and there's no bloat in the software stack because anything I don't need isn't there.

1. This will almost certainly be challenged in court, and that will take a while.

2. Some amount of gear is about to undergo a US-washing in order to evade this: "Yeah, it was designed in China and built in Vietnam, but final assembly was done in Lubbock, soooooo....it's US-made".

3. If the challenge in (1) is unsuccessful, the price of a US-made router will double. That's what happens competition is removed from markets.

4. Also, the US vendors will do their best to kill open-source firmware/software -- say, by introducing undocumented components or issuing firmware updates that break software or by labeling it a national security risk.

5. Everyone trying to cope with the mess will be faced with fewer choices and those choices will cost more...so as various devices hit EOL, folks may decide to keep running them (in spite of the security risks) rather than buy pricey new stuff. Or maybe they'll buy gray market gear.

6. Bottom line: everyone trying to run operations while aiming for the balance of cost and security now has a worse set of choices than they had yesterday.

7. The only thing left for the administration to do is to declare "MISSION ACCOMPLISHED" in huge letters and move on to tampering with the next delicate piece of machinery; perhaps someone who doesn't know the difference between fission and fusion could craft nuclear policy, or someone who doesn't know the difference between bacteria and viruses could run th....oh. Wait. My bad, already happened.

Why not "replace him as CEO"?

Although it could be done much more cheaply just by using a Magic 8-Ball equipped with appropriate choices, e.g., "be creepy as hell", "invest billions in tech nobody wants", "find another way to invade privacy", etc.

At the moment, Reddit (and other sites) are being targeted because the attackers gain something: profit or publicity or political advantage or something else. If those attackers find that age verification mechanisms/services are standing in the way of that gain, they won't just give up and go away. They'll target the age verification process itself.

That targeting could take a number of forms: an obvious one is to hack them and arrange for them to "verify" a selected set of identities. A less obvious one is to bribe or blackmail people inside the verification service; yes, it's low-tech but that's why it's worked for millennia. Another, and this one is on the table because of the amount of money and power in play, is to set up a shell corporation and buy the verification service. There are other approaches as well.

I strongly doubt that any of these verification services have the means to defend themselves from these attacks: they're running on thin margins and mounting an effective defense would be quite expensive. Moreover: why would they? It would be far more profitable to squeeze out as much revenue as possible before the roof caves in, conceal the assets, declare bankruptcy, shut down, reopen under a new name elsewhere, and repeat. (This business model has already been proven to work -- by telemarketers.)

I would add "...and throw some campaign contributions at politicians" but I'm not sure that's necessary. There are already plenty of them grandstanding on the "FOR THE CHILLLLLDRRRRENNNNNN" platform, so I there's little need to purchase any of them.

How unsurprising. What is surprising is that his fanboys haven't yet realized that he's a clueless ignorant newbie with delusions of grandeur -- someone who actually thinks he knows better than the people who've won pretty much every prize in computing, including the Turing Award.

(There's a Wikipedia entry on it, but I recommend Chesterton’s Fence: A Lesson in Thinking.)

The Bloomberg Terminal is a critical piece of financial infrastructure. It has its issues, to be sure, but it's stable, functional, and has been tested under serious duress for a very long time...so it works. This is not some unimportant app or transient service or game; it's actually important in the real world.

Could it be replaced? Sure. But it's not going to replaced by the kind of slop that vibe coding churns out. If it's replaced, it will be replaced by the work product of superb designers, excruciatingly careful developers, and fanatical testers working together for years with professionals who've been in the field for decades.

I've been in this field for close to half a century, and I'm getting increasingly annoyed by the ignorance, illiteracy, and arrogance of young and inexperienced tech bros whose world view is so constricted, so limited, so myopic that it never occurs to them that no, they do not know the answers to everything, and yes, some of us cranky geezers who have actually been there and done that might know a thing or two that has thus far eluded them and maybe, just maybe, they ought to shut up, sit down, pay attention, take notes, and learn -- if they're capable of learning.

"When someone shows you who they are, believe them the first time."

It's entirely possible that they couldn't. Let me explain, as someone who has had to deal with the financial affairs of other family members multiple times.

First, not everyone keeps their financial affairs well-organized. For every person that has a neatly-labeled box of file folders, there's one who has thrown important papers into random boxes along with old newspapers, magazines, and junk mail. LOTS of random boxes. So merely trying to answer the question "What accounts does this person have?" takes a lot of work and a lot of time -- and even if that work's done carefully, it might still result in omissions.

Second, some people keep their financial affairs on a computer/online, which means that someone trying to help has to work through all that -- all the security, all the ID verification, all processes, all the bureaucracy, everything. This can be brutally difficult: it means hours and hours on hold with customer service departments that want to avoid doing anything resembling service. It means endless letter exchanges. It means retaining an attorney and getting documents notarized. It means sending things overnight recipient-signature required and then being told they never arrived. It means filling out forms and submitting them only to find out that they're the wrong forms -- even though they were the ones you were told to submit.

Third, it means dealing with people whose only goal is to get your problem off their desk and out of their queue -- so they'll close help desk tickets without bothering to ask if the problem is actually solved or even bothering to inform you. (When you call back, you'll be told that no such ticket exists. When you insist, they'll tell you that it was resolved and thus closed.) If you somehow manage to get past this, you'll be told things like "our anti-fraud procedures prevent you from paying your grandmother's electric bill" -- yes, really. Or -- and I'm not making this up -- "we need to talk to your mother directly". Oh? My mom? The one whose death certificate I sent you four months ago? The one whose leftover $2.17 cable bill I'm trying to pay so that I can close the account and never have to speak to you again? That one?!

I could rant about this for pages, but the bottom line: trying to step in and handle someone's financial affairs is a full-time job. It requires constant attention, endless phone calls and letters, and a mountain of work.

....I encourage my targets to use AI assistants for everything.

"A plague o' both your houses!"

I can only hope that this lawsuit escalates into...well, I hope it escalates without upper bound. Parasites suing parasites is the kind of quality legal theater that we can all appreciate.

Users: we need an ad-blocker in the browser (a la uBlock Origin)
Mozilla: we're taking away the status bar.
Users: we need control over Javascript (a la NoScript)
Mozilla: we're changing the UI.
Users: we need privacy protection/anti-fingerprinting/anti-tracking (a la Privacy Badger)
Mozilla: we're adding AI
Users: we'd like you to fix the bugs and performance issues
Mozilla: we're redesigning the UI for the N'th time even though the one we had 10,15,20 years ago was pretty much ok
Users: have you noticed that your market share is in the single digits?
Mozilla: give us money!

1. Nobody did any kind of due diligence whatsoever on this guy. NONE. And then they showed the same level of attentiveness to what he was doing: NONE. So while their lawsuit may have some merits, this entire sordid affair could have been prevented if someone, anyone, had exhibited minimal competence.

2. I have no doubt that this is epidemic in the entire AI sector. Of course it is: everyone is too busy hyping their products/services or soliciting venture capitalists or making insane predictions ("AI will solve global warming") to spend any time doing the nuts-and-bolt work of running a solid business operation. This reminds me very much of the dot-com boom 30 years ago, when the exact same thing happened: any hustler who could talk a good game could land a job and a huge paycheck and then bail out before the roof caved in. (Some of them were sued, but most weren't, because the companies that failed either no longer existed or lacked the resources to engage in protracted litigation.)