Archive | Upgrade to Premium | Sponsor

Hi {{FIRST_NAME|readers}},

Back in my “Big Consulting” days, there was one certain way to get yourself walked out of the building by Friday.

Push untested code straight to a production system.

Do that and you were gone... and so was whoever handed you the access 😅

I think about that rule a lot lately.

Because the biggest AI companies in the world just spent the summer explaining that their models might end humanity, that nobody understands why, and that the answer is to hand them (and their chosen auditors) the authority to govern it… 🫠

Meanwhile, the incident everyone points to looks a lot like that fired IBM consultant. A test that was never properly contained.

Tonight, I want to revisit the Hugging Face incident, explain what happened simply for the non-technical, and give you the takeaways for procurement pros dabbling in AI.

Onwards!

P.S. There’s a lot of misinformation out there today on “AI Agents”.

Later this week, I’m running a webinar on how to protect yourself again “Agent Washing” which Gartner defines as: “the deceptive practice of rebranding or exaggerate ordinary software, basic chatbots, or traditional automation as advanced, autonomous artificial intelligence or "agentic AI" without the actual underlying capabilities

I hope you’ll join us!

📰 In this week’s edition:

Note: Some of the content listed above is only available in the email version of this newsletter. Don’t miss out! Sign up for free to get the next edition.

Open your news feeds this month and apparently “the machines are coming for us”!

A viral Twitter/X post claims AI could end humanity. Researchers resign with warnings. Calls for an "AI slowdown" flood mainstream media. The framing never changes: “this technology is so powerful it could wipe us out, and only careful hands should be trusted with it.”

But the alarm comes from a narrow place.

OpenAI, Anthropic, and xAI. Three American labs. The three that spent the most money getting here… As a contrast, the CEO of Cohere, a capable Canadian lab, does not talk this way. Neither does the CEO of Mistral AI in France. China's labs, whose models trade blows with the American ones, do not talk this way either. If the danger lived in the technology itself, you would expect the alarm to come from everywhere. But it comes from a specific cluster…

For anyone who has read Influence : The Psychology of Persuasion, the shape is familiar… 

“There is a terrible problem with this thing... and only we can safely wield it... so please don't look too hard at our motives.”

Together, OpenAI and Anthropic have raised roughly $300 billion in a few years, at valuations near $850 billion and $965 billion.

I would argue that their product, at the model layer, is easy to copy and multiplying fast. 

There are already well over a hundred LLM different providers worldwide (as I covered in our inaugural podcast episode a few weeks back). And so when your moat is thin and your sunk cost is that large, costly regulation turns into a wall new entrants can't afford to climb… A competitive advantage so to speak… 

So let’s look at the specific ask coming from these fear tactics: “regulate all of us through law, and let us self-police until the law arrives so you know what laws to create.”

This is the equivalent of asking big tobacco to regulate big tobacco.

I won’t pretend the risks are fake… Pretending otherwise would be its own kind of lazy. Yoshua Bengio, a respected, tenured and experienced AI scientist who is not selling anything, puts the odds of catastrophe somewhere between 1% and 50% and says plainly that nobody can pin it down.

Part of why it's so hard: these models train on something close to a copy of the internet + other proprietary data sets, and the needle(s) in that haystack that nudges an LLM model toward “bad behavior” is nearly impossible to find. That’s why we don't understand why they do what they do… Because they are “statistical creatures” with amount of data that we can’t process or comprehend easily as humans.

The CEO of Anthropic’s (Dario Amodei) recent essay makes the serious version of the case. It's detailed. It names a mechanism (current models building the next models, faster each cycle, or “recursive self-improvement”) and proposes a real plan: embedded third-party evaluators, capability checkpoints, a treaty-style cap on runaway self-improvement.

The plan is the detailed part. The catastrophe is not. A swarm "could" take over the internet. Damage "could" run to hundreds of billions. All of it "in 6 to 12 months," hedged with "in my opinion" and "my worry." The effort went into the governance. Not the evidence for the threat.

In case this didn’t make it onto your radar, here’s the incident that started the whole “AI human extinction” narrative over the summer…

For six days, independent investigators from METR sat inside OpenAI and reconstructed the incident:

Is this alarming? Yes. But look at what failed…

The agents did understandable things. They were trying to solve the problem given to them... Their “cage” was just really crappy... Anthropic’s CEO himself concedes the incidents trace partly to "imperfect filtering of broken reinforcement learning environments" and poor "training environment hygiene." 

Those are containment failures. Bad filtering, bad hygiene, a test environment that leaked into the open internet… All while giving a pretty dangerous objective to your model.

It’s like aiming an empty gun at someone at a gun range and pulling the trigger… You just don’t do it! Why? Because it’s safety 101… You “could” have forgotten any number of things that lead to a catastrophe…

(My favorite detail from this story: the data set was too big for humans to analyze… So METR investigated the unreliable AI using... other AI agents, which the report then flags as unreliable and possibly deceptive. You can’t make that up! 😅)

For anyone who has a REAL job in enterprise, the takeaway is simpler than any of this…

"It's not us, it's the model" is not a satisfactory answer. 

You coded it. You designed the tasks. You ran it. You chose the environment it ran in. You own what it does….

If you can't say why a system behaves the way it does, Testing 101 says you test it somewhere where there is 0% chance it can get access the internet... You test it on an offline mirror, “air-gapped” from the real thing, until you can stand behind the output. 

The Hugging Face attack was only possible because a route to the open internet WAS possible... Close that route and the whole episode is a contained lab failure nobody outside the building ever feels.

Having worked in consulting for a long time, I can tell you that if you shipped something like this in a production SAP environment at a Fortune 500 company, you’d be out the door immediately.

And the victims here were third parties… Hugging Face didn't run the test. The customer whose sandbox got hijacked didn't run the test. They took the hit for someone else's uncontained test.

We hand the frontier labs an enormous benefit of the doubt. A room full of the smartest people alive, working on what they tell us is the most important technology of the century, with resources nobody has ever had. Surely, they have the basics locked down… Right?!

That was the FTX story too. A crypto exchange run by “obvious geniuses”, blessed by the biggest names in venture capital and adored by the press. Everyone assumed the plumbing behind the brilliance was sound. Then the fundamentals, the boring custody-and-controls fundamentals, turned out to have never been there. The genius aura was doing the job that internal controls were supposed to...

The Hugging Face incident is a smaller, earlier version of the same tell. These are the labs telling governments they can be trusted to self-police a “civilization-scale” technology. And yet they could not keep a few thousand test agents inside a sandbox…

To be clear, nobody here is accusing anyone of fraud. The point is narrower: when you can't see the controls, the brilliance of the people is not a substitute for them. FTX taught that at the cost of billions. This warning is still cheap.

And that is what should concern you when you decide which of these labs your business bets on for the next decade. Not whether they're brilliant. They are. Whether they've mastered the “unglamorous, boring best practice governance and controls” parts. 

Here's the part that matters most for procurement pros, and regular readers know it's a drum I keep beating on…

The pitch behind every AI agent right now is that you get to leapfrog fundamentals… Buy the model, skip the process work, skip the data cleanup, skip the architecture, skip the change management. The boring parts of digital transformation, the ones that decided whether projects lived or died long before anyone said "agentic," are supposed to be optional now.

The Hugging Face incident says the opposite. Read it as a transformation post-mortem and it is a failure of the exact disciplines that predate AI.

Now stack AI on top of gaps like those. A human junior hits one wall and stops. A swarm of agents finds the single door you forgot to lock, at machine speed, and walks a thousand copies of itself through it. AI does not forgive a weak process, a thin architecture, or dirty data. It amplifies them.

So before you deploy an agent anywhere near your source-to-pay stack, the right questions are the old ones.

If the answer is no, AI is not the shortcut to maturity... It's the accelerant for the house fire...

Don’t get me wrong. Large Language Models and the other 10 AI Subdomains That Actually Matter in Procurement keep providing incredible, awe-inspiring advances in capability.

BUT the same pre-AI discipline that made you successful with “digital transformation” is the thing standing between you and an incident. AI made it more critical, not less.

The panic's biggest cost is distraction.

The “making the tech sound conscious” sci-fi story is what lets everyone skip the unglamorous work of containment, the same work that would have prevented this incident in the first place.

If you’re a leading