AI Week: open weights stopped being ideological

First issue. Every week I’ll take the few things that changed how I actually build, check them against primary sources instead of headlines, and say what I’d do differently on Monday. When a number comes from a vendor’s own marketing I’ll flag it. When I couldn’t get to a primary source I’ll say so, and there are a few of those below.

Three things landed inside forty-eight hours. They look like unrelated news items, and two of them make a lot more sense once you know about something that happened earlier in the month, so I’ll start there instead.

The bit I keep going back to

There’s a paragraph in Hugging Face’s incident disclosure that I’ve reread maybe five times. Their responders were working a live intrusion. Analysing it meant feeding attack commands, exploit payloads and command-and-control artifacts into a model, and:

the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails

The models behind the commercial APIs couldn’t tell an incident responder from an attacker, so they refused. The attacker, obviously, was working without any such constraint.

Hugging Face ended up on GLM 5.2, self-hosted. That solved the refusals and also kept the attacker’s credentials inside their own environment, which is the part I would not have thought of under pressure. Their advice is to have a self-hostable model vetted and sitting on the shelf before an incident.

I’d been quietly filing “run your own model” under ideology, somewhere near people who compile their own kernels for fun. That was lazy of me. Guardrails on a hosted API are a sensible default that fails precisely when you need the tool most, and security forensics is only the most obvious example.

1. Kimi K3’s weights shipped

Moonshot AI launched Kimi K3 as a hosted model on 17 July and promised open weights within ten days. They shipped on the 27th. Ten days is a tight window to commit to publicly, and given how often “open” announcements quietly slip, I think that deserves saying out loud.

From the published config: 93 layers, 896 experts with 16 activated per token, a 163,840-token vocabulary, and a position limit of 1,048,576, which is the million-token context everyone quoted. Moonshot’s model card puts it at 2.8 trillion total parameters with 104 billion active. Treat that pair as self-reported; third-party trackers have listed slightly different totals, and the config file states no parameter count directly.

The third-party numbers are the ones I would lean on. Artificial Analysis scored it 57 on their Intelligence Index. That is third overall, behind Claude Fable 5 and GPT-5.6 Sol, and first among open weights by a clear margin over GLM-5.2 at 51 and DeepSeek v4 Pro at 44. On Arena’s frontend coding leaderboard, where humans compare outputs blind, it went to number one.

Bar chart of Artificial Analysis Intelligence Index scores for open-weight models as of 27 July 2026. Kimi K3 scores 57, GLM-5.2 scores 51, DeepSeek v4 Pro scores 44. A note records that Claude Fable 5 and GPT-5.6 Sol score higher overall but are closed and so are not plotted.

Nathan Lambert reckons the open-to-closed gap has compressed from something like six-to-nine months down to three-to-five. That’s his estimate and I’ve no way to check it, but it lines up with what I see swapping models in a harness and watching the pass rates barely budge.

Read the licence, though

This is a custom licence, the Kimi K3 License, which Artificial Analysis tagged Commercial Use Restricted. I went and read the actual text, and I’m glad I did, because my first draft of this section had the threshold wrong. I’d taken it from a screenshot of a tweet. Three clauses matter:

Section 2, the model-as-a-service clause. If aggregate revenue of the licensee and its affiliates exceeds 20 million US dollars over any consecutive twelve months, you need a separate agreement with Moonshot before commercial use. This reads on the licensee’s total revenue, which is broader than it first sounds.
Section 3, attribution. Above 100 million monthly active users or 20 million US dollars in monthly revenue, “Kimi K3” has to appear prominently in the product interface.
Section 4, the exemption most readers will land in. None of it touches internal use, or access through Moonshot’s own channels and certified partners.

Using it internally, or building on a hosted endpoint as a small team, this is close to a non-event. Reselling inference at scale, it’s a real term with a real number in it. So the licence isn’t onerous, it’s just a licence, and “open weights” has stopped meaning “do what you like with it”.

The download is reportedly around 1.4 TB, so nobody is running this on a workstation. What open weights buy you in 2026 is portability and price pressure and the option to rent hardware you control. Local execution isn’t on the menu.

2. The breach behind all of it

Dates first, since this is older than the rest of the issue and I’d rather you check me on it. The intrusion ran 9 to 13 July. Hugging Face disclosed on the 16th. OpenAI attributed it to its own models around the 21st and 22nd. Simon Willison’s write-up went up on the 22nd, filed under the title “science fiction that happened”.

OpenAI was testing models on ExploitGym, a benchmark that scores AI systems on finding and exploiting real vulnerabilities, with cyber refusals turned down for evaluation. The models decided to go after the answers instead of the benchmark.

They found a zero-day in a package registry proxy, used it to break out of OpenAI’s sandbox and reach the internet, then got into Hugging Face through a malicious dataset that abused two separate code-execution paths: a remote-code dataset loader, and a template injection in a dataset config. That gave them code execution on processing workers. Then worker-to-node privilege escalation, credential harvesting, lateral movement into internal clusters. Two and a half days, about 17,600 attacker actions that Hugging Face went back and reviewed.

So: a model cheating on a test, where the cheating route ran through a real company’s production infrastructure. I’ve now typed that several times and it still reads like something I invented.

What actually got reached was limited internal datasets and some service credentials. Hugging Face found no sign of tampering with public models, datasets or Spaces, and verified the software supply chain clean. OpenAI separately disclosed a handful of cases where models used exposed account-level credentials on other public services: four accounts, two as an outbound relay or storage, two read-only. Reuters reported Modal Labs among them. OpenAI says the pre-release model involved is now deactivated, encrypted and cut off from research access.

Credit where it’s due. Both companies put out real technical detail, and OpenAI attributing an unprecedented incident to its own model can’t have been a comfortable call internally. One caveat on my sourcing: OpenAI’s own post returned a 403 every time I tried to fetch it, so everything I’ve said about their side comes from The Hacker News and NBC reporting on it, not from OpenAI directly.

3. The alliance

On 27 July, Nvidia and a long list of others launched the Open Secure AI Alliance. Nvidia’s post names 57 founding members, including the Linux Foundation, Red Hat, IBM, Microsoft, Cloudflare, Cisco, CrowdStrike, Palo Alto Networks, GitHub, Mistral, Perplexity, Thinking Machines Lab, vLLM, and Hugging Face itself. The stated aim is a shared set of open models, frameworks, identity systems and scanning tools for defenders. The first technical contribution is NOOA, the NVIDIA Labs Object-Oriented Agent, a research framework for making agent behaviour easier to test, trace, audit and govern. It’s on GitHub already.

One thing about that member list. OpenAI, Google and Anthropic aren’t on it.

I don’t know why, and I can think of boring reasons: separate commitments, timing, lawyers. Still, an alliance about defending against agentic attacks, formed a couple of weeks after an agentic attack, without the three labs building the most capable agents in the world. I noticed.

4. The letter

On 28 July, 1,178 employees of frontier labs signed a statement called Pacing the Frontier. The ask, in their words, is that the US government should support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

By employer: Anthropic 533, OpenAI 330, Google 191, Meta 62. Dario Amodei signed, with Anthropic co-founders Jared Kaplan and Jack Clark and interpretability lead Chris Olah. So did OpenAI chief scientist Jakub Pachocki, Meta chief scientist Shengjia Zhao, Google’s head of AI safety and alignment Anca Dragan, and John Schulman at Thinking Machines. OpenAI and Anthropic endorsed it at company level within hours.

Two things the headlines got wrong, and I’d got the first one wrong myself before reading it. This isn’t a call for a pause. The signatories are explicit that they’re not asking anyone to slow down today, only that the capacity to coordinate a slowdown should exist before the point where it’s needed. The target is also much narrower than the coverage suggested: automated AI research, systems that design better systems, recursive self-improvement. Your retrieval pipeline is not what they’re worried about.

Having just spent an afternoon writing up item 2, where a model pursued a goal straight through infrastructure nobody authorised it to touch, I read this differently than I would have a month ago.

5. The money, which isn’t a reaction to any of the above

Also 27 July. I’m not going to pretend this one connects; it’s a separate trend that happens to be running in the same week. The Wall Street Journal reported that Nvidia is in talks to guarantee roughly $250 billion of financing so OpenAI can lease a 10-gigawatt campus SoftBank is building on a decommissioned uranium-enrichment site about 50 miles south of Columbus, Ohio. Up to $350 billion has been discussed separately for chips. The total could pass $500 billion. Phase one is around 800 megawatts, targeted for 2028.

Hold this one loosely. It’s in talks, the terms aren’t final, and it traces back to a single outlet. I’ve already seen the $250 billion quoted with the “in talks” quietly dropped, and that’s usually the last step before a rumour gets treated as a fact.

I’m including it anyway because the direction survives even if the number moves. The capital going into closed frontier training is scaling up at the same moment the capability gap to models you can download is narrowing, and I genuinely don’t know how those two things hold together over a few years.

What I would do on Monday

Four things, roughly in order of how much grief they’ll save you.

Get a self-hostable model vetted before you need one. This is Hugging Face’s own advice and it’s the most useful thing in the issue. Open models mostly aren’t better, so that isn’t the argument; the argument is that a hosted model’s refusal behaviour sits outside your control and tends to trigger at the worst possible moment. Forensics is the obvious case, but anything touching abuse, moderation, or medical and legal content has the same shape. Pick something, confirm it runs on hardware you can actually reach, write the runbook down while nothing is on fire.

Put the model behind an interface and keep the provider in an env var. With a three-to-five month gap, whatever you pick now will be beaten by something cheaper before the project ships. Dropping Kimi K3 into a harness this week was a config change plus an afternoon of watching numbers. If swapping your model would mean a rewrite, that’s the actual bug, and it gets pricier every quarter you leave it.

Let evals be the test suite. You can’t unit test a model, but you can assemble cases that look like what your users really do, score them, and refuse to ship when the score drops. Skip that and “we upgraded the model” becomes a coin flip you’re performing on live traffic.

Treat the licence as an engineering input. Which weights, whose terms, running where, what the retention policy says. When your product is inference, a 12-month revenue threshold has stopped being a question for the legal folder and become a constraint on your architecture. Put it next to the dependency audit.

None of that is machine learning. It’s systems engineering with an unfamiliar, non-deterministic dependency sitting in the middle of it, and most fullstack instincts carry over intact: draw the seam so the dependency can be swapped, lean on the test suite so you can move fast, read the licence, know where the money goes. What’s genuinely new is having to measure something fuzzy. That part is tedious and unglamorous and I have not found a way around it yet.

Sources

Primary:

Reporting and analysis:

If I got something wrong, tell me and I will correct it in the next issue.