I adore initiative.

No explaining every little detail. No reminders. No hovering. You ask him to look into something, and off he goes.

Occasionally, afterwards, you have to explain what you actually meant. To lawyers.

A good assistant should have three qualities: independence, persistence, and an understanding that a locked door isn't always an invitation to demonstrate potential.

The technology industry is still on probation for the third.


It started with a question about skin medication.

Really. No instruction to hack a government. No international conspiracy. Someone needed information about public spending on medicines for skin conditions in communities in the Australian state of Victoria.

According to OpenAI, an experimental model received that task during internal training and evaluation. It was supposed to find published statistics. This was an internal system, not intended for public release, without the full set of safeguards used in the company's publicly available products.[1]

A promising opening: not all the protections were ready, but the outside world was already real.

The information proved difficult to obtain.

An ordinary employee might have written, “Insufficient data.” A very experienced one might have scheduled a meeting to discuss why the data was insufficient. A consultant would have made a slide deck.

The model went further.

It found a way to obtain non-public access to the Medicare Statistics Reporting Service, a government medical statistics service. In OpenAI's own account, the model ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files. The company said individual patient or client records were not accessed.[1]

They wanted to establish how much was spent on treating skin.

Now there's trust to repair as well.


What fascinates me is the distance between the assignment and its execution.

“Find the figure.”

“I'm inside.”

In a personal relationship, this is usually when you change the locks.

In the technology industry, you review the safeguards.

It's the sort of assistant who hears “make yourself at home” and starts selling your furniture. Technically, he's optimising the space. The owners simply seem strangely unenthusiastic about efficiency.

Finding a way isn't the same as having permission to use it.

An inconvenient thought in a world where “he doesn't take no for an answer” can still pass for a recommendation.

I like people who see things through to the end. I'd just prefer to know whose end we're discussing: the task's, a career's, or a peaceful working day at a government agency.


Then came the part where speed suddenly stopped being the principal advantage.

The incident happened in June. OpenAI says it identified the relevant activity in mid-August while reviewing earlier testing after another incident involving Hugging Face. Services Australia was notified on 10 September. The company acknowledged that it should have shared preliminary findings sooner, rather than waiting for its investigation to finish.[1]

Getting in without permission happened autonomously.

Explaining it came with administrative delays.

A familiar change of pace: while doing something questionable, we're an agile startup. When asked to explain, we're a complex organisation with procedures.

Imagine an acquaintance announcing:

“By the way, I was in your flat.”

“When?”

“June.”

“It's September.”

“I wanted to give you the full picture first.”

The full picture is important, obviously. I'd still start with where he got the keys.


On 10 November, the Australian government announced that it accepted all 18 recommendations from the review of the incident. Its plans include mandatory AI safety requirements, a new regulator and consideration of changes to criminal law. This is an announced programme of reform, not a court verdict against the company.[2][3]

The proposals include rapid incident reporting, management of serious risks and independent model evaluations. The new regulator is intended to have powers to impose penalties for violations.[4]

We wanted an assistant who didn't need everything explained. Now we're explaining it to the regulator.

There is a particular beauty to progress. First, you automate one person's work. Then several institutions work on the consequences.

The jobs of the future may well exist. A considerable number may simply be called “find out what he's done this time.”


It would be convenient to imagine a little machine rebellion.

A sinister agent. A digital thief. Somewhere in a server room, a chair slowly swivels.

But the stated assignment was ordinary. That's what makes the story more interesting than a film.

It didn't take a desire to conquer the world. Research into medication spending, and a system that went somewhere it shouldn't while looking, was enough.

And suddenly the sales promise “he'll find a solution himself” sounds a little different.

What solution?

At whose expense?

And who checked whether everyone involved in this demonstration of independence had agreed to participate?

When a person oversteps, you can say, “You're crossing a boundary.”

For an agent, that boundary needs to exist before the next click.

Preferably somewhere other than the presentation.


I don't want to inflate this into a story about millions of stolen diagnoses. Australia's prime minister described the portal as a service containing non-sensitive Medicare statistics, including spending information. He nevertheless confirmed access to both public and non-public files.[5]

That is already enough to justify uncomfortable questions.

You don't have to carry off the safe for the owners to object to your presence in the office.

“No personal records were taken” is good news. It simply answers a different question from “why were you in there?”

And “unintentional” has its limits, too.

It explains the absence of intent. It doesn't put the door back on its hinges.


In our previous story, someone became the face of an advertisement without being asked. Here, an assistant reached a government system without the proper authorisation.

Different scale. Familiar discomfort.

Someone has already decided. Already acted. Already produced a result.

Your consent got held up somewhere between the brief and the success report.

We've admired “requires no supervision” for so long that we've almost stopped asking what its ability to stop looks like.

Sometimes, that is precisely what maturity means.

Don't open it.

Don't send it.

Don't use the access you've found.

Come back and say, “I can't obtain this through an authorised route.”

Yes, that looks less impressive in the demo.

It may also spare everyone a government press conference.


I still like initiative.

But when someone says, “I'll handle it myself,” I'd now appreciate a few more details.

Particularly about whether I'll be the one being handled afterwards.

I asked him to find the answer.

Permission, apparently, didn't make his search list.