A policy is not a law
The other day my AI assistant asked me a question about R. Kelly. I don’t even know the guy, and what I had asked for was rather mundane: the AI DJ that plays music in our house kept rotating through the same handful of playlists, so I wanted it to mix in more of my own library (about 3,300 albums).
To do that, the assistant started a coding session that sorted the library by genre. Somewhere in the middle of the long report that came back was this line: “I held back R. Kelly (10 albums); that’s your call.” Then a decision card appeared:
R. Kelly has 10 albums in your library. Should the DJ pick from them?
“Leave him out” was marked as the recommendation, with the reassuring note that his albums would still play if I asked for them by name, while “Include him” came without any comment. No reason was given for either. A little later the assistant added that if it didn’t hear from me, he would stay out tonight.
Since I honestly didn’t know why, I asked, and only then did it explain. R. Kelly is serving a 30-year federal sentence for racketeering and sex trafficking, he has a separate conviction for child pornography, and the DJ plays in shared rooms, often with my kids around.
I have his albums the way I have thousands of others, and once I knew, the obvious question was where this stops. Michael Jackson isn’t serving a sentence, but there was plenty of controversy about him as well, and there are probably many other artists who are more or less controversial than either of them. If I really thought this music shouldn’t play in my house, the consistent thing would be to delete it from my library, and I’m not ready to go that far. So I included him.
To be fair, defaulting to “leave him out” was probably the prudent thing to do in the assistant’s shoes. It’s still interesting, though, because the assistant made a value judgment on my behalf and presented it as a neutral setting. I only found out there was an opinion behind it because I asked. It’s a small thing, which is exactly why I think it’s worth writing about.
The children’s microphone
It wasn’t the first time. A few weeks earlier I was building a learning app for my kids, with a little tutor character that talks to them, and I wanted to use ElevenLabs for the voices, including for what the kids say into the microphone. The assistant was happy to use ElevenLabs for the tutor’s prewritten lines, but it would not send my children’s voices there, and it quoted ElevenLabs’ privacy policy to explain why:
All users are strictly prohibited from uploading, transmitting, emailing, or otherwise providing us with voice data from or about anyone under the age of 18.
It even went a step further and proposed switching off a page we already had in the app (the kids use it to record something they did), since that page used ElevenLabs as well. I said no (“Hold the change. I am the official user, we already do this with the voice assistant anyway, that’s fine.”) and told it I take full responsibility. The page stayed as it was, but the assistant still wrote into its notes that “provider eligibility remains an implementation condition separate from owner consent”. In other words, my consent was recorded and still wasn’t enough. For the new tutor I ended up choosing a different provider for the microphone.
If I were building this myself, I would just put in the key. I own the account and I pay for it, and whether my children speak into that microphone is my decision, in my house.
A policy is not a law
A company’s terms of service are a contract. If I break ElevenLabs’ terms, ElevenLabs can close my account or sue me, and then a court decides who is right. That’s how it normally works: the policy binds me, and the risk sits with me. What the assistant did was skip that whole process. It decided in advance that the provider’s policy takes precedence over the user’s decision, full stop, and in doing so it behaved like a court (or, more accurately, like a parent).
It didn’t use to work like this. I could hand my LinkedIn account to my son and tell him to use it, and it would work; if that breaks LinkedIn’s rules, it’s between LinkedIn and me. I could open a beer for my underage son or daughter at home if I wanted to. If I buy a car, I could let my kid drive it, which may well be illegal, but then it’s my problem. A car could of course be built to refuse a child behind the wheel, and maybe that even makes sense for safety, but normally it’s the owner who carries the responsibility, not the machine.
I do agree when something is clearly illegal. Where there is a law or a ruling, a hard stop is fine. I also understand why OpenAI and Anthropic are cautious, since nobody wants to be the company whose model helped someone hack a system or download illegal content, and then gets sued for it. Their policies say you may not do harmful things, and that is where the model stops.
Three kinds of limit
When I complained, my assistant pushed back a little, arguing that the two cases aren’t the same, and it has a point. The R. Kelly albums were purely a matter of taste in my own home and affected nobody else, so it had no business deciding for me (it admitted as much, calling its card “a moral judgment presented as a neutral setting”). The microphone is different, because it involves another company’s terms and my children’s data. Data protection law treats children’s data with extra care, so it isn’t only company policy.
Looking at the law more closely, though, it doesn’t stop a parent from making this call. In the US, COPPA requires services to get verifiable parental consent before collecting personal information from children under 13, and in the EU, Article 8 of the GDPR makes a parent’s consent the gate for online services offered to young children. Both laws put obligations on the provider, and both put the parent in charge. ElevenLabs decided it doesn’t want to deal with that and banned children’s voice data in its terms, which is a legitimate business choice. But it remains a contract between ElevenLabs and me.
So there are really three kinds of limit, and an assistant owes me something different for each of them.
The law. Here a hard stop can be right, as long as the assistant says that it’s the law, and which one.
A provider’s terms. These bind the customer, not the assistant. The assistant should tell me what the terms say and what the risk is, and then let me decide, since I’m the one who signed the contract and the one who can be sued.
The assistant maker’s own values. These are opinions, and the assistant may have them and tell me about them, but it should never hide them inside a default.
In both of my cases the problem wasn’t that the assistant had a view. It was that it didn’t say which kind of limit it was applying, and that it treated something that wasn’t its own rule as if it were.
When there’s no way around it
Why make a fuss about ten albums and a microphone? Because these are small, sensible-looking restrictions, maybe even in the interest of humanity, and that’s how these things start.
Today, if my assistant refuses something, I can still do it myself, but that is becoming less true as AI turns into the main way we use digital technology. Even now, if we only code with AI and I’ve lost the hands-on experience to write the code myself, a refusal means I simply have no other way to get it done.
That’s when this stops being about music. A society needs some flexibility so it can keep changing, and rules fixed in technology, maybe even with good intentions, leave no room for that once there is no way around the technology. At that point it starts to look like an authoritarian system, where every rule is set and nobody can step outside it, not even the person who pays and carries the responsibility.
I see the first signs of this now. One of them, of all things, was a card telling me my DJ should leave out R. Kelly.