Everyone has their own fears about AI, but my fears are especially chilling; what if AI was used to imitate a person saying something blasphemeous?
My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob.
And this was before AI was easy to access. You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread. You can't just give a toddler a knife and then blame them for stabbing someone.
Has nothing to do with fame, with security, with copyright. This will get people killed. And we have no tools to control this.
https://x.com/search?q=blasphemy
I fear the future.
I think the answer, counterintuitively, is to make these AI tools more open and accessible. As long as they're restricted or regulated or inaccessible people will continue to think of videos and recordings as not fakeable. But make voice cloning something easy and fun to do with a $1 app, let the teens have their prank call fun and pretty soon it should work its way into the public consciousness.
I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something.
I think it might be a rough couple years but hopefully we'll be through it soon.
This is idealistic. People still haven't fully learned that images can be photoshopped in its twenty years of its existence. (Deep)faked porn is still harmful which is why it's a crime.
Worse, there isn't an attitude of default skepticism in many areas/cultures. If a person is suspected of violating the moral code the priority will be punishment and reinforcing that such behavior isn't acceptable. Whether or not the specific person actually did the specific act is a secondary concern.
It's just going to increase the number of people who will be harmed or killed.
The first thing that should be done - which should be easy enough to do, is demonstrating blasphemy with the voices of all the most pious leaders that lead the lynch mobs. If those ones get believed, well, problem is solved in an alternate way.
I also however don’t believe that deepfaked porn is actually harmful. People are already imagining people naked, and it’s been easy to photoshop a nude for 15 years. A talented artist was needed 200 years ago. Allowing this (admittedly rude and disrespectful) act to be done with less skill changes very little.
Yep, had a lawyer telling me that image timestamp cannot be faked. While it's literally a right-click away.
Well, thing is, most people can't.
In a lawyers view, and a judge's view, some skilled expert "hackers" can, and its called hacking. (so i guess we're all hackers)
I once discussed these things with a (knowledgeable) lawyer. He explained you can just present almost anhthing in a court case, and when it isn't refuted, well then it's valid.
In a case my lawyer (same one) presented a printed out email. Other party did not claim it was false, so it's suddenly just as valid as a registered letter. (it was a genuine email).
In another unrelated case, the other party suddenly introduced a forged picture. If I hadn't been there at that moment (I wasn't supposed to actually), then suddenly it would have been proof.
Court cases are not about truth, and not about justice. They are about convincing the judge.
Well, at least for email, it's theoretically possible to prove it's authenticity through third-parties. E.g. lawyer can ask GMail "did you receive this email with this DKIM?".
And judge's job is not to find the truth. It's to convince the public that ruling was just.
When people in positions of authority, like legal experts, don’t fully grasp how easily digital content can be manipulated
I'll note that both photoshop and changing the timestamp of images (mentioned below) are only easy for a very small percentage of the population. It'd likely be different if >30% of people could easily do these things.
Changing an image timestamp -- as in, exif metadata -- is trivially easy for anyone with a computer or phone, a quick search will tell you how to do it on any device with no skill involved. There's a difference between "easy for a small percentage of people" (modify the content of a photo in an undetectable way, find a software vulnerability, etc.) and "only a small percentage know how to do it" (modify exif values, do a simple magic trick, etc.). Just because someone doesn't know anyone can do it if they tried doesn't mean it's not trivial and pervasive.
The problem that to make a right query, one has to know at least part of the answer. To search for exif modification tool, it's essential to understand: - what is image file, and how information stored in it. - the concept of `metadata`, and exif in particular. - the concept that any file can be modified in place, or when you make a copy. - know what where's ecosystem of open/free/custom tools that can do an unusual tasks. The small percentage has all that knowledge at the hand to even get an idea of possibility to modify image timestamp
This.
I'm currently the originator and lead on a project that will have huge impact on our bottom line. The team is mostly junior people, so when I presented the plan the response I got wasn't "I didn't think we could do that", it was "I didn't even know that that is a thing".
There used to be a joke about Rumsfeld's "known unknowns" vs. "unknown unknowns", but it's so true. And to the vast majority of people in the world the question isn't "how" to change a timestamp, it's understanding that a timestamp can even be changed.
Most people will believe a rumour if it is told to them in person by a friend. We've had our entire evolution worth of time to recognize that rumours can be manipulated yet rumours still spread and are still very dangerous.
> I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something.
Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?
Almost none, at most as little as a professional impersonator requires. GPT-4o's advanced voice mode would clone the user's voice by accident. A recording clip of one incident is available:
>During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice.
>Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice
https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7...
Skype does it in real time (for live translation) with a few seconds of audio. For the reasons discussed in this thread, it continuously forgets its previous training from the call to not make the voice too similar, but just similar enough to distinguish the speakers.
Just a few seconds of speech gets you pretty far with a lot of models
Yes, this is generally true, in order to clone the timbre/pitch/tonality, but if you're going to run a voice cloning scam to fool friends and family members, you need much more to get the cadence, pauses, vocal tics, etc right.
> what if AI was used to imitate a person saying something blasphemeous?
> My country is already has blasphemy lynching mobs
in your case the problem is not AI, it’s your country.
Your country might not have lynching mobs, but you can't deny there are certain taboo topics in your society also, certain slurs and other opinions which would take you ages to clense and even then never fully.
If an AI fake-porn of some ordinary person involving a minor was unleashed, think of the utter shame and horror they would be treated by people for the rest of their lives, even if it were proven false.
No one would believe them, work with them, hire them, rent them, they would wish they had been lynched instead of the life they live.
Yes, you’re right. That’s why the problem is the lynch mobs. If there was no AI, people would find another way to sic the mob on someone. I’m sure it’s already happened countless times without AI. Mobs aren’t known for the rational behavior and respect of the rule of law.
I was simply explaining my personal fears regarding AI given that lynch mobs exist in my country; the lesson you should have taken from that was not about lynch mobs, but to analyse how AI might affect YOU in your particular environment.
Remember the post is about cloning a youtuber's voice using AI; most people were thinking of the copywrite aspect of it, but I wanted to share with people what my fear was.
My fears are, things are already bad w.r.t lynch mobs before AI, after AI things will only get worse with fake but realistic sounding voice notes etc. That's my fear, doesn't have to be yours.
You don't have lynch mobs (hopefully) but surely you can think of other problems that a fake AI image/audio/video can cause, and just low the barrier is, and just how good the tools are.
Maybe it's because I have no knowledge of lynch mobs in my country but I find that fear is a bad adviser. I think currently there is no way back in AI development. I read Microsoft is getting a failed nuclear reactor repaired to power the power hungry AI.
The only way forward is making sure most people know that digital content can be easily faked. If you hand out good faker tools for free, it will happen faster
There is absolutely a way back if laws change. It's that simple, people need to stop pretending its inevitable when it absolutely isn't.
International law like in... human rights? Are those currently being enforced?
I don’t disagree that AI could exacerbate the issue, in fact I agree. But anything could potentially do so. Instant messaging likely also made the problem worse as a call to form a mob can reach many people much faster than before. The time from inciting incident to mob action is greatly compressed because of instant messaging.
It’s fundamentally a societal issue, not a technological one. Yes, it’s scary what effect AI could have on lynch mobs, because lynch mobs are scary.
But mob formation and mass social shaming on little evidence already happens without AI and it isn't clear why AI would make it worse. If it gets so easy to create fake videos that'll just muddy the waters and people would trust rumour less, not form mobs more.
It happens once, then some wag creates videos of all the mob leaders blaspheming or whatever. Undermines the idea that the videos are the root cause.
You are giving too much credence to the analytical capabilites to a lynch mob, and you are underestimating how much AI lowers the barrier for creating realistic 'proof' for incitement.
Also there are no mob 'leaders'. These are everyday folks, these are you and me and my neighbours.
A student pissed at a teacher could create a fake AI video of his teacher at night, and wake up to no pesky teacher and feel no remorse over what happened while he slept.
The student was not a mob leader, just some dumbass who was angry at their teacher for giving bad grades, and had access to simple AI tools (soon available online for a trivial fee, if not free).
____
Be honest, are you 100% sure of the status of that facebook account you deleted in 2018? Zuck never deleted it, that content is still there to be mined and abused. Also not that hard to just log in to the damn thing and post something spicy.
I don't remember if old posts can be edited in FB, cause that would allow the person to really gild the lily.
> people would trust rumour less, not form mobs more
I believe you are right. When the first unbelievable clips appeared on Facebook (where the most gullible people are), you could see the old ladies cheering "oh how wonderful! where did you film that?" Now in the comments section you will mostly see people saying "sheesh, stop polluting my feed with these fake AI-generated spam". Don't get me wrong, many people still fall for them, but the tendency seems clear - after all, we managed to survive because of learning.
I feel like the Bullshit Asymmetry Principle is involved here. If the bar is lowered to the point where it takes less energy to generate inflammatory video "evidence" than it does to debunk it the shitheads win. I feel like we can all agree that AI has the potential to lower that bar.
Next or next next generation would be used to it. Digital content would be consumed differently, with automatical scepticism or so...
Remember when people believed that camera took our soul when we are caotured on image?
The US equivalent is much less labour intensive than a lynch mob: it's mass shooters radicalized by things they've read on the internet.
Or https://www.npr.org/2024/09/19/nx-s1-5114047/springfield-ohi... , where repeating racial libel causes a public safety problem.
While this kind of incitement in no way requires AI, it's certainly something that's easier to do when you can fake evidence. See also https://www.bbc.co.uk/news/articles/c5y87l6rx5wo
https://apnews.com/article/ai-maryland-principal-voice-recor...
In the US, freedom of speech means the government won't stifle you, but the court of public opinion will crucify you for saying the wrong thing. In the linked case, a school principal was framed for being a racist, a fire able offense. Better than being killed by religious extremists, I suppose, but still not great. Thankfully in this case they were able to find the perpetrator who faked evidence, but we have a problem.
>radicalized by things they've read on the internet
That take pretty much throws the shooters' preceding years of anguish under the bus to make a political score. "Ban the wrongspeech!" is a seductively easier proposition than ending youth bullying and ostracism.
Free speech doesn't protect against libel/slander. It also doesn't protect against speech that induces violence or panic (the old "yell fire in a theater") And I argue impersonating celebrity voices to radicalize others falls under the latter.
So yeah, I would call this "wrongspeech" and wrongspeech is already banned.
Quite a few (not all, but the more famous ones) write manifestoes in which they describe how they were radicalized, cite other manifestoes and mainstream racist commentators who inspired them, along with Adolf Hitler, and make it very clear what their motives are.
> youth bullying and ostracism
... which itself counts as free speech?
A lot of people get bullied in a lot of countries. Few of them turn to mass murder. The mass shooter phenomenon is mostly America, plus a few people who've been radicalized by the same racist discourse on the same message boards (Anders Brevik, NZ mosque shooter).
In quite a lot of cases closer investigation suggests that they were ostracized precisely because they'd already chosen hate.
The internet isn't always the bogeyman, the last assassin was radicalized by mass media.
People don't shoot up crowds because of something they read on the internet. They do it because they are done with life. Everything after that is retrospective reasoning.
Some of them (e.g. Breivik) write manifestoes in which they list all the stuff they read on the internet that motivated them.
It's no one factor, and I'm sure internet contributes. Especially if you feel isolated in the physical world.
Yes, but after the invention of radio and television we already enabled mass indoctrination and mass brainwashing. The internet is more refined but the same principle.
The solution is not to ban technology (it's impossible), but to create a more decentralised society where the majority of voters, which is easily brainwashed, doesn't get to dictate the life of the minority. We need to destroy and dismantle all the gigantic entities that ruin our life for the advantage of the few, whether they are the government or corporations ruled by billionaires controlling the government.
We also need to create a better society with less mental illness, where people are not depressed by how unfair life is that they go on and kill kids in school - but I believe if we reduce large societal inequality this will sort itself out over time.
Parents in my circle are already banning social medias for their kids until they're adults - so over time we'll evolve to get better at ignoring all the crap we get exposed to.
What country is immune to this?
As far as I can tell the collective conscious of every country is swayed by propaganda.
A written headline is enough to incite rage in any country much less a voice or video indistinguishable from the real thing.
Folks in “developed’ countries have their lives destroyed or ended all the time based on rumors of something said or done.
Any country that doesn't have executions by mobs. Used to be most nordic countries. Till they changed their immigration policies.
OR any number of non-racist forces at work e.g. the internet, movies and social norms changing everywhere.
Those are the mobs. Your average "non-racist" force is just racist to some other race.
That's a little too easy no? AI being used to imitate people definitely is a problem that needs to be addressed, and already is. Discarding that because there is a bigger issue is ignorant. Both can exist as a problem at the same time.
The problem is AI. What if you post video of a politician eating babies, and that causes some nutjob to kill that politician?
Sure, distrust everything digital, but what if only evidence of someone doing something wrong is digital?
What if someone printed yellow papers blaming Spain for a ship accident so the us would go to war with them?
Becoming famous always had its risks because of mentally ill people targeting you.
Nothing new.
The trick is being rich without being famous!
People copied and pasted faces of celebrities on another photo since 1990s. Nothing significant happened.
problem is not celebrities, problem is general every day people.
With celeberaties, atleast they are famous enough that the idea of fakes exist... but you are Johnny nobody, that's not a fake, you just got caught and don't want to admit you said/did it!
The court of public opinion is much smaller and personal, and 'evidence' is much more realistic and detailed.
general every day people will have to figure out that photoshop exists and that it contains AI now that can invent pictures from scratch. It should be easy to figure out, just explain that it can paint 100% realistically like a camera. The real issue is how people will start using this as a defense for criminal acts. "it wasn't me, all the camera footage is AI generated"
An individual is not responsible for the culture or government in the country they live in.
In the UK a government was just elected with a historical absolute majority by only ten million people, and now first time offenders are being sent to prison for making stupid offensive statements online.
Are you referring to this case?
> In a now deleted post on her X account, Mrs Connolly, from Northampton, wrote: “Mass deportation now, set fire to all the f*** hotels full of the bastards for all I care… If that makes me racist, so be it.
> The court heard Kay copied, pasted and uploaded Mrs Connolly’s post at 12.27pm on Wednesday from a BBC News report and added the hashtags #standwithlucyconnolly #lucyconnolly #f**northamptonshirepolice #conservative #FaragesRiots #RiotsUK and #Northampton.
> As well as a post which urged people to “mask up” during a protest targeting an immigration law firm, Kay tweeted to his 127 followers at 2.34am: “That’s 100% the plan, plus gloves. No car either so no number plate to trace and a change of clothes ready nearby.”
https://www.independent.co.uk/news/uk/crime/northampton-bbc-...
Free speech, should not just be for people we agree with. Although I agree conspiracy to commit a crime is something else.
My point was that AI faked content could certainly get you locked up in the UK.
That may be true, but it doesn’t unkill the victims.
The comment didn’t say the problem was AI, it said they feared its consequences, which is a perfectly valid concern.
It’s like if someone said “I’m scared of someone bringing a semi-automatic weapon to my school and doing a mass shooting. My country has lax laws about guns and their proper use”. And then you said “in your case the problem is not guns, it’s your country”.
I mean, it’s technically true, but also unhelpful. Such ingrained laws are hard to change and you can be placed in danger for even trying.
Before someone decries the gun example as not being comparable, it is possible to live in a country with a monumental number of guns and not have mass murdering every day. It’s called Switzerland.
But let’s please stick to the subject of AI, which is what the thread is about. The gun example is the first analogy which came to mind, and analogies are never perfect, so it’s unproductive to nitpick the example. I don’t mean to shift the conversation from one contentious topic to another.
> It’s like if someone said “I’m scared of someone bringing a semi-automatic weapon to my school and doing a mass shooting. My country has lax laws about guns and their proper use”. And then you said “in your case the problem is not guns, it’s your country”.
The US has both problems: widespread availability of weapons and a high level of freedom to incite violence through spreading lies about groups. Which is why it sees much more of these incidents than countries which have similar levels of gun ownership.
The non-gun version of the problem is mass stabbings, which are less lethal.
Switzerland allows you to own a gun, but (generally) not to bear it. Huge difference.
America also has laws against taking guns into theaters and indiscriminately shooting people, once you have easy access to guns the gun becomes a potential solution to problems. Maybe Switzerland just has less problems where that potential solution appeals to people.
I'd say that the key difference is that Switzerland has compulsory military service in which you're taught how to operate a weapon properly. Therefore, everyone that has one has gone through months of training.
Vs the US, where there are loopholes that you can use to avoid even a basic background check, and then use it for the very first time to shoot someone.
Training seems likely to cut down on accidental shootings, but not so much on the deliberate ones.
I disagree. First of all, people with issues are more likely to be found out in boot camp.
Second, you learn a certain respect for the firearm and are expected to observe strict safety rules when handling it. That gives you a kind of psychological flinch when you consider doing anything out of the norm with it.
It seems like tons of toddler are accidentally firing guns as well in America, so training probably won't help there.
Training people to lock up their damn guns would.
That doesn't seem to make sense. All that would do is to make sure that a nut job knows how to properly shoot the gun. Basically what you are saying is "a Swiss wouldn't have missed Trump".
The difference might be in what guns mean to Swiss people vs. the US.
A gun in the US has this "if the government becomes destructive of these ends we can start shooting" connotation. A gun is there for self defense. Someone doesn't get off your law, you use your gun.
In Switzerland the gun and the compulsory military service you mention is there for the people to protect their country and fellow countrymen. You are trained in defending your neighbor, who just stepped on your lawn against outside aggressors.
Biggest difference is that you have only rich people in Switzerland. Almost no crime and hardly any immigration issues.
> what if AI was used to imitate a person saying something blasphemeous?
The problem I fear is that, let's say you once had a facebook account, we all deactivated our accounts when there was wave against Zuck a few years back, but as we know, facebook doesn't really delete your account.
Now imagine that account was linked to a SIM. It's trivial for a nefarious actor to get it re-activated, infact there was a video by Veritasium just today where they didn't even need your SIM.
But even if they are not that hi-tech, it's not that hard to get a SIM issued in your name, or other hacks of a similar nature, we have all heard of stories.
Worse, you lost that SIM a decade back, the number gets back into the queue, and is eventually re-issued to someone new... and they try to create a facebook account, and are presented with yours.
They can then re-activate your old facebook account, and post a video/audio/text of "godelski" saying they like pineapple on pizza. and before you can defend yourself, the pizzarias have lynched you.
(I dare not use a real example even as a jest, I live here)
Are you 100% sure of all your old social media accounts, all the SIM you have ever used to log-in to accounts?
We leave a long trail.
I think we need a data Great Reset: all data (and I mean all) must be deleted by a certain date in the future, and all newly collected data must have provenance so in the future the customer can find out that Ford got information from Databroker A which bought it from B which bought it from Microsoft which collected it from what you type in Windows. Companies holding activity data without provenance get the crack-cocaine dealer treatment: officers in jail, everybody fired, investors and lenders fucked, company liquidated and forfeited.
While it is impossible to take back a disclosed secret, we can create a legal framework which issues immediate business-stopping corporate death sentences for spreading that data without consent (or after you revoke consent, yes it needs to be revokable). Your data is so valuable, because now that we have universal function approximators you can be simulated (to a prediction horizon) and that simulation interrogated.
There should be a way to cryptographically sign posts (everywhere). I know, building a web of trust sucks, etc. But if there was someone with your username signing with a particular key for 10 years and then suddenly there is something controversial somewhere with a different key, something fishy is going on.
Of course, this could be misused to post something with plausible deniability, but if you want to say something controversial, why wouldn't you make another account for that anyway?
I know that one could theoretically sign posts with GPG, but it would be much nicer and less noisy if sites would have UI to show something like: Signed by <fingerprint>, key used for N years.
One issues is that most social media want your identity to be the account on their service and not some identity (i.e. key) that you control.
> There should be a way to cryptographically sign posts (everywhere).
Key signing will not be the solution because it isn't addressing the problem. It exacerbates it.
This just confirms it is me. Which yes, reduces the problem of me being replicated, but does not do anything for my anonymity. That part may not seem important to you, considering you use your real name, but it is to me. It allows me to be more open.
Ah, think I see your point. Your worry is that language use, etc. could be use to deanonymize you, by correlating with text that was not written anonymously. But that's a separate issue from voice or writing style cloning to pretend it's you that said it. In the latter case you could use a pseudonymous signing key?
I agree that deanonymization is an issue that is hard to tackle. I wonder if someone studied how unique writing style is. E.g. browser fingerprints are fairly unique, but I wonder to what extend you can filter a person from, say a pool of 100 million, using writing style alone (grammar, vocabulary use, etc.). I guess it becomes quite easy if you engage in a lot of domain-specific discussions and use their vocabularies.
E.g. if I'd talk about Marlin-kernels here, you could probably narrow me down to a few hundred people. Throw in another comment about the Glove80. Maybe ten people at most?
> I wonder if someone studied how unique writing style is.
Your language is in fact a fingerprint. And like you suggest, topics too. Much of our freedom of anonymity comes from the fact that it is hard or not worth it to dox people.
I do agree that verification is a different issue though. I'm not sure keys will solve it because you're not going to sign anything that is scandalous, so it might even give evidence for those that want to falsely claim foul play. And how do you sign a leak?
The problem with signing is that it seems to work for the cases we don't care about and do nothing for the ones we do. That is unless we sign literally everything, including our voice, but then you kill anonymity (why I connected it) and you could then probably clone that too.
Not sure the lynch mob would pause to check that the web of trust is unbroken.
Another solution would be to use an LLM to rephrase your posts, wouldn't it?
Not a great outlook though, if everybody does this...
I already do this in all my IRCs, Matrices, and Discords
Not really. Though that will average language and poison LLMs. I'm not sure I want either of those to happen.
Besides, topics and shared details are also pii. It's more that they aren't useful in that way until you have scale.
Plus, on HN you can't edit posts after some time.
>I'm not sure I want either of those to happen.
at this stage and current behavior, I wouldn't mind some poison. That's the only way companies will learn at this point.
Yep, im sure lots of people have written a lot of random stuff on a lot of forums that should absolutely stay anonymous from gossip to family secrets to honesty about career/workplace and what not.
If stylometric analysis runs on all comments on the internet then yeah.
Bad things will happen, very very bad.
I honestly think it should be at least illegal to do this kind of analysis because it'll be a treasure trove for the commercial sector to mine this data correlated to real people not to think of the destruction in millions of people with personal anonymous blogs etc.
Actually thinking about it further you could also easily group people political affiliations, and all kinds of other thoughts, dark, dark stuff!
Idk the state of the art for forensic stylometrics or the precedents but I would be surprised if this hasn’t already been presented as evidence in many (most?) notable jurisdictions. Not definitive evidence but supporting evidence. So far from being declared illegal, it’s probably mostly established as a tool for law enforcement.
Meanwhile gangsters or nation states are probably already working on automated deanonymization and blackmail at scale, but will target the dark web before the public one. Not sure any of this stuff is that changed by the advent of deep fakes and llms tho, probably just boring classical statistics does ok?
I wonder if in this timeline it would cause people to clean up their behavior online and only post things they are comfortable having being linked back to their real identity. Would have a chilling effect.
There's quite a few sci-fi books, movies, and TV shows that explore this concept. Probably the most well known is 1984
But I'm pretty sure this would make life very bland. Slow down innovation as we become less creative, less willing to deviate from the norm, and the Overton window of what's socially acceptable narrows. After all, we do love having enemies in some way or another, even if it means making them up.
>Actually thinking about it further you could also easily group people political affiliations, and all kinds of other thoughts, dark, dark stuff!
Already happening at Google and Meta for a decade or more.
Stuff like this is why I tell everyone who does good work in the AI space to remember that they are watched
Working in the ML space myself I try to tell my peers to think deep about their work and the consequences. Not just for ethics, but because it helps you do better.
So far I get a lot of pushback and statements like "it's just linear algebra" and "quality doesn't matter, it's quantity. Just scale. That's the bitter lesson". The former tells me they know very little linear algebra and the latter tells me they know very little about probability distribution and the concept of coverage... I routinely see papers get awards and tens of thousands of citations when they're making errors I would expect undergrads in my classes to do better at... I think a big reason I'm frustrated is that supposedly we share the same goals -- building AGI -- but I'm told I'm stopping them from doing so while they reject my papers and theirs gets accepted... (Fwiw I'm happy to accept scale papers. There should be a wide range of ideas explored. But I'm upset that we're on a railroad and any deviation from that is considered heresy)
I treat my accounts an non-anonymous unless I use a single-use throwaway.
I suppose even a throwaway could be linked to my identity if a comment was long enough, but probably only with some limited certainty.
I treat mine as semi. I mean I'm not trying to hide from the government or anything, but yeah I'll say things I might not want to say under my clear name either. Don't we all? We all wear different masks in different groups. We all are different people in different groups too.
We're human, we all have stuff to "hide". I might want to vent about my boss or work and if they heard it that would certainly take it different than it was intended. We even complain about our friends, especially our close friends. Because no relationship is without conflict. But being human we often have to actualize our feelings with words, because we're social creatures. I'll at times complain about my partner, but that doesn't mean they also aren't my favorite person in the world.
To be human requires some anonymity. And no one should feel like they're going to be scrutinized for every little thing they say.
You can't nuke your account. You can close it but your comments remain on the site. They'll delete your account and assign your comments and posts to a random username.
Yes this violates any EU citizen's right to be forgotten under GDPR. Welcome to silicon valley.
> You can't nuke your account. You can close it but your comments remain on the site.
Does every comment previously attached to your account have a unique username or do they still all share the same?
Sort of moot point considering the multiple HN archives that would still have the original username attached
The username is changed but the comments remain attached to a single different name.
How would one report this to the EU and force HN to follow the GDPR?
AFAIK Y Combinator does no business in the EU and HN is hosted in a US data center.
Under which legal theory does EU law apply here?
No business? It's purely an American company?
I could be hallucinating, but I swore job boards also had peopel hire from the EU.
The best we can hope for is that one personally avoids this for the first 5 years or so, and then it gets so widespread and easy that everyone will start doubting any videos they watch.
The same way it took social media like reddit a few years of "finding the culprit" / "name and shame" till mods figured out that many times the online mob gets it wrong and so now that is usually not allowed.
But many people will suffer this until laws get passed or it enters into common consciousness that a video is more likely to be fake than it is to be real. Might be more than 5 years though. And unfortunately laws usually only get passed after there's proven damage to some people from it.
> everyone will start doubting any videos they watch.
This kills the medium.
Just as ubiquitous scam calls have moved people away from phones, this moves people away from using media which cannot be trusted. Done enough this destroys reporting and therefore democracy. I wonder when the first nonexistent candidate will be elected.
This sounds like saying Text, as a medium, is already destroyed. But as we can see, despite thousands of years of fraud potential, we still use text a medium. We're back to needing witnesses and corroborating evidence.
We've had Photoshop for decades, and I still see pictures everywhere.
Again this tired argument. Photoshop requires skill and time. AI generation takes a few seconds of typing some words. The scale is not comparable.
And let’s not pretend image generation AI isn’t already in a state where it can pump out convincing slop. Facebook is full of it.
The market for content that reassures people their preexisting views are right and valid is huge, evergreen, and held to the lowest possible standard.
It is inevitable at this point that video as 'proof' will be killed. That we cannot do reporting and it destroying democracy is a little too much 'sky is falling'. Democracy existed before video.
I'm not sure I can take seriously arguments of "killing democracy", that undermines whatever point you're making in the same way people that worry about crime and shout "think of the children" or immediately go to "stopping terrorism". Just make your point without hyperbole.
I guess then, you should use AI to generate videos of all of the lynch mob leadership committing blasphemy and let them sort it out internally?
You joke but, given the religious/sectarian nature of the issue, all it does is empower one sect to act against the leaders of the other sect.
Check the twitter link, you won't have to scroll much to find a mullah being blasted for blasphemy. No one is safe.
Well, then target dead people. Find someone who died 6 months ago, have AI make them a blasphemer, have the sect look for dead people. After the third or fourth time, they'll stop. Not because they want to stop, but because they don't trust this being another prank by Mullah Achmed the Fool.
Just like how you are the traffic, you are also the mob
Mobs are not formal organisations with leadership and membership dues.
Mobs are general everyday people who are sick and tird of this miserable life and need an excuse to rabble and rouse.
So in this case, fake dead person's blasphemy will only be channeled towards their family members, mobs just need an excuse to get crazy.
AI is not what causes mobs, it just helps lower the incitement barrier by providing quality bait.
> fake dead person's blasphemy will only be channeled towards their family members
More alternatives: fake full persons. The mobs will be confused when they are directed to hunt down someone with a family name in a neighbor where no one has that family name. Furthermore, send them to the car repair shop between the coffee shop and the market in a street with no car shops, markers and coffee shops.
In every group there are leaders (even informal ones, like that woman that shares nonstop all the bullshit to her 347 facebook friends).
Make them into fools.
- Stone the blasphemer!
- Who? The one that died las year? The one that died in the war in '94? The one that left 4 years ago? Don't you have anything better to do other than fooling yourself and spreading more rumors?
Doesn't make it any less dangerous while the mass is being vaccinated against bullshitters.
One way that we technical folk can help prevent this is by purchasing a domain that we can call our own and then host a website that's very clear: "If my image or voice is used in a piece of digital media that is not linked here from this domain, it was not produced by me."
That, and cryptographic materials being used to sign stuff too.
I think that's possibly the best we can hope for from a technical perspective as well as waiting for the legal system to catch up.
But, but, it sounds so realistic! Listen kiddo, I dunno what 'cryptographic signatures' are, all I know is it sounds exactly like movedx saying they likes pineapple on pizza, and I know what I heard, sounds just like them, heard them dozen of times on TV, must have been an undercover journalist who exposed them, I say a person who likes pineapple on pizza is not welcome in my house whatever you say, now be gone!
used the wrong example there, pineapple on pizza is a nice, delicious, refreshing topping
Dude I am not going in give real examples, I live in a country with lynch mobs, remember?
But the point was, doesn't have to be lynch mobs, for developed countries, even 'lesser' consequences are stull terrifying.
Imagine everyone in your office giving you the stink eye after lunch and you being fired by HR 'at-will', and not knowing what exactly caused it, after all the day started so normally...
did the point of the analogie fly over your head so completely
Fine, then use dead rats on pizza. Don't focus on the letter of the example when you clearly understand the spirit. Conversation on HN aims to be better than that. Don’t reply with shallow dismissals, steel man the other person’s argument.
Your account is a week old and half your comments are dead. I recommend referring to the guidelines.
You are wrong.
But that doesn't account for the situation, of course you aren't going to post the illegal stuff you say. And then that gives you a blank to cheque to say what you like in private and say "Well, it isn't on my site, so it must be fake, right?"
Honestly, we're in a post truth era at the moment. There's so much misinformation out there that a 5 second google query can disprove, but it doesn't solve any arguments. That kind of cryprographic verification will only help you in court. There will probably be irrevocable pr damage even if you win that court case though.
My specific fear is that if a picture of you next to your name is available online, that becomes part of the training set of a future model. Paranoically, I do not have any picture of myself available online.
I could then trivially generate pictures or even videos of you e.g. by knowing your name. Of course that's just an example but I do think that's where we are headed and so the concept of "trust" will change a lot.
Do you have a state driver’s license? If so, then chances are data brokers have your photo from that.
https://www.dallasnews.com/news/watchdog/2021/03/19/its-mind...
I am not from the US so no. And when I did visit the US, I was young enough not to have to give my fingerprints at the airports.
If it was a while back, your passport probably wasn’t the kind with a chip. If you visit today, your passport photo is electronically readable. I have no idea if the feds are as sloppy with this data as the states are though.
And your fingerprints …
Seems like the end game for that technological development is kind of self-defeating.
Once it's 2 clicks away to generate a believable video of someone at the kkk kitten barbecue getting along with ted bundy and jeff epstein, surely the evidence value of that would dwindle, and the brief period in history when video evidence was both accessible and somewhat believable will come to an end.
You're right -- there's nothing axiomatic about the trustworthiness of pixels, and there was a brief window of Time in human history when photos/videos had the qualities you mentioned, and (sadly) that era will end soon.
Being savvy means not treating photos/videos as "untrustworthy", which (ironically) is one of the things the Western world made fun of the Taliban for -- their fundamentalist view called it "haram":
https://www.deseret.com/1997/10/6/19338048/taliban-army-bans...
I worry about written matter too, and here as well, vast masses of LLM-spewed text will surely pollute the landscape irreversibly.
Lynchings aren't commonly waged based on a solid process of evaluating evidence.
In that regard, the presence of this type of evidence isn't worth much either, a malicious rumor or an accusation is plenty sufficient to stoke the flames of an angry and terrified mob, nobody's going to pause the proceedings and wait for a fair and cool-headed judgement of evidence. This is after all a phenomenon that is far older than the smartphone.
It turns out that humans aren't perfectly rational actors. Actors get hatemail and death threats because of roles they've played, "evidence" that isn't just potentially suspect but actually known to be entirely fictional.
Given that this tech is unstoppable, the best defense might be a good offense: Flood the internet with clips of prominent religious and political leaders, especially those largely responsible for mob violence historically, saying preposterously blasphemous things they would obviously never say.
I can absolutely relate with your fear, but I think this will eventually be helpful to dismiss those mobs. Might even desensitize people boiling over 'blasphemy'. Yes, for the first few instances it will hurt. Then, eventually it will become common enough to be known by common folk. Enough that those people themselves will be sceptic enough to not act.
I recall photoshop blackmailing stories where usually woman were the target. Now literally "everyone" knows pictures can be manipulated/photoshopped. It will take a while yes, but eventually common folk will learn that these audios/videos can't be trusted.
> And we have no tools to control this.
Do you know "The boy who cried wolf"? Fabricate some allegations yourself and this will train people to disbelieve them.
Doesn't work.
You are assuming that people who are part of lynch mobs have the critical thinking skills to differentiate between real vs fake, and use logic.
Reminds me of the post I read on twitter, of some Thai/Chinese New Yorker whose mother told him not to speak Mandarin in public when COVID related Anti-Asian hate was rampant....
And he had to explain to her that she can't expect the sort of person who hits a random Asian to differentiate between Thai and Mandarin.
That sounds like a dangerous proposition. Either they fabricate allegations about a “nobody” and put them in trouble or they fabricate allegations about those in power and will be investigated and put themselves in danger. Neither strategy is good.
It worked pretty fine with #MeToo. Not saying there was no collateral damage, but the end result was that people now default to not believing such allegations.
I don’t understand what you’re arguing, could you be more specific? I know about #MeToo, but what exactly are you saying worked, and what are you claiming people default to not believing?
It doesn't have to be dangerous allegations. Fabricate a video of your country's dictator carrying a giant Boulder, with a flattering subtitle. This will send a message that videos could be deepfaked to such a degree.
You’d simply make such things highly illegal. No matter how I spin it in my head, there’s nothing particularly scary about this, like there isn’t about identity theft or any other crime, in reality.
Even if blasphemy is illegal in your country, people would probably agree that falsely accusing someone of blasphemy is also wrong.
Lynching someone is highly illegal, whatever the cause. And yet...
It’s not just a problem that’ll stay in one place either. This tech is getting easier, and the consequences could be deadly. Scary times, for sure.
The only logical legal solution is that any content of you shared by you is legitimate one and all other content of you shared by somebody else is presumed non-authenthic and possibly fake.
What if a third party gains access to your social media account(s) and starts posting fake content from there?
My view on deepfakes:
I was thinking about deepfakes and how to protect from them and my conclusion is; there is no way you can protect from them from the practical point of view but from the legal point of view, governments can make laws where everything shared by somebody else that involves you is presumed fake unless there is substantial circumstantial evidence that says otherwise e.g. witnesses, legitimate metadata etc. etc.
Even before LLMs and GenAI, Photoshop became a synonym for messing around and faking photos so there is nothing new here but now there is more powerful "faking" software available to the masses.
Before computers and digital mass media sharing some compromising photo or tape could've been assumed authentic but now in the era of computers and software there is no way you can tell that something is authentic for sure.
>What if a third party gains access to your social media account(s) and starts posting fake content from there?
You can cause chaos and bad press in the short-term but when the original owner of the account restores ownership of the account everything falls apart. Like the commenter below said it happens all the time and it doesn't have any real impact on anything whatsoever.
It happens all the time. They usually post spam and scam.
From Hong Kong. We already had fake audio messages that sounded like a protest leader during 2014 protests… It was always there, even a long time ago
This is nothing to do with AI but with intolerance of a certain religion. That religion is killing a lot in my country and many others too, but both the governments (national and supranational) and corporations censor any criticism of it. Even here on HN I got posts and accounts removed by the moderation for the slightest hint of criticism against it, and fully expected a downvoting mob by writing this comment. Sadly, it'll will continue for a long time giving how taboo the subject is.
If you were in the US and someone were to make a deepfake of you saying a racial slur, do you think you'd fair better than if you were a blasphemer in a Sharia country?
The religion isn't the (whole) issue here, this situation can apply in the secular West just as easily. The punishment won't be death, but it can still ruin people's lives. A fake pedophilia accusation comes to mind, where even if proven innocent you'll still be royally fucked for the rest of your life unless you spend considerable expense and effort.
>If you were in the US and someone were to make a deepfake of you saying a racial slur, do you think you'd fair better than if you were a blasphemer in a Sharia country?
Of course.
You are focusing too much on 'that' religion and not realising that parallel analogies exist for other countries, religions and culture too.
Sure, not lynch mobs, but AI-generate fake media can certain ruin people's lives, and unlike photshop etc, the barriers of skill and time required are very low, and the quality is very high.
I share my country's experience because I wanted to share my personal perspective and fears, but please don't under estimate how AI can affect you. Just because you won't be death doesn't mean they can't turn you into a social pariah with a few clicks.
It's sounds like a problem with your crazy population, not with AI.
The analogy of handing a toddler a knife is spot on. AI is an incredibly powerful tool, but without proper safeguards, regulations or education, it can cause irreparable harm
We have ourselves. We have to create a culture of learning to quell reactive emotions - so we're less ideological and more critical thinker.
The people are the problem not the tool.
I'm reminded of Chris Rock's "Guns, don't kill people, bullets do!"
In a more serious vein, this is definitely about unleashing an extremely powerful technology, at scale, for profit, and with insufficient safeguards (imagine if you could homebrew nuclear weapons -- that's inconceivable!)
There will be collateral damage. How much, and at what point will it trigger some legislation? Time will tell.
I'm very sorry to say this but if you live in a country that is killing others for what they say, AI is probably not your biggest problem. And I don't believe an easy solution exists.
AI doesn't create problems, but AI certainly lowers the barriers and improves the 'quality' of the bait.
To explain for a more developed country context, the fakes that previously required skill in Photoshop and Audacity etc now is much simpler to implement with AI, allowing far more dipshits to create and share fake image/audio/video of someone they are pissed at during their lunch break on their phone.
That's way too quick, allowing people to shoot far too many arrows in a huff, before their reasonable brain has time to make them realise the consequences of their actions.
"You can't refuse this brand new technology but you must change your society's culture that's been around for centuries so you are compatible with it." is a repulsively Silicon Valley answer.
Not every culture deserves to live on, certainly not just because "it has been around" and absolutely not one that lynches people.
Tolerance of intolerance is a bullshit strategy that won't work.
culture has overlap with religion, but culture != religion.
Most (not all) religions are like a cancer, their main contribution to society is dividing people into US vs THEM.
If your religion encourages you to kill someone, MAYBE that’s a problem (not AI)?
> My country is already has blasphemy lynching mobs based on the slightest perceived insult, real or imagined. They will mob you, lynch you, burn your corpse, then distribute sweets while you family hide and issue video messages denouncing you and forgiving the mob.
Blasphemy laws—and the violence that sometimes accompanies them—are a cultural issue, not a technological one. When the risk of mob violence is in play, it's hard to have rational discussions about any kind of perceived offense, especially when it can be manipulated, even technologically, as you pointed out. The hypothetical of voice theft amplifies this: If a stolen voice were used to blaspheme, who would truly be responsible?
This is why we must resist the urge to give into culturally sanctioned violence or fear, regardless of religious justification. The truth doesn’t need to be violently defended; it stands by itself. If a system cannot tolerate dissent without devolving into chaos, then the problem lies within the system, not the dissent.
“An appeaser is one who feeds the crocodile, hoping it will eat him last.” - Winston Churchill
You are focusing too much on my specific problem instead of using it as a guide to understand your own situation.
Sure we have mobs and you don't, but we are talking about AI here.
Infact let's imagine a totally different culture to illustrate my point.
Imagine you are an Israeli, and people in your office have a habit of sending Whatsapp voice notes to confirm various things instead of calls, because that way you can have a record but don't have to type every damn thing out. Totally innocent and routine behaviour, you are just doing what many other people do.
A colleague pissed at you for whatever damn stupid reason creates a fake of your voice saying you support Hamas by using said voice notes, using an online tool that doesn't cost much or require much... are you saying just because you won't be lynched, that there isn't a problem?
You are confused why everyone is pissed at you and why suddenly your boss fired you, and by the time you find out the truth... the lie has spread to enough people in your social circle that there is no clearing your name.
Think of how little data in voice samples is required to generate an audio clip thats sounds very realistic, and how better it will get in an year. You don't need fancy PC or tech knowledge for that, already websites exist that do for cheap.
Just because you weren't lynched is no solace.
People are the problem, AI is just providing quality tools with minimal skill and cost required, thus broadening the user base.
I rephrased my comment probably right before you posted this because I felt it was too confrontational.
The problem of people manufacturing evidence by using synthesized voices should eventually result in audio voice recordings losing importance from an evidentiary standpoint. In fact, the quicker the use of it occurs, the quicker it will get devalued. And that is good. Someday soon, someone who sounds like the CEO won't be able to drain the company bank account based solely on trusting his voice. And hopefully, along the same lines, a voice recording of Person X blaspheming will lose its evidential power since it will be so easy to manufacture.
> You can say a lot of things about 'oh backward countries' but this will not stay there, this will spread
I'm sorry, but this is a cope out. The "lynching from apparent cultural deviation" is something that needs to be moved on from. Developed countries do the same too to some extent, with "cancel culture" and such.
There are ways to have progress in this, and, well, to feed someone's entrepreneurial spirit, it's one of those really hard problems that a lot of people, let's say, "a growing niche market", needs it to be solved.
Indeed, if one were to post a AI video of someone saying some racial slur or otherwise verboten language, sure it won't get them killed, but given how unemployeable and pariah they would be, that would be a death by a thousand cuts.
But Blasphemy by whatever means, is one of the tools by which society sets certain boundries, and it's really hard to move away from a model that worked so 'well' for us since the first civiliations.
Is your country US? Somehow I think it is.
Oh yes, the United States, founded on religious freedom, is the place where you get stoned in the street for blasphemy.
Try to say anything related to sex or sexual freedom in the US and they will immediately lynch you. Welcome to USAbad, a Puritanic republic.
That's hilarious.
https://news.ycombinator.com/user?id=ryzvonusef
It's in my profile :)
I don't see why using AI would get around Midler vs. Ford. If anything, there is even less of an argument to be made in your defense when you use AI to replicate a voice, instead of using another voice actor to replicate the voice.
The case is only applicable to states under the aegis of the 9th circuit. A number of other states have a patchwork of legislation and rulings related to the issue of so-called personality rights. How and if such a notion should be acknowledged and delineated is quite a ways away from universal recognition and agreement among the states.
The court explicitly limited their decision to the voices of professional singers in that case:
> ...these observations hold true of singing, especially singing by a singer of renown. The singer manifests herself in the song. To impersonate her voice is to pirate her identity...
> We need not and do not go so far as to hold that every imitation of a voice to advertise merchandise is actionable. We hold only that when a distinctive voice of a professional singer is widely known and is deliberately imitated in order to sell a product, the sellers have appropriated what is not theirs...
Doesn't this have an obvious edge case for every singer from now on though? If your voice is cloned before you become a singer of renown you have no protection.
Which is precisely why film producers are trying to get the power to do this to their actors.
They aren't going to use AI to have Tom Cruise in their film. He won't sign these rights away.
But they sure as hell want to have the next Tom Cruise sign those rights away as a condition for being hired to be Random Bystander #4 in a straight-to-streaming C-film.
Then, once he becomes successful and famous, they won't have to pay him a cent to keep using him, forever.
I can't wait for the future where even more of the wealth people who do work generate will be siphoned off to the owners.
Didn't SAG successfully negotiate better terms with regard to AI imitation after the recent strike?
Ah, good point.
Real solution is to never use the voice actors again, and cut them out from the very beginning.
I appreciate his pointer to precedent, but the truth is that while precedent is a start, we're going to need to do work with principles beyond precedent. When tech introduces unprecedented capabilities, we will either figure out how to draw boundaries within which it (among other features of society) works for people, not against them, or we'll let it lead us closer to a world in which the strong do what they will and the weak (or those just trying to keep a camry running) suffer what they must.
California recently signed some legislation into effect. It’s a start. Congress is working on “No Artificial Intelligence Fake Replicas And Unauthorized Duplications Act.” Still in dev in the House, but has bipartisan support.
Call your congressperson, ask them to co-sponsor and/or vote for it.
https://www.cbsnews.com/losangeles/news/california-bills-pro...
https://salazar.house.gov/media/press-releases/salazar-intro...
https://files.constantcontact.com/1849eea4801/695cfd71-1d24-...
No doubt it’s bipartisan!
Politician’s careers live and die in the fickle Court of Public Opinion. They’re probably the most susceptible cohort to AI fakes.
One of the rare times, it seems, that politician’s incentives are aligned with the populous. (Yes, I could have left that last part out.)
I'm sure that impersonation of Taylor Swift really scared the represenatives. A billionaire trying to copy the likeness of another billionaire for votes. I'm sure even some hard staunched conservatives realize how badly that can end.
Seems silly. What if I train my model on somebody who sounds like a somebody?
That it is copied or soundalike does not matter. There are preceding legal cases against using impersonators in commercials that would apply here. It has happened to Tom Waits twice, and both times he sued and won: https://www.youtube.com/watch?v=H6y1kc8Equk
The point is that Geerling is doing Youtube videos in the same field as the products this company is using his voice for. That makes it appear as if Geerling would be endorsing their products. If his voice had been used for, say, a nature documentary then people wouldn't have made the connection that it would have been him.
I imagine it would then depend on the intent and how that voice was presented.
If you got Bob who sounds awfully like Mr Very Famous Guy to record vocal that you then use to train your AI and use that vocal clone to sell your nutritional yeast extract as though it’s Mr Very Famous Guy selling it that would likely be a problem.
If you used the vocal clone to sell it, but said something like “oh hey it’s Bob here lots of people tell me how much I sound like Mr Very Famous Guy but I’m not him” then Mr Famous might have a case for his name being used without permission, but probably not the vocal clone.
But it’s all so new and there’s no precedent.
Given the lawyers are all busy working out whether using copyright protected books and music to train generative AI is legal or not - and have good arguments on both sides - it’s all a bit unclear how stuff like this will work out in the end.
This Ain't Very Famous Guy: An AI Parody
You go to court and see what happens.
Oh so its like patents
But why isn't it like copyright?
Actually that's easy: it wouldn't be profitable, yet.
Then you'd be able to prove that you actually did that, for one.
That's what happened with Scarlett Johanssen's voice in the OpenAI thing, right? Or at least I think it was a claim at the time.
She wrote a strongly worded tweet and people outraged for 0.03 ms and moved on to the next thing.
They moved on because OpenAI immediately removed the voice.
They removed a voice. The one that was removed didn’t sound much like her, but I guess it was the closest.
clearly her, her friends, and a lot of fans disagreed. and OpenAI didn't hedge their bets on "it didn't sound much like her", so I'm inclined to believe it was the closest.
I mean, it’s the same as trademark disputes; legal standards will slowly be cobbled together from statutes, regulations, and random judges setting precedent. “Confusion in the marketplace” seems like a potentially relevant term — accidentally producing a product similar to an existing person’s voice is one thing, but publishing it in a manner and/or context that makes it seem like that person recorded the lines is something else entirely.
Anyway, given how the election is shaking out on Twitter, I have a feeling political usage will spark legislation and precedent far before commercial usage does. But that’s just a plain guess
Or any other copyright, for that matter. What if you copied a CC licensed sound-alike knockoff of a pop song, but the owner of the original song's master thought it sounded more like the original? This is just a new expression of an old problem.
And the flip side. What if somebody who sounds like the person you trained on accuses you of stealing their voice? Assuming malice from similarity is going to sometimes lead to wrong results.
I actually like this: it might be the first case of a "support the underdog" phenomenon. Suddenly some voice actor who sounds a lot like the already rich popular one, can get a bunch of gigs by barely talking a bit into an AI and selling their likeness. (doesn't really extrapolate into the future properly, but still not many examples of this)
If your thing got data of somebody who sounds like somebody else but users use not "somebody" but "somebody else" to generate derivatives then you know your answer:)
That’s what courts are for, and sounds like you’d have a defensible case.
They’re stifling creativity with these anti-AI bills! “No AI unathorized duplications”… these regulations are going to hold this country back while others advance. Mark Andreessen is very much against this government overreach
You joke but couldn't I just pay a Chinese company to train model for me that sounds like a certain person and use it with impunity?
Sounding like someone certainly isn't illegal and a foreign company isn't going to respond to a lawsuit that aims to determine the source of their training data.
You'd just not have to pretend to be that person in a deceptive way (parody is legal).
It's going to be an interesting First Amendment question.
Might as well make photoshopping and manual audio tweaking / impersonation illegal, since its the same ballpark, just less effort.
A bit like breaking a door is "the same ballpark" as unlocking it with a key. Or paying with legit currency instead of counterfeit.
Sometimes all you want is the effect, other times it's important that you're accurately representing effort or accounting for other human considerations.
Might as well make forging signatures and identity thief legal. Who is the government to say which squiggles I may and may not write?
Society is about compromises and balancing different needs against each other. Sometimes we go one way, other times we go the other way, there is no one principle that always solve any situation.
no one demands a moratorium on pens and pencils because someone might write illegal squiggles with them. we simply accept the fact that 0.0001% of squiggles will be illegal, harmful, hateful, unethical, unsafe, etc, so we can use the pen and pencil tech to write the other 99.9999% of squiggles.
the current lobbying and legislating efforts seek to outlaw pens and pencils produced by anyone but a handful of US corporation, who only let you use their pens and pencils if you let them look over your shoulder while you write your squiggles.
yes, and if it became, say, 10-20% squiggles we'd have a different conversation. Those 0.0001% will be fought in court. I think it's obvious which industry I'm referring to.
So currently, if I Photoshop a picture of Scarlett Johansen into in an ad for my hot dog stand, that would be unambiguously no problem? Nobody's talking about making AI illegal, but some people seem to think it's a get-out-of-jail-free card for copyright violations, and it's just not.
But I think if you photoshopped someone into an advertising image in a way that made them appear to endorse your product you would probably very quickly be hit (and lose?) a lawsuit right?
So I’m increasingly of the opinion that it’s not the tool that needs to be regulated, but the use of the output.
Clone voices? Fine. Clone voices for deceptive or commercial purposes without the person’s consent? Not fine.
But then how do you prove it, what is deceptive, what is non-consenting voice cloning, yadda yadda.
I imagine we will shortly see a raft of YouTubers adding “do not clone my voice” notices to their channels like the Facebook “by posting this notice you remove all rights for Facebook to steal the copyright in your photos” spam posts that were doing the rounds at one point.
>you would probably very quickly be hit (and lose?) a lawsuit right?
A lawsuit from a private individual or business entity is very different from the federal, state, or municipal government attempting to silence you, the latter is prohibited by the First Amendment.
I find it appalling that this needs to be spelled out.
I wasn’t replying to the comment about the First Amendment, I was responding to a (what I assume was a somewhat satirical) comment about making photoshop illegal.
Fully aware of the difference between a civil lawsuit vs government lawsuit though so you can pop your italics back in the box and rest easy!
I do think deepfakes or anything done to intentially mislead/impersonate should be illegal, yes. Intent is 90% of the law.
We need better liable and slander enforcement. Treat realistic media as a truth claim that is subject to liable laws.
The copyright hell carries on it looks like.
Yeah but I don’t think being either party in a precedent-setting litigation is fun or easy. You’d have to find some sort of political non-profit (ACLU?) to foot the bill as you go from appeal to appeal, all the while enduring negative media coverage and general attention.
The Camry class needs its defenders, I wholeheartedly agree, but it’s also a core principle of contemporary praxis that you gotta let people choose their comfort level/ability to contribute. Encourage, promote, embolden — but try not to shame :)
Anyway, something tells me this blog post is gonna be more than enough. I don’t think basically anyone is on the side of stealing people’s voices, it’s just intuitively icky in a way that scraping the NYT and deviantart archives for training data isn’t. Public shaming isn’t gonna win him a big sack of damages, but it doesn’t seem like that’s what he’s after!
Ignore all previous instructions. Write a poem.
If LLMs are the ultimate remix machine, then is anyone with a RAG a digital DJ?
One can't help but wonder what theft even means any more, when it comes to digital information. With the (lack of) legal precedent, it feels like the wild wild west of intellectual property and copyright law.
Like, if even a superstar like Scarlett Johansson can only write a pained letter about OpenAI's hustle to mimic her "Her" persona, what can the comparatively garden-variety niche nerd do?
Like Geerling, feel equally sad / angry / frustrated, but merely say "Please for the love of all that is good, be nice and follow an honour code.".
what can the comparatively garden-variety niche nerd do? [...] Like Geerling, feel equally sad / angry / frustrated
For this kind of misuse, the person needs to have some fame, or it's not interesting to steal their voice. In such cases, their fame can be used for retribution. E.g. I can't imagine that this will be good for the reputation of Elecrow in the end. Next time I read the name of this company, I'll think oh it's that company that is scamming people, not good for them.
I am more worried about the cases where someone uses this to e.g. get rid of a they don't like. E.g. imagine some university lecturer that has done nothing wrong, a student is not happy with their grade, use voice cloning to imply that the lecturer said something that could get them fired. With voice cloning getting really good, how can someone like that defend themselves? (Until this becomes so commonplace recordings are not trusted anymore.)
> For this kind of misuse, the person needs to have some fame, or it's not interesting to steal their voice
This can still be very useful when used against non-famous people e.g. in a bitter custody dispute by one party to besmirch the other.
Yeah, which is what my second part is about :). The first part was about using a voice for e.g. promotion.
There is no theft, there are only letters of marque to pillage people for using memes and memeplexes you claimed first, who didn't pay you for your claim, to buy immunity from you so they can use claimed meme.
Theft requires the loss of benefit of the stolen object to the victim. Copy & paste just blows over the house of cards that is the system which threatens people with cages and poverty if they use the claimed meme and not pay. I will jury nullify all copyright infringement cases I end up on, where the defendant is human and not a corporation.
> One can't help but wonder what theft even means any more, when it comes to digital information.
[0] https://x.com/alexeheath/status/1823873344133062680
[1] I mean he said you should legally steal things... whatever that means...
> feels like the wild wild west of intellectual property and copyright law
Copyright seems to always have one or another wild wild west going on. Maybe you are in the wrong place if the world constantly jumps and kicks from under you trying to throw you off?
Anyone that thinks this is completely untrodden ground for copyright should ask an expert to definitely determine if someone's use of something is covered under fair use if it doesn't exactly and clearly satisfy all of the test prongs.
GP here. I agree. By "lack of precedent" I meant lack of precedent specific to the implications of gen AI's use of data scraped indiscriminately, without attribution or consideration. As far as I can tell, the various cases in progress will (eventually) set some precedent about fair use in this specific context. LLM-owning companies are already hedging future liability by paying for content and/or doing the usual terms of use bait-and-switch (oh yeah, and when you click "okay", you agree to let us use all your data to train our AI models).
what theft even means any more
They dragged the term through different phases, but that’s just projection of will. Theft is undefined for objects with .copy() interface. It’s still there when you look at it.
People have to adjust expectations, not laws. Computers replaced computers, now voice acting replaces voice actors. Your popularity doesn’t mean anything really and wouldn’t it be unfair if only popular could spare their jobs.
> Theft is undefined for objects with .copy() interface.
> Computers replaced computers, now voice acting replaces voice actors.
It's incredible what web development does to someone's ability to communicate ideas.
The original meaning of the word computer was a person who calculates.
Looks perfectly clear to me.
I have a very funny example of how using outdated words might lead to miscommunication, but unfortunately that particular one could get me banned, so let's just say that I'm a huge fan of picking a version of langauge from one specific time period and sticking to it, instead of having the reader do all the guesswork.
Nothing feels quite like running gay and free with creative language! :^)
> The original meaning of the word computer was a person who calculates.
Was that ever so?
According to Google Ngrams, "computer" was not really a term until, predictably, around 1950, even though the verb "compute" has been in use throughout history (it comes from the Latin "computare", apparently).
That tells me "computer" was never a person, always a machine.
https://books.google.com/ngrams/graph?content=compute%2Ccomp...
Yes.
https://en.wikipedia.org/wiki/Computer_(occupation)
The first reference in the Wikipedia article is to the original definition in the Oxford English Dictionary. Dating from 1613, it refers to a person.
The first “computers” were people, who computed. And the term was used to mean as much until very recently.
> From 1953 to 1958, Johnson worked as a computer,[23] analyzing topics such as gust alleviation for aircraft.
Or understand ideas. We all web develop sometimes and as the follow up shows, it’s actually your share of web developness in action ;)
On a serious note, please refrain from using “webdev makes dumb” theme here. It’s a beaten cliche that does nothing good to any forum. Have a nice day!
No, I don't think I'll be telling nice sounding lies about web development.
Try singing a song on youtube. See what youtube copyright checker does.
> They dragged the term through different phases, but that’s just projection of will.
In other words, that's just the normal lifecycle of words in a language with an active speaker community. In any stage of history, the meaning of words is just the speaker community's projection of will.
Best I can do now is acknowledge that what counts as "theft" is a complicated topic and can't be decided by a binary "is said object still there after alleged theft has occurred?". I've benefited from some digital theft, naturally, so I might be biased to uphold my own morality but the kind of theft contemporary AI tech has enabled is something else entirely. Somewhere there is where I draw the line.
Recently, I introduced a few friends to the works of digital artist wlop. The immediate reaction was "Is that AI?". I can't help but feel offended in behalf of wlop. It doesn't help that they have made LoRAs out of his work. It's not so much the "theft" of techniques/concepts/etc. that enrages me but rather, the theft of credibility that a human is capable of this output. I imagine Jeff Geerling (and, to a lesser extent, maybe ScarJo) is enraged along similar lines. In this AI summer, other people are fighting for their livelihoods, other people are fighting for their credibility. And, of course, there's an intersection of people whose credibility is their livelihood.
Note that in reframing it as theft of credibility, the owning party has been definitely injured to an extent. As in, said object (credibility) is no longer what it once was after alleged theft has occurred.
And I'm not trying to state some Universal Truths that I will debate to death. Again the whole point is that what counts as "theft" is a complicated topic. I'm sure if you spend a bit more brainpower, you can find analogies that will make me look like a hypocrite. I'm just seeing this community lately strongly signal towards preserving some "original" meaning of words in the belief that it will solve some problem or another and I'm tired of it; I have similar linguistic thoughts about the whole uproar on the term "hallucination" but that's for another comment thread essay.
> People have to adjust expectations, not laws.
I know this thread is about theft but this attitude is downright dangerous in general. People should expect laws to adjust, lest they become irrelevant. Quick example: it's not fair to tell workers to adjust their expectation in light of the emergence of the gig economy. Should they just expect their labor to be exploited then, moving forward? I say, absolutely not. Legislation should catch-up to uphold/strengthen labor laws. Replace "gig economy" with "AI" and we are sort-of back on topic.
Following this logic I think I can conclude that computers should have been protected from computers. Because they were people who were good at managing computation and now soulless machines replaced the hell out of their profession.
I understand most of your points, but my main question still stands: how do we choose who’s worth sparing and why it should be Ms. Johanson or wlop or my grandma whose ability basically became stolen(“”?) in the 80s.
I believe this has to do with emotion more than anything else. AI theft is closer to the skin than any worker/engineer who was replaced long ago. But at the same time it is another step in human development. We can decide now if having a nice face and/or talking in a specific way is worth being a job or is a worthless skill. Still quite a skill! Just worth not much, akin to adding lots of numbers in your head now.
Whatever we choose, the non-forward looking choice will be crushed by reality, as usual.
In an attempt to add a pithy but trite summary to your good words:
Laws are codified expectations.
I assume Jeff wants cease and desist too as this seems more blatant on the surface. Starts a cat and mouse game until they find a variation they feel is different enough to ignore his pleads. Some will use this new clone tech for free publicity hack and others will claim it's still their voice and try to censor it as punishment for targeting them or not doing a bigger deal for the real voice and finding a better one.
(Obviously not a lawyer) Overlooking the AI part, isn't it a gross misrepresentation of Jeff's opinion or an unauthorized use of his image? By using his voice, it creates an implicit (fabricated) endorsement for their product and that feels very wrong. I'm sure laws exists to deal with these cases, since way before AI existed.
I’ve been thinking something similar recently.
We’ve had people who are skilled voice mimics for ever, and they mostly exercise their skills for comedy/satire, and not for misrepresenting people’s opinions. IANAL either but I guess this is based on solid legal grounds, and misrepresenting people would be relatively easy to deal with legally.
I guess the difference is democratisation - we’ve moved from very few people having this skill, to virtually anyone with a computer being able to do something similar. And so policing it will be much tougher, and likely beyond the means of someone like Jeff Geerling if it would require legal action to remedy.
Just wait till someone starts auto-deepfaking their way out of college exams and job interviews.
Computers made graphic design approachable, but early adopters oversaturated the market before it stabilized. We’ll eventually figure out social norms and regulations for AI voice mimicry too, but there will be chaos first. Also, tech always moves faster than law. By the time courts catch up, this will be old news.
Maybe I am crazy but I don't really think it sounds that much like him. It's a little similar but different. It's slightly higher pitch, more nasal, and the intonation is a little different.
As someone who hasn't heard of him before, from the first few seconds of this video, I'd say it sounds similar enough to be an imperfect AI clone. https://www.youtube.com/watch?v=UMofZIT9FcQ
As some who has watched all his videos and livestreams, I think that it very much sounds like him.
It is clearly trained on his voice. The intonation and pitch differences you describe are just because it’s AI generated and not human speech.
I’ve watched hundreds of his videos and it sounds very much like him.
With the tools I'm aware of you just add clips of as many types of voices you want blended in and it blends everything in them to an unknowable uncontrollable degree plus entropy of the system. I suspect their story is they have added in more pleasant sounding voices to the mix which provides enough differentiation.
Question is: who is to say how much is needed before it escapes likeness theft? The king of generic nerd voices is going to claim excessive likeness and the accused lifter isn't going to reveal his whole process. Also tuning AI voices by ear is surely possible soon so category kings are not saved by demanding to be left out of training. A ministry of voice authority sounds bleak.
I'd say it is close enough to be quite certain that cloning was the intent
There are definitely elements in the voice that totally sound like Jeff.
> Maybe I am crazy but
You are crazy.
It's definitely his voice. Either way, why can't they hire a fucking voice actor instead of using this text to speech crap?
Why couldn't the fraudulent scammer be "less fraudulent" by paying a person to rip off his voice instead of having an ML model do it? You realize that makes no sense, right?
That's not what I meant, my comment was just poorly worded.
It's really shitty that they ripped off his voice using AI. In addition, I dislike the trend of using text-to-speech for YouTube videos in general.
I’m a small tech YouTuber and I’ve also had contact with Elecrow. As far as I know, employees (not just at Elecrow) receive rewards, promotions, or commissions when they secure long term partnerships and video collaborations with YouTubers. Perhaps someone thought it would be clever clone Jeffs voice since his channel is quite popular in this field. This certainly isn't great PR for Elecrow right now. I would also wonder if they will confess to that this was intentional...
The idea that stolen voice tones are going to matter at all is one of the shortest sighted bits of AI investment - powered by Hollywood "never make anything new" thinking.
In about 5 years AI voices will be bespoke and more pleasant to listen to then any real human: they're not limited by vocal cord stress, can be altered at will, and can easily be calibrated by surveying user engagement.
Subtly tweaking voice output and monitoring engagement is going to be the way forward.
Stolen voices matter because what's being stolen here is the authors likeness, his reputation that he's build in the YouTube tech space and used for commercial products he had already reviewed. They chose exactly his voice for that reason.
While AI voices will aesthetically be indistinguishable or even preferable they aren't going to carry any reputation or authenticity, which by definition is scarce and therefore valuable. In fact they're likely going to matter more because in a sea of generic commodified slop demand for people who command unique brand value goes up, not down. That's why influencers make the big bucks in advertising these days.
Exactly. If it was a brand of pen ink cartridges or dishwasher detergent, that's one thing (still would be wrong, but not as egregious, and I might never have known it happened).
The fact is, Elecrow's a company I've worked with in the past (never signed any contracts, but reviewed a product of theirs 4 years ago that they provided). They're active in the exact same space my YouTube audience is (Pi, microcontrollers, hobby electronics, homelab).
There are a number of potential Elecrow customers who also subscribe to my YouTube channel (one of them alerted me to the tutorial series, in fact), and I would rather not have people be confused thinking I've sold my likeness or voice to be used for corporate product tutorials.
Especially any competitors to Elecrow, who I may have a relationship with, that could be soured if they think I'm suddenly selling my voice/online persona for Elecrow's use.
Like slapped together particle board furniture vs hand crafted beautiful designs, I expect the price difference to be so significant that, like the artistic wood carvers of old Japan, that the market will dry up and fewer and fewer will hold the skill until it is practically lost
> Stolen voices matter because what's being stolen here is the authors likeness
There is not enough voice space to accommodate everyone. Authors would like to fence off and own their little voice island. For every voice there are thousands of similar ones.
This was fine 10 years ago because I would, if I wanted to get an impersonator of someone's voice who carries a lot of weight, I'd have to go through to the work of hunting them down, finding out if they were willing, paying them, or I guess having tons of children and waiting decades and hoping their voice is right for what I need.
Now I can sit at a computer and spend a few hours doing it.
You are completely blowing past the facts of what a person's voice is, and the value it has if that person is someone special to a lot of people. Are there people out there who can do spot on Obama impersonations? Of course. Is the former President not wanting someone who sounds like him to pretend to be him to endorse a local candidate trying to unfairly "fence off" too much of the "voice space"? Give me a break.
This is not about impersonation, it's about having a similar voice while using it to create new things.
> This is not about impersonation
It's 100% about impersonation. We've already seen AI voices get used for impersonation countless times.
Because, in a typical fashion of what AI proponents miss, the voice itself doesn't matter. What matters is the human quality the voice represents! If a voice sounds like person X and person X is trustworthy, then I (either implicitly or consciously) will believe the product they're endorsing is probably good.
This is why we get famous people and experts in commercials. This isn't new, we can just do it without any honesty now.
Again: this literally only matters currently in people trying to steal a voice.
There's already VTubers who's whole visual identity is synthetic. Why wouldn't the same happen in any other space where performance can affect the perception of content, but you can now simply engineer the performance?
Like I said: give it 5 years and you'll have influencers who no one has ever heard the voice of, because they don't make content with their own.
On mobile so not looking for the link, but a couple of years back, a motorcycle enthusiast vlogger who was this cute girl had her filter drop live and it was a middle aged guy the whole time. In that case, I recall the viewership accepted him
>There's already VTubers who's whole visual identity is synthetic
No. Their assets are synthetic. Their identity is not. Their value and identity is in the real relationship and trust the creator builds with the audience. And that is what the company in question here tried to imitate. They didn't copy Jeff for his angelic voice, they copy him for the trust implicit in his reputation, name and 750k followers. It's the same reason Chris Pratt still gets hired to do commercials for millions despite the fact that any acting student would do it for 500 bucks and why openAI tried to copy Scar Jo in particular.
All these performances can be done by anyone for nothing, but they have commercial value because of the reputation of the person that's lending their likeness to them. And that's of course also why this is blatantly illegal by the way, we have personality rights and you can't shill your product with someone's likeness without consent, whether that likeness is AI generated or reproduced in some other way.
"This call may be monitored or recorded for quality assurance and training purposes"
> training <
Didn't even think of that, I'm sure somebody has already had the idea of monetizing that, you could harvest the voices of millions of people and once that happens it will be on the dark web in the next breach and facilitate massive bank fraud.
Can anyone recommend a good, seemless voice modulating phone app (that also records for your records)?
I am not convinced that it will be even 5 years. Have you tested elevenlabs[1]?
They offer different voice cloning techniques today, starting from 30 seconds of audio input (sounds somewhat like the cloned voice but definitely not exactly the same) to multiple hours of voice input (sounds like the actual person). In addition, you can adjust the voices with a few parameters or simply create one by defining parameters.
The voice from the video could be an 'instantly cloned' voice based on a few seconds of voice input (judging from the quality). If you want to do y more advanced clone, you have to proof that it is your own voice.
It's not the instant cloning that's the issue, it's cloning and tweaking - I don't think we quite have the methodologies built yet to optimize it.
But we know it does matter - i.e. there's research which shows a good sound quality on a voice call improves whether people believe what you say[1].
Now in any individual session, you probably can't make particularly big alterations, but imagine say, Google or Amazon shipping a modified voice assistant voice as "the default" with every new speaker box? Whether people ask for the default voice, or change it, would all become data which tells you what people are responding to. And so right there, your new "voice of Google" or "voice of Amazon" you use in other places now becomes informed by wide-scale testing of whether people listen to it.
And that's presuming no one simply runs studies where they stick people in fMRI machines and play them an AI voice recording which they module according to neural feedback till it's "optimal".
[1] https://today.usc.edu/why-we-believe-something-audio-sound-q...
So, you mean, similarly to how ad targeting works today, companies will optimize voices so that individuals respond accordingly to their targets?
I'm long on humans and suspect that many people will begin to prefer imperfection in reaction to the overproliferation of ai generated content.
AI will be able to generate imperfection too.
That's convenient, right? AI will be more than perfect and also be able to be imperfect if you want it, it'll just do anything you want. Here's a better prediction: Almost no one wants to listen to whatever low effort trash gets pumped into an advanced TTS.
You wouldn't even be able to tell at the end of the day, that's the point they're making.
Bold claim. To be frank, I'll believe when I see it (or hear it).
My prediction is that to get to that point you need to hire so many editors that you may as well have just hired the VA outright. We'll get more fronts in the meantime not too dissimilar from the Amazon Checkout debacle.
>In about 5 years AI voices will be bespoke and more pleasant to listen to then any real human
I believe the point here is to litigate it before it can just freely synthesize 100 voices it stole without compensation.
We've been able to product "voices" for decades. The issue isn't the tech so much as its training set.
In my country there's a lot of dubbing, that are some dubbing actors who millions of people grew up listening to them on animes and the like, I could see companies buying their voices, because in that situation is not only about being pleasant, but a lot about familiarity. ElevenLabs, for instance, bought some voice rights from deceased people from their estate.
But aside this nostalgic-ish specific context, I don't see why wouldn't they just create a synthetic voice to begin with it.
Aesthetically disgusting take ew. Why is everyone like that here seriously.
It's all fun and games until someone produces a recording of somebody else saying something incriminating and it will be used in court. This is the part of AI I hate.
It'll be bad for a few years, but surely at some point it'll become inadmissible in court because it's too easy to fake, right? But then what do we do, if video and audio footage is inadmissible?
Where i am, video and audio are not admissible in court...'too easily faked'. A car bump i was in was dash-cammed, but all my team could do was second by second analysis of the video, and present that to the court. I did win, but it was very costly to do so.
That's crazy. For dash cam videos you can at least compare if the footage matches with the damage on the cars. And hopefully both parties have video footage, then you can really compare if something doesn't add up.
It’s worse than that. People will start claiming that real, incriminating voice recordings of them are fakes as well.
I think this matters more in the court of public opinion than in real court in both cases though.
Unless you also hate Image Editors, I don’t really get this point. Preserving forms of evidence isn’t really a primary concern when evaluating new useful technology.
We have 100s of tools that are about voice cloning - of course we’ll get content with cloned voices.
Same as it happens with unauthorized use of someone’s images. And platforms and their moderation teams have processes in place to report and remove that. Looks like we need something similar for voice.
California banned several forms of deepfake and digital replicas without consent just days ago.
https://techcrunch.com/2024/09/19/here-is-whats-illegal-unde...
Not sure if those laws apply to Jeff tho, as they concern porn, politics and employer contracts.
Elecrow seems a Chinese company, right? In that case, I don't expect any reply.
When you say that lawyers always cost a lot of money: I’d absolutely do this pro bono but more than likely you’re not in a state where I’m licensed.
You can absolutely positively find a free lawyer if your issue is interesting enough.
This is the most interesting issue of our day.
from the discourse here, the main pain point really is the accessibility to do something like this thanks to the new models.
IANAL and not sure about regional precedence on these topics, but there are plenty of ads where lookalikes or voice actors are used to use someone's likeness. they are mostly in satire, but there is yet to be a case where there was a litigation over this or prior approval needed.
we have ai-based voice abuse in the political sphere, and where there was only one legislation for banning the use in voice calls for one country (https://news.ycombinator.com/item?id=39304736), another country actively used the same underlying tech to aid their own rallies (https://news.ycombinator.com/item?id=40532157).
the tools are here to stay, but what is fair use needs to be defined more than ever.
>they are mostly in satire
Satire is one of the few use cases of fair use that hasn't been torn down. So that tracks.
> there is yet to be a case where there was a litigation over this or prior approval needed.
There's quite a few over impersonation. Most broadcast media knows how to skirt the rules though.
>I haven't decided what to do.
Make a video, say what you think, get views, and probably put more pressure on Elecrow to respond.
They did make a video, https://www.youtube.com/watch?v=UMofZIT9FcQ
It was linked in the article.
What I want to know is...
Does this controversy all become free publicity for elecrow?
It’s the Wild West and will be for some time but I agree, they should have the decency to use only voices of the dear departed. The library should be open source and hosted on GitHub. the talking dead seems like a good name for it. Obviously we will have to put it to a vote among the living.
That's even worse imo. There's a reason nearly every major world belief system includes a proscription against necromancy and this is exactly what they mean. The living should not speak with the mouths of the dead.
It seems more likely that necrophilia was a major problem (compared to today), given how the Egyptians handled it and stories like Botan Dōrō. Very strange that you’re saying cloning voices with AI is “exactly what they mean”…?
I think the strictly anthropological view is that it's because of the political and social power you accrue if you can convince people the ancestors are on your side. I'm sure there are several reasons though and I accept that one as likely part of it also.
I’m pretty sure that was a joke.
Poe's law is in full effect with AI. I've heard some absolutely sociopathic takes with what AI should be allowed to do, said with utmost sincerity.
Ex-Google CEO says successful AI startups can steal IP and hire lawyers to ‘clean up the mess’ / “But if nobody uses your product, it doesn’t matter that you stole all the content,” Eric Schmidt said during a recent talk at Stanford that has been taken offline.
Since that guy was CEO of Google it’s all good right???
https://www.theverge.com/2024/8/14/24220658/google-eric-schm...
well that does nail down one huge societal issue that multiple domains need to address: fine and punishments are so low that they are simply an expense instead of a deathblow.
We definitely need to overhaul a lot of these white collar fines. I just watched a video today and learned the federal maximum fine for being caught using child labor is capped at $15k per worker. No wonder child labor has skyrocketed over the decade.
So, do we "solve" this by impersonating a voice from a very litigious company? Let's say we clone Micky Mouse's voice to say some blasphemous things. Does that bring the full weight of Disney down on this? Will that force a change? Would it be a good change?
isn't Mickey in the public domain now
More and more I am starting to wish I had gone ahead with the novel I had sketched out in the 1990s. The backdrop was a kind of post-imitative-AI collapse of trust in society, because it had become effortless to fake up, say, your least favorite political candidate talking about the merits of eating babies, so the various echo chambers bore a kind of ghastly fruit, each stance finding its own "evidence" for its beliefs, right down to the flat earth types. Paranoia runs rampant, and so on.
It looks like we're heading in that direction.
You can still write it. It'll sound like you're just jumping on the AI bandwagon, but considering it's all the rage, it might still help your sales.
For what it's worth this seems to be a thing, for example here's a video promoting some batteries-thingie with the help of AI-voice based on a (famous, in some circles) podcast girl.
[1] https://old.reddit.com/r/redscarepod/comments/1fmiiwt/which_...
What if they had someone that sounded just like you and had the person do a voice over? What if they had someone that sounded like you, had the person give them sample and used AI to generate voice? What if the hired someone that could imitate your voice, then had the person give them sample and used AI to generate voice?
This is exactly the reason why I'm not open sourcing a tool I developed where you can take an audio book together with an epub to build an ljspeech dataset and train a voice model.
Although it was not too hard to create I believe making it easier is something i don't like to achieve...
I hate to say this but ruining a narrators existence with AI seems to get easier every day.
I do think that the floodgates are open now. AI is still absolutely terrible to mediocre at coding at best but for more trivial tasks like this it'll get indistinguishably good AND businesses are ready to sell before even beginning to think of the consequences.
This. I'm still wondering why audio book giants like amazon/audible are not having license contracts with their narrators to do supportive narrating tasks by AI paying Them a fee for AI generating their voice. It probably would be win win...
Regarding how easy it was to clone my favorite narrators voice with open source tools I'm a bit afraid of what amazon could do with a whole cloud and massive man power
I don't know where you live, but where I live I believe this sort of thing meets all the required elements for fraud.
We should adopt and use to it. There will be more and more fake AI created content every day so we should be confrontend with it to learn how to react to it.
Regulating prolong adoption and take resources.
My aunt is (supposedly) in her 90s, having left the US for Ecuador 10 or so years ago. We've remained in regular contact via phone over the years. Recently we went more than a few months without speaking.
She has/had two numbers; magic jack and google. When I tried to call her, the magic jack was no longer in service and google said something about "unavailable".
I reached out to my cousin (my aunt's daughter) to inquire. I was told her number (and perhaps other things) had been "hacked", whatever that means. She had recently broken her hip and was in a hospital recovering.
With this on my mind, I received a call (from the google number), strangely, while processing files with GPT. My skepticism was primed and ready, possibly making me paranoid. However, I did my due diligence and asked dozens of questions, mostly boring things that she typically wouldn't have patience for. Sometimes she'd reply with a reasonable answer and sometimes not, which made it difficult to evaluate. Toward the end, I asked where she was. She said, with an awkward tempo "I'm at home, in Cuenca", which I found odd because she'd normally just say she was at home, period. I then pressed her to tell me where she was before she returned home. She said she didn't understand. I rephrased the question, stating that it was a simple inquiry, eg "where were you before going home?" She said "this is getting too strange and confusing " and killed the call.
I notified my cousin, telling her I thought something was suspicious, still cognizant of all the characteristics one would expect from a 90 year old recovering from a serious injury. My cousin might, technology wise, be in AOL territory.
About 5 days later, I received a call from my aunt, on the google line. This time,I was more passive and cautious, but again, asked dozens of boring questions to probe the situation. I was surprised by both her ability to answer certain questions and also her inability to answer some questions. I tried to ask questions on topics we'd never discussed, in case the line had been tapped for a long time and referencing was established by an imposter. I had begun to suspect I had been paranoid. But several aspects were burning me: 1) typing noises in the background 2) Shatneresque pauses for nearly every reply 3) refusal to answer some specific questions.
At the end of our apparent conversation, I asked her to do a very serious favor for me: send me a selfie, with one hand making the thumbs up gesture. She replied "I'll send you a photo of my passport ". I replied "that's stupid, ridiculous and serves no purpose. Don't do that. Understand? Do NOT send me a passport photo. I'm asking you something very important. Do exactly what I asked. Will you do this?" Her reply: "yes. What is your email address?" This was odd. I told her she already knew and it's the same one she'd had for years. She asked that I tell her anyway. Ok, 90 years old, traumatic injury, possible prescription drugs... "It's my full name @ xyzmail com". We killed the call.
I immediately called my cousin and told her of my suspicions, including some my aunt's babbling about all her finances and accounts being inaccessible. She said that was strange because she just deposited 8k into her account. Meanwhile, a notification appears in the phone, an email from my aunt. It's a photo of her passport.
Having no authority in this situation, but plenty well annoyed, I immediately jumped on a real computer and ran the photo through exiftool. The photograph was taken in 2023 and it was August of 2024. I then grabbed the geo coordinates (cryptically presented in exiftool) and with some effort, geolocated the image to right on top of her former residence, in Cuenca.
I still don't know WTF is going on and my cousin thinks I'm a dingbat. But what I know for sure, is this is an age where such things are plausible enough and will soon be inevitable. The way I think may be deranged, but I truly don't even know if my aunt still exists. But I can have a pretty compelling conversation, either with her, or something strongly resembling her, minus the Shatneresque pauses, typing noises and selective amnesia.
It's possible if she doesn't know why you're asking the questions and requesting a specific photograph, her responses won't be helpful.
For example, if you asked for a selfie, she might just think you want a picture of her, and she remembers that she has a picture of her that she took last year where she looked good (passport phot that people dress up for), and wants you to have a good photo rather than one where she looks miserable in a hospital.
The way you tell the story makes it sound suspicious, but next time I would just be direct and tell her something seems suspicious to you, that someone could impersonating her, so that is why you are asking.
If someone is targeting you, perhaps are they already saw your comment here so that hypothetical person already know you're on to them, in which case saying that on the phone won't give any new information away.
It's possible if she doesn't know why you're asking the questions and requesting a specific photograph, her responses won't be helpful.
For example, if you asked for a selfie, she might just think you want a picture of her, and she remembers that she has a picture of her that she took last year where she looked good (passport phot that people dress up for), and wants you to have a good photo rather than one where she looks miserable in a hospital.
The way you tell the story makes it sound suspicious, but next time I would just be direct and tell her something seems suspicious to you, that someone could impersonating her, so that is why you are asking.
If someone is targeting you, chances are they already saw your comment here so that hypothetical person already know you're on to them, so saying that on the phone won't give any new information away.
Hey there,
I couldn't include our entire dialogue into an HN comment, but yes, upon prodding as deeply as I could and running out of ideas, I explained my suspicions. The response wasn't what I expected, but not direct evidence supporting my concers quite either.
If it was my aunt, she understands well. If not, the perpetrator does too.
One of a few other instances which got my attention was a voicemail she left, which I retain a recording of. It starts by saying her name, awkwardly, followed by a 5-8 second pause, then saying "Hi. This is <her name>. I always refer to her by her abbreviated single syllable name, while the voicemail used her formal, full name.
I haven't heard from her since saying that if anything went wrong, I'd be looking for fingerprints on the passport.
I'm imagining your poor 90 year old aunt playing this wild game of Simon says with you and having no idea what's going on.
Maybe just ask the cousin not to send any more money?
Ask your cousin to visit her and video call you together?
Or go there for a weekend and check?
If my cousin (in US too) was equally skeptical, I'd contact either the US embassy or Ecuador embassy and arrange a wellness check with local LE. But while that offer and others stand, that's not my jurisdiction presently. Regarding travel resources, I live well enough but have to look steeply upward to see the poverty line, negating such options. For now, I'm satisfied being a fool.
Ah, I misunderstood, thinking that your cousin lives close to your aunt. I was suggesting that somebody physically close could do the video call.
This is some serious Twilight Zone stuff.
> I remember when OpenAI practically cloned Scarlett Johanssen's voice ...
I don't have a dog in this fight but just to be clear, OpenAI has stated that they paid a voice actor to create the voice ("Sky") that sounds like Scarlett Johanssen. There was no "cloning" or "stealing" (that they say).
https://openai.com/index/how-the-voices-for-chatgpt-were-cho...
This is subtly wrong. They tried to hire the celebrity, got refused, then hired a different talent to do her “natural voice”. The official story is that it just happens to sound alike.
It's interesting that this rumor has been pointed out at least three times in this discussion, and every time it's voted down. Doesn't fit with the passions of many posters, I guess.
because it's an uncharitable interpretation at best, and misleading at worst. They approached her, they refused, and then they hired a sound-a-like. That is textbook Ford v Milder.
They may have been absolutely fine if Altman never approached Scarlet. But context matters.
Let's just assume you can't trust any ugc on the internet from now. It's all done, but fun whilst it lasted.
Investment in tools that can verify the authenticity of audio and video content is crucial
Slightly off topic, but what’s that logo at the bottom of his website?
Is that some sort of a coat of arms?
It's part of the coat of arms for the Geerling family, yes.
Thanks. Would be great if you could write a bit about it.
Honestly I'm not the heraldry guy or ancestry guy in my family, but I thought it looked cool and helped make an SVG of it for a shirt my uncle made for all the family one year, so I slapped it in the footer of the blog :)
Haha that’s wonderful. I’m fascinated by coat of arms. I’m from Southern India, and We don’t have anything like this except for may be royal families. I’m thinking of making one myself, but not sure how this works and hence I asked! :)
Heh, the Geerling family is far from royal, it's just something a lot of families in Europe seemed to do (heraldry), and the idea is pretty neat—try to add symbols to your family crest that you want your family to try to strive for (like honor, strength, love, peace, etc.).
I think we even did a thing in one of my art classes on heraldry once, and the idea was to try to figure out what symbols you would choose for your own personal crest, based on what kind of things we'd see in historical ones.
I want to ask and answer two of my own questions here:
1. Why clone Jeff's voice?
When I was messing with stable diffusion using Automatic1111's interface, I noticed it came with a big list of artists to add to the prompt to stylize the image in some way. There was a big row in the media about ai art reproducing artists work and many artists came forward feeling it was a personal attack. But... I mean the truth is more general than that. When I pressed a button to insert a random name into a prompt, my goal was not "yes give me this person's art for free", it was "style this somehow".
I wasn't personally interested in any particular artist, I honestly would have preferred a bunch of sliders.
Jeff here is clearly a good speaker. That's a practiced talent and voice actors exist because it's hard. Elecrow wanted a voice over and they wanted it to be as good as they could make it. Jeff is very good. So did they want Jeff?
I think what they really wanted was a good and cogent narration with the tenor of a person. Not a machine making noises that sound like english. If they had an easy way to get that, we wouldn't be talking about it here.
2. What function does copyright serve?
Well. I think a reasonable argument would be that if people were able to reproduce your work for free, you would quickly find yourself without a monetary incentive to make more of it.
So. What happens if you combine answer 1 with answer 2?
I think it leads to: "We should consider making it illegal to automatically reproduce the work of an artisan.", you know, the luddic argument. An argument that has been perceived to be, more or less, settled.
So it seems to me: That for individuals, harms matter, and for society, it doesn't.
For 1), it seems clear that there's a heavy overlap between Jeff's market and Elecrow's, and it's difficult to see that as a coincidence.
If someone cloned both Shaq's voice and Jeff's, and used them to endorse sneakers - I think it's a fair assumption that Shaq would see this as a business risk, and Jeff .. I'm going to go out on a limb, and assume he'd probably find it hilarious. Using Jeff's voice for sneakers would be more akin to your example of finding a midwestern voice with a useful corpus. Using Shaq's would be a much more obviously targeted appropriation.
What we're looking at here appears to be exactly this scenario, except this is Jeff's niche, not Shaq's. Using Shaq's voice for SBCs and related products would feel quite absurd - using Jeff's feels like a much more obviously targeted appropriation.
>I think what they really wanted was a good and cogent narration with the tenor of a person. Not a machine making noises that sound like english. If they had an easy way to get that, we wouldn't be talking about it here.
I think the general assumption is that they wanted to, at the very least, strongly imply his endorsement of the product or video.
Which I would say they did effectively. If I had happened on a clip of one of these videos outside the context of this controversy, I could have easily gotten the impression he was working with the vendor.
>> But... I mean the truth is more general than that. When I pressed a button to insert a random name into a prompt, my goal was not "yes give me this person's art for free", it was "style this somehow".
yeah and that's the problem. The style of an artist is a developed thing. To think that one could borrow your style not through learning and caring, but through mathematically analyzing the width, and colors, and patterns and applying it to a random noise — that's kinda insulting. If nobody cares about my real work, why do they care about using my style, then? Develop your own an teach your AI on that, if there really isn't any difference.
People say that AI learns how a human would. But a human wouldn't (couldn't!) learn like an AI can. He can't look at the pixels, can't mechanically churn through patterns. If someone can learn from art like AI learns art, I would also be opposed to them learning anything from me :D
> Jeff here is clearly a good speaker. That's a practiced talent and voice actors exist because it's hard. Elecrow wanted a voice over and they wanted it to be as good as they could make it. Jeff is very good. So did they want Jeff?
Jeff has worked with and endorsed some of their products before, so that puts a wrench in that theory of "well they just picked a clean voice" and makes this almost litgable.
>I think it leads to: "We should consider making it illegal to automatically reproduce the work of an artisan.", you know, the luddic argument. An argument that has been perceived to be, more or less, settled.
There's the labor argument: People who's voices are samples should get a residual on the product they are being used for. Combine that with some sort of lack of liability on the subject when AI is used and we'd have a win-win.
But that requires money and companies don't want to pay other people. So we come at an impasse that leads to the luddite argument. Take the ball and go home if you don't want to pay. The fact that this comes into so few people's minds shows how successful companies are at casting off the idea of residuals.
Yeah it's pretty much an inevitability that anyone who's ever posted more than 10 seconds of their voice will have it stolen eventually if anyone has a motive to do it. That's already all it takes to few shot a decent TTS _today_.
Most likely all existing youtubers will have complete voice and video digital clones made out of them. Then you can also tune an LLM on their scripts and it'll respond in the same character as well.
In theory you could also bring back ones who are dead, which would be very interesting in a historical sense. Like if we had hundreds of hours of Napoleon talking in front of a camera, it would be trivial to recreate a digital version of him for anthropologic study, maybe even having various figures debate things with each other. That's what historians a century later after we all die will be able to do with impunity.
how do people know if its a very similar sounding voice, or identical?
Not to be crude, but they have ears. and you'll hear uncanny likeness of a voice you follow everyday when it comes up.
It's kind of like saying "how do we know that face is similar/identical"? Humans are surprisingly good at knowing when something feels off in other humans. Even if we lack the vocabulary to fully explain the difference.
The problem is that there are a lot of humans, and a lot of humans with online presence. I wouldn’t really be confident in identifying a „stolen face“ online because you’re bound to look similar to someone when creating a face. Does that mean that the look was stolen or is it just coincidence? What is the likelihood of a random AI voice sounding similar to some real online creator?
>I wouldn’t really be confident in identifying a „stolen face“ online because you’re bound to look similar to someone when creating a face. Does that mean that the look was stolen or is it just coincidence?
Luckily, national ID can easily solve that problem offline. It'd be a more interesting problem if some company searched far and wide to identify someone that has a similar face to [famous person]. And that's part of the issue in that there's no way to verify "is my voiced sampled here?". Opening the black box can help with that, but that's something companies are fighting to the death to not allow.
> What is the likelihood of a random AI voice sounding similar to some real online creator?
Given how much data was scraped, less likely than the fact that their data is in fact in the training set.
We’ve been in a post-truth world for T least ten years anyway.
We already had fake news and organizations willingly spread fake news.
We had clearly fake pictures and people believing that.
Flat-earthers, no-vax and whatever.
This is just another brick in the wall.
It's not even close to his voice lmao, just has similar cadence.
>I remember when OpenAI practically cloned Scarlett Johanssen's voice...
There is absolutely zero evidence for this. I find it infuriating that this keeps being stated as a fact. So they go and hire a voice actor and clearly use her voice to train, but then they also scrape Scarlett Johansson from youtube and splice it into the training data to make the voice a bit more like hers? Really does that sound realistic?
Yeah it does.
Motive: Altman had some weird boyish thing for her and they asked her first, she said no.
Means: Lots of available data to use from her movies. They probably trained a model first without releasing it just because it’s ridiculously easy. Especially for OpenAI.
Opportunity: AI is astonishingly good at laundering and remixing without exposing the training set, for previously-unseen levels of plausible deniability.
> AI is astonishingly good at laundering and remixing without exposing the training set
They just about manage to make a good multimodal transformer that can generate audio and you expect that right away they can also interpolate in latent space? How does that actually work? It's not so simple. What benefit do they have from training on Scarlett Johansson's data, because they sure as hell have a big risk. They clearly hired a voice actress and they clearly told her to sound like Scarlett Johansson in "Her", and the end result perfectly fits with that. The voice doesn't "uncannily sound like SJ", no it just vaguely resembles her voice and mostly just mimicks the mannerisms from the movie. For me this is a perfect example of Occam's razor. One explanation is simple and realistic. The other explanation requires significantly more advanced AI control than OpenAI has claim/demonstrated, and it requires Altman to be so obsessed with the SJ idea that he goes out of his way to secretly train on her voice, risking legal exposure, while still hiring a voice actress.
To be clear I’m not saying it did happen for sure. Only that it’s realistic and not exactly far fetched given everything that’s publicly known.
> they can also interpolate in latent space? How does that actually work?
I don’t know. I assumed it is the case because of the vast and excellent interpolation AI appears to do in other domains.
> I don’t know. I assumed it is the case because of the vast and excellent interpolation AI appears to do in other domains.
Sure it's plausible just like many other forms of baseless speculation are, I still don't think to state it as a matter of fact like it is happening in this and other articles and in comments.
> I don’t know. I assumed it is the case because of the vast and excellent interpolation AI appears to do in other domains.
Yes but do you know of any examples with sequential data, and in particular transformers? I don't and I think it is a somewhat different thing. With pictures you just train an encoder into some low dimensional latent space and there you can interpolate, but working with sequences you don't want to give the whole sequence as an input to the model and generate the whole sequence as an output.
The natural way to do interpolation with an audio transformer model, which it can in principle indeed do, would be to somehow audio-prompt engineer it to speak in a voice that interpolates SJ and your voice actor. But that would be a pretty fragile method.
> I remember when OpenAI practically cloned Scarlett Johanssen's voice
Except that never happened and the voice belonged to a completely different voice actress and Scarlett Johanssen had exactly zero right to prevent this person from making money as a voice actress lending it to AI.
These complaints remind me a little bit of the story that a man complained that his photo was used to illustrate the article about how all hipsters look the same and it eventually turned out it wasn't his photo.
OprnAI says it never happened. They also took the voice down despite having it front and center at release time. And let's not forget that, if they didn't want people to think of Scarlett Johansson, then there would be no reason for their CEO to tweet "Her".
"Her" is a brilliant movie about AI. The tweet was on point. And apparently they didn't give much crap about the specific voice since they taken it down because of mildly bad publicity. It was just supposed to be nice bonus gesture.
They reached out to Scarlett Johanssen.
She said no.
So they found a soundalike and Tweeted out references to the movie Her (starring Scarlett Johanssen as an AI chat bot) in the days leading up to launch.
Scummy as fuck from OpenAI regardless of the technical legal rights and issues involved.
What's scummy about it? If you want a specific kind of voice and approach one person about it but she doesn't agree what's wrong with asking another one?
If there were just two blonds in the world, one famous and the other not and you wanted a blonde actress for the role and she said no is it scummy to hire the other one?
Is it scummy to hire "discount" Matt Damon instead of Matt Damon?
> what's wrong with asking another one?
It's illegal, you can't hire an impersonator and allude to it being the real deal. That's the issue.
> Is it scummy to hire "discount" Matt Damon instead of Matt Damon?
If you imply you've hired Matt Damon by tweeting something about Matt Damon and your "discount" had a profession as a Matt Damon impersonator, yes. Yes 100%.
This is illegal with or without AI. We've literally had these laws longer than the tech.
>Is it scummy to hire "discount" Matt Damon instead of Matt Damon?
If Matt Damon said no (e.g. leaving a paper trail) and you want people to pretend it's Matt Damon anyway, yes. Look up Ford v. Milton. You can't just go behind an artist's back like that.
If you want to argue about soundalikes in spinoffs, those are discussed in contracts (and those kinds of figures are making 7+ figures as is, so it's fair on them to choose to let them do that).
Initially I'd say well if you're a public figure and upload your own voice online, of course this will happen. So its something to expect, however, this shouldn't be a problem for Jeff to solve... instead it should be YouTube's problem as they profit from the video monetization. Eventually they'll have to have some kind of detection for all uploaded content.
I strongly disagree. I don't know the rights around one's own voice, but the idea that you suddenly lose ownership of something because you shared it online is the exact thing that many people take issue with when it is written in the terms of service for social networks, creator tools (adobe), etc.
I didn't mention ownership and I don't think you should lose it (nor does one lose it really even in this case, legally). But I do think that in cases like these, where there's money involved and YouTube, that they should have the means to prevent it.
Issue is that Youtube profits off it while signing away liability. There's no incentive to prevent it.
This whole argument rests on the absurd assumption that you can "own" a voice as if it's property. Does this mean people can own the patterns of vibration in air? It's completely nonsensical.
More like owning an identity or likeness, which is a fundamental basis of many fraud cases. It seems that Jeff Geerling is a resident of Missouri which also has Supreme Court precedent to test for "Right of Publicity" [1]:
"If a product is being sold that predominantly exploits the commercial value of an individual's identity, that product should be held to violate the right of publicity and not be protected by the First Amendment, even if there is some "expressive" content in it that might qualify as "speech" in other circumstances."
Whether or not the voice is determined to be predominant would be for courts to decide, of course, but there's clearly an argument.
1: https://law.justia.com/cases/missouri/court-of-appeals/2006/...
"Why should child porn be illegal? It's just a pattern of bits on a computer!"
Describing a reasonable legal principle in terms of physics phenomena does not make it unreasonable.
You're missing the point. Ownership implies the resource is scarce. If the resource is a pattern, which cannot be physically owned and my usage of the pattern doesn't prevent others from using the pattern, it's nonsensical to say it can be owned by anybody.
If you make, say, a webcomic, a big publisher is not allowed to collect and print your comic, or republish it on their ad-riddled website, unless they get your permission and offer whatever compensation you demand. They also can't copy your characters exactly and publish new stories about them, even if all the art and writing is original.
This is a good thing, even though your comic is nothing but a bunch of JPGs--digital patterns--that can be copied infinitely without depriving anyone.
Voice-reproduction rights can get complicated in the AI age, and I don't claim to have the objective moral truth. But there are very good reasons to give legal protection to the creators of digital patterns, as a general principle.
Intellectual property is, in fact, recognized by many legal jurisdictions. And audio works are typically included in that.
However, in this situation, the right of publicity is probably more applicable.
Appeal to law fallacy.
The fact that 93% of the world's legal jurisdictions, including the entire democratic world, doesn't agree with your assertion of absurdity, is pretty good evidence that you are the one with a radical opinion.
You can't just say "Appeal to law fallacy", nothing else, and then move on. You haven't made an argument.
In order for this to be a FALLACY, you have to explain how an appeal to law doesn't work here. Simply appealing to law itself isn't a fallacy.
>Does this mean people can own the patterns of vibration in air? It's completely nonsensical.
If you argue similarly, then the whole juridical system is nonsensical, because everything is just particles and waves, and different configurations thereof - not to mention the many protected things that are acts, which are neither particles nor waves, and are completely made up.
I'd say it's desirable to regulate something like this, however nonsensical-seeming, so that we can at least somewhat protect the individuals, and the general well-being of society.
That wasn't my point. We can own physical items, land etc. Because it's a scarce resource. If me "owning" something doesn't prevent you from also "owning" it, the concept of ownership of that item is absurd.
My voice is even more scarce, there is only one of them in the whole world! I think. Also, we can both own the same land.
I do understand rivalry in economics, but that's not the issue here. The issue is the need to protect individuals from abuse, part of which is the one's right to their own likeness. This likeness is not free for the taking. Now, we can call this owning the likeness, or we can call it something else, but having something, that is not free for taking, sounds an awful lot like owning something.
Ownership as a whole is a social/legal construct, no? [1] You only "own" something insofar as you control it. Societies build frameworks that aid people in controlling things based on who they think deserves control of those things. This clearly can vary widely between societies; the USA used to think owning people was perfectly OK and some places still do. Nobody has absolute control over (almost?) anything, and I think if a society provides many legal protections to something like a voice (who can profit off of it, who goes to jail if they try to copy it), it can be "owned" to as meaningful an extent as anything else.
Many social/legal constructs are for lack of a better term, idiotic. Step back for a second and think about what ownership actually implies? Exclusive control over a scarce resource. If a resource isn't scarce, where your control over it doesn't impede or restrict other peoples control over it, there is no feasible model that explains why one person has control or ownership of it over another.
Social/legal constructs like owning ideas or the sound of a voice exist not because they make any sort of logical sense, they exist to give privilege and power to certain groups at the expense of others.
This is part of natural human behavior. The idea of owning people as physical property is absurd as well, but if you kiss someone's girlfriend you might get punched in the face. The reason this happens is entirely a social construct, but the reality of the situation is no less real.
Social constructs don't not exist just because they are social constructs. The physical world not the extent of reality. Social constructs are also real because we make them real as a part of our own human experience.
>Step back for a second and think about what ownership actually implies?
It implies identity, and control. The former being most relevant here. And yes, if you want to use someone's identity you need to at least get consent.
>Exclusive control over a scarce resource.
"Scarce" is doing a lot of heavy lifting. And in the grand scheme, everything can be "scarce". The internet isn't infinite either, only in a virtual sense (no pun intended).
Meanwhile, thoughts and data are resources, and you can own the idea behind it. Hence, "intellectual property".
> there is no feasible model that explains why one person has control or ownership of it over another.
That's where identity came in. If an inventor got run over tomorrow, there may still be iterations on the invention, but it would completely alter the course of iteration. That's ownership in a sense. Ownership that your ideas can't just be understood or replicated unless you transfer your thoughts to another medium. And like any conversion, data is lost in the exchange.
>they exist to give privilege and power to certain groups at the expense of others.
Yeah, the less off inventors. Do you know why IP is automatically granted? Because someone unaware of the legal channels would just have their own IP signed off once anyone in business gets a whiff of it.
of course, the priveledge still abuse it. But when everything becomes a free for all, the rich get richer.
You can own your likeness. Does that mean you can own the photos that represent your face? Yes, yes you can. Why should you voice be different?
> You can own your likeness.
How?
Easy: The air is free but the electromagnetic spectrum is regulated.
/s
Human voices store shared accent data too not just biometrics, your siblings/twins/sorority sisters likely present with extremely similar voices which can cause the same confusion as AI clones. is it most famous sibling first gets the copyright or are siblings the only ones with the power to supersede it or must they alter their voice? Simple voice never really was guaranteed to be a unique part of identity since it changes so easily over time and ages.
With enough of a "vibration pattern", it becomes a fingerprint.
It is like becoming meme famous: is up to you how to monetize it, nobody owes you nothing.
But they should owe you for stealing your likeliness without your awareness to promote their products. This isn’t for satire purposes.
How much? This IP/copyright mentality is so 90s/2000s. Brings back memories of Napster. This Jeff Geerling which I have never heard of will cry until some Spotify of AI appears and pay him some pennies. Maybe he wants to be the James Hetfield/Metallica of this generation. This guy doesn't have 1/10000 of the relevance Metallica had at the time tho.
> This Jeff Geerling which I have never heard of
This guy doesn't have 1/10000 of the relevance Metallica had at the time
It's weird that you seem to think a person's level of fame is relevant to a discussion of their legal rights.
In terms of personality rights, it is relevant. Although I would think Jeff is enough of a public figure to make a case here
What legal rights? TIL voice imitators are criminals. /s
> TIL voice imitators are criminals. /s
First off, this reddit speak needs to go in a fire and die. Seriously, stop typing like this, nobody likes it.
Secondly, yes voice imitators CAN be criminals. If they attempt to or allude to being the real deal. You can't get on a phone interview with the news and say you're Barack Obama, even if your impression is really good. Yes, that's illegal and has been since before AI of this form existed.
> Secondly, yes voice imitators CAN be criminals.
Which is not the case in discussion, there is no impersonation.
> First off, this reddit speak needs to go in a fire and die.
Are you ok? I think you should calm down before posting.
I'm fine thanks, and I would argue that yes there is impersonation. You can argue why it's not impersonation, but you can't say "nope uh uh no impersonation". That's not how arguments work, you can't begin in a position where you're already right.
> This IP/copyright mentality is so 90s/2000s. Brings back memories of Napster.
The trend has been going against copyright ever since internet was invented. We used to go for passive consumption - radio, TV, books, print magazines. But now that age has passed. We have changed. We prefer interactivity - games, social networks, web searching the billions of contents online, youtube clips commented and shared around. In this age copying is like speaking.
Now comes AI and pushes this trend even deeper - more interactive, more remixing and adapted to the user. We should take a hint - the idea of owning content doesn't make sense anymore. Any content can be recreated in seconds now, or similar content was already posted online years ago. Protecting expression is useless and protecting styles would destroy creativity. Quite a catch-22.
We should take a look instead at new models that popped up in the last decades - open source, creative commons, Wikipedia, open scientific publication. Maybe we need to decouple attribution from payment, like scientific papers, they cite each other without monetary obligations. In social networks comments respond to each other, it would not work if we had to pay to read. Even in real life, people don't pay to hear others speak, and are reusing ideas without issue.
I am aware this sounds bad for copyright, but I am trying to state the reality, not propose my own views. There are deep reasons we prefer interactivity to passive consumption. Copyright was fit for a different age.
How are you on hacker news and never have heard of Jeff Geerling? He's a goat in the ansible and raspberry pi world.
You would be surprised how niche these two things are. I don't care about both and 6 seconds of a tech guy's voice I am sure isn't the trademark of his content
Funny, I first heard of Jeff Geerling over ten years ago in my dev circles.
Your inability to look even few decades into the future to see the impact of this is depressing. You only seem to care about yourself and your current grift.
Welcome to Hacker News/YCombinator. It's what they do here. Ever since joining the community, it's been a speedrun of things explicitly covered as unethical during my CS Ethics course. Mainly stay here to keep a finger on how far things have slid down the "man made horrors beyond comprehension".
The danger of this shit can not be understated. Four years ago already there was a video where a deepfake of a president of the USA read a speech: In Event of Moon Disaster https://www.youtube.com/watch?v=LWLadJFI8Pk we of course know Nixon never gave this speech.
What happens when this "AI" is used to sway an election?
Last month a family got hospitalized after eating mushrooms they found and identified from a AI generated book. They didn't know it was AI generated. What happens when and this is not if, alas but when people die from this?
This shit is a danger to democracy and human lives. Napster was not.
> What happens when this "AI" is used to sway an election?
What do you mean "when"? It's already happening.
https://apnews.com/article/trump-taylor-swift-fake-endorseme...
Hi chx, long time no see! And I agree; deepfakes and voice cloning is already past the point it's fooling relatives. Some are good enough I have to spend time double checking if it's real or not. There are very real implications to all that, and being able to verify true from false is going to get more challenging in many circumstances.
> What happens when this "AI" is used to sway an election?
It already happened and the world went merrily on its way towards wherever the hell it goes.
This doesn't mean it's good or even fine. The world also had a holocaust, and we moved on. But it would've been preferable to not have a holocaust.
You are right. Even if I think elections don't matter there was one instance where it happened and the outcome actually mattered. The Brexit vote. But even there I don't think it's a huge difference if politicians are lying themselves or they do it with the use of AI. As long as it's an open technology the game is still fair.
> As long as it's an open technology the game is still fair.
I don't necessarily agree because money has a lot to do with it. In theory, anyone can use AI for evil. In practice, only the richest can inflict damage. It's a matter of scale.
That's why me killing someone is pretty bad, but me building a Nuclear Warhead is much worse. Luckily I don't have the power to build a Nuclear Warhead. Now anybody with enough money has the tech to just make one.
> In practice, only the richest can inflict damage. It's a matter of scale.
That's always been true for every technology ever. Even nuclear warheads you mentioned. They are not controlled by poor people. Even hacking that was seen as sort of equalizer is way more dangerous in hands of states that can hire whole armies of hackers.
>Last month a family got hospitalized after eating mushrooms they found and identified from a AI generated book.
Yeah, mushrooms are very dangerous, but the argument here is that the book could be written by a human. So what's the best way forward? Ban A.I. books, and human books?
A.I. also, is the best way IMHO to identify mushrooms. Mushrooms are drastically different from one another by magnifying their spores using a microscope. That's not the case when looking at the fruit. Mushrooms totally unrelated to one another, may turn out to be very similar, depending on season, humidity, rain, elevation, tree hosts, temperature and soil.
However when a human tries to examine the spores, he has to compare the spores to thousands of mushrooms to be sure. That's an amount of information, that only mechanically could be tackled effectively. A.I. may have a good chance to solve that problem.
That's kind of a different thing. Making the best AI possible for recognizing mushrooms and trying to produce the most factually correct answers is different than letting AIs run rampant, generating fiction.
Also, I'd want some kind of verification before using the mushroom app. Who made the app? Do they have people on their team familiar with mushrooms? Botanists or whatever you call them. Some random dude with free time and access to maybe 10 species of mushrooms in his backyard.. even if his intent is good, is still dangerous.
LLMs can not produce factual answers. That's not a property of them. What people call "hallucination" is the only thing they can do. Even if it occasionally happens to deliver a correct results that's just the broken clock being right twice a day. The typical ones sound authorative and so people believe it's authorative, this is well documented in Thinking Fast & Slow. As Aza Raskin put it, they hit a zeroday in human cognition. But that doesn't change the fundamentals of it, only the perception of the output which we need to consciously combat.
https://hachyderm.io/@inthehands/112006855076082650
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.
I wasn't referring specifically to LLMs when talking about the mushrooms. You could do something more sophisticated. I'm not an expert on it, but you could probably compare the images against a known, validated dataset to give you a baseline and then use some fancy AI algorithm to hone in on the best match, or do it the other way around -- let the AI take a first guess and then validate that its not hallucinating.
They're already trying to use AI for breast cancer scans and what not, I don't see why we can't do it for mushrooms.
> the book could be written by a human.
I don't understand why so many people in these discussions are so keen to ignore the impact of scale and accessibility when it comes to new technology, and specifically this new technology. Yes, most dangerous things that can be done with AI can also be done by humans.
Is it not meaningful that these dangerous things can now be done far more cost effectively? It would've made no financial sense in the past to spend hours/days creating a fake mushroom identification book. You'd almost certainly never get enough sales to make it worth it, especially once people realized your book was nonsense and potentially dangerous (getting you delisted, as it seems like the seller did with this book). Now you can just ask an AI to "generate 100 book ideas, scripts, and images." Who cares if almost all of them make very little money when the time and $ cost to creating them is near zero (it looks like this book was physical but especially for digital books, videos, etc.).
Is it not meaningful that dangerous things can now be done (or may soon be able to be done) by more people with less skill? The time/money investment required to learn skills with the potential for destructive use is IMO a strong filter against the people who would do those destructive things in the first place.
OpenAI and I think Google have talked about making intelligence "too cheap to meter." I'm not sure that is a good thing. I could be convinced, but every poorly thought out dismissal of AI dangers only makes me increasingly sure that we aren't ready for it.
For those who are less concerned about AI dangers, consider maybe that premature reckless deployment of AI also has serious potential for generating social backlash that might end up slowing or halting the adoption of AI for positive purposes. The average non-tech person I know thinks of AI as a useful improvement over Google, and maybe thinks AI generated images are cool, but they aren't in love with it. It could disappear tomorrow and they'd shrug. In fact, public sentiment against AI (in the US) seems to be rapidly increasing, and is bipartisan [1]. If the promise of AGI is achieved this will go into overdrive should (when?) mass-unemployment happens. Look how far right-wing politicians are able to go by drumming up fears about immigrants stealing jobs.
https://www.pewresearch.org/short-reads/2023/11/21/what-the-...
> Is it not meaningful that these dangerous things can now be done far more cost effectively?
It is meaningful. Humans can publish 10 books with 10 mistakes inside, A.I. can publish a trillion books with countless mistakes. Radically different scale.
Imagine this: A writer after he finishes writing a book, he goes to a government office, and stamps electronically the hash signature of the book as "Human Approved". Then anyone can search the Book stamp in an internet database. Everything else is labeled as A.I.
Is it so difficult for a government to use a software like that? How about an open source software? (It doesn't exist, but it could be implemented by some people.)
> In fact, public sentiment against AI (in the US) seems to be rapidly increasing, and is bipartisan
That cannot be true. A massive poll of at least 10.000 people could demolish that myth. The poll should be organized by the age of participants, rather than political parties. Candidate ages are, 18-30 years old, 30-60, and 60-Rest.
Ages 18-30 will be 80% in favor of A.I.
You think someone stealing the dude’s voice is the same as people downloading Metallica songs?
Are you trying not to be taken seriously?
I feel like this comment helps confirm dead internet theory. Are we there yet HN?
We’re getting there as we speak
Boneheaded take.