‘Scary, But Exciting’: What to Know About YouTube’s Pop Star-Imitating AI Experiment

YouTube announced one of this year’s most significant artificial intelligence developments in the music industry this week when it launched its new experimental AI feature that will allow select users to make song clips featuring AI-generated vocals of superstars including Demi Lovato, Sia and John Legend.

Dream Track officially launched Thursday as an early beta test available only to about 100 selected creators in the U.S., and YouTube has given no indication if or when the new feature would become widely available. YouTube announced several new AI music tools this week — including one in development that could help turn songwriters’ thoughts into actual music just by humming an idea — though the voice clones were the most noteworthy feature.

Both the labels and YouTube itself have been careful to emphasize that Dream Track is an experiment, and that it isn’t immediately clear how the new feature will develop in subsequent months. But regardless of what happens next, Dream Track marks the first time some of the world’s biggest artists and music companies have granted their music to be used for voice clones  — one of the more controversial uses of AI in music creation — a significant step that could pave the way on how the industry handles the technology going forward. 

The news brings many further implications and questions, some that may only be better answered in the months ahead. Here’s a breakdown of what you need to know about YouTube’s new AI music maker.

Voice clones have been going viral all year. Why does this matter now?

While fans, content creators and professional songwriters alike have been messing around with voice clones for months using AI software such as UberDruck, those clones are often unauthorized, and record labels have been pushing the rest of the industry to crack down on AI content they say infringes on their music. YouTube’s new feature, by contrast, is a direct collaboration with some of the industry’s largest copyright holders. 

AI has been among the most pressing topics in the business this year, and it has the potential to be the most disruptive technology to hit the industry since Napster and peer-to-peer file sharing in the early 2000s. The industry has been wary of AI’s ability to allow people to take artists’ works and imitate them without their consent, as well as to further flood the market with content and make it even harder for real musicians to break through the noise. 

But rather than fully rebuke AI, the industry seems to be looking to avoid another disastrous Napster moment and be more proactive in developing the tech in a way that benefits them.

As WMG CEO Robert Kyncl — a former YouTube executive himself — told analysts during WMG’s earnings call Thursday morning, per Billboard: “Imagine in the early 2000s if the file-sharing companies came to the music industry and said, ‘would you like to experiment with this new tool that we built and see how it impacts the industry and how we can work together?’ It would have been incredible.”

YouTube’s new experiment isn’t the first time the industry’s largest fixtures have gingerly navigated AI. Sony Music hired an executive vice president of AI this summer to help the company navigate its approach around the tech. Earlier this year, UMG partnered with generative music app Endel to make ambient tracks using stems the company’s artists provide. 

By August, UMG and YouTube announced a partnership to develop an AI Incubator, as well as a set of principles for how to move forward with music and AI. “Our challenge and opportunity as an industry is to establish effective tools, incentives and rewards – as well as rules of the road – that enable us to limit AI’s potential downside while promoting its promising upside,” UMG CEO Lucian Grainge said when the incubator launched.

Who is involved in the new experiment?

The nine artists participating in Dream Track — Lovato, Sia, Legend, Charli XCX, Troye Sivan, Papoose, Charlie Puth, T-Pain and Alec Benjamin — are all signed to Universal, Warner and indie music companies including EMPIRE.

EMPIRE’s CEO Ghazi said in a statement to Rolling Stone that “we’re always seeking innovative ways for our artists to participate in new technology,” he said. “Teaming up with YouTube Music and Google to develop music-centric AI tools is a positive step towards the future for our artists and music creators as a whole.”

Benjamin tells Rolling Stone in an interview that YouTube asked him to participate in the test just over a month ago, after he told a friend who worked at the company that he’d written a song loosely based on AI. He was intrigued by YouTube’s concept and felt that participating would give him a chance to be one of the first artists “to help figure out and shape how this is used.”

“It doesn’t really necessarily matter how I feel about [AI] because it’s sort of where things are headed anyways. I think there’s no use in fighting this stuff,” Benjamin says. “But I am excited about it. I’m excited to be be on the forefront of this new technology. I’m interested to see what the AI generates with my voice and what these things sound like, and how creators will interact with the AI version of my music.”

Benjamin expressed both curiosity and some nerves about how creators will use his AI voice clone in the coming months. “What’s cool about this is that it gives you a certain degree of immortality as far as your voice is concerned. Your voice can be used across time now, whether you’re there to make new recordings or not. There’s something kind of scary about that, but it’s also exciting,” Benjamin says. “What’s also scary is, what if people use my voice in a way that I don’t want them to use it.

“These are really interesting, scary questions, but I think that regardless of how the application of the technology makes me feel, it’s going to happen anyways. I’m sure there will be things that will make me uncomfortable, and I don’t even know what the things are yet,” he adds with a chuckle. “This seems to be the new wave or maybe it’s not, but we’re trying things out and we’re about to find out. You can’t stop the wave so I might as well ride it.”

So, how does it work?

Dream Track uses generative AI trained on the vocals of the artists participating. Creators with access to Dream Track can submit text-based requests with prompts of song ideas, and they select which artist AI they want to perform the song. Dream Track does the rest, putting out song clips up to 30 seconds long.

Tech YouTuber and journalist Cleo Abram, for instance, used Dream Track to create a virtual Charlie Puth to sing about her dog. 

YouTube hasn’t given details on what was used to train the AI artists. But a source familiar with the matter tells Rolling Stone that UMG “allowed a limited subset of the company’s content to be trained on for a limited time in order to facilitate the experiment.”

Benjamin, signed to Warner label Elektra, tells Rolling Stone that he hadn’t recorded any new music for the experiment but that he wasn’t told precisely what the training content on his AI was. That training almost certainly came from his previously recorded catalog. 

When asked about payment for his participation, he said he hadn’t asked any questions about the monetization process and that those details would go through his record label. Neither YouTube, UMG nor WMG gave any details regarding payments with the deal. 

Do AI developers even need the industry’s permission to make their AI models? Who owns this music?

AI music presents significant legal questions from who can own the music that gets made to what permissions are needed to train AI.

Unsurprisingly, many AI developers believe that training their AI models on copyrighted material should be considered fair use, which would allow them to skirt seeking permission and paying copyright holders to use those works. Equally unsurprisingly, the music companies — who’ve built their businesses on decades of intellectual property ownership — strongly push back on that notion.

Ownership around AI-generated works is murky as well. The copyright office has ruled that purely machine-generated works cannot receive copyright protection as humans were not sufficiently involved in the work’s creation. Does a song generated mainly by AI at the specific prompting of a human being pass that barrier?

From Rolling Stone US.