Damn, that's a lot of videos for them to contact the creators and ask for permission to use their content as machine learning training content. Unless of course, they didn't, and just went ahead with it anyway...
Does that apply to everyone who doesn't live in the EU as well? And what happens when a major AI company uses this dataset as training data, ignoring consent as usual?
"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours."
I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.
Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.
Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network.
I was wondering if youtube blocked them a lot and they got a lot of videos from other sources - but no, you can see on this HF page that 93.1% of URls are youtube.com, https://huggingface.co/datasets/laion/BVD-URLs
- let us say you ran yt-dlp inside python aiohttp
- surely your ll run a limit soon as your ip address will be flagged
- what solutions do we have to auto rotate proxies in python
- are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?
I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.
They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.
[0] https://proxybase.xyz
Another thing to thank AI for! /s