1
0
mirror of https://github.com/yt-dlp/yt-dlp.git synced 2026-01-16 20:01:29 +00:00

Compare commits

...

22 Commits

Author SHA1 Message Date
pukkandan
5ef7d9bdd8 Release 2021.03.01 2021-03-01 05:39:50 +05:30
pukkandan
62bff2c170 Add option --extractor-retries to retry on known extractor errors
* Currently only used by youtube

Fixes https://github.com/ytdl-org/youtube-dl/issues/28194
Possibly also fixes: https://github.com/ytdl-org/youtube-dl/issues/28289 (can not confirm since the issue isn't reliably reproducible)
2021-03-01 05:18:37 +05:30
pukkandan
f0884c8b3f Cleanup some code (see desc)
* `--get-comments` doesn't imply `--write-info-json` if `-J`, `-j` or `--print-json` are used
* Don't pass `config_location` to `YoutubeDL` (it is unused)
* [bilibiliaudio] Recognize the file as audio-only
* Update gitignore
* Fix typos
2021-02-28 20:56:32 +05:30
pukkandan
277d6ff5f2 Extract comments only when needed #95 (Closes #94) 2021-02-28 20:26:08 +05:30
pukkandan
1cf376f55a Add option --sleep-requests to sleep b/w requests (Closes #106)
* Also fix documentation of `sleep_interval_subtitles`

Related issues:
https://github.com/blackjack4494/yt-dlc/issues/158
https://github.com/blackjack4494/youtube-dlc/issues/195
https://github.com/ytdl-org/youtube-dl/pull/28270
https://github.com/ytdl-org/youtube-dl/pull/28144
https://github.com/ytdl-org/youtube-dl/issues/27767
https://github.com/ytdl-org/youtube-dl/issues/23638
https://github.com/ytdl-org/youtube-dl/issues/26287
https://github.com/ytdl-org/youtube-dl/issues/26319
2021-02-27 18:14:42 +05:30
pukkandan
7f7de7f94d Allow specifying path in --external-downloader 2021-02-27 16:52:27 +05:30
pukkandan
86878b6cd9 [hrfensehen] Fix wrong import 2021-02-27 15:35:41 +05:30
pukkandan
b3d1242534 [youtube] Fix inconsistent webpage_url (closes #119) 2021-02-27 14:45:56 +05:30
pukkandan
9bd2020476 [hls] Enable --hls-use-mpegts by default when downloading live-streams
* Also added option `--no-hls-use-mpegts` to disable this

Related: #96
2021-02-26 21:52:16 +05:30
pukkandan
ed9b7e3dd3 Fix bug with m3u8 format extraction 2021-02-26 18:32:28 +05:30
shirt-dev
c552ae8838 Fix get_executable_path (#117)
Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>
2021-02-26 04:28:02 +05:30
Robin Dunn
31a5e037a7 [viki] Fix viki play pass authentication (#111)
Authored by: RobinD42
2021-02-26 03:33:00 +05:30
pukkandan
3638226215 [ci] Disable download tests unless specifically invoked
Tests can be enabled/disabled using the following in the commit message
* Run Download: `ci-run-dl`
* Skip Core: `ci-skip`
* Skip Quick & Core: `ci-skip-all`
(replace "-" by a space " ")
2021-02-26 03:28:18 +05:30
pukkandan
14fdfea973 [youtube] Retry on incomplete ytInitialData
Related: #116
2021-02-26 03:23:08 +05:30
shirt-dev
b45d4e4a8e Fix completion paths, zsh pip completion install (#114) 2021-02-25 11:00:29 -05:00
pukkandan
3e39273418 Merge branch 'master' into fix-paths 2021-02-25 21:17:46 +05:30
shirt-dev
b965087396 Readthedocs improvements (#115)
Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>

:ci skip dl
2021-02-25 21:16:08 +05:30
hseg
359d6d8650 Fix completion paths, zsh pip completion install
Closes: #108, #110
2021-02-25 16:49:57 +02:00
pukkandan
0e0040519b [embedthumbnail] Fix bug with deleting original thumbnail (Closes #113)
:ci skip dl
2021-02-25 18:35:04 +05:30
pukkandan
127d075955 [documentation] Fix typos (Closes #112)
:ci skip all
2021-02-25 16:08:25 +05:30
pukkandan
bce8cbb089 [tennistv] Fix format sorting 2021-02-25 16:07:38 +05:30
pukkandan
aae273ded8 [version] update :ci skip dl 2021-02-25 02:44:10 +05:30
45 changed files with 1611 additions and 1514 deletions

1
.gitattributes vendored Normal file
View File

@@ -0,0 +1 @@
Makefile* text whitespace=-tab-in-indent

View File

@@ -21,7 +21,7 @@ assignees: ''
<!-- <!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.19. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED. - First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.24. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser. - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp. - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates. - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
--> -->
- [ ] I'm reporting a broken site support - [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running yt-dlp version **2021.02.19** - [ ] I've verified that I'm running yt-dlp version **2021.02.24**
- [ ] I've checked that all provided URLs are alive and playable in a browser - [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [ ] I've searched the bugtracker for similar issues including closed ones - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
[debug] User config: [] [debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj'] [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] yt-dlp version 2021.02.19 [debug] yt-dlp version 2021.02.24
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {} [debug] Proxy map: {}

View File

@@ -21,7 +21,7 @@ assignees: ''
<!-- <!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.19. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED. - First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.24. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser. - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights. - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
- Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates. - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
--> -->
- [ ] I'm reporting a new site support request - [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running yt-dlp version **2021.02.19** - [ ] I've verified that I'm running yt-dlp version **2021.02.24**
- [ ] I've checked that all provided URLs are alive and playable in a browser - [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that none of provided URLs violate any copyrights - [ ] I've checked that none of provided URLs violate any copyrights
- [ ] I've searched the bugtracker for similar site support requests including closed ones - [ ] I've searched the bugtracker for similar site support requests including closed ones

View File

@@ -21,13 +21,13 @@ assignees: ''
<!-- <!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.19. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED. - First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.24. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates. - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space) - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--> -->
- [ ] I'm reporting a site feature request - [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running yt-dlp version **2021.02.19** - [ ] I've verified that I'm running yt-dlp version **2021.02.24**
- [ ] I've searched the bugtracker for similar site feature requests including closed ones - [ ] I've searched the bugtracker for similar site feature requests including closed ones

View File

@@ -21,7 +21,7 @@ assignees: ''
<!-- <!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.19. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED. - First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.24. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser. - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp. - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates. - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
--> -->
- [ ] I'm reporting a broken site support issue - [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running yt-dlp version **2021.02.19** - [ ] I've verified that I'm running yt-dlp version **2021.02.24**
- [ ] I've checked that all provided URLs are alive and playable in a browser - [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [ ] I've searched the bugtracker for similar bug reports including closed ones - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
[debug] User config: [] [debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj'] [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] yt-dlp version 2021.02.19 [debug] yt-dlp version 2021.02.24
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {} [debug] Proxy map: {}

View File

@@ -21,13 +21,13 @@ assignees: ''
<!-- <!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp: Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.19. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED. - First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.02.24. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates. - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space) - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--> -->
- [ ] I'm reporting a feature request - [ ] I'm reporting a feature request
- [ ] I've verified that I'm running yt-dlp version **2021.02.19** - [ ] I've verified that I'm running yt-dlp version **2021.02.24**
- [ ] I've searched the bugtracker for similar feature requests including closed ones - [ ] I've searched the bugtracker for similar feature requests including closed ones

View File

@@ -3,7 +3,7 @@ on: [push, pull_request]
jobs: jobs:
tests: tests:
name: Core Tests name: Core Tests
if: "!contains(github.event.head_commit.message, 'ci skip all')" if: "!contains(github.event.head_commit.message, 'ci skip')"
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
strategy: strategy:
fail-fast: true fail-fast: true

View File

@@ -3,7 +3,7 @@ on: [push, pull_request]
jobs: jobs:
tests: tests:
name: Download Tests name: Download Tests
if: "!contains(github.event.head_commit.message, 'ci skip dl') && !contains(github.event.head_commit.message, 'ci skip all')" if: "contains(github.event.head_commit.message, 'ci run dl')"
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
strategy: strategy:
fail-fast: true fail-fast: true

8
.gitignore vendored
View File

@@ -8,6 +8,7 @@ dist/
zip/ zip/
tmp/ tmp/
venv/ venv/
completions/
# Misc # Misc
*~ *~
@@ -34,8 +35,9 @@ README.txt
*.spec *.spec
# Binary # Binary
youtube-dl /youtube-dl
youtube-dlc /youtube-dlc
/yt-dlp
yt-dlp.zip yt-dlp.zip
*.exe *.exe
@@ -50,12 +52,14 @@ yt-dlp.zip
*.m4v *.m4v
*.mp3 *.mp3
*.3gp *.3gp
*.webm
*.wav *.wav
*.ape *.ape
*.mkv *.mkv
*.swf *.swf
*.part *.part
*.ytdl *.ytdl
*.dump
*.frag *.frag
*.frag.urls *.frag.urls
*.aria2 *.aria2

View File

@@ -20,4 +20,3 @@ python:
version: 3 version: 3
install: install:
- requirements: docs/requirements.txt - requirements: docs/requirements.txt
- requirements: requirements.txt

View File

@@ -1,38 +0,0 @@
language: python
python:
- "2.6"
- "2.7"
- "3.2"
- "3.3"
- "3.4"
- "3.5"
- "3.6"
- "pypy"
- "pypy3"
dist: trusty
env:
- YTDL_TEST_SET=core
jobs:
include:
- python: 3.7
dist: xenial
env: YTDL_TEST_SET=core
- python: 3.8
dist: xenial
env: YTDL_TEST_SET=core
- python: 3.8-dev
dist: xenial
env: YTDL_TEST_SET=core
- env: JYTHON=true; YTDL_TEST_SET=core
- name: flake8
python: 3.8
dist: xenial
install: pip install flake8
script: flake8 .
fast_finish: true
allow_failures:
- env: YTDL_TEST_SET=download
- env: JYTHON=true; YTDL_TEST_SET=core
before_install:
- if [ "$JYTHON" == "true" ]; then ./devscripts/install_jython.sh; export PATH="$HOME/jython/bin:$PATH"; fi
script: ./devscripts/run_tests.sh

View File

@@ -23,3 +23,5 @@ tsukumi
bbepis bbepis
Pccode66 Pccode66
Ashish Ashish
RobinD42
hseg

View File

@@ -17,9 +17,31 @@
--> -->
### 2021.03.01
* Allow specifying path in `--external-downloader`
* Add option `--sleep-requests` to sleep b/w requests
* Add option `--extractor-retries` to retry on known extractor errors
* Extract comments only when needed
* `--get-comments` doesn't imply `--write-info-json` if `-J`, `-j` or `--print-json` are used
* [youtube] Retry on more known errors than just HTTP-5xx
* [tennistv] Fix format sorting
* [readthedocs] Improvements by [shirt](https://github.com/shirt-dev)
* [hls] Fix bug with m3u8 format extraction
* [bilibiliaudio] Recognize the file as audio-only
* [hrfensehen] Fix wrong import
* [youtube] Fix inconsistent `webpage_url`
* [hls] Enable `--hls-use-mpegts` by default when downloading live-streams
* [viki] Fix viki play pass authentication by [RobinD42](https://github.com/RobinD42)
* [embedthumbnail] Fix bug with deleting original thumbnail
* [build] Fix completion paths, zsh pip completion install by [hseg](https://github.com/hseg)
* [ci] Disable download tests unless specifically invoked
* Cleanup some code and fix typos
### 2021.02.24 ### 2021.02.24
* Moved project to an organization [yt-dlp](https://github.com/yt-dlp) * Moved project to an organization [yt-dlp](https://github.com/yt-dlp)
* **Completely changed project name to yt-dlp** by [Pccode66](https://github.com/Pccode66) and [pukkandan](https://github.com/pukkandan) * **Completely changed project name to yt-dlp** by [Pccode66](https://github.com/Pccode66) and [pukkandan](https://github.com/pukkandan)
* Also, `youtube-dlc` config files are no longer loaded
* **Merge youtube-dl:** Upto [commit/4460329](https://github.com/ytdl-org/youtube-dl/commit/44603290e5002153f3ebad6230cc73aef42cc2cd) (except tmz, gedi) * **Merge youtube-dl:** Upto [commit/4460329](https://github.com/ytdl-org/youtube-dl/commit/44603290e5002153f3ebad6230cc73aef42cc2cd) (except tmz, gedi)
* [Readthedocs](https://yt-dlp.readthedocs.io) support by [shirt](https://github.com/shirt-dev) * [Readthedocs](https://yt-dlp.readthedocs.io) support by [shirt](https://github.com/shirt-dev)
* [youtube] Show if video was a live stream in info (`was_live`) * [youtube] Show if video was a live stream in info (`was_live`)
@@ -28,12 +50,12 @@
* [tennistv] Fix extractor * [tennistv] Fix extractor
* [hls] Support media initialization by [shirt](https://github.com/shirt-dev) * [hls] Support media initialization by [shirt](https://github.com/shirt-dev)
* [hls] Added options `--hls-split-discontinuity` to better support media discontinuity by [shirt](https://github.com/shirt-dev) * [hls] Added options `--hls-split-discontinuity` to better support media discontinuity by [shirt](https://github.com/shirt-dev)
* [ffmpeg] Allow passing custom arguments before -i using `--ppa "ffmpeg_i1:ARGS"` synatax * [ffmpeg] Allow passing custom arguments before -i using `--ppa "ffmpeg_i1:ARGS"` syntax
* Fix `--windows-filenames` removing `/` from UNIX paths * Fix `--windows-filenames` removing `/` from UNIX paths
* [hls] Show warning if pycryptodome is not found * [hls] Show warning if pycryptodome is not found
* [documentation] Improvements * [documentation] Improvements
* Fix documentation of `Extractor Options` * Fix documentation of `Extractor Options`
* Document `all` in format selection (Closes #101) * Document `all` in format selection
* Document `playable_in_embed` in output templates * Document `playable_in_embed` in output templates
@@ -276,7 +298,7 @@
* Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams` * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
* Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details * **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details
* **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-sponsorblock-options) for details
* Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h) * Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h)
* Added `--list-formats-as-table`, `--list-formats-old` * Added `--list-formats-as-table`, `--list-formats-old`
* **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file * **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file

View File

@@ -1,10 +1,10 @@
all: yt-dlp doc man all: yt-dlp doc man
doc: README.md CONTRIBUTING.md issuetemplates supportedsites doc: README.md CONTRIBUTING.md issuetemplates supportedsites
man: README.txt yt-dlp.1 yt-dlp.bash-completion yt-dlp.zsh yt-dlp.fish man: README.txt yt-dlp.1 bash-completion zsh-completion fish-completion
clean: clean:
rm -rf yt-dlp.1.temp.md yt-dlp.1 yt-dlp.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz yt-dlp.zsh yt-dlp.fish yt_dlp/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec *.frag *.frag.urls *.frag.aria2 CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe rm -rf yt-dlp.1.temp.md yt-dlp.1 README.txt MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz completions/ yt_dlp/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec *.frag *.frag.urls *.frag.aria2 CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe
find . -name "*.pyc" -delete find . -name "*.pyc" -delete
find . -name "*.class" -delete find . -name "*.class" -delete
@@ -21,17 +21,12 @@ SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then ech
# set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2 # set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi) MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
install: yt-dlp yt-dlp.1 yt-dlp.bash-completion yt-dlp.zsh yt-dlp.fish install: yt-dlp yt-dlp.1 bash-completion zsh-completion fish-completion
install -d $(DESTDIR)$(BINDIR) install -Dm755 yt-dlp $(DESTDIR)$(BINDIR)
install -m 755 yt-dlp $(DESTDIR)$(BINDIR) install -Dm644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1
install -d $(DESTDIR)$(MANDIR)/man1 install -Dm644 completions/bash/yt-dlp $(DESTDIR)$(SHAREDIR)/bash-completion/completions/yt-dlp
install -m 644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1 install -Dm644 completions/zsh/_yt-dlp $(DESTDIR)$(SHAREDIR)/zsh/site-functions/_yt-dlp
install -d $(DESTDIR)$(SYSCONFDIR)/bash_completion.d install -Dm644 completions/fish/yt-dlp.fish $(DESTDIR)$(SHAREDIR)/fish/vendor_completions.d/yt-dlp.fish
install -m 644 yt-dlp.bash-completion $(DESTDIR)$(SYSCONFDIR)/bash_completion.d/yt-dlp
install -d $(DESTDIR)$(SHAREDIR)/zsh/site-functions
install -m 644 yt-dlp.zsh $(DESTDIR)$(SHAREDIR)/zsh/site-functions/_yt-dlp
install -d $(DESTDIR)$(SYSCONFDIR)/fish/completions
install -m 644 yt-dlp.fish $(DESTDIR)$(SYSCONFDIR)/fish/completions/yt-dlp.fish
codetest: codetest:
flake8 . flake8 .
@@ -61,7 +56,7 @@ tar: yt-dlp.tar.gz
.PHONY: all clean install test tar bash-completion pypi-files zsh-completion fish-completion ot offlinetest codetest supportedsites .PHONY: all clean install test tar bash-completion pypi-files zsh-completion fish-completion ot offlinetest codetest supportedsites
pypi-files: yt-dlp.bash-completion README.txt yt-dlp.1 yt-dlp.fish pypi-files: README.txt yt-dlp.1 bash-completion zsh-completion fish-completion
yt-dlp: yt_dlp/*.py yt_dlp/*/*.py yt-dlp: yt_dlp/*.py yt_dlp/*/*.py
mkdir -p zip mkdir -p zip
@@ -92,7 +87,7 @@ issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_
$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md .github/ISSUE_TEMPLATE/5_feature_request.md $(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md .github/ISSUE_TEMPLATE/5_feature_request.md
supportedsites: supportedsites:
$(PYTHON) devscripts/make_supportedsites.py docs/supportedsites.md $(PYTHON) devscripts/make_supportedsites.py supportedsites.md
README.txt: README.md README.txt: README.md
pandoc -f $(MARKDOWN) -t plain README.md -o README.txt pandoc -f $(MARKDOWN) -t plain README.md -o README.txt
@@ -102,20 +97,23 @@ yt-dlp.1: README.md
pandoc -s -f $(MARKDOWN) -t man yt-dlp.1.temp.md -o yt-dlp.1 pandoc -s -f $(MARKDOWN) -t man yt-dlp.1.temp.md -o yt-dlp.1
rm -f yt-dlp.1.temp.md rm -f yt-dlp.1.temp.md
yt-dlp.bash-completion: yt_dlp/*.py yt_dlp/*/*.py devscripts/bash-completion.in completions/bash/yt-dlp: yt_dlp/*.py yt_dlp/*/*.py devscripts/bash-completion.in
mkdir -p completions/bash
$(PYTHON) devscripts/bash-completion.py $(PYTHON) devscripts/bash-completion.py
bash-completion: yt-dlp.bash-completion bash-completion: completions/bash/yt-dlp
yt-dlp.zsh: yt_dlp/*.py yt_dlp/*/*.py devscripts/zsh-completion.in completions/zsh/_yt-dlp: yt_dlp/*.py yt_dlp/*/*.py devscripts/zsh-completion.in
mkdir -p completions/zsh
$(PYTHON) devscripts/zsh-completion.py $(PYTHON) devscripts/zsh-completion.py
zsh-completion: yt-dlp.zsh zsh-completion: completions/zsh/_yt-dlp
yt-dlp.fish: yt_dlp/*.py yt_dlp/*/*.py devscripts/fish-completion.in completions/fish/yt-dlp.fish: yt_dlp/*.py yt_dlp/*/*.py devscripts/fish-completion.in
mkdir -p completions/fish
$(PYTHON) devscripts/fish-completion.py $(PYTHON) devscripts/fish-completion.py
fish-completion: yt-dlp.fish fish-completion: completions/fish/yt-dlp.fish
lazy-extractors: yt_dlp/extractor/lazy_extractors.py lazy-extractors: yt_dlp/extractor/lazy_extractors.py
@@ -123,7 +121,7 @@ _EXTRACTOR_FILES = $(shell find yt_dlp/extractor -iname '*.py' -and -not -iname
yt_dlp/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES) yt_dlp/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES)
$(PYTHON) devscripts/make_lazy_extractors.py $@ $(PYTHON) devscripts/make_lazy_extractors.py $@
yt-dlp.tar.gz: yt-dlp README.md README.txt yt-dlp.1 yt-dlp.bash-completion yt-dlp.zsh yt-dlp.fish ChangeLog AUTHORS yt-dlp.tar.gz: yt-dlp README.md README.txt yt-dlp.1 bash-completion zsh-completion fish-completion ChangeLog AUTHORS
@tar -czf yt-dlp.tar.gz --transform "s|^|yt-dlp/|" --owner 0 --group 0 \ @tar -czf yt-dlp.tar.gz --transform "s|^|yt-dlp/|" --owner 0 --group 0 \
--exclude '*.DS_Store' \ --exclude '*.DS_Store' \
--exclude '*.kate-swp' \ --exclude '*.kate-swp' \
@@ -135,7 +133,6 @@ yt-dlp.tar.gz: yt-dlp README.md README.txt yt-dlp.1 yt-dlp.bash-completion yt-dl
--exclude 'docs/_build' \ --exclude 'docs/_build' \
-- \ -- \
bin devscripts test yt_dlp docs \ bin devscripts test yt_dlp docs \
ChangeLog AUTHORS LICENSE README.md README.txt \ ChangeLog AUTHORS LICENSE README.md supportedsites.md README.txt \
Makefile MANIFEST.in yt-dlp.1 yt-dlp.bash-completion \ Makefile MANIFEST.in yt-dlp.1 completions \
yt-dlp.zsh yt-dlp.fish setup.py setup.cfg \ setup.py setup.cfg yt-dlp
yt-dlp

View File

@@ -11,7 +11,7 @@
[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi)](https://pypi.org/project/yt-dlp) [![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi)](https://pypi.org/project/yt-dlp)
[![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest)](https://yt-dlp.readthedocs.io) [![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest)](https://yt-dlp.readthedocs.io)
A command-line program to download videos from youtube.com and many other [video platforms](docs/supportedsites.md) A command-line program to download videos from youtube.com and many other [video platforms](supportedsites.md)
This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which is inturn a fork of [youtube-dl](https://github.com/ytdl-org/youtube-dl) This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which is inturn a fork of [youtube-dl](https://github.com/ytdl-org/youtube-dl)
@@ -57,7 +57,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples)) * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
* **Merged with youtube-dl v2021.02.10**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) * **Merged with youtube-dl v2021.02.22**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details. * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
@@ -92,7 +92,7 @@ See [changelog](Changelog.md) or [commits](https://github.com/yt-dlp/yt-dlp/comm
**PS**: Some of these changes are already in youtube-dlc, but are still unreleased. See [this](Changelog.md#unreleased-changes-in-blackjack4494yt-dlc) for details **PS**: Some of these changes are already in youtube-dlc, but are still unreleased. See [this](Changelog.md#unreleased-changes-in-blackjack4494yt-dlc) for details
If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the amount of changes are very large. Compare [options](#options) and [supported sites](docs/supportedsites.md) with youtube-dl's to get an idea of the massive number of features/patches [youtube-dlc](https://github.com/blackjack4494/yt-dlc) has accumulated. If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the amount of changes are very large. Compare [options](#options) and [supported sites](supportedsites.md) with youtube-dl's to get an idea of the massive number of features/patches [youtube-dlc](https://github.com/blackjack4494/yt-dlc) has accumulated.
# INSTALLATION # INSTALLATION
@@ -122,12 +122,12 @@ You can also build the executable without any version info or metadata by using:
**For Unix**: **For Unix**:
You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `nosetests` You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `nosetests`
Then simply run `make`. You can also run `make youtube_dlc` instead to compile only the binary without updating any of the additional files Then simply run `make`. You can also run `make yt-dlp` instead to compile only the binary without updating any of the additional files
**Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number **Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number
# DESCRIPTION # DESCRIPTION
**yt-dlp** is a command-line program to download videos from youtube.com many other [video platforms](docs/supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like. **yt-dlp** is a command-line program to download videos from youtube.com many other [video platforms](supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
yt-dlp [OPTIONS] [--] URL [URL...] yt-dlp [OPTIONS] [--] URL [URL...]
@@ -245,7 +245,7 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
"OUTPUT TEMPLATE" for a list of available "OUTPUT TEMPLATE" for a list of available
keys) to match if the key is present, !key keys) to match if the key is present, !key
to check if the key is not present, to check if the key is not present,
key>NUMBER (like "comment_count > 12", also key>NUMBER (like "view_count > 12", also
works with >=, <, <=, !=, =) to compare works with >=, <, <=, !=, =) to compare
against a number, key = 'LITERAL' (like against a number, key = 'LITERAL' (like
"uploader = 'Mike Smith'", also works with "uploader = 'Mike Smith'", also works with
@@ -317,13 +317,19 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
ffmpeg ffmpeg
--hls-prefer-ffmpeg Use ffmpeg instead of the native HLS --hls-prefer-ffmpeg Use ffmpeg instead of the native HLS
downloader downloader
--hls-use-mpegts Use the mpegts container for HLS videos, --hls-use-mpegts Use the mpegts container for HLS videos;
allowing to play the video while allowing some players to play the video
downloading (some players may not be able while downloading, and reducing the chance
to play it) of file corruption if download is
--external-downloader NAME Use the specified external downloader. interrupted. This is enabled by default for
Currently supports aria2c, avconv, axel, live streams
curl, ffmpeg, httpie, wget --no-hls-use-mpegts Do not use the mpegts container for HLS
videos. This is default when not
downloading live streams
--external-downloader NAME Name or path of the external downloader to
use. Currently supports aria2c, avconv,
axel, curl, ffmpeg, httpie, wget
(Recommended: aria2c)
--downloader-args NAME:ARGS Give these arguments to the external --downloader-args NAME:ARGS Give these arguments to the external
downloader. Specify the downloader name and downloader. Specify the downloader name and
the arguments separated by a colon ":". You the arguments separated by a colon ":". You
@@ -397,7 +403,9 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
--no-write-playlist-metafiles Do not write playlist metadata when using --no-write-playlist-metafiles Do not write playlist metadata when using
--write-info-json, --write-description etc. --write-info-json, --write-description etc.
--get-comments Retrieve video comments to be placed in the --get-comments Retrieve video comments to be placed in the
.info.json file .info.json file. The comments are fetched
even without this option if the extraction
is known to be quick
--load-info-json FILE JSON file containing the video information --load-info-json FILE JSON file containing the video information
(created with the "--write-info-json" (created with the "--write-info-json"
option) option)
@@ -485,6 +493,8 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
--bidi-workaround Work around terminals that lack --bidi-workaround Work around terminals that lack
bidirectional text support. Requires bidiv bidirectional text support. Requires bidiv
or fribidi executable in PATH or fribidi executable in PATH
--sleep-requests SECONDS Number of seconds to sleep between requests
during data extraction
--sleep-interval SECONDS Number of seconds to sleep before each --sleep-interval SECONDS Number of seconds to sleep before each
download when used alone or a lower bound download when used alone or a lower bound
of a range for randomized sleep before each of a range for randomized sleep before each
@@ -495,7 +505,8 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
before each download (maximum possible before each download (maximum possible
number of seconds to sleep). Must only be number of seconds to sleep). Must only be
used along with --min-sleep-interval used along with --min-sleep-interval
--sleep-subtitles SECONDS Enforce sleep interval on subtitles as well --sleep-subtitles SECONDS Number of seconds to sleep before each
subtitle download
## Video Format Options: ## Video Format Options:
-f, --format FORMAT Video format code, see "FORMAT SELECTION" -f, --format FORMAT Video format code, see "FORMAT SELECTION"
@@ -641,7 +652,7 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
similar syntax to the output template can similar syntax to the output template can
also be used. The parsed parameters replace also be used. The parsed parameters replace
any existing values and can be use in any existing values and can be use in
output templateThis option can be used output template. This option can be used
multiple times. Example: --parse-metadata multiple times. Example: --parse-metadata
"title:%(artist)s - %(title)s" matches a "title:%(artist)s - %(title)s" matches a
title like "Coldplay - Paradise". Example title like "Coldplay - Paradise". Example
@@ -686,6 +697,8 @@ Then simply run `make`. You can also run `make youtube_dlc` instead to compile o
directory directory
## Extractor Options: ## Extractor Options:
--extractor-retries RETRIES Number of retries for known extractor
errors (default is 10), or "infinite"
--allow-dynamic-mpd Process dynamic DASH manifests (default) --allow-dynamic-mpd Process dynamic DASH manifests (default)
(Alias: --no-ignore-dynamic-mpd) (Alias: --no-ignore-dynamic-mpd)
--ignore-dynamic-mpd Do not process dynamic DASH manifests --ignore-dynamic-mpd Do not process dynamic DASH manifests
@@ -805,7 +818,7 @@ The available fields are:
- `dislike_count` (numeric): Number of negative ratings of the video - `dislike_count` (numeric): Number of negative ratings of the video
- `repost_count` (numeric): Number of reposts of the video - `repost_count` (numeric): Number of reposts of the video
- `average_rating` (numeric): Average rating give by users, the scale used depends on the webpage - `average_rating` (numeric): Average rating give by users, the scale used depends on the webpage
- `comment_count` (numeric): Number of comments on the video - `comment_count` (numeric): Number of comments on the video (For some extractors, comments are only downloaded at the end, and so this field cannot be used)
- `age_limit` (numeric): Age restriction for the video (years) - `age_limit` (numeric): Age restriction for the video (years)
- `is_live` (boolean): Whether this video is a live stream or a fixed-length video - `is_live` (boolean): Whether this video is a live stream or a fixed-length video
- `was_live` (boolean): Whether this video was originally a live stream - `was_live` (boolean): Whether this video was originally a live stream

View File

@@ -8,7 +8,7 @@ import sys
sys.path.insert(0, dirn(dirn((os.path.abspath(__file__))))) sys.path.insert(0, dirn(dirn((os.path.abspath(__file__)))))
import yt_dlp import yt_dlp
BASH_COMPLETION_FILE = "yt-dlp.bash-completion" BASH_COMPLETION_FILE = "completions/bash/yt-dlp"
BASH_COMPLETION_TEMPLATE = "devscripts/bash-completion.in" BASH_COMPLETION_TEMPLATE = "devscripts/bash-completion.in"

View File

@@ -10,7 +10,7 @@ sys.path.insert(0, dirn(dirn((os.path.abspath(__file__)))))
import yt_dlp import yt_dlp
from yt_dlp.utils import shell_quote from yt_dlp.utils import shell_quote
FISH_COMPLETION_FILE = 'yt-dlp.fish' FISH_COMPLETION_FILE = 'completions/fish/yt-dlp.fish'
FISH_COMPLETION_TEMPLATE = 'devscripts/fish-completion.in' FISH_COMPLETION_TEMPLATE = 'devscripts/fish-completion.in'
EXTRA_ARGS = { EXTRA_ARGS = {

View File

@@ -8,7 +8,7 @@ import sys
sys.path.insert(0, dirn(dirn((os.path.abspath(__file__))))) sys.path.insert(0, dirn(dirn((os.path.abspath(__file__)))))
import yt_dlp import yt_dlp
ZSH_COMPLETION_FILE = "yt-dlp.zsh" ZSH_COMPLETION_FILE = "completions/zsh/_yt-dlp"
ZSH_COMPLETION_TEMPLATE = "devscripts/zsh-completion.in" ZSH_COMPLETION_TEMPLATE = "devscripts/zsh-completion.in"

5
docs/Changelog.md Normal file
View File

@@ -0,0 +1,5 @@
---
orphan: true
---
```{include} ../Changelog.md
```

6
docs/LICENSE.md Normal file
View File

@@ -0,0 +1,6 @@
---
orphan: true
---
# LICENSE
```{include} ../LICENSE
```

2
docs/README.md Normal file
View File

@@ -0,0 +1,2 @@
```{include} ../README.md
```

View File

@@ -7,26 +7,21 @@ import os
# Allows to import yt-dlp # Allows to import yt-dlp
sys.path.insert(0, os.path.abspath('..')) sys.path.insert(0, os.path.abspath('..'))
from recommonmark.transform import AutoStructify
# -- General configuration ------------------------------------------------ # -- General configuration ------------------------------------------------
# The suffix of source filenames.
source_suffix = ['.rst', '.md']
# Add any Sphinx extension module names here, as strings. They can be # Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones. # ones.
extensions = [ extensions = [
'sphinx.ext.autodoc', 'myst_parser',
'recommonmark',
] ]
# Add any paths that contain templates here, relative to this directory. # Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates'] templates_path = ['_templates']
# The master toctree document. # The master toctree document.
master_doc = 'index' master_doc = 'README'
# General information about the project. # General information about the project.
project = u'yt-dlp' project = u'yt-dlp'
@@ -64,12 +59,10 @@ highlight_language = 'none'
# so a file named "default.css" will overwrite the builtin "default.css". # so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static'] # html_static_path = ['_static']
# Enable heading anchors
myst_heading_anchors = 4
def setup(app): # Suppress heading warnings
app.add_config_value('recommonmark_config', { suppress_warnings = [
'enable_math': False, 'myst.header',
'enable_inline_math': False, ]
'enable_eval_rst': True,
'enable_auto_toc_tree': True,
}, True)
app.add_transform(AutoStructify)

View File

@@ -1 +0,0 @@
../README.md

View File

@@ -1,2 +1 @@
recommonmark>=0.6.0 myst-parser
m2r2

File diff suppressed because it is too large Load Diff

6
docs/ytdlp_plugins.md Normal file
View File

@@ -0,0 +1,6 @@
---
orphan: true
---
# ytdlp_plugins
See [https://github.com/yt-dlp/yt-dlp/tree/master/ytdlp_plugins](https://github.com/yt-dlp/yt-dlp/tree/master/ytdlp_plugins).

View File

@@ -27,8 +27,9 @@ if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
print("inv") print("inv")
else: else:
files_spec = [ files_spec = [
('etc/bash_completion.d', ['yt-dlp.bash-completion']), ('share/bash-completion/completions', ['completions/bash/*']),
('etc/fish/completions', ['yt-dlp.fish']), ('share/zsh/site-functions', ['completions/zsh/*']),
('share/fish/vendor_completions.d', ['completions/fish/*']),
('share/doc/yt_dlp', ['README.txt']), ('share/doc/yt_dlp', ['README.txt']),
('share/man/man1', ['yt-dlp.1']) ('share/man/man1', ['yt-dlp.1'])
] ]
@@ -86,7 +87,7 @@ setup(
#'Funding': 'https://donate.pypi.org', #'Funding': 'https://donate.pypi.org',
}, },
classifiers=[ classifiers=[
"Topic :: Multimedia :: Video", "Topic :: Multimedia :: Video",
"Development Status :: 5 - Production/Stable", "Development Status :: 5 - Production/Stable",
"Environment :: Console", "Environment :: Console",
"Programming Language :: Python", "Programming Language :: Python",
@@ -111,6 +112,6 @@ setup(
], ],
python_requires='>=2.6', python_requires='>=2.6',
cmdclass={'build_lazy_extractors': build_lazy_extractors}, cmdclass={'build_lazy_extractors': build_lazy_extractors},
**params **params
) )

1246
supportedsites.md Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -324,6 +324,8 @@ class YoutubeDL(object):
source_address: Client-side IP address to bind to. source_address: Client-side IP address to bind to.
call_home: Boolean, true iff we are allowed to contact the call_home: Boolean, true iff we are allowed to contact the
yt-dlp servers for debugging. (BROKEN) yt-dlp servers for debugging. (BROKEN)
sleep_interval_requests: Number of seconds to sleep between requests
during extraction
sleep_interval: Number of seconds to sleep before each download when sleep_interval: Number of seconds to sleep before each download when
used alone or a lower bound of a range for randomized used alone or a lower bound of a range for randomized
sleep before each download (minimum possible number sleep before each download (minimum possible number
@@ -334,6 +336,7 @@ class YoutubeDL(object):
Must only be used along with sleep_interval. Must only be used along with sleep_interval.
Actual sleep time will be a random float from range Actual sleep time will be a random float from range
[sleep_interval; max_sleep_interval]. [sleep_interval; max_sleep_interval].
sleep_interval_subtitles: Number of seconds to sleep before each subtitle download
listformats: Print an overview of available video formats and exit. listformats: Print an overview of available video formats and exit.
list_thumbnails: Print a table of all thumbnails and exit. list_thumbnails: Print a table of all thumbnails and exit.
match_filter: A function that gets called with the info_dict of match_filter: A function that gets called with the info_dict of
@@ -378,17 +381,18 @@ class YoutubeDL(object):
Use 'default' as the name for arguments to passed to all PP Use 'default' as the name for arguments to passed to all PP
The following options are used by the extractors: The following options are used by the extractors:
dynamic_mpd: Whether to process dynamic DASH manifests (default: True) extractor_retries: Number of times to retry for known errors
dynamic_mpd: Whether to process dynamic DASH manifests (default: True)
hls_split_discontinuity: Split HLS playlists to different formats at hls_split_discontinuity: Split HLS playlists to different formats at
discontinuities such as ad breaks (default: False) discontinuities such as ad breaks (default: False)
youtube_include_dash_manifest: If True (default), DASH manifests and related youtube_include_dash_manifest: If True (default), DASH manifests and related
data will be downloaded and processed by extractor. data will be downloaded and processed by extractor.
You can reduce network I/O by disabling it if you don't You can reduce network I/O by disabling it if you don't
care about DASH. (only for youtube) care about DASH. (only for youtube)
youtube_include_hls_manifest: If True (default), HLS manifests and related youtube_include_hls_manifest: If True (default), HLS manifests and related
data will be downloaded and processed by extractor. data will be downloaded and processed by extractor.
You can reduce network I/O by disabling it if you don't You can reduce network I/O by disabling it if you don't
care about HLS. (only for youtube) care about HLS. (only for youtube)
""" """
_NUMERIC_FIELDS = set(( _NUMERIC_FIELDS = set((
@@ -406,6 +410,7 @@ class YoutubeDL(object):
_ies = [] _ies = []
_pps = {'beforedl': [], 'aftermove': [], 'normal': []} _pps = {'beforedl': [], 'aftermove': [], 'normal': []}
__prepare_filename_warned = False __prepare_filename_warned = False
_first_webpage_request = True
_download_retcode = None _download_retcode = None
_num_downloads = None _num_downloads = None
_playlist_level = 0 _playlist_level = 0
@@ -420,6 +425,7 @@ class YoutubeDL(object):
self._ies_instances = {} self._ies_instances = {}
self._pps = {'beforedl': [], 'aftermove': [], 'normal': []} self._pps = {'beforedl': [], 'aftermove': [], 'normal': []}
self.__prepare_filename_warned = False self.__prepare_filename_warned = False
self._first_webpage_request = True
self._post_hooks = [] self._post_hooks = []
self._progress_hooks = [] self._progress_hooks = []
self._download_retcode = 0 self._download_retcode = 0
@@ -2036,6 +2042,7 @@ class YoutubeDL(object):
self.to_stdout(formatSeconds(info_dict['duration'])) self.to_stdout(formatSeconds(info_dict['duration']))
print_mandatory('format') print_mandatory('format')
if self.params.get('forcejson', False): if self.params.get('forcejson', False):
self.post_extract(info_dict)
self.to_stdout(json.dumps(info_dict)) self.to_stdout(json.dumps(info_dict))
def process_info(self, info_dict): def process_info(self, info_dict):
@@ -2059,6 +2066,7 @@ class YoutubeDL(object):
if self._match_entry(info_dict, incomplete=False) is not None: if self._match_entry(info_dict, incomplete=False) is not None:
return return
self.post_extract(info_dict)
self._num_downloads += 1 self._num_downloads += 1
info_dict = self.pre_process(info_dict) info_dict = self.pre_process(info_dict)
@@ -2166,15 +2174,6 @@ class YoutubeDL(object):
else: else:
try: try:
dl(sub_filename, sub_info, subtitle=True) dl(sub_filename, sub_info, subtitle=True)
'''
if self.params.get('sleep_interval_subtitles', False):
dl(sub_filename, sub_info)
else:
sub_data = ie._request_webpage(
sub_info['url'], info_dict['id'], note=False).read()
with io.open(encodeFilename(sub_filename), 'wb') as subfile:
subfile.write(sub_data)
'''
files_to_move[sub_filename] = sub_filename_final files_to_move[sub_filename] = sub_filename_final
except (ExtractorError, IOError, OSError, ValueError, compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err: except (ExtractorError, IOError, OSError, ValueError, compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
self.report_warning('Unable to download subtitle for "%s": %s' % self.report_warning('Unable to download subtitle for "%s": %s' %
@@ -2501,6 +2500,7 @@ class YoutubeDL(object):
raise raise
else: else:
if self.params.get('dump_single_json', False): if self.params.get('dump_single_json', False):
self.post_extract(res)
self.to_stdout(json.dumps(res)) self.to_stdout(json.dumps(res))
return self._download_retcode return self._download_retcode
@@ -2549,6 +2549,24 @@ class YoutubeDL(object):
del files_to_move[old_filename] del files_to_move[old_filename]
return files_to_move, infodict return files_to_move, infodict
@staticmethod
def post_extract(info_dict):
def actual_post_extract(info_dict):
if info_dict.get('_type') in ('playlist', 'multi_video'):
for video_dict in info_dict.get('entries', {}):
actual_post_extract(video_dict)
return
if '__post_extractor' not in info_dict:
return
post_extractor = info_dict['__post_extractor']
if post_extractor:
info_dict.update(post_extractor().items())
del info_dict['__post_extractor']
return
actual_post_extract(info_dict)
def pre_process(self, ie_info): def pre_process(self, ie_info):
info = dict(ie_info) info = dict(ie_info)
for pp in self._pps['beforedl']: for pp in self._pps['beforedl']:
@@ -2940,7 +2958,7 @@ class YoutubeDL(object):
self.to_screen('[%s] %s: Thumbnail %sis already present' % self.to_screen('[%s] %s: Thumbnail %sis already present' %
(info_dict['extractor'], info_dict['id'], thumb_display_id)) (info_dict['extractor'], info_dict['id'], thumb_display_id))
else: else:
self.to_screen('[%s] %s: Downloading thumbnail %s...' % self.to_screen('[%s] %s: Downloading thumbnail %s ...' %
(info_dict['extractor'], info_dict['id'], thumb_display_id)) (info_dict['extractor'], info_dict['id'], thumb_display_id))
try: try:
uf = self.urlopen(t['url']) uf = self.urlopen(t['url'])

View File

@@ -169,25 +169,33 @@ def _real_main(argv=None):
parser.error('max sleep interval must be greater than or equal to min sleep interval') parser.error('max sleep interval must be greater than or equal to min sleep interval')
else: else:
opts.max_sleep_interval = opts.sleep_interval opts.max_sleep_interval = opts.sleep_interval
if opts.sleep_interval_subtitles is not None:
if opts.sleep_interval_subtitles < 0:
parser.error('subtitles sleep interval must be positive or 0')
if opts.sleep_interval_requests is not None:
if opts.sleep_interval_requests < 0:
parser.error('requests sleep interval must be positive or 0')
if opts.ap_mso and opts.ap_mso not in MSO_INFO: if opts.ap_mso and opts.ap_mso not in MSO_INFO:
parser.error('Unsupported TV Provider, use --ap-list-mso to get a list of supported TV Providers') parser.error('Unsupported TV Provider, use --ap-list-mso to get a list of supported TV Providers')
if opts.overwrites: if opts.overwrites:
# --yes-overwrites implies --no-continue # --yes-overwrites implies --no-continue
opts.continue_dl = False opts.continue_dl = False
def parse_retries(retries): def parse_retries(retries, name=''):
if retries in ('inf', 'infinite'): if retries in ('inf', 'infinite'):
parsed_retries = float('inf') parsed_retries = float('inf')
else: else:
try: try:
parsed_retries = int(retries) parsed_retries = int(retries)
except (TypeError, ValueError): except (TypeError, ValueError):
parser.error('invalid retry count specified') parser.error('invalid %sretry count specified' % name)
return parsed_retries return parsed_retries
if opts.retries is not None: if opts.retries is not None:
opts.retries = parse_retries(opts.retries) opts.retries = parse_retries(opts.retries)
if opts.fragment_retries is not None: if opts.fragment_retries is not None:
opts.fragment_retries = parse_retries(opts.fragment_retries) opts.fragment_retries = parse_retries(opts.fragment_retries, 'fragment ')
if opts.extractor_retries is not None:
opts.extractor_retries = parse_retries(opts.extractor_retries, 'extractor ')
if opts.buffersize is not None: if opts.buffersize is not None:
numeric_buffersize = FileDownloader.parse_bytes(opts.buffersize) numeric_buffersize = FileDownloader.parse_bytes(opts.buffersize)
if numeric_buffersize is None: if numeric_buffersize is None:
@@ -262,6 +270,11 @@ def _real_main(argv=None):
any_printing = opts.print_json any_printing = opts.print_json
download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive
# If JSON is not printed anywhere, but comments are requested, save it to file
printing_json = opts.dumpjson or opts.print_json or opts.dump_single_json
if opts.getcomments and not printing_json:
opts.writeinfojson = True
def report_conflict(arg1, arg2): def report_conflict(arg1, arg2):
write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr) write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr)
if opts.remuxvideo and opts.recodevideo: if opts.remuxvideo and opts.recodevideo:
@@ -447,6 +460,7 @@ def _real_main(argv=None):
'overwrites': opts.overwrites, 'overwrites': opts.overwrites,
'retries': opts.retries, 'retries': opts.retries,
'fragment_retries': opts.fragment_retries, 'fragment_retries': opts.fragment_retries,
'extractor_retries': opts.extractor_retries,
'skip_unavailable_fragments': opts.skip_unavailable_fragments, 'skip_unavailable_fragments': opts.skip_unavailable_fragments,
'keep_fragments': opts.keep_fragments, 'keep_fragments': opts.keep_fragments,
'buffersize': opts.buffersize, 'buffersize': opts.buffersize,
@@ -466,7 +480,7 @@ def _real_main(argv=None):
'updatetime': opts.updatetime, 'updatetime': opts.updatetime,
'writedescription': opts.writedescription, 'writedescription': opts.writedescription,
'writeannotations': opts.writeannotations, 'writeannotations': opts.writeannotations,
'writeinfojson': opts.writeinfojson or opts.getcomments, 'writeinfojson': opts.writeinfojson,
'allow_playlist_files': opts.allow_playlist_files, 'allow_playlist_files': opts.allow_playlist_files,
'getcomments': opts.getcomments, 'getcomments': opts.getcomments,
'writethumbnail': opts.writethumbnail, 'writethumbnail': opts.writethumbnail,
@@ -524,6 +538,7 @@ def _real_main(argv=None):
'fixup': opts.fixup, 'fixup': opts.fixup,
'source_address': opts.source_address, 'source_address': opts.source_address,
'call_home': opts.call_home, 'call_home': opts.call_home,
'sleep_interval_requests': opts.sleep_interval_requests,
'sleep_interval': opts.sleep_interval, 'sleep_interval': opts.sleep_interval,
'max_sleep_interval': opts.max_sleep_interval, 'max_sleep_interval': opts.max_sleep_interval,
'sleep_interval_subtitles': opts.sleep_interval_subtitles, 'sleep_interval_subtitles': opts.sleep_interval_subtitles,
@@ -541,7 +556,6 @@ def _real_main(argv=None):
'postprocessor_args': opts.postprocessor_args, 'postprocessor_args': opts.postprocessor_args,
'cn_verification_proxy': opts.cn_verification_proxy, 'cn_verification_proxy': opts.cn_verification_proxy,
'geo_verification_proxy': opts.geo_verification_proxy, 'geo_verification_proxy': opts.geo_verification_proxy,
'config_location': opts.config_location,
'geo_bypass': opts.geo_bypass, 'geo_bypass': opts.geo_bypass,
'geo_bypass_country': opts.geo_bypass_country, 'geo_bypass_country': opts.geo_bypass_country,
'geo_bypass_ip_block': opts.geo_bypass_ip_block, 'geo_bypass_ip_block': opts.geo_bypass_ip_block,

View File

@@ -53,7 +53,7 @@ def get_suitable_downloader(info_dict, params={}, default=HttpFD):
external_downloader = params.get('external_downloader') external_downloader = params.get('external_downloader')
if external_downloader is not None: if external_downloader is not None:
ed = get_external_downloader(external_downloader) ed = get_external_downloader(external_downloader)
if ed.can_download(info_dict): if ed.can_download(info_dict, external_downloader):
return ed return ed
if protocol.startswith('m3u8'): if protocol.startswith('m3u8'):

View File

@@ -312,7 +312,7 @@ class FileDownloader(object):
def report_retry(self, err, count, retries): def report_retry(self, err, count, retries):
"""Report retry in case of HTTP error 5xx""" """Report retry in case of HTTP error 5xx"""
self.to_screen( self.to_screen(
'[download] Got server HTTP error: %s. Retrying (attempt %d of %s)...' '[download] Got server HTTP error: %s. Retrying (attempt %d of %s) ...'
% (error_to_compat_str(err), count, self.format_retries(retries))) % (error_to_compat_str(err), count, self.format_retries(retries)))
def report_file_already_downloaded(self, file_name): def report_file_already_downloaded(self, file_name):
@@ -359,7 +359,7 @@ class FileDownloader(object):
max_sleep_interval = self.params.get('max_sleep_interval', min_sleep_interval) max_sleep_interval = self.params.get('max_sleep_interval', min_sleep_interval)
sleep_interval = random.uniform(min_sleep_interval, max_sleep_interval) sleep_interval = random.uniform(min_sleep_interval, max_sleep_interval)
self.to_screen( self.to_screen(
'[download] Sleeping %s seconds...' % ( '[download] Sleeping %s seconds ...' % (
int(sleep_interval) if sleep_interval.is_integer() int(sleep_interval) if sleep_interval.is_integer()
else '%.2f' % sleep_interval)) else '%.2f' % sleep_interval))
time.sleep(sleep_interval) time.sleep(sleep_interval)
@@ -369,7 +369,7 @@ class FileDownloader(object):
sleep_interval_sub = self.params.get('sleep_interval_subtitles') sleep_interval_sub = self.params.get('sleep_interval_subtitles')
if sleep_interval_sub > 0: if sleep_interval_sub > 0:
self.to_screen( self.to_screen(
'[download] Sleeping %s seconds...' % ( '[download] Sleeping %s seconds ...' % (
sleep_interval_sub)) sleep_interval_sub))
time.sleep(sleep_interval_sub) time.sleep(sleep_interval_sub)
return self.real_download(filename, info_dict), True return self.real_download(filename, info_dict), True

View File

@@ -85,16 +85,16 @@ class ExternalFD(FileDownloader):
return self.params.get('external_downloader') return self.params.get('external_downloader')
@classmethod @classmethod
def available(cls): def available(cls, path=None):
return check_executable(cls.get_basename(), [cls.AVAILABLE_OPT]) return check_executable(path or cls.get_basename(), [cls.AVAILABLE_OPT])
@classmethod @classmethod
def supports(cls, info_dict): def supports(cls, info_dict):
return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS
@classmethod @classmethod
def can_download(cls, info_dict): def can_download(cls, info_dict, path=None):
return cls.available() and cls.supports(info_dict) return cls.available(path) and cls.supports(info_dict)
def _option(self, command_option, param): def _option(self, command_option, param):
return cli_option(self.params, command_option, param) return cli_option(self.params, command_option, param)
@@ -398,7 +398,10 @@ class FFmpegFD(ExternalFD):
args += ['-fs', compat_str(self._TEST_FILE_SIZE)] args += ['-fs', compat_str(self._TEST_FILE_SIZE)]
if protocol in ('m3u8', 'm3u8_native'): if protocol in ('m3u8', 'm3u8_native'):
if self.params.get('hls_use_mpegts', False) or tmpfilename == '-': use_mpegts = (tmpfilename == '-') or self.params.get('hls_use_mpegts')
if use_mpegts is None:
use_mpegts = info_dict.get('is_live')
if use_mpegts:
args += ['-f', 'mpegts'] args += ['-f', 'mpegts']
else: else:
args += ['-f', 'mp4'] args += ['-f', 'mp4']

View File

@@ -55,11 +55,11 @@ class FragmentFD(FileDownloader):
def report_retry_fragment(self, err, frag_index, count, retries): def report_retry_fragment(self, err, frag_index, count, retries):
self.to_screen( self.to_screen(
'[download] Got server HTTP error: %s. Retrying fragment %d (attempt %d of %s)...' '[download] Got server HTTP error: %s. Retrying fragment %d (attempt %d of %s) ...'
% (error_to_compat_str(err), frag_index, count, self.format_retries(retries))) % (error_to_compat_str(err), frag_index, count, self.format_retries(retries)))
def report_skip_fragment(self, frag_index): def report_skip_fragment(self, frag_index):
self.to_screen('[download] Skipping fragment %d...' % frag_index) self.to_screen('[download] Skipping fragment %d ...' % frag_index)
def _prepare_url(self, info_dict, url): def _prepare_url(self, info_dict, url):
headers = info_dict.get('http_headers') headers = info_dict.get('http_headers')
@@ -174,7 +174,7 @@ class FragmentFD(FileDownloader):
'.ytdl file is corrupt' if is_corrupt else '.ytdl file is corrupt' if is_corrupt else
'Inconsistent state of incomplete fragment download') 'Inconsistent state of incomplete fragment download')
self.report_warning( self.report_warning(
'%s. Restarting from the beginning...' % message) '%s. Restarting from the beginning ...' % message)
ctx['fragment_index'] = resume_len = 0 ctx['fragment_index'] = resume_len = 0
if 'ytdl_corrupt' in ctx: if 'ytdl_corrupt' in ctx:
del ctx['ytdl_corrupt'] del ctx['ytdl_corrupt']

View File

@@ -29,7 +29,7 @@ class NiconicoDmcFD(FileDownloader):
heartbeat_url = heartbeat_info_dict['url'] heartbeat_url = heartbeat_info_dict['url']
heartbeat_data = heartbeat_info_dict['data'] heartbeat_data = heartbeat_info_dict['data']
heartbeat_interval = heartbeat_info_dict.get('interval', 30) heartbeat_interval = heartbeat_info_dict.get('interval', 30)
self.to_screen('[%s] Heartbeat with %s second interval...' % (self.FD_NAME, heartbeat_interval)) self.to_screen('[%s] Heartbeat with %s second interval ...' % (self.FD_NAME, heartbeat_interval))
def heartbeat(): def heartbeat():
try: try:

View File

@@ -255,10 +255,6 @@ class BiliBiliIE(InfoExtractor):
info['uploader'] = self._html_search_meta( info['uploader'] = self._html_search_meta(
'author', webpage, 'uploader', default=None) 'author', webpage, 'uploader', default=None)
comments = None
if self._downloader.params.get('getcomments', False):
comments = self._get_all_comment_pages(video_id)
raw_danmaku = self._get_raw_danmaku(video_id, cid) raw_danmaku = self._get_raw_danmaku(video_id, cid)
raw_tags = self._get_tags(video_id) raw_tags = self._get_tags(video_id)
@@ -266,11 +262,18 @@ class BiliBiliIE(InfoExtractor):
top_level_info = { top_level_info = {
'raw_danmaku': raw_danmaku, 'raw_danmaku': raw_danmaku,
'comments': comments,
'comment_count': len(comments) if comments is not None else None,
'tags': tags, 'tags': tags,
'raw_tags': raw_tags, 'raw_tags': raw_tags,
} }
if self._downloader.params.get('getcomments', False):
def get_comments():
comments = self._get_all_comment_pages(video_id)
return {
'comments': comments,
'comment_count': len(comments)
}
top_level_info['__post_extractor'] = get_comments
''' '''
# Requires https://github.com/m13253/danmaku2ass which is licenced under GPL3 # Requires https://github.com/m13253/danmaku2ass which is licenced under GPL3
@@ -555,6 +558,7 @@ class BilibiliAudioIE(BilibiliAudioBaseIE):
formats = [{ formats = [{
'url': play_data['cdns'][0], 'url': play_data['cdns'][0],
'filesize': int_or_none(play_data.get('size')), 'filesize': int_or_none(play_data.get('size')),
'vcodec': 'none'
}] }]
song = self._call_api('song/info', au_id) song = self._call_api('song/info', au_id)

View File

@@ -294,6 +294,14 @@ class InfoExtractor(object):
players on other sites. Can be True (=always allowed), players on other sites. Can be True (=always allowed),
False (=never allowed), None (=unknown), or a string False (=never allowed), None (=unknown), or a string
specifying the criteria for embedability (Eg: 'whitelist'). specifying the criteria for embedability (Eg: 'whitelist').
__post_extractor: A function to be called just before the metadata is
written to either disk, logger or console. The function
must return a dict which will be added to the info_dict.
This is usefull for additional information that is
time-consuming to extract. Note that the fields thus
extracted will not be available to output template and
match_filter. So, only "comments" and "comment_count" are
currently allowed to be extracted via this method.
The following fields should only be used when the video belongs to some logical The following fields should only be used when the video belongs to some logical
chapter or section: chapter or section:
@@ -606,6 +614,14 @@ class InfoExtractor(object):
See _download_webpage docstring for arguments specification. See _download_webpage docstring for arguments specification.
""" """
if not self._downloader._first_webpage_request:
sleep_interval = float_or_none(self._downloader.params.get('sleep_interval_requests')) or 0
if sleep_interval > 0:
self.to_screen('Sleeping %s seconds ...' % sleep_interval)
time.sleep(sleep_interval)
else:
self._downloader._first_webpage_request = False
if note is None: if note is None:
self.report_download_webpage(video_id) self.report_download_webpage(video_id)
elif note is not False: elif note is not False:
@@ -1888,8 +1904,10 @@ class InfoExtractor(object):
# media playlist and MUST NOT appear in master playlist thus we can # media playlist and MUST NOT appear in master playlist thus we can
# clearly detect media playlist with this criterion. # clearly detect media playlist with this criterion.
def _extract_m3u8_playlist_formats(format_url, m3u8_doc=None): def _extract_m3u8_playlist_formats(format_url=None, m3u8_doc=None):
if not m3u8_doc: if not m3u8_doc:
if not format_url:
return []
res = self._download_webpage_handle( res = self._download_webpage_handle(
format_url, video_id, format_url, video_id,
note=False, note=False,
@@ -1928,7 +1946,7 @@ class InfoExtractor(object):
if '#EXT-X-TARGETDURATION' in m3u8_doc: # media playlist, return as is if '#EXT-X-TARGETDURATION' in m3u8_doc: # media playlist, return as is
playlist_formats = _extract_m3u8_playlist_formats(m3u8_doc, True) playlist_formats = _extract_m3u8_playlist_formats(m3u8_doc=m3u8_doc)
for format in playlist_formats: for format in playlist_formats:
format_id = [] format_id = []

View File

@@ -4,7 +4,7 @@ from __future__ import unicode_literals
import json import json
import re import re
from yt_dlp.utils import int_or_none, unified_timestamp, unescapeHTML from ..utils import int_or_none, unified_timestamp, unescapeHTML
from .common import InfoExtractor from .common import InfoExtractor

View File

@@ -86,6 +86,7 @@ class TennisTVIE(InfoExtractor):
'https://www.tennistv.com/api/users/v1/entitlementchecknondiva', 'https://www.tennistv.com/api/users/v1/entitlementchecknondiva',
video_id, note='Checking video authorization', headers=headers, data=check_json) video_id, note='Checking video authorization', headers=headers, data=check_json)
formats = self._extract_m3u8_formats(check_result['contentUrl'], video_id, ext='mp4') formats = self._extract_m3u8_formats(check_result['contentUrl'], video_id, ext='mp4')
self._sort_formats(formats)
vdata = self._download_json( vdata = self._download_json(
'https://www.tennistv.com/api/en/v2/none/common/video/%s' % video_id, 'https://www.tennistv.com/api/en/v2/none/common/video/%s' % video_id,

View File

@@ -255,15 +255,8 @@ class VikiIE(VikiBaseIE):
def _real_extract(self, url): def _real_extract(self, url):
video_id = self._match_id(url) video_id = self._match_id(url)
resp = self._download_json( video = self._call_api(
'https://www.viki.com/api/videos/' + video_id, 'videos/%s.json' % video_id, video_id, 'Downloading video JSON')
video_id, 'Downloading video JSON', headers={
'x-client-user-agent': std_headers['User-Agent'],
'x-viki-as-id': self._APP,
'x-viki-app-ver': self._APP_VERSION,
})
video = resp['video']
self._check_errors(video) self._check_errors(video)
title = self.dict_selection(video.get('titles', {}), 'en', allow_fallback=False) title = self.dict_selection(video.get('titles', {}), 'en', allow_fallback=False)
@@ -286,30 +279,12 @@ class VikiIE(VikiBaseIE):
}) })
subtitles = {} subtitles = {}
try: for subtitle_lang, _ in (video.get('subtitle_completions') or {}).items():
# New way to fetch subtitles subtitles[subtitle_lang] = [{
new_video = self._download_json( 'ext': subtitles_format,
'https://www.viki.com/api/videos/%s' % video_id, video_id, 'url': self._prepare_call(
'Downloading new video JSON to get subtitles', fatal=False, 'videos/%s/subtitles/%s.%s' % (video_id, subtitle_lang, subtitles_format)),
headers={ } for subtitles_format in ('srt', 'vtt')]
'x-client-user-agent': std_headers['User-Agent'],
'x-viki-as-id': self._APP,
'x-viki-app-ver': self._APP_VERSION,
})
for sub in new_video.get('streamSubtitles').get('dash'):
subtitles[sub.get('srclang')] = [{
'ext': 'vtt',
'url': sub.get('src'),
'completion': sub.get('percentage'),
}]
except AttributeError:
# fall-back to the old way if there isn't a streamSubtitles attribute
for subtitle_lang, _ in (video.get('subtitle_completions') or {}).items():
subtitles[subtitle_lang] = [{
'ext': subtitles_format,
'url': self._prepare_call(
'videos/%s/subtitles/%s.%s' % (video_id, subtitle_lang, subtitles_format)),
} for subtitles_format in ('srt', 'vtt')]
result = { result = {
'id': video_id, 'id': video_id,
@@ -386,23 +361,20 @@ class VikiIE(VikiBaseIE):
'filesize': int_or_none(urlh.headers.get('Content-Length')), 'filesize': int_or_none(urlh.headers.get('Content-Length')),
}) })
for format_id, format_dict in (resp.get('streams') or {}).items(): streams = self._call_api(
add_format(format_id, format_dict) 'videos/%s/streams.json' % video_id, video_id,
if not formats: 'Downloading video streams JSON')
streams = self._call_api(
'videos/%s/streams.json' % video_id, video_id,
'Downloading video streams JSON')
if 'external' in streams: if 'external' in streams:
result.update({ result.update({
'_type': 'url_transparent', '_type': 'url_transparent',
'url': streams['external']['url'], 'url': streams['external']['url'],
}) })
return result return result
for format_id, stream_dict in streams.items(): for format_id, stream_dict in streams.items():
for protocol, format_dict in stream_dict.items(): for protocol, format_dict in stream_dict.items():
add_format(format_id, format_dict, protocol) add_format(format_id, format_dict, protocol)
self._sort_formats(formats) self._sort_formats(formats)
result['formats'] = formats result['formats'] = formats

View File

@@ -1452,8 +1452,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
url, smuggled_data = unsmuggle_url(url, {}) url, smuggled_data = unsmuggle_url(url, {})
video_id = self._match_id(url) video_id = self._match_id(url)
base_url = self.http_scheme() + '//www.youtube.com/' base_url = self.http_scheme() + '//www.youtube.com/'
webpage_url = base_url + 'watch?v=' + video_id + '&has_verified=1&bpctr=9999999999' webpage_url = base_url + 'watch?v=' + video_id
webpage = self._download_webpage(webpage_url, video_id, fatal=False) webpage = self._download_webpage(
webpage_url + '&has_verified=1&bpctr=9999999999',
video_id, fatal=False)
player_response = None player_response = None
if webpage: if webpage:
@@ -2010,9 +2012,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
# Get comments # Get comments
# TODO: Refactor and move to seperate function # TODO: Refactor and move to seperate function
if get_comments: def extract_comments():
expected_video_comment_count = 0 expected_video_comment_count = 0
video_comments = [] video_comments = []
comment_xsrf = xsrf_token
def find_value(html, key, num_chars=2, separator='"'): def find_value(html, key, num_chars=2, separator='"'):
pos_begin = html.find(key) + len(key) + num_chars pos_begin = html.find(key) + len(key) + num_chars
@@ -2081,7 +2084,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
self.to_screen('Downloading comments') self.to_screen('Downloading comments')
while continuations: while continuations:
continuation = continuations.pop() continuation = continuations.pop()
comment_response = get_continuation(continuation, xsrf_token) comment_response = get_continuation(continuation, comment_xsrf)
if not comment_response: if not comment_response:
continue continue
if list(search_dict(comment_response, 'externalErrorMessage')): if list(search_dict(comment_response, 'externalErrorMessage')):
@@ -2092,7 +2095,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
continue continue
# not sure if this actually helps # not sure if this actually helps
if 'xsrf_token' in comment_response: if 'xsrf_token' in comment_response:
xsrf_token = comment_response['xsrf_token'] comment_xsrf = comment_response['xsrf_token']
item_section = comment_response['response']['continuationContents']['itemSectionContinuation'] item_section = comment_response['response']['continuationContents']['itemSectionContinuation']
if first_continuation: if first_continuation:
@@ -2121,7 +2124,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
while reply_continuations: while reply_continuations:
time.sleep(1) time.sleep(1)
continuation = reply_continuations.pop() continuation = reply_continuations.pop()
replies_data = get_continuation(continuation, xsrf_token, True) replies_data = get_continuation(continuation, comment_xsrf, True)
if not replies_data or 'continuationContents' not in replies_data[1]['response']: if not replies_data or 'continuationContents' not in replies_data[1]['response']:
continue continue
@@ -2150,10 +2153,13 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
time.sleep(1) time.sleep(1)
self.to_screen('Total comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count)) self.to_screen('Total comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
info.update({ return {
'comments': video_comments, 'comments': video_comments,
'comment_count': expected_video_comment_count 'comment_count': expected_video_comment_count
}) }
if get_comments:
info['__post_extractor'] = extract_comments
self.mark_watched(video_id, player_response) self.mark_watched(video_id, player_response)
@@ -2756,28 +2762,36 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
for page_num in itertools.count(1): for page_num in itertools.count(1):
if not continuation: if not continuation:
break break
count = 0 retries = self._downloader.params.get('extractor_retries', 3)
retries = 3 count = -1
while count <= retries: last_error = None
while count < retries:
count += 1
if last_error:
self.report_warning('%s. Retrying ...' % last_error)
try: try:
# Downloading page may result in intermittent 5xx HTTP error
# that is usually worked around with a retry
browse = self._download_json( browse = self._download_json(
'https://www.youtube.com/browse_ajax', None, 'https://www.youtube.com/browse_ajax', None,
'Downloading page %d%s' 'Downloading page %d%s'
% (page_num, ' (retry #%d)' % count if count else ''), % (page_num, ' (retry #%d)' % count if count else ''),
headers=headers, query=continuation) headers=headers, query=continuation)
break
except ExtractorError as e: except ExtractorError as e:
if isinstance(e.cause, compat_HTTPError) and e.cause.code in (500, 503): if isinstance(e.cause, compat_HTTPError) and e.cause.code in (500, 503, 404):
count += 1 # Downloading page may result in intermittent 5xx HTTP error
if count <= retries: # Sometimes a 404 is also recieved. See: https://github.com/ytdl-org/youtube-dl/issues/28289
last_error = 'HTTP Error %s' % e.cause.code
if count < retries:
continue continue
raise raise
if not browse: else:
break response = try_get(browse, lambda x: x[1]['response'], dict)
response = try_get(browse, lambda x: x[1]['response'], dict)
if not response: # Youtube sometimes sends incomplete data
# See: https://github.com/ytdl-org/youtube-dl/issues/28194
if response.get('continuationContents') or response.get('onResponseReceivedActions'):
break
last_error = 'Incomplete data recieved'
if not browse or not response:
break break
known_continuation_renderers = { known_continuation_renderers = {
@@ -2998,19 +3012,32 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
return self.url_result(video_id, ie=YoutubeIE.ie_key(), video_id=video_id) return self.url_result(video_id, ie=YoutubeIE.ie_key(), video_id=video_id)
self.to_screen('Downloading playlist %s - add --no-playlist to just download video %s' % (playlist_id, video_id)) self.to_screen('Downloading playlist %s - add --no-playlist to just download video %s' % (playlist_id, video_id))
webpage = self._download_webpage(url, item_id) retries = self._downloader.params.get('extractor_retries', 3)
identity_token = self._extract_identity_token(webpage, item_id) count = -1
data = self._extract_yt_initial_data(item_id, webpage) while count < retries:
err_msg = None count += 1
for alert_type, alert_message in self._extract_alerts(data): # Sometimes youtube returns a webpage with incomplete ytInitialData
if alert_type.lower() == 'error': # See: https://github.com/yt-dlp/yt-dlp/issues/116
if err_msg: if count:
self._downloader.report_warning('YouTube said: %s - %s' % ('ERROR', err_msg)) self.report_warning('Incomplete yt initial data recieved. Retrying ...')
err_msg = alert_message webpage = self._download_webpage(
else: url, item_id,
self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message)) 'Downloading webpage%s' % ' (retry #%d)' % count if count else '')
if err_msg: identity_token = self._extract_identity_token(webpage, item_id)
raise ExtractorError('YouTube said: %s' % err_msg, expected=True) data = self._extract_yt_initial_data(item_id, webpage)
err_msg = None
for alert_type, alert_message in self._extract_alerts(data):
if alert_type.lower() == 'error':
if err_msg:
self._downloader.report_warning('YouTube said: %s - %s' % ('ERROR', err_msg))
err_msg = alert_message
else:
self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
if err_msg:
raise ExtractorError('YouTube said: %s' % err_msg, expected=True)
if data.get('contents') or data.get('currentVideoEndpoint'):
break
tabs = try_get( tabs = try_get(
data, lambda x: x['contents']['twoColumnBrowseResultsRenderer']['tabs'], list) data, lambda x: x['contents']['twoColumnBrowseResultsRenderer']['tabs'], list)
if tabs: if tabs:

View File

@@ -347,7 +347,7 @@ def parseOpts(overrideArguments=None):
'Specify any key (see "OUTPUT TEMPLATE" for a list of available keys) to ' 'Specify any key (see "OUTPUT TEMPLATE" for a list of available keys) to '
'match if the key is present, ' 'match if the key is present, '
'!key to check if the key is not present, ' '!key to check if the key is not present, '
'key>NUMBER (like "comment_count > 12", also works with ' 'key>NUMBER (like "view_count > 12", also works with '
'>=, <, <=, !=, =) to compare against a number, ' '>=, <, <=, !=, =) to compare against a number, '
'key = \'LITERAL\' (like "uploader = \'Mike Smith\'", also works with !=) ' 'key = \'LITERAL\' (like "uploader = \'Mike Smith\'", also works with !=) '
'to match against a string literal ' 'to match against a string literal '
@@ -369,7 +369,7 @@ def parseOpts(overrideArguments=None):
help='Download only the video, if the URL refers to a video and a playlist') help='Download only the video, if the URL refers to a video and a playlist')
selection.add_option( selection.add_option(
'--yes-playlist', '--yes-playlist',
action='store_false', dest='noplaylist', default=False, action='store_false', dest='noplaylist',
help='Download the playlist, if the URL refers to a video and a playlist') help='Download the playlist, if the URL refers to a video and a playlist')
selection.add_option( selection.add_option(
'--age-limit', '--age-limit',
@@ -634,16 +634,24 @@ def parseOpts(overrideArguments=None):
help='Use ffmpeg instead of the native HLS downloader') help='Use ffmpeg instead of the native HLS downloader')
downloader.add_option( downloader.add_option(
'--hls-use-mpegts', '--hls-use-mpegts',
dest='hls_use_mpegts', action='store_true', dest='hls_use_mpegts', action='store_true', default=None,
help=( help=(
'Use the mpegts container for HLS videos, allowing to play the ' 'Use the mpegts container for HLS videos; '
'video while downloading (some players may not be able to play it)')) 'allowing some players to play the video while downloading, '
'and reducing the chance of file corruption if download is interrupted. '
'This is enabled by default for live streams'))
downloader.add_option(
'--no-hls-use-mpegts',
dest='hls_use_mpegts', action='store_false',
help=(
'Do not use the mpegts container for HLS videos. '
'This is default when not downloading live streams'))
downloader.add_option( downloader.add_option(
'--external-downloader', '--external-downloader',
dest='external_downloader', metavar='NAME', dest='external_downloader', metavar='NAME',
help=( help=(
'Use the specified external downloader. ' 'Name or path of the external downloader to use. '
'Currently supports %s' % ', '.join(list_external_downloaders()))) 'Currently supports %s (Recommended: aria2c)' % ', '.join(list_external_downloaders())))
downloader.add_option( downloader.add_option(
'--downloader-args', '--external-downloader-args', '--downloader-args', '--external-downloader-args',
metavar='NAME:ARGS', dest='external_downloader_args', default={}, type='str', metavar='NAME:ARGS', dest='external_downloader_args', default={}, type='str',
@@ -688,6 +696,10 @@ def parseOpts(overrideArguments=None):
'--bidi-workaround', '--bidi-workaround',
dest='bidi_workaround', action='store_true', dest='bidi_workaround', action='store_true',
help='Work around terminals that lack bidirectional text support. Requires bidiv or fribidi executable in PATH') help='Work around terminals that lack bidirectional text support. Requires bidiv or fribidi executable in PATH')
workarounds.add_option(
'--sleep-requests', metavar='SECONDS',
dest='sleep_interval_requests', type=float,
help='Number of seconds to sleep between requests during data extraction')
workarounds.add_option( workarounds.add_option(
'--sleep-interval', '--min-sleep-interval', metavar='SECONDS', '--sleep-interval', '--min-sleep-interval', metavar='SECONDS',
dest='sleep_interval', type=float, dest='sleep_interval', type=float,
@@ -706,7 +718,7 @@ def parseOpts(overrideArguments=None):
workarounds.add_option( workarounds.add_option(
'--sleep-subtitles', metavar='SECONDS', '--sleep-subtitles', metavar='SECONDS',
dest='sleep_interval_subtitles', default=0, type=int, dest='sleep_interval_subtitles', default=0, type=int,
help='Enforce sleep interval on subtitles as well') help='Number of seconds to sleep before each subtitle download')
verbosity = optparse.OptionGroup(parser, 'Verbosity and Simulation Options') verbosity = optparse.OptionGroup(parser, 'Verbosity and Simulation Options')
verbosity.add_option( verbosity.add_option(
@@ -973,7 +985,9 @@ def parseOpts(overrideArguments=None):
filesystem.add_option( filesystem.add_option(
'--get-comments', '--get-comments',
action='store_true', dest='getcomments', default=False, action='store_true', dest='getcomments', default=False,
help='Retrieve video comments to be placed in the .info.json file') help=(
'Retrieve video comments to be placed in the .info.json file. '
'The comments are fetched even without this option if the extraction is known to be quick'))
filesystem.add_option( filesystem.add_option(
'--load-info-json', '--load-info', '--load-info-json', '--load-info',
dest='load_info_filename', metavar='FILE', dest='load_info_filename', metavar='FILE',
@@ -1129,7 +1143,7 @@ def parseOpts(overrideArguments=None):
'Give field name to extract data from, and format of the field seperated by a ":". ' 'Give field name to extract data from, and format of the field seperated by a ":". '
'Either regular expression with named capture groups or a ' 'Either regular expression with named capture groups or a '
'similar syntax to the output template can also be used. ' 'similar syntax to the output template can also be used. '
'The parsed parameters replace any existing values and can be use in output template' 'The parsed parameters replace any existing values and can be use in output template. '
'This option can be used multiple times. ' 'This option can be used multiple times. '
'Example: --parse-metadata "title:%(artist)s - %(title)s" matches a title like ' 'Example: --parse-metadata "title:%(artist)s - %(title)s" matches a title like '
'"Coldplay - Paradise". ' '"Coldplay - Paradise". '
@@ -1204,6 +1218,10 @@ def parseOpts(overrideArguments=None):
help=optparse.SUPPRESS_HELP) help=optparse.SUPPRESS_HELP)
extractor = optparse.OptionGroup(parser, 'Extractor Options') extractor = optparse.OptionGroup(parser, 'Extractor Options')
extractor.add_option(
'--extractor-retries',
dest='extractor_retries', metavar='RETRIES', default=10,
help='Number of retries for known extractor errors (default is %default), or "infinite"')
extractor.add_option( extractor.add_option(
'--allow-dynamic-mpd', '--no-ignore-dynamic-mpd', '--allow-dynamic-mpd', '--no-ignore-dynamic-mpd',
action='store_true', dest='dynamic_mpd', default=True, action='store_true', dest='dynamic_mpd', default=True,

View File

@@ -193,4 +193,6 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
info['__thumbnail_filename'], os.path.splitext(original_thumbnail)[1][1:]) info['__thumbnail_filename'], os.path.splitext(original_thumbnail)[1][1:])
if original_thumbnail == thumbnail_filename: if original_thumbnail == thumbnail_filename:
files_to_delete = [] files_to_delete = []
elif original_thumbnail != thumbnail_filename:
files_to_delete.append(original_thumbnail)
return files_to_delete, info return files_to_delete, info

View File

@@ -5945,9 +5945,13 @@ def make_dir(path, to_screen=None):
def get_executable_path(): def get_executable_path():
path = os.path.dirname(sys.argv[0]) from zipimport import zipimporter
if os.path.basename(sys.argv[0]) == '__main__': # Running from source if hasattr(sys, 'frozen'): # Running from PyInstaller
path = os.path.join(path, '..') path = os.path.dirname(sys.executable)
elif isinstance(globals().get('__loader__'), zipimporter): # Running from ZIP
path = os.path.join(os.path.dirname(__file__), '../..')
else:
path = os.path.join(os.path.dirname(__file__), '..')
return os.path.abspath(path) return os.path.abspath(path)

View File

@@ -1,3 +1,3 @@
from __future__ import unicode_literals from __future__ import unicode_literals
__version__ = '2021.02.19' __version__ = '2021.02.24'