Release 2021.05.11

* and some documentation improvements

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.24.1. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.03.24.1**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.03.24.1
+ [debug] yt-dlp version 2021.04.22
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.24.1. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.03.24.1**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.24.1. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.03.24.1**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.24.1. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.03.24.1**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.03.24.1
+ [debug] yt-dlp version 2021.04.22
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.03.24.1. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.03.24.1**
+- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/core.yml

@@ -6,7 +6,7 @@ jobs:
     if: "!contains(github.event.head_commit.message, 'ci skip')"
     runs-on: ${{ matrix.os }}
     strategy:
-      fail-fast: true
+      fail-fast: false
       matrix:
         os: [ubuntu-18.04]
         # TODO: python 2.6
@@ -41,11 +41,18 @@ jobs:
     - name: Install Jython
       if: ${{ matrix.python-impl == 'jython' }}
       run: |
-        wget http://search.maven.org/remotecontent?filepath=org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
+        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
         java -jar jython-installer.jar -s -d "$HOME/jython"
         echo "$HOME/jython/bin" >> $GITHUB_PATH
     - name: Install nose
+      if: ${{ matrix.python-impl != 'jython' }}
       run: pip install nose
+    - name: Install nose (Jython)
+      if: ${{ matrix.python-impl == 'jython' }}
+      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
+      run: |
+        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
+        pip install nose-1.3.7-py2-none-any.whl
     - name: Run tests
       continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
       env:

.github/workflows/download.yml

@@ -41,11 +41,18 @@ jobs:
     - name: Install Jython
       if: ${{ matrix.python-impl == 'jython' }}
       run: |
-        wget http://search.maven.org/remotecontent?filepath=org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
+        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
         java -jar jython-installer.jar -s -d "$HOME/jython"
         echo "$HOME/jython/bin" >> $GITHUB_PATH
     - name: Install nose
+      if: ${{ matrix.python-impl != 'jython' }}
       run: pip install nose
+    - name: Install nose (Jython)
+      if: ${{ matrix.python-impl == 'jython' }}
+      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
+      run: |
+        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
+        pip install nose-1.3.7-py2-none-any.whl
     - name: Run tests
       continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
       env:

CONTRIBUTORS

@@ -35,3 +35,12 @@ damianoamatruda
 2ShedsJackson
 CXwudi
 xtkoba
+llacb47
+hheimbuerger
+B0pol
+lkho
+fstirlitz
+Lamieur
+tsukumijima
+Hadi0609
+b5eff52

Changelog.md

@@ -6,6 +6,7 @@
 * Run `make doc`
 * Update Changelog.md and CONTRIBUTORS
 * Change "Merged with ytdl" version in Readme.md if needed
+* Add new/fixed extractors in "new features" section of Readme.md
 * Commit to master as `Release <version>`
 * Push to origin/release using `git push origin master:release`
     build task will now run
@@ -18,8 +19,128 @@
 -->
 
 
+### 2021.05.11
+
+* **Deprecate support for python versions < 3.6**
+* **Subtitle extraction from manifests** by [fstirlitz](https://github.com/fstirlitz). See [be6202f12b97858b9d716e608394b51065d0419f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
+* **Improve output template:**
+    * Allow slicing lists/strings using `field.start:end:step`
+    * A field can also be used as offset like `field1+num+field2`
+    * A default value can be given using `field|default`
+    * Prevent invalid fields from causing errors
+* Merge youtube-dl: Upto [commit/a726009](https://github.com/ytdl-org/youtube-dl/commit/a7260099873acc6dc7d76cafad2f6b139087afd0)
+* **Remove options** `-l`, `-t`, `-A` completely and disable `--auto-number`, `--title`, `--literal`, `--id`
+* [Plugins] Prioritize plugins over standard extractors and prevent plugins from overwriting the standard extractor classes
+* [downloader] Fix `quiet` and `to_stderr`
+* [fragment] Ensure the file is closed on error
+* [fragment] Make sure first segment is not skipped
+* [aria2c] Fix whitespace being stripped off
+* [embedthumbnail] Fix bug where jpeg thumbnails were converted again
+* [FormatSort] Fix for when some formats have quality and others don't
+* [utils] Add `network_exceptions`
+* [utils] Escape URL while sanitizing
+* [ukcolumn] Add Extractor
+* [whowatch] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
+* [CBS] Improve `_VALID_URL` to support movies
+* [crackle] Improve extraction
+* [curiositystream] Fix collections
+* [francetvinfo] Improve video id extraction
+* [generic] Respect the encoding in manifest
+* [limelight] Obey `allow_unplayable_formats`
+* [mediasite] Generalize URL pattern by [fstirlitz](https://github.com/fstirlitz)
+* [mxplayer] Add MxplayerShowIE by [Ashish0804](https://github.com/Ashish0804)
+* [nebula] Move to nebula.app by [Lamieur](https://github.com/Lamieur)
+* [niconico] Fix HLS formats by [CXwudi](https://github.com/CXwudi), [tsukumijima](https://github.com/tsukumijima), [nao20010128nao](https://github.com/nao20010128nao) and [pukkandan](https://github.com/pukkandan)
+* [niconico] Fix title and thumbnail extraction by [CXwudi](https://github.com/CXwudi)
+* [plutotv] Extract subtitles from manifests
+* [plutotv] Fix format extraction for some urls
+* [rmcdecouverte] Improve `_VALID_URL`
+* [sonyliv] Fix `title` and `series` extraction by [Ashish0804](https://github.com/Ashish0804)
+* [tubi] Raise "no video formats" error when video url is empty
+* [youtube:tab] Detect playlists inside community posts
+* [youtube] Add `oembed` to reserved names
+* [zee5] Fix extraction for some URLs by [Hadi0609](https://github.com/Hadi0609)
+* [zee5] Fix py2 compatibility
+* Fix `playlist_index` and add `playlist_autonumber`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details
+* Add experimental option `--check-formats` to test the URLs before format selection
+* Option `--compat-options` to revert some of yt-dlp's changes
+    * Deprecates `--list-formats-as-table`, `--list-formats-old`
+* Fix number of digits in `%(playlist_index)s`
+* Fix case sensitivity of format selector
+* Revert "[core] be able to hand over id and title using url_result"
+* Do not strip out whitespaces in `-o` and `-P`
+* Fix `preload_download_archive` writing verbose message to `stdout`
+* Move option warnings to `YoutubeDL`so that they obey `--no-warnings` and can output colors
+* Py2 compatibility for `FileNotFoundError`
+
+
+### 2021.04.22
+* **Improve output template:**
+    * Objects can be traversed like `%(field.key1.key2)s`
+    * An offset can be added to numeric fields as `%(field+N)s`
+    * Deprecates `--autonumber-start`
+* **Improve `--sub-langs`:**
+    * Treat `--sub-langs` entries as regex
+    * `all` can be used to refer to all the subtitles
+    * language codes can be prefixed with `-` to exclude it
+    * Deprecates `--all-subs`
+* Add option `--ignore-no-formats-error` to ignore the "no video format" and similar errors
+* Add option `--skip-playlist-after-errors` to skip the rest of a playlist after a given number of errors are encountered
+* Merge youtube-dl: Upto [commit/7e8b3f9](https://github.com/ytdl-org/youtube-dl/commit/7e8b3f9439ebefb3a3a4e5da9c0bd2b595976438)
+* [downloader] Fix bug in downloader selection
+* [BilibiliChannel] Fix pagination by [nao20010128nao](https://github.com/nao20010128nao) and [pukkandan](https://github.com/pukkandan)
+* [rai] Add support for http formats by [nixxo](https://github.com/nixxo)
+* [TubiTv] Add TubiTvShowIE by [Ashish0804](https://github.com/Ashish0804)
+* [twitcasting] Fix extractor
+* [viu:ott] Fix extractor and support series by [lkho](https://github.com/lkho) and [pukkandan](https://github.com/pukkandan)
+* [youtube:tab] Show unavailable videos in playlists by [colethedj](https://github.com/colethedj)
+* [youtube:tab] Reload with unavailable videos for all playlists
+* [youtube] Ignore invalid stretch ratio
+* [youtube] Improve channel syncid extraction to support ytcfg by [colethedj](https://github.com/colethedj)
+* [youtube] Standardize API calls for tabs, mixes and search by [colethedj](https://github.com/colethedj)
+* [youtube] Bugfix in `_extract_ytcfg`
+* [mildom:user:vod] Download only necessary amount of pages
+* [mildom] Remove proxy completely by [fstirlitz](https://github.com/fstirlitz)
+* [go] Fix `_VALID_URL`
+* [MetadataFromField] Improve regex and add tests
+* [Exec] Ensure backward compatibility when the command contains `%`
+* [extractor] Fix inconsistent use of `report_warning`
+* Ensure `mergeall` selects best format when multistreams are disabled
+* Improve the yt-dlp.sh script by [fstirlitz](https://github.com/fstirlitz)
+* [lazy_extractor] Do not load plugins
+* [ci] Disable fail-fast
+* [documentation] Clarify which deprecated options still work
+* [documentation] Fix typos
+
+
+### 2021.04.11
+* Add option `--convert-thumbnails` (only jpg currently supported)
+* Format selector `mergeall` to download and merge all formats
+* Pass any field to `--exec` using similar syntax to output template
+* Choose downloader for each protocol using `--downloader PROTO:NAME`
+    * Alias `--downloader` for `--external-downloader`
+    * Added `native` as an option for the downloader
+* Merge youtube-dl: Upto [commit/4fb25ff](https://github.com/ytdl-org/youtube-dl/commit/4fb25ff5a3be5206bb72e5c4046715b1529fb2c7) (except vimeo)
+* [DiscoveryPlusIndia] Add DiscoveryPlusIndiaShowIE by [Ashish0804](https://github.com/Ashish0804)
+* [NFHSNetwork] Add extractor by [llacb47](https://github.com/llacb47)
+* [nebula] Add extractor (watchnebula.com) by [hheimbuerger](https://github.com/hheimbuerger)
+* [nitter] Fix extraction of reply tweets and update instance list by [B0pol](https://github.com/B0pol)
+* [nitter] Fix thumbnails by [B0pol](https://github.com/B0pol)
+* [youtube] Fix thumbnail URL
+* [youtube] Parse API parameters from initial webpage by [colethedj](https://github.com/colethedj)
+* [youtube] Extract comments' approximate timestamp by [colethedj](https://github.com/colethedj)
+* [youtube] Fix alert extraction
+* [bilibili] Fix uploader
+* [utils] Add `datetime_from_str` and `datetime_add_months` by [colethedj](https://github.com/colethedj)
+* Run some `postprocessors` before actual download
+* Improve argument parsing for `-P`, `-o`, `-S`
+* Fix some `m3u8` not obeying `--allow-unplayable-formats`
+* Fix default of `dynamic_mpd`
+* Deprecate `--all-formats`, `--include-ads`, `--hls-prefer-native`, `--hls-prefer-ffmpeg`
+* [documentation] Improvements
+
 ### 2021.04.03
-* Merge youtube-dl: Upto 2021.04.01 ([commit/654b4f4](https://github.com/ytdl-org/youtube-dl/commit/654b4f4ff2718f38b3182c1188c5d569c14cc70a))
+* Merge youtube-dl: Upto [commit/654b4f4](https://github.com/ytdl-org/youtube-dl/commit/654b4f4ff2718f38b3182c1188c5d569c14cc70a)
 * Ability to set a specific field in the file's metadata using `--parse-metadata`
 * Ability to select n'th best format like `-f bv*.2`
 * [DiscoveryPlus] Add discoveryplus.in

README.md

@@ -1,39 +1,44 @@
+<div align="center">
+
 # YT-DLP
+A command-line program to download videos from YouTube and many other [video platforms](supportedsites.md)
 
-[![Release version](https://img.shields.io/github/v/release/yt-dlp/yt-dlp?color=brightgreen&label=Release)](https://github.com/yt-dlp/yt-dlp/releases/latest)
-[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
+<!-- GHA doesn't have for-the-badge style
 [![CI Status](https://github.com/yt-dlp/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/yt-dlp/yt-dlp/actions)
-[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord)](https://discord.gg/H5MNcFW63r)  
+-->
+[![Release version](https://img.shields.io/github/v/release/yt-dlp/yt-dlp?color=brightgreen&label=Release&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/releases/latest)
+[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg?style=for-the-badge)](LICENSE)
+[![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest&style=for-the-badge)](https://yt-dlp.readthedocs.io)
+[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord&style=for-the-badge)](https://discord.gg/H5MNcFW63r)
+[![Commits](https://img.shields.io/github/commit-activity/m/yt-dlp/yt-dlp?label=commits&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
+[![Last Commit](https://img.shields.io/github/last-commit/yt-dlp/yt-dlp/master?style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
+[![Downloads](https://img.shields.io/github/downloads/yt-dlp/yt-dlp/total?style=for-the-badge&color=blue)](https://github.com/yt-dlp/yt-dlp/releases/latest)
+[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi&style=for-the-badge)](https://pypi.org/project/yt-dlp)
 
-[![Commits](https://img.shields.io/github/commit-activity/m/yt-dlp/yt-dlp?label=commits)](https://github.com/yt-dlp/yt-dlp/commits)
-[![Last Commit](https://img.shields.io/github/last-commit/yt-dlp/yt-dlp/master)](https://github.com/yt-dlp/yt-dlp/commits)
-[![Downloads](https://img.shields.io/github/downloads/yt-dlp/yt-dlp/total)](https://github.com/yt-dlp/yt-dlp/releases/latest)
-[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi)](https://pypi.org/project/yt-dlp)
-[![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest)](https://yt-dlp.readthedocs.io)
+</div>
 
-A command-line program to download videos from youtube.com and many other [video platforms](supportedsites.md)
-
-This is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on the now inactive [youtube-dlc](https://github.com/blackjack4494/yt-dlc). The main focus of this project is adding new features and patches while also keeping up to date with the original project
+yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on the now inactive [youtube-dlc](https://github.com/blackjack4494/yt-dlc). The main focus of this project is adding new features and patches while also keeping up to date with the original project
 
 * [NEW FEATURES](#new-features)
+    * [Differences in default behavior](#differences-in-default-behavior)
 * [INSTALLATION](#installation)
+    * [Dependencies](#dependencies)
     * [Update](#update)
     * [Compile](#compile)
-* [DESCRIPTION](#description)
-* [OPTIONS](#options)
+* [USAGE AND OPTIONS](#usage-and-options)
+    * [General Options](#general-options)
     * [Network Options](#network-options)
-    * [Geo Restriction](#geo-restriction)
+    * [Geo-restriction](#geo-restriction)
     * [Video Selection](#video-selection)
     * [Download Options](#download-options)
     * [Filesystem Options](#filesystem-options)
-    * [Thumbnail images](#thumbnail-images)
+    * [Thumbnail Options](#thumbnail-options)
     * [Internet Shortcut Options](#internet-shortcut-options)
     * [Verbosity and Simulation Options](#verbosity-and-simulation-options)
     * [Workarounds](#workarounds)
     * [Video Format Options](#video-format-options)
     * [Subtitle Options](#subtitle-options)
     * [Authentication Options](#authentication-options)
-    * [Adobe Pass Options](#adobe-pass-options)
     * [Post-processing Options](#post-processing-options)
     * [SponSkrub (SponsorBlock) Options](#sponskrub-sponsorblock-options)
     * [Extractor Options](#extractor-options)
@@ -51,6 +56,7 @@ This is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on the
 * [PLUGINS](#plugins)
 * [DEPRECATED OPTIONS](#deprecated-options)
 * [MORE](#more)
+</div>
 
 
 # NEW FEATURES
@@ -60,34 +66,39 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* **Merged with youtube-dl v2021.04.01**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl [commit/a726009](https://github.com/ytdl-org/youtube-dl/commit/a7260099873acc6dc7d76cafad2f6b139087afd0)**: (v2021.04.26) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
-* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
+* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
 
 * **Youtube improvements**:
-    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and supports downloading multiple pages of content
-    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs works correctly
+    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works and supports downloading multiple pages of content
+    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs work
+    * Youtube mixes supports downloading multiple pages of content
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
 
 * **Split video by chapters**: Videos can be split into multiple files based on chapters using `--split-chapters`
 
-* **Multithreaded fragment downloads**: Fragment downloads can be natively multi-threaded. Use `--concurrent-fragments` (`-N`) option to set the number of threads used
+* **Multi-threaded fragment downloads**: Download multiple fragments of m3u8/mpd videos in parallel. Use `--concurrent-fragments` (`-N`) option to set the number of threads used
 
-* **Aria2c with HLS/DASH**: You can use aria2c as the external downloader for DASH(mpd) and HLS(m3u8) formats. No more slow ffmpeg/native downloads
+* **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow
 
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina, rumble, tennistv, amcnetworks
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi
 
-* **Plugin extractors**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
+* **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [be6202f12b97858b9d716e608394b51065d0419f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 
 * **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to using `--paths` (`-P`)
 
 * **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details
 
-* **Other new options**: `--parse-metadata`, `--list-formats-as-table`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+* **Output template improvements**: Output templates can now have date-time formatting, numeric offsets, object traversal etc. See [output template](#output-template) for details. Even more advanced operations can also be done with the help of `--parse-metadata`
 
-* **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, Date/time formatting in `-o`, faster archive checking, more [format selection options](#format-selection) etc
+* **Other new options**: `--sleep-requests`, `--convert-thumbnails`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+
+* **Improvements**: Multiple `--postprocessor-args` and `--downloader-args`, faster archive checking, more [format selection options](#format-selection) etc
+
+* **Plugin extractors**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
 
 * **Self-updater**: The releases can be updated using `yt-dlp -U`
 
@@ -99,14 +110,40 @@ See [changelog](Changelog.md) or [commits](https://github.com/yt-dlp/yt-dlp/comm
 
 If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the amount of changes are very large. Compare [options](#options) and [supported sites](supportedsites.md) with youtube-dl's to get an idea of the massive number of features/patches [youtube-dlc](https://github.com/blackjack4494/yt-dlc) has accumulated.
 
+### Differences in default behavior
+
+Some of yt-dlp's default options are different from that of youtube-dl and youtube-dlc.
+
+1. The options `--id`, `--auto-number` (`-A`), `--title` (`-t`) and `--literal` (`-l`), no longer work. See [removed options](#Removed) for details
+1. `avconv` is not supported as as an alternative to `ffmpeg`
+1. The default [output template](#output-template) is `%(title)s [%(id)s].%(ext)s`. There is no real reason for this change. This was changed before yt-dlp was ever made public and now there are no plans to change it back to `%(title)s.%(id)s.%(ext)s`. Instead, you may use `--compat-options filename`
+1. The default [format sorting](sorting-formats) is different from youtube-dl and prefers higher resolution and better codecs rather than higher bitrates. You can use the `--format-sort` option to change this to any order you prefer, or use `--compat-options format-sort` to use youtube-dl's sorting order
+1. The default format selector is `bv*+ba/b`. This means that if a combined video + audio format that is better than the best video-only format is found, the former will be prefered. Use `-f bv+ba/b` or `--compat-options format-spec` to revert this
+1. Unlike youtube-dlc, yt-dlp does not allow merging multiple audio/video streams into one file by default (since this conflicts with the use of `-f bv*+ba`). If needed, this feature must be enabled using `--audio-multistreams` and `--video-multistreams`. You can also use `--compat-options multistreams` to enable both
+1. `--ignore-errors` is enabled by default. Use `--abort-on-error` or `--compat-options abort-on-error` to abort on errors instead
+1. When writing metadata files such as thumbnails, description or infojson, the same information (if available) is also written for playlists. Use `--no-write-playlist-metafiles` or `--compat-options no-playlist-metafiles` to not write these files
+1. `playlist_index` behaves differently when used with options like `--playlist-reverse` and `--playlist-items`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details. You can use `--compat-options playlist-index` if you want to keep the earlier behavior
+1. The output of `-F` is listed in a new format. Use `--compat-options list-formats` to revert this
+1. Youtube live chat (if available) is considered as a subtitle. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent live chat from downloading
+1. Youtube channel URLs are automatically redirected to `/video`. Either append a `/featured` to the URL or use `--compat-options no-youtube-channel-redirect` to download only the videos in the home page
+1. Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
+
+For ease of use, a few more compat options are available:
+1. `--compat-options all` = Use all compat options
+1. `--compat-options youtube-dl` = `--compat-options all,-multistreams`
+1. `--compat-options youtube-dlc` = `--compat-options all,-no-live-chat,-no-youtube-channel-redirect`
+
 
 # INSTALLATION
+yt-dlp is not platform specific. So it should work on your Unix box, on Windows or on macOS
 
 You can install yt-dlp using one of the following methods:
 * Download the binary from the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest) (recommended method)
-* Use [PyPI package](https://pypi.org/project/yt-dlp): `python -m pip install --upgrade yt-dlp`
-* Use pip+git: `python -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp.git@release`
-* Install master branch: `python -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp`
+* Use [PyPI package](https://pypi.org/project/yt-dlp): `python3 -m pip install --upgrade yt-dlp`
+* Use pip+git: `python3 -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp.git@release`
+* Install master branch: `python3 -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp`
+
+Note that on some systems, you may need to use `py` or `python` instead of `python3`
 
 UNIX users (Linux, macOS, BSD) can also install the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest) one of the following ways:
 
@@ -125,8 +162,15 @@ sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o
 sudo chmod a+rx /usr/local/bin/yt-dlp
 ```
 
+### DEPENDENCIES
+Python versions 3.6+ (CPython and PyPy) are officially supported. Other versions and implementations may or maynot work correctly.
+
+On windows, [Microsoft Visual C++ 2010 Redistributable Package (x86)](https://www.microsoft.com/en-us/download/details.aspx?id=26999) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it.
+
+Although there are no other required dependencies, `ffmpeg` and `ffprobe` are highly recommended. Other optional dependencies are `sponskrub`, `AtomicParsley`, `mutagen`, `pycryptodome` and any of the supported external downloaders. Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
+
 ### UPDATE
-Starting from version `2021.02.09`, you can use `yt-dlp -U` to update if you are using the provided release.
+You can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.
 
 ### COMPILE
@@ -134,13 +178,15 @@ If you are using `pip`, simply re-run the same command that was used to install
 **For Windows**:
 To build the Windows executable, you must have pyinstaller (and optionally mutagen and pycryptodome)
 
-    python -m pip install --upgrade pyinstaller mutagen pycryptodome
+    python3 -m pip install --upgrade pyinstaller mutagen pycryptodome
 
-Once you have all the necessary dependancies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
+Once you have all the necessary dependencies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it.
 
 You can also build the executable without any version info or metadata by using:
 
     pyinstaller.exe yt_dlp\__main__.py --onefile --name yt-dlp
+    
+Note that pyinstaller [does not support](https://github.com/pyinstaller/pyinstaller#requirements-and-tested-platforms) Python installed from the Windows store without using a virtual environment
 
 **For Unix**:
 You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `nosetests`  
@@ -148,15 +194,12 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
 
 **Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number
 
-# DESCRIPTION
-**yt-dlp** is a command-line program to download videos from youtube.com many other [video platforms](supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
+# USAGE AND OPTIONS
 
     yt-dlp [OPTIONS] [--] URL [URL...]
 
-
-# OPTIONS
 `Ctrl+F` is your friend :D
-<!-- Autogenerated -->
+<!-- Auto generated -->
 
 ## General Options:
     -h, --help                       Print this help text and exit
@@ -201,6 +244,11 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --mark-watched                   Mark videos watched (YouTube only)
     --no-mark-watched                Do not mark videos watched (default)
     --no-colors                      Do not emit color codes in output
+    --compat-options OPTS            Options that can help keep compatibility
+                                     with youtube-dl and youtube-dlc
+                                     configurations by reverting some of the
+                                     changes made in yt-dlp. See "Differences in
+                                     default behavior" for details
 
 ## Network Options:
     --proxy URL                      Use the specified HTTP/HTTPS/SOCKS proxy.
@@ -213,7 +261,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     -4, --force-ipv4                 Make all connections via IPv4
     -6, --force-ipv6                 Make all connections via IPv6
 
-## Geo Restriction:
+## Geo-restriction:
     --geo-verification-proxy URL     Use this proxy to verify the IP address for
                                      some geo-restricted sites. The default
                                      proxy specified by --proxy (or none, if the
@@ -296,14 +344,14 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      a file that is in the archive
     --break-on-reject                Stop the download process when encountering
                                      a file that has been filtered out
+    --skip-playlist-after-errors N   Number of allowed failures until the rest
+                                     of the playlist is skipped
     --no-download-archive            Do not use archive file (default)
-    --include-ads                    Download advertisements as well
-                                     (experimental)
-    --no-include-ads                 Do not download advertisements (default)
 
 ## Download Options:
-    -N, --concurrent-fragments N     Number of fragments to download
-                                     concurrently (default is 1)
+    -N, --concurrent-fragments N     Number of fragments of a dash/hlsnative
+                                     video that should be download concurrently
+                                     (default is 1)
     -r, --limit-rate RATE            Maximum download rate in bytes per second
                                      (e.g. 50K or 4.2M)
     -R, --retries RETRIES            Number of retries (default is 10), or
@@ -337,10 +385,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --playlist-random                Download playlist videos in random order
     --xattr-set-filesize             Set file xattribute ytdl.filesize with
                                      expected file size
-    --hls-prefer-native              Use the native HLS downloader instead of
-                                     ffmpeg
-    --hls-prefer-ffmpeg              Use ffmpeg instead of the native HLS
-                                     downloader
     --hls-use-mpegts                 Use the mpegts container for HLS videos;
                                      allowing some players to play the video
                                      while downloading, and reducing the chance
@@ -350,10 +394,19 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-hls-use-mpegts              Do not use the mpegts container for HLS
                                      videos. This is default when not
                                      downloading live streams
-    --external-downloader NAME       Name or path of the external downloader to
-                                     use. Currently supports aria2c, avconv,
-                                     axel, curl, ffmpeg, httpie, wget
-                                     (Recommended: aria2c)
+    --downloader [PROTO:]NAME        Name or path of the external downloader to
+                                     use (optionally) prefixed by the protocols
+                                     (http, ftp, m3u8, dash, rstp, rtmp, mms) to
+                                     use it for. Currently supports native,
+                                     aria2c, avconv, axel, curl, ffmpeg, httpie,
+                                     wget (Recommended: aria2c). You can use
+                                     this option multiple times to set different
+                                     downloaders for different protocols. For
+                                     example, --downloader aria2c --downloader
+                                     "dash,m3u8:native" will use aria2c for
+                                     http/ftp downloads, and the native
+                                     downloader for dash/m3u8 downloads
+                                     (Alias: --external-downloader)
     --downloader-args NAME:ARGS      Give these arguments to the external
                                      downloader. Specify the downloader name and
                                      the arguments separated by a colon ":". You
@@ -365,7 +418,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      stdin), one URL per line. Lines starting
                                      with '#', ';' or ']' are considered as
                                      comments and ignored
-    -P, --paths TYPE:PATH            The paths where the files should be
+    -P, --paths TYPES:PATH           The paths where the files should be
                                      downloaded. Specify the type of file and
                                      the path separated by a colon ":". All the
                                      same types as --output are supported.
@@ -376,13 +429,11 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      home path after download is finished. This
                                      option is ignored if --output is an
                                      absolute path
-    -o, --output [TYPE:]TEMPLATE     Output filename template, see "OUTPUT
+    -o, --output [TYPES:]TEMPLATE    Output filename template; see "OUTPUT
                                      TEMPLATE" for details
     --output-na-placeholder TEXT     Placeholder value for unavailable meta
                                      fields in output filename template
                                      (default: "NA")
-    --autonumber-start NUMBER        Specify the start value for %(autonumber)s
-                                     (default is 1)
     --restrict-filenames             Restrict filenames to only ASCII
                                      characters, and avoid "&" and spaces in
                                      filenames
@@ -402,7 +453,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     -c, --continue                   Resume partially downloaded files/fragments
                                      (default)
     --no-continue                    Do not resume partially downloaded
-                                     fragments. If the file is unfragmented,
+                                     fragments. If the file is not fragmented,
                                      restart download of the entire file
     --part                           Use .part files instead of writing directly
                                      into output file (default)
@@ -452,7 +503,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
 
-## Thumbnail Images:
+## Thumbnail Options:
     --write-thumbnail                Write thumbnail image to disk
     --no-write-thumbnail             Do not write thumbnail image to disk
                                      (default)
@@ -474,6 +525,12 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-warnings                    Ignore warnings
     -s, --simulate                   Do not download the video and do not write
                                      anything to disk
+    --ignore-no-formats-error        Ignore "No video formats" error. Usefull
+                                     for extracting metadata even if the video
+                                     is not actually available for download
+                                     (experimental)
+    --no-ignore-no-formats-error     Throw error when no downloadable video
+                                     formats are found (default)
     --skip-download                  Do not download the video but write all
                                      related files (Alias: --no-download)
     -g, --get-url                    Simulate, quiet but print URL
@@ -494,8 +551,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --print-json                     Be quiet and print the video information as
                                      JSON (video is still being downloaded)
     --force-write-archive            Force download archive entries to be
-                                     written as far as no errors occur,even if
-                                     -s or another simulation switch is used
+                                     written as far as no errors occur, even if
+                                     -s or another simulation option is used
                                      (Alias: --force-download-archive)
     --newline                        Output progress bar as new lines
     --no-progress                    Do not print progress bar
@@ -553,25 +610,22 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      into a single file
     --no-audio-multistreams          Only one audio stream is downloaded for
                                      each output file (default)
-    --all-formats                    Download all available video formats
     --prefer-free-formats            Prefer video formats with free containers
                                      over non-free ones of same quality. Use
                                      with "-S ext" to strictly prefer free
                                      containers irrespective of quality
     --no-prefer-free-formats         Don't give any special preference to free
                                      containers (default)
+    --check-formats                  Check that the formats selected are
+                                     actually downloadable (Experimental)
     -F, --list-formats               List all available formats of requested
                                      videos
-    --list-formats-as-table          Present the output of -F in tabular form
-                                     (default)
-    --list-formats-old               Present the output of -F in the old form
-                                     (Alias: --no-list-formats-as-table)
     --merge-output-format FORMAT     If a merge is required (e.g.
                                      bestvideo+bestaudio), output to given
                                      container format. One of mkv, mp4, ogg,
                                      webm, flv. Ignored if no merge is required
     --allow-unplayable-formats       Allow unplayable formats to be listed and
-                                     downloaded. All video postprocessing will
+                                     downloaded. All video post-processing will
                                      also be turned off
     --no-allow-unplayable-formats    Do not allow unplayable formats to be
                                      listed or downloaded (default)
@@ -583,15 +637,17 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      (Alias: --write-automatic-subs)
     --no-write-auto-subs             Do not write auto-generated subtitles
                                      (default) (Alias: --no-write-automatic-subs)
-    --all-subs                       Download all the available subtitles of the
-                                     video
     --list-subs                      List all available subtitles for the video
     --sub-format FORMAT              Subtitle format, accepts formats
                                      preference, for example: "srt" or
                                      "ass/srt/best"
-    --sub-langs LANGS                Languages of the subtitles to download
-                                     (optional) separated by commas, use --list-
-                                     subs for available language tags
+    --sub-langs LANGS                Languages of the subtitles to download (can
+                                     be regex) or "all" separated by commas.
+                                     (Eg: --sub-langs en.*,ja) You can prefix
+                                     the language code with a "-" to exempt it
+                                     from the requested languages. (Eg: --sub-
+                                     langs all,-live_chat) Use --list-subs for a
+                                     list of available language tags
 
 ## Authentication Options:
     -u, --username USERNAME          Login with this account ID
@@ -600,8 +656,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     -2, --twofactor TWOFACTOR        Two-factor authentication code
     -n, --netrc                      Use .netrc authentication data
     --video-password PASSWORD        Video password (vimeo, youku)
-
-## Adobe Pass Options:
     --ap-mso MSO                     Adobe Pass multiple-system operator (TV
                                      provider) identifier, use --ap-list-mso for
                                      a list of available MSOs
@@ -638,24 +692,24 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      Specify the postprocessor/executable name
                                      and the arguments separated by a colon ":"
                                      to give the argument to the specified
-                                     postprocessor/executable. Supported
-                                     postprocessors are: SponSkrub,
-                                     ExtractAudio, VideoRemuxer, VideoConvertor,
-                                     EmbedSubtitle, Metadata, Merger,
-                                     FixupStretched, FixupM4a, FixupM3u8,
-                                     SubtitlesConvertor, EmbedThumbnail and
-                                     SplitChapters. The supported executables
-                                     are: SponSkrub, FFmpeg, FFprobe, and
-                                     AtomicParsley. You can also specify
-                                     "PP+EXE:ARGS" to give the arguments to the
-                                     specified executable only when being used
-                                     by the specified postprocessor.
-                                     Additionally, for ffmpeg/ffprobe, "_i"/"_o"
-                                     can be appended to the prefix optionally
-                                     followed by a number to pass the argument
-                                     before the specified input/output file. Eg:
-                                     --ppa "Merger+ffmpeg_i1:-v quiet". You can
-                                     use this option multiple times to give
+                                     postprocessor/executable. Supported PP are:
+                                     Merger, ExtractAudio, SplitChapters,
+                                     Metadata, EmbedSubtitle, EmbedThumbnail,
+                                     SubtitlesConvertor, ThumbnailsConvertor,
+                                     VideoRemuxer, VideoConvertor, SponSkrub,
+                                     FixupStretched, FixupM4a and FixupM3u8. The
+                                     supported executables are: AtomicParsley,
+                                     FFmpeg, FFprobe, and SponSkrub. You can
+                                     also specify "PP+EXE:ARGS" to give the
+                                     arguments to the specified executable only
+                                     when being used by the specified
+                                     postprocessor. Additionally, for
+                                     ffmpeg/ffprobe, "_i"/"_o" can be appended
+                                     to the prefix optionally followed by a
+                                     number to pass the argument before the
+                                     specified input/output file. Eg: --ppa
+                                     "Merger+ffmpeg_i1:-v quiet". You can use
+                                     this option multiple times to give
                                      different arguments to different
                                      postprocessors. (Alias: --ppa)
     -k, --keep-video                 Keep the intermediate video file on disk
@@ -685,12 +739,19 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      path to the binary or its containing
                                      directory
     --exec CMD                       Execute a command on the file after
-                                     downloading and post-processing, similar to
-                                     find's -exec syntax. Example: --exec 'adb
-                                     push {} /sdcard/Music/ && rm {}'
+                                     downloading and post-processing. Similar
+                                     syntax to the output template can be used
+                                     to pass any field as arguments to the
+                                     command. An additional field "filepath"
+                                     that contains the final path of the
+                                     downloaded file is also available. If no
+                                     fields are passed, "%(filepath)s" is
+                                     appended to the end of the command
     --convert-subs FORMAT            Convert the subtitles to another format
                                      (currently supported: srt|ass|vtt|lrc)
                                      (Alias: --convert-subtitles)
+    --convert-thumbnails FORMAT      Convert the thumbnails to another format
+                                     (currently supported: jpg)
     --split-chapters                 Split video into multiple files based on
                                      internal chapters. The "chapter:" prefix
                                      can be used with "--paths" and "--output"
@@ -813,9 +874,22 @@ The `-o` option is used to indicate a template for the output file names while `
 
 **tl;dr:** [navigate me to examples](#output-template-examples).
 
-The basic usage of `-o` is not to set any template arguments when downloading a single file, like in `yt-dlp -o funny_video.flv "https://some/video"` (hard-coding file extension like this is not recommended). However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Date/time fields can also be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it inside the parantheses separated from the field name using a `>`. For example, `%(duration>%H-%M-%S)s`.
+The simplest usage of `-o` is not to set any template arguments when downloading a single file, like in `yt-dlp -o funny_video.flv "https://some/video"` (hard-coding file extension like this is _not_ recommended and could break some post-processing).
 
-Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon ":". The different filetypes supported are `subtitle`, `thumbnail`, `description`, `annotation`, `infojson`, `pl_description`, `pl_infojson`, `chapter`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
+It may however also contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations.
+
+The field names themselves (the part inside the parenthesis) can also have some special formatting:
+1. **Object traversal**: The dictionaries and lists available in metadata can be traversed by using a `.` (dot) separator. You can also do python slicing using `:`. Eg: `%(tags.0)s`, `%(subtitles.en.-1.ext)`, `%(id.3:7:-1)s`. Note that the fields that become available using this method are not listed below. Use `-j` to see such fields
+1. **Addition**: Addition and subtraction of numeric fields can be done using `+` and `-` respectively. Eg: `%(playlist_index+10)03d`, `%(n_entries+1-playlist_index)d`
+1. **Date/time Formatting**: Date/time fields can be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it separated from the field name using a `>`. Eg: `%(duration>%H-%M-%S)s`, `%(upload_date>%Y-%m-%d)s`, `%(epoch-3600>%H-%M-%S)s`
+1. **Default**: A default value can be specified for when the field is empty using a `|` seperator. This overrides `--output-na-template`. Eg: `%(uploader|Unknown)s`
+
+To summarize, the general syntax for a field is:
+```
+%(name[.keys][addition][>strf][|default])[flags][width][.precision][length]type
+```
+
+Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon `:`. The different file types supported are `subtitle`, `thumbnail`, `description`, `annotation`, `infojson`, `pl_description`, `pl_infojson`, `chapter`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
 
 The available fields are:
 
@@ -972,15 +1046,16 @@ The general syntax for format selection is `-f FORMAT` (or `--format FORMAT`) wh
 
 **tl;dr:** [navigate me to examples](#format-selection-examples).
 
-The simplest case is requesting a specific format, for example with `-f 22` you can download the format with format code equal to 22. You can get the list of available format codes for particular video using `--list-formats` or `-F`. Note that these format codes are extractor specific. 
+The simplest case is requesting a specific format, for example with `-f 22` you can download the format with format code equal to 22. You can get the list of available format codes for particular video using `--list-formats` or `-F`. Note that these format codes are extractor specific.
 
 You can also use a file extension (currently `3gp`, `aac`, `flv`, `m4a`, `mp3`, `mp4`, `ogg`, `wav`, `webm` are supported) to download the best quality format of a particular file extension served as a single file, e.g. `-f webm` will download the best quality format with the `webm` extension served as a single file.
 
 You can also use special names to select particular edge case formats:
 
  - `all`: Select all formats
- - `b*`, `best*`: Select the best quality format irrespective of whether it contains video or audio.
- - `w*`, `worst*`: Select the worst quality format irrespective of whether it contains video or audio.
+ - `mergeall`: Select and merge all formats (Must be used with `--audio-multistreams`, `--video-multistreams` or both)
+ - `b*`, `best*`: Select the best quality format irrespective of whether it contains video or audio
+ - `w*`, `worst*`: Select the worst quality format irrespective of whether it contains video or audio
  - `b`, `best`: Select the best quality format that contains both video and audio. Equivalent to `best*[vcodec!=none][acodec!=none]`
  - `w`, `worst`: Select the worst quality format that contains both video and audio. Equivalent to `worst*[vcodec!=none][acodec!=none]`
  - `bv`, `bestvideo`: Select the best quality video-only format. Equivalent to `best*[acodec=none]`
@@ -992,7 +1067,7 @@ You can also use special names to select particular edge case formats:
  - `ba*`, `bestaudio*`: Select the best quality format that contains audio. It may also contain video. Equivalent to `best*[acodec!=none]`
  - `wa*`, `worstaudio*`: Select the worst quality format that contains audio. It may also contain video. Equivalent to `worst*[acodec!=none]`
 
-For example, to download the worst quality video-only format you can use `-f worstvideo`. It is however recomended not to use `worst` and related options. When your format selector is `worst`, the format which is worst in all respects is selected. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps` instead of `-f worst`. See [sorting formats](#sorting-formats) for more details.
+For example, to download the worst quality video-only format you can use `-f worstvideo`. It is however recommended not to use `worst` and related options. When your format selector is `worst`, the format which is worst in all respects is selected. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps` instead of `-f worst`. See [sorting formats](#sorting-formats) for more details.
 
 You can select the n'th best format of a type by using `best<type>.<n>`. For example, `best.2` will select the 2nd best combined format. Similarly, `bv*.3` will select the 3rd best format that contains a video stream.
 
@@ -1029,7 +1104,7 @@ Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends
 
 Any string comparison may be prefixed with negation `!` in order to produce an opposite comparison, e.g. `!*=` (does not contain).
 
-Note that none of the aforementioned meta fields are guaranteed to be present since this solely depends on the metadata obtained by particular extractor, i.e. the metadata offered by the video hoster. Any other field made available by the extractor can also be used for filtering.
+Note that none of the aforementioned meta fields are guaranteed to be present since this solely depends on the metadata obtained by particular extractor, i.e. the metadata offered by the website. Any other field made available by the extractor can also be used for filtering.
 
 Formats for which the value is not known are excluded unless you put a question mark (`?`) after the operator. You can combine format filters, so `-f "[height<=?720][tbr>500]"` selects up to 720p videos (or videos where the height is not known) with a bitrate of at least 500 KBit/s. You can also use the filters with `all` to download all formats that satisfy the filter. For example, `-f "all[vcodec=none]"` selects all audio-only formats.
 
@@ -1045,7 +1120,7 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
  - `lang`: Language preference as given by the extractor
  - `quality`: The quality of the format as given by the extractor
  - `source`: Preference of the source as given by the extractor
- - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
+ - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8_native` > `m3u8` > `http_dash_segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
  - `vcodec`: Video Codec (`av01` > `vp9.2` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
  - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
  - `codec`: Equivalent to `vcodec,acodec`
@@ -1065,11 +1140,11 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
  - `br`: Equivalent to using `tbr,vbr,abr`
  - `asr`: Audio sample rate in Hz
 
-Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
+Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in descending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a preferred value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two preferred values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
 The fields `hasvid`, `ie_pref`, `lang` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
 
-If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
+If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all respects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 
 **Tip**: You can use the `-v -F` to see how the formats have been sorted (worst to best).
 
@@ -1078,7 +1153,7 @@ If your format selector is `worst`, the last item is selected after sorting. Thi
 Note that on Windows you may need to use double quotes instead of single.
 
 ```bash
-# Download and merge the best best video-only format and the best audio-only format,
+# Download and merge the best video-only format and the best audio-only format,
 # or download the best combined format if video-only format is not available
 $ yt-dlp -f 'bv+ba/b'
 
@@ -1094,10 +1169,17 @@ $ yt-dlp
 # by default, bestvideo and bestaudio will have the same file name.
 $ yt-dlp -f 'bv,ba' -o '%(title)s.f%(format_id)s.%(ext)s'
 
+# Download and merge the best format that has a video stream,
+# and all audio-only formats into one file
+$ yt-dlp -f 'bv*+mergeall[vcodec=none]' --audio-multistreams
+
+# Download and merge the best format that has a video stream,
+# and the best 2 audio-only formats into one file
+$ yt-dlp -f 'bv*+ba+ba.2' --audio-multistreams
 
 
 # The following examples show the old method (without -S) of format selection
-# and how to use -S to achieve a similar but better result
+# and how to use -S to achieve a similar but (generally) better result
 
 # Download the worst video available (old method)
 $ yt-dlp -f 'wv*+wa/w'
@@ -1173,26 +1255,26 @@ $ yt-dlp -S '+codec:h264'
 
 # More complex examples
 
-# Download the best video no better than 720p prefering framerate greater than 30,
-# or the worst video (still prefering framerate greater than 30) if there is no such video
+# Download the best video no better than 720p preferring framerate greater than 30,
+# or the worst video (still preferring framerate greater than 30) if there is no such video
 $ yt-dlp -f '((bv*[fps>30]/bv*)[height<=720]/(wv*[fps>30]/wv*)) + ba / (b[fps>30]/b)[height<=720]/(w[fps>30]/w)'
 
 # Download the video with the largest resolution no better than 720p,
-# or the video with the smallest resolution available  if there is no such video,
-# prefering larger framerate for formats with the same resolution
+# or the video with the smallest resolution available if there is no such video,
+# preferring larger framerate for formats with the same resolution
 $ yt-dlp -S 'res:720,fps'
 
 
 
 # Download the video with smallest resolution no worse than 480p,
 # or the video with the largest resolution available if there is no such video,
-# prefering better codec and then larger total bitrate for the same resolution
+# preferring better codec and then larger total bitrate for the same resolution
 $ yt-dlp -S '+res:480,codec,br'
 ```
 
 # MODIFYING METADATA
 
-The metadata obtained the the extractors can be modified by using `--parse-metadata FROM:TO`. The general syntax is to give the name of a field or a template (with similar syntax to [output template](#output-template)) to extract data from, and the format to interpret it as, separated by a colon ":". Either a [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax) with named capture groups or a similar syntax to the [output template](#output-template) (only `%(field)s` formatting is supported) can be used for `TO`. The option can be used multiple times to parse and modify various fields.
+The metadata obtained the the extractors can be modified by using `--parse-metadata FROM:TO`. The general syntax is to give the name of a field or a template (with similar syntax to [output template](#output-template)) to extract data from, and the format to interpret it as, separated by a colon `:`. Either a [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax) with named capture groups or a similar syntax to the [output template](#output-template) (only `%(field)s` formatting is supported) can be used for `TO`. The option can be used multiple times to parse and modify various fields.
 
 Note that any field created by this can be used in the [output template](#output-template) and will also affect the media file's metadata added when using `--add-metadata`.
 
@@ -1227,34 +1309,59 @@ Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently
 
 These are all the deprecated options and the current alternative to achieve the same effect
 
+#### Not recommended
+While these options still work, their use is not recommended since there are other alternatives to achieve the same
+
+    --all-formats                    -f all
+    --all-subs                       --sub-langs all --write-subs
+    --autonumber-size NUMBER         Use string formatting. Eg: %(autonumber)03d
+    --autonumber-start NUMBER        Use internal field formatting like %(autonumber+NUMBER)s
+    --metadata-from-title FORMAT     --parse-metadata "%(title)s:FORMAT"
+    --hls-prefer-native              --downloader "m3u8:native"
+    --hls-prefer-ffmpeg              --downloader "m3u8:ffmpeg"
+    --list-formats-old               --compat-options list-formats (Alias: --no-list-formats-as-table)
+    --list-formats-as-table          --compat-options -list-formats [Default] (Alias: --no-list-formats-old)
+    --sponskrub-args ARGS            --ppa "sponskrub:ARGS"
+    --test                           Used by developers for testing extractors. Not intended for the end user
+
+
+#### Old aliases
+These are aliases that are no longer documented for various reasons
+
+    --avconv-location                --ffmpeg-location
     --cn-verification-proxy URL      --geo-verification-proxy URL
+    --dump-headers                   --print-traffic
+    --dump-intermediate-pages        --dump-pages
+    --force-write-download-archive   --force-write-archive
+    --load-info                      --load-info-json
+    --no-split-tracks                --no-split-chapters
+    --no-write-srt                   --no-write-subs
+    --prefer-unsecure                --prefer-insecure
+    --rate-limit RATE                --limit-rate RATE
+    --split-tracks                   --split-chapters
+    --srt-lang LANGS                 --sub-langs LANGS
+    --trim-file-names LENGTH         --trim-filenames LENGTH
+    --write-srt                      --write-subs
+    --yes-overwrites                 --force-overwrites
+
+#### No longer supported
+These options may no longer work as intended
+
+    --prefer-avconv                  avconv is not officially supported by yt-dlp (Alias: --no-prefer-ffmpeg)
+    --prefer-ffmpeg                  Default (Alias: --no-prefer-avconv)
+    -C, --call-home                  Not implemented
+    --no-call-home                   Default
+    --include-ads                    No longer supported
+    --no-include-ads                 Default
+    --youtube-print-sig-code         No longer supported
+
+#### Removed
+These options were deprecated since 2014 and have now been entirely removed
+
     --id                             -o "%(id)s.%(ext)s"
     -A, --auto-number                -o "%(autonumber)s-%(id)s.%(ext)s"
     -t, --title                      -o "%(title)s-%(id)s.%(ext)s"
     -l, --literal                    -o accepts literal names
-    --autonumber-size NUMBER         Use string formatting. Eg: %(autonumber)03d
-    --metadata-from-title FORMAT     --parse-metadata "%(title)s:FORMAT"
-    --prefer-avconv                  avconv is no longer officially supported (Alias: --no-prefer-ffmpeg)
-    --prefer-ffmpeg                  Default (Alias: --no-prefer-avconv)
-    --avconv-location                avconv is no longer officially supported
-    -C, --call-home                  Not implemented
-    --no-call-home                   Default
-    --write-srt                      --write-subs
-    --no-write-srt                   --no-write-subs
-    --srt-lang LANGS                 --sub-langs LANGS
-    --prefer-unsecure                --prefer-insecure
-    --rate-limit RATE                --limit-rate RATE
-    --force-write-download-archive   --force-write-archive
-    --dump-intermediate-pages        --dump-pages
-    --dump-headers                   --print-traffic
-    --youtube-print-sig-code         No longer supported
-    --trim-file-names LENGTH         --trim-filenames LENGTH
-    --yes-overwrites                 --force-overwrites
-    --load-info                      --load-info-json
-    --split-tracks                   --split-chapters
-    --no-split-tracks                --no-split-chapters
-    --sponskrub-args ARGS            --ppa "sponskrub:ARGS"
-    --test                           Only used for testing extractors
 
 
 # MORE

devscripts/make_lazy_extractors.py

@@ -14,9 +14,14 @@ lazy_extractors_filename = sys.argv[1]
 if os.path.exists(lazy_extractors_filename):
     os.remove(lazy_extractors_filename)
 
+# Block plugins from loading
+os.rename('ytdlp_plugins', 'ytdlp_plugins_blocked')
+
 from yt_dlp.extractor import _ALL_CLASSES
 from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor
 
+os.rename('ytdlp_plugins_blocked', 'ytdlp_plugins')
+
 with open('devscripts/lazy_load_template.py', 'rt') as f:
     module_template = f.read()
 

pyinst.py

@@ -69,6 +69,7 @@ PyInstaller.__main__.run([
     '--onefile',
     '--icon=devscripts/cloud.ico',
     '--exclude-module=youtube_dl',
+    '--exclude-module=youtube_dlc',
     '--exclude-module=test',
     '--exclude-module=ytdlp_plugins',
     '--hidden-import=mutagen',

setup.py

@@ -18,13 +18,13 @@ DESCRIPTION = 'Command-line program to download videos from YouTube.com and many
 LONG_DESCRIPTION = '\n\n'.join((
     'Official repository: <https://github.com/yt-dlp/yt-dlp>',
     '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
-    open("README.md", "r", encoding="utf-8").read()))
+    open('README.md', 'r', encoding='utf-8').read()))
 
 REQUIREMENTS = ['mutagen', 'pycryptodome']
 
 
 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
-    print("inv")
+    print('inv')
 else:
     files_spec = [
         ('share/bash-completion/completions', ['completions/bash/yt-dlp']),
@@ -67,17 +67,17 @@ class build_lazy_extractors(Command):
         )
 
 
-packages = find_packages(exclude=("youtube_dl", "test", "ytdlp_plugins"))
+packages = find_packages(exclude=('youtube_dl', 'test', 'ytdlp_plugins'))
 
 setup(
-    name="yt-dlp",
+    name='yt-dlp',
     version=__version__,
-    maintainer="pukkandan",
-    maintainer_email="pukkandan.ytdlp@gmail.com",
+    maintainer='pukkandan',
+    maintainer_email='pukkandan.ytdlp@gmail.com',
     description=DESCRIPTION,
     long_description=LONG_DESCRIPTION,
-    long_description_content_type="text/markdown",
-    url="https://github.com/yt-dlp/yt-dlp",
+    long_description_content_type='text/markdown',
+    url='https://github.com/yt-dlp/yt-dlp',
     packages=packages,
     install_requires=REQUIREMENTS,
     project_urls={
@@ -87,28 +87,28 @@ setup(
         #'Funding': 'https://donate.pypi.org',
     },
     classifiers=[
-        "Topic :: Multimedia :: Video",
-        "Development Status :: 5 - Production/Stable",
-        "Environment :: Console",
-        "Programming Language :: Python",
-        "Programming Language :: Python :: 2",
-        "Programming Language :: Python :: 2.6",
-        "Programming Language :: Python :: 2.7",
-        "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.2",
-        "Programming Language :: Python :: 3.3",
-        "Programming Language :: Python :: 3.4",
-        "Programming Language :: Python :: 3.5",
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: Implementation",
-        "Programming Language :: Python :: Implementation :: CPython",
-        "Programming Language :: Python :: Implementation :: IronPython",
-        "Programming Language :: Python :: Implementation :: Jython",
-        "Programming Language :: Python :: Implementation :: PyPy",
-        "License :: Public Domain",
-        "Operating System :: OS Independent",
+        'Topic :: Multimedia :: Video',
+        'Development Status :: 5 - Production/Stable',
+        'Environment :: Console',
+        'Programming Language :: Python',
+        'Programming Language :: Python :: 2',
+        'Programming Language :: Python :: 2.6',
+        'Programming Language :: Python :: 2.7',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.2',
+        'Programming Language :: Python :: 3.3',
+        'Programming Language :: Python :: 3.4',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: Implementation',
+        'Programming Language :: Python :: Implementation :: CPython',
+        'Programming Language :: Python :: Implementation :: IronPython',
+        'Programming Language :: Python :: Implementation :: Jython',
+        'Programming Language :: Python :: Implementation :: PyPy',
+        'License :: Public Domain',
+        'Operating System :: OS Independent',
     ],
     python_requires='>=2.6',
 

supportedsites.md

@@ -3,6 +3,7 @@
  - **20min**
  - **220.ro**
  - **23video**
+ - **247sports**
  - **24video**
  - **3qsdn**: 3Q SDN
  - **3sat**
@@ -129,7 +130,6 @@
  - **bitwave:stream**
  - **BleacherReport**
  - **BleacherReportCMS**
- - **blinkx**
  - **Bloomberg**
  - **BokeCC**
  - **BongaCams**
@@ -171,7 +171,8 @@
  - **cbsnews**: CBS News
  - **cbsnews:embed**
  - **cbsnews:livevideo**: CBS News Live Videos
- - **CBSSports**
+ - **cbssports**
+ - **cbssports:embed**
  - **CCMA**
  - **CCTV**: 央视网
  - **CDA**
@@ -223,7 +224,8 @@
  - **Culturebox**
  - **CultureUnplugged**
  - **curiositystream**
- - **curiositystream:collection**
+ - **curiositystream:collections**
+ - **curiositystream:series**
  - **CWTV**
  - **DagelijkseKost**: dagelijksekost.een.be
  - **DailyMail**
@@ -250,6 +252,7 @@
  - **DiscoveryNetworksDe**
  - **DiscoveryPlus**
  - **DiscoveryPlusIndia**
+ - **DiscoveryPlusIndiaShow**
  - **DiscoveryVR**
  - **Disney**
  - **dlive:stream**
@@ -483,6 +486,8 @@
  - **limelight**
  - **limelight:channel**
  - **limelight:channel_list**
+ - **LineLive**
+ - **LineLiveChannel**
  - **LineTV**
  - **linkedin:learning**
  - **linkedin:learning:course**
@@ -509,6 +514,7 @@
  - **mangomolo:live**
  - **mangomolo:video**
  - **ManyVids**
+ - **MaoriTV**
  - **Markiza**
  - **MarkizaPage**
  - **massengeschmack.tv**
@@ -578,6 +584,7 @@
  - **Mwave**
  - **MwaveMeetGreet**
  - **Mxplayer**
+ - **MxplayerShow**
  - **MyChannels**
  - **MySpace**
  - **MySpace:album**
@@ -609,6 +616,7 @@
  - **ndr:embed**
  - **ndr:embed:base**
  - **NDTV**
+ - **Nebula**
  - **NerdCubedFeed**
  - **netease:album**: 网易云音乐 - 专辑
  - **netease:djradio**: 网易云音乐 - 电台
@@ -627,6 +635,7 @@
  - **NextTV**: 壹電視
  - **Nexx**
  - **NexxEmbed**
+ - **NFHSNetwork**
  - **nfl.com** (Currently broken)
  - **nfl.com:article** (Currently broken)
  - **NhkVod**
@@ -1008,6 +1017,7 @@
  - **TruTV**
  - **Tube8**
  - **TubiTv**
+ - **TubiTvShow**
  - **Tumblr**
  - **tunein:clip**
  - **tunein:program**
@@ -1067,6 +1077,7 @@
  - **UDNEmbed**: 聯合影音
  - **UFCArabia**
  - **UFCTV**
+ - **ukcolumn**
  - **UKTVPlay**
  - **umg:de**: Universal Music Deutschland
  - **Unistra**
@@ -1185,6 +1196,7 @@
  - **Weibo**
  - **WeiboMobile**
  - **WeiqiTV**: WQTV
+ - **whowatch**
  - **WimTV**
  - **Wistia**
  - **WistiaPlaylist**
@@ -1195,7 +1207,7 @@
  - **WWE**
  - **XBef**
  - **XboxClips**
- - **XFileShare**: XFileShare based sites: Aparat, ClipWatching, GoUnlimited, GoVid, HolaVid, Streamty, TheVideoBee, Uqload, VidBom, vidlo, VidLocker, VidShare, VUp, XVideoSharing
+ - **XFileShare**: XFileShare based sites: Aparat, ClipWatching, GoUnlimited, GoVid, HolaVid, Streamty, TheVideoBee, Uqload, VidBom, vidlo, VidLocker, VidShare, VUp, WolfStream, XVideoSharing
  - **XHamster**
  - **XHamsterEmbed**
  - **XHamsterUser**

test/test_InfoExtractor.py

@@ -684,17 +684,186 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'width': 1920,
                     'height': 1080,
                     'vcodec': 'avc1.64002a',
-                }]
+                }],
+                {}
+            ),
+            (
+                'bipbop_16x9',
+                'https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8',
+                [{
+                    "format_id": "bipbop_audio-BipBop Audio 2",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/alternate_audio_aac/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "language": "eng",
+                    "ext": "mp4",
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "vcodec": "none",
+                    "audio_ext": "mp4",
+                    "video_ext": "none",
+                }, {
+                    "format_id": "41",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear0/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "tbr": 41.457,
+                    "ext": "mp4",
+                    "fps": None,
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "vcodec": "none",
+                    "acodec": "mp4a.40.2",
+                    "audio_ext": "mp4",
+                    "video_ext": "none",
+                    "abr": 41.457,
+                }, {
+                    "format_id": "263",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear1/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "tbr": 263.851,
+                    "ext": "mp4",
+                    "fps": None,
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "width": 416,
+                    "height": 234,
+                    "vcodec": "avc1.4d400d",
+                    "acodec": "mp4a.40.2",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 263.851,
+                    "abr": 0,
+                }, {
+                    "format_id": "577",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear2/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "tbr": 577.61,
+                    "ext": "mp4",
+                    "fps": None,
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "width": 640,
+                    "height": 360,
+                    "vcodec": "avc1.4d401e",
+                    "acodec": "mp4a.40.2",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 577.61,
+                    "abr": 0,
+                }, {
+                    "format_id": "915",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear3/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "tbr": 915.905,
+                    "ext": "mp4",
+                    "fps": None,
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "width": 960,
+                    "height": 540,
+                    "vcodec": "avc1.4d401f",
+                    "acodec": "mp4a.40.2",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 915.905,
+                    "abr": 0,
+                }, {
+                    "format_id": "1030",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear4/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "tbr": 1030.138,
+                    "ext": "mp4",
+                    "fps": None,
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "width": 1280,
+                    "height": 720,
+                    "vcodec": "avc1.4d401f",
+                    "acodec": "mp4a.40.2",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 1030.138,
+                    "abr": 0,
+                }, {
+                    "format_id": "1924",
+                    "format_index": None,
+                    "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/gear5/prog_index.m3u8",
+                    "manifest_url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8",
+                    "tbr": 1924.009,
+                    "ext": "mp4",
+                    "fps": None,
+                    "protocol": "m3u8",
+                    "preference": None,
+                    "quality": None,
+                    "width": 1920,
+                    "height": 1080,
+                    "vcodec": "avc1.4d401f",
+                    "acodec": "mp4a.40.2",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 1924.009,
+                    "abr": 0,
+                }],
+                {
+                    "en": [{
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/eng/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }, {
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/eng_forced/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }],
+                    "fr": [{
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/fra/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }, {
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/fra_forced/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }],
+                    "es": [{
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/spa/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }, {
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/spa_forced/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }],
+                    "ja": [{
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/jpn/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }, {
+                        "url": "https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/subtitles/jpn_forced/prog_index.m3u8",
+                        "ext": "vtt",
+                        "protocol": "m3u8_native"
+                    }],
+                }
             ),
         ]
 
-        for m3u8_file, m3u8_url, expected_formats in _TEST_CASES:
+        for m3u8_file, m3u8_url, expected_formats, expected_subs in _TEST_CASES:
             with io.open('./test/testdata/m3u8/%s.m3u8' % m3u8_file,
                          mode='r', encoding='utf-8') as f:
-                formats = self.ie._parse_m3u8_formats(
+                formats, subs = self.ie._parse_m3u8_formats_and_subtitles(
                     f.read(), m3u8_url, ext='mp4')
                 self.ie._sort_formats(formats)
                 expect_value(self, formats, expected_formats, None)
+                expect_value(self, subs, expected_subs, None)
 
     def test_parse_mpd_formats(self):
         _TEST_CASES = [
@@ -780,7 +949,8 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'tbr': 5997.485,
                     'width': 1920,
                     'height': 1080,
-                }]
+                }],
+                {},
             ), (
                 # https://github.com/ytdl-org/youtube-dl/pull/14844
                 'urls_only',
@@ -863,7 +1033,8 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'tbr': 4400,
                     'width': 1920,
                     'height': 1080,
-                }]
+                }],
+                {},
             ), (
                 # https://github.com/ytdl-org/youtube-dl/issues/20346
                 # Media considered unfragmented even though it contains
@@ -909,18 +1080,328 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'width': 360,
                     'height': 360,
                     'fps': 30,
-                }]
+                }],
+                {},
+            ), (
+                'subtitles',
+                'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd',
+                'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/',
+                [{
+                    "format_id": "audio=128001",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "ext": "m4a",
+                    "tbr": 128.001,
+                    "asr": 48000,
+                    "format_note": "DASH audio",
+                    "container": "m4a_dash",
+                    "vcodec": "none",
+                    "acodec": "mp4a.40.2",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                    "protocol": "http_dash_segments",
+                    "audio_ext": "m4a",
+                    "video_ext": "none",
+                    "abr": 128.001,
+                }, {
+                    "format_id": "video=100000",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "ext": "mp4",
+                    "width": 336,
+                    "height": 144,
+                    "tbr": 100,
+                    "format_note": "DASH video",
+                    "container": "mp4_dash",
+                    "vcodec": "avc1.4D401F",
+                    "acodec": "none",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                    "protocol": "http_dash_segments",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 100,
+                }, {
+                    "format_id": "video=326000",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "ext": "mp4",
+                    "width": 562,
+                    "height": 240,
+                    "tbr": 326,
+                    "format_note": "DASH video",
+                    "container": "mp4_dash",
+                    "vcodec": "avc1.4D401F",
+                    "acodec": "none",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                    "protocol": "http_dash_segments",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 326,
+                }, {
+                    "format_id": "video=698000",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "ext": "mp4",
+                    "width": 844,
+                    "height": 360,
+                    "tbr": 698,
+                    "format_note": "DASH video",
+                    "container": "mp4_dash",
+                    "vcodec": "avc1.4D401F",
+                    "acodec": "none",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                    "protocol": "http_dash_segments",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 698,
+                }, {
+                    "format_id": "video=1493000",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "ext": "mp4",
+                    "width": 1126,
+                    "height": 480,
+                    "tbr": 1493,
+                    "format_note": "DASH video",
+                    "container": "mp4_dash",
+                    "vcodec": "avc1.4D401F",
+                    "acodec": "none",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                    "protocol": "http_dash_segments",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 1493,
+                }, {
+                    "format_id": "video=4482000",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "ext": "mp4",
+                    "width": 1688,
+                    "height": 720,
+                    "tbr": 4482,
+                    "format_note": "DASH video",
+                    "container": "mp4_dash",
+                    "vcodec": "avc1.4D401F",
+                    "acodec": "none",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                    "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                    "protocol": "http_dash_segments",
+                    "video_ext": "mp4",
+                    "audio_ext": "none",
+                    "vbr": 4482,
+                }],
+                {
+                    "en": [
+                        {
+                            "ext": "mp4",
+                            "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                            "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/manifest.mpd",
+                            "fragment_base_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/dash/",
+                            "protocol": "http_dash_segments",
+                        }
+                    ]
+                },
             )
         ]
 
-        for mpd_file, mpd_url, mpd_base_url, expected_formats in _TEST_CASES:
+        for mpd_file, mpd_url, mpd_base_url, expected_formats, expected_subtitles in _TEST_CASES:
             with io.open('./test/testdata/mpd/%s.mpd' % mpd_file,
                          mode='r', encoding='utf-8') as f:
-                formats = self.ie._parse_mpd_formats(
+                formats, subtitles = self.ie._parse_mpd_formats_and_subtitles(
                     compat_etree_fromstring(f.read().encode('utf-8')),
                     mpd_base_url=mpd_base_url, mpd_url=mpd_url)
                 self.ie._sort_formats(formats)
                 expect_value(self, formats, expected_formats, None)
+                expect_value(self, subtitles, expected_subtitles, None)
+
+    def test_parse_ism_formats(self):
+        _TEST_CASES = [
+            (
+                'sintel',
+                'https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest',
+                [{
+                    "format_id": "audio-128",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "ext": "isma",
+                    "tbr": 128,
+                    "asr": 48000,
+                    "vcodec": "none",
+                    "acodec": "AACL",
+                    "protocol": "ism",
+                    "_download_params": {
+                        "stream_type": "audio",
+                        "duration": 8880746666,
+                        "timescale": 10000000,
+                        "width": 0,
+                        "height": 0,
+                        "fourcc": "AACL",
+                        "codec_private_data": "1190",
+                        "sampling_rate": 48000,
+                        "channels": 2,
+                        "bits_per_sample": 16,
+                        "nal_unit_length_field": 4
+                    },
+                    "audio_ext": "isma",
+                    "video_ext": "none",
+                    "abr": 128,
+                }, {
+                    "format_id": "video-100",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "ext": "ismv",
+                    "width": 336,
+                    "height": 144,
+                    "tbr": 100,
+                    "vcodec": "AVC1",
+                    "acodec": "none",
+                    "protocol": "ism",
+                    "_download_params": {
+                        "stream_type": "video",
+                        "duration": 8880746666,
+                        "timescale": 10000000,
+                        "width": 336,
+                        "height": 144,
+                        "fourcc": "AVC1",
+                        "codec_private_data": "00000001674D401FDA0544EFFC2D002CBC40000003004000000C03C60CA80000000168EF32C8",
+                        "channels": 2,
+                        "bits_per_sample": 16,
+                        "nal_unit_length_field": 4
+                    },
+                    "video_ext": "ismv",
+                    "audio_ext": "none",
+                    "vbr": 100,
+                }, {
+                    "format_id": "video-326",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "ext": "ismv",
+                    "width": 562,
+                    "height": 240,
+                    "tbr": 326,
+                    "vcodec": "AVC1",
+                    "acodec": "none",
+                    "protocol": "ism",
+                    "_download_params": {
+                        "stream_type": "video",
+                        "duration": 8880746666,
+                        "timescale": 10000000,
+                        "width": 562,
+                        "height": 240,
+                        "fourcc": "AVC1",
+                        "codec_private_data": "00000001674D401FDA0241FE23FFC3BC83BA44000003000400000300C03C60CA800000000168EF32C8",
+                        "channels": 2,
+                        "bits_per_sample": 16,
+                        "nal_unit_length_field": 4
+                    },
+                    "video_ext": "ismv",
+                    "audio_ext": "none",
+                    "vbr": 326,
+                }, {
+                    "format_id": "video-698",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "ext": "ismv",
+                    "width": 844,
+                    "height": 360,
+                    "tbr": 698,
+                    "vcodec": "AVC1",
+                    "acodec": "none",
+                    "protocol": "ism",
+                    "_download_params": {
+                        "stream_type": "video",
+                        "duration": 8880746666,
+                        "timescale": 10000000,
+                        "width": 844,
+                        "height": 360,
+                        "fourcc": "AVC1",
+                        "codec_private_data": "00000001674D401FDA0350BFB97FF06AF06AD1000003000100000300300F1832A00000000168EF32C8",
+                        "channels": 2,
+                        "bits_per_sample": 16,
+                        "nal_unit_length_field": 4
+                    },
+                    "video_ext": "ismv",
+                    "audio_ext": "none",
+                    "vbr": 698,
+                }, {
+                    "format_id": "video-1493",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "ext": "ismv",
+                    "width": 1126,
+                    "height": 480,
+                    "tbr": 1493,
+                    "vcodec": "AVC1",
+                    "acodec": "none",
+                    "protocol": "ism",
+                    "_download_params": {
+                        "stream_type": "video",
+                        "duration": 8880746666,
+                        "timescale": 10000000,
+                        "width": 1126,
+                        "height": 480,
+                        "fourcc": "AVC1",
+                        "codec_private_data": "00000001674D401FDA011C3DE6FFF0D890D871000003000100000300300F1832A00000000168EF32C8",
+                        "channels": 2,
+                        "bits_per_sample": 16,
+                        "nal_unit_length_field": 4
+                    },
+                    "video_ext": "ismv",
+                    "audio_ext": "none",
+                    "vbr": 1493,
+                }, {
+                    "format_id": "video-4482",
+                    "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                    "ext": "ismv",
+                    "width": 1688,
+                    "height": 720,
+                    "tbr": 4482,
+                    "vcodec": "AVC1",
+                    "acodec": "none",
+                    "protocol": "ism",
+                    "_download_params": {
+                        "stream_type": "video",
+                        "duration": 8880746666,
+                        "timescale": 10000000,
+                        "width": 1688,
+                        "height": 720,
+                        "fourcc": "AVC1",
+                        "codec_private_data": "00000001674D401FDA01A816F97FFC1ABC1AB440000003004000000C03C60CA80000000168EF32C8",
+                        "channels": 2,
+                        "bits_per_sample": 16,
+                        "nal_unit_length_field": 4
+                    },
+                    "video_ext": "ismv",
+                    "audio_ext": "none",
+                    "vbr": 4482,
+                }],
+                {
+                    "eng": [
+                        {
+                            "ext": "ismt",
+                            "protocol": "ism",
+                            "url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                            "manifest_url": "https://sdn-global-streaming-cache-3qsdn.akamaized.net/stream/3144/files/17/07/672975/3144-kZT4LWMQw6Rh7Kpd.ism/Manifest",
+                            "_download_params": {
+                                "stream_type": "text",
+                                "duration": 8880746666,
+                                "timescale": 10000000,
+                                "fourcc": "TTML",
+                                "codec_private_data": ""
+                            }
+                        }
+                    ]
+                },
+            ),
+        ]
+
+        for ism_file, ism_url, expected_formats, expected_subtitles in _TEST_CASES:
+            with io.open('./test/testdata/ism/%s.Manifest' % ism_file,
+                         mode='r', encoding='utf-8') as f:
+                formats, subtitles = self.ie._parse_ism_formats_and_subtitles(
+                    compat_etree_fromstring(f.read().encode('utf-8')), ism_url=ism_url)
+                self.ie._sort_formats(formats)
+                expect_value(self, formats, expected_formats, None)
+                expect_value(self, subtitles, expected_subtitles, None)
 
     def test_parse_f4m_formats(self):
         _TEST_CASES = [

test/test_YoutubeDL.py

@@ -311,8 +311,8 @@ class TestFormatSelection(unittest.TestCase):
         self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
 
     def test_youtube_format_selection(self):
+        # FIXME: Rewrite in accordance with the new format sorting options
         return
-        # disabled for now - this needs some changes
 
         order = [
             '38', '37', '46', '22', '45', '35', '44', '18', '34', '43', '6', '5', '17', '36', '13',
@@ -601,6 +601,26 @@ class TestYoutubeDL(unittest.TestCase):
         self.assertTrue(subs)
         self.assertEqual(set(subs.keys()), set(['es', 'fr']))
 
+        result = get_info({'writesubtitles': True, 'subtitleslangs': ['all', '-en']})
+        subs = result['requested_subtitles']
+        self.assertTrue(subs)
+        self.assertEqual(set(subs.keys()), set(['es', 'fr']))
+
+        result = get_info({'writesubtitles': True, 'subtitleslangs': ['en', 'fr', '-en']})
+        subs = result['requested_subtitles']
+        self.assertTrue(subs)
+        self.assertEqual(set(subs.keys()), set(['fr']))
+
+        result = get_info({'writesubtitles': True, 'subtitleslangs': ['-en', 'en']})
+        subs = result['requested_subtitles']
+        self.assertTrue(subs)
+        self.assertEqual(set(subs.keys()), set(['en']))
+
+        result = get_info({'writesubtitles': True, 'subtitleslangs': ['e.+']})
+        subs = result['requested_subtitles']
+        self.assertTrue(subs)
+        self.assertEqual(set(subs.keys()), set(['es', 'en']))
+
         result = get_info({'writesubtitles': True, 'writeautomaticsub': True, 'subtitleslangs': ['es', 'pt']})
         subs = result['requested_subtitles']
         self.assertTrue(subs)
@@ -635,6 +655,8 @@ class TestYoutubeDL(unittest.TestCase):
             'height': 1080,
             'title1': '$PATH',
             'title2': '%PATH%',
+            'timestamp': 1618488000,
+            'formats': [{'id': 'id1'}, {'id': 'id2'}]
         }
 
         def fname(templ, na_placeholder='NA'):
@@ -651,6 +673,7 @@ class TestYoutubeDL(unittest.TestCase):
         # Or by provided placeholder
         self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
         self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
+        self.assertEqual(fname('%(height)s.%(ext)s'), '1080.mp4')
         self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
         self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
         self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')
@@ -668,6 +691,12 @@ class TestYoutubeDL(unittest.TestCase):
         self.assertEqual(fname('%%(width)06d.%(ext)s'), '%(width)06d.mp4')
         self.assertEqual(fname('Hello %(title1)s'), 'Hello $PATH')
         self.assertEqual(fname('Hello %(title2)s'), 'Hello %PATH%')
+        self.assertEqual(fname('%(timestamp+-1000>%H-%M-%S)s'), '11-43-20')
+        self.assertEqual(fname('%(id+1)05d'), '01235')
+        self.assertEqual(fname('%(width+100)05d'), 'NA')
+        self.assertEqual(fname('%(formats.0)s').replace("u", ""), "{'id' - 'id1'}")
+        self.assertEqual(fname('%(formats.-1.id)s'), 'id2')
+        self.assertEqual(fname('%(formats.2)s'), 'NA')
 
     def test_format_note(self):
         ydl = YoutubeDL()

test/test_all_urls.py

@@ -72,15 +72,6 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
         self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
 
-    def test_youtube_extract(self):
-        assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)
-        assertExtractId('http://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('https://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('https://www.youtube.com/watch?feature=player_embedded&v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('https://www.youtube.com/watch_popup?v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('http://www.youtube.com/watch?v=BaW_jenozKcsharePLED17F32AD9753930', 'BaW_jenozKc')
-        assertExtractId('BaW_jenozKc', 'BaW_jenozKc')
-
     def test_facebook_matching(self):
         self.assertTrue(FacebookIE.suitable('https://www.facebook.com/Shiniknoh#!/photo.php?v=10153317450565268'))
         self.assertTrue(FacebookIE.suitable('https://www.facebook.com/cindyweather?fref=ts#!/photo.php?v=10152183998945793'))

test/test_execution.py

@@ -39,6 +39,16 @@ class TestExecution(unittest.TestCase):
         _, stderr = p.communicate()
         self.assertFalse(stderr)
 
+    def test_lazy_extractors(self):
+        try:
+            subprocess.check_call([sys.executable, 'devscripts/make_lazy_extractors.py', 'yt_dlp/extractor/lazy_extractors.py'], cwd=rootDir, stdout=_DEV_NULL)
+            subprocess.check_call([sys.executable, 'test/test_all_urls.py'], cwd=rootDir, stdout=_DEV_NULL)
+        finally:
+            try:
+                os.remove('yt_dlp/extractor/lazy_extractors.py')
+            except (IOError, OSError):
+                pass
+
 
 if __name__ == '__main__':
     unittest.main()

test/test_postprocessors.py

@@ -16,6 +16,15 @@ class TestMetadataFromField(unittest.TestCase):
         pp = MetadataFromFieldPP(None, ['title:%(title)s - %(artist)s'])
         self.assertEqual(pp._data[0]['regex'], r'(?P<title>.+)\ \-\ (?P<artist>.+)')
 
+    def test_field_to_outtmpl(self):
+        pp = MetadataFromFieldPP(None, ['title:%(title)s : %(artist)s'])
+        self.assertEqual(pp._data[0]['tmpl'], '%(title)s')
+
+    def test_in_out_seperation(self):
+        pp = MetadataFromFieldPP(None, ['%(title)s \\: %(artist)s:%(title)s : %(artist)s'])
+        self.assertEqual(pp._data[0]['in'], '%(title)s : %(artist)s')
+        self.assertEqual(pp._data[0]['out'], '%(title)s : %(artist)s')
+
 
 class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):

test/test_utils.py

@@ -23,6 +23,7 @@ from yt_dlp.utils import (
     clean_html,
     clean_podcast_url,
     date_from_str,
+    datetime_from_str,
     DateRange,
     detect_exe_version,
     determine_ext,
@@ -311,8 +312,18 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(date_from_str('yesterday'), date_from_str('now-1day'))
         self.assertEqual(date_from_str('now+7day'), date_from_str('now+1week'))
         self.assertEqual(date_from_str('now+14day'), date_from_str('now+2week'))
-        self.assertEqual(date_from_str('now+365day'), date_from_str('now+1year'))
-        self.assertEqual(date_from_str('now+30day'), date_from_str('now+1month'))
+        self.assertEqual(date_from_str('20200229+365day'), date_from_str('20200229+1year'))
+        self.assertEqual(date_from_str('20210131+28day'), date_from_str('20210131+1month'))
+
+    def test_datetime_from_str(self):
+        self.assertEqual(datetime_from_str('yesterday', precision='day'), datetime_from_str('now-1day', precision='auto'))
+        self.assertEqual(datetime_from_str('now+7day', precision='day'), datetime_from_str('now+1week', precision='auto'))
+        self.assertEqual(datetime_from_str('now+14day', precision='day'), datetime_from_str('now+2week', precision='auto'))
+        self.assertEqual(datetime_from_str('20200229+365day', precision='day'), datetime_from_str('20200229+1year', precision='auto'))
+        self.assertEqual(datetime_from_str('20210131+28day', precision='day'), datetime_from_str('20210131+1month', precision='auto'))
+        self.assertEqual(datetime_from_str('20210131+59day', precision='day'), datetime_from_str('20210131+2month', precision='auto'))
+        self.assertEqual(datetime_from_str('now+1day', precision='hour'), datetime_from_str('now+24hours', precision='auto'))
+        self.assertEqual(datetime_from_str('now+23hours', precision='hour'), datetime_from_str('now+23hours', precision='auto'))
 
     def test_daterange(self):
         _20century = DateRange("19000101", "20000101")

test/test_youtube_misc.py

@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+from __future__ import unicode_literals
+
+# Allow direct execution
+import os
+import sys
+import unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+
+from yt_dlp.extractor import YoutubeIE
+
+
+class TestYoutubeMisc(unittest.TestCase):
+    def test_youtube_extract(self):
+        assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)
+        assertExtractId('http://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('https://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('https://www.youtube.com/watch?feature=player_embedded&v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('https://www.youtube.com/watch_popup?v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('http://www.youtube.com/watch?v=BaW_jenozKcsharePLED17F32AD9753930', 'BaW_jenozKc')
+        assertExtractId('BaW_jenozKc', 'BaW_jenozKc')
+
+
+if __name__ == '__main__':
+    unittest.main()

test/testdata/ism/sintel.Manifest

@@ -0,0 +1,988 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Created with Unified Streaming Platform (version=1.10.18-20255) -->
+<SmoothStreamingMedia
+  MajorVersion="2"
+  MinorVersion="0"
+  TimeScale="10000000"
+  Duration="8880746666">
+  <StreamIndex
+    Type="audio"
+    QualityLevels="1"
+    TimeScale="10000000"
+    Name="audio"
+    Chunks="445"
+    Url="QualityLevels({bitrate})/Fragments(audio={start time})">
+    <QualityLevel
+      Index="0"
+      Bitrate="128001"
+      CodecPrivateData="1190"
+      SamplingRate="48000"
+      Channels="2"
+      BitsPerSample="16"
+      PacketSize="4"
+      AudioTag="255"
+      FourCC="AACL" />
+    <c t="0" d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="20053333" />
+    <c d="20053333" />
+    <c d="20053334" />
+    <c d="19840000" />
+    <c d="746666" />
+  </StreamIndex>
+  <StreamIndex
+    Type="text"
+    QualityLevels="1"
+    TimeScale="10000000"
+    Language="eng"
+    Subtype="CAPT"
+    Name="textstream_eng"
+    Chunks="11"
+    Url="QualityLevels({bitrate})/Fragments(textstream_eng={start time})">
+    <QualityLevel
+      Index="0"
+      Bitrate="1000"
+      CodecPrivateData=""
+      FourCC="TTML" />
+    <c t="0" d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="600000000" />
+    <c d="240000000" />
+  </StreamIndex>
+  <StreamIndex
+    Type="video"
+    QualityLevels="5"
+    TimeScale="10000000"
+    Name="video"
+    Chunks="444"
+    Url="QualityLevels({bitrate})/Fragments(video={start time})"
+    MaxWidth="1688"
+    MaxHeight="720"
+    DisplayWidth="1689"
+    DisplayHeight="720">
+    <QualityLevel
+      Index="0"
+      Bitrate="100000"
+      CodecPrivateData="00000001674D401FDA0544EFFC2D002CBC40000003004000000C03C60CA80000000168EF32C8"
+      MaxWidth="336"
+      MaxHeight="144"
+      FourCC="AVC1" />
+    <QualityLevel
+      Index="1"
+      Bitrate="326000"
+      CodecPrivateData="00000001674D401FDA0241FE23FFC3BC83BA44000003000400000300C03C60CA800000000168EF32C8"
+      MaxWidth="562"
+      MaxHeight="240"
+      FourCC="AVC1" />
+    <QualityLevel
+      Index="2"
+      Bitrate="698000"
+      CodecPrivateData="00000001674D401FDA0350BFB97FF06AF06AD1000003000100000300300F1832A00000000168EF32C8"
+      MaxWidth="844"
+      MaxHeight="360"
+      FourCC="AVC1" />
+    <QualityLevel
+      Index="3"
+      Bitrate="1493000"
+      CodecPrivateData="00000001674D401FDA011C3DE6FFF0D890D871000003000100000300300F1832A00000000168EF32C8"
+      MaxWidth="1126"
+      MaxHeight="480"
+      FourCC="AVC1" />
+    <QualityLevel
+      Index="4"
+      Bitrate="4482000"
+      CodecPrivateData="00000001674D401FDA01A816F97FFC1ABC1AB440000003004000000C03C60CA80000000168EF32C8"
+      MaxWidth="1688"
+      MaxHeight="720"
+      FourCC="AVC1" />
+    <c t="0" d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+    <c d="20000000" />
+  </StreamIndex>
+</SmoothStreamingMedia>

test/testdata/m3u8/bipbop_16x9.m3u8

@@ -0,0 +1,38 @@
+#EXTM3U
+
+#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",AUTOSELECT=YES,DEFAULT=YES
+#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 2",AUTOSELECT=NO,DEFAULT=NO,URI="alternate_audio_aac/prog_index.m3u8"
+
+
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",DEFAULT=YES,AUTOSELECT=YES,FORCED=NO,LANGUAGE="en",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/eng/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="en",URI="subtitles/eng_forced/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Français",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="fr",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/fra/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Français (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="fr",URI="subtitles/fra_forced/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Español",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="es",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/spa/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Español (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="es",URI="subtitles/spa_forced/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="日本語",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="ja",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/jpn/prog_index.m3u8"
+#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="日本語 (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="ja",URI="subtitles/jpn_forced/prog_index.m3u8"
+
+
+#EXT-X-STREAM-INF:BANDWIDTH=263851,CODECS="mp4a.40.2, avc1.4d400d",RESOLUTION=416x234,AUDIO="bipbop_audio",SUBTITLES="subs"
+gear1/prog_index.m3u8
+#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=28451,CODECS="avc1.4d400d",URI="gear1/iframe_index.m3u8"
+
+#EXT-X-STREAM-INF:BANDWIDTH=577610,CODECS="mp4a.40.2, avc1.4d401e",RESOLUTION=640x360,AUDIO="bipbop_audio",SUBTITLES="subs"
+gear2/prog_index.m3u8
+#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=181534,CODECS="avc1.4d401e",URI="gear2/iframe_index.m3u8"
+
+#EXT-X-STREAM-INF:BANDWIDTH=915905,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=960x540,AUDIO="bipbop_audio",SUBTITLES="subs"
+gear3/prog_index.m3u8
+#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=297056,CODECS="avc1.4d401f",URI="gear3/iframe_index.m3u8"
+
+#EXT-X-STREAM-INF:BANDWIDTH=1030138,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=1280x720,AUDIO="bipbop_audio",SUBTITLES="subs"
+gear4/prog_index.m3u8
+#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=339492,CODECS="avc1.4d401f",URI="gear4/iframe_index.m3u8"
+
+#EXT-X-STREAM-INF:BANDWIDTH=1924009,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=1920x1080,AUDIO="bipbop_audio",SUBTITLES="subs"
+gear5/prog_index.m3u8
+#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=669554,CODECS="avc1.4d401f",URI="gear5/iframe_index.m3u8"
+
+#EXT-X-STREAM-INF:BANDWIDTH=41457,CODECS="mp4a.40.2",AUDIO="bipbop_audio",SUBTITLES="subs"
+gear0/prog_index.m3u8

test/testdata/mpd/subtitles.mpd

@@ -0,0 +1,351 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Created with Unified Streaming Platform (version=1.10.18-20255) -->
+<MPD
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xmlns="urn:mpeg:dash:schema:mpd:2011"
+  xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
+  type="static"
+  mediaPresentationDuration="PT14M48S"
+  maxSegmentDuration="PT1M"
+  minBufferTime="PT10S"
+  profiles="urn:mpeg:dash:profile:isoff-live:2011">
+  <Period
+    id="1"
+    duration="PT14M48S">
+    <BaseURL>dash/</BaseURL>
+    <AdaptationSet
+      id="1"
+      group="1"
+      contentType="audio"
+      segmentAlignment="true"
+      audioSamplingRate="48000"
+      mimeType="audio/mp4"
+      codecs="mp4a.40.2"
+      startWithSAP="1">
+      <AudioChannelConfiguration
+        schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011"
+        value="2" />
+      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" />
+      <SegmentTemplate
+        timescale="48000"
+        initialization="3144-kZT4LWMQw6Rh7Kpd-$RepresentationID$.dash"
+        media="3144-kZT4LWMQw6Rh7Kpd-$RepresentationID$-$Time$.dash">
+        <SegmentTimeline>
+          <S t="0" d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="96256" r="2" />
+          <S d="95232" />
+          <S d="3584" />
+        </SegmentTimeline>
+      </SegmentTemplate>
+      <Representation
+        id="audio=128001"
+        bandwidth="128001">
+      </Representation>
+    </AdaptationSet>
+    <AdaptationSet
+      id="2"
+      group="3"
+      contentType="text"
+      lang="en"
+      mimeType="application/mp4"
+      codecs="stpp"
+      startWithSAP="1">
+      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle" />
+      <SegmentTemplate
+        timescale="1000"
+        initialization="3144-kZT4LWMQw6Rh7Kpd-$RepresentationID$.dash"
+        media="3144-kZT4LWMQw6Rh7Kpd-$RepresentationID$-$Time$.dash">
+        <SegmentTimeline>
+          <S t="0" d="60000" r="9" />
+          <S d="24000" />
+        </SegmentTimeline>
+      </SegmentTemplate>
+      <Representation
+        id="textstream_eng=1000"
+        bandwidth="1000">
+      </Representation>
+    </AdaptationSet>
+    <AdaptationSet
+      id="3"
+      group="2"
+      contentType="video"
+      par="960:409"
+      minBandwidth="100000"
+      maxBandwidth="4482000"
+      maxWidth="1689"
+      maxHeight="720"
+      segmentAlignment="true"
+      mimeType="video/mp4"
+      codecs="avc1.4D401F"
+      startWithSAP="1">
+      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" />
+      <SegmentTemplate
+        timescale="12288"
+        initialization="3144-kZT4LWMQw6Rh7Kpd-$RepresentationID$.dash"
+        media="3144-kZT4LWMQw6Rh7Kpd-$RepresentationID$-$Time$.dash">
+        <SegmentTimeline>
+          <S t="0" d="24576" r="443" />
+        </SegmentTimeline>
+      </SegmentTemplate>
+      <Representation
+        id="video=100000"
+        bandwidth="100000"
+        width="336"
+        height="144"
+        sar="2880:2863"
+        scanType="progressive">
+      </Representation>
+      <Representation
+        id="video=326000"
+        bandwidth="326000"
+        width="562"
+        height="240"
+        sar="115200:114929"
+        scanType="progressive">
+      </Representation>
+      <Representation
+        id="video=698000"
+        bandwidth="698000"
+        width="844"
+        height="360"
+        sar="86400:86299"
+        scanType="progressive">
+      </Representation>
+      <Representation
+        id="video=1493000"
+        bandwidth="1493000"
+        width="1126"
+        height="480"
+        sar="230400:230267"
+        scanType="progressive">
+      </Representation>
+      <Representation
+        id="video=4482000"
+        bandwidth="4482000"
+        width="1688"
+        height="720"
+        sar="86400:86299"
+        scanType="progressive">
+      </Representation>
+    </AdaptationSet>
+  </Period>
+</MPD>

yt-dlp.sh

@@ -1,2 +1,2 @@
-#!/bin/bash
-python3 "$(dirname $(realpath $0))/yt_dlp/__main__.py" "$@"
+#!/bin/sh
+exec python3 "$(dirname "$(realpath "$0")")/yt_dlp/__main__.py" "$@"

yt_dlp/__init__.py

@@ -60,6 +60,7 @@ def _real_main(argv=None):
     setproctitle('yt-dlp')
 
     parser, opts, args = parseOpts(argv)
+    warnings = []
 
     # Set user agent
     if opts.user_agent is not None:
@@ -128,16 +129,12 @@ def _real_main(argv=None):
         parser.error('account username missing\n')
     if opts.ap_password is not None and opts.ap_username is None:
         parser.error('TV Provider account username missing\n')
-    if opts.outtmpl is not None and (opts.usetitle or opts.autonumber or opts.useid):
-        parser.error('using output template conflicts with using title, video ID or auto number')
     if opts.autonumber_size is not None:
         if opts.autonumber_size <= 0:
             parser.error('auto number size must be positive')
     if opts.autonumber_start is not None:
         if opts.autonumber_start < 0:
             parser.error('auto number start must be positive or 0')
-    if opts.usetitle and opts.useid:
-        parser.error('using title conflicts with using video ID')
     if opts.username is not None and opts.password is None:
         opts.password = compat_getpass('Type account password and press [Return]: ')
     if opts.ap_username is not None and opts.ap_password is None:
@@ -177,8 +174,7 @@ def _real_main(argv=None):
             parser.error('requests sleep interval must be positive or 0')
     if opts.ap_mso and opts.ap_mso not in MSO_INFO:
         parser.error('Unsupported TV Provider, use --ap-list-mso to get a list of supported TV Providers')
-    if opts.overwrites:
-        # --yes-overwrites implies --no-continue
+    if opts.overwrites:  # --yes-overwrites implies --no-continue
         opts.continue_dl = False
     if opts.concurrent_fragment_downloads <= 0:
         raise ValueError('Concurrent fragments must be positive')
@@ -228,29 +224,86 @@ def _real_main(argv=None):
         if not re.match(remux_regex, opts.remuxvideo):
             parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
-        if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
+        if opts.convertsubtitles not in ('srt', 'vtt', 'ass', 'lrc'):
             parser.error('invalid subtitle format specified')
+    if opts.convertthumbnails is not None:
+        if opts.convertthumbnails not in ('jpg', ):
+            parser.error('invalid thumbnail format specified')
 
     if opts.date is not None:
         date = DateRange.day(opts.date)
     else:
         date = DateRange(opts.dateafter, opts.datebefore)
 
-    # Do not download videos when there are audio-only formats
+    def parse_compat_opts():
+        parsed_compat_opts, compat_opts = set(), opts.compat_opts[::-1]
+        while compat_opts:
+            actual_opt = opt = compat_opts.pop().lower()
+            if opt == 'youtube-dl':
+                compat_opts.extend(['-multistreams', 'all'])
+            elif opt == 'youtube-dlc':
+                compat_opts.extend(['-no-youtube-channel-redirect', '-no-live-chat', 'all'])
+            elif opt == 'all':
+                parsed_compat_opts.update(all_compat_opts)
+            elif opt == '-all':
+                parsed_compat_opts = set()
+            else:
+                if opt[0] == '-':
+                    opt = opt[1:]
+                    parsed_compat_opts.discard(opt)
+                else:
+                    parsed_compat_opts.update([opt])
+                if opt not in all_compat_opts:
+                    parser.error('Invalid compatibility option %s' % actual_opt)
+        return parsed_compat_opts
+
+    all_compat_opts = [
+        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'multistreams',
+        'no-playlist-metafiles', 'no-live-chat', 'playlist-index', 'list-formats',
+        'no-youtube-channel-redirect', 'no-youtube-unavailable-videos',
+    ]
+    compat_opts = parse_compat_opts()
+
+    def _unused_compat_opt(name):
+        if name not in compat_opts:
+            return False
+        compat_opts.discard(name)
+        compat_opts.update(['*%s' % name])
+        return True
+
+    def set_default_compat(compat_name, opt_name, default=True, remove_compat=False):
+        attr = getattr(opts, opt_name)
+        if compat_name in compat_opts:
+            if attr is None:
+                setattr(opts, opt_name, not default)
+                return True
+            else:
+                if remove_compat:
+                    _unused_compat_opt(compat_name)
+                return False
+        elif attr is None:
+            setattr(opts, opt_name, default)
+        return None
+
+    set_default_compat('abort-on-error', 'ignoreerrors')
+    set_default_compat('no-playlist-metafiles', 'allow_playlist_files')
+    if 'format-sort' in compat_opts:
+        opts.format_sort.extend(InfoExtractor.FormatSort.ytdl_default)
+    _video_multistreams_set = set_default_compat('multistreams', 'allow_multiple_video_streams', False, remove_compat=False)
+    _audio_multistreams_set = set_default_compat('multistreams', 'allow_multiple_audio_streams', False, remove_compat=False)
+    if _video_multistreams_set is False and _audio_multistreams_set is False:
+        _unused_compat_opt('multistreams')
+    outtmpl_default = opts.outtmpl.get('default')
+    if 'filename' in compat_opts:
+        if outtmpl_default is None:
+            outtmpl_default = '%(title)s.%(id)s.%(ext)s'
+            opts.outtmpl.update({'default': outtmpl_default})
+        else:
+            _unused_compat_opt('filename')
+
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
 
-    outtmpl = opts.outtmpl
-    if not outtmpl:
-        outtmpl = {'default': (
-            '%(title)s-%(id)s-%(format)s.%(ext)s' if opts.format == '-1' and opts.usetitle
-            else '%(id)s-%(format)s.%(ext)s' if opts.format == '-1'
-            else '%(autonumber)s-%(title)s-%(id)s.%(ext)s' if opts.usetitle and opts.autonumber
-            else '%(title)s-%(id)s.%(ext)s' if opts.usetitle
-            else '%(id)s.%(ext)s' if opts.useid
-            else '%(autonumber)s-%(id)s.%(ext)s' if opts.autonumber
-            else None)}
-    outtmpl_default = outtmpl.get('default')
     if outtmpl_default is not None and not os.path.splitext(outtmpl_default)[1] and opts.extractaudio:
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
@@ -278,7 +331,7 @@ def _real_main(argv=None):
         opts.writeinfojson = True
 
     def report_conflict(arg1, arg2):
-        write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr)
+        warnings.append('%s is ignored since %s was given' % (arg2, arg1))
 
     if opts.remuxvideo and opts.recodevideo:
         report_conflict('--recode-video', '--remux-video')
@@ -322,7 +375,22 @@ def _real_main(argv=None):
         postprocessors.append({
             'key': 'MetadataFromField',
             'formats': opts.metafromfield,
-            'when': 'beforedl'
+            # Run this immediately after extraction is complete
+            'when': 'pre_process'
+        })
+    if opts.convertsubtitles:
+        postprocessors.append({
+            'key': 'FFmpegSubtitlesConvertor',
+            'format': opts.convertsubtitles,
+            # Run this before the actual video download
+            'when': 'before_dl'
+        })
+    if opts.convertthumbnails:
+        postprocessors.append({
+            'key': 'FFmpegThumbnailsConvertor',
+            'format': opts.convertthumbnails,
+            # Run this before the actual video download
+            'when': 'before_dl'
         })
     if opts.extractaudio:
         postprocessors.append({
@@ -351,15 +419,11 @@ def _real_main(argv=None):
     # so metadata can be added here.
     if opts.addmetadata:
         postprocessors.append({'key': 'FFmpegMetadata'})
-    if opts.convertsubtitles:
-        postprocessors.append({
-            'key': 'FFmpegSubtitlesConvertor',
-            'format': opts.convertsubtitles,
-        })
     if opts.embedsubtitles:
         already_have_subtitle = opts.writesubtitles
         postprocessors.append({
             'key': 'FFmpegEmbedSubtitle',
+            # already_have_subtitle = True prevents the file from being deleted after embedding
             'already_have_subtitle': already_have_subtitle
         })
         if not already_have_subtitle:
@@ -385,6 +449,7 @@ def _real_main(argv=None):
         already_have_thumbnail = opts.writethumbnail or opts.write_all_thumbnails
         postprocessors.append({
             'key': 'EmbedThumbnail',
+            # already_have_thumbnail = True prevents the file from being deleted after embedding
             'already_have_thumbnail': already_have_thumbnail
         })
         if not already_have_thumbnail:
@@ -399,15 +464,15 @@ def _real_main(argv=None):
         postprocessors.append({
             'key': 'ExecAfterDownload',
             'exec_cmd': opts.exec_cmd,
-            'when': 'aftermove'
+            # Run this only after the files have been moved to their final locations
+            'when': 'after_move'
         })
 
     def report_args_compat(arg, name):
-        write_string(
-            'WARNING: %s given without specifying name. The arguments will be given to all %s\n' % (arg, name),
-            out=sys.stderr)
+        warnings.append('%s given without specifying name. The arguments will be given to all %s' % (arg, name))
+
     if 'default' in opts.external_downloader_args:
-        report_args_compat('--external-downloader-args', 'external downloaders')
+        report_args_compat('--downloader-args', 'external downloaders')
 
     if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
         report_args_compat('--post-processor-args', 'post-processors')
@@ -425,7 +490,6 @@ def _real_main(argv=None):
         else match_filter_func(opts.match_filter))
 
     ydl_opts = {
-        'convertsubtitles': opts.convertsubtitles,
         'usenetrc': opts.usenetrc,
         'username': opts.username,
         'password': opts.password,
@@ -451,13 +515,15 @@ def _real_main(argv=None):
         'skip_download': opts.skip_download,
         'format': opts.format,
         'allow_unplayable_formats': opts.allow_unplayable_formats,
+        'ignore_no_formats_error': opts.ignore_no_formats_error,
         'format_sort': opts.format_sort,
         'format_sort_force': opts.format_sort_force,
         'allow_multiple_video_streams': opts.allow_multiple_video_streams,
         'allow_multiple_audio_streams': opts.allow_multiple_audio_streams,
+        'check_formats': opts.check_formats,
         'listformats': opts.listformats,
         'listformats_table': opts.listformats_table,
-        'outtmpl': outtmpl,
+        'outtmpl': opts.outtmpl,
         'outtmpl_na_placeholder': opts.outtmpl_na_placeholder,
         'paths': opts.paths,
         'autonumber_size': opts.autonumber_size,
@@ -528,6 +594,7 @@ def _real_main(argv=None):
         'download_archive': download_archive_fn,
         'break_on_existing': opts.break_on_existing,
         'break_on_reject': opts.break_on_reject,
+        'skip_playlist_after_errors': opts.skip_playlist_after_errors,
         'cookiefile': opts.cookiefile,
         'nocheckcertificate': opts.no_check_certificate,
         'prefer_insecure': opts.prefer_insecure,
@@ -571,9 +638,12 @@ def _real_main(argv=None):
         'geo_bypass': opts.geo_bypass,
         'geo_bypass_country': opts.geo_bypass_country,
         'geo_bypass_ip_block': opts.geo_bypass_ip_block,
+        'warnings': warnings,
+        'compat_opts': compat_opts,
         # just for deprecation check
-        'autonumber': opts.autonumber if opts.autonumber is True else None,
-        'usetitle': opts.usetitle if opts.usetitle is True else None,
+        'autonumber': opts.autonumber or None,
+        'usetitle': opts.usetitle or None,
+        'useid': opts.useid or None,
     }
 
     with YoutubeDL(ydl_opts) as ydl:

yt_dlp/compat.py

@@ -78,6 +78,15 @@ try:
 except ImportError:  # Python 2
     import Cookie as compat_cookies
 
+if sys.version_info[0] == 2:
+    class compat_cookies_SimpleCookie(compat_cookies.SimpleCookie):
+        def load(self, rawdata):
+            if isinstance(rawdata, compat_str):
+                rawdata = str(rawdata)
+            return super(compat_cookies_SimpleCookie, self).load(rawdata)
+else:
+    compat_cookies_SimpleCookie = compat_cookies.SimpleCookie
+
 try:
     import html.entities as compat_html_entities
 except ImportError:  # Python 2
@@ -3009,10 +3018,24 @@ else:
         return ctypes.WINFUNCTYPE(*args, **kwargs)
 
 
+try:
+    compat_Pattern = re.Pattern
+except AttributeError:
+    compat_Pattern = type(re.compile(''))
+
+
+try:
+    compat_Match = re.Match
+except AttributeError:
+    compat_Match = type(re.compile('').match(''))
+
+
 __all__ = [
     'compat_HTMLParseError',
     'compat_HTMLParser',
     'compat_HTTPError',
+    'compat_Match',
+    'compat_Pattern',
     'compat_Struct',
     'compat_b64decode',
     'compat_basestring',
@@ -3020,6 +3043,7 @@ __all__ = [
     'compat_cookiejar',
     'compat_cookiejar_Cookie',
     'compat_cookies',
+    'compat_cookies_SimpleCookie',
     'compat_ctypes_WINFUNCTYPE',
     'compat_etree_Element',
     'compat_etree_fromstring',

yt_dlp/downloader/__init__.py

@@ -1,5 +1,6 @@
 from __future__ import unicode_literals
 
+from ..compat import compat_str
 from ..utils import (
     determine_protocol,
 )
@@ -30,6 +31,7 @@ from .external import (
 
 PROTOCOL_MAP = {
     'rtmp': RtmpFD,
+    'rtmp_ffmpeg': FFmpegFD,
     'm3u8_native': HlsFD,
     'm3u8': FFmpegFD,
     'mms': RtspFD,
@@ -42,6 +44,25 @@ PROTOCOL_MAP = {
 }
 
 
+def shorten_protocol_name(proto, simplify=False):
+    short_protocol_names = {
+        'm3u8_native': 'm3u8_n',
+        'rtmp_ffmpeg': 'rtmp_f',
+        'http_dash_segments': 'dash',
+        'niconico_dmc': 'dmc',
+    }
+    if simplify:
+        short_protocol_names.update({
+            'https': 'http',
+            'ftps': 'ftp',
+            'm3u8_native': 'm3u8',
+            'rtmp_ffmpeg': 'rtmp',
+            'm3u8_frag_urls': 'm3u8',
+            'dash_frag_urls': 'dash',
+        })
+    return short_protocol_names.get(proto, proto)
+
+
 def get_suitable_downloader(info_dict, params={}, default=HttpFD):
     """Get the downloader class that can handle the info dict."""
     protocol = determine_protocol(info_dict)
@@ -50,16 +71,24 @@ def get_suitable_downloader(info_dict, params={}, default=HttpFD):
     # if (info_dict.get('start_time') or info_dict.get('end_time')) and not info_dict.get('requested_formats') and FFmpegFD.can_download(info_dict):
     #     return FFmpegFD
 
-    external_downloader = params.get('external_downloader')
-    if external_downloader is not None:
+    downloaders = params.get('external_downloader')
+    external_downloader = (
+        downloaders if isinstance(downloaders, compat_str) or downloaders is None
+        else downloaders.get(shorten_protocol_name(protocol, True), downloaders.get('default')))
+    if external_downloader and external_downloader.lower() == 'native':
+        external_downloader = 'native'
+
+    if external_downloader not in (None, 'native'):
         ed = get_external_downloader(external_downloader)
         if ed.can_download(info_dict, external_downloader):
             return ed
 
-    if protocol.startswith('m3u8'):
+    if protocol in ('m3u8', 'm3u8_native'):
         if info_dict.get('is_live'):
             return FFmpegFD
-        elif _get_real_downloader(info_dict, 'frag_urls', params, None):
+        elif external_downloader == 'native':
+            return HlsFD
+        elif _get_real_downloader(info_dict, 'm3u8_frag_urls', params, None):
             return HlsFD
         elif params.get('hls_prefer_native') is True:
             return HlsFD
@@ -70,6 +99,7 @@ def get_suitable_downloader(info_dict, params={}, default=HttpFD):
 
 
 __all__ = [
-    'get_suitable_downloader',
     'FileDownloader',
+    'get_suitable_downloader',
+    'shorten_protocol_name',
 ]

yt_dlp/downloader/common.py

@@ -147,10 +147,10 @@ class FileDownloader(object):
         return int(round(number * multiplier))
 
     def to_screen(self, *args, **kargs):
-        self.ydl.to_screen(*args, **kargs)
+        self.ydl.to_stdout(*args, quiet=self.params.get('quiet'), **kargs)
 
     def to_stderr(self, message):
-        self.ydl.to_screen(message)
+        self.ydl.to_stderr(message)
 
     def to_console_title(self, message):
         self.ydl.to_console_title(message)

yt_dlp/downloader/dash.py

@@ -1,5 +1,6 @@
 from __future__ import unicode_literals
 
+import errno
 try:
     import concurrent.futures
     can_threaded_download = True
@@ -20,7 +21,7 @@ from ..utils import (
 class DashSegmentsFD(FragmentFD):
     """
     Download segments in a DASH manifest. External downloaders can take over
-    the fragment downloads by supporting the 'frag_urls' protocol
+    the fragment downloads by supporting the 'dash_frag_urls' protocol
     """
 
     FD_NAME = 'dashsegments'
@@ -30,7 +31,7 @@ class DashSegmentsFD(FragmentFD):
         fragments = info_dict['fragments'][:1] if self.params.get(
             'test', False) else info_dict['fragments']
 
-        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+        real_downloader = _get_real_downloader(info_dict, 'dash_frag_urls', self.params, None)
 
         ctx = {
             'filename': filename,
@@ -112,12 +113,14 @@ class DashSegmentsFD(FragmentFD):
                 if count > fragment_retries:
                     if not fatal:
                         return False, frag_index
+                    ctx['dest_stream'].close()
                     self.report_error('Giving up after %s fragment retries' % fragment_retries)
                     return False, frag_index
 
                 return frag_content, frag_index
 
             def append_fragment(frag_content, frag_index):
+                fatal = frag_index == 1 or not skip_unavailable_fragments
                 if frag_content:
                     fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
                     try:
@@ -126,19 +129,24 @@ class DashSegmentsFD(FragmentFD):
                         file.close()
                         self._append_fragment(ctx, frag_content)
                         return True
-                    except FileNotFoundError:
-                        if skip_unavailable_fragments:
+                    except EnvironmentError as ose:
+                        if ose.errno != errno.ENOENT:
+                            raise
+                        # FileNotFoundError
+                        if not fatal:
                             self.report_skip_fragment(frag_index)
                             return True
                         else:
+                            ctx['dest_stream'].close()
                             self.report_error(
                                 'fragment %s not found, unable to continue' % frag_index)
                             return False
                 else:
-                    if skip_unavailable_fragments:
+                    if not fatal:
                         self.report_skip_fragment(frag_index)
                         return True
                     else:
+                        ctx['dest_stream'].close()
                         self.report_error(
                             'fragment %s not found, unable to continue' % frag_index)
                         return False

yt_dlp/downloader/external.py

@@ -81,11 +81,15 @@ class ExternalFD(FileDownloader):
 
     @property
     def exe(self):
-        return self.params.get('external_downloader')
+        return self.get_basename()
 
     @classmethod
     def available(cls, path=None):
-        return check_executable(path or cls.get_basename(), [cls.AVAILABLE_OPT])
+        path = check_executable(path or cls.get_basename(), [cls.AVAILABLE_OPT])
+        if path:
+            cls.exe = path
+            return path
+        return False
 
     @classmethod
     def supports(cls, info_dict):
@@ -259,7 +263,7 @@ class WgetFD(ExternalFD):
 
 class Aria2cFD(ExternalFD):
     AVAILABLE_OPT = '-v'
-    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'frag_urls')
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'dash_frag_urls', 'm3u8_frag_urls')
 
     @staticmethod
     def supports_manifest(manifest):
@@ -286,11 +290,19 @@ class Aria2cFD(ExternalFD):
         cmd += self._bool_option('--remote-time', 'updatetime', 'true', 'false', '=')
         cmd += self._configuration_args()
 
+        # aria2c strips out spaces from the beginning/end of filenames and paths.
+        # We work around this issue by adding a "./" to the beginning of the
+        # filename and relative path, and adding a "/" at the end of the path.
+        # See: https://github.com/yt-dlp/yt-dlp/issues/276
+        # https://github.com/ytdl-org/youtube-dl/issues/20312
+        # https://github.com/aria2/aria2/issues/1373
         dn = os.path.dirname(tmpfilename)
         if dn:
-            cmd += ['--dir', dn]
+            if not os.path.isabs(dn):
+                dn = '.%s%s' % (os.path.sep, dn)
+            cmd += ['--dir', dn + os.path.sep]
         if 'fragments' not in info_dict:
-            cmd += ['--out', os.path.basename(tmpfilename)]
+            cmd += ['--out', '.%s%s' % (os.path.sep, os.path.basename(tmpfilename))]
         cmd += ['--auto-file-renaming=false']
 
         if 'fragments' in info_dict:
@@ -310,9 +322,11 @@ class Aria2cFD(ExternalFD):
 
 
 class HttpieFD(ExternalFD):
+    AVAILABLE_OPT = '--version'
+
     @classmethod
     def available(cls, path=None):
-        return check_executable(path or 'http', ['--version'])
+        return ExternalFD.available(cls, path or 'http')
 
     def _make_cmd(self, tmpfilename, info_dict):
         cmd = ['http', '--download', '--output', tmpfilename, info_dict['url']]
@@ -324,10 +338,11 @@ class HttpieFD(ExternalFD):
 
 
 class FFmpegFD(ExternalFD):
-    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'm3u8_native', 'rtsp', 'rtmp', 'rtmp_ffmpeg', 'mms')
 
     @classmethod
-    def available(cls, path=None):  # path is ignored for ffmpeg
+    def available(cls, path=None):
+        # TODO: Fix path for ffmpeg
         return FFmpegPostProcessor().available
 
     def _call_downloader(self, tmpfilename, info_dict):
@@ -484,4 +499,4 @@ def get_external_downloader(external_downloader):
         downloader . """
     # Drop .exe extension on Windows
     bn = os.path.splitext(os.path.basename(external_downloader))[0]
-    return _BY_NAME[bn]
+    return _BY_NAME.get(bn)

yt_dlp/downloader/fragment.py

@@ -31,6 +31,7 @@ class FragmentFD(FileDownloader):
                         Skip unavailable fragments (DASH and hlsnative only)
     keep_fragments:     Keep downloaded fragments on disk after downloading is
                         finished
+    _no_ytdl_file:      Don't use .ytdl file
 
     For each incomplete fragment download yt-dlp keeps on disk a special
     bookkeeping file with download state and metadata (in future such files will
@@ -69,15 +70,17 @@ class FragmentFD(FileDownloader):
         self._prepare_frag_download(ctx)
         self._start_frag_download(ctx)
 
-    @staticmethod
-    def __do_ytdl_file(ctx):
-        return not ctx['live'] and not ctx['tmpfilename'] == '-'
+    def __do_ytdl_file(self, ctx):
+        return not ctx['live'] and not ctx['tmpfilename'] == '-' and not self.params.get('_no_ytdl_file')
 
     def _read_ytdl_file(self, ctx):
         assert 'ytdl_corrupt' not in ctx
         stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'r')
         try:
-            ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
+            ytdl_data = json.loads(stream.read())
+            ctx['fragment_index'] = ytdl_data['downloader']['current_fragment']['index']
+            if 'extra_state' in ytdl_data['downloader']:
+                ctx['extra_state'] = ytdl_data['downloader']['extra_state']
         except Exception:
             ctx['ytdl_corrupt'] = True
         finally:
@@ -90,6 +93,8 @@ class FragmentFD(FileDownloader):
                 'index': ctx['fragment_index'],
             },
         }
+        if 'extra_state' in ctx:
+            downloader['extra_state'] = ctx['extra_state']
         if ctx.get('fragment_count') is not None:
             downloader['fragment_count'] = ctx['fragment_count']
         frag_index_stream.write(json.dumps({'downloader': downloader}))

yt_dlp/downloader/hls.py

@@ -1,6 +1,8 @@
 from __future__ import unicode_literals
 
+import errno
 import re
+import io
 import binascii
 try:
     from Crypto.Cipher import AES
@@ -26,13 +28,15 @@ from ..utils import (
     parse_m3u8_attributes,
     sanitize_open,
     update_url_query,
+    bug_reports_message,
 )
+from .. import webvtt
 
 
 class HlsFD(FragmentFD):
     """
     Download segments in a m3u8 manifest. External downloaders can take over
-    the fragment downloads by supporting the 'frag_urls' protocol and
+    the fragment downloads by supporting the 'm3u8_frag_urls' protocol and
     re-defining 'supports_manifest' function
     """
 
@@ -77,6 +81,8 @@ class HlsFD(FragmentFD):
         man_url = info_dict['url']
         self.to_screen('[%s] Downloading m3u8 manifest' % self.FD_NAME)
 
+        is_webvtt = info_dict['ext'] == 'vtt'
+
         urlh = self.ydl.urlopen(self._prepare_url(info_dict, man_url))
         man_url = urlh.geturl()
         s = urlh.read().decode('utf-8', 'ignore')
@@ -95,7 +101,7 @@ class HlsFD(FragmentFD):
             #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
-        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+        real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
         if real_downloader and not real_downloader.supports_manifest(s):
             real_downloader = None
         if real_downloader:
@@ -141,6 +147,8 @@ class HlsFD(FragmentFD):
         else:
             self._prepare_and_start_frag_download(ctx)
 
+        extra_state = ctx.setdefault('extra_state', {})
+
         fragment_retries = self.params.get('fragment_retries', 0)
         skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
         test = self.params.get('test', False)
@@ -291,6 +299,7 @@ class HlsFD(FragmentFD):
                         if count <= fragment_retries:
                             self.report_retry_fragment(err, frag_index, count, fragment_retries)
                 if count > fragment_retries:
+                    ctx['dest_stream'].close()
                     self.report_error('Giving up after %s fragment retries' % fragment_retries)
                     return False, frag_index
 
@@ -307,28 +316,105 @@ class HlsFD(FragmentFD):
 
                 return frag_content, frag_index
 
+            pack_fragment = lambda frag_content, _: frag_content
+
+            if is_webvtt:
+                def pack_fragment(frag_content, frag_index):
+                    output = io.StringIO()
+                    adjust = 0
+                    for block in webvtt.parse_fragment(frag_content):
+                        if isinstance(block, webvtt.CueBlock):
+                            block.start += adjust
+                            block.end += adjust
+
+                            dedup_window = extra_state.setdefault('webvtt_dedup_window', [])
+                            cue = block.as_json
+
+                            # skip the cue if an identical one appears
+                            # in the window of potential duplicates
+                            # and prune the window of unviable candidates
+                            i = 0
+                            skip = True
+                            while i < len(dedup_window):
+                                window_cue = dedup_window[i]
+                                if window_cue == cue:
+                                    break
+                                if window_cue['end'] >= cue['start']:
+                                    i += 1
+                                    continue
+                                del dedup_window[i]
+                            else:
+                                skip = False
+
+                            if skip:
+                                continue
+
+                            # add the cue to the window
+                            dedup_window.append(cue)
+                        elif isinstance(block, webvtt.Magic):
+                            # take care of MPEG PES timestamp overflow
+                            if block.mpegts is None:
+                                block.mpegts = 0
+                            extra_state.setdefault('webvtt_mpegts_adjust', 0)
+                            block.mpegts += extra_state['webvtt_mpegts_adjust'] << 33
+                            if block.mpegts < extra_state.get('webvtt_mpegts_last', 0):
+                                extra_state['webvtt_mpegts_adjust'] += 1
+                                block.mpegts += 1 << 33
+                            extra_state['webvtt_mpegts_last'] = block.mpegts
+
+                            if frag_index == 1:
+                                extra_state['webvtt_mpegts'] = block.mpegts or 0
+                                extra_state['webvtt_local'] = block.local or 0
+                                # XXX: block.local = block.mpegts = None ?
+                            else:
+                                if block.mpegts is not None and block.local is not None:
+                                    adjust = (
+                                        (block.mpegts - extra_state.get('webvtt_mpegts', 0))
+                                        - (block.local - extra_state.get('webvtt_local', 0))
+                                    )
+                                continue
+                        elif isinstance(block, webvtt.HeaderBlock):
+                            if frag_index != 1:
+                                # XXX: this should probably be silent as well
+                                # or verify that all segments contain the same data
+                                self.report_warning(bug_reports_message(
+                                    'Discarding a %s block found in the middle of the stream; '
+                                    'if the subtitles display incorrectly,'
+                                    % (type(block).__name__)))
+                                continue
+                        block.write_into(output)
+
+                    return output.getvalue().encode('utf-8')
+
             def append_fragment(frag_content, frag_index):
+                fatal = frag_index == 1 or not skip_unavailable_fragments
                 if frag_content:
                     fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
                     try:
                         file, frag_sanitized = sanitize_open(fragment_filename, 'rb')
                         ctx['fragment_filename_sanitized'] = frag_sanitized
                         file.close()
+                        frag_content = pack_fragment(frag_content, frag_index)
                         self._append_fragment(ctx, frag_content)
                         return True
-                    except FileNotFoundError:
-                        if skip_unavailable_fragments:
+                    except EnvironmentError as ose:
+                        if ose.errno != errno.ENOENT:
+                            raise
+                        # FileNotFoundError
+                        if not fatal:
                             self.report_skip_fragment(frag_index)
                             return True
                         else:
+                            ctx['dest_stream'].close()
                             self.report_error(
                                 'fragment %s not found, unable to continue' % frag_index)
                             return False
                 else:
-                    if skip_unavailable_fragments:
+                    if not fatal:
                         self.report_skip_fragment(frag_index)
                         return True
                     else:
+                        ctx['dest_stream'].close()
                         self.report_error(
                             'fragment %s not found, unable to continue' % frag_index)
                         return False

yt_dlp/downloader/ism.py

@@ -48,7 +48,7 @@ def write_piff_header(stream, params):
     language = params.get('language', 'und')
     height = params.get('height', 0)
     width = params.get('width', 0)
-    is_audio = width == 0 and height == 0
+    stream_type = params['stream_type']
     creation_time = modification_time = int(time.time())
 
     ftyp_payload = b'isml'  # major brand
@@ -77,7 +77,7 @@ def write_piff_header(stream, params):
     tkhd_payload += u32.pack(0) * 2  # reserved
     tkhd_payload += s16.pack(0)  # layer
     tkhd_payload += s16.pack(0)  # alternate group
-    tkhd_payload += s88.pack(1 if is_audio else 0)  # volume
+    tkhd_payload += s88.pack(1 if stream_type == 'audio' else 0)  # volume
     tkhd_payload += u16.pack(0)  # reserved
     tkhd_payload += unity_matrix
     tkhd_payload += u1616.pack(width)
@@ -93,19 +93,34 @@ def write_piff_header(stream, params):
     mdia_payload = full_box(b'mdhd', 1, 0, mdhd_payload)  # Media Header Box
 
     hdlr_payload = u32.pack(0)  # pre defined
-    hdlr_payload += b'soun' if is_audio else b'vide'  # handler type
-    hdlr_payload += u32.pack(0) * 3  # reserved
-    hdlr_payload += (b'Sound' if is_audio else b'Video') + b'Handler\0'  # name
+    if stream_type == 'audio':  # handler type
+        hdlr_payload += b'soun'
+        hdlr_payload += u32.pack(0) * 3  # reserved
+        hdlr_payload += b'SoundHandler\0'  # name
+    elif stream_type == 'video':
+        hdlr_payload += b'vide'
+        hdlr_payload += u32.pack(0) * 3  # reserved
+        hdlr_payload += b'VideoHandler\0'  # name
+    elif stream_type == 'text':
+        hdlr_payload += b'subt'
+        hdlr_payload += u32.pack(0) * 3  # reserved
+        hdlr_payload += b'SubtitleHandler\0'  # name
+    else:
+        assert False
     mdia_payload += full_box(b'hdlr', 0, 0, hdlr_payload)  # Handler Reference Box
 
-    if is_audio:
+    if stream_type == 'audio':
         smhd_payload = s88.pack(0)  # balance
         smhd_payload += u16.pack(0)  # reserved
         media_header_box = full_box(b'smhd', 0, 0, smhd_payload)  # Sound Media Header
-    else:
+    elif stream_type == 'video':
         vmhd_payload = u16.pack(0)  # graphics mode
         vmhd_payload += u16.pack(0) * 3  # opcolor
         media_header_box = full_box(b'vmhd', 0, 1, vmhd_payload)  # Video Media Header
+    elif stream_type == 'text':
+        media_header_box = full_box(b'sthd', 0, 0, b'')  # Subtitle Media Header
+    else:
+        assert False
     minf_payload = media_header_box
 
     dref_payload = u32.pack(1)  # entry count
@@ -117,7 +132,7 @@ def write_piff_header(stream, params):
 
     sample_entry_payload = u8.pack(0) * 6  # reserved
     sample_entry_payload += u16.pack(1)  # data reference index
-    if is_audio:
+    if stream_type == 'audio':
         sample_entry_payload += u32.pack(0) * 2  # reserved
         sample_entry_payload += u16.pack(params.get('channels', 2))
         sample_entry_payload += u16.pack(params.get('bits_per_sample', 16))
@@ -127,7 +142,7 @@ def write_piff_header(stream, params):
 
         if fourcc == 'AACL':
             sample_entry_box = box(b'mp4a', sample_entry_payload)
-    else:
+    elif stream_type == 'video':
         sample_entry_payload += u16.pack(0)  # pre defined
         sample_entry_payload += u16.pack(0)  # reserved
         sample_entry_payload += u32.pack(0) * 3  # pre defined
@@ -155,6 +170,18 @@ def write_piff_header(stream, params):
             avcc_payload += pps
             sample_entry_payload += box(b'avcC', avcc_payload)  # AVC Decoder Configuration Record
             sample_entry_box = box(b'avc1', sample_entry_payload)  # AVC Simple Entry
+        else:
+            assert False
+    elif stream_type == 'text':
+        if fourcc == 'TTML':
+            sample_entry_payload += b'http://www.w3.org/ns/ttml\0'  # namespace
+            sample_entry_payload += b'\0'  # schema location
+            sample_entry_payload += b'\0'  # auxilary mime types(??)
+            sample_entry_box = box(b'stpp', sample_entry_payload)
+        else:
+            assert False
+    else:
+        assert False
     stsd_payload += sample_entry_box
 
     stbl_payload = full_box(b'stsd', 0, 0, stsd_payload)  # Sample Description Box
@@ -221,10 +248,13 @@ class IsmFD(FragmentFD):
 
         self._prepare_and_start_frag_download(ctx)
 
+        extra_state = ctx.setdefault('extra_state', {
+            'ism_track_written': False,
+        })
+
         fragment_retries = self.params.get('fragment_retries', 0)
         skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
 
-        track_written = False
         frag_index = 0
         for i, segment in enumerate(segments):
             frag_index += 1
@@ -236,11 +266,11 @@ class IsmFD(FragmentFD):
                     success, frag_content = self._download_fragment(ctx, segment['url'], info_dict)
                     if not success:
                         return False
-                    if not track_written:
+                    if not extra_state['ism_track_written']:
                         tfhd_data = extract_box_data(frag_content, [b'moof', b'traf', b'tfhd'])
                         info_dict['_download_params']['track_id'] = u32.unpack(tfhd_data[4:8])[0]
                         write_piff_header(ctx['dest_stream'], info_dict['_download_params'])
-                        track_written = True
+                        extra_state['ism_track_written'] = True
                     self._append_fragment(ctx, frag_content)
                     break
                 except compat_urllib_error.HTTPError as err:

yt_dlp/downloader/niconico.py

@@ -24,16 +24,14 @@ class NiconicoDmcFD(FileDownloader):
 
         success = download_complete = False
         timer = [None]
-
         heartbeat_lock = threading.Lock()
         heartbeat_url = heartbeat_info_dict['url']
-        heartbeat_data = heartbeat_info_dict['data']
+        heartbeat_data = heartbeat_info_dict['data'].encode()
         heartbeat_interval = heartbeat_info_dict.get('interval', 30)
-        self.to_screen('[%s] Heartbeat with %s second interval ...' % (self.FD_NAME, heartbeat_interval))
 
         def heartbeat():
             try:
-                compat_urllib_request.urlopen(url=heartbeat_url, data=heartbeat_data.encode())
+                compat_urllib_request.urlopen(url=heartbeat_url, data=heartbeat_data)
             except Exception:
                 self.to_screen('[%s] Heartbeat failed' % self.FD_NAME)
 
@@ -42,13 +40,16 @@ class NiconicoDmcFD(FileDownloader):
                     timer[0] = threading.Timer(heartbeat_interval, heartbeat)
                     timer[0].start()
 
+        heartbeat_info_dict['ping']()
+        self.to_screen('[%s] Heartbeat with %d second interval ...' % (self.FD_NAME, heartbeat_interval))
         try:
             heartbeat()
+            if type(fd).__name__ == 'HlsFD':
+                info_dict.update(ie._extract_m3u8_formats(info_dict['url'], info_dict['id'])[0])
             success = fd.real_download(filename, info_dict)
         finally:
             if heartbeat_lock:
                 with heartbeat_lock:
                     timer[0].cancel()
                     download_complete = True
-
             return success

yt_dlp/extractor/__init__.py

@@ -12,9 +12,6 @@ except ImportError:
 
 if not _LAZY_LOADER:
     from .extractors import *
-
-    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
-
     _ALL_CLASSES = [
         klass
         for name, klass in globals().items()
@@ -22,6 +19,9 @@ if not _LAZY_LOADER:
     ]
     _ALL_CLASSES.append(GenericIE)
 
+    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
+    _ALL_CLASSES = _PLUGIN_CLASSES + _ALL_CLASSES
+
 
 def gen_extractor_classes():
     """ Return a list of supported extractors.

yt_dlp/extractor/afreecatv.py

@@ -257,7 +257,7 @@ class AfreecaTVIE(InfoExtractor):
             if flag and flag == 'SUCCEED':
                 break
             if flag == 'PARTIAL_ADULT':
-                self._downloader.report_warning(
+                self.report_warning(
                     'In accordance with local laws and regulations, underage users are restricted from watching adult content. '
                     'Only content suitable for all ages will be downloaded. '
                     'Provide account credentials if you wish to download restricted content.')
@@ -323,7 +323,7 @@ class AfreecaTVIE(InfoExtractor):
                         'url': file_url,
                         'format_id': 'http',
                     }]
-                if not formats:
+                if not formats and not self._downloader.params.get('ignore_no_formats'):
                     continue
                 self._sort_formats(formats)
                 file_info = common_entry.copy()

yt_dlp/extractor/ard.py

@@ -36,12 +36,12 @@ class ARDMediathekBaseIE(InfoExtractor):
 
         if not formats:
             if fsk:
-                raise ExtractorError(
+                self.raise_no_formats(
                     'This video is only available after 20:00', expected=True)
             elif media_info.get('_geoblocked'):
                 self.raise_geo_restricted(
                     'This video is not available due to geoblocking',
-                    countries=self._GEO_COUNTRIES)
+                    countries=self._GEO_COUNTRIES, metadata_available=True)
 
         self._sort_formats(formats)
 

yt_dlp/extractor/atresplayer.py

@@ -86,18 +86,19 @@ class AtresPlayerIE(InfoExtractor):
         title = episode['titulo']
 
         formats = []
+        subtitles = {}
         for source in episode.get('sources', []):
             src = source.get('src')
             if not src:
                 continue
             src_type = source.get('type')
             if src_type == 'application/vnd.apple.mpegurl':
-                formats.extend(self._extract_m3u8_formats(
+                formats, subtitles = self._extract_m3u8_formats(
                     src, video_id, 'mp4', 'm3u8_native',
-                    m3u8_id='hls', fatal=False))
+                    m3u8_id='hls', fatal=False)
             elif src_type == 'application/dash+xml':
-                formats.extend(self._extract_mpd_formats(
-                    src, video_id, mpd_id='dash', fatal=False))
+                formats, subtitles = self._extract_mpd_formats(
+                    src, video_id, mpd_id='dash', fatal=False)
         self._sort_formats(formats)
 
         heartbeat = episode.get('heartbeat') or {}
@@ -115,4 +116,5 @@ class AtresPlayerIE(InfoExtractor):
             'channel': get_meta('channel'),
             'season': get_meta('season'),
             'episode_number': int_or_none(get_meta('episodeNumber')),
+            'subtitles': subtitles,
         }

yt_dlp/extractor/bbc.py

@@ -11,6 +11,7 @@ from ..compat import (
     compat_etree_Element,
     compat_HTTPError,
     compat_parse_qs,
+    compat_str,
     compat_urllib_parse_urlparse,
     compat_urlparse,
 )
@@ -25,8 +26,10 @@ from ..utils import (
     js_to_json,
     parse_duration,
     parse_iso8601,
+    strip_or_none,
     try_get,
     unescapeHTML,
+    unified_timestamp,
     url_or_none,
     urlencode_postdata,
     urljoin,
@@ -761,8 +764,17 @@ class BBCIE(BBCCoUkIE):
         'only_matching': True,
     }, {
         # custom redirection to www.bbc.com
+        # also, video with window.__INITIAL_DATA__
         'url': 'http://www.bbc.co.uk/news/science-environment-33661876',
-        'only_matching': True,
+        'info_dict': {
+            'id': 'p02xzws1',
+            'ext': 'mp4',
+            'title': "Pluto may have 'nitrogen glaciers'",
+            'description': 'md5:6a95b593f528d7a5f2605221bc56912f',
+            'thumbnail': r're:https?://.+/.+\.jpg',
+            'timestamp': 1437785037,
+            'upload_date': '20150725',
+        },
     }, {
         # single video article embedded with data-media-vpid
         'url': 'http://www.bbc.co.uk/sport/rowing/35908187',
@@ -1164,12 +1176,29 @@ class BBCIE(BBCCoUkIE):
                         continue
                     formats, subtitles = self._download_media_selector(item_id)
                     self._sort_formats(formats)
+                    item_desc = None
+                    blocks = try_get(media, lambda x: x['summary']['blocks'], list)
+                    if blocks:
+                        summary = []
+                        for block in blocks:
+                            text = try_get(block, lambda x: x['model']['text'], compat_str)
+                            if text:
+                                summary.append(text)
+                        if summary:
+                            item_desc = '\n\n'.join(summary)
+                    item_time = None
+                    for meta in try_get(media, lambda x: x['metadata']['items'], list) or []:
+                        if try_get(meta, lambda x: x['label']) == 'Published':
+                            item_time = unified_timestamp(meta.get('timestamp'))
+                            break
                     entries.append({
                         'id': item_id,
                         'title': item_title,
                         'thumbnail': item.get('holdingImageUrl'),
                         'formats': formats,
                         'subtitles': subtitles,
+                        'timestamp': item_time,
+                        'description': strip_or_none(item_desc),
                     })
             for resp in (initial_data.get('data') or {}).values():
                 name = resp.get('name')
@@ -1242,7 +1271,7 @@ class BBCIE(BBCCoUkIE):
         entries = []
         for num, media_meta in enumerate(medias, start=1):
             formats, subtitles = self._extract_from_media_meta(media_meta, playlist_id)
-            if not formats:
+            if not formats and not self._downloader.params.get('ignore_no_formats'):
                 continue
             self._sort_formats(formats)
 

yt_dlp/extractor/bilibili.py

@@ -2,6 +2,7 @@
 from __future__ import unicode_literals
 
 import hashlib
+import itertools
 import json
 import re
 
@@ -120,6 +121,7 @@ class BiliBiliIE(InfoExtractor):
         'url': 'https://www.bilibili.com/video/BV1bK411W797',
         'info_dict': {
             'id': 'BV1bK411W797',
+            'title': '物语中的人物是如何吐槽自己的OP的'
         },
         'playlist_count': 17,
     }]
@@ -156,7 +158,8 @@ class BiliBiliIE(InfoExtractor):
                 if r is not None:
                     self.to_screen('Downloading anthology %s - add --no-playlist to just download video' % video_id)
                     return r
-            self.to_screen('Downloading just video %s because of --no-playlist' % video_id)
+            else:
+                self.to_screen('Downloading just video %s because of --no-playlist' % video_id)
 
         if 'anime/' not in url:
             cid = self._search_regex(
@@ -274,7 +277,7 @@ class BiliBiliIE(InfoExtractor):
         }
 
         uploader_mobj = re.search(
-            r'<a[^>]+href="(?:https?:)?//space\.bilibili\.com/(?P<id>\d+)"[^>]*>(?P<name>[^<]+)',
+            r'<a[^>]+href="(?:https?:)?//space\.bilibili\.com/(?P<id>\d+)"[^>]*>\s*(?P<name>[^<]+?)\s*<',
             webpage)
         if uploader_mobj:
             info.update({
@@ -496,28 +499,40 @@ class BiliBiliBangumiIE(InfoExtractor):
 
 class BilibiliChannelIE(InfoExtractor):
     _VALID_URL = r'https?://space.bilibili\.com/(?P<id>\d+)'
-    # May need to add support for pagination? Need to find a user with many video uploads to test
-    _API_URL = "https://api.bilibili.com/x/space/arc/search?mid=%s&pn=1&ps=25&jsonp=jsonp"
-    _TEST = {}  # TODO: Add tests
+    _API_URL = "https://api.bilibili.com/x/space/arc/search?mid=%s&pn=%d&jsonp=jsonp"
+    _TESTS = [{
+        'url': 'https://space.bilibili.com/3985676/video',
+        'info_dict': {},
+        'playlist_mincount': 112,
+    }]
+
+    def _entries(self, list_id):
+        count, max_count = 0, None
+
+        for page_num in itertools.count(1):
+            data = self._parse_json(
+                self._download_webpage(
+                    self._API_URL % (list_id, page_num), list_id,
+                    note='Downloading page %d' % page_num),
+                list_id)['data']
+
+            max_count = max_count or try_get(data, lambda x: x['page']['count'])
+
+            entries = try_get(data, lambda x: x['list']['vlist'])
+            if not entries:
+                return
+            for entry in entries:
+                yield self.url_result(
+                    'https://www.bilibili.com/video/%s' % entry['bvid'],
+                    BiliBiliIE.ie_key(), entry['bvid'])
+
+            count += len(entries)
+            if max_count and count >= max_count:
+                return
 
     def _real_extract(self, url):
         list_id = self._match_id(url)
-        json_str = self._download_webpage(self._API_URL % list_id, "None")
-
-        json_parsed = json.loads(json_str)
-        entries = [{
-            '_type': 'url',
-            'ie_key': BiliBiliIE.ie_key(),
-            'url': ('https://www.bilibili.com/video/%s' %
-                    entry['bvid']),
-            'id': entry['bvid'],
-        } for entry in json_parsed['data']['list']['vlist']]
-
-        return {
-            '_type': 'playlist',
-            'id': list_id,
-            'entries': entries
-        }
+        return self.playlist_result(self._entries(list_id), list_id)
 
 
 class BiliBiliSearchIE(SearchInfoExtractor):

yt_dlp/extractor/blinkx.py

@@ -78,8 +78,8 @@ class BlinkxIE(InfoExtractor):
             'fullid': video_id,
             'title': data['title'],
             'formats': formats,
-            'uploader': data['channel_name'],
-            'timestamp': data['pubdate_epoch'],
+            'uploader': data.get('channel_name'),
+            'timestamp': data.get('pubdate_epoch'),
             'description': data.get('description'),
             'thumbnails': thumbnails,
             'duration': duration,

yt_dlp/extractor/br.py

@@ -114,7 +114,7 @@ class BRIE(InfoExtractor):
             medias.append(media)
 
         if len(medias) > 1:
-            self._downloader.report_warning(
+            self.report_warning(
                 'found multiple medias; please '
                 'report this with the video URL to http://yt-dl.org/bug')
         if not medias:

yt_dlp/extractor/brightcove.py

@@ -545,9 +545,9 @@ class BrightcoveNewIE(AdobePassIE):
             errors = json_data.get('errors')
             if errors:
                 error = errors[0]
-                raise ExtractorError(
+                self.raise_no_formats(
                     error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
-            if (not self._downloader.params.get('allow_unplayable_formats')
+            elif (not self._downloader.params.get('allow_unplayable_formats')
                     and sources and num_drm_sources == len(sources)):
                 raise ExtractorError('This video is DRM protected.', expected=True)
 

yt_dlp/extractor/byutv.py

@@ -82,6 +82,7 @@ class BYUtvIE(InfoExtractor):
 
         info = {}
         formats = []
+        subtitles = {}
         for format_id, ep in video.items():
             if not isinstance(ep, dict):
                 continue
@@ -90,12 +91,16 @@ class BYUtvIE(InfoExtractor):
                 continue
             ext = determine_ext(video_url)
             if ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
+                m3u8_fmts, m3u8_subs = self._extract_m3u8_formats_and_subtitles(
                     video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                    m3u8_id='hls', fatal=False))
+                    m3u8_id='hls', fatal=False)
+                formats.extend(m3u8_fmts)
+                subtitles = self._merge_subtitles(subtitles, m3u8_subs)
             elif ext == 'mpd':
-                formats.extend(self._extract_mpd_formats(
-                    video_url, video_id, mpd_id='dash', fatal=False))
+                mpd_fmts, mpd_subs = self._extract_mpd_formats_and_subtitles(
+                    video_url, video_id, mpd_id='dash', fatal=False)
+                formats.extend(mpd_fmts)
+                subtitles = self._merge_subtitles(subtitles, mpd_subs)
             else:
                 formats.append({
                     'url': video_url,
@@ -114,4 +119,5 @@ class BYUtvIE(InfoExtractor):
             'display_id': display_id,
             'title': display_id,
             'formats': formats,
+            'subtitles': subtitles,
         })

yt_dlp/extractor/canvas.py

@@ -83,24 +83,31 @@ class CanvasIE(InfoExtractor):
         description = data.get('description')
 
         formats = []
+        subtitles = {}
         for target in data['targetUrls']:
             format_url, format_type = url_or_none(target.get('url')), str_or_none(target.get('type'))
             if not format_url or not format_type:
                 continue
             format_type = format_type.upper()
             if format_type in self._HLS_ENTRY_PROTOCOLS_MAP:
-                formats.extend(self._extract_m3u8_formats(
+                fmts, subs = self._extract_m3u8_formats_and_subtitles(
                     format_url, video_id, 'mp4', self._HLS_ENTRY_PROTOCOLS_MAP[format_type],
-                    m3u8_id=format_type, fatal=False))
+                    m3u8_id=format_type, fatal=False)
+                formats.extend(fmts)
+                subtitles = self._merge_subtitles(subtitles, subs)
             elif format_type == 'HDS':
                 formats.extend(self._extract_f4m_formats(
                     format_url, video_id, f4m_id=format_type, fatal=False))
             elif format_type == 'MPEG_DASH':
-                formats.extend(self._extract_mpd_formats(
-                    format_url, video_id, mpd_id=format_type, fatal=False))
+                fmts, subs = self._extract_mpd_formats_and_subtitles(
+                    format_url, video_id, mpd_id=format_type, fatal=False)
+                formats.extend(fmts)
+                subtitles = self._merge_subtitles(subtitles, subs)
             elif format_type == 'HSS':
-                formats.extend(self._extract_ism_formats(
-                    format_url, video_id, ism_id='mss', fatal=False))
+                fmts, subs = self._extract_ism_formats_and_subtitles(
+                    format_url, video_id, ism_id='mss', fatal=False)
+                formats.extend(fmts)
+                subtitles = self._merge_subtitles(subtitles, subs)
             else:
                 formats.append({
                     'format_id': format_type,
@@ -108,7 +115,6 @@ class CanvasIE(InfoExtractor):
                 })
         self._sort_formats(formats)
 
-        subtitles = {}
         subtitle_urls = data.get('subtitleUrls')
         if isinstance(subtitle_urls, list):
             for subtitle in subtitle_urls:

yt_dlp/extractor/cbs.py

@@ -27,7 +27,13 @@ class CBSBaseIE(ThePlatformFeedIE):
 
 
 class CBSIE(CBSBaseIE):
-    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:(?:cbs|paramountplus)\.com/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'
+    _VALID_URL = r'''(?x)
+        (?:
+            cbs:|
+            https?://(?:www\.)?(?:
+                (?:cbs|paramountplus)\.com/(?:shows/[^/]+/video|movies/[^/]+)/|
+                colbertlateshow\.com/(?:video|podcasts)/)
+        )(?P<id>[\w-]+)'''
 
     _TESTS = [{
         'url': 'https://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
@@ -55,6 +61,9 @@ class CBSIE(CBSBaseIE):
     }, {
         'url': 'https://www.paramountplus.com/shows/all-rise/video/QmR1WhNkh1a_IrdHZrbcRklm176X_rVc/all-rise-space/',
         'only_matching': True,
+    }, {
+        'url': 'https://www.paramountplus.com/movies/million-dollar-american-princesses-meghan-and-harry/C0LpgNwXYeB8txxycdWdR9TjxpJOsdCq',
+        'only_matching': True,
     }]
 
     def _extract_video_info(self, content_id, site='cbs', mpx_acc=2198311517):

yt_dlp/extractor/cbsnews.py

@@ -26,7 +26,7 @@ class CBSNewsEmbedIE(CBSIE):
     def _real_extract(self, url):
         item = self._parse_json(zlib.decompress(compat_b64decode(
             compat_urllib_parse_unquote(self._match_id(url))),
-            -zlib.MAX_WBITS), None)['video']['items'][0]
+            -zlib.MAX_WBITS).decode('utf-8'), None)['video']['items'][0]
         return self._extract_video_info(item['mpxRefId'], 'cbsnews')
 
 

yt_dlp/extractor/cbssports.py

@@ -1,38 +1,113 @@
 from __future__ import unicode_literals
 
-from .cbs import CBSBaseIE
+import re
+
+# from .cbs import CBSBaseIE
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    try_get,
+)
 
 
-class CBSSportsIE(CBSBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/[^/]+/(?:video|news)/(?P<id>[^/?#&]+)'
-
+# class CBSSportsEmbedIE(CBSBaseIE):
+class CBSSportsEmbedIE(InfoExtractor):
+    IE_NAME = 'cbssports:embed'
+    _VALID_URL = r'''(?ix)https?://(?:(?:www\.)?cbs|embed\.247)sports\.com/player/embed.+?
+        (?:
+            ids%3D(?P<id>[\da-f]{8}-(?:[\da-f]{4}-){3}[\da-f]{12})|
+            pcid%3D(?P<pcid>\d+)
+        )'''
     _TESTS = [{
-        'url': 'https://www.cbssports.com/nba/video/donovan-mitchell-flashes-star-potential-in-game-2-victory-over-thunder/',
-        'info_dict': {
-            'id': '1214315075735',
-            'ext': 'mp4',
-            'title': 'Donovan Mitchell flashes star potential in Game 2 victory over Thunder',
-            'description': 'md5:df6f48622612c2d6bd2e295ddef58def',
-            'timestamp': 1524111457,
-            'upload_date': '20180419',
-            'uploader': 'CBSI-NEW',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        }
+        'url': 'https://www.cbssports.com/player/embed/?args=player_id%3Db56c03a6-231a-4bbe-9c55-af3c8a8e9636%26ids%3Db56c03a6-231a-4bbe-9c55-af3c8a8e9636%26resizable%3D1%26autoplay%3Dtrue%26domain%3Dcbssports.com%26comp_ads_enabled%3Dfalse%26watchAndRead%3D0%26startTime%3D0%26env%3Dprod',
+        'only_matching': True,
     }, {
-        'url': 'https://www.cbssports.com/nba/news/nba-playoffs-2018-watch-76ers-vs-heat-game-3-series-schedule-tv-channel-online-stream/',
+        'url': 'https://embed.247sports.com/player/embed/?args=%3fplayer_id%3d1827823171591%26channel%3dcollege-football-recruiting%26pcid%3d1827823171591%26width%3d640%26height%3d360%26autoplay%3dTrue%26comp_ads_enabled%3dFalse%26uvpc%3dhttps%253a%252f%252fwww.cbssports.com%252fapi%252fcontent%252fvideo%252fconfig%252f%253fcfg%253duvp_247sports_v4%2526partner%253d247%26uvpc_m%3dhttps%253a%252f%252fwww.cbssports.com%252fapi%252fcontent%252fvideo%252fconfig%252f%253fcfg%253duvp_247sports_m_v4%2526partner_m%253d247_mobile%26utag%3d247sportssite%26resizable%3dTrue',
         'only_matching': True,
     }]
 
-    def _extract_video_info(self, filter_query, video_id):
-        return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
+    # def _extract_video_info(self, filter_query, video_id):
+    #     return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
 
+    def _real_extract(self, url):
+        uuid, pcid = re.match(self._VALID_URL, url).groups()
+        query = {'id': uuid} if uuid else {'pcid': pcid}
+        video = self._download_json(
+            'https://www.cbssports.com/api/content/video/',
+            uuid or pcid, query=query)[0]
+        video_id = video['id']
+        title = video['title']
+        metadata = video.get('metaData') or {}
+        # return self._extract_video_info('byId=%d' % metadata['mpxOutletId'], video_id)
+        # return self._extract_video_info('byGuid=' + metadata['mpxRefId'], video_id)
+
+        formats = self._extract_m3u8_formats(
+            metadata['files'][0]['url'], video_id, 'mp4',
+            'm3u8_native', m3u8_id='hls', fatal=False)
+        self._sort_formats(formats)
+
+        image = video.get('image')
+        thumbnails = None
+        if image:
+            image_path = image.get('path')
+            if image_path:
+                thumbnails = [{
+                    'url': image_path,
+                    'width': int_or_none(image.get('width')),
+                    'height': int_or_none(image.get('height')),
+                    'filesize': int_or_none(image.get('size')),
+                }]
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'description': video.get('description'),
+            'timestamp': int_or_none(try_get(video, lambda x: x['dateCreated']['epoch'])),
+            'duration': int_or_none(metadata.get('duration')),
+        }
+
+
+class CBSSportsBaseIE(InfoExtractor):
     def _real_extract(self, url):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
-        video_id = self._search_regex(
-            [r'(?:=|%26)pcid%3D(\d+)', r'embedVideo(?:Container)?_(\d+)'],
-            webpage, 'video id')
-        return self._extract_video_info('byId=%s' % video_id, video_id)
+        iframe_url = self._search_regex(
+            r'<iframe[^>]+(?:data-)?src="(https?://[^/]+/player/embed[^"]+)"',
+            webpage, 'embed url')
+        return self.url_result(iframe_url, CBSSportsEmbedIE.ie_key())
+
+
+class CBSSportsIE(CBSSportsBaseIE):
+    IE_NAME = 'cbssports'
+    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/[^/]+/video/(?P<id>[^/?#&]+)'
+    _TESTS = [{
+        'url': 'https://www.cbssports.com/college-football/video/cover-3-stanford-spring-gleaning/',
+        'info_dict': {
+            'id': 'b56c03a6-231a-4bbe-9c55-af3c8a8e9636',
+            'ext': 'mp4',
+            'title': 'Cover 3: Stanford Spring Gleaning',
+            'description': 'The Cover 3 crew break down everything you need to know about the Stanford Cardinal this spring.',
+            'timestamp': 1617218398,
+            'upload_date': '20210331',
+            'duration': 502,
+        },
+    }]
+
+
+class TwentyFourSevenSportsIE(CBSSportsBaseIE):
+    IE_NAME = '247sports'
+    _VALID_URL = r'https?://(?:www\.)?247sports\.com/Video/(?:[^/?#&]+-)?(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://247sports.com/Video/2021-QB-Jake-Garcia-senior-highlights-through-five-games-10084854/',
+        'info_dict': {
+            'id': '4f1265cb-c3b5-44a8-bb1d-1914119a0ccc',
+            'ext': 'mp4',
+            'title': '2021 QB Jake Garcia senior highlights through five games',
+            'description': 'md5:8cb67ebed48e2e6adac1701e0ff6e45b',
+            'timestamp': 1607114223,
+            'upload_date': '20201204',
+            'duration': 208,
+        },
+    }]

yt_dlp/extractor/cda.py

@@ -133,6 +133,8 @@ class CDAIE(InfoExtractor):
             'age_limit': 18 if need_confirm_age else 0,
         }
 
+        info = self._search_json_ld(webpage, video_id, default={})
+
         # Source: https://www.cda.pl/js/player.js?t=1606154898
         def decrypt_file(a):
             for p in ('_XDDD', '_CDA', '_ADC', '_CXD', '_QWE', '_Q5', '_IKSDE'):
@@ -197,7 +199,7 @@ class CDAIE(InfoExtractor):
                 handler = self._download_webpage
 
             webpage = handler(
-                self._BASE_URL + href, video_id,
+                urljoin(self._BASE_URL, href), video_id,
                 'Downloading %s version information' % resolution, fatal=False)
             if not webpage:
                 # Manually report warning because empty page is returned when
@@ -209,6 +211,4 @@ class CDAIE(InfoExtractor):
 
         self._sort_formats(formats)
 
-        info = self._search_json_ld(webpage, video_id, default={})
-
         return merge_dicts(info_dict, info)

yt_dlp/extractor/channel9.py

@@ -5,7 +5,6 @@ import re
 from .common import InfoExtractor
 from ..utils import (
     clean_html,
-    ExtractorError,
     int_or_none,
     parse_iso8601,
     qualities,
@@ -187,14 +186,13 @@ class Channel9IE(InfoExtractor):
                     'quality': quality(q, q_url),
                 })
 
-            self._sort_formats(formats)
-
             slides = content_data.get('Slides')
             zip_file = content_data.get('ZipFile')
 
             if not formats and not slides and not zip_file:
-                raise ExtractorError(
+                self.raise_no_formats(
                     'None of recording, slides or zip are available for %s' % content_path)
+            self._sort_formats(formats)
 
             subtitles = {}
             for caption in content_data.get('Captions', []):

yt_dlp/extractor/common.py

@@ -9,15 +9,13 @@ import netrc
 import os
 import random
 import re
-import socket
-import ssl
 import sys
 import time
 import math
 
 from ..compat import (
     compat_cookiejar_Cookie,
-    compat_cookies,
+    compat_cookies_SimpleCookie,
     compat_etree_Element,
     compat_etree_fromstring,
     compat_getpass,
@@ -58,6 +56,7 @@ from ..utils import (
     js_to_json,
     JSON_LD_RE,
     mimetype2ext,
+    network_exceptions,
     orderedSet,
     parse_bitrate,
     parse_codecs,
@@ -157,7 +156,7 @@ class InfoExtractor(object):
                     * player_url SWF Player URL (used for rtmpdump).
                     * protocol   The protocol that will be used for the actual
                                  download, lower-case.
-                                 "http", "https", "rtsp", "rtmp", "rtmpe",
+                                 "http", "https", "rtsp", "rtmp", "rtmp_ffmpeg", "rtmpe",
                                  "m3u8", "m3u8_native" or "http_dash_segments".
                     * fragment_base_url
                                  Base URL for fragments. Each fragment's path
@@ -558,6 +557,10 @@ class InfoExtractor(object):
                     ie_result = self._real_extract(url)
                     if self._x_forwarded_for_ip:
                         ie_result['__x_forwarded_for_ip'] = self._x_forwarded_for_ip
+                    subtitles = ie_result.get('subtitles')
+                    if (subtitles and 'live_chat' in subtitles
+                            and 'no-live-chat' in self._downloader.params.get('compat_opts')):
+                        del subtitles['live_chat']
                     return ie_result
                 except GeoRestrictedError as e:
                     if self.__maybe_fake_ip_and_retry(e.countries):
@@ -659,12 +662,9 @@ class InfoExtractor(object):
                 url_or_request = update_url_query(url_or_request, query)
             if data is not None or headers:
                 url_or_request = sanitized_Request(url_or_request, data, headers)
-        exceptions = [compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error]
-        if hasattr(ssl, 'CertificateError'):
-            exceptions.append(ssl.CertificateError)
         try:
             return self._downloader.urlopen(url_or_request)
-        except tuple(exceptions) as err:
+        except network_exceptions as err:
             if isinstance(err, compat_urllib_error.HTTPError):
                 if self.__can_accept_status_code(err, expected_status):
                     # Retain reference to error to prevent file object from
@@ -683,7 +683,7 @@ class InfoExtractor(object):
             if fatal:
                 raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
             else:
-                self._downloader.report_warning(errmsg)
+                self.report_warning(errmsg)
                 return False
 
     def _download_webpage_handle(self, url_or_request, video_id, note=None, errnote=None, fatal=True, encoding=None, data=None, headers={}, query={}, expected_status=None):
@@ -968,15 +968,27 @@ class InfoExtractor(object):
         """Report attempt to log in."""
         self.to_screen('Logging in')
 
-    @staticmethod
-    def raise_login_required(msg='This video is only available for registered users'):
+    def raise_login_required(
+            self, msg='This video is only available for registered users', metadata_available=False):
+        if metadata_available and self._downloader.params.get('ignore_no_formats_error'):
+            self.report_warning(msg)
         raise ExtractorError(
-            '%s. Use --username and --password or --netrc to provide account credentials.' % msg,
+            '%s. Use --cookies, --username and --password or --netrc to provide account credentials' % msg,
             expected=True)
 
-    @staticmethod
-    def raise_geo_restricted(msg='This video is not available from your location due to geo restriction', countries=None):
-        raise GeoRestrictedError(msg, countries=countries)
+    def raise_geo_restricted(
+            self, msg='This video is not available from your location due to geo restriction',
+            countries=None, metadata_available=False):
+        if metadata_available and self._downloader.params.get('ignore_no_formats_error'):
+            self.report_warning(msg)
+        else:
+            raise GeoRestrictedError(msg, countries=countries)
+
+    def raise_no_formats(self, msg, expected=False, video_id=None):
+        if expected and self._downloader.params.get('ignore_no_formats_error'):
+            self.report_warning(msg, video_id)
+        else:
+            raise ExtractorError(msg, expected=expected, video_id=video_id)
 
     # Methods for following #608
     @staticmethod
@@ -1044,7 +1056,7 @@ class InfoExtractor(object):
         elif fatal:
             raise RegexNotFoundError('Unable to extract %s' % _name)
         else:
-            self._downloader.report_warning('unable to extract %s' % _name + bug_reports_message())
+            self.report_warning('unable to extract %s' % _name + bug_reports_message())
             return None
 
     def _html_search_regex(self, pattern, string, name, default=NO_DEFAULT, fatal=True, flags=0, group=None):
@@ -1072,7 +1084,7 @@ class InfoExtractor(object):
                     raise netrc.NetrcParseError(
                         'No authenticators for %s' % netrc_machine)
             except (IOError, netrc.NetrcParseError) as err:
-                self._downloader.report_warning(
+                self.report_warning(
                     'parsing .netrc: %s' % error_to_compat_str(err))
 
         return username, password
@@ -1247,7 +1259,7 @@ class InfoExtractor(object):
         elif fatal:
             raise RegexNotFoundError('Unable to extract JSON-LD')
         else:
-            self._downloader.report_warning('unable to extract JSON-LD %s' % bug_reports_message())
+            self.report_warning('unable to extract JSON-LD %s' % bug_reports_message())
             return {}
 
     def _json_ld(self, json_ld, video_id, fatal=True, expected_type=None):
@@ -1308,6 +1320,7 @@ class InfoExtractor(object):
 
         def extract_video_object(e):
             assert e['@type'] == 'VideoObject'
+            author = e.get('author')
             info.update({
                 'url': url_or_none(e.get('contentUrl')),
                 'title': unescapeHTML(e.get('name')),
@@ -1315,7 +1328,11 @@ class InfoExtractor(object):
                 'thumbnail': url_or_none(e.get('thumbnailUrl') or e.get('thumbnailURL')),
                 'duration': parse_duration(e.get('duration')),
                 'timestamp': unified_timestamp(e.get('uploadDate')),
-                'uploader': str_or_none(e.get('author')),
+                # author can be an instance of 'Organization' or 'Person' types.
+                # both types can have 'name' property(inherited from 'Thing' type). [1]
+                # however some websites are using 'Text' type instead.
+                # 1. https://schema.org/VideoObject
+                'uploader': author.get('name') if isinstance(author, dict) else author if isinstance(author, compat_str) else None,
                 'filesize': float_or_none(e.get('contentSize')),
                 'tbr': int_or_none(e.get('bitrate')),
                 'width': int_or_none(e.get('width')),
@@ -1402,7 +1419,10 @@ class InfoExtractor(object):
 
         default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
                    'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
-                   'proto', 'ext', 'has_audio', 'source', 'format_id')  # These must not be aliases
+                   'proto', 'ext', 'hasaud', 'source', 'format_id')  # These must not be aliases
+        ytdl_default = ('hasaud', 'quality', 'tbr', 'filesize', 'vbr',
+                        'height', 'width', 'proto', 'vext', 'abr', 'aext',
+                        'fps', 'fs_approx', 'source', 'format_id')
 
         settings = {
             'vcodec': {'type': 'ordered', 'regex': True,
@@ -1422,7 +1442,7 @@ class InfoExtractor(object):
             'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
             'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
             'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
-            'quality': {'convert': 'float_none'},
+            'quality': {'convert': 'float_none', 'default': -1},
             'filesize': {'convert': 'bytes'},
             'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
             'id': {'convert': 'string', 'field': 'format_id'},
@@ -1575,7 +1595,8 @@ class InfoExtractor(object):
                              else None)
 
         def print_verbose_info(self, to_screen):
-            to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
+            if self._sort_user:
+                to_screen('[debug] Sort order given by user: %s' % ', '.join(self._sort_user))
             if self._sort_extractor:
                 to_screen('[debug] Sort order given by extractor: %s' % ', '.join(self._sort_extractor))
             to_screen('[debug] Formats sorted by: %s' % ', '.join(['%s%s%s' % (
@@ -1603,7 +1624,7 @@ class InfoExtractor(object):
                 value = self._resolve_field_value(field, value, True)
 
             # try to convert to number
-            val_num = float_or_none(value)
+            val_num = float_or_none(value, default=self._get_field_setting(field, 'default'))
             is_num = self._get_field_setting(field, 'convert') != 'string' and val_num is not None
             if is_num:
                 value = val_num
@@ -1664,6 +1685,8 @@ class InfoExtractor(object):
 
     def _sort_formats(self, formats, field_preference=[]):
         if not formats:
+            if self._downloader.params.get('ignore_no_formats_error'):
+                return
             raise ExtractorError('No video formats found')
         format_sort = self.FormatSort()  # params and to_screen are taken from the downloader
         format_sort.evaluate_params(self._downloader.params, field_preference)
@@ -1859,11 +1882,21 @@ class InfoExtractor(object):
             'format_note': 'Quality selection URL',
         }
 
-    def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
-                              entry_protocol='m3u8', preference=None, quality=None,
-                              m3u8_id=None, note=None, errnote=None,
-                              fatal=True, live=False, data=None, headers={},
-                              query={}):
+    def _extract_m3u8_formats(self, *args, **kwargs):
+        fmts, subs = self._extract_m3u8_formats_and_subtitles(*args, **kwargs)
+        if subs:
+            self.report_warning(bug_reports_message(
+                "Ignoring subtitle tracks found in the HLS manifest; "
+                "if any subtitle tracks are missing,"
+            ))
+        return fmts
+
+    def _extract_m3u8_formats_and_subtitles(
+            self, m3u8_url, video_id, ext=None, entry_protocol='m3u8',
+            preference=None, quality=None, m3u8_id=None, note=None,
+            errnote=None, fatal=True, live=False, data=None, headers={},
+            query={}):
+
         res = self._download_webpage_handle(
             m3u8_url, video_id,
             note=note or 'Downloading m3u8 information',
@@ -1871,29 +1904,34 @@ class InfoExtractor(object):
             fatal=fatal, data=data, headers=headers, query=query)
 
         if res is False:
-            return []
+            return [], {}
 
         m3u8_doc, urlh = res
         m3u8_url = urlh.geturl()
 
-        return self._parse_m3u8_formats(
+        return self._parse_m3u8_formats_and_subtitles(
             m3u8_doc, m3u8_url, ext=ext, entry_protocol=entry_protocol,
             preference=preference, quality=quality, m3u8_id=m3u8_id,
             note=note, errnote=errnote, fatal=fatal, live=live, data=data,
             headers=headers, query=query, video_id=video_id)
 
-    def _parse_m3u8_formats(self, m3u8_doc, m3u8_url, ext=None,
-                            entry_protocol='m3u8', preference=None, quality=None,
-                            m3u8_id=None, live=False, note=None, errnote=None,
-                            fatal=True, data=None, headers={}, query={}, video_id=None):
-        if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
-            return []
+    def _parse_m3u8_formats_and_subtitles(
+            self, m3u8_doc, m3u8_url, ext=None, entry_protocol='m3u8',
+            preference=None, quality=None, m3u8_id=None, live=False, note=None,
+            errnote=None, fatal=True, data=None, headers={}, query={},
+            video_id=None):
 
-        if re.search(r'#EXT-X-SESSION-KEY:.*?URI="skd://', m3u8_doc):  # Apple FairPlay
-            return []
+        if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
+            return [], {}
+
+        if (not self._downloader.params.get('allow_unplayable_formats')
+                and re.search(r'#EXT-X-SESSION-KEY:.*?URI="skd://', m3u8_doc)):  # Apple FairPlay
+            return [], {}
 
         formats = []
 
+        subtitles = {}
+
         format_url = lambda u: (
             u
             if re.match(r'^https?://', u)
@@ -1980,7 +2018,7 @@ class InfoExtractor(object):
                 }
                 formats.append(f)
 
-            return formats
+            return formats, subtitles
 
         groups = {}
         last_stream_inf = {}
@@ -1992,6 +2030,21 @@ class InfoExtractor(object):
             if not (media_type and group_id and name):
                 return
             groups.setdefault(group_id, []).append(media)
+            # <https://tools.ietf.org/html/rfc8216#section-4.3.4.1>
+            if media_type == 'SUBTITLES':
+                lang = media['LANGUAGE']  # XXX: normalise?
+                url = format_url(media['URI'])
+                sub_info = {
+                    'url': url,
+                    'ext': determine_ext(url),
+                }
+                if sub_info['ext'] == 'm3u8':
+                    # Per RFC 8216 §3.1, the only possible subtitle format m3u8
+                    # files may contain is WebVTT:
+                    # <https://tools.ietf.org/html/rfc8216#section-3.1>
+                    sub_info['ext'] = 'vtt'
+                    sub_info['protocol'] = 'm3u8_native'
+                subtitles.setdefault(lang, []).append(sub_info)
             if media_type not in ('VIDEO', 'AUDIO'):
                 return
             media_url = media.get('URI')
@@ -2139,7 +2192,7 @@ class InfoExtractor(object):
                         formats.append(http_f)
 
                 last_stream_inf = {}
-        return formats
+        return formats, subtitles
 
     @staticmethod
     def _xpath_ns(path, namespace=None):
@@ -2382,23 +2435,44 @@ class InfoExtractor(object):
             })
         return entries
 
-    def _extract_mpd_formats(self, mpd_url, video_id, mpd_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
+    def _extract_mpd_formats(self, *args, **kwargs):
+        fmts, subs = self._extract_mpd_formats_and_subtitles(*args, **kwargs)
+        if subs:
+            self.report_warning(bug_reports_message(
+                "Ignoring subtitle tracks found in the DASH manifest; "
+                "if any subtitle tracks are missing,"
+            ))
+        return fmts
+
+    def _extract_mpd_formats_and_subtitles(
+            self, mpd_url, video_id, mpd_id=None, note=None, errnote=None,
+            fatal=True, data=None, headers={}, query={}):
         res = self._download_xml_handle(
             mpd_url, video_id,
             note=note or 'Downloading MPD manifest',
             errnote=errnote or 'Failed to download MPD manifest',
             fatal=fatal, data=data, headers=headers, query=query)
         if res is False:
-            return []
+            return [], {}
         mpd_doc, urlh = res
         if mpd_doc is None:
-            return []
+            return [], {}
         mpd_base_url = base_url(urlh.geturl())
 
-        return self._parse_mpd_formats(
+        return self._parse_mpd_formats_and_subtitles(
             mpd_doc, mpd_id, mpd_base_url, mpd_url)
 
-    def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', mpd_url=None):
+    def _parse_mpd_formats(self, *args, **kwargs):
+        fmts, subs = self._parse_mpd_formats_and_subtitles(*args, **kwargs)
+        if subs:
+            self.report_warning(bug_reports_message(
+                "Ignoring subtitle tracks found in the DASH manifest; "
+                "if any subtitle tracks are missing,"
+            ))
+        return fmts
+
+    def _parse_mpd_formats_and_subtitles(
+            self, mpd_doc, mpd_id=None, mpd_base_url='', mpd_url=None):
         """
         Parse formats from MPD manifest.
         References:
@@ -2406,9 +2480,9 @@ class InfoExtractor(object):
             http://standards.iso.org/ittf/PubliclyAvailableStandards/c065274_ISO_IEC_23009-1_2014.zip
          2. https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP
         """
-        if not self._downloader.params.get('dynamic_mpd'):
+        if not self._downloader.params.get('dynamic_mpd', True):
             if mpd_doc.get('type') == 'dynamic':
-                return []
+                return [], {}
 
         namespace = self._search_regex(r'(?i)^{([^}]+)?}MPD$', mpd_doc.tag, 'namespace', default=None)
 
@@ -2480,6 +2554,7 @@ class InfoExtractor(object):
 
         mpd_duration = parse_duration(mpd_doc.get('mediaPresentationDuration'))
         formats = []
+        subtitles = {}
         for period in mpd_doc.findall(_add_ns('Period')):
             period_duration = parse_duration(period.get('duration')) or mpd_duration
             period_ms_info = extract_multisegment_info(period, {
@@ -2497,11 +2572,9 @@ class InfoExtractor(object):
                     representation_attrib.update(representation.attrib)
                     # According to [1, 5.3.7.2, Table 9, page 41], @mimeType is mandatory
                     mime_type = representation_attrib['mimeType']
-                    content_type = mime_type.split('/')[0]
-                    if content_type == 'text':
-                        # TODO implement WebVTT downloading
-                        pass
-                    elif content_type in ('video', 'audio'):
+                    content_type = representation_attrib.get('contentType', mime_type.split('/')[0])
+
+                    if content_type in ('video', 'audio', 'text'):
                         base_url = ''
                         for element in (representation, adaptation_set, period, mpd_doc):
                             base_url_e = element.find(_add_ns('BaseURL'))
@@ -2518,21 +2591,28 @@ class InfoExtractor(object):
                         url_el = representation.find(_add_ns('BaseURL'))
                         filesize = int_or_none(url_el.attrib.get('{http://youtube.com/yt/2012/10/10}contentLength') if url_el is not None else None)
                         bandwidth = int_or_none(representation_attrib.get('bandwidth'))
-                        f = {
-                            'format_id': '%s-%s' % (mpd_id, representation_id) if mpd_id else representation_id,
-                            'manifest_url': mpd_url,
-                            'ext': mimetype2ext(mime_type),
-                            'width': int_or_none(representation_attrib.get('width')),
-                            'height': int_or_none(representation_attrib.get('height')),
-                            'tbr': float_or_none(bandwidth, 1000),
-                            'asr': int_or_none(representation_attrib.get('audioSamplingRate')),
-                            'fps': int_or_none(representation_attrib.get('frameRate')),
-                            'language': lang if lang not in ('mul', 'und', 'zxx', 'mis') else None,
-                            'format_note': 'DASH %s' % content_type,
-                            'filesize': filesize,
-                            'container': mimetype2ext(mime_type) + '_dash',
-                        }
-                        f.update(parse_codecs(representation_attrib.get('codecs')))
+                        if content_type in ('video', 'audio'):
+                            f = {
+                                'format_id': '%s-%s' % (mpd_id, representation_id) if mpd_id else representation_id,
+                                'manifest_url': mpd_url,
+                                'ext': mimetype2ext(mime_type),
+                                'width': int_or_none(representation_attrib.get('width')),
+                                'height': int_or_none(representation_attrib.get('height')),
+                                'tbr': float_or_none(bandwidth, 1000),
+                                'asr': int_or_none(representation_attrib.get('audioSamplingRate')),
+                                'fps': int_or_none(representation_attrib.get('frameRate')),
+                                'language': lang if lang not in ('mul', 'und', 'zxx', 'mis') else None,
+                                'format_note': 'DASH %s' % content_type,
+                                'filesize': filesize,
+                                'container': mimetype2ext(mime_type) + '_dash',
+                            }
+                            f.update(parse_codecs(representation_attrib.get('codecs')))
+                        elif content_type == 'text':
+                            f = {
+                                'ext': mimetype2ext(mime_type),
+                                'manifest_url': mpd_url,
+                                'filesize': filesize,
+                            }
                         representation_ms_info = extract_multisegment_info(representation, adaption_set_ms_info)
 
                         def prepare_template(template_name, identifiers):
@@ -2679,26 +2759,38 @@ class InfoExtractor(object):
                         else:
                             # Assuming direct URL to unfragmented media.
                             f['url'] = base_url
-                        formats.append(f)
+                        if content_type in ('video', 'audio'):
+                            formats.append(f)
+                        elif content_type == 'text':
+                            subtitles.setdefault(lang or 'und', []).append(f)
                     else:
                         self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
-        return formats
+        return formats, subtitles
 
-    def _extract_ism_formats(self, ism_url, video_id, ism_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
+    def _extract_ism_formats(self, *args, **kwargs):
+        fmts, subs = self._extract_ism_formats_and_subtitles(*args, **kwargs)
+        if subs:
+            self.report_warning(bug_reports_message(
+                "Ignoring subtitle tracks found in the ISM manifest; "
+                "if any subtitle tracks are missing,"
+            ))
+        return fmts
+
+    def _extract_ism_formats_and_subtitles(self, ism_url, video_id, ism_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
         res = self._download_xml_handle(
             ism_url, video_id,
             note=note or 'Downloading ISM manifest',
             errnote=errnote or 'Failed to download ISM manifest',
             fatal=fatal, data=data, headers=headers, query=query)
         if res is False:
-            return []
+            return [], {}
         ism_doc, urlh = res
         if ism_doc is None:
-            return []
+            return [], {}
 
-        return self._parse_ism_formats(ism_doc, urlh.geturl(), ism_id)
+        return self._parse_ism_formats_and_subtitles(ism_doc, urlh.geturl(), ism_id)
 
-    def _parse_ism_formats(self, ism_doc, ism_url, ism_id=None):
+    def _parse_ism_formats_and_subtitles(self, ism_doc, ism_url, ism_id=None):
         """
         Parse formats from ISM manifest.
         References:
@@ -2706,26 +2798,28 @@ class InfoExtractor(object):
             https://msdn.microsoft.com/en-us/library/ff469518.aspx
         """
         if ism_doc.get('IsLive') == 'TRUE':
-            return []
+            return [], {}
         if (not self._downloader.params.get('allow_unplayable_formats')
                 and ism_doc.find('Protection') is not None):
-            return []
+            return [], {}
 
         duration = int(ism_doc.attrib['Duration'])
         timescale = int_or_none(ism_doc.get('TimeScale')) or 10000000
 
         formats = []
+        subtitles = {}
         for stream in ism_doc.findall('StreamIndex'):
             stream_type = stream.get('Type')
-            if stream_type not in ('video', 'audio'):
+            if stream_type not in ('video', 'audio', 'text'):
                 continue
             url_pattern = stream.attrib['Url']
             stream_timescale = int_or_none(stream.get('TimeScale')) or timescale
             stream_name = stream.get('Name')
+            stream_language = stream.get('Language', 'und')
             for track in stream.findall('QualityLevel'):
                 fourcc = track.get('FourCC', 'AACL' if track.get('AudioTag') == '255' else None)
                 # TODO: add support for WVC1 and WMAP
-                if fourcc not in ('H264', 'AVC1', 'AACL'):
+                if fourcc not in ('H264', 'AVC1', 'AACL', 'TTML'):
                     self.report_warning('%s is not a supported codec' % fourcc)
                     continue
                 tbr = int(track.attrib['Bitrate']) // 1000
@@ -2768,33 +2862,52 @@ class InfoExtractor(object):
                     format_id.append(stream_name)
                 format_id.append(compat_str(tbr))
 
-                formats.append({
-                    'format_id': '-'.join(format_id),
-                    'url': ism_url,
-                    'manifest_url': ism_url,
-                    'ext': 'ismv' if stream_type == 'video' else 'isma',
-                    'width': width,
-                    'height': height,
-                    'tbr': tbr,
-                    'asr': sampling_rate,
-                    'vcodec': 'none' if stream_type == 'audio' else fourcc,
-                    'acodec': 'none' if stream_type == 'video' else fourcc,
-                    'protocol': 'ism',
-                    'fragments': fragments,
-                    '_download_params': {
-                        'duration': duration,
-                        'timescale': stream_timescale,
-                        'width': width or 0,
-                        'height': height or 0,
-                        'fourcc': fourcc,
-                        'codec_private_data': track.get('CodecPrivateData'),
-                        'sampling_rate': sampling_rate,
-                        'channels': int_or_none(track.get('Channels', 2)),
-                        'bits_per_sample': int_or_none(track.get('BitsPerSample', 16)),
-                        'nal_unit_length_field': int_or_none(track.get('NALUnitLengthField', 4)),
-                    },
-                })
-        return formats
+                if stream_type == 'text':
+                    subtitles.setdefault(stream_language, []).append({
+                        'ext': 'ismt',
+                        'protocol': 'ism',
+                        'url': ism_url,
+                        'manifest_url': ism_url,
+                        'fragments': fragments,
+                        '_download_params': {
+                            'stream_type': stream_type,
+                            'duration': duration,
+                            'timescale': stream_timescale,
+                            'fourcc': fourcc,
+                            'language': stream_language,
+                            'codec_private_data': track.get('CodecPrivateData'),
+                        }
+                    })
+                elif stream_type in ('video', 'audio'):
+                    formats.append({
+                        'format_id': '-'.join(format_id),
+                        'url': ism_url,
+                        'manifest_url': ism_url,
+                        'ext': 'ismv' if stream_type == 'video' else 'isma',
+                        'width': width,
+                        'height': height,
+                        'tbr': tbr,
+                        'asr': sampling_rate,
+                        'vcodec': 'none' if stream_type == 'audio' else fourcc,
+                        'acodec': 'none' if stream_type == 'video' else fourcc,
+                        'protocol': 'ism',
+                        'fragments': fragments,
+                        '_download_params': {
+                            'stream_type': stream_type,
+                            'duration': duration,
+                            'timescale': stream_timescale,
+                            'width': width or 0,
+                            'height': height or 0,
+                            'fourcc': fourcc,
+                            'language': stream_language,
+                            'codec_private_data': track.get('CodecPrivateData'),
+                            'sampling_rate': sampling_rate,
+                            'channels': int_or_none(track.get('Channels', 2)),
+                            'bits_per_sample': int_or_none(track.get('BitsPerSample', 16)),
+                            'nal_unit_length_field': int_or_none(track.get('NALUnitLengthField', 4)),
+                        },
+                    })
+        return formats, subtitles
 
     def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None, quality=None):
         def absolute_url(item_url):
@@ -2919,7 +3032,16 @@ class InfoExtractor(object):
                 entries.append(media_info)
         return entries
 
-    def _extract_akamai_formats(self, manifest_url, video_id, hosts={}):
+    def _extract_akamai_formats(self, *args, **kwargs):
+        fmts, subs = self._extract_akamai_formats_and_subtitles(*args, **kwargs)
+        if subs:
+            self.report_warning(bug_reports_message(
+                "Ignoring subtitle tracks found in the manifests; "
+                "if any subtitle tracks are missing,"
+            ))
+        return fmts
+
+    def _extract_akamai_formats_and_subtitles(self, manifest_url, video_id, hosts={}):
         signed = 'hdnea=' in manifest_url
         if not signed:
             # https://learn.akamai.com/en-us/webhelp/media-services-on-demand/stream-packaging-user-guide/GUID-BE6C0F73-1E06-483B-B0EA-57984B91B7F9.html
@@ -2928,6 +3050,7 @@ class InfoExtractor(object):
                 '', manifest_url).strip('?')
 
         formats = []
+        subtitles = {}
 
         hdcore_sign = 'hdcore=3.7.0'
         f4m_url = re.sub(r'(https?://[^/]+)/i/', r'\1/z/', manifest_url).replace('/master.m3u8', '/manifest.f4m')
@@ -2946,10 +3069,11 @@ class InfoExtractor(object):
         hls_host = hosts.get('hls')
         if hls_host:
             m3u8_url = re.sub(r'(https?://)[^/]+', r'\1' + hls_host, m3u8_url)
-        m3u8_formats = self._extract_m3u8_formats(
+        m3u8_formats, m3u8_subtitles = self._extract_m3u8_formats_and_subtitles(
             m3u8_url, video_id, 'mp4', 'm3u8_native',
             m3u8_id='hls', fatal=False)
         formats.extend(m3u8_formats)
+        subtitles = self._merge_subtitles(subtitles, m3u8_subtitles)
 
         http_host = hosts.get('http')
         if http_host and m3u8_formats and not signed:
@@ -2973,7 +3097,7 @@ class InfoExtractor(object):
                             formats.append(http_f)
                         i += 1
 
-        return formats
+        return formats, subtitles
 
     def _extract_wowza_formats(self, url, video_id, m3u8_entry_protocol='m3u8_native', skip_protocols=[]):
         query = compat_urlparse.urlparse(url).query
@@ -3196,7 +3320,7 @@ class InfoExtractor(object):
             if fatal:
                 raise ExtractorError(msg)
             else:
-                self._downloader.report_warning(msg)
+                self.report_warning(msg)
         return res
 
     def _float(self, v, name, fatal=False, **kwargs):
@@ -3206,7 +3330,7 @@ class InfoExtractor(object):
             if fatal:
                 raise ExtractorError(msg)
             else:
-                self._downloader.report_warning(msg)
+                self.report_warning(msg)
         return res
 
     def _set_cookie(self, domain, name, value, expire_time=None, port=None,
@@ -3218,13 +3342,10 @@ class InfoExtractor(object):
         self._downloader.cookiejar.set_cookie(cookie)
 
     def _get_cookies(self, url):
-        """ Return a compat_cookies.SimpleCookie with the cookies for the url """
+        """ Return a compat_cookies_SimpleCookie with the cookies for the url """
         req = sanitized_Request(url)
         self._downloader.cookiejar.add_cookie_header(req)
-        cookie = req.get_header('Cookie')
-        if cookie and sys.version_info[0] == 2:
-            cookie = str(cookie)
-        return compat_cookies.SimpleCookie(cookie)
+        return compat_cookies_SimpleCookie(req.get_header('Cookie'))
 
     def _apply_first_set_cookie_header(self, url_handle, cookie):
         """
@@ -3301,12 +3422,22 @@ class InfoExtractor(object):
         return ret
 
     @classmethod
-    def _merge_subtitles(cls, subtitle_dict1, subtitle_dict2):
-        """ Merge two subtitle dictionaries, language by language. """
-        ret = dict(subtitle_dict1)
-        for lang in subtitle_dict2:
-            ret[lang] = cls._merge_subtitle_items(subtitle_dict1.get(lang, []), subtitle_dict2[lang])
-        return ret
+    def _merge_subtitles(cls, *dicts, **kwargs):
+        """ Merge subtitle dictionaries, language by language. """
+
+        target = (lambda target=None: target)(**kwargs)
+        # The above lambda extracts the keyword argument 'target' from kwargs
+        # while ensuring there are no stray ones. When Python 2 support
+        # is dropped, remove it and change the function signature to:
+        #
+        #     def _merge_subtitles(cls, *dicts, target=None):
+
+        if target is None:
+            target = {}
+        for d in dicts:
+            for lang, subs in d.items():
+                target[lang] = cls._merge_subtitle_items(target.get(lang, []), subs)
+        return target
 
     def extract_automatic_captions(self, *args, **kwargs):
         if (self._downloader.params.get('writeautomaticsub', False)
@@ -3385,7 +3516,7 @@ class SearchInfoExtractor(InfoExtractor):
             if n <= 0:
                 raise ExtractorError('invalid download number %s for query "%s"' % (n, query))
             elif n > self._MAX_RESULTS:
-                self._downloader.report_warning('%s returns max %i results (you requested %i)' % (self._SEARCH_KEY, self._MAX_RESULTS, n))
+                self.report_warning('%s returns max %i results (you requested %i)' % (self._SEARCH_KEY, self._MAX_RESULTS, n))
                 n = self._MAX_RESULTS
             return self._get_n_results(query, n)
 

yt_dlp/extractor/corus.py

@@ -131,7 +131,7 @@ class CorusIE(ThePlatformFeedIE):
             formats.extend(self._parse_smil_formats(
                 smil, smil_url, video_id, namespace))
         if not formats and video.get('drm'):
-            raise ExtractorError('This video is DRM protected.', expected=True)
+            self.raise_no_formats('This video is DRM protected.', expected=True)
         self._sort_formats(formats)
 
         subtitles = {}

yt_dlp/extractor/crackle.py

@@ -12,6 +12,7 @@ from ..utils import (
     determine_ext,
     float_or_none,
     int_or_none,
+    orderedSet,
     parse_age_limit,
     parse_duration,
     url_or_none,
@@ -66,135 +67,179 @@ class CrackleIE(InfoExtractor):
         },
     }
 
+    def _download_json(self, url, *args, **kwargs):
+        # Authorization generation algorithm is reverse engineered from:
+        # https://www.sonycrackle.com/static/js/main.ea93451f.chunk.js
+        timestamp = time.strftime('%Y%m%d%H%M', time.gmtime())
+        h = hmac.new(b'IGSLUQCBDFHEOIFM', '|'.join([url, timestamp]).encode(), hashlib.sha1).hexdigest().upper()
+        headers = {
+            'Accept': 'application/json',
+            'Authorization': '|'.join([h, timestamp, '117', '1']),
+        }
+        return InfoExtractor._download_json(self, url, *args, headers=headers, **kwargs)
+
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        country_code = self._downloader.params.get('geo_bypass_country', None)
-        countries = [country_code] if country_code else (
-            'US', 'AU', 'CA', 'AS', 'FM', 'GU', 'MP', 'PR', 'PW', 'MH', 'VI')
+        geo_bypass_country = self._downloader.params.get('geo_bypass_country', None)
+        countries = orderedSet((geo_bypass_country, 'US', 'AU', 'CA', 'AS', 'FM', 'GU', 'MP', 'PR', 'PW', 'MH', 'VI', ''))
+        num_countries, num = len(countries) - 1, 0
 
-        last_e = None
+        media = {}
+        for num, country in enumerate(countries):
+            if num == 1:  # start hard-coded list
+                self.report_warning('%s. Trying with a list of known countries' % (
+                    'Unable to obtain video formats from %s API' % geo_bypass_country if geo_bypass_country
+                    else 'No country code was given using --geo-bypass-country'))
+            elif num == num_countries:  # end of list
+                geo_info = self._download_json(
+                    'https://web-api-us.crackle.com/Service.svc/geo/country',
+                    video_id, fatal=False, note='Downloading geo-location information from crackle API',
+                    errnote='Unable to fetch geo-location information from crackle') or {}
+                country = geo_info.get('CountryCode')
+                if country is None:
+                    continue
+                self.to_screen('%s identified country as %s' % (self.IE_NAME, country))
+                if country in countries:
+                    self.to_screen('Downloading from %s API was already attempted. Skipping...' % country)
+                    continue
 
-        for country in countries:
+            if country is None:
+                continue
             try:
-                # Authorization generation algorithm is reverse engineered from:
-                # https://www.sonycrackle.com/static/js/main.ea93451f.chunk.js
-                media_detail_url = 'https://web-api-us.crackle.com/Service.svc/details/media/%s/%s?disableProtocols=true' % (video_id, country)
-                timestamp = time.strftime('%Y%m%d%H%M', time.gmtime())
-                h = hmac.new(b'IGSLUQCBDFHEOIFM', '|'.join([media_detail_url, timestamp]).encode(), hashlib.sha1).hexdigest().upper()
                 media = self._download_json(
-                    media_detail_url, video_id, 'Downloading media JSON as %s' % country,
-                    'Unable to download media JSON', headers={
-                        'Accept': 'application/json',
-                        'Authorization': '|'.join([h, timestamp, '117', '1']),
-                    })
+                    'https://web-api-us.crackle.com/Service.svc/details/media/%s/%s?disableProtocols=true' % (video_id, country),
+                    video_id, note='Downloading media JSON from %s API' % country,
+                    errnote='Unable to download media JSON')
             except ExtractorError as e:
                 # 401 means geo restriction, trying next country
                 if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
-                    last_e = e
                     continue
                 raise
 
-            media_urls = media.get('MediaURLs')
-            if not media_urls or not isinstance(media_urls, list):
+            status = media.get('status')
+            if status.get('messageCode') != '0':
+                raise ExtractorError(
+                    '%s said: %s %s - %s' % (
+                        self.IE_NAME, status.get('messageCodeDescription'), status.get('messageCode'), status.get('message')),
+                    expected=True)
+
+            # Found video formats
+            if isinstance(media.get('MediaURLs'), list):
+                break
+
+        ignore_no_formats = self._downloader.params.get('ignore_no_formats_error')
+        allow_unplayable_formats = self._downloader.params.get('allow_unplayable_formats')
+
+        if not media or (not media.get('MediaURLs') and not ignore_no_formats):
+            raise ExtractorError(
+                'Unable to access the crackle API. Try passing your country code '
+                'to --geo-bypass-country. If it still does not work and the '
+                'video is available in your country')
+        title = media['Title']
+
+        formats, subtitles = [], {}
+        has_drm = False
+        for e in media.get('MediaURLs') or []:
+            if e.get('UseDRM'):
+                has_drm = True
+                if not allow_unplayable_formats:
+                    continue
+            format_url = url_or_none(e.get('Path'))
+            if not format_url:
                 continue
-
-            title = media['Title']
-
-            formats = []
-            for e in media['MediaURLs']:
-                if not self._downloader.params.get('allow_unplayable_formats') and e.get('UseDRM') is True:
-                    continue
-                format_url = url_or_none(e.get('Path'))
-                if not format_url:
-                    continue
-                ext = determine_ext(format_url)
-                if ext == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        format_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                        m3u8_id='hls', fatal=False))
-                elif ext == 'mpd':
-                    formats.extend(self._extract_mpd_formats(
-                        format_url, video_id, mpd_id='dash', fatal=False))
-                elif format_url.endswith('.ism/Manifest'):
-                    formats.extend(self._extract_ism_formats(
-                        format_url, video_id, ism_id='mss', fatal=False))
-                else:
-                    mfs_path = e.get('Type')
-                    mfs_info = self._MEDIA_FILE_SLOTS.get(mfs_path)
-                    if not mfs_info:
-                        continue
-                    formats.append({
-                        'url': format_url,
-                        'format_id': 'http-' + mfs_path.split('.')[0],
-                        'width': mfs_info['width'],
-                        'height': mfs_info['height'],
-                    })
-            self._sort_formats(formats)
-
-            description = media.get('Description')
-            duration = int_or_none(media.get(
-                'DurationInSeconds')) or parse_duration(media.get('Duration'))
-            view_count = int_or_none(media.get('CountViews'))
-            average_rating = float_or_none(media.get('UserRating'))
-            age_limit = parse_age_limit(media.get('Rating'))
-            genre = media.get('Genre')
-            release_year = int_or_none(media.get('ReleaseYear'))
-            creator = media.get('Directors')
-            artist = media.get('Cast')
-
-            if media.get('MediaTypeDisplayValue') == 'Full Episode':
-                series = media.get('ShowName')
-                episode = title
-                season_number = int_or_none(media.get('Season'))
-                episode_number = int_or_none(media.get('Episode'))
+            ext = determine_ext(format_url)
+            if ext == 'm3u8':
+                fmts, subs = self._extract_m3u8_formats_and_subtitles(
+                    format_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                    m3u8_id='hls', fatal=False)
+                formats.extend(fmts)
+                subtitles = self._merge_subtitles(subtitles, subs)
+            elif ext == 'mpd':
+                fmts, subs = self._extract_mpd_formats_and_subtitles(
+                    format_url, video_id, mpd_id='dash', fatal=False)
+                formats.extend(fmts)
+                subtitles = self._merge_subtitles(subtitles, subs)
+            elif format_url.endswith('.ism/Manifest'):
+                fmts, subs = self._extract_ism_formats_and_subtitles(
+                    format_url, video_id, ism_id='mss', fatal=False)
+                formats.extend(fmts)
+                subtitles = self._merge_subtitles(subtitles, subs)
             else:
-                series = episode = season_number = episode_number = None
+                mfs_path = e.get('Type')
+                mfs_info = self._MEDIA_FILE_SLOTS.get(mfs_path)
+                if not mfs_info:
+                    continue
+                formats.append({
+                    'url': format_url,
+                    'format_id': 'http-' + mfs_path.split('.')[0],
+                    'width': mfs_info['width'],
+                    'height': mfs_info['height'],
+                })
+        if not formats and has_drm and not ignore_no_formats:
+            raise ExtractorError('The video is DRM protected', expected=True)
+        self._sort_formats(formats)
 
-            subtitles = {}
-            cc_files = media.get('ClosedCaptionFiles')
-            if isinstance(cc_files, list):
-                for cc_file in cc_files:
-                    if not isinstance(cc_file, dict):
-                        continue
-                    cc_url = url_or_none(cc_file.get('Path'))
-                    if not cc_url:
-                        continue
-                    lang = cc_file.get('Locale') or 'en'
-                    subtitles.setdefault(lang, []).append({'url': cc_url})
+        description = media.get('Description')
+        duration = int_or_none(media.get(
+            'DurationInSeconds')) or parse_duration(media.get('Duration'))
+        view_count = int_or_none(media.get('CountViews'))
+        average_rating = float_or_none(media.get('UserRating'))
+        age_limit = parse_age_limit(media.get('Rating'))
+        genre = media.get('Genre')
+        release_year = int_or_none(media.get('ReleaseYear'))
+        creator = media.get('Directors')
+        artist = media.get('Cast')
 
-            thumbnails = []
-            images = media.get('Images')
-            if isinstance(images, list):
-                for image_key, image_url in images.items():
-                    mobj = re.search(r'Img_(\d+)[xX](\d+)', image_key)
-                    if not mobj:
-                        continue
-                    thumbnails.append({
-                        'url': image_url,
-                        'width': int(mobj.group(1)),
-                        'height': int(mobj.group(2)),
-                    })
+        if media.get('MediaTypeDisplayValue') == 'Full Episode':
+            series = media.get('ShowName')
+            episode = title
+            season_number = int_or_none(media.get('Season'))
+            episode_number = int_or_none(media.get('Episode'))
+        else:
+            series = episode = season_number = episode_number = None
 
-            return {
-                'id': video_id,
-                'title': title,
-                'description': description,
-                'duration': duration,
-                'view_count': view_count,
-                'average_rating': average_rating,
-                'age_limit': age_limit,
-                'genre': genre,
-                'creator': creator,
-                'artist': artist,
-                'release_year': release_year,
-                'series': series,
-                'episode': episode,
-                'season_number': season_number,
-                'episode_number': episode_number,
-                'thumbnails': thumbnails,
-                'subtitles': subtitles,
-                'formats': formats,
-            }
+        cc_files = media.get('ClosedCaptionFiles')
+        if isinstance(cc_files, list):
+            for cc_file in cc_files:
+                if not isinstance(cc_file, dict):
+                    continue
+                cc_url = url_or_none(cc_file.get('Path'))
+                if not cc_url:
+                    continue
+                lang = cc_file.get('Locale') or 'en'
+                subtitles.setdefault(lang, []).append({'url': cc_url})
 
-        raise last_e
+        thumbnails = []
+        images = media.get('Images')
+        if isinstance(images, list):
+            for image_key, image_url in images.items():
+                mobj = re.search(r'Img_(\d+)[xX](\d+)', image_key)
+                if not mobj:
+                    continue
+                thumbnails.append({
+                    'url': image_url,
+                    'width': int(mobj.group(1)),
+                    'height': int(mobj.group(2)),
+                })
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'duration': duration,
+            'view_count': view_count,
+            'average_rating': average_rating,
+            'age_limit': age_limit,
+            'genre': genre,
+            'creator': creator,
+            'artist': artist,
+            'release_year': release_year,
+            'series': series,
+            'episode': episode,
+            'season_number': season_number,
+            'episode_number': episode_number,
+            'thumbnails': thumbnails,
+            'subtitles': subtitles,
+            'formats': formats,
+        }

yt_dlp/extractor/crunchyroll.py

@@ -428,7 +428,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
             r'<div class="showmedia-trailer-notice">(.+?)</div>',
             webpage, 'trailer-notice', default='')
         if note_m:
-            raise ExtractorError(note_m)
+            raise ExtractorError(note_m, expected=True)
 
         mobj = re.search(r'Page\.messaging_box_controller\.addItems\(\[(?P<msg>{.+?})\]\)', webpage)
         if mobj:

yt_dlp/extractor/curiositystream.py

@@ -25,12 +25,12 @@ class CuriosityStreamBaseIE(InfoExtractor):
             raise ExtractorError(
                 '%s said: %s' % (self.IE_NAME, error), expected=True)
 
-    def _call_api(self, path, video_id):
+    def _call_api(self, path, video_id, query=None):
         headers = {}
         if self._auth_token:
             headers['X-Auth-Token'] = self._auth_token
         result = self._download_json(
-            self._API_BASE_URL + path, video_id, headers=headers)
+            self._API_BASE_URL + path, video_id, headers=headers, query=query)
         self._handle_errors(result)
         return result['data']
 
@@ -52,62 +52,75 @@ class CuriosityStreamIE(CuriosityStreamBaseIE):
     _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/video/(?P<id>\d+)'
     _TEST = {
         'url': 'https://app.curiositystream.com/video/2',
-        'md5': '262bb2f257ff301115f1973540de8983',
         'info_dict': {
             'id': '2',
             'ext': 'mp4',
             'title': 'How Did You Develop The Internet?',
             'description': 'Vint Cerf, Google\'s Chief Internet Evangelist, describes how he and Bob Kahn created the internet.',
-        }
+        },
+        'params': {
+            'format': 'bestvideo',
+            # m3u8 download
+            'skip_download': True,
+        },
     }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        media = self._call_api('media/' + video_id, video_id)
-        title = media['title']
 
         formats = []
-        for encoding in media.get('encodings', []):
-            m3u8_url = encoding.get('master_playlist_url')
-            if m3u8_url:
-                formats.extend(self._extract_m3u8_formats(
-                    m3u8_url, video_id, 'mp4', 'm3u8_native',
-                    m3u8_id='hls', fatal=False))
-            encoding_url = encoding.get('url')
-            file_url = encoding.get('file_url')
-            if not encoding_url and not file_url:
-                continue
-            f = {
-                'width': int_or_none(encoding.get('width')),
-                'height': int_or_none(encoding.get('height')),
-                'vbr': int_or_none(encoding.get('video_bitrate')),
-                'abr': int_or_none(encoding.get('audio_bitrate')),
-                'filesize': int_or_none(encoding.get('size_in_bytes')),
-                'vcodec': encoding.get('video_codec'),
-                'acodec': encoding.get('audio_codec'),
-                'container': encoding.get('container_type'),
-            }
-            for f_url in (encoding_url, file_url):
-                if not f_url:
+        for encoding_format in ('m3u8', 'mpd'):
+            media = self._call_api('media/' + video_id, video_id, query={
+                'encodingsNew': 'true',
+                'encodingsFormat': encoding_format,
+            })
+            for encoding in media.get('encodings', []):
+                playlist_url = encoding.get('master_playlist_url')
+                if encoding_format == 'm3u8':
+                    # use `m3u8` entry_protocol until EXT-X-MAP is properly supported by `m3u8_native` entry_protocol
+                    formats.extend(self._extract_m3u8_formats(
+                        playlist_url, video_id, 'mp4',
+                        m3u8_id='hls', fatal=False))
+                elif encoding_format == 'mpd':
+                    formats.extend(self._extract_mpd_formats(
+                        playlist_url, video_id, mpd_id='dash', fatal=False))
+                encoding_url = encoding.get('url')
+                file_url = encoding.get('file_url')
+                if not encoding_url and not file_url:
                     continue
-                fmt = f.copy()
-                rtmp = re.search(r'^(?P<url>rtmpe?://(?P<host>[^/]+)/(?P<app>.+))/(?P<playpath>mp[34]:.+)$', f_url)
-                if rtmp:
-                    fmt.update({
-                        'url': rtmp.group('url'),
-                        'play_path': rtmp.group('playpath'),
-                        'app': rtmp.group('app'),
-                        'ext': 'flv',
-                        'format_id': 'rtmp',
-                    })
-                else:
-                    fmt.update({
-                        'url': f_url,
-                        'format_id': 'http',
-                    })
-                formats.append(fmt)
+                f = {
+                    'width': int_or_none(encoding.get('width')),
+                    'height': int_or_none(encoding.get('height')),
+                    'vbr': int_or_none(encoding.get('video_bitrate')),
+                    'abr': int_or_none(encoding.get('audio_bitrate')),
+                    'filesize': int_or_none(encoding.get('size_in_bytes')),
+                    'vcodec': encoding.get('video_codec'),
+                    'acodec': encoding.get('audio_codec'),
+                    'container': encoding.get('container_type'),
+                }
+                for f_url in (encoding_url, file_url):
+                    if not f_url:
+                        continue
+                    fmt = f.copy()
+                    rtmp = re.search(r'^(?P<url>rtmpe?://(?P<host>[^/]+)/(?P<app>.+))/(?P<playpath>mp[34]:.+)$', f_url)
+                    if rtmp:
+                        fmt.update({
+                            'url': rtmp.group('url'),
+                            'play_path': rtmp.group('playpath'),
+                            'app': rtmp.group('app'),
+                            'ext': 'flv',
+                            'format_id': 'rtmp',
+                        })
+                    else:
+                        fmt.update({
+                            'url': f_url,
+                            'format_id': 'http',
+                        })
+                    formats.append(fmt)
         self._sort_formats(formats)
 
+        title = media['title']
+
         subtitles = {}
         for closed_caption in media.get('closed_captions', []):
             sub_url = closed_caption.get('file')
@@ -130,32 +143,45 @@ class CuriosityStreamIE(CuriosityStreamBaseIE):
         }
 
 
-class CuriosityStreamCollectionIE(CuriosityStreamBaseIE):
-    IE_NAME = 'curiositystream:collection'
-    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/(?:collection|series)/(?P<id>\d+)'
+class CuriosityStreamCollectionsIE(CuriosityStreamBaseIE):
+    IE_NAME = 'curiositystream:collections'
+    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/collections/(?P<id>\d+)'
+    _API_BASE_URL = 'https://api.curiositystream.com/v2/collections/'
     _TESTS = [{
-        'url': 'https://app.curiositystream.com/collection/2',
+        'url': 'https://curiositystream.com/collections/86',
+        'info_dict': {
+            'id': '86',
+            'title': 'Staff Picks',
+            'description': 'Wondering where to start? Here are a few of our favorite series and films... from our couch to yours.',
+        },
+        'playlist_mincount': 7,
+    }]
+
+    def _real_extract(self, url):
+        collection_id = self._match_id(url)
+        collection = self._call_api(collection_id, collection_id)
+        entries = []
+        for media in collection.get('media', []):
+            media_id = compat_str(media.get('id'))
+            media_type, ie = ('series', CuriosityStreamSeriesIE) if media.get('is_collection') else ('video', CuriosityStreamIE)
+            entries.append(self.url_result(
+                'https://curiositystream.com/%s/%s' % (media_type, media_id),
+                ie=ie.ie_key(), video_id=media_id))
+        return self.playlist_result(
+            entries, collection_id,
+            collection.get('title'), collection.get('description'))
+
+
+class CuriosityStreamSeriesIE(CuriosityStreamCollectionsIE):
+    IE_NAME = 'curiositystream:series'
+    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/series/(?P<id>\d+)'
+    _API_BASE_URL = 'https://api.curiositystream.com/v2/series/'
+    _TESTS = [{
+        'url': 'https://app.curiositystream.com/series/2',
         'info_dict': {
             'id': '2',
             'title': 'Curious Minds: The Internet',
             'description': 'How is the internet shaping our lives in the 21st Century?',
         },
-        'playlist_mincount': 17,
-    }, {
-        'url': 'https://curiositystream.com/series/2',
-        'only_matching': True,
+        'playlist_mincount': 16,
     }]
-
-    def _real_extract(self, url):
-        collection_id = self._match_id(url)
-        collection = self._call_api(
-            'collections/' + collection_id, collection_id)
-        entries = []
-        for media in collection.get('media', []):
-            media_id = compat_str(media.get('id'))
-            entries.append(self.url_result(
-                'https://curiositystream.com/video/' + media_id,
-                CuriosityStreamIE.ie_key(), media_id))
-        return self.playlist_result(
-            entries, collection_id,
-            collection.get('title'), collection.get('description'))

yt_dlp/extractor/deezer.py

@@ -14,7 +14,7 @@ from ..utils import (
 class DeezerBaseInfoExtractor(InfoExtractor):
     def get_data(self, url):
         if not self._downloader.params.get('test'):
-            self._downloader.report_warning('For now, this extractor only supports the 30 second previews. Patches welcome!')
+            self.report_warning('For now, this extractor only supports the 30 second previews. Patches welcome!')
 
         mobj = re.match(self._VALID_URL, url)
         data_id = mobj.group('id')

yt_dlp/extractor/discoveryplusindia.py

@@ -0,0 +1,100 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+import re
+
+from ..compat import compat_str
+from ..utils import try_get
+from .common import InfoExtractor
+from .dplay import DPlayIE
+
+
+class DiscoveryPlusIndiaIE(DPlayIE):
+    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.in/videos?' + DPlayIE._PATH_REGEX
+    _TESTS = [{
+        'url': 'https://www.discoveryplus.in/videos/how-do-they-do-it/fugu-and-more?seasonId=8&type=EPISODE',
+        'info_dict': {
+            'id': '27104',
+            'ext': 'mp4',
+            'display_id': 'how-do-they-do-it/fugu-and-more',
+            'title': 'Fugu and More',
+            'description': 'The Japanese catch, prepare and eat the deadliest fish on the planet.',
+            'duration': 1319,
+            'timestamp': 1582309800,
+            'upload_date': '20200221',
+            'series': 'How Do They Do It?',
+            'season_number': 8,
+            'episode_number': 2,
+            'creator': 'Discovery Channel',
+        },
+        'params': {
+            'format': 'bestvideo',
+            'skip_download': True,
+        },
+        'skip': 'Cookies (not necessarily logged in) are needed'
+    }]
+
+    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
+        headers['x-disco-params'] = 'realm=%s' % realm
+        headers['x-disco-client'] = 'WEB:UNKNOWN:dplus-india:17.0.0'
+
+    def _download_video_playback_info(self, disco_base, video_id, headers):
+        return self._download_json(
+            disco_base + 'playback/v3/videoPlaybackInfo',
+            video_id, headers=headers, data=json.dumps({
+                'deviceInfo': {
+                    'adBlocker': False,
+                },
+                'videoId': video_id,
+            }).encode('utf-8'))['data']['attributes']['streaming']
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        return self._get_disco_api_info(
+            url, display_id, 'ap2-prod-direct.discoveryplus.in', 'dplusindia', 'in')
+
+
+class DiscoveryPlusIndiaShowIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.in/show/(?P<show_name>[^/]+)/?(?:[?#]|$)'
+    _TESTS = [{
+        'url': 'https://www.discoveryplus.in/show/how-do-they-do-it',
+        'playlist_mincount': 140,
+        'info_dict': {
+            'id': 'how-do-they-do-it',
+        },
+    }
+    ]
+
+    def _entries(self, show_name):
+        headers = {
+            'x-disco-client': 'WEB:UNKNOWN:dplus-india:prod',
+            'x-disco-params': 'realm=dplusindia',
+            'referer': 'https://www.discoveryplus.in/',
+        }
+        show_url = 'https://ap2-prod-direct.discoveryplus.in/cms/routes/show/{}?include=default'.format(show_name)
+        show_json = self._download_json(show_url,
+                                        video_id=show_name,
+                                        headers=headers)['included'][4]['attributes']['component']
+        show_id = show_json['mandatoryParams'].split('=')[-1]
+        season_url = 'https://ap2-prod-direct.discoveryplus.in/content/videos?sort=episodeNumber&filter[seasonNumber]={}&filter[show.id]={}&page[size]=100&page[number]={}'
+        for season in show_json['filters'][0]['options']:
+            season_id = season['id']
+            total_pages, page_num = 1, 0
+            while page_num < total_pages:
+                season_json = self._download_json(season_url.format(season_id, show_id, compat_str(page_num + 1)),
+                                                  video_id=show_id, headers=headers,
+                                                  note='Downloading JSON metadata%s' % (' page %d' % page_num if page_num else ''))
+                if page_num == 0:
+                    total_pages = try_get(season_json, lambda x: x['meta']['totalPages'], int) or 1
+                episodes_json = season_json['data']
+                for episode in episodes_json:
+                    video_id = episode['attributes']['path']
+                    yield self.url_result(
+                        'https://discoveryplus.in/videos/%s' % video_id,
+                        ie=DiscoveryPlusIndiaIE.ie_key(), video_id=video_id)
+                page_num += 1
+
+    def _real_extract(self, url):
+        show_name = re.match(self._VALID_URL, url).group('show_name')
+        return self.playlist_result(self._entries(show_name), playlist_id=show_name)

yt_dlp/extractor/disney.py

@@ -9,7 +9,6 @@ from ..utils import (
     unified_strdate,
     compat_str,
     determine_ext,
-    ExtractorError,
     update_url_query,
 )
 
@@ -140,7 +139,7 @@ class DisneyIE(InfoExtractor):
                 'vcodec': 'none' if (width == 0 and height == 0) else None,
             })
         if not formats and video_data.get('expired'):
-            raise ExtractorError(
+            self.raise_no_formats(
                 '%s said: %s' % (self.IE_NAME, page_data['translations']['video_expired']),
                 expected=True)
         self._sort_formats(formats)

yt_dlp/extractor/dispeak.py

@@ -32,6 +32,18 @@ class DigitallySpeakingIE(InfoExtractor):
         # From http://www.gdcvault.com/play/1013700/Advanced-Material
         'url': 'http://sevt.dispeak.com/ubm/gdc/eur10/xml/11256_1282118587281VNIT.xml',
         'only_matching': True,
+    }, {
+        # From https://gdcvault.com/play/1016624, empty speakerVideo
+        'url': 'https://sevt.dispeak.com/ubm/gdc/online12/xml/201210-822101_1349794556671DDDD.xml',
+        'info_dict': {
+            'id': '201210-822101_1349794556671DDDD',
+            'ext': 'flv',
+            'title': 'Pre-launch - Preparing to Take the Plunge',
+        },
+    }, {
+        # From http://www.gdcvault.com/play/1014846/Conference-Keynote-Shigeru, empty slideVideo
+        'url': 'http://events.digitallyspeaking.com/gdc/project25/xml/p25-miyamoto1999_1282467389849HSVB.xml',
+        'only_matching': True,
     }]
 
     def _parse_mp4(self, metadata):
@@ -85,25 +97,19 @@ class DigitallySpeakingIE(InfoExtractor):
                 'quality': 1,
                 'format_id': audio.get('code'),
             })
-        slide_video_path = xpath_text(metadata, './slideVideo', fatal=True)
-        formats.append({
-            'url': 'rtmp://%s/ondemand?ovpfv=1.1' % akamai_url,
-            'play_path': remove_end(slide_video_path, '.flv'),
-            'ext': 'flv',
-            'format_note': 'slide deck video',
-            'quality': -2,
-            'format_id': 'slides',
-            'acodec': 'none',
-        })
-        speaker_video_path = xpath_text(metadata, './speakerVideo', fatal=True)
-        formats.append({
-            'url': 'rtmp://%s/ondemand?ovpfv=1.1' % akamai_url,
-            'play_path': remove_end(speaker_video_path, '.flv'),
-            'ext': 'flv',
-            'format_note': 'speaker video',
-            'quality': -1,
-            'format_id': 'speaker',
-        })
+        for video_key, format_id, preference in (
+                ('slide', 'slides', -2), ('speaker', 'speaker', -1)):
+            video_path = xpath_text(metadata, './%sVideo' % video_key)
+            if not video_path:
+                continue
+            formats.append({
+                'url': 'rtmp://%s/ondemand?ovpfv=1.1' % akamai_url,
+                'play_path': remove_end(video_path, '.flv'),
+                'ext': 'flv',
+                'format_note': '%s video' % video_key,
+                'quality': preference,
+                'format_id': format_id,
+            })
         return formats
 
     def _real_extract(self, url):

yt_dlp/extractor/dplay.py

@@ -296,51 +296,6 @@ class DPlayIE(InfoExtractor):
             url, display_id, host, 'dplay' + country, country)
 
 
-class DiscoveryPlusIndiaIE(DPlayIE):
-    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.in/videos?' + DPlayIE._PATH_REGEX
-    _TESTS = [{
-        'url': 'https://www.discoveryplus.in/videos/how-do-they-do-it/fugu-and-more?seasonId=8&type=EPISODE',
-        'info_dict': {
-            'id': '27104',
-            'ext': 'mp4',
-            'display_id': 'how-do-they-do-it/fugu-and-more',
-            'title': 'Fugu and More',
-            'description': 'The Japanese catch, prepare and eat the deadliest fish on the planet.',
-            'duration': 1319,
-            'timestamp': 1582309800,
-            'upload_date': '20200221',
-            'series': 'How Do They Do It?',
-            'season_number': 8,
-            'episode_number': 2,
-            'creator': 'Discovery Channel',
-        },
-        'params': {
-            'format': 'bestvideo',
-            'skip_download': True,
-        },
-        'skip': 'Cookies (not necessarily logged in) are needed'
-    }]
-
-    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
-        headers['x-disco-params'] = 'realm=%s' % realm
-        headers['x-disco-client'] = 'WEB:UNKNOWN:dplus-india:17.0.0'
-
-    def _download_video_playback_info(self, disco_base, video_id, headers):
-        return self._download_json(
-            disco_base + 'playback/v3/videoPlaybackInfo',
-            video_id, headers=headers, data=json.dumps({
-                'deviceInfo': {
-                    'adBlocker': False,
-                },
-                'videoId': video_id,
-            }).encode('utf-8'))['data']['attributes']['streaming']
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        return self._get_disco_api_info(
-            url, display_id, 'ap2-prod-direct.discoveryplus.in', 'dplusindia', 'in')
-
-
 class DiscoveryPlusIE(DPlayIE):
     _VALID_URL = r'https?://(?:www\.)?discoveryplus\.com/video' + DPlayIE._PATH_REGEX
     _TESTS = [{

yt_dlp/extractor/elonet.py

@@ -1,9 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import os
 import re
-import tempfile
 
 from .common import InfoExtractor
 from ..utils import (
@@ -12,12 +10,12 @@ from ..utils import (
     try_get,
 )
 from ..compat import compat_str
-from ..downloader.hls import HlsFD
 
 
 class ElonetIE(InfoExtractor):
     _VALID_URL = r'https?://elonet\.finna\.fi/Record/kavi\.elonet_elokuva_(?P<id>[0-9]+)'
-    _TEST = {
+    _TESTS = [{
+        # m3u8 with subtitles
         'url': 'https://elonet.finna.fi/Record/kavi.elonet_elokuva_107867',
         'md5': '8efc954b96c543711707f87de757caea',
         'info_dict': {
@@ -27,62 +25,17 @@ class ElonetIE(InfoExtractor):
             'description': 'Valkoinen peura (1952) on Erik Blombergin ohjaama ja yhdessä Mirjami Kuosmasen kanssa käsikirjoittama tarunomainen kertomus valkoisen peuran hahmossa lii...',
             'thumbnail': 'https://elonet.finna.fi/Cover/Show?id=kavi.elonet_elokuva_107867&index=0&size=large',
         },
-    }
-
-    def _download_m3u8_chunked_subtitle(self, chunklist_url):
-        """
-        Download VTT subtitles from pieces in manifest URL.
-        Return a string containing joined chunks with extra headers removed.
-        """
-        with tempfile.NamedTemporaryFile(delete=True) as outfile:
-            fname = outfile.name
-        hlsdl = HlsFD(self._downloader, {})
-        hlsdl.download(compat_str(fname), {"url": chunklist_url})
-        with open(fname, 'r') as fin:
-            # Remove (some) headers
-            fdata = re.sub(r'X-TIMESTAMP-MAP.*\n+|WEBVTT\n+', '', fin.read())
-        os.remove(fname)
-        return "WEBVTT\n\n" + fdata
-
-    def _parse_m3u8_subtitles(self, m3u8_doc, m3u8_url):
-        """
-        Parse subtitles from HLS / m3u8 manifest.
-        """
-        subtitles = {}
-        baseurl = m3u8_url[:m3u8_url.rindex('/') + 1]
-        for line in m3u8_doc.split('\n'):
-            if 'EXT-X-MEDIA:TYPE=SUBTITLES' in line:
-                lang = self._search_regex(
-                    r'LANGUAGE="(.+?)"', line, 'lang', default=False)
-                uri = self._search_regex(
-                    r'URI="(.+?)"', line, 'uri', default=False)
-                if lang and uri:
-                    data = self._download_m3u8_chunked_subtitle(baseurl + uri)
-                    subtitles[lang] = [{'ext': 'vtt', 'data': data}]
-        return subtitles
-
-    def _parse_mpd_subtitles(self, mpd_doc):
-        """
-        Parse subtitles from MPD manifest.
-        """
-        ns = '{urn:mpeg:dash:schema:mpd:2011}'
-        subtitles = {}
-        for aset in mpd_doc.findall(".//%sAdaptationSet[@mimeType='text/vtt']" % (ns)):
-            lang = aset.attrib.get('lang', 'unk')
-            url = aset.find("./%sRepresentation/%sBaseURL" % (ns, ns)).text
-            subtitles[lang] = [{'ext': 'vtt', 'url': url}]
-        return subtitles
-
-    def _get_subtitles(self, fmt, doc, url):
-        if fmt == 'm3u8':
-            subs = self._parse_m3u8_subtitles(doc, url)
-        elif fmt == 'mpd':
-            subs = self._parse_mpd_subtitles(doc)
-        else:
-            self._downloader.report_warning(
-                "Cannot download subtitles from '%s' streams." % (fmt))
-            subs = {}
-        return subs
+    }, {
+        # DASH with subtitles
+        'url': 'https://elonet.finna.fi/Record/kavi.elonet_elokuva_116539',
+        'info_dict': {
+            'id': '116539',
+            'ext': 'mp4',
+            'title': 'Minulla on tiikeri',
+            'description': 'Pienellä pojalla, joka asuu kerrostalossa, on kotieläimenä tiikeri. Se on kuitenkin salaisuus. Kerrostalon räpätäti on Kotilaisen täti, joka on aina vali...',
+            'thumbnail': 'https://elonet.finna.fi/Cover/Show?id=kavi.elonet_elokuva_116539&index=0&size=large&source=Solr',
+        }
+    }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
@@ -101,8 +54,8 @@ class ElonetIE(InfoExtractor):
             self._parse_json(json_s, video_id),
             lambda x: x[0]["src"], compat_str)
         formats = []
+        subtitles = {}
         if re.search(r'\.m3u8\??', src):
-            fmt = 'm3u8'
             res = self._download_webpage_handle(
                 # elonet servers have certificate problems
                 src.replace('https:', 'http:'), video_id,
@@ -111,11 +64,10 @@ class ElonetIE(InfoExtractor):
             if res:
                 doc, urlh = res
                 url = urlh.geturl()
-                formats = self._parse_m3u8_formats(doc, url)
+                formats, subtitles = self._parse_m3u8_formats_and_subtitles(doc, url)
                 for f in formats:
                     f['ext'] = 'mp4'
         elif re.search(r'\.mpd\??', src):
-            fmt = 'mpd'
             res = self._download_xml_handle(
                 src, video_id,
                 note='Downloading MPD manifest',
@@ -123,7 +75,7 @@ class ElonetIE(InfoExtractor):
             if res:
                 doc, urlh = res
                 url = base_url(urlh.geturl())
-                formats = self._parse_mpd_formats(doc, mpd_base_url=url)
+                formats, subtitles = self._parse_mpd_formats_and_subtitles(doc, mpd_base_url=url)
         else:
             raise ExtractorError("Unknown streaming format")
 
@@ -133,5 +85,5 @@ class ElonetIE(InfoExtractor):
             'description': description,
             'thumbnail': thumbnail,
             'formats': formats,
-            'subtitles': self.extract_subtitles(fmt, doc, url),
+            'subtitles': subtitles,
         }

yt_dlp/extractor/extractors.py

@@ -151,7 +151,6 @@ from .bleacherreport import (
     BleacherReportIE,
     BleacherReportCMSIE,
 )
-from .blinkx import BlinkxIE
 from .bloomberg import BloombergIE
 from .bokecc import BokeCCIE
 from .bongacams import BongaCamsIE
@@ -210,7 +209,11 @@ from .cbsnews import (
     CBSNewsIE,
     CBSNewsLiveVideoIE,
 )
-from .cbssports import CBSSportsIE
+from .cbssports import (
+    CBSSportsEmbedIE,
+    CBSSportsIE,
+    TwentyFourSevenSportsIE,
+)
 from .ccc import (
     CCCIE,
     CCCPlaylistIE,
@@ -284,7 +287,8 @@ from .ctvnews import CTVNewsIE
 from .cultureunplugged import CultureUnpluggedIE
 from .curiositystream import (
     CuriosityStreamIE,
-    CuriosityStreamCollectionIE,
+    CuriosityStreamCollectionsIE,
+    CuriosityStreamSeriesIE,
 )
 from .cwtv import CWTVIE
 from .dailymail import DailyMailIE
@@ -309,6 +313,10 @@ from .democracynow import DemocracynowIE
 from .dfb import DFBIE
 from .dhm import DHMIE
 from .digg import DiggIE
+from .discoveryplusindia import (
+    DiscoveryPlusIndiaIE,
+    DiscoveryPlusIndiaShowIE,
+)
 from .dotsub import DotsubIE
 from .douyutv import (
     DouyuShowIE,
@@ -317,7 +325,6 @@ from .douyutv import (
 from .dplay import (
     DPlayIE,
     DiscoveryPlusIE,
-    DiscoveryPlusIndiaIE,
     HGTVDeIE,
 )
 from .dreisat import DreiSatIE
@@ -630,7 +637,11 @@ from .limelight import (
     LimelightChannelIE,
     LimelightChannelListIE,
 )
-from .line import LineTVIE
+from .line import (
+    LineTVIE,
+    LineLiveIE,
+    LineLiveChannelIE,
+)
 from .linkedin import (
     LinkedInLearningIE,
     LinkedInLearningCourseIE,
@@ -668,6 +679,7 @@ from .mangomolo import (
     MangomoloLiveIE,
 )
 from .manyvids import ManyVidsIE
+from .maoritv import MaoriTVIE
 from .markiza import (
     MarkizaIE,
     MarkizaPageIE,
@@ -748,7 +760,10 @@ from .mtv import (
 )
 from .muenchentv import MuenchenTVIE
 from .mwave import MwaveIE, MwaveMeetGreetIE
-from .mxplayer import MxplayerIE
+from .mxplayer import (
+    MxplayerIE,
+    MxplayerShowIE,
+)
 from .mychannels import MyChannelsIE
 from .myspace import MySpaceIE, MySpaceAlbumIE
 from .myspass import MySpassIE
@@ -791,8 +806,9 @@ from .ndr import (
     NJoyEmbedIE,
 )
 from .ndtv import NDTVIE
-from .netzkino import NetzkinoIE
+from .nebula import NebulaIE
 from .nerdcubed import NerdCubedFeedIE
+from .netzkino import NetzkinoIE
 from .neteasemusic import (
     NetEaseMusicIE,
     NetEaseMusicAlbumIE,
@@ -817,6 +833,7 @@ from .nexx import (
     NexxIE,
     NexxEmbedIE,
 )
+from .nfhsnetwork import NFHSNetworkIE
 from .nfl import (
     NFLIE,
     NFLArticleIE,
@@ -1326,7 +1343,10 @@ from .trovo import (
 from .trunews import TruNewsIE
 from .trutv import TruTVIE
 from .tube8 import Tube8IE
-from .tubitv import TubiTvIE
+from .tubitv import (
+    TubiTvIE,
+    TubiTvShowIE,
+)
 from .tumblr import TumblrIE
 from .tunein import (
     TuneInClipIE,
@@ -1421,6 +1441,7 @@ from .ufctv import (
     UFCTVIE,
     UFCArabiaIE,
 )
+from .ukcolumn import UkColumnIE
 from .uktvplay import UKTVPlayIE
 from .digiteka import DigitekaIE
 from .dlive import (
@@ -1580,6 +1601,7 @@ from .weibo import (
 )
 from .weiqitv import WeiqiTVIE
 from .wimtv import WimTVIE
+from .whowatch import WhoWatchIE
 from .wistia import (
     WistiaIE,
     WistiaPlaylistIE,

yt_dlp/extractor/facebook.py

@@ -3,14 +3,11 @@ from __future__ import unicode_literals
 
 import json
 import re
-import socket
 
 from .common import InfoExtractor
 from ..compat import (
     compat_etree_fromstring,
-    compat_http_client,
     compat_str,
-    compat_urllib_error,
     compat_urllib_parse_unquote,
     compat_urllib_parse_unquote_plus,
 )
@@ -23,6 +20,7 @@ from ..utils import (
     int_or_none,
     js_to_json,
     limit_length,
+    network_exceptions,
     parse_count,
     qualities,
     sanitized_Request,
@@ -348,7 +346,7 @@ class FacebookIE(InfoExtractor):
                     login_results, 'login error', default=None, group='error')
                 if error:
                     raise ExtractorError('Unable to login: %s' % error, expected=True)
-                self._downloader.report_warning('unable to log in: bad username/password, or exceeded login rate limit (~3/min). Check credentials or wait.')
+                self.report_warning('unable to log in: bad username/password, or exceeded login rate limit (~3/min). Check credentials or wait.')
                 return
 
             fb_dtsg = self._search_regex(
@@ -369,9 +367,9 @@ class FacebookIE(InfoExtractor):
             check_response = self._download_webpage(check_req, None,
                                                     note='Confirming login')
             if re.search(r'id="checkpointSubmitButton"', check_response) is not None:
-                self._downloader.report_warning('Unable to confirm login, you have to login in your browser and authorize the login.')
-        except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
-            self._downloader.report_warning('unable to log in: %s' % error_to_compat_str(err))
+                self.report_warning('Unable to confirm login, you have to login in your browser and authorize the login.')
+        except network_exceptions as err:
+            self.report_warning('unable to log in: %s' % error_to_compat_str(err))
             return
 
     def _real_initialize(self):
@@ -625,8 +623,6 @@ class FacebookIE(InfoExtractor):
             subtitles_src = f[0].get('subtitles_src')
             if subtitles_src:
                 subtitles.setdefault('en', []).append({'url': subtitles_src})
-        if not formats:
-            raise ExtractorError('Cannot find video formats')
 
         process_formats(formats)
 

yt_dlp/extractor/francetv.py

@@ -151,6 +151,7 @@ class FranceTVIE(InfoExtractor):
                     videos.append(fallback_info['video'])
 
         formats = []
+        subtitles = {}
         for video in videos:
             video_url = video.get('url')
             if not video_url:
@@ -171,10 +172,12 @@ class FranceTVIE(InfoExtractor):
                     sign(video_url, format_id) + '&hdcore=3.7.0&plugin=aasp-3.7.0.39.44',
                     video_id, f4m_id=format_id, fatal=False))
             elif ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
+                m3u8_fmts, m3u8_subs = self._extract_m3u8_formats_and_subtitles(
                     sign(video_url, format_id), video_id, 'mp4',
                     entry_protocol='m3u8_native', m3u8_id=format_id,
-                    fatal=False))
+                    fatal=False)
+                formats.extend(m3u8_fmts)
+                subtitles = self._merge_subtitles(subtitles, m3u8_subs)
             elif ext == 'mpd':
                 formats.extend(self._extract_mpd_formats(
                     sign(video_url, format_id), video_id, mpd_id=format_id, fatal=False))
@@ -199,13 +202,12 @@ class FranceTVIE(InfoExtractor):
             title += ' - %s' % subtitle
         title = title.strip()
 
-        subtitles = {}
-        subtitles_list = [{
-            'url': subformat['url'],
-            'ext': subformat.get('format'),
-        } for subformat in info.get('subtitles', []) if subformat.get('url')]
-        if subtitles_list:
-            subtitles['fr'] = subtitles_list
+        subtitles.setdefault('fr', []).extend(
+            [{
+                'url': subformat['url'],
+                'ext': subformat.get('format'),
+            } for subformat in info.get('subtitles', []) if subformat.get('url')]
+        )
 
         return {
             'id': video_id,
@@ -357,6 +359,22 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
             'skip_download': True,
         },
         'add_ie': [FranceTVIE.ie_key()],
+    }, {
+        'note': 'Only an image exists in initial webpage instead of the video',
+        'url': 'https://www.francetvinfo.fr/sante/maladie/coronavirus/covid-19-en-inde-une-situation-catastrophique-a-new-dehli_4381095.html',
+        'info_dict': {
+            'id': '7d204c9e-a2d3-11eb-9e4c-000d3a23d482',
+            'ext': 'mp4',
+            'title': 'Covid-19 : une situation catastrophique à New Dehli',
+            'thumbnail': str,
+            'duration': 76,
+            'timestamp': 1619028518,
+            'upload_date': '20210421',
+        },
+        'params': {
+            'skip_download': True,
+        },
+        'add_ie': [FranceTVIE.ie_key()],
     }, {
         'url': 'http://www.francetvinfo.fr/elections/europeennes/direct-europeennes-regardez-le-debat-entre-les-candidats-a-la-presidence-de-la-commission_600639.html',
         'only_matching': True,
@@ -384,6 +402,10 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
     }, {
         'url': 'http://france3-regions.francetvinfo.fr/limousin/emissions/jt-1213-limousin',
         'only_matching': True,
+    }, {
+        # "<figure id=" pattern (#28792)
+        'url': 'https://www.francetvinfo.fr/culture/patrimoine/incendie-de-notre-dame-de-paris/notre-dame-de-paris-de-l-incendie-de-la-cathedrale-a-sa-reconstruction_4372291.html',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -401,7 +423,7 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
             (r'player\.load[^;]+src:\s*["\']([^"\']+)',
              r'id-video=([^@]+@[^"]+)',
              r'<a[^>]+href="(?:https?:)?//videos\.francetv\.fr/video/([^@]+@[^"]+)"',
-             r'data-id=["\']([\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'),
+             r'(?:data-id|<figure[^<]+\bid)=["\']([\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'),
             webpage, 'video id')
 
         return self._make_url_result(video_id)

yt_dlp/extractor/funimation.py

@@ -16,7 +16,7 @@ from ..utils import (
 
 
 class FunimationIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?funimation(?:\.com|now\.uk)/shows/[^/]+/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?funimation(?:\.com|now\.uk)/(?:[^/]+/)?shows/[^/]+/(?P<id>[^/?#&]+)'
 
     _NETRC_MACHINE = 'funimation'
     _TOKEN = None
@@ -51,6 +51,10 @@ class FunimationIE(InfoExtractor):
     }, {
         'url': 'https://www.funimationnow.uk/shows/puzzle-dragons-x/drop-impact/simulcast/',
         'only_matching': True,
+    }, {
+        # with lang code
+        'url': 'https://www.funimation.com/en/shows/hacksign/role-play/',
+        'only_matching': True,
     }]
 
     def _login(self):

yt_dlp/extractor/gdcvault.py

@@ -5,7 +5,10 @@ import re
 from .common import InfoExtractor
 from .kaltura import KalturaIE
 from ..utils import (
+    HEADRequest,
+    remove_start,
     sanitized_Request,
+    smuggle_url,
     urlencode_postdata,
 )
 
@@ -100,6 +103,26 @@ class GDCVaultIE(InfoExtractor):
                 'format': 'mp4-408',
             },
         },
+        {
+            # Kaltura embed, whitespace between quote and embedded URL in iframe's src
+            'url': 'https://www.gdcvault.com/play/1025699',
+            'info_dict': {
+                'id': '0_zagynv0a',
+                'ext': 'mp4',
+                'title': 'Tech Toolbox',
+                'upload_date': '20190408',
+                'uploader_id': 'joe@blazestreaming.com',
+                'timestamp': 1554764629,
+            },
+            'params': {
+                'skip_download': True,
+            },
+        },
+        {
+            # HTML5 video
+            'url': 'http://www.gdcvault.com/play/1014846/Conference-Keynote-Shigeru',
+            'only_matching': True,
+        },
     ]
 
     def _login(self, webpage_url, display_id):
@@ -120,38 +143,78 @@ class GDCVaultIE(InfoExtractor):
         request = sanitized_Request(login_url, urlencode_postdata(login_form))
         request.add_header('Content-Type', 'application/x-www-form-urlencoded')
         self._download_webpage(request, display_id, 'Logging in')
-        webpage = self._download_webpage(webpage_url, display_id, 'Getting authenticated video page')
+        start_page = self._download_webpage(webpage_url, display_id, 'Getting authenticated video page')
         self._download_webpage(logout_url, display_id, 'Logging out')
 
-        return webpage
+        return start_page
 
     def _real_extract(self, url):
         video_id, name = re.match(self._VALID_URL, url).groups()
         display_id = name or video_id
 
-        webpage = self._download_webpage(url, display_id)
+        webpage_url = 'http://www.gdcvault.com/play/' + video_id
+        start_page = self._download_webpage(webpage_url, display_id)
 
-        title = self._html_search_regex(
-            r'<td><strong>Session Name:?</strong></td>\s*<td>(.*?)</td>',
-            webpage, 'title')
+        direct_url = self._search_regex(
+            r's1\.addVariable\("file",\s*encodeURIComponent\("(/[^"]+)"\)\);',
+            start_page, 'url', default=None)
+        if direct_url:
+            title = self._html_search_regex(
+                r'<td><strong>Session Name:?</strong></td>\s*<td>(.*?)</td>',
+                start_page, 'title')
+            video_url = 'http://www.gdcvault.com' + direct_url
+            # resolve the url so that we can detect the correct extension
+            video_url = self._request_webpage(
+                HEADRequest(video_url), video_id).geturl()
 
-        PLAYER_REGEX = r'<iframe src=\"(?P<manifest_url>.*?)\".*?</iframe>'
-        manifest_url = self._html_search_regex(
-            PLAYER_REGEX, webpage, 'manifest_url')
+            return {
+                'id': video_id,
+                'display_id': display_id,
+                'url': video_url,
+                'title': title,
+            }
 
-        partner_id = self._search_regex(
-            r'/p(?:artner_id)?/(\d+)', manifest_url, 'partner id',
-            default='1670711')
+        embed_url = KalturaIE._extract_url(start_page)
+        if embed_url:
+            embed_url = smuggle_url(embed_url, {'source_url': url})
+            ie_key = 'Kaltura'
+        else:
+            PLAYER_REGEX = r'<iframe src="(?P<xml_root>.+?)/(?:gdc-)?player.*?\.html.*?".*?</iframe>'
 
-        kaltura_id = self._search_regex(
-            r'entry_id=(?P<id>(?:[^&])+)', manifest_url,
-            'kaltura id', group='id')
+            xml_root = self._html_search_regex(
+                PLAYER_REGEX, start_page, 'xml root', default=None)
+            if xml_root is None:
+                # Probably need to authenticate
+                login_res = self._login(webpage_url, display_id)
+                if login_res is None:
+                    self.report_warning('Could not login.')
+                else:
+                    start_page = login_res
+                    # Grab the url from the authenticated page
+                    xml_root = self._html_search_regex(
+                        PLAYER_REGEX, start_page, 'xml root')
+
+            xml_name = self._html_search_regex(
+                r'<iframe src=".*?\?xml(?:=|URL=xml/)(.+?\.xml).*?".*?</iframe>',
+                start_page, 'xml filename', default=None)
+            if not xml_name:
+                info = self._parse_html5_media_entries(url, start_page, video_id)[0]
+                info.update({
+                    'title': remove_start(self._search_regex(
+                        r'>Session Name:\s*<.*?>\s*<td>(.+?)</td>', start_page,
+                        'title', default=None) or self._og_search_title(
+                        start_page, default=None), 'GDC Vault - '),
+                    'id': video_id,
+                    'display_id': display_id,
+                })
+                return info
+            embed_url = '%s/xml/%s' % (xml_root, xml_name)
+            ie_key = 'DigitallySpeaking'
 
         return {
             '_type': 'url_transparent',
-            'url': 'kaltura:%s:%s' % (partner_id, kaltura_id),
-            'ie_key': KalturaIE.ie_key(),
             'id': video_id,
             'display_id': display_id,
-            'title': title,
+            'url': embed_url,
+            'ie_key': ie_key,
         }

yt_dlp/extractor/generic.py

@@ -2376,7 +2376,7 @@ class GenericIE(InfoExtractor):
 
             if default_search in ('auto', 'auto_warning', 'fixup_error'):
                 if re.match(r'^[^\s/]+\.[^\s/]+/', url):
-                    self._downloader.report_warning('The url doesn\'t specify the protocol, trying with http')
+                    self.report_warning('The url doesn\'t specify the protocol, trying with http')
                     return self.url_result('http://' + url)
                 elif default_search != 'fixup_error':
                     if default_search == 'auto_warning':
@@ -2385,7 +2385,7 @@ class GenericIE(InfoExtractor):
                                 'Invalid URL:  %r . Call yt-dlp like this:  yt-dlp -v "https://www.youtube.com/watch?v=BaW_jenozKc"  ' % url,
                                 expected=True)
                         else:
-                            self._downloader.report_warning(
+                            self.report_warning(
                                 'Falling back to youtube search for  %s . Set --default-search "auto" to suppress this warning.' % url)
                     return self.url_result('ytsearch:' + url)
 
@@ -2444,8 +2444,9 @@ class GenericIE(InfoExtractor):
         m = re.match(r'^(?P<type>audio|video|application(?=/(?:ogg$|(?:vnd\.apple\.|x-)?mpegurl)))/(?P<format_id>[^;\s]+)', content_type)
         if m:
             format_id = compat_str(m.group('format_id'))
+            subtitles = {}
             if format_id.endswith('mpegurl'):
-                formats = self._extract_m3u8_formats(url, video_id, 'mp4')
+                formats, subtitles = self._extract_m3u8_formats_and_subtitles(url, video_id, 'mp4')
             elif format_id == 'f4m':
                 formats = self._extract_f4m_formats(url, video_id)
             else:
@@ -2457,11 +2458,12 @@ class GenericIE(InfoExtractor):
                 info_dict['direct'] = True
             self._sort_formats(formats)
             info_dict['formats'] = formats
+            info_dict['subtitles'] = subtitles
             return info_dict
 
         if not self._downloader.params.get('test', False) and not is_intentional:
             force = self._downloader.params.get('force_generic_extractor', False)
-            self._downloader.report_warning(
+            self.report_warning(
                 '%s on generic information extractor.' % ('Forcing' if force else 'Falling back'))
 
         if not full_response:
@@ -2488,7 +2490,7 @@ class GenericIE(InfoExtractor):
         # Maybe it's a direct link to a video?
         # Be careful not to download the whole thing!
         if not is_html(first_bytes):
-            self._downloader.report_warning(
+            self.report_warning(
                 'URL could be a direct video link, returning it as such.')
             info_dict.update({
                 'direct': True,
@@ -2506,11 +2508,14 @@ class GenericIE(InfoExtractor):
 
         # Is it an RSS feed, a SMIL file, an XSPF playlist or a MPD manifest?
         try:
-            doc = compat_etree_fromstring(webpage.encode('utf-8'))
+            try:
+                doc = compat_etree_fromstring(webpage)
+            except compat_xml_parse_error:
+                doc = compat_etree_fromstring(webpage.encode('utf-8'))
             if doc.tag == 'rss':
                 return self._extract_rss(url, video_id, doc)
             elif doc.tag == 'SmoothStreamingMedia':
-                info_dict['formats'] = self._parse_ism_formats(doc, url)
+                info_dict['formats'], info_dict['subtitles'] = self._parse_ism_formats_and_subtitles(doc, url)
                 self._sort_formats(info_dict['formats'])
                 return info_dict
             elif re.match(r'^(?:{[^}]+})?smil$', doc.tag):
@@ -2524,7 +2529,7 @@ class GenericIE(InfoExtractor):
                         xspf_base_url=full_response.geturl()),
                     video_id)
             elif re.match(r'(?i)^(?:{[^}]+})?MPD$', doc.tag):
-                info_dict['formats'] = self._parse_mpd_formats(
+                info_dict['formats'], info_dict['subtitles'] = self._parse_mpd_formats_and_subtitles(
                     doc,
                     mpd_base_url=full_response.geturl().rpartition('/')[0],
                     mpd_url=url)

yt_dlp/extractor/go.py

@@ -4,10 +4,14 @@ from __future__ import unicode_literals
 import re
 
 from .adobepass import AdobePassIE
+from ..compat import compat_str
 from ..utils import (
     int_or_none,
     determine_ext,
     parse_age_limit,
+    remove_start,
+    remove_end,
+    try_get,
     urlencode_postdata,
     ExtractorError,
 )
@@ -46,15 +50,15 @@ class GoIE(AdobePassIE):
     }
     _VALID_URL = r'''(?x)
                     https?://
-                        (?:
-                            (?:(?P<sub_domain>%s)\.)?go|
-                            (?P<sub_domain_2>abc|freeform|disneynow|fxnow\.fxnetworks)
+                        (?P<sub_domain>
+                            (?:%s\.)?go|fxnow\.fxnetworks|
+                            (?:www\.)?(?:abc|freeform|disneynow)
                         )\.com/
                         (?:
                             (?:[^/]+/)*(?P<id>[Vv][Dd][Kk][Aa]\w+)|
                             (?:[^/]+/)*(?P<display_id>[^/?\#]+)
                         )
-                    ''' % '|'.join(list(_SITE_INFO.keys()))
+                    ''' % r'\.|'.join(list(_SITE_INFO.keys()))
     _TESTS = [{
         'url': 'http://abc.go.com/shows/designated-survivor/video/most-recent/VDKA3807643',
         'info_dict': {
@@ -116,6 +120,18 @@ class GoIE(AdobePassIE):
             # m3u8 download
             'skip_download': True,
         },
+    }, {
+        'url': 'https://abc.com/shows/modern-family/episode-guide/season-01/101-pilot',
+        'info_dict': {
+            'id': 'VDKA22600213',
+            'ext': 'mp4',
+            'title': 'Pilot',
+            'description': 'md5:74306df917cfc199d76d061d66bebdb4',
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        },
     }, {
         'url': 'http://abc.go.com/shows/the-catch/episode-guide/season-01/10-the-wedding',
         'only_matching': True,
@@ -133,6 +149,9 @@ class GoIE(AdobePassIE):
     }, {
         'url': 'https://disneynow.com/shows/minnies-bow-toons/video/happy-campers/vdka4872013',
         'only_matching': True,
+    }, {
+        'url': 'https://www.freeform.com/shows/cruel-summer/episode-guide/season-01/01-happy-birthday-jeanette-turner',
+        'only_matching': True,
     }]
 
     def _extract_videos(self, brand, video_id='-1', show_id='-1'):
@@ -143,24 +162,36 @@ class GoIE(AdobePassIE):
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
-        sub_domain = mobj.group('sub_domain') or mobj.group('sub_domain_2')
+        sub_domain = remove_start(remove_end(mobj.group('sub_domain') or '', '.go'), 'www.')
         video_id, display_id = mobj.group('id', 'display_id')
         site_info = self._SITE_INFO.get(sub_domain, {})
         brand = site_info.get('brand')
         if not video_id or not site_info:
             webpage = self._download_webpage(url, display_id or video_id)
-            video_id = self._search_regex(
-                (
-                    # There may be inner quotes, e.g. data-video-id="'VDKA3609139'"
-                    # from http://freeform.go.com/shows/shadowhunters/episodes/season-2/1-this-guilty-blood
-                    r'data-video-id=["\']*(VDKA\w+)',
-                    # https://github.com/ytdl-org/youtube-dl/pull/25216/files
-                    #  The following is based on the pull request on the line above. Changed the ABC.com URL to a show available now.
-                    # https://abc.com/shows/the-rookie/episode-guide/season-02/19-the-q-word
-                    r'\bvideoIdCode["\']\s*:\s*["\'](vdka\w+)',
-                    # Deprecated  fallback pattern
-                    r'\b(?:video)?id["\']\s*:\s*["\'](VDKA\w+)'
-                ), webpage, 'video id', default=video_id)
+            data = self._parse_json(
+                self._search_regex(
+                    r'["\']__abc_com__["\']\s*\]\s*=\s*({.+?})\s*;', webpage,
+                    'data', default='{}'),
+                display_id or video_id, fatal=False)
+            # https://abc.com/shows/modern-family/episode-guide/season-01/101-pilot
+            layout = try_get(data, lambda x: x['page']['content']['video']['layout'], dict)
+            video_id = None
+            if layout:
+                video_id = try_get(
+                    layout,
+                    (lambda x: x['videoid'], lambda x: x['video']['id']),
+                    compat_str)
+            if not video_id:
+                video_id = self._search_regex(
+                    (
+                        # There may be inner quotes, e.g. data-video-id="'VDKA3609139'"
+                        # from http://freeform.go.com/shows/shadowhunters/episodes/season-2/1-this-guilty-blood
+                        r'data-video-id=["\']*(VDKA\w+)',
+                        # page.analytics.videoIdCode
+                        r'\bvideoIdCode["\']\s*:\s*["\']((?:vdka|VDKA)\w+)',
+                        # https://abc.com/shows/the-rookie/episode-guide/season-02/03-the-bet
+                        r'\b(?:video)?id["\']\s*:\s*["\'](VDKA\w+)'
+                    ), webpage, 'video id', default=video_id)
             if not site_info:
                 brand = self._search_regex(
                     (r'data-brand=\s*["\']\s*(\d+)',

yt_dlp/extractor/googledrive.py

@@ -253,7 +253,7 @@ class GoogleDriveIE(InfoExtractor):
                             or 'unable to extract confirmation code')
 
         if not formats and reason:
-            raise ExtractorError(reason, expected=True)
+            self.raise_no_formats(reason, expected=True)
 
         self._sort_formats(formats)
 

yt_dlp/extractor/hotstar.py

@@ -184,7 +184,7 @@ class HotStarIE(HotStarBaseIE):
                     geo_restricted = True
                 continue
         if not formats and geo_restricted:
-            self.raise_geo_restricted(countries=['IN'])
+            self.raise_geo_restricted(countries=['IN'], metadata_available=True)
         self._sort_formats(formats)
 
         for f in formats:

yt_dlp/extractor/iprima.py

@@ -136,7 +136,7 @@ class IPrimaIE(InfoExtractor):
                 extract_formats(src)
 
         if not formats and '>GEO_IP_NOT_ALLOWED<' in playerpage:
-            self.raise_geo_restricted(countries=['CZ'])
+            self.raise_geo_restricted(countries=['CZ'], metadata_available=True)
 
         self._sort_formats(formats)
 

yt_dlp/extractor/iqiyi.py

@@ -280,7 +280,7 @@ class IqiyiIE(InfoExtractor):
                 msg = 'error %s' % code
                 if validation_result.get('msg'):
                     msg += ': ' + validation_result['msg']
-            self._downloader.report_warning('unable to log in: ' + msg)
+            self.report_warning('unable to log in: ' + msg)
             return False
 
         return True

yt_dlp/extractor/jamendo.py

@@ -29,34 +29,51 @@ class JamendoIE(InfoExtractor):
             'id': '196219',
             'display_id': 'stories-from-emona-i',
             'ext': 'flac',
-            'title': 'Maya Filipič - Stories from Emona I',
-            'artist': 'Maya Filipič',
+            # 'title': 'Maya Filipič - Stories from Emona I',
+            'title': 'Stories from Emona I',
+            # 'artist': 'Maya Filipič',
             'track': 'Stories from Emona I',
             'duration': 210,
             'thumbnail': r're:^https?://.*\.jpg',
             'timestamp': 1217438117,
             'upload_date': '20080730',
+            'license': 'by-nc-nd',
+            'view_count': int,
+            'like_count': int,
+            'average_rating': int,
+            'tags': ['piano', 'peaceful', 'newage', 'strings', 'upbeat'],
         }
     }, {
         'url': 'https://licensing.jamendo.com/en/track/1496667/energetic-rock',
         'only_matching': True,
     }]
 
+    def _call_api(self, resource, resource_id):
+        path = '/api/%ss' % resource
+        rand = compat_str(random.random())
+        return self._download_json(
+            'https://www.jamendo.com' + path, resource_id, query={
+                'id[]': resource_id,
+            }, headers={
+                'X-Jam-Call': '$%s*%s~' % (hashlib.sha1((path + rand).encode()).hexdigest(), rand)
+            })[0]
+
     def _real_extract(self, url):
         track_id, display_id = self._VALID_URL_RE.match(url).groups()
-        webpage = self._download_webpage(
-            'https://www.jamendo.com/track/' + track_id, track_id)
-        models = self._parse_json(self._html_search_regex(
-            r"data-bundled-models='([^']+)",
-            webpage, 'bundled models'), track_id)
-        track = models['track']['models'][0]
+        # webpage = self._download_webpage(
+        #     'https://www.jamendo.com/track/' + track_id, track_id)
+        # models = self._parse_json(self._html_search_regex(
+        #     r"data-bundled-models='([^']+)",
+        #     webpage, 'bundled models'), track_id)
+        # track = models['track']['models'][0]
+        track = self._call_api('track', track_id)
         title = track_name = track['name']
-        get_model = lambda x: try_get(models, lambda y: y[x]['models'][0], dict) or {}
-        artist = get_model('artist')
-        artist_name = artist.get('name')
-        if artist_name:
-            title = '%s - %s' % (artist_name, title)
-        album = get_model('album')
+        # get_model = lambda x: try_get(models, lambda y: y[x]['models'][0], dict) or {}
+        # artist = get_model('artist')
+        # artist_name = artist.get('name')
+        # if artist_name:
+        #     title = '%s - %s' % (artist_name, title)
+        # album = get_model('album')
 
         formats = [{
             'url': 'https://%s.jamendo.com/?trackid=%s&format=%s&from=app-97dab294'
@@ -74,7 +91,7 @@ class JamendoIE(InfoExtractor):
 
         urls = []
         thumbnails = []
-        for _, covers in track.get('cover', {}).items():
+        for covers in (track.get('cover') or {}).values():
             for cover_id, cover_url in covers.items():
                 if not cover_url or cover_url in urls:
                     continue
@@ -88,13 +105,14 @@ class JamendoIE(InfoExtractor):
                 })
 
         tags = []
-        for tag in track.get('tags', []):
+        for tag in (track.get('tags') or []):
             tag_name = tag.get('name')
             if not tag_name:
                 continue
             tags.append(tag_name)
 
         stats = track.get('stats') or {}
+        license = track.get('licenseCC') or []
 
         return {
             'id': track_id,
@@ -103,11 +121,11 @@ class JamendoIE(InfoExtractor):
             'title': title,
             'description': track.get('description'),
             'duration': int_or_none(track.get('duration')),
-            'artist': artist_name,
+            # 'artist': artist_name,
             'track': track_name,
-            'album': album.get('name'),
+            # 'album': album.get('name'),
             'formats': formats,
-            'license': '-'.join(track.get('licenseCC', [])) or None,
+            'license': '-'.join(license) if license else None,
             'timestamp': int_or_none(track.get('dateCreated')),
             'view_count': int_or_none(stats.get('listenedAll')),
             'like_count': int_or_none(stats.get('favorited')),
@@ -116,9 +134,9 @@ class JamendoIE(InfoExtractor):
         }
 
 
-class JamendoAlbumIE(InfoExtractor):
+class JamendoAlbumIE(JamendoIE):
     _VALID_URL = r'https?://(?:www\.)?jamendo\.com/album/(?P<id>[0-9]+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'https://www.jamendo.com/album/121486/duck-on-cover',
         'info_dict': {
             'id': '121486',
@@ -151,17 +169,7 @@ class JamendoAlbumIE(InfoExtractor):
         'params': {
             'playlistend': 2
         }
-    }
-
-    def _call_api(self, resource, resource_id):
-        path = '/api/%ss' % resource
-        rand = compat_str(random.random())
-        return self._download_json(
-            'https://www.jamendo.com' + path, resource_id, query={
-                'id[]': resource_id,
-            }, headers={
-                'X-Jam-Call': '$%s*%s~' % (hashlib.sha1((path + rand).encode()).hexdigest(), rand)
-            })[0]
+    }]
 
     def _real_extract(self, url):
         album_id = self._match_id(url)
@@ -169,7 +177,7 @@ class JamendoAlbumIE(InfoExtractor):
         album_name = album.get('name')
 
         entries = []
-        for track in album.get('tracks', []):
+        for track in (album.get('tracks') or []):
             track_id = track.get('id')
             if not track_id:
                 continue

yt_dlp/extractor/kaltura.py

@@ -120,7 +120,7 @@ class KalturaIE(InfoExtractor):
     def _extract_urls(webpage):
         # Embed codes: https://knowledge.kaltura.com/embedding-kaltura-media-players-your-site
         finditer = (
-            re.finditer(
+            list(re.finditer(
                 r"""(?xs)
                     kWidget\.(?:thumb)?[Ee]mbed\(
                     \{.*?
@@ -128,8 +128,8 @@ class KalturaIE(InfoExtractor):
                         (?P<q2>['"])_?(?P<partner_id>(?:(?!(?P=q2)).)+)(?P=q2),.*?
                         (?P<q3>['"])entry_?[Ii]d(?P=q3)\s*:\s*
                         (?P<q4>['"])(?P<id>(?:(?!(?P=q4)).)+)(?P=q4)(?:,|\s*\})
-                """, webpage)
-            or re.finditer(
+                """, webpage))
+            or list(re.finditer(
                 r'''(?xs)
                     (?P<q1>["'])
                         (?:https?:)?//cdnapi(?:sec)?\.kaltura\.com(?::\d+)?/(?:(?!(?P=q1)).)*\b(?:p|partner_id)/(?P<partner_id>\d+)(?:(?!(?P=q1)).)*
@@ -142,16 +142,16 @@ class KalturaIE(InfoExtractor):
                         \[\s*(?P<q2_1>["'])entry_?[Ii]d(?P=q2_1)\s*\]\s*=\s*
                     )
                     (?P<q3>["'])(?P<id>(?:(?!(?P=q3)).)+)(?P=q3)
-                ''', webpage)
-            or re.finditer(
+                ''', webpage))
+            or list(re.finditer(
                 r'''(?xs)
-                    <(?:iframe[^>]+src|meta[^>]+\bcontent)=(?P<q1>["'])
+                    <(?:iframe[^>]+src|meta[^>]+\bcontent)=(?P<q1>["'])\s*
                       (?:https?:)?//(?:(?:www|cdnapi(?:sec)?)\.)?kaltura\.com/(?:(?!(?P=q1)).)*\b(?:p|partner_id)/(?P<partner_id>\d+)
                       (?:(?!(?P=q1)).)*
                       [?&;]entry_id=(?P<id>(?:(?!(?P=q1))[^&])+)
                       (?:(?!(?P=q1)).)*
                     (?P=q1)
-                ''', webpage)
+                ''', webpage))
         )
         urls = []
         for mobj in finditer:

yt_dlp/extractor/keezmovies.py

@@ -101,7 +101,7 @@ class KeezMoviesIE(InfoExtractor):
 
         if not formats:
             if 'title="This video is no longer available"' in webpage:
-                raise ExtractorError(
+                self.raise_no_formats(
                     'Video %s is no longer available' % video_id, expected=True)
 
         try:

yt_dlp/extractor/lbry.py

@@ -122,6 +122,26 @@ class LBRYIE(LBRYBaseIE):
             'channel_url': 'https://lbry.tv/@LBRYFoundation:0ed629d2b9c601300cacf7eabe9da0be79010212',
             'vcodec': 'none',
         }
+    }, {
+        # HLS
+        'url': 'https://odysee.com/@gardeningincanada:b/plants-i-will-never-grow-again.-the:e',
+        'md5': 'fc82f45ea54915b1495dd7cb5cc1289f',
+        'info_dict': {
+            'id': 'e51671357333fe22ae88aad320bde2f6f96b1410',
+            'ext': 'mp4',
+            'title': 'PLANTS I WILL NEVER GROW AGAIN. THE BLACK LIST PLANTS FOR A CANADIAN GARDEN | Gardening in Canada 🍁',
+            'description': 'md5:9c539c6a03fb843956de61a4d5288d5e',
+            'timestamp': 1618254123,
+            'upload_date': '20210412',
+            'release_timestamp': 1618254002,
+            'release_date': '20210412',
+            'tags': list,
+            'duration': 554,
+            'channel': 'Gardening In Canada',
+            'channel_id': 'b8be0e93b423dad221abe29545fbe8ec36e806bc',
+            'channel_url': 'https://odysee.com/@gardeningincanada:b8be0e93b423dad221abe29545fbe8ec36e806bc',
+            'formats': 'mincount:3',
+        }
     }, {
         'url': 'https://odysee.com/@BrodieRobertson:5/apple-is-tracking-everything-you-do-on:e',
         'only_matching': True,
@@ -168,10 +188,18 @@ class LBRYIE(LBRYBaseIE):
         streaming_url = self._call_api_proxy(
             'get', claim_id, {'uri': uri}, 'streaming url')['streaming_url']
         info = self._parse_stream(result, url)
+        urlh = self._request_webpage(
+            streaming_url, display_id, note='Downloading streaming redirect url info')
+        if determine_ext(urlh.geturl()) == 'm3u8':
+            info['formats'] = self._extract_m3u8_formats(
+                urlh.geturl(), display_id, 'mp4', entry_protocol='m3u8_native',
+                m3u8_id='hls')
+            self._sort_formats(info['formats'])
+        else:
+            info['url'] = streaming_url
         info.update({
             'id': claim_id,
             'title': title,
-            'url': streaming_url,
         })
         return info
 

yt_dlp/extractor/limelight.py

@@ -160,7 +160,10 @@ class LimelightBaseIE(InfoExtractor):
         for mobile_url in mobile_item.get('mobileUrls', []):
             media_url = mobile_url.get('mobileUrl')
             format_id = mobile_url.get('targetMediaPlatform')
-            if not media_url or format_id in ('Widevine', 'SmoothStreaming') or media_url in urls:
+            if not media_url or media_url in urls:
+                continue
+            if (format_id in ('Widevine', 'SmoothStreaming')
+                    and not self._downloader.params.get('allow_unplayable_formats', False)):
                 continue
             urls.append(media_url)
             ext = determine_ext(media_url)

yt_dlp/extractor/line.py

@@ -4,7 +4,12 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..utils import js_to_json
+from ..compat import compat_str
+from ..utils import (
+    int_or_none,
+    js_to_json,
+    str_or_none,
+)
 
 
 class LineTVIE(InfoExtractor):
@@ -71,7 +76,7 @@ class LineTVIE(InfoExtractor):
 
         self._sort_formats(formats)
 
-        if not formats[0].get('width'):
+        if formats and not formats[0].get('width'):
             formats[0]['vcodec'] = 'none'
 
         title = self._og_search_title(webpage)
@@ -88,3 +93,137 @@ class LineTVIE(InfoExtractor):
                            for thumbnail in video_info.get('thumbnails', {}).get('list', [])],
             'view_count': video_info.get('meta', {}).get('count'),
         }
+
+
+class LineLiveBaseIE(InfoExtractor):
+    _API_BASE_URL = 'https://live-api.line-apps.com/web/v4.0/channel/'
+
+    def _parse_broadcast_item(self, item):
+        broadcast_id = compat_str(item['id'])
+        title = item['title']
+        is_live = item.get('isBroadcastingNow')
+
+        thumbnails = []
+        for thumbnail_id, thumbnail_url in (item.get('thumbnailURLs') or {}).items():
+            if not thumbnail_url:
+                continue
+            thumbnails.append({
+                'id': thumbnail_id,
+                'url': thumbnail_url,
+            })
+
+        channel = item.get('channel') or {}
+        channel_id = str_or_none(channel.get('id'))
+
+        return {
+            'id': broadcast_id,
+            'title': self._live_title(title) if is_live else title,
+            'thumbnails': thumbnails,
+            'timestamp': int_or_none(item.get('createdAt')),
+            'channel': channel.get('name'),
+            'channel_id': channel_id,
+            'channel_url': 'https://live.line.me/channels/' + channel_id if channel_id else None,
+            'duration': int_or_none(item.get('archiveDuration')),
+            'view_count': int_or_none(item.get('viewerCount')),
+            'comment_count': int_or_none(item.get('chatCount')),
+            'is_live': is_live,
+        }
+
+
+class LineLiveIE(LineLiveBaseIE):
+    _VALID_URL = r'https?://live\.line\.me/channels/(?P<channel_id>\d+)/broadcast/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://live.line.me/channels/4867368/broadcast/16331360',
+        'md5': 'bc931f26bf1d4f971e3b0982b3fab4a3',
+        'info_dict': {
+            'id': '16331360',
+            'title': '振りコピ講座😙😙😙',
+            'ext': 'mp4',
+            'timestamp': 1617095132,
+            'upload_date': '20210330',
+            'channel': '白川ゆめか',
+            'channel_id': '4867368',
+            'view_count': int,
+            'comment_count': int,
+            'is_live': False,
+        }
+    }, {
+        # archiveStatus == 'DELETED'
+        'url': 'https://live.line.me/channels/4778159/broadcast/16378488',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        channel_id, broadcast_id = re.match(self._VALID_URL, url).groups()
+        broadcast = self._download_json(
+            self._API_BASE_URL + '%s/broadcast/%s' % (channel_id, broadcast_id),
+            broadcast_id)
+        item = broadcast['item']
+        info = self._parse_broadcast_item(item)
+        protocol = 'm3u8' if info['is_live'] else 'm3u8_native'
+        formats = []
+        for k, v in (broadcast.get(('live' if info['is_live'] else 'archived') + 'HLSURLs') or {}).items():
+            if not v:
+                continue
+            if k == 'abr':
+                formats.extend(self._extract_m3u8_formats(
+                    v, broadcast_id, 'mp4', protocol,
+                    m3u8_id='hls', fatal=False))
+                continue
+            f = {
+                'ext': 'mp4',
+                'format_id': 'hls-' + k,
+                'protocol': protocol,
+                'url': v,
+            }
+            if not k.isdigit():
+                f['vcodec'] = 'none'
+            formats.append(f)
+        if not formats:
+            archive_status = item.get('archiveStatus')
+            if archive_status != 'ARCHIVED':
+                self.raise_no_formats('this video has been ' + archive_status.lower(), expected=True)
+        self._sort_formats(formats)
+        info['formats'] = formats
+        return info
+
+
+class LineLiveChannelIE(LineLiveBaseIE):
+    _VALID_URL = r'https?://live\.line\.me/channels/(?P<id>\d+)(?!/broadcast/\d+)(?:[/?&#]|$)'
+    _TEST = {
+        'url': 'https://live.line.me/channels/5893542',
+        'info_dict': {
+            'id': '5893542',
+            'title': 'いくらちゃん',
+            'description': 'md5:c3a4af801f43b2fac0b02294976580be',
+        },
+        'playlist_mincount': 29
+    }
+
+    def _archived_broadcasts_entries(self, archived_broadcasts, channel_id):
+        while True:
+            for row in (archived_broadcasts.get('rows') or []):
+                share_url = str_or_none(row.get('shareURL'))
+                if not share_url:
+                    continue
+                info = self._parse_broadcast_item(row)
+                info.update({
+                    '_type': 'url',
+                    'url': share_url,
+                    'ie_key': LineLiveIE.ie_key(),
+                })
+                yield info
+            if not archived_broadcasts.get('hasNextPage'):
+                return
+            archived_broadcasts = self._download_json(
+                self._API_BASE_URL + channel_id + '/archived_broadcasts',
+                channel_id, query={
+                    'lastId': info['id'],
+                })
+
+    def _real_extract(self, url):
+        channel_id = self._match_id(url)
+        channel = self._download_json(self._API_BASE_URL + channel_id, channel_id)
+        return self.playlist_result(
+            self._archived_broadcasts_entries(channel.get('archivedBroadcasts') or {}, channel_id),
+            channel_id, channel.get('title'), channel.get('information'))

yt_dlp/extractor/lynda.py

@@ -331,7 +331,7 @@ class LyndaCourseIE(LyndaBaseIE):
                     })
 
         if unaccessible_videos > 0:
-            self._downloader.report_warning(
+            self.report_warning(
                 '%s videos are only available for members (or paid members) and will not be downloaded. '
                 % unaccessible_videos + self._ACCOUNT_CREDENTIALS_HINT)
 

yt_dlp/extractor/maoritv.py

@@ -0,0 +1,31 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class MaoriTVIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?maoritelevision\.com/shows/(?:[^/]+/)+(?P<id>[^/?&#]+)'
+    _TEST = {
+        'url': 'https://www.maoritelevision.com/shows/korero-mai/S01E054/korero-mai-series-1-episode-54',
+        'md5': '5ade8ef53851b6a132c051b1cd858899',
+        'info_dict': {
+            'id': '4774724855001',
+            'ext': 'mp4',
+            'title': 'Kōrero Mai, Series 1 Episode 54',
+            'upload_date': '20160226',
+            'timestamp': 1456455018,
+            'description': 'md5:59bde32fd066d637a1a55794c56d8dcb',
+            'uploader_id': '1614493167001',
+        },
+    }
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/1614493167001/HJlhIQhQf_default/index.html?videoId=%s'
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
+        brightcove_id = self._search_regex(
+            r'data-main-video-id=["\'](\d+)', webpage, 'brightcove id')
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id,
+            'BrightcoveNew', brightcove_id)

yt_dlp/extractor/medaltv.py

@@ -15,33 +15,39 @@ from ..utils import (
 
 
 class MedalTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?medal\.tv/clips/(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://(?:www\.)?medal\.tv/clips/(?P<id>[^/?#&]+)'
     _TESTS = [{
-        'url': 'https://medal.tv/clips/34934644/3Is9zyGMoBMr',
+        'url': 'https://medal.tv/clips/2mA60jWAGQCBH',
         'md5': '7b07b064331b1cf9e8e5c52a06ae68fa',
         'info_dict': {
-            'id': '34934644',
+            'id': '2mA60jWAGQCBH',
             'ext': 'mp4',
             'title': 'Quad Cold',
             'description': 'Medal,https://medal.tv/desktop/',
             'uploader': 'MowgliSB',
             'timestamp': 1603165266,
             'upload_date': '20201020',
-            'uploader_id': 10619174,
+            'uploader_id': '10619174',
         }
     }, {
-        'url': 'https://medal.tv/clips/36787208',
+        'url': 'https://medal.tv/clips/2um24TWdty0NA',
         'md5': 'b6dc76b78195fff0b4f8bf4a33ec2148',
         'info_dict': {
-            'id': '36787208',
+            'id': '2um24TWdty0NA',
             'ext': 'mp4',
             'title': 'u tk me i tk u bigger',
             'description': 'Medal,https://medal.tv/desktop/',
             'uploader': 'Mimicc',
             'timestamp': 1605580939,
             'upload_date': '20201117',
-            'uploader_id': 5156321,
+            'uploader_id': '5156321',
         }
+    }, {
+        'url': 'https://medal.tv/clips/37rMeFpryCC-9',
+        'only_matching': True,
+    }, {
+        'url': 'https://medal.tv/clips/2WRj40tpY_EU9',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -97,11 +103,11 @@ class MedalTVIE(InfoExtractor):
         error = clip.get('error')
         if not formats and error:
             if error == 404:
-                raise ExtractorError(
+                self.raise_no_formats(
                     'That clip does not exist.',
                     expected=True, video_id=video_id)
             else:
-                raise ExtractorError(
+                self.raise_no_formats(
                     'An unknown error occurred ({0}).'.format(error),
                     video_id=video_id)
 

yt_dlp/extractor/mediasite.py

@@ -26,7 +26,7 @@ _ID_RE = r'(?:[0-9a-f]{32,34}|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0
 
 
 class MediasiteIE(InfoExtractor):
-    _VALID_URL = r'(?xi)https?://[^/]+/Mediasite/(?:Play|Showcase/(?:default|livebroadcast)/Presentation)/(?P<id>%s)(?P<query>\?[^#]+|)' % _ID_RE
+    _VALID_URL = r'(?xi)https?://[^/]+/Mediasite/(?:Play|Showcase/[^/#?]+/Presentation)/(?P<id>%s)(?P<query>\?[^#]+|)' % _ID_RE
     _TESTS = [
         {
             'url': 'https://hitsmediaweb.h-its.org/mediasite/Play/2db6c271681e4f199af3c60d1f82869b1d',

yt_dlp/extractor/mildom.py

@@ -1,22 +1,20 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import base64
 from datetime import datetime
 import itertools
 import json
-import base64
 import re
 
 from .common import InfoExtractor
 from ..utils import (
-    ExtractorError, std_headers,
+    std_headers,
     update_url_query,
     random_uuidv4,
     try_get,
 )
 from ..compat import (
-    compat_urlparse,
-    compat_urllib_parse_urlencode,
     compat_str,
 )
 
@@ -48,29 +46,24 @@ class MildomBaseIE(InfoExtractor):
 
     def _fetch_dispatcher_config(self):
         if not self._DISPATCHER_CONFIG:
-            try:
-                tmp = self._download_json(
-                    'https://disp.mildom.com/serverListV2', 'initialization',
-                    note='Downloading dispatcher_config', data=json.dumps({
-                        'protover': 0,
-                        'data': base64.b64encode(json.dumps({
-                            'fr': 'web',
-                            'sfr': 'pc',
-                            'devi': 'Windows',
-                            'la': 'ja',
-                            'gid': None,
-                            'loc': '',
-                            'clu': '',
-                            'wh': '1919*810',
-                            'rtm': self.iso_timestamp(),
-                            'ua': std_headers['User-Agent'],
-                        }).encode('utf8')).decode('utf8').replace('\n', ''),
-                    }).encode('utf8'))
-                self._DISPATCHER_CONFIG = self._parse_json(base64.b64decode(tmp['data']), 'initialization')
-            except ExtractorError:
-                self._DISPATCHER_CONFIG = self._download_json(
-                    'https://bookish-octo-barnacle.vercel.app/api/mildom/dispatcher_config', 'initialization',
-                    note='Downloading dispatcher_config fallback')
+            tmp = self._download_json(
+                'https://disp.mildom.com/serverListV2', 'initialization',
+                note='Downloading dispatcher_config', data=json.dumps({
+                    'protover': 0,
+                    'data': base64.b64encode(json.dumps({
+                        'fr': 'web',
+                        'sfr': 'pc',
+                        'devi': 'Windows',
+                        'la': 'ja',
+                        'gid': None,
+                        'loc': '',
+                        'clu': '',
+                        'wh': '1919*810',
+                        'rtm': self.iso_timestamp(),
+                        'ua': std_headers['User-Agent'],
+                    }).encode('utf8')).decode('utf8').replace('\n', ''),
+                }).encode('utf8'))
+            self._DISPATCHER_CONFIG = self._parse_json(base64.b64decode(tmp['data']), 'initialization')
         return self._DISPATCHER_CONFIG
 
     @staticmethod
@@ -145,15 +138,10 @@ class MildomIE(MildomBaseIE):
             'Referer': 'https://www.mildom.com/',
             'Origin': 'https://www.mildom.com',
         }, note='Downloading m3u8 information')
+
         del stream_query['streamReqId'], stream_query['timestamp']
         for fmt in formats:
-            # Uses https://github.com/nao20010128nao/bookish-octo-barnacle by @nao20010128nao as a proxy
-            parsed = compat_urlparse.urlparse(fmt['url'])
-            parsed = parsed._replace(
-                netloc='bookish-octo-barnacle.vercel.app',
-                query=compat_urllib_parse_urlencode(stream_query, True),
-                path='/api/mildom' + parsed.path)
-            fmt['url'] = compat_urlparse.urlunparse(parsed)
+            fmt.setdefault('http_headers', {})['Referer'] = 'https://www.mildom.com/'
 
         self._sort_formats(formats)
 
@@ -200,16 +188,16 @@ class MildomVodIE(MildomBaseIE):
                 lambda x: x['author_info']['login_name'],
             ), compat_str)
 
-        audio_formats = [{
+        formats = [{
             'url': autoplay['audio_url'],
             'format_id': 'audio',
             'protocol': 'm3u8_native',
             'vcodec': 'none',
             'acodec': 'aac',
+            'ext': 'm4a'
         }]
-        video_formats = []
         for fmt in autoplay['video_link']:
-            video_formats.append({
+            formats.append({
                 'format_id': 'video-%s' % fmt['name'],
                 'url': fmt['url'],
                 'protocol': 'm3u8_native',
@@ -217,23 +205,9 @@ class MildomVodIE(MildomBaseIE):
                 'height': fmt['level'],
                 'vcodec': 'h264',
                 'acodec': 'aac',
+                'ext': 'mp4'
             })
 
-        stream_query = self._common_queries({
-            'is_lhls': '0',
-        })
-        del stream_query['timestamp']
-        formats = audio_formats + video_formats
-        for fmt in formats:
-            fmt['ext'] = 'mp4'
-            parsed = compat_urlparse.urlparse(fmt['url'])
-            stream_query['path'] = parsed.path[5:]
-            parsed = parsed._replace(
-                netloc='bookish-octo-barnacle.vercel.app',
-                query=compat_urllib_parse_urlencode(stream_query, True),
-                path='/api/mildom/vod2/proxy')
-            fmt['url'] = compat_urlparse.urlunparse(parsed)
-
         self._sort_formats(formats)
 
         return {
@@ -259,16 +233,7 @@ class MildomUserVodIE(MildomBaseIE):
         'playlist_mincount': 351,
     }]
 
-    def _real_extract(self, url):
-        user_id = self._match_id(url)
-
-        self._downloader.report_warning('To download ongoing live, please use "https://www.mildom.com/%s" instead. This will list up VODs belonging to user.' % user_id)
-
-        profile = self._call_api(
-            'https://cloudac.mildom.com/nonolive/gappserv/user/profileV2', user_id,
-            query={'user_id': user_id}, note='Downloading user profile')['user_info']
-
-        results = []
+    def _entries(self, user_id):
         for page in itertools.count(1):
             reply = self._call_api(
                 'https://cloudac.mildom.com/nonolive/videocontent/profile/playbackList',
@@ -279,7 +244,16 @@ class MildomUserVodIE(MildomBaseIE):
                 })
             if not reply:
                 break
-            results.extend('https://www.mildom.com/playback/%s/%s' % (user_id, x['v_id']) for x in reply)
-        return self.playlist_result([
-            self.url_result(u, ie=MildomVodIE.ie_key()) for u in results
-        ], user_id, 'Uploads from %s' % profile['loginname'])
+            for x in reply:
+                yield self.url_result('https://www.mildom.com/playback/%s/%s' % (user_id, x['v_id']))
+
+    def _real_extract(self, url):
+        user_id = self._match_id(url)
+        self.to_screen('This will download all VODs belonging to user. To download ongoing live video, use "https://www.mildom.com/%s" instead' % user_id)
+
+        profile = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/user/profileV2', user_id,
+            query={'user_id': user_id}, note='Downloading user profile')['user_info']
+
+        return self.playlist_result(
+            self._entries(user_id), user_id, 'Uploads from %s' % profile['loginname'])

yt_dlp/extractor/mixcloud.py

@@ -157,7 +157,7 @@ class MixcloudIE(MixcloudBaseIE):
                 })
 
         if not formats and cloudcast.get('isExclusive'):
-            self.raise_login_required()
+            self.raise_login_required(metadata_available=True)
 
         self._sort_formats(formats)
 

yt_dlp/extractor/mtv.py

@@ -275,7 +275,9 @@ class MTVServicesInfoExtractor(InfoExtractor):
 
     @staticmethod
     def _extract_child_with_type(parent, t):
-        return next(c for c in parent['children'] if c.get('type') == t)
+        for c in parent['children']:
+            if c.get('type') == t:
+                return c
 
     def _extract_mgid(self, webpage):
         try:
@@ -306,7 +308,8 @@ class MTVServicesInfoExtractor(InfoExtractor):
             data = self._parse_json(self._search_regex(
                 r'__DATA__\s*=\s*({.+?});', webpage, 'data'), None)
             main_container = self._extract_child_with_type(data, 'MainContainer')
-            video_player = self._extract_child_with_type(main_container, 'VideoPlayer')
+            ab_testing = self._extract_child_with_type(main_container, 'ABTesting')
+            video_player = self._extract_child_with_type(ab_testing or main_container, 'VideoPlayer')
             mgid = video_player['props']['media']['video']['config']['uri']
 
         return mgid

yt_dlp/extractor/mxplayer.py

@@ -3,6 +3,7 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
+from ..compat import compat_str
 from ..utils import (
     ExtractorError,
     js_to_json,
@@ -14,7 +15,7 @@ from ..utils import (
 
 
 class MxplayerIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?mxplayer\.in/(?:show|movie)/(?:(?P<display_id>[-/a-z0-9]+)-)?(?P<id>[a-z0-9]+)'
+    _VALID_URL = r'https?://(?:www\.)?mxplayer\.in/(?:movie|show/[-\w]+/[-\w]+)/(?P<display_id>[-\w]+)-(?P<id>\w+)'
     _TESTS = [{
         'url': 'https://www.mxplayer.in/movie/watch-knock-knock-hindi-dubbed-movie-online-b9fa28df3bfb8758874735bbd7d2655a?watch=true',
         'info_dict': {
@@ -117,7 +118,7 @@ class MxplayerIE(InfoExtractor):
         self._sort_formats(formats)
         return {
             'id': video_id,
-            'display_id': display_id.replace('/', '-'),
+            'display_id': display_id,
             'title': video_dict['title'] or self._og_search_title(webpage),
             'formats': formats,
             'description': video_dict.get('description'),
@@ -125,3 +126,46 @@ class MxplayerIE(InfoExtractor):
             'series': try_get(video_dict, lambda x: x['container']['container']['title']),
             'thumbnails': thumbnails,
         }
+
+
+class MxplayerShowIE(InfoExtractor):
+    _VALID_URL = r'(?:https?://)(?:www\.)?mxplayer\.in/show/(?P<display_id>[-\w]+)-(?P<id>\w+)/?(?:$|[#?])'
+    _TESTS = [{
+        'url': 'https://www.mxplayer.in/show/watch-chakravartin-ashoka-samrat-series-online-a8f44e3cc0814b5601d17772cedf5417',
+        'playlist_mincount': 440,
+        'info_dict': {
+            'id': 'a8f44e3cc0814b5601d17772cedf5417',
+            'title': 'Watch Chakravartin Ashoka Samrat Series Online',
+        }
+    }]
+
+    _API_SHOW_URL = "https://api.mxplay.com/v1/web/detail/tab/tvshowseasons?type=tv_show&id={}&device-density=2&platform=com.mxplay.desktop&content-languages=hi,en"
+    _API_EPISODES_URL = "https://api.mxplay.com/v1/web/detail/tab/tvshowepisodes?type=season&id={}&device-density=1&platform=com.mxplay.desktop&content-languages=hi,en&{}"
+
+    def _entries(self, show_id):
+        show_json = self._download_json(
+            self._API_SHOW_URL.format(show_id),
+            video_id=show_id, headers={'Referer': 'https://mxplayer.in'})
+        page_num = 0
+        for season in show_json.get('items') or []:
+            season_id = try_get(season, lambda x: x['id'], compat_str)
+            next_url = ''
+            while next_url is not None:
+                page_num += 1
+                season_json = self._download_json(
+                    self._API_EPISODES_URL.format(season_id, next_url),
+                    video_id=season_id,
+                    headers={'Referer': 'https://mxplayer.in'},
+                    note='Downloading JSON metadata page %d' % page_num)
+                for episode in season_json.get('items') or []:
+                    video_url = episode['webUrl']
+                    yield self.url_result(
+                        'https://mxplayer.in%s' % video_url,
+                        ie=MxplayerIE.ie_key(), video_id=video_url.split('-')[-1])
+                next_url = season_json.get('next')
+
+    def _real_extract(self, url):
+        display_id, show_id = re.match(self._VALID_URL, url).groups()
+        return self.playlist_result(
+            self._entries(show_id), playlist_id=show_id,
+            playlist_title=display_id.replace('-', ' ').title())

yt_dlp/extractor/nebula.py

@@ -0,0 +1,201 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (
+    ExtractorError,
+    parse_iso8601,
+    try_get,
+    urljoin,
+)
+
+
+class NebulaIE(InfoExtractor):
+
+    _VALID_URL = r'https?://(?:www\.)?(?:watchnebula\.com|nebula\.app)/videos/(?P<id>[-\w]+)'
+    _TESTS = [
+        {
+            'url': 'https://nebula.app/videos/that-time-disney-remade-beauty-and-the-beast',
+            'md5': 'fe79c4df8b3aa2fea98a93d027465c7e',
+            'info_dict': {
+                'id': '5c271b40b13fd613090034fd',
+                'ext': 'mp4',
+                'title': 'That Time Disney Remade Beauty and the Beast',
+                'description': 'Note: this video was originally posted on YouTube with the sponsor read included. We weren’t able to remove it without reducing video quality, so it’s presented here in its original context.',
+                'upload_date': '20180731',
+                'timestamp': 1533009600,
+                'channel': 'Lindsay Ellis',
+                'uploader': 'Lindsay Ellis',
+            },
+            'params': {
+                'usenetrc': True,
+            },
+            'skip': 'All Nebula content requires authentication',
+        },
+        {
+            'url': 'https://nebula.app/videos/the-logistics-of-d-day-landing-craft-how-the-allies-got-ashore',
+            'md5': '6d4edd14ce65720fa63aba5c583fb328',
+            'info_dict': {
+                'id': '5e7e78171aaf320001fbd6be',
+                'ext': 'mp4',
+                'title': 'Landing Craft - How The Allies Got Ashore',
+                'description': r're:^In this episode we explore the unsung heroes of D-Day, the landing craft.',
+                'upload_date': '20200327',
+                'timestamp': 1585348140,
+                'channel': 'The Logistics of D-Day',
+                'uploader': 'The Logistics of D-Day',
+            },
+            'params': {
+                'usenetrc': True,
+            },
+            'skip': 'All Nebula content requires authentication',
+        },
+        {
+            'url': 'https://nebula.app/videos/money-episode-1-the-draw',
+            'md5': '8c7d272910eea320f6f8e6d3084eecf5',
+            'info_dict': {
+                'id': '5e779ebdd157bc0001d1c75a',
+                'ext': 'mp4',
+                'title': 'Episode 1: The Draw',
+                'description': r'contains:There’s free money on offer… if the players can all work together.',
+                'upload_date': '20200323',
+                'timestamp': 1584980400,
+                'channel': 'Tom Scott Presents: Money',
+                'uploader': 'Tom Scott Presents: Money',
+            },
+            'params': {
+                'usenetrc': True,
+            },
+            'skip': 'All Nebula content requires authentication',
+        },
+        {
+            'url': 'https://watchnebula.com/videos/money-episode-1-the-draw',
+            'only_matching': True,
+        },
+    ]
+    _NETRC_MACHINE = 'watchnebula'
+
+    def _retrieve_nebula_auth(self, video_id):
+        """
+        Log in to Nebula, and returns a Nebula API token
+        """
+
+        username, password = self._get_login_info()
+        if not (username and password):
+            self.raise_login_required()
+
+        self.report_login()
+        data = json.dumps({'email': username, 'password': password}).encode('utf8')
+        response = self._download_json(
+            'https://api.watchnebula.com/api/v1/auth/login/',
+            data=data, fatal=False, video_id=video_id,
+            headers={
+                'content-type': 'application/json',
+                # Submitting the 'sessionid' cookie always causes a 403 on auth endpoint
+                'cookie': ''
+            },
+            note='Authenticating to Nebula with supplied credentials',
+            errnote='Authentication failed or rejected')
+        if not response or not response.get('key'):
+            self.raise_login_required()
+        return response['key']
+
+    def _retrieve_zype_api_key(self, page_url, display_id):
+        """
+        Retrieves the Zype API key
+        """
+
+        # Find the js that has the API key from the webpage and download it
+        webpage = self._download_webpage(page_url, video_id=display_id)
+        main_script_relpath = self._search_regex(
+            r'<script[^>]*src="(?P<script_relpath>[^"]*main.[0-9a-f]*.chunk.js)"[^>]*>', webpage,
+            group='script_relpath', name='script relative path', fatal=True)
+        main_script_abspath = urljoin(page_url, main_script_relpath)
+        main_script = self._download_webpage(main_script_abspath, video_id=display_id,
+                                             note='Retrieving Zype API key')
+
+        api_key = self._search_regex(
+            r'REACT_APP_ZYPE_API_KEY\s*:\s*"(?P<api_key>[\w-]*)"', main_script,
+            group='api_key', name='API key', fatal=True)
+
+        return api_key
+
+    def _call_zype_api(self, path, params, video_id, api_key, note):
+        """
+        A helper for making calls to the Zype API.
+        """
+        query = {'api_key': api_key, 'per_page': 1}
+        query.update(params)
+        return self._download_json('https://api.zype.com' + path, video_id, query=query, note=note)
+
+    def _call_nebula_api(self, path, video_id, access_token, note):
+        """
+        A helper for making calls to the Nebula API.
+        """
+        return self._download_json('https://api.watchnebula.com/api/v1' + path, video_id, headers={
+            'Authorization': 'Token {access_token}'.format(access_token=access_token)
+        }, note=note)
+
+    def _fetch_zype_access_token(self, video_id, nebula_token):
+        user_object = self._call_nebula_api('/auth/user/', video_id, nebula_token, note='Retrieving Zype access token')
+        access_token = try_get(user_object, lambda x: x['zype_auth_info']['access_token'], compat_str)
+        if not access_token:
+            if try_get(user_object, lambda x: x['is_subscribed'], bool):
+                # TODO: Reimplement the same Zype token polling the Nebula frontend implements
+                # see https://github.com/ytdl-org/youtube-dl/pull/24805#issuecomment-749231532
+                raise ExtractorError(
+                    'Unable to extract Zype access token from Nebula API authentication endpoint. '
+                    'Open an arbitrary video in a browser with this account to generate a token',
+                    expected=True)
+            raise ExtractorError('Unable to extract Zype access token from Nebula API authentication endpoint')
+        return access_token
+
+    def _extract_channel_title(self, video_meta):
+        # TODO: Implement the API calls giving us the channel list,
+        # so that we can do the title lookup and then figure out the channel URL
+        categories = video_meta.get('categories', []) if video_meta else []
+        # the channel name is the value of the first category
+        for category in categories:
+            if category.get('value'):
+                return category['value'][0]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        nebula_token = self._retrieve_nebula_auth(display_id)
+        api_key = self._retrieve_zype_api_key(url, display_id)
+
+        response = self._call_zype_api('/videos', {'friendly_title': display_id},
+                                       display_id, api_key, note='Retrieving metadata from Zype')
+        if len(response.get('response') or []) != 1:
+            raise ExtractorError('Unable to find video on Zype API')
+        video_meta = response['response'][0]
+
+        video_id = video_meta['_id']
+        zype_access_token = self._fetch_zype_access_token(display_id, nebula_token=nebula_token)
+
+        channel_title = self._extract_channel_title(video_meta)
+
+        return {
+            'id': video_id,
+            'display_id': display_id,
+            '_type': 'url_transparent',
+            'ie_key': 'Zype',
+            'url': 'https://player.zype.com/embed/%s.html?access_token=%s' % (video_id, zype_access_token),
+            'title': video_meta.get('title'),
+            'description': video_meta.get('description'),
+            'timestamp': parse_iso8601(video_meta.get('published_at')),
+            'thumbnails': [
+                {
+                    'id': tn.get('name'),  # this appears to be null
+                    'url': tn['url'],
+                    'width': tn.get('width'),
+                    'height': tn.get('height'),
+                } for tn in video_meta.get('thumbnails', [])],
+            'duration': video_meta.get('duration'),
+            'channel': channel_title,
+            'uploader': channel_title,  # we chose uploader = channel name
+            # TODO: uploader_url, channel_id, channel_url
+        }

yt_dlp/extractor/nfhsnetwork.py

@@ -0,0 +1,144 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+from ..utils import (
+    try_get,
+    unified_strdate,
+    unified_timestamp
+)
+
+
+class NFHSNetworkIE(InfoExtractor):
+    IE_NAME = 'NFHSNetwork'
+    _VALID_URL = r'https?://(?:www\.)?nfhsnetwork\.com/events/[\w-]+/(?P<id>(?:gam|evt|dd|)?[\w\d]{0,10})'
+    _TESTS = [{
+        # Auto-generated two-team sport (pixellot)
+        'url': 'https://www.nfhsnetwork.com/events/rockford-high-school-rockford-mi/gamcf7e54cfbc',
+        'info_dict': {
+            'id': 'gamcf7e54cfbc',
+            'ext': 'mp4',
+            'title': 'Rockford vs Spring Lake - Girls Varsity Lacrosse 03/27/2021',
+            'uploader': 'MHSAA - Michigan: Rockford High School, Rockford, MI',
+            'uploader_id': 'cd2622cf76',
+            'uploader_url': 'https://www.nfhsnetwork.com/schools/rockford-high-school-rockford-mi',
+            'location': 'Rockford, Michigan',
+            'timestamp': 1616859000,
+            'upload_date': '20210327'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }, {
+        # Non-sport activity with description
+        'url': 'https://www.nfhsnetwork.com/events/limon-high-school-limon-co/evt4a30e3726c',
+        'info_dict': {
+            'id': 'evt4a30e3726c',
+            'ext': 'mp4',
+            'title': 'Drama Performance Limon High School vs. Limon High School - 12/13/2020',
+            'description': 'Join the broadcast of the Limon High School Musical Performance at 2 PM.',
+            'uploader': 'CHSAA: Limon High School, Limon, CO',
+            'uploader_id': '7d2d121332',
+            'uploader_url': 'https://www.nfhsnetwork.com/schools/limon-high-school-limon-co',
+            'location': 'Limon, Colorado',
+            'timestamp': 1607893200,
+            'upload_date': '20201213'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }, {
+        # Postseason game
+        'url': 'https://www.nfhsnetwork.com/events/nfhs-network-special-events/dd8de71d45',
+        'info_dict': {
+            'id': 'dd8de71d45',
+            'ext': 'mp4',
+            'title': '2015 UA Holiday Classic Tournament: National Division  - 12/26/2015',
+            'uploader': 'SoCal Sports Productions',
+            'uploader_id': '063dba0150',
+            'uploader_url': 'https://www.nfhsnetwork.com/affiliates/socal-sports-productions',
+            'location': 'San Diego, California',
+            'timestamp': 1451187000,
+            'upload_date': '20151226'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }, {
+        # Video with no broadcasts object
+        'url': 'https://www.nfhsnetwork.com/events/wiaa-wi/9aa2f92f82',
+        'info_dict': {
+            'id': '9aa2f92f82',
+            'ext': 'mp4',
+            'title': 'Competitive Equity  - 01/21/2015',
+            'description': 'Committee members discuss points of their research regarding a competitive equity plan',
+            'uploader': 'WIAA - Wisconsin: Wisconsin Interscholastic Athletic Association',
+            'uploader_id': 'a49f7d1002',
+            'uploader_url': 'https://www.nfhsnetwork.com/associations/wiaa-wi',
+            'location': 'Stevens Point, Wisconsin',
+            'timestamp': 1421856000,
+            'upload_date': '20150121'
+        },
+        'params': {
+            # m3u8 download
+            'skip_download': True,
+        }
+    }
+    ]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        data = self._download_json(
+            'https://cfunity.nfhsnetwork.com/v2/game_or_event/' + video_id,
+            video_id)
+        publisher = data.get('publishers')[0]  # always exists
+        broadcast = (publisher.get('broadcasts') or publisher.get('vods'))[0]  # some (older) videos don't have a broadcasts object
+        uploader = publisher.get('formatted_name') or publisher.get('name')
+        uploaderID = publisher.get('publisher_key')
+        pubType = publisher.get('type')
+        uploaderPrefix = (
+            "schools" if pubType == "school"
+            else "associations" if "association" in pubType
+            else "affiliates" if (pubType == "publisher" or pubType == "affiliate")
+            else "schools")
+        uploaderPage = 'https://www.nfhsnetwork.com/%s/%s' % (uploaderPrefix, publisher.get('slug'))
+        location = '%s, %s' % (data.get('city'), data.get('state_name'))
+        description = broadcast.get('description')
+        isLive = broadcast.get('on_air') or broadcast.get('status') == 'on_air' or False
+
+        timestamp = unified_timestamp(data.get('local_start_time'))
+        upload_date = unified_strdate(data.get('local_start_time'))
+
+        title = (
+            self._og_search_title(webpage)
+            or self._html_search_regex(r'<h1 class="sr-hidden">(.*?)</h1>', webpage, 'title'))
+        title = title.split('|')[0].strip()
+
+        video_type = 'broadcasts' if isLive else 'vods'
+        key = broadcast.get('key') if isLive else try_get(publisher, lambda x: x['vods'][0]['key'])
+        m3u8_url = self._download_json(
+            'https://cfunity.nfhsnetwork.com/v2/%s/%s/url' % (video_type, key),
+            video_id).get('video_url')
+
+        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', live=isLive)
+        self._sort_formats(formats, ['res', 'tbr'])
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'description': description,
+            'timestamp': timestamp,
+            'uploader': uploader,
+            'uploader_id': uploaderID,
+            'uploader_url': uploaderPage,
+            'location': location,
+            'upload_date': upload_date,
+            'is_live': isLive
+        }

yt_dlp/extractor/niconico.py

@@ -164,6 +164,11 @@ class NiconicoIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.|secure\.|sp\.)?nicovideo\.jp/watch/(?P<id>(?:[a-z]{2})?[0-9]+)'
     _NETRC_MACHINE = 'niconico'
 
+    _API_HEADERS = {
+        'X-Frontend-ID': '6',
+        'X-Frontend-Version': '0'
+    }
+
     def _real_initialize(self):
         self._login()
 
@@ -190,53 +195,55 @@ class NiconicoIE(InfoExtractor):
             if compat_parse_qs(parts.query).get('message', [None])[0] == 'cant_login':
                 login_ok = False
         if not login_ok:
-            self._downloader.report_warning('unable to log in: bad username or password')
+            self.report_warning('unable to log in: bad username or password')
         return login_ok
 
     def _get_heartbeat_info(self, info_dict):
 
         video_id, video_src_id, audio_src_id = info_dict['url'].split(':')[1].split('/')
 
-        # Get video webpage for API data.
-        webpage, handle = self._download_webpage_handle(
-            'http://www.nicovideo.jp/watch/' + video_id, video_id)
-
-        api_data = self._parse_json(self._html_search_regex(
-            'data-api-data="([^"]+)"', webpage,
-            'API data', default='{}'), video_id)
+        api_data = (
+            info_dict.get('_api_data')
+            or self._parse_json(
+                self._html_search_regex(
+                    'data-api-data="([^"]+)"',
+                    self._download_webpage('http://www.nicovideo.jp/watch/' + video_id, video_id),
+                    'API data', default='{}'),
+                video_id))
 
         session_api_data = try_get(api_data, lambda x: x['media']['delivery']['movie']['session'])
         session_api_endpoint = try_get(session_api_data, lambda x: x['urls'][0])
 
-        # ping
-        self._download_json(
-            'https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch', video_id,
-            query={'t': try_get(api_data, lambda x: x['video']['dmcInfo']['tracking_id'])},
-            headers={
-                'Origin': 'https://www.nicovideo.jp',
-                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
-                'X-Frontend-Id': '6',
-                'X-Frontend-Version': '0'
-            })
+        def ping():
+            status = try_get(
+                self._download_json(
+                    'https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch', video_id,
+                    query={'t': try_get(api_data, lambda x: x['media']['delivery']['trackingId'])},
+                    note='Acquiring permission for downloading video',
+                    headers=self._API_HEADERS),
+                lambda x: x['meta']['status'])
+            if status != 200:
+                self.report_warning('Failed to acquire permission for playing video. The video may not download.')
 
         yesno = lambda x: 'yes' if x else 'no'
 
         # m3u8 (encryption)
-        if 'encryption' in (try_get(api_data, lambda x: x['media']['delivery']['movie']) or {}):
+        if try_get(api_data, lambda x: x['media']['delivery']['encryption']) is not None:
             protocol = 'm3u8'
+            encryption = self._parse_json(session_api_data['token'], video_id)['hls_encryption']
             session_api_http_parameters = {
                 'parameters': {
                     'hls_parameters': {
                         'encryption': {
-                            'hls_encryption_v1': {
-                                'encrypted_key': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['encrypted_key']),
-                                'key_uri': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['key_uri'])
+                            encryption: {
+                                'encrypted_key': try_get(api_data, lambda x: x['media']['delivery']['encryption']['encryptedKey']),
+                                'key_uri': try_get(api_data, lambda x: x['media']['delivery']['encryption']['keyUri'])
                             }
                         },
                         'transfer_preset': '',
-                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
-                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
-                        'segment_duration': 6000
+                        'use_ssl': yesno(session_api_endpoint['isSsl']),
+                        'use_well_known_port': yesno(session_api_endpoint['isWellKnownPort']),
+                        'segment_duration': 6000,
                     }
                 }
             }
@@ -310,7 +317,8 @@ class NiconicoIE(InfoExtractor):
             'url': session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT',
             'data': json.dumps(session_response['data']),
             # interval, convert milliseconds to seconds, then halve to make a buffer.
-            'interval': float_or_none(session_api_data.get('heartbeatLifetime'), scale=2000),
+            'interval': float_or_none(session_api_data.get('heartbeatLifetime'), scale=3000),
+            'ping': ping
         }
 
         return info_dict, heartbeat_info_dict
@@ -400,7 +408,7 @@ class NiconicoIE(InfoExtractor):
         # Get HTML5 videos info
         quality_info = try_get(api_data, lambda x: x['media']['delivery']['movie'])
         if not quality_info:
-            raise ExtractorError('The video can\'t downloaded.', expected=True)
+            raise ExtractorError('The video can\'t be downloaded', expected=True)
 
         for audio_quality in quality_info.get('audios') or {}:
             for video_quality in quality_info.get('videos') or {}:
@@ -412,9 +420,7 @@ class NiconicoIE(InfoExtractor):
         # Get flv/swf info
         timestamp = None
         video_real_url = try_get(api_data, lambda x: x['video']['smileInfo']['url'])
-        if not video_real_url:
-            self.report_warning('Unable to obtain smile video information')
-        else:
+        if video_real_url:
             is_economy = video_real_url.endswith('low')
 
             if is_economy:
@@ -486,14 +492,12 @@ class NiconicoIE(InfoExtractor):
                     'filesize': filesize
                 })
 
-        if len(formats) == 0:
-            raise ExtractorError('Unable to find video info.')
-
         self._sort_formats(formats)
 
         # Start extracting information
         title = (
-            get_video_info_web(['originalTitle', 'title'])
+            get_video_info_xml('title')  # prefer to get the untranslated original title
+            or get_video_info_web(['originalTitle', 'title'])
             or self._og_search_title(webpage, default=None)
             or self._html_search_regex(
                 r'<span[^>]+class="videoHeaderTitle"[^>]*>([^<]+)</span>',
@@ -507,7 +511,9 @@ class NiconicoIE(InfoExtractor):
 
         thumbnail = (
             self._html_search_regex(r'<meta property="og:image" content="([^"]+)">', webpage, 'thumbnail data', default=None)
-            or get_video_info_web(['thumbnail_url', 'largeThumbnailURL', 'thumbnailURL'])
+            or dict_get(  # choose highest from 720p to 240p
+                get_video_info_web('thumbnail'),
+                ['ogp', 'player', 'largeUrl', 'middleUrl', 'url'])
             or self._html_search_meta('image', webpage, 'thumbnail', default=None)
             or video_detail.get('thumbnail'))
 
@@ -582,6 +588,7 @@ class NiconicoIE(InfoExtractor):
 
         return {
             'id': video_id,
+            '_api_data': api_data,
             'title': title,
             'formats': formats,
             'thumbnail': thumbnail,
@@ -616,24 +623,19 @@ class NiconicoPlaylistIE(InfoExtractor):
         'only_matching': True,
     }]
 
+    _API_HEADERS = {
+        'X-Frontend-ID': '6',
+        'X-Frontend-Version': '0'
+    }
+
     def _real_extract(self, url):
         list_id = self._match_id(url)
-        webpage = self._download_webpage(url, list_id)
-
-        header = self._parse_json(self._html_search_regex(
-            r'data-common-header="([^"]+)"', webpage,
-            'webpage header'), list_id)
-        frontendId = header.get('initConfig').get('frontendId')
-        frontendVersion = header.get('initConfig').get('frontendVersion')
 
         def get_page_data(pagenum, pagesize):
             return self._download_json(
                 'http://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
                 query={'page': 1 + pagenum, 'pageSize': pagesize},
-                headers={
-                    'X-Frontend-Id': frontendId,
-                    'X-Frontend-Version': frontendVersion,
-                }).get('data').get('mylist')
+                headers=self._API_HEADERS).get('data').get('mylist')
 
         data = get_page_data(0, 1)
         title = data.get('name')
@@ -669,20 +671,20 @@ class NiconicoUserIE(InfoExtractor):
         'playlist_mincount': 101,
     }
     _API_URL = "https://nvapi.nicovideo.jp/v1/users/%s/videos?sortKey=registeredAt&sortOrder=desc&pageSize=%s&page=%s"
-    _api_headers = {
-        'X-Frontend-ID': '6',
-        'X-Frontend-Version': '0',
-        'X-Niconico-Language': 'en-us'
-    }
     _PAGE_SIZE = 100
 
+    _API_HEADERS = {
+        'X-Frontend-ID': '6',
+        'X-Frontend-Version': '0'
+    }
+
     def _entries(self, list_id, ):
         total_count = 1
         count = page_num = 0
         while count < total_count:
             json_parsed = self._download_json(
                 self._API_URL % (list_id, self._PAGE_SIZE, page_num + 1), list_id,
-                headers=self._api_headers,
+                headers=self._API_HEADERS,
                 note='Downloading JSON metadata%s' % (' page %d' % page_num if page_num else ''))
             if not page_num:
                 total_count = int_or_none(json_parsed['data'].get('totalCount'))

yt_dlp/extractor/nitter.py

@@ -11,60 +11,100 @@ from ..utils import (
     determine_ext,
 )
 import re
+import random
 
 
 class NitterIE(InfoExtractor):
     # Taken from https://github.com/zedeus/nitter/wiki/Instances
-    INSTANCES = ('nitter.net',
-                 'nitter.snopyta.org',
-                 'nitter.42l.fr',
-                 'nitter.nixnet.services',
-                 'nitter.13ad.de',
-                 'nitter.pussthecat.org',
-                 'nitter.mastodont.cat',
-                 'nitter.dark.fail',
-                 'nitter.tedomum.net',
-                 'nitter.cattube.org',
-                 'nitter.fdn.fr',
-                 'nitter.1d4.us',
-                 'nitter.kavin.rocks',
-                 'tweet.lambda.dance',
-                 'nitter.cc',
-                 'nitter.weaponizedhumiliation.com',
-                 '3nzoldnxplag42gqjs23xvghtzf6t6yzssrtytnntc6ppc7xxuoneoad.onion',
-                 'nitter.l4qlywnpwqsluw65ts7md3khrivpirse744un3x7mlskqauz5pyuzgqd.onion',
-                 'nitterlgj3n5fgwesu3vxc5h67ruku33nqaoeoocae2mvlzhsu6k7fqd.onion')
+
+    NON_HTTP_INSTANCES = (
+        '3nzoldnxplag42gqjs23xvghtzf6t6yzssrtytnntc6ppc7xxuoneoad.onion',
+        'nitter.l4qlywnpwqsluw65ts7md3khrivpirse744un3x7mlskqauz5pyuzgqd.onion',
+        'nitter7bryz3jv7e3uekphigvmoyoem4al3fynerxkj22dmoxoq553qd.onion',
+        'npf37k3mtzwxreiw52ccs5ay4e6qt2fkcs2ndieurdyn2cuzzsfyfvid.onion',
+        'nitter.v6vgyqpa7yefkorazmg5d5fimstmvm2vtbirt6676mt7qmllrcnwycqd.onion',
+        'i23nv6w3juvzlw32xzoxcqzktegd4i4fu3nmnc2ewv4ggiu4ledwklad.onion',
+        '26oq3gioiwcmfojub37nz5gzbkdiqp7fue5kvye7d4txv4ny6fb4wwid.onion',
+
+        'nitter.i2p',
+        'u6ikd6zndl3c4dsdq4mmujpntgeevdk5qzkfb57r4tnfeccrn2qa.b32.i2p',
+
+        'nitterlgj3n5fgwesu3vxc5h67ruku33nqaoeoocae2mvlzhsu6k7fqd.onion',
+    )
+
+    HTTP_INSTANCES = (
+        'nitter.42l.fr',
+        'nitter.pussthecat.org',
+        'nitter.nixnet.services',
+        'nitter.mastodont.cat',
+        'nitter.tedomum.net',
+        'nitter.fdn.fr',
+        'nitter.1d4.us',
+        'nitter.kavin.rocks',
+        'tweet.lambda.dance',
+        'nitter.cc',
+        'nitter.vxempire.xyz',
+        'nitter.unixfox.eu',
+        'nitter.domain.glass',
+        'nitter.himiko.cloud',
+        'nitter.eu',
+        'nitter.namazso.eu',
+        'nitter.mailstation.de',
+        'nitter.actionsack.com',
+        'nitter.cattube.org',
+        'nitter.dark.fail',
+        'birdsite.xanny.family',
+        'nitter.40two.app',
+        'nitter.skrep.in',
+
+        # not in the list anymore
+        'nitter.snopyta.org',
+    )
+
+    DEAD_INSTANCES = (
+        # maintenance
+        'nitter.ethibox.fr',
+
+        # official, rate limited
+        'nitter.net',
+        # offline
+        'nitter.13ad.de',
+        'nitter.weaponizedhumiliation.com',
+    )
+
+    INSTANCES = NON_HTTP_INSTANCES + HTTP_INSTANCES + DEAD_INSTANCES
 
     _INSTANCES_RE = '(?:' + '|'.join([re.escape(instance) for instance in INSTANCES]) + ')'
     _VALID_URL = r'https?://%(instance)s/(?P<uploader_id>.+)/status/(?P<id>[0-9]+)(#.)?' % {'instance': _INSTANCES_RE}
-    current_instance = INSTANCES[0]  # the test and official instance
+    current_instance = random.choice(HTTP_INSTANCES)
+
     _TESTS = [
         {
             # GIF (wrapped in mp4)
-            'url': 'https://' + current_instance + '/firefox/status/1314279897502629888#m',
+            'url': 'https://%s/firefox/status/1314279897502629888#m' % current_instance,
             'info_dict': {
                 'id': '1314279897502629888',
                 'ext': 'mp4',
-                'title': 'Firefox 🔥 - You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension.   Report harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg  #UnfckTheInternet',
-                'description': 'You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension.   Report harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg  #UnfckTheInternet',
+                'title': 'Firefox 🔥 - You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension. \n\nReport harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg\n\n#UnfckTheInternet',
+                'description': 'You know the old saying, if you see something say something. Now you actually can with the YouTube regrets extension. \n\nReport harmful YouTube recommendations so others can avoid watching them. ➡️ https://mzl.la/3iFIiyg\n\n#UnfckTheInternet',
                 'thumbnail': r're:^https?://.*\.jpg$',
                 'uploader': 'Firefox 🔥',
                 'uploader_id': 'firefox',
-                'uploader_url': 'https://' + current_instance + '/firefox',
+                'uploader_url': 'https://%s/firefox' % current_instance,
                 'upload_date': '20201008',
                 'timestamp': 1602183720,
             },
         }, {  # normal video
-            'url': 'https://' + current_instance + '/Le___Doc/status/1299715685392756737#m',
+            'url': 'https://%s/Le___Doc/status/1299715685392756737#m' % current_instance,
             'info_dict': {
                 'id': '1299715685392756737',
                 'ext': 'mp4',
-                'title': 'Le Doc - "Je ne prédis jamais rien" D Raoult, Août 2020...',
-                'description': '"Je ne prédis jamais rien" D Raoult, Août 2020...',
+                'title': 'Le Doc - "Je ne prédis jamais rien"\nD Raoult, Août 2020...',
+                'description': '"Je ne prédis jamais rien"\nD Raoult, Août 2020...',
                 'thumbnail': r're:^https?://.*\.jpg$',
                 'uploader': 'Le Doc',
                 'uploader_id': 'Le___Doc',
-                'uploader_url': 'https://' + current_instance + '/Le___Doc',
+                'uploader_url': 'https://%s/Le___Doc' % current_instance,
                 'upload_date': '20200829',
                 'timestamp': 1598711341,
                 'view_count': int,
@@ -73,31 +113,51 @@ class NitterIE(InfoExtractor):
                 'comment_count': int,
             },
         }, {  # video embed in a "Streaming Political Ads" box
-            'url': 'https://' + current_instance + '/mozilla/status/1321147074491092994#m',
+            'url': 'https://%s/mozilla/status/1321147074491092994#m' % current_instance,
             'info_dict': {
                 'id': '1321147074491092994',
                 'ext': 'mp4',
-                'title': "Mozilla - Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?  This isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few.   Learn more ➡️ https://mzl.la/StreamingAds",
-                'description': "Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?  This isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few.   Learn more ➡️ https://mzl.la/StreamingAds",
+                'title': "Mozilla - Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?\n\nThis isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few. \n\nLearn more ➡️ https://mzl.la/StreamingAds",
+                'description': "Are you being targeted with weird, ominous or just plain annoying political ads while streaming your favorite shows?\n\nThis isn't a real political ad, but if you're watching streaming TV in the U.S., chances are you've seen quite a few. \n\nLearn more ➡️ https://mzl.la/StreamingAds",
                 'thumbnail': r're:^https?://.*\.jpg$',
                 'uploader': 'Mozilla',
                 'uploader_id': 'mozilla',
-                'uploader_url': 'https://' + current_instance + '/mozilla',
+                'uploader_url': 'https://%s/mozilla' % current_instance,
                 'upload_date': '20201027',
                 'timestamp': 1603820982
             },
-        },
+        }, {  # not the first tweet but main-tweet
+            'url': 'https://%s/TheNaturalNu/status/1379050895539724290#m' % current_instance,
+            'info_dict': {
+                'id': '1379050895539724290',
+                'ext': 'mp4',
+                'title': 'Dorothy Zbornak - This had me hollering!!',
+                'description': 'This had me hollering!!',
+                'thumbnail': r're:^https?://.*\.jpg$',
+                'uploader': 'Dorothy Zbornak',
+                'uploader_id': 'TheNaturalNu',
+                'uploader_url': 'https://%s/TheNaturalNu' % current_instance,
+                'timestamp': 1617626329,
+                'upload_date': '20210405'
+            }
+        }
     ]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
         parsed_url = compat_urlparse.urlparse(url)
-        base_url = parsed_url.scheme + '://' + parsed_url.netloc
+        base_url = '%s://%s' % (parsed_url.scheme, parsed_url.netloc)
 
         self._set_cookie(parsed_url.netloc, 'hlsPlayback', 'on')
-        webpage = self._download_webpage(url, video_id)
+        full_webpage = self._download_webpage(url, video_id)
 
-        video_url = base_url + self._html_search_regex(r'(?:<video[^>]+data-url|<source[^>]+src)="([^"]+)"', webpage, 'video url')
+        main_tweet_start = full_webpage.find('class="main-tweet"')
+        if main_tweet_start > 0:
+            webpage = full_webpage[main_tweet_start:]
+        if not webpage:
+            webpage = full_webpage
+
+        video_url = '%s%s' % (base_url, self._html_search_regex(r'(?:<video[^>]+data-url|<source[^>]+src)="([^"]+)"', webpage, 'video url'))
         ext = determine_ext(video_url)
 
         if ext == 'unknown_video':
@@ -108,33 +168,34 @@ class NitterIE(InfoExtractor):
                 'ext': ext
             }]
 
-        title = (
-            self._og_search_description(webpage).replace('\n', ' ')
-            or self._html_search_regex(r'<div class="tweet-content[^>]+>([^<]+)</div>', webpage, 'title'))
+        title = self._og_search_description(full_webpage)
+        if not title:
+            title = self._html_search_regex(r'<div class="tweet-content[^>]+>([^<]+)</div>', webpage, 'title')
         description = title
 
         mobj = re.match(self._VALID_URL, url)
         uploader_id = (
             mobj.group('uploader_id')
-            or self._html_search_regex(r'<a class="fullname"[^>]+title="([^"]+)"', webpage, 'uploader name', fatal=False))
+            or self._html_search_regex(r'<a class="fullname"[^>]+title="([^"]+)"', webpage, 'uploader name', fatal=False)
+        )
 
         if uploader_id:
-            uploader_url = base_url + '/' + uploader_id
+            uploader_url = '%s/%s' % (base_url, uploader_id)
 
         uploader = self._html_search_regex(r'<a class="fullname"[^>]+title="([^"]+)"', webpage, 'uploader name', fatal=False)
 
         if uploader:
-            title = uploader + ' - ' + title
+            title = '%s - %s' % (uploader, title)
 
         view_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-play[^>]*></span>\s([^<]+)</div>', webpage, 'view count', fatal=False))
         like_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-heart[^>]*></span>\s([^<]+)</div>', webpage, 'like count', fatal=False))
         repost_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-retweet[^>]*></span>\s([^<]+)</div>', webpage, 'repost count', fatal=False))
         comment_count = parse_count(self._html_search_regex(r'<span[^>]+class="icon-comment[^>]*></span>\s([^<]+)</div>', webpage, 'repost count', fatal=False))
 
-        thumbnail = base_url + (self._html_search_meta('og:image', webpage, 'thumbnail url')
-                                or self._html_search_regex(r'<video[^>]+poster="([^"]+)"', webpage, 'thumbnail url', fatal=False))
-
-        thumbnail = remove_end(thumbnail, '%3Asmall')  # if parsed with regex, it should contain this
+        thumbnail = self._html_search_meta('og:image', full_webpage, 'thumbnail url')
+        if not thumbnail:
+            thumbnail = '%s%s' % (base_url, self._html_search_regex(r'<video[^>]+poster="([^"]+)"', webpage, 'thumbnail url', fatal=False))
+            thumbnail = remove_end(thumbnail, '%3Asmall')
 
         thumbnails = []
         thumbnail_ids = ('thumb', 'small', 'large', 'medium', 'orig')

yt_dlp/extractor/npo.py

@@ -247,7 +247,7 @@ class NPOIE(NPOBaseIE):
 
         if not formats:
             if not self._downloader.params.get('allow_unplayable_formats') and drm:
-                raise ExtractorError('This video is DRM protected.', expected=True)
+                self.raise_no_formats('This video is DRM protected.', expected=True)
             return
 
         self._sort_formats(formats)

yt_dlp/extractor/nytimes.py

@@ -46,6 +46,7 @@ class NYTimesBaseIE(InfoExtractor):
 
         urls = []
         formats = []
+        subtitles = {}
         for video in video_data.get('renditions', []):
             video_url = video.get('url')
             format_id = video.get('type')
@@ -54,9 +55,11 @@ class NYTimesBaseIE(InfoExtractor):
             urls.append(video_url)
             ext = mimetype2ext(video.get('mimetype')) or determine_ext(video_url)
             if ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
+                m3u8_fmts, m3u8_subs = self._extract_m3u8_formats_and_subtitles(
                     video_url, video_id, 'mp4', 'm3u8_native',
-                    m3u8_id=format_id or 'hls', fatal=False))
+                    m3u8_id=format_id or 'hls', fatal=False)
+                formats.extend(m3u8_fmts)
+                subtitles = self._merge_subtitles(subtitles, m3u8_subs)
             elif ext == 'mpd':
                 continue
             #     formats.extend(self._extract_mpd_formats(
@@ -96,6 +99,7 @@ class NYTimesBaseIE(InfoExtractor):
             'uploader': video_data.get('byline'),
             'duration': float_or_none(video_data.get('duration'), 1000),
             'formats': formats,
+            'subtitles': subtitles,
             'thumbnails': thumbnails,
         }
 

yt_dlp/extractor/odnoklassniki.py

@@ -260,7 +260,7 @@ class OdnoklassnikiIE(InfoExtractor):
         if not formats:
             payment_info = metadata.get('paymentInfo')
             if payment_info:
-                raise ExtractorError('This video is paid, subscribe to download it', expected=True)
+                self.raise_no_formats('This video is paid, subscribe to download it', expected=True)
 
         self._sort_formats(formats)
 

yt_dlp/extractor/ooyala.py

@@ -10,7 +10,6 @@ from ..compat import (
 )
 from ..utils import (
     determine_ext,
-    ExtractorError,
     float_or_none,
     int_or_none,
     try_get,
@@ -85,7 +84,7 @@ class OoyalaBaseIE(InfoExtractor):
                     'fps': float_or_none(stream.get('framerate')),
                 })
         if not formats and not auth_data.get('authorized'):
-            raise ExtractorError('%s said: %s' % (
+            self.raise_no_formats('%s said: %s' % (
                 self.IE_NAME, auth_data['message']), expected=True)
         self._sort_formats(formats)
 

yt_dlp/extractor/philharmoniedeparis.py

@@ -79,7 +79,7 @@ class PhilharmonieDeParisIE(InfoExtractor):
                 formats.extend(self._extract_m3u8_formats(
                     m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',
                     m3u8_id='hls', fatal=False))
-            if not formats:
+            if not formats and not self._downloader.params.get('ignore_no_formats'):
                 return
             self._sort_formats(formats)
             return {

yt_dlp/extractor/pluralsight.py

@@ -393,7 +393,7 @@ query viewClip {
                 # To somewhat reduce the probability of these consequences
                 # we will sleep random amount of time before each call to ViewClip.
                 self._sleep(
-                    random.randint(2, 5), display_id,
+                    random.randint(5, 10), display_id,
                     '%(video_id)s: Waiting for %(timeout)s seconds to avoid throttling')
 
                 if not viewclip:

yt_dlp/extractor/plutotv.py

@@ -78,45 +78,60 @@ class PlutoTVIE(InfoExtractor):
         },
     ]
 
-    def _to_ad_free_formats(self, video_id, formats):
-        ad_free_formats = []
-        m3u8_urls = set()
-        for format in formats:
+    def _to_ad_free_formats(self, video_id, formats, subtitles):
+        ad_free_formats, ad_free_subtitles, m3u8_urls = [], {}, set()
+        for fmt in formats:
             res = self._download_webpage(
-                format.get('url'), video_id, note='Downloading m3u8 playlist',
+                fmt.get('url'), video_id, note='Downloading m3u8 playlist',
                 fatal=False)
             if not res:
                 continue
             first_segment_url = re.search(
                 r'^(https?://.*/)0\-(end|[0-9]+)/[^/]+\.ts$', res,
                 re.MULTILINE)
-            if not first_segment_url:
+            if first_segment_url:
+                m3u8_urls.add(
+                    compat_urlparse.urljoin(first_segment_url.group(1), '0-end/master.m3u8'))
+                continue
+            first_segment_url = re.search(
+                r'^(https?://.*/).+\-0+\.ts$', res,
+                re.MULTILINE)
+            if first_segment_url:
+                m3u8_urls.add(
+                    compat_urlparse.urljoin(first_segment_url.group(1), 'master.m3u8'))
                 continue
-            m3u8_urls.add(
-                compat_urlparse.urljoin(first_segment_url.group(1), '0-end/master.m3u8'))
 
         for m3u8_url in m3u8_urls:
-            ad_free_formats.extend(
-                self._extract_m3u8_formats(
-                    m3u8_url, video_id, 'mp4', 'm3u8_native',
-                    m3u8_id='hls', fatal=False))
-        self._sort_formats(ad_free_formats)
-        return ad_free_formats
+            fmts, subs = self._extract_m3u8_formats_and_subtitles(
+                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False)
+            ad_free_formats.extend(fmts)
+            ad_free_subtitles = self._merge_subtitles(ad_free_subtitles, subs)
+        if ad_free_formats:
+            formats, subtitles = ad_free_formats, ad_free_subtitles
+        else:
+            self._downloader.report_warning('Unable to find ad-free formats')
+        return formats, subtitles
 
     def _get_video_info(self, video_json, slug, series_name=None):
         video_id = video_json.get('_id', slug)
-        formats = []
+        formats, subtitles = [], {}
         for video_url in try_get(video_json, lambda x: x['stitched']['urls'], list) or []:
             if video_url.get('type') != 'hls':
                 continue
             url = url_or_none(video_url.get('url'))
-            formats.extend(
-                self._extract_m3u8_formats(
-                    url, video_id, 'mp4', 'm3u8_native',
-                    m3u8_id='hls', fatal=False))
+
+            fmts, subs = self._extract_m3u8_formats_and_subtitles(
+                url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False)
+            formats.extend(fmts)
+            subtitles = self._merge_subtitles(subtitles, subs)
+
+        formats, subtitles = self._to_ad_free_formats(video_id, formats, subtitles)
+        self._sort_formats(formats)
+
         info = {
             'id': video_id,
-            'formats': self._to_ad_free_formats(video_id, formats),
+            'formats': formats,
+            'subtitles': subtitles,
             'title': video_json.get('name'),
             'description': video_json.get('description'),
             'duration': float_or_none(video_json.get('duration'), scale=1000),

yt_dlp/extractor/pornhub.py

@@ -398,6 +398,16 @@ class PornHubIE(PornHubBaseIE):
         formats = []
 
         def add_format(format_url, height=None):
+            ext = determine_ext(format_url)
+            if ext == 'mpd':
+                formats.extend(self._extract_mpd_formats(
+                    format_url, video_id, mpd_id='dash', fatal=False))
+                return
+            if ext == 'm3u8':
+                formats.extend(self._extract_m3u8_formats(
+                    format_url, video_id, 'mp4', entry_protocol='m3u8_native',
+                    m3u8_id='hls', fatal=False))
+                return
             tbr = None
             mobj = re.search(r'(?P<height>\d+)[pP]?_(?P<tbr>\d+)[kK]', format_url)
             if mobj:
@@ -417,16 +427,6 @@ class PornHubIE(PornHubBaseIE):
                     r'/(\d{6}/\d{2})/', video_url, 'upload data', default=None)
                 if upload_date:
                     upload_date = upload_date.replace('/', '')
-            ext = determine_ext(video_url)
-            if ext == 'mpd':
-                formats.extend(self._extract_mpd_formats(
-                    video_url, video_id, mpd_id='dash', fatal=False))
-                continue
-            elif ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
-                    video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                    m3u8_id='hls', fatal=False))
-                continue
             if '/video/get_media' in video_url:
                 medias = self._download_json(video_url, video_id, fatal=False)
                 if isinstance(medias, list):