Release 2021.06.23

Update to ytdl-commit-379f52a
[liveleak] Remove extractor 379f52a495
2026-01-13 10:21:30 +00:00 · 2021-06-23 07:34:55 +05:30 · 2021-06-23 07:34:55 +05:30 · 2021-06-23 07:27:53 +05:30 · 2021-06-23 06:04:42 +05:30 · 2021-06-23 05:42:39 +05:30
59 changed files with 1432 additions and 991 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -1 +1,4 @@
+* text=auto
+
 Makefile* text whitespace=-tab-in-indent
+*.sh text eol=lf
--- a/.github/ISSUE_TEMPLATE/1_broken_site.md
+++ b/.github/ISSUE_TEMPLATE/1_broken_site.md
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.06.01
+ [debug] yt-dlp version 2021.06.09
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.github/ISSUE_TEMPLATE/2_site_support_request.md
+++ b/.github/ISSUE_TEMPLATE/2_site_support_request.md
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
--- a/.github/ISSUE_TEMPLATE/3_site_feature_request.md
+++ b/.github/ISSUE_TEMPLATE/3_site_feature_request.md
@@ -21,13 +21,13 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

 - [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones


--- a/.github/ISSUE_TEMPLATE/4_bug_report.md
+++ b/.github/ISSUE_TEMPLATE/4_bug_report.md
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.06.01
+ [debug] yt-dlp version 2021.06.09
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.github/ISSUE_TEMPLATE/5_feature_request.md
+++ b/.github/ISSUE_TEMPLATE/5_feature_request.md
@@ -21,13 +21,13 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

 - [ ] I'm reporting a feature request
- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones


--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -95,14 +95,15 @@ jobs:

    steps:
    - uses: actions/checkout@v2
-    - name: Set up Python
+    # 3.8 is used for Win7 support
+    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
          python-version: '3.8'
    - name: Upgrade pip and enable wheel support
      run: python -m pip install --upgrade pip setuptools wheel
    - name: Install Requirements
-      run: pip install pyinstaller mutagen pycryptodome
+      run: pip install pyinstaller mutagen pycryptodome websockets
    - name: Bump version
      id: bump_version
      run: python devscripts/update-version.py
@@ -137,15 +138,16 @@ jobs:

    steps:
    - uses: actions/checkout@v2
-    - name: Set up Python 3.4.4 32-Bit
+    # 3.7 is used for Vista support. See https://github.com/yt-dlp/yt-dlp/issues/390
+    - name: Set up Python 3.7 32-Bit
      uses: actions/setup-python@v2
      with:
-          python-version: '3.4.4'
+          python-version: '3.7'
          architecture: 'x86'
    - name: Upgrade pip and enable wheel support
-      run: python -m pip install pip==19.1.1 setuptools==43.0.0 wheel==0.33.6
-    - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4 pefile==2019.4.18
+      run: python -m pip install --upgrade pip setuptools wheel
+    - name: Install Requirements
+      run: pip install pyinstaller mutagen pycryptodome websockets
    - name: Bump version
      id: bump_version
      run: python devscripts/update-version.py
--- a/.github/workflows/core.yml
+++ b/.github/workflows/core.yml
@@ -9,11 +9,13 @@ jobs:
      fail-fast: false
      matrix:
        os: [ubuntu-18.04]
-        python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
+        # py3.9 is in quick-test
+        python-version: [3.7, 3.8, pypy-3.6, pypy-3.7]
        run-tests-ext: [sh]
        include:
+        # atleast one of the tests must be in windows
        - os: windows-latest
-          python-version: 3.4  # Windows x86 build is still in 3.4
+          python-version: 3.6
          run-tests-ext: bat
    steps:
    - uses: actions/checkout@v2
--- a/.github/workflows/download.yml
+++ b/.github/workflows/download.yml
@@ -9,11 +9,11 @@ jobs:
      fail-fast: true
      matrix:
        os: [ubuntu-18.04]
-        python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
+        python-version: [3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
        run-tests-ext: [sh]
        include:
        - os: windows-latest
-          python-version: 3.4  # Windows x86 build is still in 3.4
+          python-version: 3.6
          run-tests-ext: bat
    steps:
    - uses: actions/checkout@v2
--- a/Changelog.md
+++ b/Changelog.md
@@ -19,15 +19,54 @@
 -->


+### 2021.06.23
+
+* Merge youtube-dl: Upto [commit/379f52a](https://github.com/ytdl-org/youtube-dl/commit/379f52a4954013767219d25099cce9e0f9401961)
+* **Add option `--throttled-rate`** below which video data is re-extracted
+* [fragment] **Merge during download for `-N`**, and refactor `hls`/`dash`
+* [websockets] Add `WebSocketFragmentFD`by [nao20010128nao](https://github.com/nao20010128nao), [pukkandan](https://github.com/pukkandan)
+* Allow `images` formats in addition to video/audio
+* [downloader/mhtml] Add new downloader for slideshows/storyboards by [fstirlitz](https://github.com/fstirlitz)
+* [youtube] Temporary **fix for age-gate**
+* [youtube] Support ongoing live chat by [siikamiika](https://github.com/siikamiika)
+* [youtube] Improve SAPISID cookie handling by [colethedj](https://github.com/colethedj)
+* [youtube] Login is not needed for `:ytrec`
+* [youtube] Non-fatal alert reporting for unavailable videos page by [colethedj](https://github.com/colethedj)
+* [twitcasting] Websocket support by [nao20010128nao](https://github.com/nao20010128nao)
+* [mediasite] Extract slides by [fstirlitz](https://github.com/fstirlitz)
+* [funimation] Extract subtitles 
+* [pornhub] Extract `cast`
+* [hotstar] Use server time for authentication instead of local time
+* [EmbedThumbnail] Fix for already downloaded thumbnail
+* [EmbedThumbnail] Add compat-option `embed-thumbnail-atomicparsley`
+* Expand `--check-formats` to thumbnails
+* Fix id sanitization in filenames
+* Skip fixup of existing files and add `--fixup force` to force it
+* Better error handling of syntax errors in `-f`
+* Use `NamedTemporaryFile` for `--check-formats`
+* [aria2c] Lower `--min-split-size` for HTTP downloads
+* [options] Rename `--add-metadata` to `--embed-metadata`
+* [utils] Improve `LazyList` and add tests
+* [build] Build Windows x86 version with py3.7 and remove redundant tests by [pukkandan](https://github.com/pukkandan), [shirt](https://github.com/shirt-dev)
+* [docs] Clarify that `--embed-metadata` embeds chapter markers
+* [cleanup] Refactor fixup
+
+
+### 2021.06.09
+
+* Fix bug where `%(field)d` in filename template throws error
+* Improve offset parsing in outtmpl
+* [test] More rigorous tests for `prepare_filename`
+
 ### 2021.06.08

 * Remove support for obsolete Python versions: Only 3.6+ is now supported
 * Merge youtube-dl: Upto [commit/c2350ca](https://github.com/ytdl-org/youtube-dl/commit/c2350cac243ba1ec1586fe85b0d62d1b700047a2)
 * [hls] Fix decryption for multithreaded downloader
 * [extractor] Fix pre-checking archive for some extractors
-* [extractor] Fix FourCC fallback when parsing ISM [fstirlitz](https://github.com/fstirlitz)
-* [twitcasting] Add TwitCastingUserIE, TwitCastingLiveIE [pukkandan](https://github.com/pukkandan), [nao20010128nao](https://github.com/nao20010128nao)
-* [vidio] Add VidioPremierIE and VidioLiveIE [minEplaYerspe](Https://github.com/MinePlayersPE)
+* [extractor] Fix FourCC fallback when parsing ISM by [fstirlitz](https://github.com/fstirlitz)
+* [twitcasting] Add TwitCastingUserIE, TwitCastingLiveIE by [pukkandan](https://github.com/pukkandan), [nao20010128nao](https://github.com/nao20010128nao)
+* [vidio] Add VidioPremierIE and VidioLiveIE by [MinePlayersPE](Https://github.com/MinePlayersPE)
 * [viki] Fix extraction from [ytdl-org/youtube-dl@59e583f](https://github.com/ytdl-org/youtube-dl/commit/59e583f7e8530ca92776c866897d895c072e2a82)
 * [youtube] Support shorts URL
 * [zoom] Extract transcripts as subtitles
@@ -35,13 +74,13 @@
 * Fix and refactor `prepare_outtmpl`
 * Make more fields available for `--print` when used with `--flat-playlist`
 * [utils] Generalize `traverse_dict` to `traverse_obj`
-* [downloader/ffmpeg] Hide FFmpeg banner unless in verbose mode [fstirlitz](https://github.com/fstirlitz)
+* [downloader/ffmpeg] Hide FFmpeg banner unless in verbose mode by [fstirlitz](https://github.com/fstirlitz)
 * [build] Release `yt-dlp.tar.gz`
-* [build,update] Add GNU-style SHA512 and prepare updater for simlar SHA256 [nihil-admirari](https://github.com/nihil-admirari)
-* [pyinst] Show Python version in exe metadata [nihil-admirari](https://github.com/nihil-admirari)
+* [build,update] Add GNU-style SHA512 and prepare updater for simlar SHA256 by [nihil-admirari](https://github.com/nihil-admirari)
+* [pyinst] Show Python version in exe metadata by [nihil-admirari](https://github.com/nihil-admirari)
 * [docs] Improve documentation of dependencies
 * [cleanup] Mark unused files
-* [cleanup] Point all shebang to `python3` [fstirlitz](https://github.com/fstirlitz)
+* [cleanup] Point all shebang to `python3` by [fstirlitz](https://github.com/fstirlitz)
 * [cleanup] Remove duplicate file `trovolive.py`


--- a/README.md
+++ b/README.md
@@ -66,7 +66,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/

 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))

-* **Merged with youtube-dl [commit/c2350ca](https://github.com/ytdl-org/youtube-dl/commit/c2350cac243ba1ec1586fe85b0d62d1b700047a2)**: (v2021.06.06) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl [commit/379f52a](https://github.com/ytdl-org/youtube-dl/commit/379f52a4954013767219d25099cce9e0f9401961)**: (v2021.06.06) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)

 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--write-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.

@@ -131,6 +131,7 @@ Some of yt-dlp's default options are different from that of youtube-dl and youtu
 * Youtube channel URLs are automatically redirected to `/video`. Append a `/featured` to the URL to download only the videos in the home page. If the channel does not have a videos tab, we try to download the equivalent `UU` playlist instead. Also, `/live` URLs raise an error if there are no live videos instead of silently downloading the entire channel. You may use `--compat-options no-youtube-channel-redirect` to revert all these redirections
 * Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
 * If `ffmpeg` is used as the downloader, the downloading and merging of formats happen in a single step when possible. Use `--compat-options no-direct-merge` to revert this
+* Thumbnail embedding in `mp4` is done with mutagen if possible. Use `--compat-options embed-thumbnail-atomicparsley` to force the use of AtomicParsley instead

 For ease of use, a few more compat options are available:
 * `--compat-options all`: Use all compat options
@@ -181,6 +182,7 @@ While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly
 * [**sponskrub**](https://github.com/faissaloo/SponSkrub) - For using the [sponskrub options](#sponskrub-sponsorblock-options). Licenced under [GPLv3+](https://github.com/faissaloo/SponSkrub/blob/master/LICENCE.md)
 * [**mutagen**](https://github.com/quodlibet/mutagen) - For embedding thumbnail in certain formats. Licenced under [GPLv2+](https://github.com/quodlibet/mutagen/blob/master/COPYING)
 * [**pycryptodome**](https://github.com/Legrandin/pycryptodome) - For decrypting various data. Licenced under [BSD2](https://github.com/Legrandin/pycryptodome/blob/master/LICENSE.rst)
+* [**websockets**](https://github.com/aaugustin/websockets) - For downloading over websocket. Licenced under [BSD3](https://github.com/aaugustin/websockets/blob/main/LICENSE)
 * [**AtomicParsley**](https://github.com/wez/atomicparsley) - For embedding thumbnail in mp4/m4a if mutagen is not present. Licenced under [GPLv2+](https://github.com/wez/atomicparsley/blob/master/COPYING)
 * [**rtmpdump**](http://rtmpdump.mplayerhq.hu) - For downloading `rtmp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](http://rtmpdump.mplayerhq.hu)
 * [**mplayer**](http://mplayerhq.hu/design7/info.html) or [**mpv**](https://mpv.io) - For downloading `rstp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](https://github.com/mpv-player/mpv/blob/master/Copyright)
@@ -189,14 +191,14 @@ While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly

 To use or redistribute the dependencies, you must agree to their respective licensing terms.

-Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
+Note that the windows releases are already built with the python interpreter, mutagen, pycryptodome and websockets included.

 ### COMPILE

 **For Windows**:
-To build the Windows executable, you must have pyinstaller (and optionally mutagen and pycryptodome)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen, pycryptodome, websockets)

-    python3 -m pip install --upgrade pyinstaller mutagen pycryptodome
+    python3 -m pip install --upgrade pyinstaller mutagen pycryptodome websockets

 Once you have all the necessary dependencies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it.

@@ -372,6 +374,9 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     (default is 1)
    -r, --limit-rate RATE            Maximum download rate in bytes per second
                                     (e.g. 50K or 4.2M)
+    --throttled-rate RATE            Minimum download rate in bytes per second
+                                     below which throttling is assumed and the
+                                     video data is re-extracted (e.g. 100K)
    -R, --retries RETRIES            Number of retries (default is 10), or
                                     "infinite"
    --fragment-retries RETRIES       Number of retries for a fragment (default
@@ -710,7 +715,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     Metadata, EmbedSubtitle, EmbedThumbnail,
                                     SubtitlesConvertor, ThumbnailsConvertor,
                                     VideoRemuxer, VideoConvertor, SponSkrub,
-                                     FixupStretched, FixupM4a and FixupM3u8. The
+                                     FixupStretched, FixupM4a, FixupM3u8,
+                                     FixupTimestamp and FixupDuration. The
                                     supported executables are: AtomicParsley,
                                     FFmpeg, FFprobe, and SponSkrub. You can
                                     also specify "PP+EXE:ARGS" to give the
@@ -734,10 +740,13 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
    --embed-subs                     Embed subtitles in the video (only for mp4,
                                     webm and mkv videos)
    --no-embed-subs                  Do not embed subtitles (default)
-    --embed-thumbnail                Embed thumbnail in the audio as cover art
+    --embed-thumbnail                Embed thumbnail in the video as cover art
    --no-embed-thumbnail             Do not embed thumbnail (default)
-    --add-metadata                   Write metadata to the video file
-    --no-add-metadata                Do not write metadata (default)
+    --embed-metadata                 Embed metadata including chapter markers
+                                     (if supported by the format) to the video
+                                     file (Alias: --add-metadata)
+    --no-embed-metadata              Do not write metadata (default)
+                                     (Alias: --no-add-metadata)
    --parse-metadata FROM:TO         Parse additional metadata like title/artist
                                     from other fields; see "MODIFYING METADATA"
                                     for details
@@ -747,7 +756,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                     file. One of never (do nothing), warn (only
                                     emit a warning), detect_or_warn (the
                                     default; fix file if we can, warn
-                                     otherwise)
+                                     otherwise), force (try fixing even if file
+                                     already exists
    --ffmpeg-location PATH           Location of the ffmpeg binary; either the
                                     path to the binary or its containing
                                     directory
@@ -1140,7 +1150,7 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
 - `lang`: Language preference as given by the extractor
 - `quality`: The quality of the format as given by the extractor
 - `source`: Preference of the source as given by the extractor
- - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8_native` > `m3u8` > `http_dash_segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
+ - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8_native`/`m3u8` > `http_dash_segments`> `websocket_frag` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
 - `vcodec`: Video Codec (`av01` > `vp9.2` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
 - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
 - `codec`: Equivalent to `vcodec,acodec`
--- a/pyinst.py
+++ b/pyinst.py
@@ -6,6 +6,7 @@ import sys
 # import os
 import platform

+from PyInstaller.utils.hooks import collect_submodules
 from PyInstaller.utils.win32.versioninfo import (
    VarStruct, VarFileInfo, StringStruct, StringTable,
    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
@@ -66,16 +67,15 @@ VERSION_FILE = VSVersionInfo(
    ]
 )

+dependancies = ['Crypto', 'mutagen'] + collect_submodules('websockets')
+excluded_modules = ['test', 'ytdlp_plugins', 'youtube-dl', 'youtube-dlc']
+
 PyInstaller.__main__.run([
    '--name=yt-dlp%s' % _x86,
    '--onefile',
    '--icon=devscripts/cloud.ico',
-    '--exclude-module=youtube_dl',
-    '--exclude-module=youtube_dlc',
-    '--exclude-module=test',
-    '--exclude-module=ytdlp_plugins',
-    '--hidden-import=mutagen',
-    '--hidden-import=Crypto',
+    *[f'--exclude-module={module}' for module in excluded_modules],
+    *[f'--hidden-import={module}' for module in dependancies],
    '--upx-exclude=vcruntime140.dll',
    'yt_dlp/__main__.py',
 ])
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,3 @@
 mutagen
 pycryptodome
+websockets
--- a/setup.py
+++ b/setup.py
@@ -19,7 +19,7 @@ LONG_DESCRIPTION = '\n\n'.join((
    '**PS**: Some links in this document will not work since this is a copy of the README.md from Github',
    open('README.md', 'r', encoding='utf-8').read()))

-REQUIREMENTS = ['mutagen', 'pycryptodome']
+REQUIREMENTS = ['mutagen', 'pycryptodome', 'websockets']

 if sys.argv[1:2] == ['py2exe']:
    raise NotImplementedError('py2exe is not currently supported; instead, use "pyinst.py" to build with pyinstaller')
--- a/supportedsites.md
+++ b/supportedsites.md
@@ -225,8 +225,7 @@
 - **Culturebox**
 - **CultureUnplugged**
 - **curiositystream**
- - **curiositystream:collections**
- - **curiositystream:series**
+ - **curiositystream:collection**
 - **CWTV**
 - **DagelijkseKost**: dagelijksekost.een.be
 - **DailyMail**
@@ -497,8 +496,6 @@
 - **LinuxAcademy**
 - **LiTV**
 - **LiveJournal**
- - **LiveLeak**
- - **LiveLeakEmbed**
 - **livestream**
 - **livestream:original**
 - **LnkGo**
--- a/test/test_YoutubeDL.py
+++ b/test/test_YoutubeDL.py
@@ -17,7 +17,7 @@ from yt_dlp.compat import compat_str, compat_urllib_error
 from yt_dlp.extractor import YoutubeIE
 from yt_dlp.extractor.common import InfoExtractor
 from yt_dlp.postprocessor.common import PostProcessor
-from yt_dlp.utils import ExtractorError, float_or_none, match_filter_func
+from yt_dlp.utils import ExtractorError, int_or_none, match_filter_func

 TEST_URL = 'http://localhost/sample.mp4'

@@ -461,14 +461,13 @@ class TestFormatSelection(unittest.TestCase):

    def test_invalid_format_specs(self):
        def assert_syntax_error(format_spec):
-            ydl = YDL({'format': format_spec})
-            info_dict = _make_result([{'format_id': 'foo', 'url': TEST_URL}])
-            self.assertRaises(SyntaxError, ydl.process_ie_result, info_dict)
+            self.assertRaises(SyntaxError, YDL, {'format': format_spec})

        assert_syntax_error('bestvideo,,best')
        assert_syntax_error('+bestaudio')
        assert_syntax_error('bestvideo+')
        assert_syntax_error('/')
+        assert_syntax_error('[720<height]')

    def test_format_filtering(self):
        formats = [
@@ -664,92 +663,115 @@ class TestYoutubeDL(unittest.TestCase):
        'formats': [{'id': 'id1'}, {'id': 'id2'}, {'id': 'id3'}]
    }

-    def test_prepare_outtmpl(self):
-        def out(tmpl, **params):
+    def test_prepare_outtmpl_and_filename(self):
+        def test(tmpl, expected, *, info=None, **params):
            params['outtmpl'] = tmpl
            ydl = YoutubeDL(params)
            ydl._num_downloads = 1
-            err = ydl.validate_outtmpl(tmpl)
-            if err:
-                raise err
-            outtmpl, tmpl_dict = ydl.prepare_outtmpl(tmpl, self.outtmpl_info)
-            return outtmpl % tmpl_dict
+            self.assertEqual(ydl.validate_outtmpl(tmpl), None)

-        self.assertEqual(out('%(id)s.%(ext)s'), '1234.mp4')
-        self.assertEqual(out('%(duration_string)s'), '27:46:40')
-        self.assertTrue(float_or_none(out('%(epoch)d')))
-        self.assertEqual(out('%(resolution)s'), '1080p')
-        self.assertEqual(out('%(playlist_index)s'), '001')
-        self.assertEqual(out('%(autonumber)s'), '00001')
-        self.assertEqual(out('%(autonumber+2)03d', autonumber_start=3), '005')
-        self.assertEqual(out('%(autonumber)s', autonumber_size=3), '001')
+            outtmpl, tmpl_dict = ydl.prepare_outtmpl(tmpl, info or self.outtmpl_info)
+            out = outtmpl % tmpl_dict
+            fname = ydl.prepare_filename(info or self.outtmpl_info)

-        self.assertEqual(out('%%'), '%')
-        self.assertEqual(out('%%%%'), '%%')
-        self.assertEqual(out('%(invalid@tmpl|def)s', outtmpl_na_placeholder='none'), 'none')
-        self.assertEqual(out('%()s'), 'NA')
-        self.assertEqual(out('%s'), '%s')
-        self.assertEqual(out('%d'), '%d')
-        self.assertRaises(ValueError, out, '%')
-        self.assertRaises(ValueError, out, '%(title)')
+            if callable(expected):
+                self.assertTrue(expected(out))
+                self.assertTrue(expected(fname))
+            elif isinstance(expected, compat_str):
+                self.assertEqual((out, fname), (expected, expected))
+            else:
+                self.assertEqual((out, fname), expected)

+        # Auto-generated fields
+        test('%(id)s.%(ext)s', '1234.mp4')
+        test('%(duration_string)s', ('27:46:40', '27-46-40'))
+        test('%(epoch)d', int_or_none)
+        test('%(resolution)s', '1080p')
+        test('%(playlist_index)s', '001')
+        test('%(autonumber)s', '00001')
+        test('%(autonumber+2)03d', '005', autonumber_start=3)
+        test('%(autonumber)s', '001', autonumber_size=3)
+
+        # Escaping %
+        test('%%', '%')
+        test('%%%%', '%%')
+        test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
+        test('%(width)06d.%(ext)s', 'NA.mp4')
+        test('%(width)06d.%%(ext)s', 'NA.%(ext)s')
+        test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
+
+        # ID sanitization
+        test('%(id)s', '_abcd', info={'id': '_abcd'})
+        test('%(some_id)s', '_abcd', info={'some_id': '_abcd'})
+        test('%(formats.0.id)s', '_abcd', info={'formats': [{'id': '_abcd'}]})
+        test('%(id)s', '-abcd', info={'id': '-abcd'})
+        test('%(id)s', '.abcd', info={'id': '.abcd'})
+        test('%(id)s', 'ab__cd', info={'id': 'ab__cd'})
+        test('%(id)s', ('ab:cd', 'ab -cd'), info={'id': 'ab:cd'})
+
+        # Invalid templates
+        self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%'), ValueError))
+        self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%(title)'), ValueError))
+        test('%(invalid@tmpl|def)s', 'none', outtmpl_na_placeholder='none')
+        test('%()s', 'NA')
+        test('%s', '%s')
+        test('%d', '%d')
+
+        # NA placeholder
        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(x|def)s-%(id)s.%(ext)s'
-        self.assertEqual(out(NA_TEST_OUTTMPL), 'NA-NA-def-1234.mp4')
-        self.assertEqual(out(NA_TEST_OUTTMPL, outtmpl_na_placeholder='none'), 'none-none-def-1234.mp4')
-        self.assertEqual(out(NA_TEST_OUTTMPL, outtmpl_na_placeholder=''), '--def-1234.mp4')
+        test(NA_TEST_OUTTMPL, 'NA-NA-def-1234.mp4')
+        test(NA_TEST_OUTTMPL, 'none-none-def-1234.mp4', outtmpl_na_placeholder='none')
+        test(NA_TEST_OUTTMPL, '--def-1234.mp4', outtmpl_na_placeholder='')

+        # String formatting
        FMT_TEST_OUTTMPL = '%%(height)%s.%%(ext)s'
-        self.assertEqual(out(FMT_TEST_OUTTMPL % 's'), '1080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % 'd'), '1080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '6d'), '  1080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '-6d'), '1080  .mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '06d'), '001080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % ' 06d'), ' 01080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '   06d'), ' 01080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '0 6d'), ' 01080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '0   6d'), ' 01080.mp4')
-        self.assertEqual(out(FMT_TEST_OUTTMPL % '   0   6d'), ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % 's', '1080.mp4')
+        test(FMT_TEST_OUTTMPL % 'd', '1080.mp4')
+        test(FMT_TEST_OUTTMPL % '6d', '  1080.mp4')
+        test(FMT_TEST_OUTTMPL % '-6d', '1080  .mp4')
+        test(FMT_TEST_OUTTMPL % '06d', '001080.mp4')
+        test(FMT_TEST_OUTTMPL % ' 06d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '   06d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '0 6d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '0   6d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '   0   6d', ' 01080.mp4')

-        self.assertEqual(out('%(id)d'), '1234')
-        self.assertEqual(out('%(height)c'), '1')
-        self.assertEqual(out('%(ext)c'), 'm')
-        self.assertEqual(out('%(id)d %(id)r'), "1234 '1234'")
-        self.assertEqual(out('%(ext)s-%(ext|def)d'), 'mp4-def')
-        self.assertEqual(out('%(width|0)04d'), '0000')
-        self.assertEqual(out('%(width|)d', outtmpl_na_placeholder='none'), '')
+        # Type casting
+        test('%(id)d', '1234')
+        test('%(height)c', '1')
+        test('%(ext)c', 'm')
+        test('%(id)d %(id)r', "1234 '1234'")
+        test('%(id)r %(height)r', "'1234' 1080")
+        test('%(ext)s-%(ext|def)d', 'mp4-def')
+        test('%(width|0)04d', '0000')
+        test('a%(width|)d', 'a', outtmpl_na_placeholder='none')

+        # Internal formatting
        FORMATS = self.outtmpl_info['formats']
-        self.assertEqual(out('%(timestamp+-1000>%H-%M-%S)s'), '11-43-20')
-        self.assertEqual(out('%(id+1-height+3)05d'), '00158')
-        self.assertEqual(out('%(width+100)05d'), 'NA')
-        self.assertEqual(out('%(formats.0)s'), str(FORMATS[0]))
-        self.assertEqual(out('%(height.0)03d'), '001')
-        self.assertEqual(out('%(formats.-1.id)s'), str(FORMATS[-1]['id']))
-        self.assertEqual(out('%(formats.3)s'), 'NA')
-        self.assertEqual(out('%(formats.:2:-1)r'), repr(FORMATS[:2:-1]))
-        self.assertEqual(out('%(formats.0.id.-1+id)f'), '1235.000000')
+        test('%(timestamp-1000>%H-%M-%S)s', '11-43-20')
+        test('%(id+1-height+3)05d', '00158')
+        test('%(width+100)05d', 'NA')
+        test('%(formats.0) 15s', ('% 15s' % FORMATS[0], '% 15s' % str(FORMATS[0]).replace(':', ' -')))
+        test('%(formats.0)r', (repr(FORMATS[0]), repr(FORMATS[0]).replace(':', ' -')))
+        test('%(height.0)03d', '001')
+        test('%(-height.0)04d', '-001')
+        test('%(formats.-1.id)s', FORMATS[-1]['id'])
+        test('%(formats.0.id.-1)d', FORMATS[0]['id'][-1])
+        test('%(formats.3)s', 'NA')
+        test('%(formats.:2:-1)r', repr(FORMATS[:2:-1]))
+        test('%(formats.0.id.-1+id)f', '1235.000000')
+        test('%(formats.0.id.-1+formats.1.id.-1)d', '3')

-    def test_prepare_filename(self):
-        def fname(templ):
-            params = {'outtmpl': templ}
-            ydl = YoutubeDL(params)
-            return ydl.prepare_filename(self.outtmpl_info)
+        # Empty filename
+        test('%(foo|)s-%(bar|)s.%(ext)s', '-.mp4')
+        # test('%(foo|)s.%(ext)s', ('.mp4', '_.mp4'))  # fixme
+        # test('%(foo|)s', ('', '_'))  # fixme

-        self.assertEqual(fname('%%'), '%')
-        self.assertEqual(fname('%%%%'), '%%')
-        self.assertEqual(fname('%%(width)06d.%(ext)s'), '%(width)06d.mp4')
-        self.assertEqual(fname('%(width)06d.%(ext)s'), 'NA.mp4')
-        self.assertEqual(fname('%(width)06d.%%(ext)s'), 'NA.%(ext)s')
-        self.assertEqual(fname('%%(width)06d.%(ext)s'), '%(width)06d.mp4')
-
-        self.assertEqual(fname('Hello %(title1)s'), 'Hello $PATH')
-        self.assertEqual(fname('Hello %(title2)s'), 'Hello %PATH%')
-
-        self.assertEqual(fname('%(title3)s'), 'foo_bar_test')
-        self.assertEqual(fname('%(formats.0)s'), "{'id' - 'id1'}")
-
-        self.assertEqual(fname('%(id)r %(height)r'), "'1234' 1080")
-        self.assertEqual(fname('%(formats.0)r'), "{'id' - 'id1'}")
+        # Path expansion and escaping
+        test('Hello %(title1)s', 'Hello $PATH')
+        test('Hello %(title2)s', 'Hello %PATH%')
+        test('%(title3)s', ('foo/bar\\test', 'foo_bar_test'))
+        test('folder/%(title3)s', ('folder/foo/bar\\test', 'folder%sfoo_bar_test' % os.path.sep))

    def test_format_note(self):
        ydl = YoutubeDL()
--- a/test/test_unicode_literals.py.disabled
+++ b/test/test_unicode_literals.py.disabled
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -12,6 +12,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

 # Various small unit tests
 import io
+import itertools
 import json
 import xml.etree.ElementTree

@@ -108,6 +109,7 @@ from yt_dlp.utils import (
    cli_bool_option,
    parse_codecs,
    iri_to_uri,
+    LazyList,
 )
 from yt_dlp.compat import (
    compat_chr,
@@ -126,6 +128,7 @@ class TestUtil(unittest.TestCase):
        self.assertTrue(timeconvert('bougrg') is None)

    def test_sanitize_filename(self):
+        self.assertEqual(sanitize_filename(''), '')
        self.assertEqual(sanitize_filename('abc'), 'abc')
        self.assertEqual(sanitize_filename('abc_d-e'), 'abc_d-e')

@@ -1524,6 +1527,47 @@ Line 1
        self.assertEqual(clean_podcast_url('https://www.podtrac.com/pts/redirect.mp3/chtbl.com/track/5899E/traffic.megaphone.fm/HSW7835899191.mp3'), 'https://traffic.megaphone.fm/HSW7835899191.mp3')
        self.assertEqual(clean_podcast_url('https://play.podtrac.com/npr-344098539/edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3'), 'https://edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3')

+    def test_LazyList(self):
+        it = list(range(10))
+
+        self.assertEqual(list(LazyList(it)), it)
+        self.assertEqual(LazyList(it).exhaust(), it)
+        self.assertEqual(LazyList(it)[5], it[5])
+
+        self.assertEqual(LazyList(it)[::2], it[::2])
+        self.assertEqual(LazyList(it)[1::2], it[1::2])
+        self.assertEqual(LazyList(it)[6:2:-2], it[6:2:-2])
+        self.assertEqual(LazyList(it)[::-1], it[::-1])
+
+        self.assertTrue(LazyList(it))
+        self.assertFalse(LazyList(range(0)))
+        self.assertEqual(len(LazyList(it)), len(it))
+        self.assertEqual(repr(LazyList(it)), repr(it))
+        self.assertEqual(str(LazyList(it)), str(it))
+
+        self.assertEqual(list(reversed(LazyList(it))), it[::-1])
+        self.assertEqual(list(reversed(LazyList(it))[1:3:7]), it[::-1][1:3:7])
+
+    def test_LazyList_laziness(self):
+
+        def test(ll, idx, val, cache):
+            self.assertEqual(ll[idx], val)
+            self.assertEqual(getattr(ll, '_LazyList__cache'), list(cache))
+
+        ll = LazyList(range(10))
+        test(ll, 0, 0, range(1))
+        test(ll, 5, 5, range(6))
+        test(ll, -3, 7, range(10))
+
+        ll = reversed(LazyList(range(10)))
+        test(ll, -1, 0, range(1))
+        test(ll, 3, 6, range(10))
+
+        ll = LazyList(itertools.count())
+        test(ll, 10, 10, range(11))
+        reversed(ll)
+        test(ll, -15, 14, range(15))
+

 if __name__ == '__main__':
    unittest.main()
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -20,6 +20,7 @@ import re
 import shutil
 import subprocess
 import sys
+import tempfile
 import time
 import tokenize
 import traceback
@@ -67,6 +68,7 @@ from .utils import (
    STR_FORMAT_RE,
    formatSeconds,
    GeoRestrictedError,
+    HEADRequest,
    int_or_none,
    iri_to_uri,
    ISO3166Utils,
@@ -86,7 +88,6 @@ from .utils import (
    preferredencoding,
    prepend_extension,
    process_communicate_or_kill,
-    random_uuidv4,
    register_socks_protocols,
    RejectedVideoReached,
    render_table,
@@ -100,6 +101,7 @@ from .utils import (
    str_or_none,
    strftime_or_none,
    subtitles_filename,
+    ThrottledDownload,
    to_high_limit_path,
    traverse_obj,
    UnavailableVideoError,
@@ -126,13 +128,14 @@ from .downloader import (
 )
 from .downloader.rtmp import rtmpdump_version
 from .postprocessor import (
+    get_postprocessor,
+    FFmpegFixupDurationPP,
    FFmpegFixupM3u8PP,
    FFmpegFixupM4aPP,
    FFmpegFixupStretchedPP,
+    FFmpegFixupTimestampPP,
    FFmpegMergerPP,
    FFmpegPostProcessor,
-    # FFmpegSubtitlesConvertorPP,
-    get_postprocessor,
    MoveFilesAfterDownloadPP,
 )
 from .version import __version__
@@ -390,15 +393,15 @@ class YoutubeDL(object):
    compat_opts:       Compatibility options. See "Differences in default behavior".
                       Note that only format-sort, format-spec, no-live-chat,
                       no-attach-info-json, playlist-index, list-formats,
-                       no-direct-merge, no-youtube-channel-redirect,
-                       and no-youtube-unavailable-videos works when used via the API
+                       no-direct-merge, embed-thumbnail-atomicparsley,
+                       no-youtube-unavailable-videos, no-youtube-channel-redirect,
+                       works when used via the API

    The following parameters are not used by YoutubeDL itself, they are used by
    the downloader (see yt_dlp/downloader/common.py):
-    nopart, updatetime, buffersize, ratelimit, min_filesize, max_filesize, test,
-    noresizebuffer, retries, continuedl, noprogress, consoletitle,
-    xattr_set_filesize, external_downloader_args, hls_use_mpegts,
-    http_chunk_size.
+    nopart, updatetime, buffersize, ratelimit, throttledratelimit, min_filesize,
+    max_filesize, test, noresizebuffer, retries, continuedl, noprogress, consoletitle,
+    xattr_set_filesize, external_downloader_args, hls_use_mpegts, http_chunk_size.

    The following options are used by the post processors:
    prefer_ffmpeg:     If False, use avconv instead of ffmpeg if both are available,
@@ -472,8 +475,7 @@ class YoutubeDL(object):

        if sys.version_info < (3, 6):
            self.report_warning(
-                'Support for Python version %d.%d have been deprecated and will break in future versions of yt-dlp! '
-                'Update to Python 3.6 or above' % sys.version_info[:2])
+                'Python version %d.%d is not supported! Please update to Python 3.6 or above' % sys.version_info[:2])

        def check_deprecated(param, option, suggestion):
            if self.params.get(param) is not None:
@@ -539,6 +541,11 @@ class YoutubeDL(object):

        self.outtmpl_dict = self.parse_outtmpl()

+        # Creating format selector here allows us to catch syntax errors before the extraction
+        self.format_selector = (
+            None if self.params.get('format') is None
+            else self.build_format_selector(self.params['format']))
+
        self._setup_opener()

        """Preload the archive, if any is specified"""
@@ -564,14 +571,9 @@ class YoutubeDL(object):
            self.add_default_info_extractors()

        for pp_def_raw in self.params.get('postprocessors', []):
-            pp_class = get_postprocessor(pp_def_raw['key'])
            pp_def = dict(pp_def_raw)
-            del pp_def['key']
-            if 'when' in pp_def:
-                when = pp_def['when']
-                del pp_def['when']
-            else:
-                when = 'post_process'
+            when = pp_def.pop('when', 'post_process')
+            pp_class = get_postprocessor(pp_def.pop('key'))
            pp = pp_class(self, **compat_kwargs(pp_def))
            self.add_post_processor(pp, when=when)

@@ -813,6 +815,21 @@ class YoutubeDL(object):
                    'Put  from __future__ import unicode_literals  at the top of your code file or consider switching to Python 3.x.')
        return outtmpl_dict

+    def get_output_path(self, dir_type='', filename=None):
+        paths = self.params.get('paths', {})
+        assert isinstance(paths, dict)
+        path = os.path.join(
+            expand_path(paths.get('home', '').strip()),
+            expand_path(paths.get(dir_type, '').strip()) if dir_type else '',
+            filename or '')
+
+        # Temporary fix for #4787
+        # 'Treat' all problem characters by passing filename through preferredencoding
+        # to workaround encoding issues with subprocess on python2 @ Windows
+        if sys.version_info < (3, 0) and sys.platform == 'win32':
+            path = encodeFilename(path, True).decode(preferredencoding())
+        return sanitize_path(path, force=self.params.get('windowsfilenames'))
+
    @staticmethod
    def validate_outtmpl(tmpl):
        ''' @return None or Exception object '''
@@ -847,23 +864,24 @@ class YoutubeDL(object):
            'autonumber': self.params.get('autonumber_size') or 5,
        }

-        EXTERNAL_FORMAT_RE = STR_FORMAT_RE.format('[^)]*')
-        # Field is of the form key1.key2...
-        # where keys (except first) can be string, int or slice
-        FIELD_RE = r'\w+(?:\.(?:\w+|[-\d]*(?::[-\d]*){0,2}))*'
-        INTERNAL_FORMAT_RE = re.compile(r'''(?x)
-            (?P<negate>-)?
-            (?P<fields>{0})
-            (?P<maths>(?:[-+]-?(?:\d+(?:\.\d+)?|{0}))*)
-            (?:>(?P<strf_format>.+?))?
-            (?:\|(?P<default>.*?))?
-            $'''.format(FIELD_RE))
-        MATH_OPERATORS_RE = re.compile(r'(?<![-+])([-+])')
+        TMPL_DICT = {}
+        EXTERNAL_FORMAT_RE = re.compile(STR_FORMAT_RE.format('[^)]*'))
        MATH_FUNCTIONS = {
            '+': float.__add__,
            '-': float.__sub__,
        }
-        tmpl_dict = {}
+        # Field is of the form key1.key2...
+        # where keys (except first) can be string, int or slice
+        FIELD_RE = r'\w+(?:\.(?:\w+|{num}|{num}?(?::{num}?){{1,2}}))*'.format(num=r'(?:-?\d+)')
+        MATH_FIELD_RE = r'''{field}|{num}'''.format(field=FIELD_RE, num=r'-?\d+(?:.\d+)?')
+        MATH_OPERATORS_RE = r'(?:%s)' % '|'.join(map(re.escape, MATH_FUNCTIONS.keys()))
+        INTERNAL_FORMAT_RE = re.compile(r'''(?x)
+            (?P<negate>-)?
+            (?P<fields>{field})
+            (?P<maths>(?:{math_op}{math_field})*)
+            (?:>(?P<strf_format>.+?))?
+            (?:\|(?P<default>.*?))?
+            $'''.format(field=FIELD_RE, math_op=MATH_OPERATORS_RE, math_field=MATH_FIELD_RE))

        get_key = lambda k: traverse_obj(
            info_dict, k.split('.'), is_user_input=True, traverse_string=True)
@@ -877,24 +895,27 @@ class YoutubeDL(object):
                if value is not None:
                    value *= -1
            # Do maths
-            if mdict['maths']:
+            offset_key = mdict['maths']
+            if offset_key:
                value = float_or_none(value)
                operator = None
-                for item in MATH_OPERATORS_RE.split(mdict['maths'])[1:]:
-                    if item == '' or value is None:
-                        return None
-                    if operator:
-                        item, multiplier = (item[1:], -1) if item[0] == '-' else (item, 1)
-                        offset = float_or_none(item)
-                        if offset is None:
-                            offset = float_or_none(get_key(item))
-                        try:
-                            value = operator(value, multiplier * offset)
-                        except (TypeError, ZeroDivisionError):
-                            return None
-                        operator = None
-                    else:
+                while offset_key:
+                    item = re.match(
+                        MATH_FIELD_RE if operator else MATH_OPERATORS_RE,
+                        offset_key).group(0)
+                    offset_key = offset_key[len(item):]
+                    if operator is None:
                        operator = MATH_FUNCTIONS[item]
+                        continue
+                    item, multiplier = (item[1:], -1) if item[0] == '-' else (item, 1)
+                    offset = float_or_none(item)
+                    if offset is None:
+                        offset = float_or_none(get_key(item))
+                    try:
+                        value = operator(value, multiplier * offset)
+                    except (TypeError, ZeroDivisionError):
+                        return None
+                    operator = None
            # Datetime formatting
            if mdict['strf_format']:
                value = strftime_or_none(value, mdict['strf_format'])
@@ -909,7 +930,7 @@ class YoutubeDL(object):
            fmt = outer_mobj.group('format')
            mobj = re.match(INTERNAL_FORMAT_RE, key)
            if mobj is None:
-                value, default = None, na
+                value, default, mobj = None, na, {'fields': ''}
            else:
                mobj = mobj.groupdict()
                default = mobj['default'] if mobj['default'] is not None else na
@@ -919,7 +940,6 @@ class YoutubeDL(object):
                fmt = '0{:d}d'.format(field_size_compat_map[key])

            value = default if value is None else value
-            key += '\0%s' % fmt

            if fmt == 'c':
                value = compat_str(value)
@@ -936,11 +956,13 @@ class YoutubeDL(object):
                    # If value is an object, sanitize might convert it to a string
                    # So we convert it to repr first
                    value, fmt = repr(value), '%ss' % fmt[:-1]
-                value = sanitize(key, value)
-            tmpl_dict[key] = value
+                if fmt[-1] in 'csr':
+                    value = sanitize(mobj['fields'].split('.')[-1], value)
+            key += '\0%s' % fmt
+            TMPL_DICT[key] = value
            return '%({key}){fmt}'.format(key=key, fmt=fmt)

-        return re.sub(EXTERNAL_FORMAT_RE, create_key, outtmpl), tmpl_dict
+        return EXTERNAL_FORMAT_RE.sub(create_key, outtmpl), TMPL_DICT

    def _prepare_filename(self, info_dict, tmpl_type='default'):
        try:
@@ -985,12 +1007,11 @@ class YoutubeDL(object):

    def prepare_filename(self, info_dict, dir_type='', warn=False):
        """Generate the output filename."""
-        paths = self.params.get('paths', {})
-        assert isinstance(paths, dict)
+
        filename = self._prepare_filename(info_dict, dir_type or 'default')

        if warn and not self.__prepare_filename_warned:
-            if not paths:
+            if not self.params.get('paths'):
                pass
            elif filename == '-':
                self.report_warning('--paths is ignored when an outputting to stdout')
@@ -1000,18 +1021,7 @@ class YoutubeDL(object):
        if filename == '-' or not filename:
            return filename

-        homepath = expand_path(paths.get('home', '').strip())
-        assert isinstance(homepath, compat_str)
-        subdir = expand_path(paths.get(dir_type, '').strip()) if dir_type else ''
-        assert isinstance(subdir, compat_str)
-        path = os.path.join(homepath, subdir, filename)
-
-        # Temporary fix for #4787
-        # 'Treat' all problem characters by passing filename through preferredencoding
-        # to workaround encoding issues with subprocess on python2 @ Windows
-        if sys.version_info < (3, 0) and sys.platform == 'win32':
-            path = encodeFilename(path, True).decode(preferredencoding())
-        return sanitize_path(path, force=self.params.get('windowsfilenames'))
+        return self.get_output_path(dir_type, filename)

    def _match_entry(self, info_dict, incomplete=False, silent=False):
        """ Returns None if the file should be downloaded """
@@ -1135,6 +1145,10 @@ class YoutubeDL(object):
                self.report_error(msg)
            except ExtractorError as e:  # An error we somewhat expected
                self.report_error(compat_str(e), e.format_traceback())
+            except ThrottledDownload:
+                self.to_stderr('\r')
+                self.report_warning('The download speed is below throttle limit. Re-extracting data')
+                return wrapper(self, *args, **kwargs)
            except (MaxDownloadsReached, ExistingVideoReached, RejectedVideoReached):
                raise
            except Exception as e:
@@ -1483,12 +1497,11 @@ class YoutubeDL(object):
            '!=': operator.ne,
        }
        operator_rex = re.compile(r'''(?x)\s*
-            (?P<key>width|height|tbr|abr|vbr|asr|filesize|filesize_approx|fps)
-            \s*(?P<op>%s)(?P<none_inclusive>\s*\?)?\s*
-            (?P<value>[0-9.]+(?:[kKmMgGtTpPeEzZyY]i?[Bb]?)?)
-            $
+            (?P<key>width|height|tbr|abr|vbr|asr|filesize|filesize_approx|fps)\s*
+            (?P<op>%s)(?P<none_inclusive>\s*\?)?\s*
+            (?P<value>[0-9.]+(?:[kKmMgGtTpPeEzZyY]i?[Bb]?)?)\s*
            ''' % '|'.join(map(re.escape, OPERATORS.keys())))
-        m = operator_rex.search(filter_spec)
+        m = operator_rex.fullmatch(filter_spec)
        if m:
            try:
                comparison_value = int(m.group('value'))
@@ -1509,13 +1522,12 @@ class YoutubeDL(object):
                '$=': lambda attr, value: attr.endswith(value),
                '*=': lambda attr, value: value in attr,
            }
-            str_operator_rex = re.compile(r'''(?x)
-                \s*(?P<key>[a-zA-Z0-9._-]+)
-                \s*(?P<negation>!\s*)?(?P<op>%s)(?P<none_inclusive>\s*\?)?
-                \s*(?P<value>[a-zA-Z0-9._-]+)
-                \s*$
+            str_operator_rex = re.compile(r'''(?x)\s*
+                (?P<key>[a-zA-Z0-9._-]+)\s*
+                (?P<negation>!\s*)?(?P<op>%s)(?P<none_inclusive>\s*\?)?\s*
+                (?P<value>[a-zA-Z0-9._-]+)\s*
                ''' % '|'.join(map(re.escape, STR_OPERATORS.keys())))
-            m = str_operator_rex.search(filter_spec)
+            m = str_operator_rex.fullmatch(filter_spec)
            if m:
                comparison_value = m.group('value')
                str_op = STR_OPERATORS[m.group('op')]
@@ -1525,7 +1537,7 @@ class YoutubeDL(object):
                    op = str_op

        if not m:
-            raise ValueError('Invalid filter specification %r' % filter_spec)
+            raise SyntaxError('Invalid filter specification %r' % filter_spec)

        def _filter(f):
            actual_value = f.get(m.group('key'))
@@ -1680,9 +1692,12 @@ class YoutubeDL(object):
            formats_info.extend(format_2.get('requested_formats', (format_2,)))

            if not allow_multiple_streams['video'] or not allow_multiple_streams['audio']:
-                get_no_more = {"video": False, "audio": False}
+                get_no_more = {'video': False, 'audio': False}
                for (i, fmt_info) in enumerate(formats_info):
-                    for aud_vid in ["audio", "video"]:
+                    if fmt_info.get('acodec') == fmt_info.get('vcodec') == 'none':
+                        formats_info.pop(i)
+                        continue
+                    for aud_vid in ['audio', 'video']:
                        if not allow_multiple_streams[aud_vid] and fmt_info.get(aud_vid[0] + 'codec') != 'none':
                            if get_no_more[aud_vid]:
                                formats_info.pop(i)
@@ -1735,18 +1750,20 @@ class YoutubeDL(object):
        def _check_formats(formats):
            for f in formats:
                self.to_screen('[info] Testing format %s' % f['format_id'])
-                paths = self.params.get('paths', {})
-                temp_file = os.path.join(
-                    expand_path(paths.get('home', '').strip()),
-                    expand_path(paths.get('temp', '').strip()),
-                    'ytdl.%s.f%s.check-format' % (random_uuidv4(), f['format_id']))
+                temp_file = tempfile.NamedTemporaryFile(
+                    suffix='.tmp', delete=False,
+                    dir=self.get_output_path('temp') or None)
+                temp_file.close()
                try:
-                    dl, _ = self.dl(temp_file, f, test=True)
+                    dl, _ = self.dl(temp_file.name, f, test=True)
                except (ExtractorError, IOError, OSError, ValueError) + network_exceptions:
                    dl = False
                finally:
-                    if os.path.exists(temp_file):
-                        os.remove(temp_file)
+                    if os.path.exists(temp_file.name):
+                        try:
+                            os.remove(temp_file.name)
+                        except OSError:
+                            self.report_warning('Unable to delete temporary file "%s"' % temp_file.name)
                if dl:
                    yield f
                else:
@@ -1788,7 +1805,9 @@ class YoutubeDL(object):
                            yield f
                elif format_spec == 'mergeall':
                    def selector_function(ctx):
-                        formats = list(_check_formats(ctx['formats']))
+                        formats = ctx['formats']
+                        if check_formats:
+                            formats = list(_check_formats(formats))
                        if not formats:
                            return
                        merged_format = formats[-1]
@@ -1809,14 +1828,16 @@ class YoutubeDL(object):
                        format_modified = mobj.group('mod') is not None

                        format_fallback = not format_type and not format_modified  # for b, w
-                        filter_f = (
+                        _filter_f = (
                            (lambda f: f.get('%scodec' % format_type) != 'none')
                            if format_type and format_modified  # bv*, ba*, wv*, wa*
                            else (lambda f: f.get('%scodec' % not_format_type) == 'none')
                            if format_type  # bv, ba, wv, wa
                            else (lambda f: f.get('vcodec') != 'none' and f.get('acodec') != 'none')
                            if not format_modified  # b, w
-                            else None)  # b*, w*
+                            else lambda f: True)  # b*, w*
+                        filter_f = lambda f: _filter_f(f) and (
+                            f.get('vcodec') != 'none' or f.get('acodec') != 'none')
                    else:
                        filter_f = ((lambda f: f.get('ext') == format_spec)
                                    if format_spec in ['mp4', 'flv', 'webm', '3gp', 'm4a', 'mp3', 'ogg', 'aac', 'wav']  # extension
@@ -1909,8 +1930,7 @@ class YoutubeDL(object):
        self.cookiejar.add_cookie_header(pr)
        return pr.get_header('Cookie')

-    @staticmethod
-    def _sanitize_thumbnails(info_dict):
+    def _sanitize_thumbnails(self, info_dict):
        thumbnails = info_dict.get('thumbnails')
        if thumbnails is None:
            thumbnail = info_dict.get('thumbnail')
@@ -1923,12 +1943,25 @@ class YoutubeDL(object):
                t.get('height') if t.get('height') is not None else -1,
                t.get('id') if t.get('id') is not None else '',
                t.get('url')))
+
+            def test_thumbnail(t):
+                self.to_screen('[info] Testing thumbnail %s' % t['id'])
+                try:
+                    self.urlopen(HEADRequest(t['url']))
+                except network_exceptions as err:
+                    self.to_screen('[info] Unable to connect to thumbnail %s URL "%s" - %s. Skipping...' % (
+                        t['id'], t['url'], error_to_compat_str(err)))
+                    return False
+                return True
+
            for i, t in enumerate(thumbnails):
-                t['url'] = sanitize_url(t['url'])
-                if t.get('width') and t.get('height'):
-                    t['resolution'] = '%dx%d' % (t['width'], t['height'])
                if t.get('id') is None:
                    t['id'] = '%d' % i
+                if t.get('width') and t.get('height'):
+                    t['resolution'] = '%dx%d' % (t['width'], t['height'])
+                t['url'] = sanitize_url(t['url'])
+            if self.params.get('check_formats'):
+                info_dict['thumbnails'] = reversed(LazyList(filter(test_thumbnail, thumbnails[::-1])))

    def process_video_result(self, info_dict, download=True):
        assert info_dict.get('_type', 'video') == 'video'
@@ -2114,12 +2147,11 @@ class YoutubeDL(object):
            self.list_formats(info_dict)
            return

-        req_format = self.params.get('format')
-        if req_format is None:
+        format_selector = self.format_selector
+        if format_selector is None:
            req_format = self._default_format_spec(info_dict, download=download)
            self.write_debug('Default format spec: %s' % req_format)
-
-        format_selector = self.build_format_selector(req_format)
+            format_selector = self.build_format_selector(req_format)

        # While in format selection we may need to have an access to the original
        # format set in order to calculate some metrics or do some processing.
@@ -2653,65 +2685,53 @@ class YoutubeDL(object):
                return

            if success and full_filename != '-':
-                # Fixup content
-                fixup_policy = self.params.get('fixup')
-                if fixup_policy is None:
-                    fixup_policy = 'detect_or_warn'

-                INSTALL_FFMPEG_MESSAGE = 'Install ffmpeg to fix this automatically.'
+                def fixup():
+                    do_fixup = True
+                    fixup_policy = self.params.get('fixup')
+                    vid = info_dict['id']

-                stretched_ratio = info_dict.get('stretched_ratio')
-                if stretched_ratio is not None and stretched_ratio != 1:
-                    if fixup_policy == 'warn':
-                        self.report_warning('%s: Non-uniform pixel ratio (%s)' % (
-                            info_dict['id'], stretched_ratio))
-                    elif fixup_policy == 'detect_or_warn':
-                        stretched_pp = FFmpegFixupStretchedPP(self)
-                        if stretched_pp.available:
-                            info_dict['__postprocessors'].append(stretched_pp)
+                    if fixup_policy in ('ignore', 'never'):
+                        return
+                    elif fixup_policy == 'warn':
+                        do_fixup = False
+                    elif fixup_policy != 'force':
+                        assert fixup_policy in ('detect_or_warn', None)
+                        if not info_dict.get('__real_download'):
+                            do_fixup = False
+
+                    def ffmpeg_fixup(cndn, msg, cls):
+                        if not cndn:
+                            return
+                        if not do_fixup:
+                            self.report_warning(f'{vid}: {msg}')
+                            return
+                        pp = cls(self)
+                        if pp.available:
+                            info_dict['__postprocessors'].append(pp)
                        else:
-                            self.report_warning(
-                                '%s: Non-uniform pixel ratio (%s). %s'
-                                % (info_dict['id'], stretched_ratio, INSTALL_FFMPEG_MESSAGE))
-                    else:
-                        assert fixup_policy in ('ignore', 'never')
+                            self.report_warning(f'{vid}: {msg}. Install ffmpeg to fix this automatically')

-                if (info_dict.get('requested_formats') is None
-                        and info_dict.get('container') == 'm4a_dash'
-                        and info_dict.get('ext') == 'm4a'):
-                    if fixup_policy == 'warn':
-                        self.report_warning(
-                            '%s: writing DASH m4a. '
-                            'Only some players support this container.'
-                            % info_dict['id'])
-                    elif fixup_policy == 'detect_or_warn':
-                        fixup_pp = FFmpegFixupM4aPP(self)
-                        if fixup_pp.available:
-                            info_dict['__postprocessors'].append(fixup_pp)
-                        else:
-                            self.report_warning(
-                                '%s: writing DASH m4a. '
-                                'Only some players support this container. %s'
-                                % (info_dict['id'], INSTALL_FFMPEG_MESSAGE))
-                    else:
-                        assert fixup_policy in ('ignore', 'never')
+                    stretched_ratio = info_dict.get('stretched_ratio')
+                    ffmpeg_fixup(
+                        stretched_ratio not in (1, None),
+                        f'Non-uniform pixel ratio {stretched_ratio}',
+                        FFmpegFixupStretchedPP)

-                if ('protocol' in info_dict
-                        and get_suitable_downloader(info_dict, self.params).__name__ == 'HlsFD'):
-                    if fixup_policy == 'warn':
-                        self.report_warning('%s: malformed AAC bitstream detected.' % (
-                            info_dict['id']))
-                    elif fixup_policy == 'detect_or_warn':
-                        fixup_pp = FFmpegFixupM3u8PP(self)
-                        if fixup_pp.available:
-                            info_dict['__postprocessors'].append(fixup_pp)
-                        else:
-                            self.report_warning(
-                                '%s: malformed AAC bitstream detected. %s'
-                                % (info_dict['id'], INSTALL_FFMPEG_MESSAGE))
-                    else:
-                        assert fixup_policy in ('ignore', 'never')
+                    ffmpeg_fixup(
+                        (info_dict.get('requested_formats') is None
+                         and info_dict.get('container') == 'm4a_dash'
+                         and info_dict.get('ext') == 'm4a'),
+                        'writing DASH m4a. Only some players support this container',
+                        FFmpegFixupM4aPP)

+                    downloader = (get_suitable_downloader(info_dict, self.params).__name__
+                                  if 'protocol' in info_dict else None)
+                    ffmpeg_fixup(downloader == 'HlsFD', 'malformed AAC bitstream detected', FFmpegFixupM3u8PP)
+                    ffmpeg_fixup(downloader == 'WebSocketFragmentFD', 'malformed timestamps detected', FFmpegFixupTimestampPP)
+                    ffmpeg_fixup(downloader == 'WebSocketFragmentFD', 'malformed duration detected', FFmpegFixupDurationPP)
+
+                fixup()
                try:
                    info_dict = self.post_process(dl_filename, info_dict, files_to_move)
                except PostProcessingError as err:
@@ -2793,7 +2813,7 @@ class YoutubeDL(object):
            info_dict['epoch'] = int(time.time())
            reject = lambda k, v: k in remove_keys
        filter_fn = lambda obj: (
-            list(map(filter_fn, obj)) if isinstance(obj, (list, tuple, set))
+            list(map(filter_fn, obj)) if isinstance(obj, (LazyList, list, tuple, set))
            else obj if not isinstance(obj, dict)
            else dict((k, filter_fn(v)) for k, v in obj.items() if not reject(k, v)))
        return filter_fn(info_dict)
@@ -2904,6 +2924,8 @@ class YoutubeDL(object):
    @staticmethod
    def format_resolution(format, default='unknown'):
        if format.get('vcodec') == 'none':
+            if format.get('acodec') == 'none':
+                return 'images'
            return 'audio only'
        if format.get('resolution') is not None:
            return format['resolution']
@@ -3031,7 +3053,7 @@ class YoutubeDL(object):
                hideEmpty=new_format)))

    def list_thumbnails(self, info_dict):
-        thumbnails = info_dict.get('thumbnails')
+        thumbnails = list(info_dict.get('thumbnails'))
        if not thumbnails:
            self.to_screen('[info] No thumbnails present for %s' % info_dict['id'])
            return
@@ -3241,6 +3263,7 @@ class YoutubeDL(object):

            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
                ret.append(suffix + thumb_ext)
+                t['filepath'] = thumb_filename
                self.to_screen('[%s] %s: Thumbnail %sis already present' %
                               (info_dict['extractor'], info_dict['id'], thumb_display_id))
            else:
--- a/yt_dlp/init.py
+++ b/yt_dlp/init.py
@@ -151,6 +151,11 @@ def _real_main(argv=None):
        if numeric_limit is None:
            parser.error('invalid rate limit specified')
        opts.ratelimit = numeric_limit
+    if opts.throttledratelimit is not None:
+        numeric_limit = FileDownloader.parse_bytes(opts.throttledratelimit)
+        if numeric_limit is None:
+            parser.error('invalid rate limit specified')
+        opts.throttledratelimit = numeric_limit
    if opts.min_filesize is not None:
        numeric_limit = FileDownloader.parse_bytes(opts.min_filesize)
        if numeric_limit is None:
@@ -268,6 +273,7 @@ def _real_main(argv=None):
        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'no-playlist-metafiles',
        'multistreams', 'no-live-chat', 'playlist-index', 'list-formats', 'no-direct-merge',
        'no-youtube-channel-redirect', 'no-youtube-unavailable-videos', 'no-attach-info-json',
+        'embed-thumbnail-atomicparsley',
    ]
    compat_opts = parse_compat_opts()

@@ -551,6 +557,7 @@ def _real_main(argv=None):
        'ignoreerrors': opts.ignoreerrors,
        'force_generic_extractor': opts.force_generic_extractor,
        'ratelimit': opts.ratelimit,
+        'throttledratelimit': opts.throttledratelimit,
        'overwrites': opts.overwrites,
        'retries': opts.retries,
        'fragment_retries': opts.fragment_retries,
--- a/yt_dlp/compat.py
+++ b/yt_dlp/compat.py
@@ -3030,6 +3030,21 @@ except AttributeError:
    compat_Match = type(re.compile('').match(''))


+import asyncio
+try:
+    compat_asyncio_run = asyncio.run
+except AttributeError:
+    def compat_asyncio_run(coro):
+        try:
+            loop = asyncio.get_event_loop()
+        except RuntimeError:
+            loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+        loop.run_until_complete(coro)
+
+    asyncio.run = compat_asyncio_run
+
+
 __all__ = [
    'compat_HTMLParseError',
    'compat_HTMLParser',
@@ -3037,6 +3052,7 @@ __all__ = [
    'compat_Match',
    'compat_Pattern',
    'compat_Struct',
+    'compat_asyncio_run',
    'compat_b64decode',
    'compat_basestring',
    'compat_chr',
--- a/yt_dlp/downloader/init.py
+++ b/yt_dlp/downloader/init.py
@@ -22,8 +22,10 @@ from .http import HttpFD
 from .rtmp import RtmpFD
 from .rtsp import RtspFD
 from .ism import IsmFD
+from .mhtml import MhtmlFD
 from .niconico import NiconicoDmcFD
-from .youtube_live_chat import YoutubeLiveChatReplayFD
+from .websocket import WebSocketFragmentFD
+from .youtube_live_chat import YoutubeLiveChatFD
 from .external import (
    get_external_downloader,
    FFmpegFD,
@@ -39,8 +41,11 @@ PROTOCOL_MAP = {
    'f4m': F4mFD,
    'http_dash_segments': DashSegmentsFD,
    'ism': IsmFD,
+    'mhtml': MhtmlFD,
    'niconico_dmc': NiconicoDmcFD,
-    'youtube_live_chat_replay': YoutubeLiveChatReplayFD,
+    'websocket_frag': WebSocketFragmentFD,
+    'youtube_live_chat': YoutubeLiveChatFD,
+    'youtube_live_chat_replay': YoutubeLiveChatFD,
 }


@@ -50,6 +55,7 @@ def shorten_protocol_name(proto, simplify=False):
        'rtmp_ffmpeg': 'rtmp_f',
        'http_dash_segments': 'dash',
        'niconico_dmc': 'dmc',
+        'websocket_frag': 'WSfrag',
    }
    if simplify:
        short_protocol_names.update({
--- a/yt_dlp/downloader/common.py
+++ b/yt_dlp/downloader/common.py
@@ -32,6 +32,7 @@ class FileDownloader(object):
    verbose:            Print additional info to stdout.
    quiet:              Do not print messages to stdout.
    ratelimit:          Download speed limit, in bytes/sec.
+    throttledratelimit: Assume the download is being throttled below this speed (bytes/sec)
    retries:            Number of times to retry for HTTP error 5xx
    buffersize:         Size of download buffer in bytes.
    noresizebuffer:     Do not automatically resize the download buffer.
--- a/yt_dlp/downloader/dash.py
+++ b/yt_dlp/downloader/dash.py
@@ -1,21 +1,9 @@
 from __future__ import unicode_literals

-import errno
-try:
-    import concurrent.futures
-    can_threaded_download = True
-except ImportError:
-    can_threaded_download = False
-
 from ..downloader import _get_real_downloader
 from .fragment import FragmentFD

-from ..compat import compat_urllib_error
-from ..utils import (
-    DownloadError,
-    sanitize_open,
-    urljoin,
-)
+from ..utils import urljoin


 class DashSegmentsFD(FragmentFD):
@@ -43,9 +31,6 @@ class DashSegmentsFD(FragmentFD):
        else:
            self._prepare_and_start_frag_download(ctx)

-        fragment_retries = self.params.get('fragment_retries', 0)
-        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
-
        fragments_to_download = []
        frag_index = 0
        for i, fragment in enumerate(fragments):
@@ -76,116 +61,5 @@ class DashSegmentsFD(FragmentFD):
            if not success:
                return False
        else:
-            def download_fragment(fragment):
-                i = fragment['index']
-                frag_index = fragment['frag_index']
-                fragment_url = fragment['url']
-
-                ctx['fragment_index'] = frag_index
-
-                # In DASH, the first segment contains necessary headers to
-                # generate a valid MP4 file, so always abort for the first segment
-                fatal = i == 0 or not skip_unavailable_fragments
-                count = 0
-                while count <= fragment_retries:
-                    try:
-                        success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
-                        if not success:
-                            return False, frag_index
-                        break
-                    except compat_urllib_error.HTTPError as err:
-                        # YouTube may often return 404 HTTP error for a fragment causing the
-                        # whole download to fail. However if the same fragment is immediately
-                        # retried with the same request data this usually succeeds (1-2 attempts
-                        # is usually enough) thus allowing to download the whole file successfully.
-                        # To be future-proof we will retry all fragments that fail with any
-                        # HTTP error.
-                        count += 1
-                        if count <= fragment_retries:
-                            self.report_retry_fragment(err, frag_index, count, fragment_retries)
-                    except DownloadError:
-                        # Don't retry fragment if error occurred during HTTP downloading
-                        # itself since it has own retry settings
-                        if not fatal:
-                            break
-                        raise
-
-                if count > fragment_retries:
-                    if not fatal:
-                        return False, frag_index
-                    ctx['dest_stream'].close()
-                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
-                    return False, frag_index
-
-                return frag_content, frag_index
-
-            def append_fragment(frag_content, frag_index):
-                fatal = frag_index == 1 or not skip_unavailable_fragments
-                if frag_content:
-                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
-                    try:
-                        file, frag_sanitized = sanitize_open(fragment_filename, 'rb')
-                        ctx['fragment_filename_sanitized'] = frag_sanitized
-                        file.close()
-                        self._append_fragment(ctx, frag_content)
-                        return True
-                    except EnvironmentError as ose:
-                        if ose.errno != errno.ENOENT:
-                            raise
-                        # FileNotFoundError
-                        if not fatal:
-                            self.report_skip_fragment(frag_index)
-                            return True
-                        else:
-                            ctx['dest_stream'].close()
-                            self.report_error(
-                                'fragment %s not found, unable to continue' % frag_index)
-                            return False
-                else:
-                    if not fatal:
-                        self.report_skip_fragment(frag_index)
-                        return True
-                    else:
-                        ctx['dest_stream'].close()
-                        self.report_error(
-                            'fragment %s not found, unable to continue' % frag_index)
-                        return False
-
-            max_workers = self.params.get('concurrent_fragment_downloads', 1)
-            if can_threaded_download and max_workers > 1:
-                self.report_warning('The download speed shown is only of one thread. This is a known issue')
-                _download_fragment = lambda f: (f, download_fragment(f)[1])
-                with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
-                    futures = [pool.submit(_download_fragment, fragment) for fragment in fragments_to_download]
-                    # timeout must be 0 to return instantly
-                    done, not_done = concurrent.futures.wait(futures, timeout=0)
-                    try:
-                        while not_done:
-                            # Check every 1 second for KeyboardInterrupt
-                            freshly_done, not_done = concurrent.futures.wait(not_done, timeout=1)
-                            done |= freshly_done
-                    except KeyboardInterrupt:
-                        for future in not_done:
-                            future.cancel()
-                        # timeout must be none to cancel
-                        concurrent.futures.wait(not_done, timeout=None)
-                        raise KeyboardInterrupt
-
-                for fragment, frag_index in map(lambda x: x.result(), futures):
-                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
-                    down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
-                    fragment['fragment_filename_sanitized'] = frag_sanitized
-                    frag_content = down.read()
-                    down.close()
-                    result = append_fragment(frag_content, frag_index)
-                    if not result:
-                        return False
-            else:
-                for fragment in fragments_to_download:
-                    frag_content, frag_index = download_fragment(fragment)
-                    result = append_fragment(frag_content, frag_index)
-                    if not result:
-                        return False
-
-            self._finish_frag_download(ctx)
+            self.download_and_append_fragments(ctx, fragments_to_download, info_dict)
        return True
--- a/yt_dlp/downloader/external.py
+++ b/yt_dlp/downloader/external.py
@@ -280,6 +280,8 @@ class Aria2cFD(ExternalFD):
               '--file-allocation=none', '-x16', '-j16', '-s16']
        if 'fragments' in info_dict:
            cmd += ['--allow-overwrite=true', '--allow-piece-length-change=true']
+        else:
+            cmd += ['--min-split-size', '1M']

        if info_dict.get('http_headers') is not None:
            for key, val in info_dict['http_headers'].items():
@@ -345,6 +347,10 @@ class FFmpegFD(ExternalFD):
        # TODO: Fix path for ffmpeg
        return FFmpegPostProcessor().available

+    def on_process_started(self, proc, stdin):
+        """ Override this in subclasses  """
+        pass
+
    def _call_downloader(self, tmpfilename, info_dict):
        urls = [f['url'] for f in info_dict.get('requested_formats', [])] or [info_dict['url']]
        ffpp = FFmpegPostProcessor(downloader=self)
@@ -472,6 +478,8 @@ class FFmpegFD(ExternalFD):
        self._debug_cmd(args)

        proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)
+        if url in ('-', 'pipe:'):
+            self.on_process_started(proc, proc.stdin)
        try:
            retval = proc.wait()
        except BaseException as e:
@@ -480,7 +488,7 @@ class FFmpegFD(ExternalFD):
            # produces a file that is playable (this is mostly useful for live
            # streams). Note that Windows is not affected and produces playable
            # files (see https://github.com/ytdl-org/youtube-dl/issues/8300).
-            if isinstance(e, KeyboardInterrupt) and sys.platform != 'win32':
+            if isinstance(e, KeyboardInterrupt) and sys.platform != 'win32' and url not in ('-', 'pipe:'):
                process_communicate_or_kill(proc, b'q')
            else:
                proc.kill()
--- a/yt_dlp/downloader/fragment.py
+++ b/yt_dlp/downloader/fragment.py
@@ -4,9 +4,26 @@ import os
 import time
 import json

+try:
+    from Crypto.Cipher import AES
+    can_decrypt_frag = True
+except ImportError:
+    can_decrypt_frag = False
+
+try:
+    import concurrent.futures
+    can_threaded_download = True
+except ImportError:
+    can_threaded_download = False
+
 from .common import FileDownloader
 from .http import HttpFD
+from ..compat import (
+    compat_urllib_error,
+    compat_struct_pack,
+)
 from ..utils import (
+    DownloadError,
    error_to_compat_str,
    encodeFilename,
    sanitize_open,
@@ -56,7 +73,7 @@ class FragmentFD(FileDownloader):

    def report_retry_fragment(self, err, frag_index, count, retries):
        self.to_screen(
-            '[download] Got server HTTP error: %s. Retrying fragment %d (attempt %d of %s) ...'
+            '\r[download] Got server HTTP error: %s. Retrying fragment %d (attempt %d of %s) ...'
            % (error_to_compat_str(err), frag_index, count, self.format_retries(retries)))

    def report_skip_fragment(self, frag_index):
@@ -112,11 +129,15 @@ class FragmentFD(FileDownloader):
            return False, None
        if fragment_info_dict.get('filetime'):
            ctx['fragment_filetime'] = fragment_info_dict.get('filetime')
-        down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
+        ctx['fragment_filename_sanitized'] = fragment_filename
+        return True, self._read_fragment(ctx)
+
+    def _read_fragment(self, ctx):
+        down, frag_sanitized = sanitize_open(ctx['fragment_filename_sanitized'], 'rb')
        ctx['fragment_filename_sanitized'] = frag_sanitized
        frag_content = down.read()
        down.close()
-        return True, frag_content
+        return frag_content

    def _append_fragment(self, ctx, frag_content):
        try:
@@ -304,3 +325,106 @@ class FragmentFD(FileDownloader):
            'tmpfilename': tmpfilename,
            'fragment_index': 0,
        })
+
+    def download_and_append_fragments(self, ctx, fragments, info_dict, pack_func=None):
+        fragment_retries = self.params.get('fragment_retries', 0)
+        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
+        test = self.params.get('test', False)
+        if not pack_func:
+            pack_func = lambda frag_content, _: frag_content
+
+        def download_fragment(fragment, ctx):
+            frag_index = ctx['fragment_index'] = fragment['frag_index']
+            headers = info_dict.get('http_headers', {})
+            byte_range = fragment.get('byte_range')
+            if byte_range:
+                headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
+
+            # Never skip the first fragment
+            fatal = (fragment.get('index') or frag_index) == 0 or not skip_unavailable_fragments
+            count, frag_content = 0, None
+            while count <= fragment_retries:
+                try:
+                    success, frag_content = self._download_fragment(ctx, fragment['url'], info_dict, headers)
+                    if not success:
+                        return False, frag_index
+                    break
+                except compat_urllib_error.HTTPError as err:
+                    # Unavailable (possibly temporary) fragments may be served.
+                    # First we try to retry then either skip or abort.
+                    # See https://github.com/ytdl-org/youtube-dl/issues/10165,
+                    # https://github.com/ytdl-org/youtube-dl/issues/10448).
+                    count += 1
+                    if count <= fragment_retries:
+                        self.report_retry_fragment(err, frag_index, count, fragment_retries)
+                except DownloadError:
+                    # Don't retry fragment if error occurred during HTTP downloading
+                    # itself since it has own retry settings
+                    if not fatal:
+                        break
+                    raise
+
+            if count > fragment_retries:
+                if not fatal:
+                    return False, frag_index
+                ctx['dest_stream'].close()
+                self.report_error('Giving up after %s fragment retries' % fragment_retries)
+                return False, frag_index
+            return frag_content, frag_index
+
+        def decrypt_fragment(fragment, frag_content):
+            decrypt_info = fragment.get('decrypt_info')
+            if not decrypt_info or decrypt_info['METHOD'] != 'AES-128':
+                return frag_content
+            iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', fragment['media_sequence'])
+            decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+            # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
+            # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+            # not what it decrypts to.
+            if test:
+                return frag_content
+            return AES.new(decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+
+        def append_fragment(frag_content, frag_index, ctx):
+            if not frag_content:
+                fatal = frag_index == 1 or not skip_unavailable_fragments
+                if not fatal:
+                    self.report_skip_fragment(frag_index)
+                    return True
+                else:
+                    ctx['dest_stream'].close()
+                    self.report_error(
+                        'fragment %s not found, unable to continue' % frag_index)
+                    return False
+            self._append_fragment(ctx, pack_func(frag_content, frag_index))
+            return True
+
+        max_workers = self.params.get('concurrent_fragment_downloads', 1)
+        if can_threaded_download and max_workers > 1:
+
+            def _download_fragment(fragment):
+                try:
+                    ctx_copy = ctx.copy()
+                    frag_content, frag_index = download_fragment(fragment, ctx_copy)
+                    return fragment, frag_content, frag_index, ctx_copy.get('fragment_filename_sanitized')
+                except Exception:
+                    # Return immediately on exception so that it is raised in the main thread
+                    return
+
+            self.report_warning('The download speed shown is only of one thread. This is a known issue and patches are welcome')
+            with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
+                for fragment, frag_content, frag_index, frag_filename in pool.map(_download_fragment, fragments):
+                    ctx['fragment_filename_sanitized'] = frag_filename
+                    ctx['fragment_index'] = frag_index
+                    result = append_fragment(decrypt_fragment(fragment, frag_content), frag_index, ctx)
+                    if not result:
+                        return False
+        else:
+            for fragment in fragments:
+                frag_content, frag_index = download_fragment(fragment, ctx)
+                result = append_fragment(decrypt_fragment(fragment, frag_content), frag_index, ctx)
+                if not result:
+                    return False
+
+        self._finish_frag_download(ctx)
--- a/yt_dlp/downloader/hls.py
+++ b/yt_dlp/downloader/hls.py
@@ -1,32 +1,18 @@
 from __future__ import unicode_literals

-import errno
 import re
 import io
 import binascii
-try:
-    from Crypto.Cipher import AES
-    can_decrypt_frag = True
-except ImportError:
-    can_decrypt_frag = False
-try:
-    import concurrent.futures
-    can_threaded_download = True
-except ImportError:
-    can_threaded_download = False

 from ..downloader import _get_real_downloader
-from .fragment import FragmentFD
+from .fragment import FragmentFD, can_decrypt_frag
 from .external import FFmpegFD

 from ..compat import (
-    compat_urllib_error,
    compat_urlparse,
-    compat_struct_pack,
 )
 from ..utils import (
    parse_m3u8_attributes,
-    sanitize_open,
    update_url_query,
    bug_reports_message,
 )
@@ -151,10 +137,6 @@ class HlsFD(FragmentFD):

        extra_state = ctx.setdefault('extra_state', {})

-        fragment_retries = self.params.get('fragment_retries', 0)
-        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
-        test = self.params.get('test', False)
-
        format_index = info_dict.get('format_index')
        extra_query = None
        extra_param_to_segment_url = info_dict.get('extra_param_to_segment_url')
@@ -258,7 +240,7 @@ class HlsFD(FragmentFD):
                media_sequence += 1

        # We only download the first fragment during the test
-        if test:
+        if self.params.get('test', False):
            fragments = [fragments[0] if fragments else None]

        if real_downloader:
@@ -272,55 +254,6 @@ class HlsFD(FragmentFD):
            if not success:
                return False
        else:
-            def decrypt_fragment(fragment, frag_content):
-                decrypt_info = fragment['decrypt_info']
-                if decrypt_info['METHOD'] != 'AES-128':
-                    return frag_content
-                iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', fragment['media_sequence'])
-                decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                    self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-                # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
-                # not what it decrypts to.
-                if test:
-                    return frag_content
-                return AES.new(decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
-
-            def download_fragment(fragment):
-                frag_index = fragment['frag_index']
-                frag_url = fragment['url']
-                byte_range = fragment['byte_range']
-
-                ctx['fragment_index'] = frag_index
-
-                count = 0
-                headers = info_dict.get('http_headers', {})
-                if byte_range:
-                    headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
-                while count <= fragment_retries:
-                    try:
-                        success, frag_content = self._download_fragment(
-                            ctx, frag_url, info_dict, headers)
-                        if not success:
-                            return False, frag_index
-                        break
-                    except compat_urllib_error.HTTPError as err:
-                        # Unavailable (possibly temporary) fragments may be served.
-                        # First we try to retry then either skip or abort.
-                        # See https://github.com/ytdl-org/youtube-dl/issues/10165,
-                        # https://github.com/ytdl-org/youtube-dl/issues/10448).
-                        count += 1
-                        if count <= fragment_retries:
-                            self.report_retry_fragment(err, frag_index, count, fragment_retries)
-                if count > fragment_retries:
-                    ctx['dest_stream'].close()
-                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
-                    return False, frag_index
-
-                return decrypt_fragment(fragment, frag_content), frag_index
-
-            pack_fragment = lambda frag_content, _: frag_content
-
            if is_webvtt:
                def pack_fragment(frag_content, frag_index):
                    output = io.StringIO()
@@ -388,75 +321,7 @@ class HlsFD(FragmentFD):
                        block.write_into(output)

                    return output.getvalue().encode('utf-8')
-
-            def append_fragment(frag_content, frag_index):
-                fatal = frag_index == 1 or not skip_unavailable_fragments
-                if frag_content:
-                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
-                    try:
-                        file, frag_sanitized = sanitize_open(fragment_filename, 'rb')
-                        ctx['fragment_filename_sanitized'] = frag_sanitized
-                        file.close()
-                        frag_content = pack_fragment(frag_content, frag_index)
-                        self._append_fragment(ctx, frag_content)
-                        return True
-                    except EnvironmentError as ose:
-                        if ose.errno != errno.ENOENT:
-                            raise
-                        # FileNotFoundError
-                        if not fatal:
-                            self.report_skip_fragment(frag_index)
-                            return True
-                        else:
-                            ctx['dest_stream'].close()
-                            self.report_error(
-                                'fragment %s not found, unable to continue' % frag_index)
-                            return False
-                else:
-                    if not fatal:
-                        self.report_skip_fragment(frag_index)
-                        return True
-                    else:
-                        ctx['dest_stream'].close()
-                        self.report_error(
-                            'fragment %s not found, unable to continue' % frag_index)
-                        return False
-
-            max_workers = self.params.get('concurrent_fragment_downloads', 1)
-            if can_threaded_download and max_workers > 1:
-                self.report_warning('The download speed shown is only of one thread. This is a known issue')
-                _download_fragment = lambda f: (f, download_fragment(f)[1])
-                with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
-                    futures = [pool.submit(_download_fragment, fragment) for fragment in fragments]
-                    # timeout must be 0 to return instantly
-                    done, not_done = concurrent.futures.wait(futures, timeout=0)
-                    try:
-                        while not_done:
-                            # Check every 1 second for KeyboardInterrupt
-                            freshly_done, not_done = concurrent.futures.wait(not_done, timeout=1)
-                            done |= freshly_done
-                    except KeyboardInterrupt:
-                        for future in not_done:
-                            future.cancel()
-                        # timeout must be none to cancel
-                        concurrent.futures.wait(not_done, timeout=None)
-                        raise KeyboardInterrupt
-
-                for fragment, frag_index in map(lambda x: x.result(), futures):
-                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
-                    down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
-                    fragment['fragment_filename_sanitized'] = frag_sanitized
-                    frag_content = down.read()
-                    down.close()
-                    result = append_fragment(decrypt_fragment(fragment, frag_content), frag_index)
-                    if not result:
-                        return False
            else:
-                for fragment in fragments:
-                    frag_content, frag_index = download_fragment(fragment)
-                    result = append_fragment(frag_content, frag_index)
-                    if not result:
-                        return False
-
-            self._finish_frag_download(ctx)
+                pack_fragment = None
+            self.download_and_append_fragments(ctx, fragments, info_dict, pack_fragment)
        return True
--- a/yt_dlp/downloader/http.py
+++ b/yt_dlp/downloader/http.py
@@ -18,6 +18,7 @@ from ..utils import (
    int_or_none,
    sanitize_open,
    sanitized_Request,
+    ThrottledDownload,
    write_xattr,
    XAttrMetadataError,
    XAttrUnavailableError,
@@ -223,6 +224,7 @@ class HttpFD(FileDownloader):
            # measure time over whole while-loop, so slow_down() and best_block_size() work together properly
            now = None  # needed for slow_down() in the first loop run
            before = start  # start measuring
+            throttle_start = None

            def retry(e):
                to_stdout = ctx.tmpfilename == '-'
@@ -313,6 +315,18 @@ class HttpFD(FileDownloader):
                if data_len is not None and byte_counter == data_len:
                    break

+                if speed and speed < (self.params.get('throttledratelimit') or 0):
+                    # The speed must stay below the limit for 3 seconds
+                    # This prevents raising error when the speed temporarily goes down
+                    if throttle_start is None:
+                        throttle_start = now
+                    elif now - throttle_start > 3:
+                        if ctx.stream is not None and ctx.tmpfilename != '-':
+                            ctx.stream.close()
+                        raise ThrottledDownload()
+                else:
+                    throttle_start = None
+
            if not is_test and ctx.chunk_size and ctx.data_len is not None and byte_counter < ctx.data_len:
                ctx.resume_len = byte_counter
                # ctx.block_size = block_size
--- a/yt_dlp/downloader/mhtml.py
+++ b/yt_dlp/downloader/mhtml.py
@@ -0,0 +1,202 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import io
+import quopri
+import re
+import uuid
+
+from .fragment import FragmentFD
+from ..utils import (
+    escapeHTML,
+    formatSeconds,
+    srt_subtitles_timecode,
+    urljoin,
+)
+from ..version import __version__ as YT_DLP_VERSION
+
+
+class MhtmlFD(FragmentFD):
+    FD_NAME = 'mhtml'
+
+    _STYLESHEET = """\
+html, body {
+    margin: 0;
+    padding: 0;
+    height: 100vh;
+}
+
+html {
+    overflow-y: scroll;
+    scroll-snap-type: y mandatory;
+}
+
+body {
+    scroll-snap-type: y mandatory;
+    display: flex;
+    flex-flow: column;
+}
+
+body > figure {
+    max-width: 100vw;
+    max-height: 100vh;
+    scroll-snap-align: center;
+}
+
+body > figure > figcaption {
+    text-align: center;
+    height: 2.5em;
+}
+
+body > figure > img {
+    display: block;
+    margin: auto;
+    max-width: 100%;
+    max-height: calc(100vh - 5em);
+}
+"""
+    _STYLESHEET = re.sub(r'\s+', ' ', _STYLESHEET)
+    _STYLESHEET = re.sub(r'\B \B|(?<=[\w\-]) (?=[^\w\-])|(?<=[^\w\-]) (?=[\w\-])', '', _STYLESHEET)
+
+    @staticmethod
+    def _escape_mime(s):
+        return '=?utf-8?Q?' + (b''.join(
+            bytes((b,)) if b >= 0x20 else b'=%02X' % b
+            for b in quopri.encodestring(s.encode('utf-8'), header=True)
+        )).decode('us-ascii') + '?='
+
+    def _gen_cid(self, i, fragment, frag_boundary):
+        return '%u.%s@yt-dlp.github.io.invalid' % (i, frag_boundary)
+
+    def _gen_stub(self, *, fragments, frag_boundary, title):
+        output = io.StringIO()
+
+        output.write((
+            '<!DOCTYPE html>'
+            '<html>'
+            '<head>'
+            ''  '<meta name="generator" content="yt-dlp {version}">'
+            ''  '<title>{title}</title>'
+            ''  '<style>{styles}</style>'
+            '<body>'
+        ).format(
+            version=escapeHTML(YT_DLP_VERSION),
+            styles=self._STYLESHEET,
+            title=escapeHTML(title)
+        ))
+
+        t0 = 0
+        for i, frag in enumerate(fragments):
+            output.write('<figure>')
+            try:
+                t1 = t0 + frag['duration']
+                output.write((
+                    '<figcaption>Slide #{num}: {t0} – {t1} (duration: {duration})</figcaption>'
+                ).format(
+                    num=i + 1,
+                    t0=srt_subtitles_timecode(t0),
+                    t1=srt_subtitles_timecode(t1),
+                    duration=formatSeconds(frag['duration'], msec=True)
+                ))
+            except (KeyError, ValueError, TypeError):
+                t1 = None
+                output.write((
+                    '<figcaption>Slide #{num}</figcaption>'
+                ).format(num=i + 1))
+            output.write('<img src="cid:{cid}">'.format(
+                cid=self._gen_cid(i, frag, frag_boundary)))
+            output.write('</figure>')
+            t0 = t1
+
+        return output.getvalue()
+
+    def real_download(self, filename, info_dict):
+        fragment_base_url = info_dict.get('fragment_base_url')
+        fragments = info_dict['fragments'][:1] if self.params.get(
+            'test', False) else info_dict['fragments']
+        title = info_dict['title']
+        origin = info_dict['webpage_url']
+
+        ctx = {
+            'filename': filename,
+            'total_frags': len(fragments),
+        }
+
+        self._prepare_and_start_frag_download(ctx)
+
+        extra_state = ctx.setdefault('extra_state', {
+            'header_written': False,
+            'mime_boundary': str(uuid.uuid4()).replace('-', ''),
+        })
+
+        frag_boundary = extra_state['mime_boundary']
+
+        if not extra_state['header_written']:
+            stub = self._gen_stub(
+                fragments=fragments,
+                frag_boundary=frag_boundary,
+                title=title
+            )
+
+            ctx['dest_stream'].write((
+                'MIME-Version: 1.0\r\n'
+                'From: <nowhere@yt-dlp.github.io.invalid>\r\n'
+                'To: <nowhere@yt-dlp.github.io.invalid>\r\n'
+                'Subject: {title}\r\n'
+                'Content-type: multipart/related; '
+                ''  'boundary="{boundary}"; '
+                ''  'type="text/html"\r\n'
+                'X.yt-dlp.Origin: {origin}\r\n'
+                '\r\n'
+                '--{boundary}\r\n'
+                'Content-Type: text/html; charset=utf-8\r\n'
+                'Content-Length: {length}\r\n'
+                '\r\n'
+                '{stub}\r\n'
+            ).format(
+                origin=origin,
+                boundary=frag_boundary,
+                length=len(stub),
+                title=self._escape_mime(title),
+                stub=stub
+            ).encode('utf-8'))
+            extra_state['header_written'] = True
+
+        for i, fragment in enumerate(fragments):
+            if (i + 1) <= ctx['fragment_index']:
+                continue
+
+            fragment_url = urljoin(fragment_base_url, fragment['path'])
+            success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
+            if not success:
+                continue
+
+            mime_type = b'image/jpeg'
+            if frag_content.startswith(b'\x89PNG\r\n\x1a\n'):
+                mime_type = b'image/png'
+            if frag_content.startswith((b'GIF87a', b'GIF89a')):
+                mime_type = b'image/gif'
+            if frag_content.startswith(b'RIFF') and frag_content[8:12] == 'WEBP':
+                mime_type = b'image/webp'
+
+            frag_header = io.BytesIO()
+            frag_header.write(
+                b'--%b\r\n' % frag_boundary.encode('us-ascii'))
+            frag_header.write(
+                b'Content-ID: <%b>\r\n' % self._gen_cid(i, fragment, frag_boundary).encode('us-ascii'))
+            frag_header.write(
+                b'Content-type: %b\r\n' % mime_type)
+            frag_header.write(
+                b'Content-length: %u\r\n' % len(frag_content))
+            frag_header.write(
+                b'Content-location: %b\r\n' % fragment_url.encode('us-ascii'))
+            frag_header.write(
+                b'X.yt-dlp.Duration: %f\r\n' % fragment['duration'])
+            frag_header.write(b'\r\n')
+            self._append_fragment(
+                ctx, frag_header.getvalue() + frag_content + b'\r\n')
+
+        ctx['dest_stream'].write(
+            b'--%b--\r\n\r\n' % frag_boundary.encode('us-ascii'))
+        self._finish_frag_download(ctx)
+        return True
--- a/yt_dlp/downloader/websocket.py
+++ b/yt_dlp/downloader/websocket.py
@@ -0,0 +1,59 @@
+import os
+import signal
+import asyncio
+import threading
+
+try:
+    import websockets
+    has_websockets = True
+except ImportError:
+    has_websockets = False
+
+from .common import FileDownloader
+from .external import FFmpegFD
+
+
+class FFmpegSinkFD(FileDownloader):
+    """ A sink to ffmpeg for downloading fragments in any form """
+
+    def real_download(self, filename, info_dict):
+        info_copy = info_dict.copy()
+        info_copy['url'] = '-'
+
+        async def call_conn(proc, stdin):
+            try:
+                await self.real_connection(stdin, info_dict)
+            except (BrokenPipeError, OSError):
+                pass
+            finally:
+                try:
+                    stdin.flush()
+                    stdin.close()
+                except OSError:
+                    pass
+                os.kill(os.getpid(), signal.SIGINT)
+
+        class FFmpegStdinFD(FFmpegFD):
+            @classmethod
+            def get_basename(cls):
+                return FFmpegFD.get_basename()
+
+            def on_process_started(self, proc, stdin):
+                thread = threading.Thread(target=asyncio.run, daemon=True, args=(call_conn(proc, stdin), ))
+                thread.start()
+
+        return FFmpegStdinFD(self.ydl, self.params or {}).download(filename, info_copy)
+
+    async def real_connection(self, sink, info_dict):
+        """ Override this in subclasses """
+        raise NotImplementedError('This method must be implemented by subclasses')
+
+
+class WebSocketFragmentFD(FFmpegSinkFD):
+    async def real_connection(self, sink, info_dict):
+        async with websockets.connect(info_dict['url'], extra_headers=info_dict.get('http_headers', {})) as ws:
+            while True:
+                recv = await ws.recv()
+                if isinstance(recv, str):
+                    recv = recv.encode('utf8')
+                sink.write(recv)
--- a/yt_dlp/downloader/youtube_live_chat.py
+++ b/yt_dlp/downloader/youtube_live_chat.py
@@ -1,20 +1,23 @@
 from __future__ import division, unicode_literals

 import json
+import time

 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
 from ..utils import (
    try_get,
+    dict_get,
+    int_or_none,
    RegexNotFoundError,
 )
 from ..extractor.youtube import YoutubeBaseInfoExtractor as YT_BaseIE


-class YoutubeLiveChatReplayFD(FragmentFD):
-    """ Downloads YouTube live chat replays fragment by fragment """
+class YoutubeLiveChatFD(FragmentFD):
+    """ Downloads YouTube live chats fragment by fragment """

-    FD_NAME = 'youtube_live_chat_replay'
+    FD_NAME = 'youtube_live_chat'

    def real_download(self, filename, info_dict):
        video_id = info_dict['video_id']
@@ -31,6 +34,8 @@ class YoutubeLiveChatReplayFD(FragmentFD):

        ie = YT_BaseIE(self.ydl)

+        start_time = int(time.time() * 1000)
+
        def dl_fragment(url, data=None, headers=None):
            http_headers = info_dict.get('http_headers', {})
            if headers:
@@ -38,36 +43,70 @@ class YoutubeLiveChatReplayFD(FragmentFD):
                http_headers.update(headers)
            return self._download_fragment(ctx, url, info_dict, http_headers, data)

-        def download_and_parse_fragment(url, frag_index, request_data):
+        def parse_actions_replay(live_chat_continuation):
+            offset = continuation_id = None
+            processed_fragment = bytearray()
+            for action in live_chat_continuation.get('actions', []):
+                if 'replayChatItemAction' in action:
+                    replay_chat_item_action = action['replayChatItemAction']
+                    offset = int(replay_chat_item_action['videoOffsetTimeMsec'])
+                processed_fragment.extend(
+                    json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
+            if offset is not None:
+                continuation_id = try_get(
+                    live_chat_continuation,
+                    lambda x: x['continuations'][0]['liveChatReplayContinuationData']['continuation'])
+            self._append_fragment(ctx, processed_fragment)
+            return continuation_id, offset
+
+        live_offset = 0
+
+        def parse_actions_live(live_chat_continuation):
+            nonlocal live_offset
+            continuation_id = None
+            processed_fragment = bytearray()
+            for action in live_chat_continuation.get('actions', []):
+                timestamp = self.parse_live_timestamp(action)
+                if timestamp is not None:
+                    live_offset = timestamp - start_time
+                # compatibility with replay format
+                pseudo_action = {
+                    'replayChatItemAction': {'actions': [action]},
+                    'videoOffsetTimeMsec': str(live_offset),
+                    'isLive': True,
+                }
+                processed_fragment.extend(
+                    json.dumps(pseudo_action, ensure_ascii=False).encode('utf-8') + b'\n')
+            continuation_data_getters = [
+                lambda x: x['continuations'][0]['invalidationContinuationData'],
+                lambda x: x['continuations'][0]['timedContinuationData'],
+            ]
+            continuation_data = try_get(live_chat_continuation, continuation_data_getters, dict)
+            if continuation_data:
+                continuation_id = continuation_data.get('continuation')
+                timeout_ms = int_or_none(continuation_data.get('timeoutMs'))
+                if timeout_ms is not None:
+                    time.sleep(timeout_ms / 1000)
+            self._append_fragment(ctx, processed_fragment)
+            return continuation_id, live_offset
+
+        if info_dict['protocol'] == 'youtube_live_chat_replay':
+            parse_actions = parse_actions_replay
+        elif info_dict['protocol'] == 'youtube_live_chat':
+            parse_actions = parse_actions_live
+
+        def download_and_parse_fragment(url, frag_index, request_data, headers):
            count = 0
            while count <= fragment_retries:
                try:
-                    success, raw_fragment = dl_fragment(url, request_data, {'content-type': 'application/json'})
+                    success, raw_fragment = dl_fragment(url, request_data, headers)
                    if not success:
                        return False, None, None
-                    try:
-                        data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
-                    except RegexNotFoundError:
-                        data = None
-                    if not data:
-                        data = json.loads(raw_fragment)
+                    data = json.loads(raw_fragment)
                    live_chat_continuation = try_get(
                        data,
                        lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
-                    offset = continuation_id = None
-                    processed_fragment = bytearray()
-                    for action in live_chat_continuation.get('actions', []):
-                        if 'replayChatItemAction' in action:
-                            replay_chat_item_action = action['replayChatItemAction']
-                            offset = int(replay_chat_item_action['videoOffsetTimeMsec'])
-                        processed_fragment.extend(
-                            json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
-                    if offset is not None:
-                        continuation_id = try_get(
-                            live_chat_continuation,
-                            lambda x: x['continuations'][0]['liveChatReplayContinuationData']['continuation'])
-                    self._append_fragment(ctx, processed_fragment)
-
+                    continuation_id, offset = parse_actions(live_chat_continuation)
                    return True, continuation_id, offset
                except compat_urllib_error.HTTPError as err:
                    count += 1
@@ -100,7 +139,11 @@ class YoutubeLiveChatReplayFD(FragmentFD):
        innertube_context = try_get(ytcfg, lambda x: x['INNERTUBE_CONTEXT'])
        if not api_key or not innertube_context:
            return False
-        url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
+        visitor_data = try_get(innertube_context, lambda x: x['client']['visitorData'], str)
+        if info_dict['protocol'] == 'youtube_live_chat_replay':
+            url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
+        elif info_dict['protocol'] == 'youtube_live_chat':
+            url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat?key=' + api_key

        frag_index = offset = 0
        while continuation_id is not None:
@@ -111,8 +154,11 @@ class YoutubeLiveChatReplayFD(FragmentFD):
            }
            if frag_index > 1:
                request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
+            headers = ie._generate_api_headers(ytcfg, visitor_data=visitor_data)
+            headers.update({'content-type': 'application/json'})
+            fragment_request_data = json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n'
            success, continuation_id, offset = download_and_parse_fragment(
-                url, frag_index, json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n')
+                url, frag_index, fragment_request_data, headers)
            if not success:
                return False
            if test:
@@ -120,3 +166,39 @@ class YoutubeLiveChatReplayFD(FragmentFD):

        self._finish_frag_download(ctx)
        return True
+
+    @staticmethod
+    def parse_live_timestamp(action):
+        action_content = dict_get(
+            action,
+            ['addChatItemAction', 'addLiveChatTickerItemAction', 'addBannerToLiveChatCommand'])
+        if not isinstance(action_content, dict):
+            return None
+        item = dict_get(action_content, ['item', 'bannerRenderer'])
+        if not isinstance(item, dict):
+            return None
+        renderer = dict_get(item, [
+            # text
+            'liveChatTextMessageRenderer', 'liveChatPaidMessageRenderer',
+            'liveChatMembershipItemRenderer', 'liveChatPaidStickerRenderer',
+            # ticker
+            'liveChatTickerPaidMessageItemRenderer',
+            'liveChatTickerSponsorItemRenderer',
+            # banner
+            'liveChatBannerRenderer',
+        ])
+        if not isinstance(renderer, dict):
+            return None
+        parent_item_getters = [
+            lambda x: x['showItemEndpoint']['showLiveChatItemEndpoint']['renderer'],
+            lambda x: x['contents'],
+        ]
+        parent_item = try_get(renderer, parent_item_getters, dict)
+        if parent_item:
+            renderer = dict_get(parent_item, [
+                'liveChatTextMessageRenderer', 'liveChatPaidMessageRenderer',
+                'liveChatMembershipItemRenderer', 'liveChatPaidStickerRenderer',
+            ])
+            if not isinstance(renderer, dict):
+                return None
+        return int_or_none(renderer.get('timestampUsec'), 1000)
--- a/yt_dlp/extractor/appleconnect.py
+++ b/yt_dlp/extractor/appleconnect.py
@@ -9,10 +9,10 @@ from ..utils import (


 class AppleConnectIE(InfoExtractor):
-    _VALID_URL = r'https?://itunes\.apple\.com/\w{0,2}/?post/idsa\.(?P<id>[\w-]+)'
-    _TEST = {
+    _VALID_URL = r'https?://itunes\.apple\.com/\w{0,2}/?post/(?:id)?sa\.(?P<id>[\w-]+)'
+    _TESTS = [{
        'url': 'https://itunes.apple.com/us/post/idsa.4ab17a39-2720-11e5-96c5-a5b38f6c42d3',
-        'md5': 'e7c38568a01ea45402570e6029206723',
+        'md5': 'c1d41f72c8bcaf222e089434619316e4',
        'info_dict': {
            'id': '4ab17a39-2720-11e5-96c5-a5b38f6c42d3',
            'ext': 'm4v',
@@ -22,7 +22,10 @@ class AppleConnectIE(InfoExtractor):
            'upload_date': '20150710',
            'timestamp': 1436545535,
        },
-    }
+    }, {
+        'url': 'https://itunes.apple.com/us/post/sa.0fe0229f-2457-11e5-9f40-1bb645f2d5d9',
+        'only_matching': True,
+    }]

    def _real_extract(self, url):
        video_id = self._match_id(url)
@@ -36,7 +39,7 @@ class AppleConnectIE(InfoExtractor):

        video_data = self._parse_json(video_json, video_id)
        timestamp = str_to_int(self._html_search_regex(r'data-timestamp="(\d+)"', webpage, 'timestamp'))
-        like_count = str_to_int(self._html_search_regex(r'(\d+) Loves', webpage, 'like count'))
+        like_count = str_to_int(self._html_search_regex(r'(\d+) Loves', webpage, 'like count', default=None))

        return {
            'id': video_id,
--- a/yt_dlp/extractor/bilibili.py
+++ b/yt_dlp/extractor/bilibili.py
@@ -281,7 +281,7 @@ class BiliBiliIE(InfoExtractor):
            webpage)
        if uploader_mobj:
            info.update({
-                'uploader': uploader_mobj.group('name'),
+                'uploader': uploader_mobj.group('name').strip(),
                'uploader_id': uploader_mobj.group('id'),
            })

--- a/yt_dlp/extractor/canvas.py
+++ b/yt_dlp/extractor/canvas.py
@@ -24,7 +24,7 @@ class CanvasIE(InfoExtractor):
    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza|dako)/assets/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://mediazone.vrt.be/api/v1/ketnet/assets/md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
-        'md5': '68993eda72ef62386a15ea2cf3c93107',
+        'md5': '37b2b7bb9b3dcaa05b67058dc3a714a9',
        'info_dict': {
            'id': 'md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
            'display_id': 'md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
@@ -32,9 +32,9 @@ class CanvasIE(InfoExtractor):
            'title': 'Nachtwacht: De Greystook',
            'description': 'Nachtwacht: De Greystook',
            'thumbnail': r're:^https?://.*\.jpg$',
-            'duration': 1468.04,
+            'duration': 1468.02,
        },
-        'expected_warnings': ['is not a supported codec', 'Unknown MIME type'],
+        'expected_warnings': ['is not a supported codec'],
    }, {
        'url': 'https://mediazone.vrt.be/api/v1/canvas/assets/mz-ast-5e5f90b6-2d72-4c40-82c2-e134f884e93e',
        'only_matching': True,
--- a/yt_dlp/extractor/common.py
+++ b/yt_dlp/extractor/common.py
@@ -290,6 +290,7 @@ class InfoExtractor(object):
    categories:     A list of categories that the video falls in, for example
                    ["Sports", "Berlin"]
    tags:           A list of tags assigned to the video, e.g. ["sweden", "pop music"]
+    cast:           A list of the video cast
    is_live:        True, False, or None (=unknown). Whether this video is a
                    live stream that goes on instead of a fixed-length video.
    was_live:       True, False, or None (=unknown). Whether this video was
@@ -1473,7 +1474,7 @@ class InfoExtractor(object):
    class FormatSort:
        regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<separator>[~:])(?P<limit>.*?))?)? *$'

-        default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
+        default = ('hidden', 'aud_or_vid', 'hasvid', 'ie_pref', 'lang', 'quality',
                   'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
                   'proto', 'ext', 'hasaud', 'source', 'format_id')  # These must not be aliases
        ytdl_default = ('hasaud', 'quality', 'tbr', 'filesize', 'vbr',
@@ -1486,7 +1487,7 @@ class InfoExtractor(object):
            'acodec': {'type': 'ordered', 'regex': True,
                       'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
            'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
-                      'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
+                      'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', '.*dash', 'ws|websocket', '', 'mms|rtsp', 'none', 'f4']},
            'vext': {'type': 'ordered', 'field': 'video_ext',
                     'order': ('mp4', 'webm', 'flv', '', 'none'),
                     'order_free': ('webm', 'mp4', 'flv', '', 'none')},
@@ -1494,6 +1495,9 @@ class InfoExtractor(object):
                     'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
                     'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
            'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
+            'aud_or_vid': {'visible': False, 'forced': True, 'type': 'multiple', 'default': 1,
+                           'field': ('vcodec', 'acodec'),
+                           'function': lambda it: int(any(v != 'none' for v in it))},
            'ie_pref': {'priority': True, 'type': 'extractor'},
            'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
            'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
@@ -1701,9 +1705,7 @@ class InfoExtractor(object):

                def wrapped_function(values):
                    values = tuple(filter(lambda x: x is not None, values))
-                    return (self._get_field_setting(field, 'function')(*values) if len(values) > 1
-                            else values[0] if values
-                            else None)
+                    return self._get_field_setting(field, 'function')(values) if values else None

                value = wrapped_function((get_value(f) for f in actual_fields))
            else:
@@ -1719,7 +1721,7 @@ class InfoExtractor(object):
            if not format.get('ext') and 'url' in format:
                format['ext'] = determine_ext(format['url'])
            if format.get('vcodec') == 'none':
-                format['audio_ext'] = format['ext']
+                format['audio_ext'] = format['ext'] if format.get('acodec') != 'none' else 'none'
                format['video_ext'] = 'none'
            else:
                format['video_ext'] = format['ext']
@@ -2125,6 +2127,7 @@ class InfoExtractor(object):
                        format_id.append(str(format_index))
                    f = {
                        'format_id': '-'.join(format_id),
+                        'format_note': name,
                        'format_index': format_index,
                        'url': manifest_url,
                        'manifest_url': m3u8_url,
@@ -2636,7 +2639,7 @@ class InfoExtractor(object):
                    mime_type = representation_attrib['mimeType']
                    content_type = representation_attrib.get('contentType', mime_type.split('/')[0])

-                    if content_type in ('video', 'audio', 'text'):
+                    if content_type in ('video', 'audio', 'text') or mime_type == 'image/jpeg':
                        base_url = ''
                        for element in (representation, adaptation_set, period, mpd_doc):
                            base_url_e = element.find(_add_ns('BaseURL'))
@@ -2653,9 +2656,15 @@ class InfoExtractor(object):
                        url_el = representation.find(_add_ns('BaseURL'))
                        filesize = int_or_none(url_el.attrib.get('{http://youtube.com/yt/2012/10/10}contentLength') if url_el is not None else None)
                        bandwidth = int_or_none(representation_attrib.get('bandwidth'))
+                        if representation_id is not None:
+                            format_id = representation_id
+                        else:
+                            format_id = content_type
+                        if mpd_id:
+                            format_id = mpd_id + '-' + format_id
                        if content_type in ('video', 'audio'):
                            f = {
-                                'format_id': '%s-%s' % (mpd_id, representation_id) if mpd_id else representation_id,
+                                'format_id': format_id,
                                'manifest_url': mpd_url,
                                'ext': mimetype2ext(mime_type),
                                'width': int_or_none(representation_attrib.get('width')),
@@ -2675,6 +2684,17 @@ class InfoExtractor(object):
                                'manifest_url': mpd_url,
                                'filesize': filesize,
                            }
+                        elif mime_type == 'image/jpeg':
+                            # See test case in VikiIE
+                            # https://www.viki.com/videos/1175236v-choosing-spouse-by-lottery-episode-1
+                            f = {
+                                'format_id': format_id,
+                                'ext': 'mhtml',
+                                'manifest_url': mpd_url,
+                                'format_note': 'DASH storyboards (jpeg)',
+                                'acodec': 'none',
+                                'vcodec': 'none',
+                            }
                        representation_ms_info = extract_multisegment_info(representation, adaption_set_ms_info)

                        def prepare_template(template_name, identifiers):
@@ -2693,7 +2713,8 @@ class InfoExtractor(object):
                                    t += c
                            # Next, $...$ templates are translated to their
                            # %(...) counterparts to be used with % operator
-                            t = t.replace('$RepresentationID$', representation_id)
+                            if representation_id is not None:
+                                t = t.replace('$RepresentationID$', representation_id)
                            t = re.sub(r'\$(%s)\$' % '|'.join(identifiers), r'%(\1)d', t)
                            t = re.sub(r'\$(%s)%%([^$]+)\$' % '|'.join(identifiers), r'%(\1)\2', t)
                            t.replace('$$', '$')
@@ -2810,7 +2831,7 @@ class InfoExtractor(object):
                                'url': mpd_url or base_url,
                                'fragment_base_url': base_url,
                                'fragments': [],
-                                'protocol': 'http_dash_segments',
+                                'protocol': 'http_dash_segments' if mime_type != 'image/jpeg' else 'mhtml',
                            })
                            if 'initialization_url' in representation_ms_info:
                                initialization_url = representation_ms_info['initialization_url']
@@ -2821,7 +2842,7 @@ class InfoExtractor(object):
                        else:
                            # Assuming direct URL to unfragmented media.
                            f['url'] = base_url
-                        if content_type in ('video', 'audio'):
+                        if content_type in ('video', 'audio') or mime_type == 'image/jpeg':
                            formats.append(f)
                        elif content_type == 'text':
                            subtitles.setdefault(lang or 'und', []).append(f)
--- a/yt_dlp/extractor/curiositystream.py
+++ b/yt_dlp/extractor/curiositystream.py
@@ -143,9 +143,9 @@ class CuriosityStreamIE(CuriosityStreamBaseIE):
        }


-class CuriosityStreamCollectionsIE(CuriosityStreamBaseIE):
-    IE_NAME = 'curiositystream:collections'
-    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/collections/(?P<id>\d+)'
+class CuriosityStreamCollectionIE(CuriosityStreamBaseIE):
+    IE_NAME = 'curiositystream:collection'
+    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/(?:collections?|series)/(?P<id>\d+)'
    _API_BASE_URL = 'https://api.curiositystream.com/v2/collections/'
    _TESTS = [{
        'url': 'https://curiositystream.com/collections/86',
@@ -155,6 +155,20 @@ class CuriosityStreamCollectionsIE(CuriosityStreamBaseIE):
            'description': 'Wondering where to start? Here are a few of our favorite series and films... from our couch to yours.',
        },
        'playlist_mincount': 7,
+    }, {
+        'url': 'https://app.curiositystream.com/collection/2',
+        'info_dict': {
+            'id': '2',
+            'title': 'Curious Minds: The Internet',
+            'description': 'How is the internet shaping our lives in the 21st Century?',
+        },
+        'playlist_mincount': 16,
+    }, {
+        'url': 'https://curiositystream.com/series/2',
+        'only_matching': True,
+    }, {
+        'url': 'https://curiositystream.com/collections/36',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
@@ -163,25 +177,10 @@ class CuriosityStreamCollectionsIE(CuriosityStreamBaseIE):
        entries = []
        for media in collection.get('media', []):
            media_id = compat_str(media.get('id'))
-            media_type, ie = ('series', CuriosityStreamSeriesIE) if media.get('is_collection') else ('video', CuriosityStreamIE)
+            media_type, ie = ('series', CuriosityStreamCollectionIE) if media.get('is_collection') else ('video', CuriosityStreamIE)
            entries.append(self.url_result(
                'https://curiositystream.com/%s/%s' % (media_type, media_id),
                ie=ie.ie_key(), video_id=media_id))
        return self.playlist_result(
            entries, collection_id,
            collection.get('title'), collection.get('description'))
-
-
-class CuriosityStreamSeriesIE(CuriosityStreamCollectionsIE):
-    IE_NAME = 'curiositystream:series'
-    _VALID_URL = r'https?://(?:app\.)?curiositystream\.com/series/(?P<id>\d+)'
-    _API_BASE_URL = 'https://api.curiositystream.com/v2/series/'
-    _TESTS = [{
-        'url': 'https://app.curiositystream.com/series/2',
-        'info_dict': {
-            'id': '2',
-            'title': 'Curious Minds: The Internet',
-            'description': 'How is the internet shaping our lives in the 21st Century?',
-        },
-        'playlist_mincount': 16,
-    }]
--- a/yt_dlp/extractor/egghead.py
+++ b/yt_dlp/extractor/egghead.py
@@ -22,16 +22,19 @@ class EggheadBaseIE(InfoExtractor):
 class EggheadCourseIE(EggheadBaseIE):
    IE_DESC = 'egghead.io course'
    IE_NAME = 'egghead:course'
-    _VALID_URL = r'https://egghead\.io/courses/(?P<id>[^/?#&]+)'
-    _TEST = {
+    _VALID_URL = r'https://(?:app\.)?egghead\.io/(?:course|playlist)s/(?P<id>[^/?#&]+)'
+    _TESTS = [{
        'url': 'https://egghead.io/courses/professor-frisby-introduces-composable-functional-javascript',
        'playlist_count': 29,
        'info_dict': {
-            'id': '72',
+            'id': '432655',
            'title': 'Professor Frisby Introduces Composable Functional JavaScript',
            'description': 're:(?s)^This course teaches the ubiquitous.*You\'ll start composing functionality before you know it.$',
        },
-    }
+    }, {
+        'url': 'https://app.egghead.io/playlists/professor-frisby-introduces-composable-functional-javascript',
+        'only_matching': True,
+    }]

    def _real_extract(self, url):
        playlist_id = self._match_id(url)
@@ -65,7 +68,7 @@ class EggheadCourseIE(EggheadBaseIE):
 class EggheadLessonIE(EggheadBaseIE):
    IE_DESC = 'egghead.io lesson'
    IE_NAME = 'egghead:lesson'
-    _VALID_URL = r'https://egghead\.io/(?:api/v1/)?lessons/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https://(?:app\.)?egghead\.io/(?:api/v1/)?lessons/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://egghead.io/lessons/javascript-linear-data-flow-with-container-style-types-box',
        'info_dict': {
@@ -88,6 +91,9 @@ class EggheadLessonIE(EggheadBaseIE):
    }, {
        'url': 'https://egghead.io/api/v1/lessons/react-add-redux-to-a-react-application',
        'only_matching': True,
+    }, {
+        'url': 'https://app.egghead.io/lessons/javascript-linear-data-flow-with-container-style-types-box',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
--- a/yt_dlp/extractor/extractors.py
+++ b/yt_dlp/extractor/extractors.py
@@ -291,8 +291,7 @@ from .ctvnews import CTVNewsIE
 from .cultureunplugged import CultureUnpluggedIE
 from .curiositystream import (
    CuriosityStreamIE,
-    CuriosityStreamCollectionsIE,
-    CuriosityStreamSeriesIE,
+    CuriosityStreamCollectionIE,
 )
 from .cwtv import CWTVIE
 from .dailymail import DailyMailIE
@@ -655,10 +654,6 @@ from .linkedin import (
 from .linuxacademy import LinuxAcademyIE
 from .litv import LiTVIE
 from .livejournal import LiveJournalIE
-from .liveleak import (
-    LiveLeakIE,
-    LiveLeakEmbedIE,
-)
 from .livestream import (
    LivestreamIE,
    LivestreamOriginalIE,
--- a/yt_dlp/extractor/funimation.py
+++ b/yt_dlp/extractor/funimation.py
@@ -10,8 +10,9 @@ from ..utils import (
    determine_ext,
    int_or_none,
    js_to_json,
+    urlencode_postdata,
+    urljoin,
    ExtractorError,
-    urlencode_postdata
 )


@@ -109,6 +110,7 @@ class FunimationIE(InfoExtractor):
        if series:
            title = '%s - %s' % (series, title)
        description = self._html_search_meta(['description', 'og:description'], webpage, fatal=True)
+        subtitles = self.extract_subtitles(url, video_id, display_id)

        try:
            headers = {}
@@ -153,6 +155,24 @@ class FunimationIE(InfoExtractor):
            'season_number': int_or_none(title_data.get('seasonNum') or _search_kane('season')),
            'episode_number': int_or_none(title_data.get('episodeNum')),
            'episode': episode,
+            'subtitles': subtitles,
            'season_id': title_data.get('seriesId'),
            'formats': formats,
        }
+
+    def _get_subtitles(self, url, video_id, display_id):
+        player_url = urljoin(url, '/player/' + video_id)
+        player_page = self._download_webpage(player_url, display_id)
+        text_tracks_json_string = self._search_regex(
+            r'"textTracks": (\[{.+?}\])',
+            player_page, 'subtitles data', default='')
+        text_tracks = self._parse_json(
+            text_tracks_json_string, display_id, js_to_json, fatal=False) or []
+        subtitles = {}
+        for text_track in text_tracks:
+            url_element = {'url': text_track.get('src')}
+            language = text_track.get('language')
+            if text_track.get('type') == 'CC':
+                language += '_CC'
+            subtitles.setdefault(language, []).append(url_element)
+        return subtitles
--- a/yt_dlp/extractor/generic.py
+++ b/yt_dlp/extractor/generic.py
@@ -84,7 +84,6 @@ from .jwplatform import JWPlatformIE
 from .digiteka import DigitekaIE
 from .arkena import ArkenaIE
 from .instagram import InstagramIE
-from .liveleak import LiveLeakIE
 from .threeqsdn import ThreeQSDNIE
 from .theplatform import ThePlatformIE
 from .kaltura import KalturaIE
@@ -1632,31 +1631,6 @@ class GenericIE(InfoExtractor):
                'upload_date': '20160409',
            },
        },
-        # LiveLeak embed
-        {
-            'url': 'http://www.wykop.pl/link/3088787/',
-            'md5': '7619da8c820e835bef21a1efa2a0fc71',
-            'info_dict': {
-                'id': '874_1459135191',
-                'ext': 'mp4',
-                'title': 'Man shows poor quality of new apartment building',
-                'description': 'The wall is like a sand pile.',
-                'uploader': 'Lake8737',
-            },
-            'add_ie': [LiveLeakIE.ie_key()],
-        },
-        # Another LiveLeak embed pattern (#13336)
-        {
-            'url': 'https://milo.yiannopoulos.net/2017/06/concealed-carry-robbery/',
-            'info_dict': {
-                'id': '2eb_1496309988',
-                'ext': 'mp4',
-                'title': 'Thief robs place where everyone was armed',
-                'description': 'md5:694d73ee79e535953cf2488562288eee',
-                'uploader': 'brazilwtf',
-            },
-            'add_ie': [LiveLeakIE.ie_key()],
-        },
        # Duplicated embedded video URLs
        {
            'url': 'http://www.hudl.com/athlete/2538180/highlights/149298443',
@@ -3204,11 +3178,6 @@ class GenericIE(InfoExtractor):
            return self.url_result(
                self._proto_relative_url(instagram_embed_url), InstagramIE.ie_key())

-        # Look for LiveLeak embeds
-        liveleak_urls = LiveLeakIE._extract_urls(webpage)
-        if liveleak_urls:
-            return self.playlist_from_matches(liveleak_urls, video_id, video_title)
-
        # Look for 3Q SDN embeds
        threeqsdn_url = ThreeQSDNIE._extract_url(webpage)
        if threeqsdn_url:
--- a/yt_dlp/extractor/hotstar.py
+++ b/yt_dlp/extractor/hotstar.py
@@ -27,8 +27,8 @@ from ..utils import (
 class HotStarBaseIE(InfoExtractor):
    _AKAMAI_ENCRYPTION_KEY = b'\x05\xfc\x1a\x01\xca\xc9\x4b\xc4\x12\xfc\x53\x12\x07\x75\xf9\xee'

-    def _call_api_impl(self, path, video_id, query):
-        st = int(time.time())
+    def _call_api_impl(self, path, video_id, query, st=None):
+        st = int_or_none(st) or int(time.time())
        exp = st + 6000
        auth = 'st=%d~exp=%d~acl=/*' % (st, exp)
        auth += '~hmac=' + hmac.new(self._AKAMAI_ENCRYPTION_KEY, auth.encode(), hashlib.sha256).hexdigest()
@@ -75,9 +75,9 @@ class HotStarBaseIE(InfoExtractor):
            'tas': 10000,
        })

-    def _call_api_v2(self, path, video_id):
+    def _call_api_v2(self, path, video_id, st=None):
        return self._call_api_impl(
-            '%s/content/%s' % (path, video_id), video_id, {
+            '%s/content/%s' % (path, video_id), video_id, st=st, query={
                'desired-config': 'audio_channel:stereo|dynamic_range:sdr|encryption:plain|ladder:tv|package:dash|resolution:hd|subs-tag:HotstarVIP|video_codec:vp9',
                'device-id': compat_str(uuid.uuid4()),
                'os-name': 'Windows',
@@ -131,7 +131,8 @@ class HotStarIE(HotStarBaseIE):
    def _real_extract(self, url):
        video_id = self._match_id(url)

-        webpage = self._download_webpage(url, video_id)
+        webpage, urlh = self._download_webpage_handle(url, video_id)
+        st = urlh.headers.get('x-origin-date')
        app_state = self._parse_json(self._search_regex(
            r'<script>window\.APP_STATE\s*=\s*({.+?})</script>',
            webpage, 'app state'), video_id)
@@ -155,7 +156,7 @@ class HotStarIE(HotStarBaseIE):
        formats = []
        geo_restricted = False
        # change to v2 in the future
-        playback_sets = self._call_api_v2('play/v1/playback', video_id)['playBackSets']
+        playback_sets = self._call_api_v2('play/v1/playback', video_id, st=st)['playBackSets']
        for playback_set in playback_sets:
            if not isinstance(playback_set, dict):
                continue
--- a/yt_dlp/extractor/liveleak.py
+++ b/yt_dlp/extractor/liveleak.py
@@ -1,191 +0,0 @@
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import int_or_none
-
-
-class LiveLeakIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:\w+\.)?liveleak\.com/view\?.*?\b[it]=(?P<id>[\w_]+)'
-    _TESTS = [{
-        'url': 'http://www.liveleak.com/view?i=757_1364311680',
-        'md5': '0813c2430bea7a46bf13acf3406992f4',
-        'info_dict': {
-            'id': '757_1364311680',
-            'ext': 'mp4',
-            'description': 'extremely bad day for this guy..!',
-            'uploader': 'ljfriel2',
-            'title': 'Most unlucky car accident',
-            'thumbnail': r're:^https?://.*\.jpg$'
-        }
-    }, {
-        'url': 'http://www.liveleak.com/view?i=f93_1390833151',
-        'md5': 'd3f1367d14cc3c15bf24fbfbe04b9abf',
-        'info_dict': {
-            'id': 'f93_1390833151',
-            'ext': 'mp4',
-            'description': 'German Television Channel NDR does an exclusive interview with Edward Snowden.\r\nUploaded on LiveLeak cause German Television thinks the rest of the world isn\'t intereseted in Edward Snowden.',
-            'uploader': 'ARD_Stinkt',
-            'title': 'German Television does first Edward Snowden Interview (ENGLISH)',
-            'thumbnail': r're:^https?://.*\.jpg$'
-        }
-    }, {
-        # Prochan embed
-        'url': 'http://www.liveleak.com/view?i=4f7_1392687779',
-        'md5': '42c6d97d54f1db107958760788c5f48f',
-        'info_dict': {
-            'id': '4f7_1392687779',
-            'ext': 'mp4',
-            'description': "The guy with the cigarette seems amazingly nonchalant about the whole thing...  I really hope my friends' reactions would be a bit stronger.\r\n\r\nAction-go to 0:55.",
-            'uploader': 'CapObveus',
-            'title': 'Man is Fatally Struck by Reckless Car While Packing up a Moving Truck',
-            'age_limit': 18,
-        },
-        'skip': 'Video is dead',
-    }, {
-        # Covers https://github.com/ytdl-org/youtube-dl/pull/5983
-        # Multiple resolutions
-        'url': 'http://www.liveleak.com/view?i=801_1409392012',
-        'md5': 'c3a449dbaca5c0d1825caecd52a57d7b',
-        'info_dict': {
-            'id': '801_1409392012',
-            'ext': 'mp4',
-            'description': 'Happened on 27.7.2014. \r\nAt 0:53 you can see people still swimming at near beach.',
-            'uploader': 'bony333',
-            'title': 'Crazy Hungarian tourist films close call waterspout in Croatia',
-            'thumbnail': r're:^https?://.*\.jpg$'
-        }
-    }, {
-        # Covers https://github.com/ytdl-org/youtube-dl/pull/10664#issuecomment-247439521
-        'url': 'http://m.liveleak.com/view?i=763_1473349649',
-        'add_ie': ['Youtube'],
-        'info_dict': {
-            'id': '763_1473349649',
-            'ext': 'mp4',
-            'title': 'Reporters and public officials ignore epidemic of black on asian violence in Sacramento | Colin Flaherty',
-            'description': 'Colin being the warrior he is and showing the injustice Asians in Sacramento are being subjected to.',
-            'uploader': 'Ziz',
-            'upload_date': '20160908',
-            'uploader_id': 'UCEbta5E_jqlZmEJsriTEtnw'
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'https://www.liveleak.com/view?i=677_1439397581',
-        'info_dict': {
-            'id': '677_1439397581',
-            'title': 'Fuel Depot in China Explosion caught on video',
-        },
-        'playlist_count': 3,
-    }, {
-        'url': 'https://www.liveleak.com/view?t=HvHi_1523016227',
-        'only_matching': True,
-    }, {
-        # No original video
-        'url': 'https://www.liveleak.com/view?t=C26ZZ_1558612804',
-        'only_matching': True,
-    }]
-
-    @staticmethod
-    def _extract_urls(webpage):
-        return re.findall(
-            r'<iframe[^>]+src="(https?://(?:\w+\.)?liveleak\.com/ll_embed\?[^"]*[ift]=[\w_]+[^"]+)"',
-            webpage)
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-
-        video_title = self._og_search_title(webpage).replace('LiveLeak.com -', '').strip()
-        video_description = self._og_search_description(webpage)
-        video_uploader = self._html_search_regex(
-            r'By:.*?(\w+)</a>', webpage, 'uploader', fatal=False)
-        age_limit = int_or_none(self._search_regex(
-            r'you confirm that you are ([0-9]+) years and over.',
-            webpage, 'age limit', default=None))
-        video_thumbnail = self._og_search_thumbnail(webpage)
-
-        entries = self._parse_html5_media_entries(url, webpage, video_id)
-        if not entries:
-            # Maybe an embed?
-            embed_url = self._search_regex(
-                r'<iframe[^>]+src="((?:https?:)?//(?:www\.)?(?:prochan|youtube)\.com/embed[^"]+)"',
-                webpage, 'embed URL')
-            return {
-                '_type': 'url_transparent',
-                'url': embed_url,
-                'id': video_id,
-                'title': video_title,
-                'description': video_description,
-                'uploader': video_uploader,
-                'age_limit': age_limit,
-            }
-
-        for idx, info_dict in enumerate(entries):
-            formats = []
-            for a_format in info_dict['formats']:
-                if not a_format.get('height'):
-                    a_format['height'] = int_or_none(self._search_regex(
-                        r'([0-9]+)p\.mp4', a_format['url'], 'height label',
-                        default=None))
-                formats.append(a_format)
-
-                # Removing '.*.mp4' gives the raw video, which is essentially
-                # the same video without the LiveLeak logo at the top (see
-                # https://github.com/ytdl-org/youtube-dl/pull/4768)
-                orig_url = re.sub(r'\.mp4\.[^.]+', '', a_format['url'])
-                if a_format['url'] != orig_url:
-                    format_id = a_format.get('format_id')
-                    format_id = 'original' + ('-' + format_id if format_id else '')
-                    if self._is_valid_url(orig_url, video_id, format_id):
-                        formats.append({
-                            'format_id': format_id,
-                            'url': orig_url,
-                            'quality': 1,
-                        })
-            self._sort_formats(formats)
-            info_dict['formats'] = formats
-
-            # Don't append entry ID for one-video pages to keep backward compatibility
-            if len(entries) > 1:
-                info_dict['id'] = '%s_%s' % (video_id, idx + 1)
-            else:
-                info_dict['id'] = video_id
-
-            info_dict.update({
-                'title': video_title,
-                'description': video_description,
-                'uploader': video_uploader,
-                'age_limit': age_limit,
-                'thumbnail': video_thumbnail,
-            })
-
-        return self.playlist_result(entries, video_id, video_title)
-
-
-class LiveLeakEmbedIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?liveleak\.com/ll_embed\?.*?\b(?P<kind>[ift])=(?P<id>[\w_]+)'
-
-    # See generic.py for actual test cases
-    _TESTS = [{
-        'url': 'https://www.liveleak.com/ll_embed?i=874_1459135191',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.liveleak.com/ll_embed?f=ab065df993c1',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        kind, video_id = re.match(self._VALID_URL, url).groups()
-
-        if kind == 'f':
-            webpage = self._download_webpage(url, video_id)
-            liveleak_url = self._search_regex(
-                r'(?:logourl\s*:\s*|window\.open\()(?P<q1>[\'"])(?P<url>%s)(?P=q1)' % LiveLeakIE._VALID_URL,
-                webpage, 'LiveLeak URL', group='url')
-        else:
-            liveleak_url = 'http://www.liveleak.com/view?%s=%s' % (kind, video_id)
-
-        return self.url_result(liveleak_url, ie=LiveLeakIE.ie_key())
--- a/yt_dlp/extractor/mediasite.py
+++ b/yt_dlp/extractor/mediasite.py
@@ -122,6 +122,52 @@ class MediasiteIE(InfoExtractor):
                r'(?xi)<iframe\b[^>]+\bsrc=(["\'])(?P<url>(?:(?:https?:)?//[^/]+)?/Mediasite/Play/%s(?:\?.*?)?)\1' % _ID_RE,
                webpage)]

+    def __extract_slides(self, *, stream_id, snum, Stream, duration, images):
+        slide_base_url = Stream['SlideBaseUrl']
+
+        fname_template = Stream['SlideImageFileNameTemplate']
+        if fname_template != 'slide_{0:D4}.jpg':
+            self.report_warning('Unusual slide file name template; report a bug if slide downloading fails')
+        fname_template = re.sub(r'\{0:D([0-9]+)\}', r'{0:0\1}', fname_template)
+
+        fragments = []
+        for i, slide in enumerate(Stream['Slides']):
+            if i == 0:
+                if slide['Time'] > 0:
+                    default_slide = images.get('DefaultSlide')
+                    if default_slide is None:
+                        default_slide = images.get('DefaultStreamImage')
+                    if default_slide is not None:
+                        default_slide = default_slide['ImageFilename']
+                    if default_slide is not None:
+                        fragments.append({
+                            'path': default_slide,
+                            'duration': slide['Time'] / 1000,
+                        })
+
+            next_time = try_get(None, [
+                lambda _: Stream['Slides'][i + 1]['Time'],
+                lambda _: duration,
+                lambda _: slide['Time'],
+            ], expected_type=(int, float))
+
+            fragments.append({
+                'path': fname_template.format(slide.get('Number', i + 1)),
+                'duration': (next_time - slide['Time']) / 1000
+            })
+
+        return {
+            'format_id': '%s-%u.slides' % (stream_id, snum),
+            'ext': 'mhtml',
+            'url': slide_base_url,
+            'protocol': 'mhtml',
+            'acodec': 'none',
+            'vcodec': 'none',
+            'format_note': 'Slides',
+            'fragments': fragments,
+            'fragment_base_url': slide_base_url,
+        }
+
    def _real_extract(self, url):
        url, data = unsmuggle_url(url, {})
        mobj = re.match(self._VALID_URL, url)
@@ -198,10 +244,15 @@ class MediasiteIE(InfoExtractor):
                        'ext': mimetype2ext(VideoUrl.get('MimeType')),
                    })

-            # TODO: if Stream['HasSlideContent']:
-            # synthesise an MJPEG video stream '%s-%u.slides' % (stream_type, snum)
-            # from Stream['Slides']
-            # this will require writing a custom downloader...
+            if Stream.get('HasSlideContent', False):
+                images = player_options['PlayerLayoutOptions']['Images']
+                stream_formats.append(self.__extract_slides(
+                    stream_id=stream_id,
+                    snum=snum,
+                    Stream=Stream,
+                    duration=presentation.get('Duration'),
+                    images=images,
+                ))

            # disprefer 'secondary' streams
            if stream_type != 0:
--- a/yt_dlp/extractor/nrk.py
+++ b/yt_dlp/extractor/nrk.py
@@ -58,7 +58,7 @@ class NRKBaseIE(InfoExtractor):

    def _call_api(self, path, video_id, item=None, note=None, fatal=True, query=None):
        return self._download_json(
-            urljoin('http://psapi.nrk.no/', path),
+            urljoin('https://psapi.nrk.no/', path),
            video_id, note or 'Downloading %s JSON' % item,
            fatal=fatal, query=query,
            headers={'Accept-Encoding': 'gzip, deflate, br'})
--- a/yt_dlp/extractor/orf.py
+++ b/yt_dlp/extractor/orf.py
@@ -98,6 +98,9 @@ class ORFTVthekIE(InfoExtractor):
                elif ext == 'f4m':
                    formats.extend(self._extract_f4m_formats(
                        src, video_id, f4m_id=format_id, fatal=False))
+                elif ext == 'mpd':
+                    formats.extend(self._extract_mpd_formats(
+                        src, video_id, mpd_id=format_id, fatal=False))
                else:
                    formats.append({
                        'format_id': format_id,
--- a/yt_dlp/extractor/pornhub.py
+++ b/yt_dlp/extractor/pornhub.py
@@ -14,6 +14,7 @@ from ..compat import (
 )
 from .openload import PhantomJSwrapper
 from ..utils import (
+    clean_html,
    determine_ext,
    ExtractorError,
    int_or_none,
@@ -30,6 +31,7 @@ from ..utils import (

 class PornHubBaseIE(InfoExtractor):
    _NETRC_MACHINE = 'pornhub'
+    _PORNHUB_HOST_RE = r'(?:(?P<host>pornhub(?:premium)?\.(?:com|net|org))|pornhubthbh7ap3u\.onion)'

    def _download_webpage_handle(self, *args, **kwargs):
        def dl(*args, **kwargs):
@@ -122,11 +124,13 @@ class PornHubIE(PornHubBaseIE):
    _VALID_URL = r'''(?x)
                    https?://
                        (?:
-                            (?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net|org))/(?:(?:view_video\.php|video/show)\?viewkey=|embed/)|
+                            (?:[^/]+\.)?
+                            %s
+                            /(?:(?:view_video\.php|video/show)\?viewkey=|embed/)|
                            (?:www\.)?thumbzilla\.com/video/
                        )
                        (?P<id>[\da-z]+)
-                    '''
+                    ''' % PornHubBaseIE._PORNHUB_HOST_RE
    _TESTS = [{
        'url': 'http://www.pornhub.com/view_video.php?viewkey=648719015',
        'md5': 'a6391306d050e4547f62b3f485dd9ba9',
@@ -145,6 +149,7 @@ class PornHubIE(PornHubBaseIE):
            'age_limit': 18,
            'tags': list,
            'categories': list,
+            'cast': list,
        },
    }, {
        # non-ASCII title
@@ -236,6 +241,13 @@ class PornHubIE(PornHubBaseIE):
    }, {
        'url': 'https://www.pornhubpremium.com/view_video.php?viewkey=ph5f75b0f4b18e3',
        'only_matching': True,
+    }, {
+        # geo restricted
+        'url': 'https://www.pornhub.com/view_video.php?viewkey=ph5a9813bfa7156',
+        'only_matching': True,
+    }, {
+        'url': 'http://pornhubthbh7ap3u.onion/view_video.php?viewkey=ph5a9813bfa7156',
+        'only_matching': True,
    }]

    @staticmethod
@@ -275,6 +287,11 @@ class PornHubIE(PornHubBaseIE):
                'PornHub said: %s' % error_msg,
                expected=True, video_id=video_id)

+        if any(re.search(p, webpage) for p in (
+                r'class=["\']geoBlocked["\']',
+                r'>\s*This content is unavailable in your country')):
+            self.raise_geo_restricted()
+
        # video_title from flashvars contains whitespace instead of non-ASCII (see
        # http://www.pornhub.com/view_video.php?viewkey=1331683002), not relying
        # on that anymore.
@@ -408,17 +425,14 @@ class PornHubIE(PornHubBaseIE):
                    format_url, video_id, 'mp4', entry_protocol='m3u8_native',
                    m3u8_id='hls', fatal=False))
                return
-            tbr = None
-            mobj = re.search(r'(?P<height>\d+)[pP]?_(?P<tbr>\d+)[kK]', format_url)
-            if mobj:
-                if not height:
-                    height = int(mobj.group('height'))
-                tbr = int(mobj.group('tbr'))
+            if not height:
+                height = int_or_none(self._search_regex(
+                    r'(?P<height>\d+)[pP]?_\d+[kK]', format_url, 'height',
+                    default=None))
            formats.append({
                'url': format_url,
                'format_id': '%dp' % height if height else None,
                'height': height,
-                'tbr': tbr,
            })

        for video_url, height in video_urls:
@@ -440,7 +454,10 @@ class PornHubIE(PornHubBaseIE):
                        add_format(video_url, height)
                continue
            add_format(video_url)
-        self._sort_formats(formats)
+
+        # field_preference is unnecessary here, but kept for code-similarity with youtube-dl
+        self._sort_formats(
+            formats, field_preference=('height', 'width', 'fps', 'format_id'))

        video_uploader = self._html_search_regex(
            r'(?s)From:&nbsp;.+?<(?:a\b[^>]+\bhref=["\']/(?:(?:user|channel)s|model|pornstar)/|span\b[^>]+\bclass=["\']username)[^>]+>(.+?)<',
@@ -464,7 +481,7 @@ class PornHubIE(PornHubBaseIE):
                r'(?s)<div[^>]+\bclass=["\'].*?\b%sWrapper[^>]*>(.+?)</div>'
                % meta_key, webpage, meta_key, default=None)
            if div:
-                return re.findall(r'<a[^>]+\bhref=[^>]+>([^<]+)', div)
+                return [clean_html(x).strip() for x in re.findall(r'(?s)<a[^>]+\bhref=[^>]+>.+?</a>', div)]

        info = self._search_json_ld(webpage, video_id, default={})
        # description provided in JSON-LD is irrelevant
@@ -485,6 +502,7 @@ class PornHubIE(PornHubBaseIE):
            'age_limit': 18,
            'tags': extract_list('tags'),
            'categories': extract_list('categories'),
+            'cast': extract_list('pornstars'),
            'subtitles': subtitles,
        }, info)

@@ -513,7 +531,7 @@ class PornHubPlaylistBaseIE(PornHubBaseIE):


 class PornHubUserIE(PornHubPlaylistBaseIE):
-    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net|org))/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/?#&]+))(?:[?#&]|/(?!videos)|$)'
+    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?%s/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/?#&]+))(?:[?#&]|/(?!videos)|$)' % PornHubBaseIE._PORNHUB_HOST_RE
    _TESTS = [{
        'url': 'https://www.pornhub.com/model/zoe_ph',
        'playlist_mincount': 118,
@@ -542,6 +560,9 @@ class PornHubUserIE(PornHubPlaylistBaseIE):
        # Same as before, multi page
        'url': 'https://www.pornhubpremium.com/pornstar/lily-labeau',
        'only_matching': True,
+    }, {
+        'url': 'https://pornhubthbh7ap3u.onion/model/zoe_ph',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
@@ -617,7 +638,7 @@ class PornHubPagedPlaylistBaseIE(PornHubPlaylistBaseIE):


 class PornHubPagedVideoListIE(PornHubPagedPlaylistBaseIE):
-    _VALID_URL = r'https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net|org))/(?P<id>(?:[^/]+/)*[^/?#&]+)'
+    _VALID_URL = r'https?://(?:[^/]+\.)?%s/(?P<id>(?:[^/]+/)*[^/?#&]+)' % PornHubBaseIE._PORNHUB_HOST_RE
    _TESTS = [{
        'url': 'https://www.pornhub.com/model/zoe_ph/videos',
        'only_matching': True,
@@ -722,6 +743,9 @@ class PornHubPagedVideoListIE(PornHubPagedPlaylistBaseIE):
    }, {
        'url': 'https://de.pornhub.com/playlist/4667351',
        'only_matching': True,
+    }, {
+        'url': 'https://pornhubthbh7ap3u.onion/model/zoe_ph/videos',
+        'only_matching': True,
    }]

    @classmethod
@@ -732,7 +756,7 @@ class PornHubPagedVideoListIE(PornHubPagedPlaylistBaseIE):


 class PornHubUserVideosUploadIE(PornHubPagedPlaylistBaseIE):
-    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?(?P<host>pornhub(?:premium)?\.(?:com|net|org))/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/]+)/videos/upload)'
+    _VALID_URL = r'(?P<url>https?://(?:[^/]+\.)?%s/(?:(?:user|channel)s|model|pornstar)/(?P<id>[^/]+)/videos/upload)' % PornHubBaseIE._PORNHUB_HOST_RE
    _TESTS = [{
        'url': 'https://www.pornhub.com/pornstar/jenny-blighe/videos/upload',
        'info_dict': {
@@ -742,4 +766,7 @@ class PornHubUserVideosUploadIE(PornHubPagedPlaylistBaseIE):
    }, {
        'url': 'https://www.pornhub.com/model/zoe_ph/videos/upload',
        'only_matching': True,
+    }, {
+        'url': 'http://pornhubthbh7ap3u.onion/pornstar/jenny-blighe/videos/upload',
+        'only_matching': True,
    }]
--- a/yt_dlp/extractor/twitcasting.py
+++ b/yt_dlp/extractor/twitcasting.py
@@ -5,12 +5,14 @@ import itertools
 import re

 from .common import InfoExtractor
+from ..downloader.websocket import has_websockets
 from ..utils import (
    clean_html,
    float_or_none,
    get_element_by_class,
    get_element_by_id,
    parse_duration,
+    qualities,
    str_to_int,
    try_get,
    unified_timestamp,
@@ -89,9 +91,24 @@ class TwitCastingIE(InfoExtractor):
            video_js_data = video_js_data[0]
            m3u8_url = try_get(video_js_data, lambda x: x['source']['url'])

+        stream_server_data = self._download_json(
+            'https://twitcasting.tv/streamserver.php?target=%s&mode=client' % uploader_id, video_id,
+            'Downloading live info', fatal=False)
+
        is_live = 'data-status="online"' in webpage
+        formats = []
        if is_live and not m3u8_url:
            m3u8_url = 'https://twitcasting.tv/%s/metastream.m3u8' % uploader_id
+        if is_live and has_websockets and stream_server_data:
+            qq = qualities(['base', 'mobilesource', 'main'])
+            for mode, ws_url in stream_server_data['llfmp4']['streams'].items():
+                formats.append({
+                    'url': ws_url,
+                    'format_id': 'ws-%s' % mode,
+                    'ext': 'mp4',
+                    'quality': qq(mode),
+                    'protocol': 'websocket_frag',  # TwitCasting simply sends moof atom directly over WS
+                })

        thumbnail = video_js_data.get('thumbnailUrl') or self._og_search_thumbnail(webpage)
        description = clean_html(get_element_by_id(
@@ -106,10 +123,9 @@ class TwitCastingIE(InfoExtractor):
            r'data-toggle="true"[^>]+datetime="([^"]+)"',
            webpage, 'datetime', None))

-        formats = None
        if m3u8_url:
-            formats = self._extract_m3u8_formats(
-                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', live=is_live)
+            formats.extend(self._extract_m3u8_formats(
+                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', live=is_live))
        self._sort_formats(formats)

        return {
--- a/yt_dlp/extractor/umg.py
+++ b/yt_dlp/extractor/umg.py
@@ -28,7 +28,7 @@ class UMGDeIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
        video_data = self._download_json(
-            'https://api.universal-music.de/graphql',
+            'https://graphql.universal-music.de/',
            video_id, query={
                'query': '''{
  universalMusic(channel:16) {
@@ -56,11 +56,9 @@ class UMGDeIE(InfoExtractor):
        formats = []

        def add_m3u8_format(format_id):
-            m3u8_formats = self._extract_m3u8_formats(
+            formats.extend(self._extract_m3u8_formats(
                hls_url_template % format_id, video_id, 'mp4',
-                'm3u8_native', m3u8_id='hls', fatal='False')
-            if m3u8_formats and m3u8_formats[0].get('height'):
-                formats.extend(m3u8_formats)
+                'm3u8_native', m3u8_id='hls', fatal=False))

        for f in video_data.get('formats', []):
            f_url = f.get('url')
--- a/yt_dlp/extractor/viki.py
+++ b/yt_dlp/extractor/viki.py
@@ -142,6 +142,7 @@ class VikiIE(VikiBaseIE):
    IE_NAME = 'viki'
    _VALID_URL = r'%s(?:videos|player)/(?P<id>[0-9]+v)' % VikiBaseIE._VALID_URL_BASE
    _TESTS = [{
+        'note': 'Free non-DRM video with storyboards in MPD',
        'url': 'https://www.viki.com/videos/1175236v-choosing-spouse-by-lottery-episode-1',
        'info_dict': {
            'id': '1175236v',
@@ -155,7 +156,6 @@ class VikiIE(VikiBaseIE):
        'params': {
            'format': 'bestvideo',
        },
-        'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
    }, {
        'url': 'http://www.viki.com/videos/1023585v-heirs-episode-14',
        'info_dict': {
@@ -173,7 +173,6 @@ class VikiIE(VikiBaseIE):
            'format': 'bestvideo',
        },
        'skip': 'Blocked in the US',
-        'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
    }, {
        # clip
        'url': 'http://www.viki.com/videos/1067139v-the-avengers-age-of-ultron-press-conference',
@@ -225,7 +224,6 @@ class VikiIE(VikiBaseIE):
        'params': {
            'format': 'bestvideo',
        },
-        'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
    }, {
        # youtube external
        'url': 'http://www.viki.com/videos/50562v-poor-nastya-complete-episode-1',
@@ -264,7 +262,6 @@ class VikiIE(VikiBaseIE):
        'params': {
            'format': 'bestvideo',
        },
-        'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
    }]

    def _real_extract(self, url):
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -301,12 +301,22 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
    _YT_INITIAL_BOUNDARY_RE = r'(?:var\s+meta|</script|\n)'

    def _generate_sapisidhash_header(self):
-        sapisid_cookie = self._get_cookies('https://www.youtube.com').get('SAPISID')
+        # Sometimes SAPISID cookie isn't present but __Secure-3PAPISID is.
+        # See: https://github.com/yt-dlp/yt-dlp/issues/393
+        yt_cookies = self._get_cookies('https://www.youtube.com')
+        sapisid_cookie = dict_get(
+            yt_cookies, ('__Secure-3PAPISID', 'SAPISID'))
        if sapisid_cookie is None:
            return
        time_now = round(time.time())
-        sapisidhash = hashlib.sha1((str(time_now) + " " + sapisid_cookie.value + " " + "https://www.youtube.com").encode("utf-8")).hexdigest()
-        return "SAPISIDHASH %s_%s" % (time_now, sapisidhash)
+        # SAPISID cookie is required if not already present
+        if not yt_cookies.get('SAPISID'):
+            self._set_cookie(
+                '.youtube.com', 'SAPISID', sapisid_cookie.value, secure=True, expire_time=time_now + 3600)
+        # SAPISIDHASH algorithm from https://stackoverflow.com/a/32065323
+        sapisidhash = hashlib.sha1(
+            f'{time_now} {sapisid_cookie.value} https://www.youtube.com'.encode('utf-8')).hexdigest()
+        return f'SAPISIDHASH {time_now}_{sapisidhash}'

    def _call_api(self, ep, query, video_id, fatal=True, headers=None,
                  note='Downloading API JSON', errnote='Unable to download API page',
@@ -454,20 +464,15 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        # Invidious instances taken from https://github.com/iv-org/documentation/blob/master/Invidious-Instances.md
        r'(?:www\.)?invidious\.pussthecat\.org',
        r'(?:www\.)?invidious\.zee\.li',
-        r'(?:(?:www|au)\.)?ytprivate\.com',
-        r'(?:www\.)?invidious\.namazso\.eu',
        r'(?:www\.)?invidious\.ethibox\.fr',
-        r'(?:www\.)?w6ijuptxiku4xpnnaetxvnkc5vqcdu7mgns2u77qefoixi63vbvnpnqd\.onion',
-        r'(?:www\.)?kbjggqkzv65ivcqj6bumvp337z6264huv5kpkwuv6gu5yjiskvan7fad\.onion',
        r'(?:www\.)?invidious\.3o7z6yfxhbw7n3za4rss6l434kmv55cgw2vuziwuigpwegswvwzqipyd\.onion',
-        r'(?:www\.)?grwp24hodrefzvjjuccrkw3mjq4tzhaaq32amf33dzpmuxe7ilepcmad\.onion',
        # youtube-dl invidious instances list
        r'(?:(?:www|no)\.)?invidiou\.sh',
        r'(?:(?:www|fi)\.)?invidious\.snopyta\.org',
        r'(?:www\.)?invidious\.kabi\.tk',
        r'(?:www\.)?invidious\.mastodon\.host',
        r'(?:www\.)?invidious\.zapashcanon\.fr',
-        r'(?:www\.)?invidious\.kavin\.rocks',
+        r'(?:www\.)?(?:invidious(?:-us)?|piped)\.kavin\.rocks',
        r'(?:www\.)?invidious\.tinfoil-hat\.net',
        r'(?:www\.)?invidious\.himiko\.cloud',
        r'(?:www\.)?invidious\.reallyancient\.tech',
@@ -494,6 +499,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        r'(?:www\.)?invidious\.toot\.koeln',
        r'(?:www\.)?invidious\.fdn\.fr',
        r'(?:www\.)?watch\.nettohikari\.com',
+        r'(?:www\.)?invidious\.namazso\.eu',
+        r'(?:www\.)?invidious\.silkky\.cloud',
+        r'(?:www\.)?invidious\.exonip\.de',
+        r'(?:www\.)?invidious\.riverside\.rocks',
+        r'(?:www\.)?invidious\.blamefran\.net',
+        r'(?:www\.)?invidious\.moomoo\.de',
+        r'(?:www\.)?ytb\.trom\.tf',
+        r'(?:www\.)?yt\.cyberhost\.uk',
        r'(?:www\.)?kgg2m7yk5aybusll\.onion',
        r'(?:www\.)?qklhadlycap4cnod\.onion',
        r'(?:www\.)?axqzx4s6s54s32yentfqojs3x5i7faxza6xo3ehd4bzzsg2ii4fv2iid\.onion',
@@ -502,6 +515,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        r'(?:www\.)?invidious\.l4qlywnpwqsluw65ts7md3khrivpirse744un3x7mlskqauz5pyuzgqd\.onion',
        r'(?:www\.)?owxfohz4kjyv25fvlqilyxast7inivgiktls3th44jhk3ej3i7ya\.b32\.i2p',
        r'(?:www\.)?4l2dgddgsrkf2ous66i6seeyi6etzfgrue332grh2n7madpwopotugyd\.onion',
+        r'(?:www\.)?w6ijuptxiku4xpnnaetxvnkc5vqcdu7mgns2u77qefoixi63vbvnpnqd\.onion',
+        r'(?:www\.)?kbjggqkzv65ivcqj6bumvp337z6264huv5kpkwuv6gu5yjiskvan7fad\.onion',
+        r'(?:www\.)?grwp24hodrefzvjjuccrkw3mjq4tzhaaq32amf33dzpmuxe7ilepcmad\.onion',
+        r'(?:www\.)?hpniueoejy4opn7bc4ftgazyqjoeqwlvh2uiku2xqku6zpoa4bf5ruid\.onion',
    )
    _VALID_URL = r"""(?x)^
                     (
@@ -1866,6 +1883,16 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            'comment_count': len(comments),
        }

+    @staticmethod
+    def _get_video_info_params(video_id):
+        return {
+            'video_id': video_id,
+            'eurl': 'https://youtube.googleapis.com/v/' + video_id,
+            'html5': '1',
+            'c': 'TVHTML5',
+            'cver': '6.20180913',
+        }
+
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
        video_id = self._match_id(url)
@@ -1898,16 +1925,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                    base_url + 'get_video_info', video_id,
                    'Fetching youtube music info webpage',
                    'unable to download youtube music info webpage', query={
-                        'video_id': video_id,
-                        'eurl': 'https://youtube.googleapis.com/v/' + video_id,
+                        **self._get_video_info_params(video_id),
                        'el': 'detailpage',
                        'c': 'WEB_REMIX',
                        'cver': '0.1',
                        'cplayer': 'UNIPLAYER',
-                        'html5': '1',
-                    }, fatal=False)),
+                    }, fatal=False) or ''),
                lambda x: x['player_response'][0],
-                compat_str) or '{}', video_id)
+                compat_str) or '{}', video_id, fatal=False)
            ytm_streaming_data = ytm_player_response.get('streamingData') or {}

        player_response = None
@@ -1926,12 +1951,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            pr = self._parse_json(try_get(compat_parse_qs(
                self._download_webpage(
                    base_url + 'get_video_info', video_id,
-                    'Refetching age-gated info webpage',
-                    'unable to download video info webpage', query={
-                        'video_id': video_id,
-                        'eurl': 'https://youtube.googleapis.com/v/' + video_id,
-                        'html5': '1',
-                    }, fatal=False)),
+                    'Refetching age-gated info webpage', 'unable to download video info webpage',
+                    query=self._get_video_info_params(video_id), fatal=False)),
                lambda x: x['player_response'][0],
                compat_str) or '{}', video_id)
            if pr:
@@ -2325,18 +2346,17 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            initial_data = self._call_api(
                'next', {'videoId': video_id}, video_id, fatal=False, api_key=self._extract_api_key(ytcfg))

-        if not is_live:
-            try:
-                # This will error if there is no livechat
-                initial_data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
-                info['subtitles']['live_chat'] = [{
-                    'url': 'https://www.youtube.com/watch?v=%s' % video_id,  # url is needed to set cookies
-                    'video_id': video_id,
-                    'ext': 'json',
-                    'protocol': 'youtube_live_chat_replay',
-                }]
-            except (KeyError, IndexError, TypeError):
-                pass
+        try:
+            # This will error if there is no livechat
+            initial_data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
+            info['subtitles']['live_chat'] = [{
+                'url': 'https://www.youtube.com/watch?v=%s' % video_id,  # url is needed to set cookies
+                'video_id': video_id,
+                'ext': 'json',
+                'protocol': 'youtube_live_chat' if is_live else 'youtube_live_chat_replay',
+            }]
+        except (KeyError, IndexError, TypeError):
+            pass

        if initial_data:
            chapters = self._extract_chapters_from_json(
@@ -3592,7 +3612,13 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):

            else:
                # Youtube may send alerts if there was an issue with the continuation page
-                self._extract_and_report_alerts(response, expected=False)
+                try:
+                    self._extract_and_report_alerts(response, expected=False)
+                except ExtractorError as e:
+                    if fatal:
+                        raise
+                    self.report_warning(error_to_compat_str(e))
+                    return
                if not check_get_keys or dict_get(response, check_get_keys):
                    break
                # Youtube sometimes sends incomplete data
@@ -4062,6 +4088,7 @@ class YoutubeRecommendedIE(YoutubeFeedsInfoExtractor):
    IE_DESC = 'YouTube.com recommended videos, ":ytrec" for short (requires authentication)'
    _VALID_URL = r'https?://(?:www\.)?youtube\.com/?(?:[?#]|$)|:ytrec(?:ommended)?'
    _FEED_NAME = 'recommended'
+    _LOGIN_REQUIRED = False
    _TESTS = [{
        'url': ':ytrec',
        'only_matching': True,
--- a/yt_dlp/options.py
+++ b/yt_dlp/options.py
@@ -599,6 +599,10 @@ def parseOpts(overrideArguments=None):
        '-r', '--limit-rate', '--rate-limit',
        dest='ratelimit', metavar='RATE',
        help='Maximum download rate in bytes per second (e.g. 50K or 4.2M)')
+    downloader.add_option(
+        '--throttled-rate',
+        dest='throttledratelimit', metavar='RATE',
+        help='Minimum download rate in bytes per second below which throttling is assumed and the video data is re-extracted (e.g. 100K)')
    downloader.add_option(
        '-R', '--retries',
        dest='retries', metavar='RETRIES', default=10,
@@ -1165,7 +1169,7 @@ def parseOpts(overrideArguments=None):
            'to give the argument to the specified postprocessor/executable. Supported PP are: '
            'Merger, ExtractAudio, SplitChapters, Metadata, EmbedSubtitle, EmbedThumbnail, '
            'SubtitlesConvertor, ThumbnailsConvertor, VideoRemuxer, VideoConvertor, '
-            'SponSkrub, FixupStretched, FixupM4a and FixupM3u8. '
+            'SponSkrub, FixupStretched, FixupM4a, FixupM3u8, FixupTimestamp and FixupDuration. '
            'The supported executables are: AtomicParsley, FFmpeg, FFprobe, and SponSkrub. '
            'You can also specify "PP+EXE:ARGS" to give the arguments to the specified executable '
            'only when being used by the specified postprocessor. Additionally, for ffmpeg/ffprobe, '
@@ -1200,19 +1204,19 @@ def parseOpts(overrideArguments=None):
    postproc.add_option(
        '--embed-thumbnail',
        action='store_true', dest='embedthumbnail', default=False,
-        help='Embed thumbnail in the audio as cover art')
+        help='Embed thumbnail in the video as cover art')
    postproc.add_option(
        '--no-embed-thumbnail',
        action='store_false', dest='embedthumbnail',
        help='Do not embed thumbnail (default)')
    postproc.add_option(
-        '--add-metadata',
+        '--embed-metadata', '--add-metadata',
        action='store_true', dest='addmetadata', default=False,
-        help='Write metadata to the video file')
+        help='Embed metadata including chapter markers (if supported by the format) to the video file (Alias: --add-metadata)')
    postproc.add_option(
-        '--no-add-metadata',
+        '--no-embed-metadata', '--no-add-metadata',
        action='store_false', dest='addmetadata',
-        help='Do not write metadata (default)')
+        help='Do not write metadata (default)  (Alias: --no-add-metadata)')
    postproc.add_option(
        '--metadata-from-title',
        metavar='FORMAT', dest='metafromtitle',
@@ -1230,10 +1234,12 @@ def parseOpts(overrideArguments=None):
    postproc.add_option(
        '--fixup',
        metavar='POLICY', dest='fixup', default=None,
+        choices=('never', 'ignore', 'warn', 'detect_or_warn', 'force'),
        help=(
            'Automatically correct known faults of the file. '
            'One of never (do nothing), warn (only emit a warning), '
-            'detect_or_warn (the default; fix file if we can, warn otherwise)'))
+            'detect_or_warn (the default; fix file if we can, warn otherwise), '
+            'force (try fixing even if file already exists'))
    postproc.add_option(
        '--prefer-avconv', '--no-prefer-ffmpeg',
        action='store_false', dest='prefer_ffmpeg',
--- a/yt_dlp/postprocessor/init.py
+++ b/yt_dlp/postprocessor/init.py
@@ -5,7 +5,9 @@ from .ffmpeg import (
    FFmpegPostProcessor,
    FFmpegEmbedSubtitlePP,
    FFmpegExtractAudioPP,
+    FFmpegFixupDurationPP,
    FFmpegFixupStretchedPP,
+    FFmpegFixupTimestampPP,
    FFmpegFixupM3u8PP,
    FFmpegFixupM4aPP,
    FFmpegMergerPP,
@@ -35,9 +37,11 @@ __all__ = [
    'FFmpegEmbedSubtitlePP',
    'FFmpegExtractAudioPP',
    'FFmpegSplitChaptersPP',
+    'FFmpegFixupDurationPP',
    'FFmpegFixupM3u8PP',
    'FFmpegFixupM4aPP',
    'FFmpegFixupStretchedPP',
+    'FFmpegFixupTimestampPP',
    'FFmpegMergerPP',
    'FFmpegMetadataPP',
    'FFmpegSubtitlesConvertorPP',
--- a/yt_dlp/postprocessor/common.py
+++ b/yt_dlp/postprocessor/common.py
@@ -1,5 +1,6 @@
 from __future__ import unicode_literals

+import functools
 import os

 from ..compat import compat_str
@@ -67,6 +68,25 @@ class PostProcessor(object):
        """Sets the downloader for this PP."""
        self._downloader = downloader

+    @staticmethod
+    def _restrict_to(*, video=True, audio=True, images=True):
+        allowed = {'video': video, 'audio': audio, 'images': images}
+
+        def decorator(func):
+            @functools.wraps(func)
+            def wrapper(self, info):
+                format_type = (
+                    'video' if info.get('vcodec') != 'none'
+                    else 'audio' if info.get('acodec') != 'none'
+                    else 'images')
+                if allowed[format_type]:
+                    return func(self, info)
+                else:
+                    self.to_screen('Skipping %s' % format_type)
+                    return [], info
+            return wrapper
+        return decorator
+
    def run(self, information):
        """Run the PostProcessor.

--- a/yt_dlp/postprocessor/embedthumbnail.py
+++ b/yt_dlp/postprocessor/embedthumbnail.py
@@ -16,6 +16,7 @@ try:
 except ImportError:
    has_mutagen = False

+from .common import PostProcessor
 from .ffmpeg import (
    FFmpegPostProcessor,
    FFmpegThumbnailsConvertorPP,
@@ -62,6 +63,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
    def _report_run(self, exe, filename):
        self.to_screen('%s: Adding thumbnail to "%s"' % (exe, filename))

+    @PostProcessor._restrict_to(images=False)
    def run(self, info):
        filename = info['filepath']
        temp_filename = prepend_extension(filename, 'temp')
@@ -123,8 +125,9 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
            self.run_ffmpeg(filename, temp_filename, options)

        elif info['ext'] in ['m4a', 'mp4', 'mov']:
+            prefer_atomicparsley = 'embed-thumbnail-atomicparsley' in self.get_param('compat_opts', [])
            # Method 1: Use mutagen
-            if not has_mutagen:
+            if not has_mutagen or prefer_atomicparsley:
                success = False
            else:
                try:
@@ -143,7 +146,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
                    success = False

            # Method 2: Use ffmpeg+ffprobe
-            if not success:
+            if not success and not prefer_atomicparsley:
                success = True
                try:
                    options = ['-c', 'copy', '-map', '0', '-dn', '-map', '1']
--- a/yt_dlp/postprocessor/ffmpeg.py
+++ b/yt_dlp/postprocessor/ffmpeg.py
@@ -310,6 +310,7 @@ class FFmpegExtractAudioPP(FFmpegPostProcessor):
        except FFmpegPostProcessorError as err:
            raise AudioConversionError(err.msg)

+    @PostProcessor._restrict_to(images=False)
    def run(self, information):
        path = information['filepath']
        orig_ext = information['ext']
@@ -419,6 +420,7 @@ class FFmpegVideoConvertorPP(FFmpegPostProcessor):
            return ['-c:v', 'libxvid', '-vtag', 'XVID']
        return []

+    @PostProcessor._restrict_to(images=False)
    def run(self, information):
        path, source_ext = information['filepath'], information['ext'].lower()
        target_ext = self._target_ext(source_ext)
@@ -456,6 +458,7 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
        super(FFmpegEmbedSubtitlePP, self).__init__(downloader)
        self._already_have_subtitle = already_have_subtitle

+    @PostProcessor._restrict_to(images=False)
    def run(self, information):
        if information['ext'] not in ('mp4', 'webm', 'mkv'):
            self.to_screen('Subtitles can only be embedded in mp4, webm or mkv files')
@@ -523,6 +526,7 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):


 class FFmpegMetadataPP(FFmpegPostProcessor):
+    @PostProcessor._restrict_to(images=False)
    def run(self, info):
        metadata = {}

@@ -625,6 +629,7 @@ class FFmpegMetadataPP(FFmpegPostProcessor):


 class FFmpegMergerPP(FFmpegPostProcessor):
+    @PostProcessor._restrict_to(images=False)
    def run(self, info):
        filename = info['filepath']
        temp_filename = prepend_extension(filename, 'temp')
@@ -656,55 +661,71 @@ class FFmpegMergerPP(FFmpegPostProcessor):
        return True


-class FFmpegFixupStretchedPP(FFmpegPostProcessor):
+class FFmpegFixupPostProcessor(FFmpegPostProcessor):
+    def _fixup(self, msg, filename, options):
+        temp_filename = prepend_extension(filename, 'temp')
+
+        self.to_screen(f'{msg} of "{filename}"')
+        self.run_ffmpeg(filename, temp_filename, options)
+
+        os.remove(encodeFilename(filename))
+        os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+
+
+class FFmpegFixupStretchedPP(FFmpegFixupPostProcessor):
+    @PostProcessor._restrict_to(images=False, audio=False)
    def run(self, info):
        stretched_ratio = info.get('stretched_ratio')
-        if stretched_ratio is None or stretched_ratio == 1:
-            return [], info
-
-        filename = info['filepath']
-        temp_filename = prepend_extension(filename, 'temp')
-
-        options = ['-c', 'copy', '-map', '0', '-dn', '-aspect', '%f' % stretched_ratio]
-        self.to_screen('Fixing aspect ratio in "%s"' % filename)
-        self.run_ffmpeg(filename, temp_filename, options)
-
-        os.remove(encodeFilename(filename))
-        os.rename(encodeFilename(temp_filename), encodeFilename(filename))
-
+        if stretched_ratio not in (None, 1):
+            self._fixup('Fixing aspect ratio', info['filepath'], [
+                '-c', 'copy', '-map', '0', '-dn', '-aspect', '%f' % stretched_ratio])
        return [], info


-class FFmpegFixupM4aPP(FFmpegPostProcessor):
+class FFmpegFixupM4aPP(FFmpegFixupPostProcessor):
+    @PostProcessor._restrict_to(images=False, video=False)
    def run(self, info):
-        if info.get('container') != 'm4a_dash':
-            return [], info
-
-        filename = info['filepath']
-        temp_filename = prepend_extension(filename, 'temp')
-
-        options = ['-c', 'copy', '-map', '0', '-dn', '-f', 'mp4']
-        self.to_screen('Correcting container in "%s"' % filename)
-        self.run_ffmpeg(filename, temp_filename, options)
-
-        os.remove(encodeFilename(filename))
-        os.rename(encodeFilename(temp_filename), encodeFilename(filename))
-
+        if info.get('container') == 'm4a_dash':
+            self._fixup('Correcting container', info['filepath'], [
+                '-c', 'copy', '-map', '0', '-dn', '-f', 'mp4'])
        return [], info


-class FFmpegFixupM3u8PP(FFmpegPostProcessor):
+class FFmpegFixupM3u8PP(FFmpegFixupPostProcessor):
+    @PostProcessor._restrict_to(images=False)
    def run(self, info):
-        filename = info['filepath']
-        if self.get_audio_codec(filename) == 'aac':
-            temp_filename = prepend_extension(filename, 'temp')
+        if self.get_audio_codec(info['filepath']) == 'aac':
+            self._fixup('Fixing malformed AAC bitstream', info['filepath'], [
+                '-c', 'copy', '-map', '0', '-dn', '-f', 'mp4', '-bsf:a', 'aac_adtstoasc'])
+        return [], info

-            options = ['-c', 'copy', '-map', '0', '-dn', '-f', 'mp4', '-bsf:a', 'aac_adtstoasc']
-            self.to_screen('Fixing malformed AAC bitstream in "%s"' % filename)
-            self.run_ffmpeg(filename, temp_filename, options)

-            os.remove(encodeFilename(filename))
-            os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+class FFmpegFixupTimestampPP(FFmpegFixupPostProcessor):
+
+    def __init__(self, downloader=None, trim=0.001):
+        # "trim" should be used when the video contains unintended packets
+        super(FFmpegFixupTimestampPP, self).__init__(downloader)
+        assert isinstance(trim, (int, float))
+        self.trim = str(trim)
+
+    @PostProcessor._restrict_to(images=False)
+    def run(self, info):
+        required_version = '4.4'
+        if is_outdated_version(self._versions[self.basename], required_version):
+            self.report_warning(
+                'A re-encode is needed to fix timestamps in older versions of ffmpeg. '
+                f'Please install ffmpeg {required_version} or later to fixup without re-encoding')
+            opts = ['-vf', 'setpts=PTS-STARTPTS']
+        else:
+            opts = ['-c', 'copy', '-bsf', 'setts=ts=TS-STARTPTS']
+        self._fixup('Fixing frame timestamp', info['filepath'], opts + ['-map', '0', '-dn', '-ss', self.trim])
+        return [], info
+
+
+class FFmpegFixupDurationPP(FFmpegFixupPostProcessor):
+    @PostProcessor._restrict_to(images=False)
+    def run(self, info):
+        self._fixup('Fixing video duration', info['filepath'], ['-c', 'copy', '-map', '0', '-dn'])
        return [], info


@@ -805,6 +826,7 @@ class FFmpegSplitChaptersPP(FFmpegPostProcessor):
            ['-ss', compat_str(chapter['start_time']),
             '-t', compat_str(chapter['end_time'] - chapter['start_time'])])

+    @PostProcessor._restrict_to(images=False)
    def run(self, info):
        chapters = info.get('chapters') or []
        if not chapters:
--- a/yt_dlp/postprocessor/sponskrub.py
+++ b/yt_dlp/postprocessor/sponskrub.py
@@ -41,6 +41,7 @@ class SponSkrubPP(PostProcessor):
                return None
        return path

+    @PostProcessor._restrict_to(images=False)
    def run(self, information):
        if self.path is None:
            return [], information
--- a/yt_dlp/update.py
+++ b/yt_dlp/update.py
@@ -89,13 +89,9 @@ def run_update(ydl):

    err = None
    if isinstance(globals().get('__loader__'), zipimporter):
-        # We only support python 3.6 or above
-        if sys.version_info < (3, 6):
-            err = 'This is the last release of yt-dlp for Python version %d.%d! Please update to Python 3.6 or above' % sys.version_info[:2]
+        pass
    elif hasattr(sys, 'frozen'):
-        # Python 3.6 supports only vista and above
-        if sys.getwindowsversion()[0] < 6:
-            err = 'This is the last release of yt-dlp for your version of Windows. Please update to Windows Vista or above'
+        pass
    else:
        err = 'It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Please use that to update'
    if err:
--- a/yt_dlp/utils.py
+++ b/yt_dlp/utils.py
@@ -2107,6 +2107,8 @@ def sanitize_filename(s, restricted=False, is_id=False):
            return '_'
        return char

+    if s == '':
+        return ''
    # Handle timestamps
    s = re.sub(r'[0-9]+(?::[0-9]+)+', lambda m: m.group(0).replace(':', '_'), s)
    result = ''.join(map(replace_insane, s))
@@ -2242,6 +2244,17 @@ def unescapeHTML(s):
        r'&([^&;]+;)', lambda m: _htmlentity_transform(m.group(1)), s)


+def escapeHTML(text):
+    return (
+        text
+        .replace('&', '&amp;')
+        .replace('<', '&lt;')
+        .replace('>', '&gt;')
+        .replace('"', '&quot;')
+        .replace("'", '&#39;')
+    )
+
+
 def process_communicate_or_kill(p, *args, **kwargs):
    try:
        return p.communicate(*args, **kwargs)
@@ -2321,13 +2334,14 @@ def decodeOption(optval):
    return optval


-def formatSeconds(secs, delim=':'):
+def formatSeconds(secs, delim=':', msec=False):
    if secs > 3600:
-        return '%d%s%02d%s%02d' % (secs // 3600, delim, (secs % 3600) // 60, delim, secs % 60)
+        ret = '%d%s%02d%s%02d' % (secs // 3600, delim, (secs % 3600) // 60, delim, secs % 60)
    elif secs > 60:
-        return '%d%s%02d' % (secs // 60, delim, secs % 60)
+        ret = '%d%s%02d' % (secs // 60, delim, secs % 60)
    else:
-        return '%d' % secs
+        ret = '%d' % secs
+    return '%s.%03d' % (ret, secs % 1) if msec else ret


 def make_HTTPS_handler(params, **kwargs):
@@ -2490,6 +2504,11 @@ class RejectedVideoReached(YoutubeDLError):
    pass


+class ThrottledDownload(YoutubeDLError):
+    """ Download speed below --throttled-rate. """
+    pass
+
+
 class MaxDownloadsReached(YoutubeDLError):
    """ --max-downloads limit has been reached. """
    pass
@@ -3952,10 +3971,14 @@ class LazyList(collections.Sequence):
    def __init__(self, iterable):
        self.__iterable = iter(iterable)
        self.__cache = []
+        self.__reversed = False

    def __iter__(self):
-        for item in self.__cache:
-            yield item
+        if self.__reversed:
+            # We need to consume the entire iterable to iterate in reverse
+            yield from self.exhaust()[::-1]
+            return
+        yield from self.__cache
        for item in self.__iterable:
            self.__cache.append(item)
            yield item
@@ -3963,29 +3986,39 @@ class LazyList(collections.Sequence):
    def exhaust(self):
        ''' Evaluate the entire iterable '''
        self.__cache.extend(self.__iterable)
+        return self.__cache
+
+    @staticmethod
+    def _reverse_index(x):
+        return -(x + 1)

    def __getitem__(self, idx):
        if isinstance(idx, slice):
            step = idx.step or 1
-            start = idx.start if idx.start is not None else 1 if step > 0 else -1
+            start = idx.start if idx.start is not None else 0 if step > 0 else -1
            stop = idx.stop if idx.stop is not None else -1 if step > 0 else 0
+            if self.__reversed:
+                start, stop, step = map(self._reverse_index, (start, stop, step))
+                idx = slice(start, stop, step)
        elif isinstance(idx, int):
+            if self.__reversed:
+                idx = self._reverse_index(idx)
            start = stop = idx
        else:
            raise TypeError('indices must be integers or slices')
        if start < 0 or stop < 0:
            # We need to consume the entire iterable to be able to slice from the end
            # Obviously, never use this with infinite iterables
-            self.exhaust()
-        else:
-            n = max(start, stop) - len(self.__cache) + 1
-            if n > 0:
-                self.__cache.extend(itertools.islice(self.__iterable, n))
+            return self.exhaust()[idx]
+
+        n = max(start, stop) - len(self.__cache) + 1
+        if n > 0:
+            self.__cache.extend(itertools.islice(self.__iterable, n))
        return self.__cache[idx]

    def __bool__(self):
        try:
-            self[0]
+            self[-1] if self.__reversed else self[0]
        except IndexError:
            return False
        return True
@@ -3994,6 +4027,17 @@ class LazyList(collections.Sequence):
        self.exhaust()
        return len(self.__cache)

+    def __reversed__(self):
+        self.__reversed = not self.__reversed
+        return self
+
+    def __repr__(self):
+        # repr and str should mimic a list. So we exhaust the iterable
+        return repr(self.exhaust())
+
+    def __str__(self):
+        return repr(self.exhaust())
+

 class PagedList(object):
    def __len__(self):
@@ -6202,6 +6246,8 @@ def traverse_obj(obj, keys, *, casesense=True, is_user_input=False, traverse_str
            if is_user_input:
                key = (int_or_none(key) if ':' not in key
                       else slice(*map(int_or_none, key.split(':'))))
+                if key is None:
+                    return None
            if not isinstance(obj, (list, tuple)):
                if traverse_string:
                    obj = compat_str(obj)
--- a/yt_dlp/version.py
+++ b/yt_dlp/version.py
@@ -1,3 +1,3 @@
 from __future__ import unicode_literals

-__version__ = '2021.06.01'
+__version__ = '2021.06.09'
Author	SHA1	Message	Date
pukkandan	6aecd87106	Release 2021.06.23	2021-06-23 07:34:55 +05:30
pukkandan	ed807c1837	Update to ytdl-commit-379f52a [liveleak] Remove extractor `379f52a495`	2021-06-23 07:34:55 +05:30
Mevious	29f63c9672	[funimation] Extract subtitles (#434 ) Closes #420, https://github.com/ytdl-org/youtube-dl/issues/25645 Related: https://github.com/ytdl-org/youtube-dl/pull/24906 Authored by: Mevious	2021-06-23 07:27:53 +05:30
pukkandan	9fc0de5796	[hotstar] Use server time for authentication instead of local time Closes #396	2021-06-23 06:04:42 +05:30
siikamiika	c60ee3a218	[youtube_live_chat] Support ongoing live chat (#422 ) Authored by: siikamiika	2021-06-23 05:42:39 +05:30
pukkandan	8a77e5e6bc	[cleanup] Revert unnecessary changes in `51d9739f80`	2021-06-23 05:34:40 +05:30
pukkandan	51d9739f80	Add option `--throttled-rate` below which video data is re-extracted Currently only for HTTP downloads Closes #430, workaround for https://github.com/ytdl-org/youtube-dl/issues/29326	2021-06-23 05:29:58 +05:30
pukkandan	4c7853de14	[fragment] Merge during download for `-N`, and refactor `hls`/`dash` (#364 )	2021-06-22 00:29:50 +05:30
pukkandan	e6779b9400	[twitcasting] Websocket support (#399 ) Closes #392 Authored by: nao20010128nao	2021-06-21 22:56:45 +05:30
pukkandan	e36d50c5dd	[websockets] Add `WebSocketFragmentFD` (#399 ) Necessary for #392 Co-authored by: nao20010128nao, pukkandan	2021-06-21 22:56:36 +05:30
pukkandan	ff0f78e1fe	[aria2c] Lower `--min-split-size` for HTTP downloads This makes downloading smaller files much faster	2021-06-20 19:28:54 +05:30
pukkandan	7e067091e8	[options] Rename `--add-metadata` to `--embed-metadata` and clarify that it embeds chapter markers	2021-06-20 04:59:35 +05:30
pukkandan	f89b3e2d7a	Skip fixup of existing files and add `--fixup force` to force it	2021-06-20 04:59:34 +05:30
pukkandan	fd7cfb6444	[cleanup] Refactor fixup	2021-06-20 04:26:11 +05:30
pukkandan	4e6767b5f2	[youtube] Temporary fix for age-gate Related: https://stackoverflow.com/a/67629882 https://github.com/yt-dlp/yt-dlp/issues/319 https://github.com/ytdl-org/youtube-dl/issues/29333 https://github.com/ytdl-org/youtube-dl/issues/29086	2021-06-18 20:32:52 +05:30
pukkandan	9fea350f0d	Fix id sanitization in filenames Closes #415	2021-06-17 02:32:24 +05:30
pukkandan	e858a9d6d3	[EmbedThumbnail] Add compat-option `embed-thumbnail-atomicparsley` to force use of atomicparsley for embedding thumbnails in mp4 Related: #411	2021-06-16 22:33:32 +05:30
pukkandan	7e87e27c52	[postprocessor] Fix `_restrict_to` when a codec is not set	2021-06-14 14:09:22 +05:30
pukkandan	d0fb4bd16f	[pornhub] Extract `cast` Closes #406, https://github.com/ytdl-org/youtube-dl/pull/27384	2021-06-13 21:38:08 +05:30
felix	3fd4c2a543	[mediasite] Extract slides (#343 ) Fixes: https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762 https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231 https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239 Authored by: fstirlitz	2021-06-13 20:36:40 +05:30
felix	cdb19aa4c2	[downloader/mhtml] Add new downloader (#343 ) This downloader is intended to be used for streams that consist of a timed sequence of stand-alone images, such as slideshows or thumbnail streams This can be used for implementing: https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762 https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231 https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239 https://github.com/ytdl-org/youtube-dl/issues/9868 https://github.com/ytdl-org/youtube-dl/pull/14951 Authored by: fstirlitz	2021-06-13 20:36:40 +05:30
pukkandan	4d85fbbdbb	Fix bug in `8326b00aab`	2021-06-13 14:36:13 +05:30
pukkandan	551f93885e	Ignore `images` formats from merge	2021-06-13 04:16:42 +05:30
pukkandan	8326b00aab	Allow `images` formats Necessary for #343. * They are identified by `vcodec=acodec='none'` * These formats show as the worst in `-F` * Any postprocessor that expects audio/video will be skipped * `b` and all related selectors will skip such formats This commit also does not add any selector for downloading such formats. They have to be explicitly requested by the `format_id`. Implementation of a selector is left for when #389 is resolved	2021-06-13 03:45:53 +05:30
pukkandan	b0249bcaf0	Expand `--check-formats` to thumbnails Closes #402	2021-06-13 03:45:53 +05:30
pukkandan	21cd8fae49	Use `NamedTemporaryFile` for `--check-formats`	2021-06-13 03:45:53 +05:30
pukkandan	45db527fa6	[youtube] Login is not needed for `:ytrec`	2021-06-13 03:45:53 +05:30
pukkandan	28419ca2c8	[utils] Improve `LazyList` * Add `repr` and `str` that mimics `list` * Add `reversed`. Unlike `[::-1]`, reversed does not exhaust the iterable and modifies the `LazyList` in-place * Add tests	2021-06-13 03:45:53 +05:30
pukkandan	8ba8714880	[EmbedThumbnail] Fix for already downloaded thumbnail	2021-06-11 19:13:24 +05:30
pukkandan	187986a857	Better error handling of syntax errors in `-f`	2021-06-11 19:13:22 +05:30
coletdjnz	4ba001080f	[youtube] Non-fatal alert reporting for unavailable videos page (#401 ) Co-Authored by: colethedj, pukkandan	2021-06-10 21:12:56 +00:00
coletdjnz	1974e99f4b	[youtube] Improve SAPISID cookie handling (closes #393 ) (#395 ) Author: colethedj	2021-06-10 21:02:57 +00:00
pukkandan	0181adefc6	[build] Build Windows x86 version with py3.7 and remove redundant tests Closes #390 :ci skip Co-authored by: pukkandan, shirt-dev	2021-06-10 01:41:04 +05:30
pukkandan	fd3c633d26	[version] update :ci skip all	2021-06-10 01:36:46 +05:30
pukkandan	0d47c278d1	Release 2021.06.09	2021-06-10 00:49:48 +05:30
pukkandan	385a27fad1	Improve offset parsing in outtmpl	2021-06-09 20:01:57 +05:30
pukkandan	5c6542ce69	[test] More rigorous tests for `prepare_filename` All tests of `prepare_outtmpl` is now also run on `prepare_filename`	2021-06-09 20:01:56 +05:30
pukkandan	639f1cea92	Fix `%d` and empty default in outtmpl Closes #388	2021-06-09 15:37:15 +05:30
pukkandan	b5c5d84f60	Revert "[build] Build Windows x86 version with py3.8" This reverts commit `aa75e51f99`. See #390 This is being reverted instead of modified due to #388	2021-06-09 15:37:15 +05:30
pukkandan	aa75e51f99	[build] Build Windows x86 version with py3.8 and remove redundant tests :ci skip Ao-authored by: pukkandan, shirt-dev	2021-06-09 02:18:55 +05:30
pukkandan	884ce9d05d	[version] update :ci skip all	2021-06-09 02:18:55 +05:30